MailStats

I wrote this script during swotvac to parse my sendmail logs and report on the number of emails processed and spams tagged during the preceding day. I'd just implemented SPF, so I was interested in determining how widely adopted it is. It would also be interesting to count how much ham and spam would have been reject by SPF - I may add that one day.

It uses a patched version of logtail, but works with the normal one if you don't care about rotation issues.

#!/usr/bin/perl -w
# Check maillogs and generate nice spam summary statistics

use IO::File;

# Variables

my $LOGTAIL="/root/logtailng.pl";
my $LOGTAILSTATE="/root/maillog.state";
my $LOGFILE="/var/log/mail/mail.log";
my $LOCALDOMAIN="(localhost|tristesse\.org)"; # Assume these are local

# Things we want to look out for

my @userunknown;

my $spffail = 0;
my $spfnone = 0;
my $spfpass = 0;
my $spfneut = 0;
my $acceptedcount = 0;
my $sentcount = 0;
my $spamcount = 0;

# Code goes here

my $fd = new IO::File;

$fd->open("$LOGTAIL -f $LOGFILE -o $LOGTAILSTATE |");

while (<$fd>) {
       if (/User unknown/) {
               my ($user) = /\<([^>]+)>/;
               push @userunknown, $user;
               next;
       }

       if (/identified spam/) {
               $spamcount++;
       }

       if (/stat=Sent/ && / sm-mta/) {
               # Locally sent mail will also produce a sendmail line.
               # Ignore duplicates.
               if (/to=[<]*[^\@]+\@$LOCALDOMAIN[>]*/) {
                       $acceptedcount++;
               } elsif (/to=[<]*[^\@]+[>]*[,\s]+/) {
                       # Match things like to=joel, to=root
                       $acceptedcount++;
               } else {
                       $sentcount++;
               }
       }

       if (/Received-SPF: (\w+) /) {

               # Sending locally triggers spfmilter. Filter it.
               next if /localhost is always allowed/;

               my $spfstat = $1;
               if ($spfstat eq "none") {
                       $spfnone++;
               } elsif ($spfstat eq "softfail") {
                       $spffail++;
               } elsif ($spfstat eq "pass") {
                       $spfpass++;
               } elsif ($spfstat eq "neutral") {
                       $spfneut++;
               }
       }

}

$fd->close;

my $spampercent = 0;
$spampercent = sprintf("%.3f", $spamcount/$acceptedcount * 100) unless !$acceptedcount;

print <<EOF
In summary:

Mail accepted $acceptedcount
Mail sent     $sentcount
Spam tagged   $spamcount
Spam %        $spampercent

SPF Failures: $spffail
SPF No record $spfnone
SPF Passes    $spfpass
SPF Neutral   $spfneut

Unknown users $#userunknown

EOF
;

Here is a "spam for today" script that reads Procmail logfiles. It is incredibly evil. You will need to customise it, so don't be scared! - WzDD

#!/bin/sh

TODAY=`date +"%b %d"`
TODAY=$TODAY.\*`date +"%Y"`

TOTAL=`grep "$TODAY" ~/.procmail-logfile |wc -l`
SPAM=`grep -A 2 "$TODAY" .procmail-logfile |grep "Folder: spam"|wc -l`
PERCENT=`python -c "print '%2.1f' % (float($SPAM)*100/$TOTAL)"`
echo Received $TOTAL in total, $SPAM spam \($PERCENT% spam\)