musings on performance

Maybe maia-users isn't the right place to post this, but I
just have
tremendous pressure building to somehow get the thoughts
I've been
having available to somebody who might care, or especially
might
point out what stupid mistake I've made.

First, let me list a few thoughts about how I used to do my
own
personal spam filtering.

I first thought it might be worthwhile to do any spam
filtering at
all when there was mention on slashdot of Paul Graham's
ideas about
using Bayesian statistics.  My saved lynx bookmark says that
was
August 16, 2002.  There was another slashdot article on
November 3,
2002 about Graham's POPFile, which used Perl to run a
localhost POP
server that had filtered messages from the ISP mail server. 
I guess
for this audience we don't need a glossary to say what POP,
Perl and
ISP mean???  I didn't use POP, I used a Unix mbox, so I
never used
Graham's software directly myself.  According to the
slashdot bookmarks
around it, I first went to sourceforge for Eric S. Raymond's
bogofilter
between November 22 and 24, 2002.  I have about 10
spam-filter-related
bookmarks in that two-day period.

So I started using bogofilter in late 2002, and it was still
my sole
filter as of May 2006.  It got about 2 or 3 false positives
per year
and 2 or 3 false negatives per day over that interval, for a
mail load
of a few hundred ham messages per day, and the spam rate
went from
about 60 per day in late 2002 to nearly 300 per day in
August through
November 2004 (a peak 50% higher than almost any other
month) and
stayed over 100 a day until late 2005 and tapered down to
about 80
per day into 2006.  In May 2006 I installed spamassassin on
my desktop
Gentoo Linux and modified my filter script to run SA on
whatever
bogofilter considered ham, and that got the false negatives
down
to 4 or 5 per week.  I thought that was pretty good.

By "my filter script" I mean I actually
incorporated all messages
into my MH (Mail Handler, Marshall Rose's Rand Corporation
mail
client system) inbox.  Then I would run a script that would
run
bogofilter on each inbox message and refile the spam to a
spam
sub-sub-folder (sub-folder covers a year, sub-sub-folder
covers
a month.)  And manually refile whatever spams remained there
also,
and retrieve the occasional ham from it back to inbox.

I noticed that when a message went to SA, that SA took a lot
longer
to analyze it than bogofilter did.  Maybe 2 or 3 seconds
instead of
2 or 3 messages per second.  This on a 2 GHz AMD XP 3500 --
probably
disk speed didn't matter much, but it is an IDE, so that's
relatively
slow in modern disk technology.

And since I hadn't found Maia and all the terrific advice
from Robert
and David, and others, I didn't know to use sa-learn daily
to update
SA rules, or even what rules SA was using (it's just
whatever gets
installed, and maybe updated if gentoo "emerge
world" happens to
have a new SA with updated rules when I do that at 6-12 week
intervals.)
The SARE rules from openprotect seem to do a great job at
eliminating
those last few false negatives.  And using Bayesian with SA
helps
quite a bit, too.  Although, if SA didn't even see what the
bogofilter
Bayesian could pre-filter, wow, lots of time would be saved
poring
through rules that really didn't matter anyway.  Well, at
least that's
what I'm thinking.

In August 2006 I started using the postfix-amavis-SA-clamav
suite for
other users, and I knew they were going to take more
resources than the
700 MHz AMD K7 with 40 GB IDE drive computer I was using for
that e-mail
system could provide.  I thought a 3+ GHz 64-bit Pentium
with SATA
would be more than adequate, though.  And, without Maia, it
was, as
slow as SA is.  However, I still hadn't discovered sa-learn
or
openprotect and SARE, or how to make Bayesian and AWL do
anything
intelligent, so the efficiency was still horrid.  It was
doing
quarantining, but there were no utilities to show what was
quarantined for each user, or to allow users to release
false
positives from the quarantine, or to report anything to
spamcop,
spamhaus, or anywhere.  It did look like a good SA threshold
for
most users was about 2.9, although, unlike bogofilter, that
still
had way too high a rate of false positives.  But the default
5.0
had even more false negatives.  There wasn't really a
performance
problem, though, on the new hardware.  I had considered
getting
dual processors and using mirroring for reliability with two
SATA
drives, but not striping for performance, but the system
priced
out triple the one I chose.

Because of the quarantine problem, I began looking for what
I'd call
a "quarantine manager."  And pretty much the only
one I found at all
was Maia Mailguard.  That was about November 2006.  I looked
for about
6 weeks before I found Maia.  And the way Maia is supposed
to work
seems to me just about ideal, although the users still find
it more
complex and confusing than they'd like.  But the problem is,
what
happened to the speed?

Sigh.

It was snappy for awhile.  Even after a few thousand
messages had
passed through, although it was getting a little sluggish,
it still
brought up web pages in under 5 seconds, and the postfix
queue was
never more than a handful of messages.  But by the time it
had gone
through a few tens of thousands of messages, although the
current
store was still under 2,000 messages, the web pages were
taking
30 seconds and up.  Now that it's pushing 80,000 messages,
hardly
any web page ever comes up in less than a minute, the logs
are
getting TIMED OUT messages, the queue has frequently been
dozens
of messages dating back an hour or more, and once was about
1,200
that were up to 36 hours since initial delivery.  That time
I had
to manually remove files from the /var/spool/postfix/*
directories,
it was just spiraling into oblivion.

When I run psql and do simple select statements, what should
be
practically instant response is 30 seconds and up.  The
"ms" entries
in the log are up to several minutes, very rarely only 10
seconds
or so.  The "top" page says postgres is in biowai,
perl is in biowai.
When I use the web interface to confirm/report/whatever, if
I restart
postgres first, it can be pretty snappy, until the cron perl
scripts
kick in, then it's deadly slow again.  And restarting
postgres just
creates a backlog of incoming messages to process, too, so
it's not
really a good thing to do.  It seems to corrupt the data,
too, making
entries in the spam/ham lists show up with nothing but
"no subject"
and the welcome page count of spam disagrees with the actual
list
(it says there are 35, but clicking on the report button
just drops
back to the welcome page because it can't find any to list,
for
instance.)  I used the rc.shutdown script to stop postgres,
and
the rc.local one to start it back up, yet something bad
happened,
it appears, though not really causing any problem more
serious than
counts that disagree, as far as I know.

So, I'm thinking, there's something wrong here.  This
computer should
be about to handle thousands of messages per day, but 950 or
so is
just crippling it to the point where nobody would tolerate
how slow
it responds.  Oh, "top" also says the CPU idle is
90% or more, though
the load is sometimes around 4.  I don't see how
multi-processors
would make this better.  Maybe it's I/O bound.  But, then,
"top" says
I have 400 MB of free RAM, and I've tried to raise some of
the small
postgres server defaults so postgres will get smarter about
caching
data in RAM instead of disk, but it just keeps getting
slower.
There's very little more total I/O required than without
Maia, certainly
not double.  Maybe using sa-learn to add new rules has
increased SA
processing time somewhat, but going from a few seconds to
several
minutes of wall clock time per message seems outrageous to
me.

I've tried to read up on postgres tuning.  I did discover
some things
like raising the default 1024 in work_mem (I'm trying 16384
currently)
and 16384 maintenance_work_mem (131072.)  And the shm and
sem kernel
sizes, and using ipcs to check what's happening there.  I
enabled
the stat lines in /var/postgresql/data/postgresql.conf but I
haven't
seen much about how to see what the stats are.  I did find a
perl
script named pgtop (not easily, since "i /pgtop/"
in my cpan doesn't
find it at cpan.utah.edu, but it is at cpan.pair.com.)  But
that
doesn't tell me much, either.  I think there are counts of
operations
and maybe something about timing in there, but I have no
clue how to
ferret it out.  In phpPgAdmin, I see only "public"
functions, but in
psql, if I do "df" I see lots of pg_catalog
(that's a schema) functions,
including many that begin "pg_stat_" but I have no
clue how to use
those.  Maybe if I study the pgtop source, there's a clue
there.
The postgresql manual seems murky, although it has several
pages
about stats.

I'm still using Berkeley DB for SA Bayesian and AWL, because
on
the system where I tried moving them to postgres, the
system
actually got *slower* (a *lot* *slower*) and I had to
revert
to Berkeley DB so it would keep up with incoming messages. 
Of
course, on that system, there's no free RAM and it's using
as
much swap as RAM, so maybe the experience would be opposite
on the faster system.  I'm just plain afraid to try it! 
Plain
old fear.

So I'm kind of stymied again.  This is not going to hack it
for
production, though, that's obvious, and I'm thinking it's
not because
I don't have a 4-way CPU and 4-way stripe on my disks.  You
guys
are talking about heartbeat and DRDB for redundancy, and I'm
seeing
horrendous resource consumption.  Why are we not only not on
the
same page, we're not in the same book, maybe not the same
library??

I think it would be faster to do the bogofilter first, SA
only on
what bogofilter thinks is ham, though I'm not sure how that
would
translate into scores that SA itemizes so well.  SA in C
instead of
Perl might help by a factor of 10 or more.  The Maia scripts
in
C instead of Perl might be somewhat faster.  The Maia web
site
in C instead of PHP would be faster, but maybe that's
dwarfed by
the rest of the slowness.  But somehow there's resource
consumption
here that just is orders of magnitude worse than it should
be.
Maybe I just haven't found the right postgres conf item to
tweak,
but it shouldn't be this difficult to find how to make it
faster,
and it just shouldn't be this slow anyway.

I think postgres is probably at the heart of this, though I
may
be wrong, since I don't really have anything but
"top" and how
long "select ..." in psql takes to judge postgres'
performance.
And it could be that the perl scripts are dominating the
I/O
system so much that postgres can't get its foot in the door
(although I'd think those scripts use postgres quite a lot,
too.)

This reminds me of both Java and bittorrent.  I've tried
both, a little.
They're both way too slow to use for anything useful, in my
opinion,
although I admit I haven't tested them all that thoroughly. 
I do see
that plenty of other people do share my opinion of Java,
however.  Yet
lots of people swear by them.  Are those people just finding
some magic
potion I never see, are they tolerant of abysmal
performance, am I
particularly stupid, what's going on here?

_______________________________________________
Maia-users mailing list
Maia-usersrenaissoft.com
http://www.renaissoft.com/mailman/listinfo/maia-users

musings on performance

Thread: musings on performance