Re: process-quarantine.pl CPU deadlock

Robert,

Thank you for your detailed response.  After implimenting
all your suggested 
changes, the issue still remains.  I've determined it's not
a result of 
large emails, too many messages, ect...  It's basically SA
hanging on 
certain emails and sitting there in an infinate loop.  I
enabled "all" debug 
logging on process-quarantine-sub.pl and this is a snippet
of what I get 
every time it's ran by hand.  

It's hanging *indefinately* on "decoding
quoted-printable" while using up 
all available CPU.

I've made sure perl-MIME-tools is fully up to date.

For now, I've had to totally disable the processing of
quarantines to keep 
the loop from reoccuring.

<---snip--->

[25903] dbg: conf: finish parsing
[25903] dbg: plugin:
Mail::SpamAssassin::Plugin::ReplaceTags=HASH(0x9212554) 
implements 'finish_parsing_end', priority 0
[25903] dbg: replacetags: replacing tags
[25903] dbg: replacetags: done replacing tags
[25903] dbg: bayes: using username: amavis
[25903] dbg: bayes: database connection established
[25903] dbg: bayes: found bayes db version 3
[25903] dbg: bayes: Using userid: 1
[25903] dbg: config: score set 3 chosen.
[25903] dbg: message: main message type:
multipart/alternative
[25903] dbg: learn: learning spam
[25903] dbg: plugin:
Mail::SpamAssassin::Plugin::WLBLEval=HASH(0x9399c68) 
implements 'check_wb_list', priority 0
[25903] dbg: eval: all '*From' addrs: tamalbarringtonhomesinc.com
[25903] dbg: eval: all '*To' addrs: salesfastserv.com
[25903] dbg: conf: trusted_networks are not configured; it
is recommended 
that you configure trusted_networks manually
[25903] dbg: received-header: parsed as [ ip=84.54.253.90
rdns=n19s253-
90.helios.ru helo=n19s253-90.helios.ru
by=mailguard.ash01.fast-serv.com 
ident= envfrom= intl=0 id=371A58800AD auth= msa=0 ]
[25903] dbg: received-header: do not trust any hosts from
here on
[25903] dbg: received-header: relay 84.54.253.90 trusted? no
internal? no 
msa? no
[25903] dbg: metadata: X-Spam-Relays-Trusted:
[25903] dbg: metadata: X-Spam-Relays-Untrusted: [
ip=84.54.253.90 
rdns=n19s253-90.helios.ru helo=n19s253-90.helios.ru
by=mailguard.ash01.fast-
serv.com ident= envfrom= intl=0 id=371A58800AD auth= msa=0
]
[25903] dbg: metadata: X-Spam-Relays-Internal:
[25903] dbg: metadata: X-Spam-Relays-External: [
ip=84.54.253.90 
rdns=n19s253-90.helios.ru helo=n19s253-90.helios.ru
by=mailguard.ash01.fast-
serv.com ident= envfrom= intl=0 id=371A58800AD auth= msa=0
]
[25903] dbg: message: ---- MIME PARSER START ----
[25903] dbg: message: parsing multipart, got boundary: ----
=_NextPart_000_0004_01C88F13.03795243
[25903] dbg: message: found part of type text/plain,
boundary: ----
=_NextPart_000_0004_01C88F13.03795243
[25903] dbg: message: added part, type: text/plain
[25903] dbg: message: found part of type text/html,
boundary: ----
=_NextPart_000_0004_01C88F13.03795243
[25903] dbg: message: added part, type: text/html
[25903] dbg: message: parsing normal part
[25903] dbg: message: parsing normal part
[25903] dbg: message: ---- MIME PARSER END ----
[25903] dbg: message: decoding quoted-printable
[25903] dbg: message: decoding quoted-printable

...and here it sits for hours and hours.

-----

> That complaint message isn't telling you that you need
to take any
> action, it's just telling you that it tried to start a
second 
> instance of the process-quarantine script and then
aborted.  There 
> shouldn't be anything to "kill off" in that
scenario, since the 
> initial process should be writing a lockfile that
prevents other 
> instances from running until the first one finishes. 
In other words,
>  if you have process-quarantine running hourly and the
job takes 
> more than an hour, you should see a notice in the logs
to the effect 
> of, "I tried to start on schedule, but there was
another process 
> running, so I aborted."  No manual intervention on
your part is 
> necessary in that situation, you shouldn't have to
"kill off" 
> anything--the new process should have aborted itself
when it saw the 
> existing lockfile.
> 
> On a very busy site, in fact, it's possible for
process-quarantine to
> run almost continuously, because at the end of every
process-quarantine
> run it checks to see if there are any new items waiting
to be
> processed--items that were confirmed/reported since the
last run
> started.  Only when there are no more items left to
process will the
> script terminate.  As such, it's safe to run the script
on an hourly
> schedule, even if it takes more than an hour to
run--the lockfiles
> prevent concurrency problems.
> 
> As for what's eating up all of your CPU during those
runs, my guess
> would be that some of those mail items happen to be
quite large, and
> that you're trying to process too many of them in each
pass.  In your
> /etc/maia.conf file, try reducing the $default_limit
setting to
> something small, e.g. 5 items per pass (and certainly
nothing larger
> than 20).  The effect this has is to cause the
process-quarantine script
> to purge its memory buffers after a small number of
items have been
> processed, so that perl doesn't continue to allocate
new memory ad
> infinitum.  Perl's garbage-collection leaves a lot to
be desired, so 
> if you've got thousands of items to process you're
better off doing 
> so in small batches rather than big ones.
> 
> Another setting you'll want to tweak in /etc/maia.conf
is the
> $default_max_size setting, which should match the
> $sa_mail_body_size_limit you set in /etc/amavisd.conf. 
This will
> prevent process-quarantine from trying to process items
larger than a
> certain size, so that you don't end up wasting a lot of
extra cycles
> churning through the guts of a 350 MB AVI file or
something looking for
> usable Bayes tokens.  The default setting of 256 kB
should be fine; not
> much spam is larger than that these days, so you won't
miss out on many
> useful tokens by skipping larger items.
> 
> - --
> Robert LeBlanc <rjlrenaissoft.com>
> Renaissoft, Inc.
> Maia Mailguard <http://www.maiamail
guard.com/>

_______________________________________________
Maia-users mailing list
Maia-usersrenaissoft.com
http://www.renaissoft.com/mailman/listinfo/maia-users

Re: process-quarantine.pl CPU deadlock

Thread: Re: process-quarantine.pl CPU deadlock