Big delays in processing incoming mail due SA checks

Hi folks,

We recently added 300 domains to two our spamfilters working
in parallel
(DNS load-balance with equal MX record values). The total
number of domains
is 350 and that started bringing them down to its knees
during the day when
we receive the peak of incoming spam. The active mail queue
builds up in
both of them and it introduces quite a delay in mail
processing. I
investigated maillog file and was amazed to find that it
takes up to 80
seconds to process mail. See the excerpt of the maillog
showing the TIMING
equals to 21228 ms.

============================================================
================
===========
Mar 27 22:18:25 mail03 amavis[19642]: (19642-07) SPAM,
<gutierrezbossanova.com> -> <comtoseecomtosee.com>, Yes, hits=25.061 tag=1
tag2=3 kill=5 tests=BAYES_99=3.5, FH_BAD_OEV1441=2.401,
HTML_MESSAGE=0.001,
RAZOR2_CF_RANGE_51_100=0.5, RAZOR2_CF_RANGE_E8_51_100=1.5,
RAZOR2_CHECK=0.5,
RCVD_IN_BL_SPAMCOP_NET=1.96, RCVD_IN_XBL=3.033,
SPAMMY_XMAILER=2.337,
SPF_SOFTFAIL=0.596, URIBL_BLACK=1.955,
URIBL_JP_SURBL=1.501,
URIBL_SBL=1.499, URIBL_WS_SURBL=1.5,
XMAILER_MIMEOLE_OL_72641=2.278,
quarantine spam-1634605 (maia-spam-quarantine)

Mar 27 22:18:25 mail03 amavis[19642]: (19642-07) Blocked
SPAM,
[87.117.27.142] [64.127.117.231] <gutierrezbossanova.com> ->
<comtoseecomtosee.com>, Message-ID:
<616c01c89093$0e7db210$8e1b7557Dante>,
Hits: 25.061, 21224 ms                                      
 
Mar 27 22:18:25 mail03 postfix/smtp[19635]: A032B1AF412:
to=<comtoseecomtosee.com>,
relay=127.0.0.1[127.0.0.1]:10024, delay=24,
delays=2.3/0/0/21, dsn=2.7.1, status=sent (250 2.7.1 Ok,
discarded, UBE,
id=19642-07)                                                
 
Mar 27 22:18:25 mail03 amavis[19642]: (19642-07) TIMING
[total 21228 ms] -
SMTP EHLO: 3 (0%), SMTP pre-MAIL: 1 (0%), lookup_sql: 384
(2%), SMTP
pre-DATA-flush: 2 (0%), SMTP DATA: 99 (0%), body_hash: 1
(0%), maia_connect:
3 (0%), maia_read_system_config: 2 (0%),
maia_get_mysql_size_limit: 1 (0%),
lookup_sql: 3 (0%), mime_decode: 17 (0%), get-file-type2: 25
(0%),
decompose_part: 1 (0%), parts_decode: 0 (0%), AV-scan-1: 14
(0%),
spam-wb-list: 6 (0%), SA msg read: 1 (0%), SA parse: 3 (0%),
SA check: 20556
(97%), update_cache: 2 (0%), maia_store_mail: 10 (0%),
maia_record_tests: 67
(0%), maia_set_mail_status: 5 (0%), post-do_spam: 4 (0%),
deal_with_mail_size: 0 (0%), main_log_entry: 15 (0%),
update_snmp: 1 (0%),
maia_cleanup: 1 (0%), maia_disconnect: 0 (0%),
unlink-2-files: 1 (0%),
rundown: 0 (0%)   
============================================================
================
=================

It turns out that Spamassassin checks take the majority of
time - 20556 ms.
That observation was made during evening time when the load
is not too high.
At the daytime it is higher. So the question to be asked -
what can I do
with SA to minimize the time. I attached two files from SA
configuration
(local.cf and v310.pre) if this could be of any help. 
Hope for any sound thoughts or suggestions. 
Thanks in advance.

Eugene



_______________________________________________
Maia-users mailing list
Maia-usersrenaissoft.com
http://www.renaissoft.com/mailman/listinfo/maia-users

On 3/30/08, Eugene Pefti wrote:
> Hi folks,
>
> We recently added 300 domains to two our spamfilters
working in parallel
> (DNS load-balance with equal MX record values). The
total number of domains
> is 350 and that started bringing them down to its knees
during the day when
> we receive the peak of incoming spam. The active mail
queue builds up in
> both of them and it introduces quite a delay in mail
processing. I
> investigated maillog file and was amazed to find that
it takes up to 80
> seconds to process mail. See the excerpt of the maillog
showing the TIMING
> equals to 21228 ms.
>
>  SA check: 20556 (97%),

Pyzor often ties up the system for 5 seconds. It can help to
use a
mirror for pyzor - or disable pyzor. Check your:
/var/amavis/.pyzor/servers
file (depending on where the home directory of your amavis
user is)
and see if you are using:
82.94.255.100:24441
If not, set it to that and then use:
su amavis -c 'pyzor ping'
to test. It helps quite a bit with RBLs if you have a local
caching
DNS server. You can run some tests to see where SA spends
time:

http://marc.info/?l=amavis-user&m=117874388132138

-- 
Gary V
_______________________________________________
Maia-users mailing list
Maia-usersrenaissoft.com
http://www.renaissoft.com/mailman/listinfo/maia-users

On 3/30/08, Gary V wrote:
> On 3/30/08, Eugene Pefti wrote:
> > Hi folks,
> >
> > We recently added 300 domains to two our
spamfilters working in parallel
> > (DNS load-balance with equal MX record values).
The total number of domains
> > is 350 and that started bringing them down to its
knees during the day when
> > we receive the peak of incoming spam. The active
mail queue builds up in
> > both of them and it introduces quite a delay in
mail processing. I
> > investigated maillog file and was amazed to find
that it takes up to 80
> > seconds to process mail. See the excerpt of the
maillog showing the TIMING
> > equals to 21228 ms.
> >
> >  SA check: 20556 (97%),
>
> Pyzor often ties up the system for 5 seconds. It can
help to use a
> mirror for pyzor - or disable pyzor. Check your:
> /var/amavis/.pyzor/servers
> file (depending on where the home directory of your
amavis user is)
> and see if you are using:
> 82.94.255.100:24441
> If not, set it to that and then use:
> su amavis -c 'pyzor ping'
> to test. It helps quite a bit with RBLs if you have a
local caching
> DNS server. You can run some tests to see where SA
spends time:
>
> http://marc.info/?l=amavis-user&m=117874388132138
>

Additionally, run:
su amavis -c 'razor-admin -discover'
just in case razor servers have changed, but you are not yet
at the
point where automatic rediscovery will occur.

-- 
Gary V
_______________________________________________
Maia-users mailing list
Maia-usersrenaissoft.com
http://www.renaissoft.com/mailman/listinfo/maia-users