Amavisd-new slowness issues

On 3/20/08, Casartello, Thomas wrote:
>
> As my MySQL database has grown, I've been having some
slowness issues with
> Maia and its slowing down email with some email taking
a long time to get
> through. Is there anything I can do to speed things up
with Amavisd-new and
> MySQL settings? My Innodb buffer pool size is currently
1 GB. That's the
> only MySQL tweek I have in there right now.  My amavis
$max_servers = 2. I
> have about 10000 users and it's a pretty busy system.
It's on a box with 4
> GB of memory and twin dual core CPUs. Any help would be
appreciated.
>
> Thomas E. Casartello, Jr.

Of course there are a lot of reasons scanning can be slow,
but it
looks like you mentioned $max_servers because you are
thinking this
may be an issue, and I think you are correct. You need to
increase
concurrency. Scanning introduces a high amount of latency.
You can
compensate by increasing concurrency (up to some limit
where
performance degrades, or you chew up enough RAM to start
swap
thrashing). For your machine I would try setting this at 6,
and see
how that goes, but don't forget that when you change
$max_servers, you
also have to change the maxproc in master.cf for the amavis
transport
to the same number, e.g.:

$max_servers = 6;

smtp-amavis unix -	-	n	-	6  smtp
    -o smtp_data_done_timeout=1200
    -o smtp_send_xforward_command=yes
    -o disable_dns_lookups=yes

Here is a recent post on the amavis list with some links
that may help
pinpoint the issue:

http://marc.info/?l=amavis-user&m=120594774420664
notably:
http://marc.info/?l=amavis-user&m=117874388132138

-- 
Gary V
_______________________________________________
Maia-users mailing list
Maia-usersrenaissoft.com
http://www.renaissoft.com/mailman/listinfo/maia-users

Gary V wrote:
> On 3/20/08, Casartello, Thomas wrote:
>   
>> As my MySQL database has grown, I've been having
some slowness issues with
>> Maia and its slowing down email with some email
taking a long time to get
>> through. Is there anything I can do to speed things
up with Amavisd-new and
>> MySQL settings? My Innodb buffer pool size is
currently 1 GB. That's the
>> only MySQL tweek I have in there right now.  My
amavis $max_servers = 2. I
>> have about 10000 users and it's a pretty busy
system. It's on a box with 4
>> GB of memory and twin dual core CPUs. Any help
would be appreciated.
>>
>> Thomas E. Casartello, Jr.
>>     
>
> Of course there are a lot of reasons scanning can be
slow, but it
> looks like you mentioned $max_servers because you are
thinking this
> may be an issue, and I think you are correct. You need
to increase
> concurrency. Scanning introduces a high amount of
latency. You can
> compensate by increasing concurrency (up to some limit
where
> performance degrades, or you chew up enough RAM to
start swap
> thrashing). For your machine I would try setting this
at 6, and see
> how that goes, but don't forget that when you change
$max_servers, you
> also have to change the maxproc in master.cf for the
amavis transport
> to the same number, e.g.:
>
> $max_servers = 6;
>
> smtp-amavis unix -	-	n	-	6  smtp
>     -o smtp_data_done_timeout=1200
>     -o smtp_send_xforward_command=yes
>     -o disable_dns_lookups=yes
>
> Here is a recent post on the amavis list with some
links that may help
> pinpoint the issue:
>
> http://marc.info/?l=amavis-user&m=120594774420664
> notably:
> http://marc.info/?l=amavis-user&m=117874388132138
>   
Given this is the Maia list,  you mean amavisd-maia, not
amavisd-new, yes?

You mention 10k users, but not anything about mail volume.
Thats a fair 
amount of users to have on one machine, depending on volume,
avg message 
size etc.

Assumptions - you're either not running a greylist process,
or its 
behaving fine, you've worked your MTA setup to eliminate as
much mail at 
the door as possible (verify_recipients, rbls), and have no
network 
problems.

The sequence I would recommend is:
- I agree your $max_servers setting is too low, bump it up
some as Gary 
recommends.
- DNS - running a caching server on the Maia machine?
Depending on your 
plugins and rulesets, there can be quite a lot of dns
traffic - URI 
lookups, Razor/Pyzor/DCC, BL lookups. These can kill SA
times, and a 
local dns server can help allot.
- are any scripts running much longer than expected (12hour

expire-quarantine runs, process-quarantine not completing
regularly)?
- analyze your system to find the bottlenecks. It sounds
like you have 
enough memory and CPU, so I would check disks first. The
details depend 
on your OS.
- mysqld tunings can definitely help - there's been lots of
traffic on 
this list on this topic. But if the disks can't keep up,
nothing else 
will truely solve the problem.
- mysql maintenance - with InnoDB, deleted space is reused,
but not 
reclaimed - if you delete half your database, the file sizes
on disk 
won't change. If you haven't ever, or recently, done a
dump/restore, 
give that a try (I know, late night, not pleasant, also not
one of the 
highest priorities)

If you check the list archives, you'll see I had a serious
issue 
recently and was helped quite allot by people here, but in
the end, my 
issue was all about disk speed, so perhaps I have that on my
mind .

Seriously, disks and network lookups are always going to be
the slowest 
part of the picture.
_______________________________________________
Maia-users mailing list
Maia-usersrenaissoft.com
http://www.renaissoft.com/mailman/listinfo/maia-users

Sorry yes I meant amavisd-maia. The actual major problem I
was having is
that amavis was allowing Spam assassin to scan e-mails about
to 20 MB. I
decreased this to 200 KB. Large emails were causing SA
Timeouts, and
since I only had two in my max servers, it was clogging up
the system.
Downing the value to 200 KB fixed my immediate problem, but
I'll look
into some of the database issues you mentioned.

Thanks a bunch to both you and Gary for responding.

-----Original Message-----
From: maia-users-bouncesrenaissoft.com
[mailto:maia-users-bouncesrenaissoft.com] On Behalf
Of Tim Palmer
Sent: Thursday, March 20, 2008 9:16 PM
Cc: maia-usersrenaissoft.com
Subject: Re: [Maia-users] Amavisd-new slowness issues

Gary V wrote:
> On 3/20/08, Casartello, Thomas wrote:
>   
>> As my MySQL database has grown, I've been having
some slowness issues
with
>> Maia and its slowing down email with some email
taking a long time to
get
>> through. Is there anything I can do to speed things
up with
Amavisd-new and
>> MySQL settings? My Innodb buffer pool size is
currently 1 GB. That's
the
>> only MySQL tweek I have in there right now.  My
amavis $max_servers =
2. I
>> have about 10000 users and it's a pretty busy
system. It's on a box
with 4
>> GB of memory and twin dual core CPUs. Any help
would be appreciated.
>>
>> Thomas E. Casartello, Jr.
>>     
>
> Of course there are a lot of reasons scanning can be
slow, but it
> looks like you mentioned $max_servers because you are
thinking this
> may be an issue, and I think you are correct. You need
to increase
> concurrency. Scanning introduces a high amount of
latency. You can
> compensate by increasing concurrency (up to some limit
where
> performance degrades, or you chew up enough RAM to
start swap
> thrashing). For your machine I would try setting this
at 6, and see
> how that goes, but don't forget that when you change
$max_servers, you
> also have to change the maxproc in master.cf for the
amavis transport
> to the same number, e.g.:
>
> $max_servers = 6;
>
> smtp-amavis unix -	-	n	-	6  smtp
>     -o smtp_data_done_timeout=1200
>     -o smtp_send_xforward_command=yes
>     -o disable_dns_lookups=yes
>
> Here is a recent post on the amavis list with some
links that may help
> pinpoint the issue:
>
> http://marc.info/?l=amavis-user&m=120594774420664
> notably:
> http://marc.info/?l=amavis-user&m=117874388132138
>   
Given this is the Maia list,  you mean amavisd-maia, not
amavisd-new,
yes?

You mention 10k users, but not anything about mail volume.
Thats a fair 
amount of users to have on one machine, depending on volume,
avg message

size etc.

Assumptions - you're either not running a greylist process,
or its 
behaving fine, you've worked your MTA setup to eliminate as
much mail at

the door as possible (verify_recipients, rbls), and have no
network 
problems.

The sequence I would recommend is:
- I agree your $max_servers setting is too low, bump it up
some as Gary 
recommends.
- DNS - running a caching server on the Maia machine?
Depending on your 
plugins and rulesets, there can be quite a lot of dns
traffic - URI 
lookups, Razor/Pyzor/DCC, BL lookups. These can kill SA
times, and a 
local dns server can help allot.
- are any scripts running much longer than expected (12hour

expire-quarantine runs, process-quarantine not completing
regularly)?
- analyze your system to find the bottlenecks. It sounds
like you have 
enough memory and CPU, so I would check disks first. The
details depend 
on your OS.
- mysqld tunings can definitely help - there's been lots of
traffic on 
this list on this topic. But if the disks can't keep up,
nothing else 
will truely solve the problem.
- mysql maintenance - with InnoDB, deleted space is reused,
but not 
reclaimed - if you delete half your database, the file sizes
on disk 
won't change. If you haven't ever, or recently, done a
dump/restore, 
give that a try (I know, late night, not pleasant, also not
one of the 
highest priorities)

If you check the list archives, you'll see I had a serious
issue 
recently and was helped quite allot by people here, but in
the end, my 
issue was all about disk speed, so perhaps I have that on my
mind .

Seriously, disks and network lookups are always going to be
the slowest 
part of the picture.
_______________________________________________
Maia-users mailing list
Maia-usersrenaissoft.com
http://www.renaissoft.com/mailman/listinfo/maia-users
_______________________________________________
Maia-users mailing list
Maia-usersrenaissoft.com
http://www.renaissoft.com/mailman/listinfo/maia-users

(my only real religion about posting top/bottom is that is
should be 
consistent. Since I bottom posted originally, please see my
response on 
the bottom)

Casartello, Thomas wrote:
> Sorry yes I meant amavisd-maia. The actual major
problem I was having is
> that amavis was allowing Spam assassin to scan e-mails
about to 20 MB. I
> decreased this to 200 KB. Large emails were causing SA
Timeouts, and
> since I only had two in my max servers, it was clogging
up the system.
> Downing the value to 200 KB fixed my immediate problem,
but I'll look
> into some of the database issues you mentioned.
>
> Thanks a bunch to both you and Gary for responding.
>
> -----Original Message-----
> From: maia-users-bouncesrenaissoft.com
> [mailto:maia-users-bouncesrenaissoft.com] On Behalf
Of Tim Palmer
> Sent: Thursday, March 20, 2008 9:16 PM
> Cc: maia-usersrenaissoft.com
> Subject: Re: [Maia-users] Amavisd-new slowness issues
>
> Gary V wrote:
>   
>> On 3/20/08, Casartello, Thomas wrote:
>>   
>>     
>>> As my MySQL database has grown, I've been
having some slowness issues
>>>       
> with
>   
>>> Maia and its slowing down email with some email
taking a long time to
>>>       
> get
>   
>>> through. Is there anything I can do to speed
things up with
>>>       
> Amavisd-new and
>   
>>> MySQL settings? My Innodb buffer pool size is
currently 1 GB. That's
>>>       
> the
>   
>>> only MySQL tweek I have in there right now.  My
amavis $max_servers =
>>>       
> 2. I
>   
>>> have about 10000 users and it's a pretty busy
system. It's on a box
>>>       
> with 4
>   
>>> GB of memory and twin dual core CPUs. Any help
would be appreciated.
>>>
>>> Thomas E. Casartello, Jr.
>>>     
>>>       
>> Of course there are a lot of reasons scanning can
be slow, but it
>> looks like you mentioned $max_servers because you
are thinking this
>> may be an issue, and I think you are correct. You
need to increase
>> concurrency. Scanning introduces a high amount of
latency. You can
>> compensate by increasing concurrency (up to some
limit where
>> performance degrades, or you chew up enough RAM to
start swap
>> thrashing). For your machine I would try setting
this at 6, and see
>> how that goes, but don't forget that when you
change $max_servers, you
>> also have to change the maxproc in master.cf for
the amavis transport
>> to the same number, e.g.:
>>
>> $max_servers = 6;
>>
>> smtp-amavis unix -	-	n	-	6  smtp
>>     -o smtp_data_done_timeout=1200
>>     -o smtp_send_xforward_command=yes
>>     -o disable_dns_lookups=yes
>>
>> Here is a recent post on the amavis list with some
links that may help
>> pinpoint the issue:
>>
>> http://marc.info/?l=amavis-user&m=120594774420664
>> notably:
>> http://marc.info/?l=amavis-user&m=117874388132138
>>   
>>     
> Given this is the Maia list,  you mean amavisd-maia,
not amavisd-new,
> yes?
>
> You mention 10k users, but not anything about mail
volume. Thats a fair 
> amount of users to have on one machine, depending on
volume, avg message
>
> size etc.
>
> Assumptions - you're either not running a greylist
process, or its 
> behaving fine, you've worked your MTA setup to
eliminate as much mail at
>
> the door as possible (verify_recipients, rbls), and
have no network 
> problems.
>
> The sequence I would recommend is:
> - I agree your $max_servers setting is too low, bump it
up some as Gary 
> recommends.
> - DNS - running a caching server on the Maia machine?
Depending on your 
> plugins and rulesets, there can be quite a lot of dns
traffic - URI 
> lookups, Razor/Pyzor/DCC, BL lookups. These can kill SA
times, and a 
> local dns server can help allot.
> - are any scripts running much longer than expected
(12hour 
> expire-quarantine runs, process-quarantine not
completing regularly)?
> - analyze your system to find the bottlenecks. It
sounds like you have 
> enough memory and CPU, so I would check disks first.
The details depend 
> on your OS.
> - mysqld tunings can definitely help - there's been
lots of traffic on 
> this list on this topic. But if the disks can't keep
up, nothing else 
> will truely solve the problem.
> - mysql maintenance - with InnoDB, deleted space is
reused, but not 
> reclaimed - if you delete half your database, the file
sizes on disk 
> won't change. If you haven't ever, or recently, done a
dump/restore, 
> give that a try (I know, late night, not pleasant, also
not one of the 
> highest priorities)
>
> If you check the list archives, you'll see I had a
serious issue 
> recently and was helped quite allot by people here, but
in the end, my 
> issue was all about disk speed, so perhaps I have that
on my mind . 
> Seriously, disks and network lookups are always going
to be the slowest 
> part of the picture.
> _______________________________________________
> Maia-users mailing list
> Maia-usersrenaissoft.com
> http://www.renaissoft.com/mailman/listinfo/maia-users
>
>   

Well, yes, that would be a problem, and well caught . While
I've heard 
spam is normally under 100k in size, I tend to scan to 512k.
Definitely 
think about increasing your $max_servers and don't forget
those disks!
_______________________________________________
Maia-users mailing list
Maia-usersrenaissoft.com
http://www.renaissoft.com/mailman/listinfo/maia-users

On Thu, 2008-03-20 at 21:15 -0400, Tim Palmer wrote:
>  If you haven't ever, or recently, done a dump/restore,

> give that a try (I know, late night, not pleasant, also
not one of the 
> highest priorities)

Does anybody out there actually have a machine and a setup
that would be
loaded enough to be having problems, but where a dump and
restore is
actually a reasonable thing to do? I ask this because, when
I tried it,
it literally took more than 40 hours to run, and this on a
dual
processor Xeon 64-bit system with hardware RAID disks and
4GB of RAM.
It's not just a late night, it's a weekend-long e-mail
outage. We only
have 1200 local users with perhaps 6000 aliases, and our
stack of
scanning boxes processes about 6.5 million messages per
month.

--Greg


_______________________________________________
Maia-users mailing list
Maia-usersrenaissoft.com
http://www.renaissoft.com/mailman/listinfo/maia-users

Greg Woods wrote:
> On Thu, 2008-03-20 at 21:15 -0400, Tim Palmer wrote:
>   
>>  If you haven't ever, or recently, done a
dump/restore, 
>> give that a try (I know, late night, not pleasant,
also not one of the 
>> highest priorities)
>>     
>
> Does anybody out there actually have a machine and a
setup that would be
> loaded enough to be having problems, but where a dump
and restore is
> actually a reasonable thing to do? I ask this because,
when I tried it,
> it literally took more than 40 hours to run, and this
on a dual
> processor Xeon 64-bit system with hardware RAID disks
and 4GB of RAM.
> It's not just a late night, it's a weekend-long e-mail
outage. We only
> have 1200 local users with perhaps 6000 aliases, and
our stack of
> scanning boxes processes about 6.5 million messages per
month.
>
> --Greg
>
>
>   
We did a 25G dump/restore in under 4 hours recently. I've
also dumped 
much smaller (5G) maia databases in very little time. You
don't mention 
how big your dump was, but 40 hours definitely sounds
excessive.

I don't think its needs doing very often - I almost didn't
mention it, 
and am still not sure I should have in this case.
_______________________________________________
Maia-users mailing list
Maia-usersrenaissoft.com
http://www.renaissoft.com/mailman/listinfo/maia-users

Hi Greg,

For speed up your mysql restore, add the next two lines in
the beginning of
your mysql_restore.sql file.

SET AUTOCOMMIT = 0;
SET FOREIGN_KEY_CHECKS=0;

After this insert also the next two lines at the end of
your
mysql_restore.sql file

SET FOREIGN_KEY_CHECKS = 1;
COMMIT;

This should speed up the restore.


Best Regards,

João Carmona

-----Original Message-----
From: maia-users-bouncesrenaissoft.com
[mailto:maia-users-bouncesrenaissoft.com] On Behalf
Of Greg Woods
Sent: sexta-feira, 21 de Março de 2008 16:09
To: Tim Palmer
Cc: maia-usersrenaissoft.com
Subject: Re: [Maia-users] Amavisd-new slowness issues


On Thu, 2008-03-20 at 21:15 -0400, Tim Palmer wrote:
>  If you haven't ever, or recently, done a dump/restore,

> give that a try (I know, late night, not pleasant, also
not one of the 
> highest priorities)

Does anybody out there actually have a machine and a setup
that would be
loaded enough to be having problems, but where a dump and
restore is
actually a reasonable thing to do? I ask this because, when
I tried it,
it literally took more than 40 hours to run, and this on a
dual
processor Xeon 64-bit system with hardware RAID disks and
4GB of RAM.
It's not just a late night, it's a weekend-long e-mail
outage. We only
have 1200 local users with perhaps 6000 aliases, and our
stack of
scanning boxes processes about 6.5 million messages per
month.

--Greg


_______________________________________________
Maia-users mailing list
Maia-usersrenaissoft.com
http://www.renaissoft.com/mailman/listinfo/maia-users

_______________________________________________
Maia-users mailing list
Maia-usersrenaissoft.com
http://www.renaissoft.com/mailman/listinfo/maia-users