Postfix + Amavisd-new + Amavisd-milter

Hi,

I am using Postfix and am experimenting with the inline
rejection of
virus mails using amavisd-new combined with amavisd-milter.

The reason for using the milter interface instead of
smtpd_proxy_filter
is that if you do inline rejection with smtpd_proxy_filter,
you need to
have as many amavisd processes as you have smtpd processes,
which is
really a problem. You can't also just limit drastically the
number of
concurrent smtpd processes because then they could be all
blocked by
slow clients. I guess that this is the biggest reason why
Wietse
does not recommend doing it for big sites.

Combining amavisd-new with amavisd-milter could however
solve this
problem: there is only one small multi-threaded process that
talks to
each of the smtpd processes and a connection is done to
amavisd only
when all the data is ready. It means that you can have 100
smtpd
processes and 4 backend amavisd processes... Also, note that
you can't
combine smtpd_proxy_filter with milters doing content
filtering, which
puts you in a position to choose between the two.

There are however two small problems that I encountered:

1. amavisd-milter needs to be patched to work well with
Postfix, because
   Postfix does give information about the queue-number only
after the
   RCPT-TO-phase. I have attached a patch for that.

2. amavisd-milter requires to use Unix domain sockets to
communicate
   with amavisd. This is not really a problem except that
when I was
   stress-testing my test setup with 10 backend amavisd
processes, I
   noticed that, once in a while, amavisd-milter was getting
a
   "Connection refused" on the socket.

   The problem is that even though I properly limited the
number of
   amavisd connections in amavisd-milter, amavisd does have
a listen
   backlog (queuing of connecting clients) of maximum 5
clients. That
   is, if I have configured to use at most 10 amavisd
processes and by
   chance 6 amavisd-milter processes try to establish a
connection
   simultaneously, it will fail.

   That's why, it would be useful to have in amavisd a
variable called
   $listen_backlog or something like that, which would allow
you to
   specify the queue size for the listen sockets. amavisd
would then
   just need to pass this as 'listen' parameter to
Net::Server...

   Note that this is also not really an issue with inet
sockets because
   of the way connections to inet sockets are handled: if
there is no
   free space in the connection queue, the SYN packet is
just dropped
   so that the client comes back later. With unix sockets,
it is not
   possible to do that so the client gets immediately a
connection
   refused.

By the way: I am also getting better performance with
amavisd-milter
than with smtpd_proxy_filter: about 20% more mails per
second.

Cheers
David

------------------------------------------------------------
-------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and
a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
AMaViS-user mailing list
AMaViS-userlists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user

AMaViS-FAQ:http://www.amav
is.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/ho
wto/

The mailing-list doesn't seem to like attachments so here is
the patch
for amavisd-milter inline:


--- amavisd-milter/mlfi.c.orig	2007-07-27 14:02:47.068396000
+0200
+++ amavisd-milter/mlfi.c	2007-07-27 14:02:53.058402000
+0200
 -929,6
+929,18 
 	return SMFIS_TEMPFAIL;
     }
 
+    /* Get queue id (Postfix) */
+    if (mlfi->mlfi_qid == NULL) {
+	const char *qid;
+        if ((qid = smfi_getsymval(ctx, "i")) !=
NULL) {
+            if ((mlfi->mlfi_qid = strdup(qid)) == NULL)
{
+                logqidmsg(mlfi, LOG_ERR, "could not
allocate memory");
+                mlfi_setreply_tempfail(ctx);
+                return SMFIS_TEMPFAIL;
+            }
+        }
+    }
+
     /* MTA queue id */
     if (mlfi->mlfi_qid != NULL) {
 	logqidmsg(mlfi, LOG_DEBUG, "queue_id=%s",
mlfi->mlfi_qid);

------------------------------------------------------------
-------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and
a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
AMaViS-user mailing list
AMaViS-userlists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user

AMaViS-FAQ:http://www.amav
is.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/ho
wto/

David,

> 2. amavisd-milter requires to use Unix domain sockets
to communicate
>    with amavisd. This is not really a problem except
that when I was
>    stress-testing my test setup with 10 backend amavisd
processes, I
>    noticed that, once in a while, amavisd-milter was
getting a
>    "Connection refused" on the socket.
>
>    The problem is that even though I properly limited
the number of
>    amavisd connections in amavisd-milter, amavisd does
have a listen
>    backlog (queuing of connecting clients) of maximum 5
clients. That
>    is, if I have configured to use at most 10 amavisd
processes and by
>    chance 6 amavisd-milter processes try to establish a
connection
>    simultaneously, it will fail.

I don't see why it would be 5. The program flow goes like:

amavisd leaves Net::Server's option 'listen' at a default.

Net::Server.pm turns undef into 128:
  $prop-> = Socket::SOMAXCONN()
    unless defined($prop->) &&
$prop-> =~ /^d{1,3}$/;

Net::Server::Proto::Unix.pm:
  if( $unix_type != SOCK_DGRAM ){
    $args = $prop->;  # how many
connections for kernel to queue
  }

Inserting a debug printout for $lport into
IO::Socket::UNIX.pm like:

    if(exists $arg-> && $type !=
SOCK_DGRAM) {
printf STDERR ("HERE: %sn", $arg->);
	$sock->listen($arg-> || 5) or
	    return undef;
    }

and starting 'amavisd foreground' gives me 128, the same as
for INET sockets.

>    That's why, it would be useful to have in amavisd a
variable called
>    $listen_backlog or something like that, which would
allow you to
>    specify the queue size for the listen sockets.
amavisd would then
>    just need to pass this as 'listen' parameter to
Net::Server...

--- amavisd.orig	Wed Jun 27 12:43:00 2007
+++ amavisd	Fri Jul 27 17:48:57 2007
 -282,5
+282,5 
       $child_timeout $smtpd_timeout
       %current_policy_bank %policy_bank %interface_policy
-      $unix_socketname $inet_socket_port $inet_socket_bind
+      $unix_socketname $inet_socket_port $inet_socket_bind
$listen_queue_size
       $relayhost_is_client $smtpd_recipient_limit
       $enforce_smtpd_message_size_limit_64kb_min
 -11444,4
+11444,5 
     host => (!defined($inet_socket_bind) ||
$inet_socket_bind eq '' ? '*'
                                                           :
$inet_socket_bind),
+    listen => defined $listen_queue_size ?
$listen_queue_size : undef,
     max_servers => defined $max_servers_override ?
$max_servers_override
                                : $max_servers,  # number of
pre-forked children

Unfortunately Net::Server does not allow to specify listen
backlog queue size
individually for each socket.

>    Note that this is also not really an issue with inet
sockets because
>    of the way connections to inet sockets are handled:
if there is no
>    free space in the connection queue, the SYN packet
is just dropped
>    so that the client comes back later. With unix
sockets, it is not
>    possible to do that so the client gets immediately a
connection
>    refused.

'netstat -L -a' is showing me 128 for all amavisd sockets.

  Mark

------------------------------------------------------------
-------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and
a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
AMaViS-user mailing list
AMaViS-userlists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user

AMaViS-FAQ:http://www.amav
is.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/ho
wto/

On Fri, Jul 27, 2007 at 17:52:58 +0200, Mark Martinec
wrote:
> >    The problem is that even though I properly
limited the number of
> >    amavisd connections in amavisd-milter, amavisd
does have a listen
> >    backlog (queuing of connecting clients) of
maximum 5 clients. That
> >    is, if I have configured to use at most 10
amavisd processes and by
> >    chance 6 amavisd-milter processes try to
establish a connection
> >    simultaneously, it will fail.
> 
> I don't see why it would be 5. The program flow goes
like:
> 
> amavisd leaves Net::Server's option 'listen' at a
default.
> 
> Net::Server.pm turns undef into 128:
>   $prop-> = Socket::SOMAXCONN()
>     unless defined($prop->) &&
$prop-> =~ /^d{1,3}$/;

Funnily enough, Socket::SOMAXCONN() returns 5 on Solaris,
but I am
pretty sure that it is not the real limit. It probably comes
from
SOMAXCONN in /usr/include/sys/socket.h, but I don't think
that it is the
real limit. I have:

dwswinnetou:~$ ndd /dev/tcp tcp_conn_req_max_q
1024

Maybe Net::Server should be fixed instead? It probably would
be even
better to fix Socket::SOMAXCONN... I don't know if there is
a faster way
to find out the real SOMAXCONN than running ndd though. In
the mean
time, it would be nice however to have a $listen_queue_size
option in
Amavisd-new 

Cheers
David

------------------------------------------------------------
-------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and
a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
AMaViS-user mailing list
AMaViS-userlists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user

AMaViS-FAQ:http://www.amav
is.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/ho
wto/

David,

> Maybe Net::Server should be fixed instead? It probably
would be even
> better to fix Socket::SOMAXCONN...

Since the problem is in Socket::SOMAXCONN, this is where it
should
be fixed, probably not in Net::Server. It is unlikely the
Perl
core module will be fixed, at least not for the current
version.
But certainly, if you'd be willing to file a Perl bug
report,
it would be a contribution to a common good...

> I don't know if there is a faster way 
> to find out the real SOMAXCONN than running ndd though.
In the mean
> time, it would be nice however to have a
$listen_queue_size option in
> Amavisd-new 

Already in the code, it will be in the next version
(whenever it may be).

Thanks for troubleshooting!

  Mark

------------------------------------------------------------
-------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and
a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
AMaViS-user mailing list
AMaViS-userlists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user

AMaViS-FAQ:http://www.amav
is.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/ho
wto/

David Schweikert wrote:

> On Fri, Jul 27, 2007 at 17:52:58 +0200, Mark Martinec
wrote:
> > >    The problem is that even though I properly
limited the number of
> > >    amavisd connections in amavisd-milter,
amavisd does have a listen
> > >    backlog (queuing of connecting clients) of
maximum 5 clients. That
> > >    is, if I have configured to use at most 10
amavisd processes and by
> > >    chance 6 amavisd-milter processes try to
establish a connection
> > >    simultaneously, it will fail.
> >
> > I don't see why it would be 5. The program flow
goes like:
> >
> > amavisd leaves Net::Server's option 'listen' at a
default.
> >
> > Net::Server.pm turns undef into 128:
> >   $prop-> = Socket::SOMAXCONN()
> >     unless defined($prop->) &&
$prop-> =~ /^d{1,3}$/;
>
> Funnily enough, Socket::SOMAXCONN() returns 5 on
Solaris, but I am
> pretty sure that it is not the real limit. It probably
comes from
> SOMAXCONN in /usr/include/sys/socket.h, but I don't
think that it is the
> real limit. I have:
>
> dwswinnetou:~$ ndd /dev/tcp tcp_conn_req_max_q
> 1024
>
> Maybe Net::Server should be fixed instead? It probably
would be even
> better to fix Socket::SOMAXCONN... I don't know if
there is a faster way
> to find out the real SOMAXCONN than running ndd though.
In the mean
> time, it would be nice however to have a
$listen_queue_size option in
> Amavisd-new 

One catch there, which I'd call a Net::Server bug:

If one sets the $listen_queue_size to 1024 (as you say is a
default on
Solaris), the Net::Server tests the value of a 'listen'
option and sees
the number has more than three digits, and will silently
give you the 
Socket::SOMAXCONN default, which is a 5 on Solaris!  Without
a warning!

I think that Net::Server should:
- log a warning or call 'die' if the 'listen' option is
invalid and
  is not a 0 or undef (or an empty string);
- allow values of up to 1024 at least;
- maybe even provide a more sensible default on Solaris,
instead of
  a problematic Socket::SOMAXCONN;

I'm CCing this to Paul Seamons. I'd ask at least for a
logged warning.
Btw, this thread is archived at: http://ma
rc.info/?t=118553883800013&r=1

  Mark

------------------------------------------------------------
-------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and
a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
AMaViS-user mailing list
AMaViS-userlists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user

AMaViS-FAQ:http://www.amav
is.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/ho
wto/

On 7/27/07, David Schweikert <davidschweikert.ch> wrote:
...
> 1. amavisd-milter needs to be patched to work well with
Postfix, because
>    Postfix does give information about the queue-number
only after the
>    RCPT-TO-phase. I have attached a patch for that.

I will add it to the next release.

Thanks for troubleshooting!

Keep in touch.

P.

------------------------------------------------------------
-------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and
a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
AMaViS-user mailing list
AMaViS-userlists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user

AMaViS-FAQ:http://www.amav
is.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/ho
wto/