amavisd-new, spamassassin and sendmail milter

This is a shorter request for clarification (see previous
longer post for 
details) about our attempt to replace our pure sendmail with
DNSBL 
filtering policy with one based on sendmail 8.13.1,
amavisd-new-2.1.2 in 
the "milter" configuration and spamassassin
(spamd ??) 3.0.4

What we want to achieve is that : filtering is all performed
on one of our 
"entrance" MX machines; spam is quarantined (not
tagged and not delivered 
to recipient) ; spam generates an explanatory reject error
(to inform 
generators of false positives) ; the other messages are
forwarded to the 
user workstations (NO centralized delivery) ; we exploit the
spamassassin 
auto-learning and auto-whitelisting capabilities.


Now my first question (which I did not ask yesterday) is
more of 
architectural nature :

QUESTION 0) is spamd necessary ?

  In our present configuration we start (in the order) :
spamd,
  amavisd-new with its milter, sendmail.

  I thought the milter will act as an intermediate layer
between
  sendmail and amavisd-new and the latter was talking with
spamd

  but now I have the doubt that it calls spamassassin
modules  
  directly without need of spamd 

  Shall we run spamd or not ?
  

Having realized that the default milter coming with
amavisd-new does not 
perform full spamassassin header editing, we remain with the
more radical 
doubt

QUESTION 5) bayes autolearn and autowhitelist

  Are these spamassassin features working in milter
configuration, or 
  are they forbidden ? In latter case is there a workaround
?

The above question is quite relevant. We thought to
"train" the bayes 
filter with a collection of old spam (we plan to do it next
week), but if 
it does not autolearn, is there any sense with it ?

I also have a curiosity question

QUESTION 3) Hits: - in syslog

We see in the mail log some "amavis Passed
CLEAN" entries which have a 
non-numeric spamassassin score (Hits: -).

Is this related to $sa_timeout ? Should we raise this
timeout to a value 
higher than default ? Will it be honoured in milter
configuration ?

Many thanks for any hint.

------------------------------------------------------------
----------------
Lucio Chiappetti - INAF/IASF - via Bassini 15 - I-20133
Milano (Italy)
For more info : h
ttp://www.iasf-milano.inaf.it/~lucio/personal.html
------------------------------------------------------------
----------------


_______________________________________________
AMaViS-user mailing list
AMaViS-userlists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user

AMaViS-FAQ:http://www.amav
is.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/ho
wto/

Lucio Chiappetti schrieb:
> This is a shorter request for clarification (see
previous longer post for 
> details) about our attempt to replace our pure sendmail
with DNSBL 
> filtering policy with one based on sendmail 8.13.1,
amavisd-new-2.1.2 in 
> the "milter" configuration and spamassassin
(spamd ??) 3.0.4

Why don't you use a newer version of amavisd-new?

> QUESTION 0) is spamd necessary ?

>   Shall we run spamd or not ?

No (unless amavisd-new/milter is completely different).
spamd is used
to eliminate the overhead of starting the spamassassin
command line
tool. amavisd-new uses the SpamAssassin modules directly and
therefore
does not use spamd.

> QUESTION 3) Hits: - in syslog

> We see in the mail log some "amavis Passed
CLEAN" entries which have a 
> non-numeric spamassassin score (Hits: -).

> Is this related to $sa_timeout ? Should we raise this
timeout to a value
> higher than default ? Will it be honoured in milter
configuration ?

I think "Hits: -" means that the spam scan did
not run or did not
complete successfully. Either you got a SpamAssassin timeout
(which is
noted in the logs IIRC) or you disabled spam scanning for
that
recipient.

-- 
Felix



_______________________________________________
AMaViS-user mailing list
AMaViS-userlists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user

AMaViS-FAQ:http://www.amav
is.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/ho
wto/

Lucio,

> This is a shorter request for clarification (see
previous longer post for
> details) about our attempt to replace our pure sendmail
with DNSBL
> filtering policy with one based on sendmail 8.13.1,
amavisd-new-2.1.2 in
> the "milter" configuration and spamassassin
(spamd ??) 3.0.4

Before going into details: your life would be easier by
switching to Postfix.
TLS is easy, and interfacing to IMAP, Mailman, webmail,
procmail
is no more difficult than in sendmail. Sendmail+milter setup
with heavy-weight filter like SA offers poor control on mail
rush-ins.
Not to mention that amavisd-new would gain full control on
mail
header edits. A price for keeping congestions under control
would be a loss of ability to REJECT spam (except for
greylisting
and obvious violations).

> What we want to achieve is that : filtering is all
performed on one of our
> "entrance" MX machines; spam is quarantined
(not tagged and not delivered
> to recipient) ; spam generates an explanatory reject
error (to inform
> generators of false positives) ; the other messages are
forwarded to the
> user workstations (NO centralized delivery) ; we
exploit the spamassassin
> auto-learning and auto-whitelisting capabilities.

> QUESTION 0) is spamd necessary ?

No.  The amavisd daemon behaves just like spamd and is used
in its place.

>   In our present configuration we start (in the order)
: spamd,
>   amavisd-new with its milter, sendmail.
>
>   I thought the milter will act as an intermediate
layer between
>   sendmail and amavisd-new and the latter was talking
with spamd
>
>   but now I have the doubt that it calls spamassassin
modules
>   directly without need of spamd
>
>   Shall we run spamd or not ?

If your clients do not call spamc/spamd on their own (like
for
per-recipient rulesets), you don't need spamd if you have
amavisd-new.

> Having realized that the default milter coming with
amavisd-new does not
> perform full spamassassin header editing, we remain
with the more radical
> doubt
>
> QUESTION 5) bayes autolearn and autowhitelist
>
>   Are these spamassassin features working in milter
configuration, or
>   are they forbidden ? In latter case is there a
workaround ?

Bayes autolearn and autowhitelist work just fine with
amavisd-new,
no matter how it interfaces with MTA. A global Bayes
database is used
though - which can be a benefit or a drawback, depending on
diversity
of your users.

> The above question is quite relevant. We thought to
"train" the bayes
> filter with a collection of old spam (we plan to do it
next week), but if
> it does not autolearn, is there any sense with it ?

If you have a good set of rules (SA 3.1.2 and recent SARE
rules)
and usual networks tests like DCC, Razor2 and RBL/URIBL
enabled,
there is probably no need to pre-train bayes, just let it
autolearn for a couple of days of regular mail traffic,
perhaps with temporarily lowered scores for BAYES_* rules.
Later it suffices to feed it an occasional FP and FN.

> I also have a curiosity question
>
> QUESTION 3) Hits: - in syslog
>
> We see in the mail log some "amavis Passed
CLEAN" entries which have a
> non-numeric spamassassin score (Hits: -).

A '-' score means SA didn't produce any score. It
normally means
SA was not called by amavisd because mail was too big (over 
$sa_mail_body_size_limit), or spam checking was disabled for
all recipients of the message (bypass_*) or sender was black
or whitelisted for all recipients.

> Is this related to $sa_timeout ?

Rarely.

> Should we raise this timeout to a value 
> higher than default ? Will it be honoured in milter
configuration ?

The default in amavisd-new 2.4.1 suffices (and is larger and
computed
differently than in previous versions). Setting the
$sa_timeout is
now unnecessary.

  Mark


_______________________________________________
AMaViS-user mailing list
AMaViS-userlists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user

AMaViS-FAQ:http://www.amav
is.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/ho
wto/

On Fri, 26 May 2006, Felix Schwarz wrote:
> Lucio Chiappetti schrieb:

> Why don't you use a newer version of amavisd-new?

Rule of the house : use software bundled with OS
distribution (in this 
case SuSE 9.2) unless there are good reasons not to. Are
there any ?

> > QUESTION 3) Hits: - in syslog

> I think "Hits: -" means that the spam scan
did not run or did not
> complete successfully. Either you got a SpamAssassin
timeout (which is
> noted in the logs IIRC) or you disabled spam scanning
for that
> recipient.

We have no individual setup, all users are handled equally.
And there is 
no info about timeout in OUR logs, entries look like this :

May 25 09:42:20 oort sendmail[32286]: k4P7gEmL032286:
from=<xxxx>, 
size=291278, class=0, nrcpts=1, 
msgid=<001101c67fce$d8960a20$876408c0om02>,
proto=ESMTP, daemon=MTA, 
relay=bngf.ufanet.ru [81.30.212.117]

May 25 09:42:20 oort amavis[32186]: (k4P7gEmL032286) Passed
CLEAN, 
[192.8.100.135] <xxxx> -> <yyyy>, Message-ID:

<001101c67fce$d8960a20$876408c0om02>, Hits: -

May 25 09:42:20 oort amavis[32186]: (k4P7gEmL032286) Passed
CLEAN, <xxxx> 
-> <yyyy>, Hits: -, tag=-990, tag2=4.5, kill=4.5,
L/0/0/0

May 25 09:42:22 oort sendmail[32289]: k4P7gEmL032286:
to=yyyy, 
delay=00:00:03, xdelay=00:00:02, mailer=smtp, pri=321520,
relay=zzzz. 
[x.y.z.t], dsn=2.0.0, stat=Sent (k4P7gLUl022281 Message
accepted for 
delivery)


If it were an (unlogged) timeout, should we tweak
$sa_timeout ? What are 
sensible values ?

------------------------------------------------------------
----------------
Lucio Chiappetti - INAF/IASF - via Bassini 15 - I-20133
Milano (Italy)
For more info : h
ttp://www.iasf-milano.inaf.it/~lucio/personal.html
------------------------------------------------------------
----------------


_______________________________________________
AMaViS-user mailing list
AMaViS-userlists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user

AMaViS-FAQ:http://www.amav
is.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/ho
wto/

Lucio,

> a) we presently run only spam checking, no virus
checking

I'd recommend adding (at least) clamd to the mix for two
reasons:
if amavisd-new knows the content is infected it won't call
SA
and won't do it's spam-related actions on spammy-looking
viruses.
This saves time because virus checking with clamd is much
faster
than spam checking, and it prevents backscatter in a
dual-MTA setup
(this is a non-issue in a pre-queue setup like milter if you
REJECT
mail, but REJECTing viruses can produce secondary
backscatter
from the sending MTA anyway and is undesirable
nevertheless).

> b) we defined $final_spam_destiny         = D_REJECT;
> c) we defined $QUARANTINEDIR to be a file (Unix mbox
folder)

I'd recommend quarantining to individual files (or to SQL).
The amavisd-release utility is unable to release quarantined
mail from a mbox file.

> d) we have $sa_tag_level_deflt  = -999,
$sa_tag2_level_deflt = 4.5,
>    $sa_kill_level_deflt = 4.5, $sa_dsn_cutoff_level =
-999 as
>    in the GARR recommended configuration
>
>    As a result of the above all spam above 4.5 (which
seems pretty OK)
>    is NOT delivered to the recipient, but quarantined
to the virusmails
>    folder. We plan to rotate the folder daily and send
to each user one
>    mail with the report of the spam received.

4.5 is too low for a kill level.  A 6 is more like it.

Presenting recipient with false positives gives spam filter
a bad reputation,
it is better to err slightly in the other direction.

Consider http://www.MailZu.net/ if
your decide to use SQL
for quarantining and reporting.

> QUESTION 2) Do I interpret correctly the various
statements in
> http://www.ijs.si
/software/amavisd/ meaning that no header editing AT ALL
> is performed when using the milter ?

Yes, with a milter helper that comes with a package.
It is a limitation of the old amavis client protocol.

The solution was provided by implementing alternative
protocol (AM.PDP), and Petr's milter supports it.

> f) we tried to edit $X_HEADER_TAG =
'X-Virus-Spam-Scanned' so that the
>    recipient could see that a spam scan was done and
where, but it looks
>    like that in passed mail we get only the
"standard"
>    X-Virus-Scanned: by amavisd-new
>    without indication of host nor of our editing.

This is specific to old milter helper program, which inserts
this
header by itself, without ability to obtain it from amavisd
daemon.
AM.PDP overcomes this limitation.

> g) we have $sa_auto_whitelist = 1 , and we have Bayes
filter configured
>    in spamassassin local.cf ... but we doubt that they
are operating.

$sa_auto_whitelist is irrelevant with newer versions of SA.
>From amavisd.conf-sample:
  $sa_auto_whitelist = 1; # turn on AWL in SA 2.63 or older
(irrelevant
                          # for SA 3.0, its cf option is
use_auto_whitelist)

>    The bayes and auto-whitelist files have size zero,

Make sure these files are owned/writable by uid under which
amavisd
is running (usually vscan or amavis).

Try to manually feed one spam sample to sa-learn just to see
that
it it able to update the database:
  su vscan -c 'sa-learn --spam test.msg'

Run 'amavisd debug-sa' for a short while to be able to see
if SA is
reporing any problems.

> h) we have all $warn*sender=0 and $warn*recip=1.

Let's just forget that $warn*sender settings ever existed.
It is off by default anyway.

$warn*recip=1 is mostly unproductive: replacing one spam
message
with another locally generated is more annoying than letting
spam (tagged as such) through.

> Rule of the house : use software bundled with OS
distribution (in this
> case SuSE 9.2) unless there are good reasons not to.
Are there any ?

We are slowly loosing amavisd-new-2.1.2 out of sight
(September 2004),
you should at least go to 2.3.3 or preferably to 2.4.1, if
for no other 
reasons for better maintainability (better handling and/or
clearer logging
of problems). The long list of reasons is available in
RELEASE_NOTES.

Most advice and examples on the mailing list refer to 2.4.x
(or sometimes to 2.3.3).

> May 25 09:42:20 oort amavis[32186]: (k4P7gEmL032286)
Passed CLEAN, <xxxx>
> -> <yyyy>, Hits: -, tag=-990, tag2=4.5,
kill=4.5, L/0/0/0

Are you sure you don't have spam checks bypassed?
Look at amavisd log, increase the $log_level for starters
(a 2 is a good compromise).

> If it were an (unlogged) timeout, should we tweak
$sa_timeout ?

With 2.4.1 there is no need to tweak $sa_timeout

> What are sensible values ?

Should be less than $child_timeout, which in turn should
be less than milter timeout.

If you have Bayes on a berkeley db, which handles
auto-expiry very slowly,
one indeed had to increase $sa_timeout in versions before
2.4.1.
A better solution is to have Bayes on SQL, which expires
tokens
much faster, so all the problems with slow bayes auto-expiry
and an occasional corrupted Bayes database are avoided.

  Mark


_______________________________________________
AMaViS-user mailing list
AMaViS-userlists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user

AMaViS-FAQ:http://www.amav
is.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/ho
wto/