turn off encapsulation?

hi,

thank you for all the hard work on maia.  i've installed
her, she's running, and i'm 
trying to understand what to do with her, short of getting
rid of her and using 
amavisd-new by itself instead.

i want to enable thunderbird to recognize
maia/amavisd/spamassassin results, and i am 
trying to understand what i need to do.

i am unclear if amavisd-maia is removing spam tag headers
inserted by spamassassin, or 
if amavisd-maia is instructing spamassassin not to insert
them.  the spamassassin 
manpage says by default it would add X-Spam-Status headers. 
i am assuming the 
encapsulation i am getting was done by spamassassin
("Spam detection software, running 
on the system...").

also, if such spamassassin headers are missing, perhaps the
encapsulation i'm getting 
might be defeating what's needed by sa-learn:

htt
p://mail-archives.apache.org/mod_mbox/spamassassin-users/200
506.mbox/%3c6.2.0.14.0.20050601090302.03a8cb60mail.comcast.net%3e
    sa-learn recognizes markups made by spamassassin,
including encapsulation,
    and will correctly undo the encapsulation before
learning the message.
    However, if you use some other tool such as mimedefang
to do your
    encapsulation, SA won't recognize that.

so, what i want to know is:
   1.  where to (re-)enable the X-Spam-Flag and
X-Spam-Status headers
   2.  whether i should disable encapsulation for
sa-learn's benefit, and again where to 
do that
   3.  how to skip spam and "banned" filtering
for authenticated smtp clients

thank you,
greg wm
IT Coordinator
NonviolentPeaceforce.org
_______________________________________________
Maia-users mailing list
Maia-usersrenaissoft.com
http://www.renaissoft.com/mailman/listinfo/maia-users

greg wm wrote:
> hi,
> 
> thank you for all the hard work on maia.  i've
installed her, she's running, and i'm 
> trying to understand what to do with her, short of
getting rid of her and using 
> amavisd-new by itself instead.
> 
> i want to enable thunderbird to recognize
maia/amavisd/spamassassin results, and i am 
> trying to understand what i need to do.
> 
> i am unclear if amavisd-maia is removing spam tag
headers inserted by spamassassin, or 
> if amavisd-maia is instructing spamassassin not to
insert them.  the spamassassin 
> manpage says by default it would add X-Spam-Status
headers.  i am assuming the 
> encapsulation i am getting was done by spamassassin
("Spam detection software, running 
> on the system...").

It sounds like you're trying to use a 747 (maia) merely as
a chicken coop.

The main idea of maia is to integrate the spam/virus
detection capabilities,
provide easy manageability of the spam quarantine, and to
automate tedious
tasks such as spam reporting and the training of the bayes
database.

If you toss all that aside and simply use your mail agent to
detect/sort your
spam, you're really not getting any benefit at all from
maia - but that's
fine, everybody has different needs. It sounds like you just
want a simple
standalone spamassassin setup, to mark your mail headers,
and that's it.

Joel

_______________________________________________
Maia-users mailing list
Maia-usersrenaissoft.com
http://www.renaissoft.com/mailman/listinfo/maia-users

greg wm wrote:

> i am unclear if amavisd-maia is removing spam tag
headers inserted by spamassassin, or 
> if amavisd-maia is instructing spamassassin not to
insert them.

Amavisd-maia is forced to write the X-Spam headers itself,
due to the
way that SpamAssassin is invoked.  All that amavisd-maia
gets back from
SpamAssassin is the mail's total score and a list of the
rules that were
triggered, /not/ a modified copy of the mail itself.  Any
header
modifications must be done by amavisd-maia instead.  The
SpamAssassin
features you're talking about (e.g. encapsulation) are only
available if
you use spamd instead of amavisd-maia (though that rather
defeats the
purpose of having Maia in the first place).


> also, if such spamassassin headers are missing, perhaps
the encapsulation i'm getting 
> might be defeating what's needed by sa-learn:
> 
> htt
p://mail-archives.apache.org/mod_mbox/spamassassin-users/200
506.mbox/%3c6.2.0.14.0.20050601090302.03a8cb60mail.comcast.net%3e
>     sa-learn recognizes markups made by spamassassin,
including encapsulation,
>     and will correctly undo the encapsulation before
learning the message.
>     However, if you use some other tool such as
mimedefang to do your
>     encapsulation, SA won't recognize that.
> 
> so, what i want to know is:
>    1.  where to (re-)enable the X-Spam-Flag and
X-Spam-Status headers
>    2.  whether i should disable encapsulation for
sa-learn's benefit, and again where to 
> do that
>    3.  how to skip spam and "banned"
filtering for authenticated smtp clients

Well, first of all, why are you concerned about running
sa-learn?
Maia's process-quarantine.pl script takes care of all of
that for you.
I get the impression that you're not altogether clear about
how Maia
works, and you're trying to hammer it as a square peg into
a round hole
that you're more familiar with.

With Maia, all the Bayes training and spam reporting gets
done behind
the scenes by the process-quarantine.pl script, which you
schedule to
run at hourly intervals.  Your users effectively do the
training
whenever they login to the Maia web interface to confirm the
mail in
their quarantines and caches, release items, and/or report
spam that
slipped through.  The process-quarantine.pl script simply
uses these
bits of user feedback to do the Bayes training and reporting
for you.
There's no need to run sa-learn manually.

As for the X-Spam-Status header, it will be added
automatically to mail
that scores above the threshold you've set for them.  On
your Settings
page, click on one of the e-mail addresses linked to your
account, and
you should see three available threshold scores, the first
of which is
called "Add X-Spam: Headers when Score is
>=".  You can set that to
something like -999 to ensure that that header is included
in /all/
mail, whether it's spam or not.

As for adding subject prefixes to spam items, you need to do
two things.
 First, set "Add a prefix to the subjects of
spam?" to "Yes" on your
Settings page, and second, set the subject prefix in your
amavisd.conf
file, e.g.

$sa_spam_subject_tag = '***SPAM*** ';

As for your third item, how to bypass filtering for
authenticated SMTP
clients, your best bet, frankly, is to have your mail server
route that
traffic /around/ amavisd-maia.  Maia has no way to verify
that the mail
it gets handed came from an authenticated source, after
all--it just
receives the mail from your upstream mail server, not the
original
connection from the client.  Sure, your upstream mail server
might have
added some headers to persuade amavisd-maia to treat the
mail
differently, but then so could a spammer.  Spam often
contains such
forged "authenticated-by" headers precisely with
the goal of making the
mail look more legitimate to spam filters that have no way
of verifying
the authentication stamp.

If you want to give special treatment to users that your
upstream mail
server authenticates, you should have that upstream mail
server route
their mail differently, so that it does not go through
amavisd-maia at
all.  Presumably this is because you trust them never to
send spam
themselves (or to have their own computers infected by a
worm or Trojan
that causes them to start spewing spam against their will).

On the other hand, if you're mainly concerned with
/outbound/ filtering,
the easiest solution is to just go to the System Default
user (.)'s
settings disable whatever filters you don't want.  If
you've set up all
of your domains properly so that Maia is aware of them, the
only mail
that should ever end up assigned to the System Default user
is outbound
mail (i.e. mail that isn't addressed to any of the domains
Maia knows
about), so the System Default user becomes an easy mechanism
for
defining how outbound mail gets filtered.

-- 
Robert LeBlanc <rjlrenaissoft.com>
Renaissoft, Inc.
Maia Mailguard <http://www.maiamail
guard.com/>

_______________________________________________
Maia-users mailing list
Maia-usersrenaissoft.com
http://www.renaissoft.com/mailman/listinfo/maia-users

joel wrote:
> It sounds like you're trying to use a 747 (maia)
merely as a chicken coop.

Robert LeBlanc wrote:
> Well, first of all, why are you concerned about running
sa-learn?
> Maia's process-quarantine.pl script takes care of all
of that for you.
> I get the impression that you're not altogether clear
about how Maia
> works, and you're trying to hammer it as a square peg
into a round hole
> that you're more familiar with.

what attracted me to maia is the ability for users to use
the web
interface to customize the way it works for them.  and if
they want to
use the maia quarantine and interface, that's available.

or if they want to review their spam in their mail app, we
should be
able to do that.  it would seem that maia was intended to
accomodate
this (the "labeled" choice).  thunderbird's
server-side spam filter
integration, or a thunderbird or maildrop filter, can put
tagged spam
into a spam folder.  false negatives can be moved to a
Spam-Learn folder
and false positives copied to a Ham-Learn folder.  i have
setup cron to
pass these to sa-learn, right alongside
process-quarantine.pl.  will
maia's design prevent any of this from working as i expect?

> Maia is seeing your email address <maiausernvpf.org> as "non-local" in
> both cases...You haven't told Maia that it should
process mail for the
> nvpf.org domain.

ah.  that was it.  now i'm getting the spam flag and status
headers
along with the encapsulation.  thank you.
 
 >>htt
p://mail-archives.apache.org/mod_mbox/spamassassin-users/200
506.mbox/%3c6.2.0.14.0.20050601090302.03a8cb60mail.comcast.net%3e
>>    sa-learn recognizes markups made by
spamassassin, including encapsulation,
>>    and will correctly undo the encapsulation before
learning the message.
>>    However, if you use some other tool such as
mimedefang to do your
>>    encapsulation, SA won't recognize that.

so 2 questions remain.  will sa-learn work alongside maia,
and, will the
amavisd-maia encapsulation be ok for sa-learn?

here's an example encapsulation i'm getting (if this
message appears
twice i apologize, i'm trimming the example and retrying to
get it by
maia!):

thank you,
greg wm
IT Coordinator
NonviolentPeaceforce.org

>> Return-Path: <xrqsergemail.com>
>> X-Original-To: webmasternvpf.us
>> Delivered-To: gmottsergio.nvpf.org
>> Received: from localhost (localhost.localdomain
[127.0.0.1])
>> 	by sergio.nvpf.org (Postfix) with ESMTP id
107628B316
>> 	for <webmasternvpf.us>; Wed, 28 Jun
2006 21:41:32 -0500 (CDT)
>> Resent-From: "Content-filter at sergio"
<postmastersergio.nvpf.org>
>> Resent-Date: Wed, 28 Jun 2006 21:41:26 -0500 (CDT)
>> Resent-Message-ID: <RE01390-01-5sergio>
>> Received: from sergio.nvpf.org ([127.0.0.1])
>>  by localhost (sergio [127.0.0.1]) (amavisd-new,
port 10024) with LMTP
>>  id 01390-01-5 for <webmasternvpf.us>; Wed, 28 Jun 2006 21:41:26 -0500 (CDT)
>> Content-Type: multipart/mixed;
boundary="----------=_1151548892-1390-3"
>> Content-Transfer-Encoding: binary
>> MIME-Version: 1.0
>> Received: from [snip] by sergio.nvpf.org
>>  (Postfix) with SMTP id D0FBBBAE7C 	for
<webmasternvpf.us>; Wed,
>>  28 Jun 2006 21:41:15 -0500 (CDT)
>> From: zqUsherlx <xrqsergemail.com>
>> To: webmasternvpf.us
>> Date: Wed, 28 Jun 2006 23:29:47 +0000
>> Message-ID: <dde601c69b0a$722d376a$3732cd46mail.com>
>> Subject: [probably spam]
=?iso-8859-1?B?QWxsIGJlc3Qgc29mdHdhcmUhIEVWRVIhIGl2eCBXZWQsI
DI4IEp1biAyMDA2IDIz?=
>>  =?iso-8859-1?B?OjQwOjE0?=
>> X-Mailer: Microsoft Outlook Express V6.00.2900.2180
>> X-Amavis-Modified: Original mail wrapped as
attachment (defanged) by sergio
>> X-Virus-Scanned: Maia Mailguard 1.0.0
>> X-Spam-Status: Yes, hits=24.938 tagged_above=-5
required=2
>>  tests=EXTRA_MPART_TYPE, FORGED_OUTLOOK_TAGS,
HTML_FONT_BIG, HTML_MESSAGE,
>>  MANY_EXCLAMATIONS, RCVD_IN_XBL, RCVD_NUMERIC_HELO,
SUBJECT_ENCODED_TWICE,
>>  SUBJECT_EXCESS_BASE64, TW_IV, URIBL_AB_SURBL,
URIBL_JP_SURBL, URIBL_OB_SURBL,
>>  URIBL_SBL, URIBL_SC_SURBL, URIBL_WS_SURBL
>> X-Spam-Level: ************************
>> X-Spam-Flag: YES
>>
>> This is a multi-part message in MIME format...
>>
>> ------------=_1151548892-1390-3
>> Content-Type: text/plain;
charset="iso-8859-1"
>> Content-Disposition: inline
>> Content-Transfer-Encoding: 7bit
>>
>> Spam detection software, running on the system
"sergio.nvpf.org", has
>> identified this incoming email as possible spam. 
The original message
>> has been attached to this so you can view it (if it
isn't spam) or label
>> similar future email.  If you have any questions,
see
>> the administrator of that system for details.
>> 
>> Content preview:  Grab Microsoft Windows XP Pro and
Microsoft Office XP
>>   Pro at lowest price ever. $69.95 only. You never
find the same software
>>   at such a cheap price anywhere else. Its just
Great Product at Great
>>   Price. [...]
>>
>> Content analysis details:   (24.9 points, 5.0
required)
>>
>>  pts rule name              description
>> ---- ----------------------
--------------------------------------------------
>>  1.5 SUBJECT_ENCODED_TWICE  Subject: MIME encoded
twice
>>  0.8 EXTRA_MPART_TYPE       Header has extraneous
Content-type:...type= entry
>>  1.3 RCVD_NUMERIC_HELO      Received: contains an
IP address used for HELO
>>  0.1 TW_IV                  BODY: Odd Letter
Triples with IV
>>  0.0 HTML_MESSAGE           BODY: HTML included in
message
>>  0.3 HTML_FONT_BIG          BODY: HTML tag for a
big font size
>>  3.1 RCVD_IN_XBL            RBL: Received via a
relay in Spamhaus XBL
>>                             [snip]
_______________________________________________
Maia-users mailing list
Maia-usersrenaissoft.com
http://www.renaissoft.com/mailman/listinfo/maia-users

greg wm wrote:
> so 2 questions remain.  will sa-learn work alongside
maia, and, will the
> amavisd-maia encapsulation be ok for sa-learn?

oh yes and a 3rd:  exactly how do i turn off encapsulation? 
as 
suggested by 
https://secure.renaissoft.com/maia/wiki/SAConfigFile, my

/etc/mail/spamassassin/local.cf alreay contains:

    report_safe 0

thank you,
greg wm
IT Coordinator
NonviolentPeaceforce.org
_______________________________________________
Maia-users mailing list
Maia-usersrenaissoft.com
http://www.renaissoft.com/mailman/listinfo/maia-users

Robert LeBlanc wrote:
> On the other hand, if you're mainly concerned with
/outbound/ filtering,
> the easiest solution is to just go to the System
Default user (.)'s
> settings disable whatever filters you don't want.  If
you've set up all
> of your domains properly so that Maia is aware of them,
the only mail
> that should ever end up assigned to the System Default
user is outbound
> mail (i.e. mail that isn't addressed to any of the
domains Maia knows
> about), so the System Default user becomes an easy
mechanism for
> defining how outbound mail gets filtered.

that's fine if there are only non-local recipients, but if
there is a 
mixture of non-local and local recipients, or if the mail is
local to 
local, it may still get filtered, tagged, and encapsulated,
right?
_______________________________________________
Maia-users mailing list
Maia-usersrenaissoft.com
http://www.renaissoft.com/mailman/listinfo/maia-users

greg wm wrote:
> greg wm wrote:
> 
>>so 2 questions remain.  will sa-learn work alongside
maia, and, will the
>>amavisd-maia encapsulation be ok for sa-learn?

Yes, if you're really desperate to make more work for
yourself, you can
use sa-learn manually in addition to Maia's automatic
training.  You
won't want any encapsulation/defanging performed, however,
since
sa-learn needs to see the mail in its complete and raw form,
and
preferably with its original, unmodified mail headers.


> oh yes and a 3rd:  exactly how do i turn off
encapsulation?

In your amavisd.conf file, encapsulation is referred to as
"defanging",
e.g. $defang_virus, $defang_spam, $defang_banned,
$defang_bad_header,
$defang_undecipherable, and $defang_all.  Set these to 0 to
disable
encapsulation for a given mail type, or 1 to enable it.

-- 
Robert LeBlanc <rjlrenaissoft.com>
Renaissoft, Inc.
Maia Mailguard <http://www.maiamail
guard.com/>

_______________________________________________
Maia-users mailing list
Maia-usersrenaissoft.com
http://www.renaissoft.com/mailman/listinfo/maia-users

greg wm wrote:
> Robert LeBlanc wrote:
> 
>>On the other hand, if you're mainly concerned with
/outbound/ filtering,
>>the easiest solution is to just go to the System
Default user (.)'s
>>settings disable whatever filters you don't want. 
If you've set up all
>>of your domains properly so that Maia is aware of
them, the only mail
>>that should ever end up assigned to the System
Default user is outbound
>>mail (i.e. mail that isn't addressed to any of the
domains Maia knows
>>about), so the System Default user becomes an easy
mechanism for
>>defining how outbound mail gets filtered.
> 
> 
> that's fine if there are only non-local recipients,
but if there is a 
> mixture of non-local and local recipients, or if the
mail is local to 
> local, it may still get filtered, tagged, and
encapsulated, right?

Correct.  I still think you're trying to solve the wrong
problem, here,
though.  What scenario, exactly, are you trying to avoid by
giving your
authenticated users a free pass through all of your filters?

You do realize, I hope, that filtering outbound mail is one
of the
things you can do to stop the spread of spam and viruses
from infected
machines on your network, right?  If one of your users'
machines
contracts a malware infection that causes his machine to
spew spam
across the Internet, the last thing you want to do is let
his machine
bypass your filters.  If ISPs routinely performed outbound
filtering the
way they should, the spam problem would be cut dramatically.

If you're concerned that the list of banned file types is
too strict,
edit the banned list in your amavisd.conf file to remove the
types you
don't want it to filter--or disable banned file filtering
altogether for
your domain.

If you're concerned that your users' mail will be wrongly
quarantined as
spam, you should remember that if their mail looks spammy to
/your/
filter, it's probably going to look pretty spammy at the
/receiver's/
end as well.  And if your filter is performing so poorly
that it's
generating a lot of false positives on your own users'
mail, that means
it's also hurting you in terms of the mail you're
receiving from
outside, and you need to correct the problem with your
filter, not
bypass it and pretend the problem doesn't exist.

Furthermore, for the Bayes to develop properly it needs to
see lots of
examples of both "good" mail and spam, and if
you send all of your
known-good mail out in a way that bypasses your filter,
you're cheating
your Bayes out of a good learning opportunity.  Instead, let
your
outbound mail be filtered, and the Bayes and AWL will
quickly come to
recognize what your users' "usual" mail looks
like.  As your users send
out more mail and Maia gets a look at it, their AWL scores
will
gradually drop, biasing the total score downward over time. 
On the
other hand, if your users regularly bypass the filters, the
filters will
treat them as strangers because it doesn't know them.

-- 
Robert LeBlanc <rjlrenaissoft.com>
Renaissoft, Inc.
Maia Mailguard <http://www.maiamail
guard.com/>

_______________________________________________
Maia-users mailing list
Maia-usersrenaissoft.com
http://www.renaissoft.com/mailman/listinfo/maia-users

Robert LeBlanc wrote:
> What scenario, exactly, are you trying to avoid by
giving your
> authenticated users a free pass through all of your
filters?

folks connected via dynamic IP authenticate via tls so we
trust them, 
but the received line doesn't reflect that:
>> Received: from [192.168.1.105]
(cust-160-189.dsl.versateladsl.be [...])
>> 	by sergio.nvpf.org (Postfix) with ESMTP id
726B28B316;
>> 	Thu, 29 Jun 2006 02:22:43 -0500 (CDT)

and spamassassin ends up treating the message with disfavor,
to wit:
>>  2.0 RCVD_IN_SORBS_DUL      RBL: SORBS: sent
directly from dynamic IP address
>>                             [... listed in
dnsbl.sorbs.net]
>>  1.7 RCVD_IN_NJABL_DUL      RBL: NJABL: dialup
sender did non-local SMTP
>>                             [... listed in
combined.njabl.org]

and after a couple of these, the AWL bangs them down with an
even bigger 
hammer:
>>  9.3 AWL                    AWL: From: address is
in the auto white-list

and what is the user to do?  it's from them, not to them. 
it's not in 
their quarantine.  how are they even to know?  the score is
too high for 
a DSN.
_______________________________________________
Maia-users mailing list
Maia-usersrenaissoft.com
http://www.renaissoft.com/mailman/listinfo/maia-users

greg wm wrote:

> folks connected via dynamic IP authenticate via tls so
we trust them, 
> but the received line doesn't reflect that:
> 
>>>Received: from [192.168.1.105]
(cust-160-189.dsl.versateladsl.be [...])
>>>	by sergio.nvpf.org (Postfix) with ESMTP id
726B28B316;
>>>	Thu, 29 Jun 2006 02:22:43 -0500 (CDT)
> 
> 
> and spamassassin ends up treating the message with
disfavor, to wit:
> 
>>> 2.0 RCVD_IN_SORBS_DUL      RBL: SORBS: sent
directly from dynamic IP address
>>>                            [... listed in
dnsbl.sorbs.net]
>>> 1.7 RCVD_IN_NJABL_DUL      RBL: NJABL: dialup
sender did non-local SMTP
>>>                            [... listed in
combined.njabl.org]

It sounds like you haven't got your trusted_networks and/or
internal_networks configured properly in SpamAssassin's
local.cf file.
As a result, SpamAssassin doesn't know how where in the
list of
"Received:" headers your local/remote boundary
is.  When you've set
trusted_networks and/or internal_networks correctly, telling
SpamAssassin explicitly what your mail hosts' IP addresses
are and what
IP ranges define your network, it will (properly) ignore
"Received:"
lines that were written by servers you control or trust. 
See
<htt
p://wiki.apache.org/spamassassin/TrustPath> for a
more detailed
explanation of what's going on, and how to solve the
problem.

-- 
Robert LeBlanc <rjlrenaissoft.com>
Renaissoft, Inc.
Maia Mailguard <http://www.maiamail
guard.com/>

_______________________________________________
Maia-users mailing list
Maia-usersrenaissoft.com
http://www.renaissoft.com/mailman/listinfo/maia-users