SaneSecurity malware signatures are not being detected by amavisd-new

I have been exchanging e-mails with Steve Basford of
SaneSecuirty 
regarding malware e-mails that are not detected and
quarantined by 
amavisd-new, yet are detected fine when scanned directly by
clamdscan.  
Here is my amavisd.conf entries for SaneSecurity and MSRBL
signature 
detection:

virus_name_to_spam_score_maps =
  (new_RE( [
qr'^(Email|HTML).(Phishing|Spam|Scam[a-z0-9]?).'i =>
0.1 ],
           [ qr'^(Email|Html).Malware.Sanesecurity.'     
    => undef ],
           [ qr'^(Email|Html)(.[^., ]*)*.Sanesecurity.'  
    => 0.1 ],
           [ qr'^(MSRBL-Images/|MSRBL-SPAM.)'   => 0.1
],
  ));

However, it does not seem to detect and quarantine any
signature that 
starts with "Email", even though clamdscan
corrected detects and reports 
the malware signature:

clamdscan test.msg
test.msg: Email.Malware.Sanesecurity.07051800 FOUND

The entries in amavisd.conf look correct, but for some
reason, malware 
signatures beginning with "Email" do not get
detected and quarantined by 
amaviad-new.  Thoughts?

Thanks,

Bill


------------------------------------------------------------
-------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and
take
control of your XML. No limits. Just data. Click to get it
now.
http://sourcefor
ge.net/powerbar/db2/
_______________________________________________
AMaViS-user mailing list
AMaViS-userlists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user

AMaViS-FAQ:http://www.amav
is.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/ho
wto/

Bill,

>  [ qr'^(Email|HTML).(Phishing|Spam|Scam[a-z0-9]?).'i
=> 0.1 ],
>  [ qr'^(Email|Html).Malware.Sanesecurity.'    =>
undef],
>  [ qr'^(Email|Html)(.[^., ]*)*.Sanesecurity.' =>
0.1 ],
>  [ qr'^(MSRBL-Images/|MSRBL-SPAM.)'   => 0.1 ],

> However, it does not seem to detect and quarantine any
signature that
> starts with "Email", even though clamdscan
corrected detects and reports
> the malware signature:
>   test.msg: Email.Malware.Sanesecurity.07051800 FOUND
> The entries in amavisd.conf look correct, but for some
reason, malware
> signatures beginning with "Email" do not get
detected and quarantined by
> amaviad-new.  Thoughts?

Why do you have the second entry (... => undef) ???
If matched, it terminates the search and reports that a
lookup
did not find enything.  You probably intended to just remove
the line.

  Mark

------------------------------------------------------------
-------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and
take
control of your XML. No limits. Just data. Click to get it
now.
http://sourcefor
ge.net/powerbar/db2/
_______________________________________________
AMaViS-user mailing list
AMaViS-userlists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user

AMaViS-FAQ:http://www.amav
is.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/ho
wto/

Mark Martinec wrote the following on 5/18/2007 11:52 AM
-0800:
> Bill,
>
>   
>>  [
qr'^(Email|HTML).(Phishing|Spam|Scam[a-z0-9]?).'i =>
0.1 ],
>>  [ qr'^(Email|Html).Malware.Sanesecurity.'   
=> undef],
>>  [ qr'^(Email|Html)(.[^., ]*)*.Sanesecurity.'
=> 0.1 ],
>>  [ qr'^(MSRBL-Images/|MSRBL-SPAM.)'   => 0.1
],
>>     
>
>   
>> However, it does not seem to detect and quarantine
any signature that
>> starts with "Email", even though
clamdscan corrected detects and reports
>> the malware signature:
>>   test.msg: Email.Malware.Sanesecurity.07051800
FOUND
>> The entries in amavisd.conf look correct, but for
some reason, malware
>> signatures beginning with "Email" do not
get detected and quarantined by
>> amaviad-new.  Thoughts?
>>     
>
> Why do you have the second entry (... => undef) ???
> If matched, it terminates the search and reports that a
lookup
> did not find enything.  You probably intended to just
remove the line.
>
>   Mark
>   
Hi Mark,

It's setup this way because that's the way you have it shown
in the 
amavisd.conf-default file that comes with the distro, and I
want the 
file to be quarantined.  However, Steve had me try the
following:
=====
Change this signature from:

Email.Malware.Sanesecurity.07051800:4:687474703a2
f2f6d61696c2e756262692e636f6d2e62722f7664526663326174742f726
6633261747461636832302e646c6c

to:
Email.Malware.Sanesecurity.07051800:0:687474703a2
f2f6d61696c2e756262692e636f6d2e62722f7664526663326174742f726
6633261747461636832302e646c6c

Ie. change the type 4 (mail file) to type 0 (all file
types).   Save and
re-load clamd....

Now re-send the ecard to yourself... is it detected now?

If it does work then it looks like amavisd-new separates the
headers
from the body...and then uses clamd to scan the body ONLY...
which might
be no type 4 (Email.) would ever work?   But I'm sure you
would have
noticed.
=====

and without any changes to the amavisd.conf file,
amavisd-new now correctly identifies the malware and
quarantines the message.  Mark/Steve, how would you suggest
we handle these going forward?

Thanks,

Bill


------------------------------------------------------------
-------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and
take
control of your XML. No limits. Just data. Click to get it
now.
http://sourcefor
ge.net/powerbar/db2/
_______________________________________________
AMaViS-user mailing list
AMaViS-userlists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user

AMaViS-FAQ:http://www.amav
is.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/ho
wto/

At 02:04 PM 5/18/2007, Bill Landry wrote:
>Mark Martinec wrote the following on 5/18/2007 11:52 AM
-0800:
> > Bill,
> >
> >
> >>  [
qr'^(Email|HTML).(Phishing|Spam|Scam[a-z0-9]?).'i =>
0.1 ],
> >>  [ qr'^(Email|Html).Malware.Sanesecurity.' 
  => undef],
> >>  [ qr'^(Email|Html)(.[^.,
]*)*.Sanesecurity.' => 0.1 ],
> >>  [ qr'^(MSRBL-Images/|MSRBL-SPAM.)'   =>
0.1 ],
> >>
> >
> >
> >> However, it does not seem to detect and
quarantine any signature that
> >> starts with "Email", even though
clamdscan corrected detects and reports
> >> the malware signature:
> >>   test.msg:
Email.Malware.Sanesecurity.07051800 FOUND
> >> The entries in amavisd.conf look correct, but
for some reason, malware
> >> signatures beginning with "Email" do
not get detected and quarantined by
> >> amaviad-new.  Thoughts?
> >>
> >
> > Why do you have the second entry (... => undef)
???
> > If matched, it terminates the search and reports
that a lookup
> > did not find enything.  You probably intended to
just remove the line.
> >
> >   Mark
> >
>Hi Mark,
>
>It's setup this way because that's the way you have it
shown in the
>amavisd.conf-default file that comes with the distro,
and I want the
>file to be quarantined.  However, Steve had me try the
following:
>=====
>Change this signature from:
>
>Email.Malware.Sanesecurity.07051800:4:687474703a2
f2f6d61696c2e756262692e636f6d2e62722f7664526663326174742f726
6633261747461636832302e646c6c
>
>to:
>Email.Malware.Sanesecurity.07051800:0:687474703a2
f2f6d61696c2e756262692e636f6d2e62722f7664526663326174742f726
6633261747461636832302e646c6c
>
>Ie. change the type 4 (mail file) to type 0 (all file
types).   Save and
>re-load clamd....
>
>Now re-send the ecard to yourself... is it detected
now?
>
>If it does work then it looks like amavisd-new separates
the headers
>from the body...and then uses clamd to scan the body
ONLY... which might
>be no type 4 (Email.) would ever work?   But I'm sure
you would have
>noticed.
>=====
>
>and without any changes to the amavisd.conf file,
amavisd-new now 
>correctly identifies the malware and quarantines the 
>message.  Mark/Steve, how would you suggest we handle
these going forward?
>
>Thanks,
>
>Bill

Bill,

Amavisd-new by default unpacks the mail and virus scans the

parts.  This is intended behavior because of historically
poor mime 
support in some commercial virus scanners.

You can adjust keep_decoded_original_maps to include MAIL
so the raw 
message is also provided to clam, or $bypass_decode_parts
which 
affects banned filename matching.  See the comments in
amavisd.conf-sample.

-- 
Noel Jones 


------------------------------------------------------------
-------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and
take
control of your XML. No limits. Just data. Click to get it
now.
http://sourcefor
ge.net/powerbar/db2/
_______________________________________________
AMaViS-user mailing list
AMaViS-userlists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user

AMaViS-FAQ:http://www.amav
is.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/ho
wto/

Bill,

> >>  [
qr'^(Email|HTML).(Phishing|Spam|Scam[a-z0-9]?).'i =>
0.1 ],
> >>  [ qr'^(Email|Html).Malware.Sanesecurity.' 
  => undef],
> >>  [ qr'^(Email|Html)(.[^.,
]*)*.Sanesecurity.' => 0.1 ],
> >>  [ qr'^(MSRBL-Images/|MSRBL-SPAM.)'   =>
0.1 ],

> It's setup this way because that's the way you have it
shown in the
> amavisd.conf-default file that comes with the distro

I have it that way, because I wanted to have the:

  ^(Email|Html).Malware.Sanesecurity.

treated as a virus, and not as a spam.

The rule stands above the
  ^(Email|Html)(.[^., ]*)*.Sanesecurity.
rule, which would have matched on such name too.

So my intention is to let Email|Html  *       .Sanesecurity
be spam, except for       Email|Html .Malware .Sanesecurity


> If it does work then it looks like amavisd-new
separates the headers
> from the body...and then uses clamd to scan the body
ONLY...

Yes, as always, except when some decoder declares it is
unable to decode, or if keep_decoded_original_maps
matches,
in this case AV scanner would also see the complete mail,
in addition to each decoded part.

  Mark

------------------------------------------------------------
-------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and
take
control of your XML. No limits. Just data. Click to get it
now.
http://sourcefor
ge.net/powerbar/db2/
_______________________________________________
AMaViS-user mailing list
AMaViS-userlists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user

AMaViS-FAQ:http://www.amav
is.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/ho
wto/

At 04:49 PM 5/18/2007, Bill Landry wrote:
>Well, this was not a good solution:
>
>keep_decoded_original_maps = (new_RE(
>   qr'^MAIL$',   # retain full original message for
virus checking (can
>be slow)
>
>as this had the effect of quarantining everything that
SaneSecurity and
>MSRBL detected, including spam, phish, image, scam,
etc., and not just
>malware.  :-(
>
>Any other suggestions?  Thanks,

I think your virus_name_to_spam_score_maps is somehow 
wrong.  Scanning the raw email doesn't change the result
text from clamdscan.

-- 
Noel Jones 


------------------------------------------------------------
-------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and
take
control of your XML. No limits. Just data. Click to get it
now.
http://sourcefor
ge.net/powerbar/db2/
_______________________________________________
AMaViS-user mailing list
AMaViS-userlists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user

AMaViS-FAQ:http://www.amav
is.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/ho
wto/

At 05:28 PM 5/18/2007, Bill Landry wrote:
>Noel Jones wrote the following on 5/18/2007 3:14 PM
-0800:
> > At 04:49 PM 5/18/2007, Bill Landry wrote:
> >
> >> Well, this was not a good solution:
> >>
> >> keep_decoded_original_maps = (new_RE(
> >>   qr'^MAIL$',   # retain full original message
for virus checking (can
> >> be slow)
> >>
> >> as this had the effect of quarantining
everything that SaneSecurity and
> >> MSRBL detected, including spam, phish, image,
scam, etc., and not just
> >> malware.  :-(
> >>
> >> Any other suggestions?  Thanks,
> >>
> >
> > I think your virus_name_to_spam_score_maps is somehow
> > wrong.  Scanning the raw email doesn't change the
result text 
> from clamdscan.
> >
> >
>Other than score changes, it's identical to what can be
found in the
>amavis.conf-default included with the distro:
>
>virus_name_to_spam_score_maps =
>   (new_RE( [
qr'^(Email|HTML).(Phishing|Spam|Scam[a-z0-9]?).'i =>
1.5 ],
>            [ qr'^(Email|Html).Malware.Sanesecurity.'
         => undef ],
>            [ qr'^(Email|Html)(.[^.,
]*)*.Sanesecurity.'       => 1.5 ],
>            [ qr'^(MSRBL-Images/|MSRBL-SPAM.)'         
         => 1.5 ],
>   ));
>
>Let me know if you see anything wrong here...

I don't see any obvious problems above.  You don't happen to
have 
virus_name_to_spam_score_maps defined more than once,
do you?

maybe Mark has other suggestions.

-- 
Noel Jones 


------------------------------------------------------------
-------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and
take
control of your XML. No limits. Just data. Click to get it
now.
http://sourcefor
ge.net/powerbar/db2/
_______________________________________________
AMaViS-user mailing list
AMaViS-userlists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user

AMaViS-FAQ:http://www.amav
is.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/ho
wto/

At 05:37 PM 5/18/2007, Noel Jones wrote:
>I don't see any obvious problems above.  You don't
happen to have
>virus_name_to_spam_score_maps defined more than once,
do you?

and here's what works for me:

virus_name_to_spam_score_maps =
   (new_RE( # [
qr'^(Email|HTML).(Phishing|Spam|Scam[a-z0-9]?).'i =>
0.1 ],
            [ qr'^(Email|Html).Malware.Sanesecurity.'    
   => undef ],
            [ qr'^(Email|Html)(.[^., ]*)*.Sanesecurity.' 
   => 15.1 ],
            [
qr'^(Email|Html).(Hdr|Img|ImgO|Bou|Stk|Loan|Cred|Job|Dipl|D
oc)
                  (.[^., ]*)* .Sanesecurity.'x => 10.1
],
            [ qr'^(MSRBL-Images/)'   => 2.1 ],
            [ qr'^(MSRBL-SPAM.)'   => 5.1 ],
   ));

-- 
Noel Jones 


------------------------------------------------------------
-------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and
take
control of your XML. No limits. Just data. Click to get it
now.
http://sourcefor
ge.net/powerbar/db2/
_______________________________________________
AMaViS-user mailing list
AMaViS-userlists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user

AMaViS-FAQ:http://www.amav
is.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/ho
wto/

Bill,

> I've noticed that when multiple message parts match
different clamav
> signatures, *all* the signature names must be listed
in
> virus_name_to_spam_score_maps for it to be considered
spam.

Yes, as documented in RELEASE_NOTES:

  [...] When a virus scanner returns
  names of viruses, and all provided names are matched by
the
  virus_name_to_spam_score_maps, and no other virus
scanner has
  anything more sinister to report, then a message is _not_
flagged
  as a virus, but a corresponding spam score is contributed
to other
  spam results [...]

This is a key issue here.

Your test example after enabling /^MAIL$/ (which requests
that
a full message is passed to virus scanners, besides each
decoded
part), clamd starts to report _two_ malware names.

As the 'Phishing.Email' was not in your virus_name_to_spam_score_maps
list, such mail did not fulfill the requirement that _all_
reported
names must be in the list for the result to be turned into
spam,
so you ended up with a quarantined 'virus'.

> So, amavisd-new splits of the headers into a temporary
file called
> email.001 (for example) and the body into a temporary
file called email.002
> (for example)

Not entirely true. There is never a part that would only
contain
a mail header. Each mail part (i.e. a temporary file to be
passed
to each virus scanner) contains either a decoded MIME part
or an
archive component of a mail, or the entire mail (if /^MAIL$/
is
in keep_decoded_original_maps, or if some decoder
declares
it can not do its job properly, e.g. due to a corrupted or
password-protected archive).

 
Steve, thanks for your help in understanding the matter!


  Mark

------------------------------------------------------------
-------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and
take
control of your XML. No limits. Just data. Click to get it
now.
http://sourcefor
ge.net/powerbar/db2/
_______________________________________________
AMaViS-user mailing list
AMaViS-userlists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user

AMaViS-FAQ:http://www.amav
is.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/ho
wto/

At 09:55 PM 5/21/2007, Bill Landry wrote:
>Mark, can you tell me why the Email.Malware are still
not detected
>without enabling /^MAIL$/?  I would like to keep virus
scan processing
>to a minimum, but if I disable /^MAIL$/, then
Email.Malware messages are
>not detected.

Most of the Email.Malware signatures are "email"
type 
signatures.   Clamav must be presented with a file
recognizable as an 
email (Received: headers and other clues) for these
signature to even 
be checked.

You must always present clamav with raw email files to use
all the 
published signatures.  In addition to the SaneSecurity
add-on 
signatures, most of the "official" clam Phish
signatures are "email" 
type, along with several official trojan & worm
signatures.

If you don't set amavisd-new to scan the full email message,
you 
effectively disable all signatures requiring an email
message.

-- 
Noel Jones 


------------------------------------------------------------
-------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and
take
control of your XML. No limits. Just data. Click to get it
now.
http://sourcefor
ge.net/powerbar/db2/
_______________________________________________
AMaViS-user mailing list
AMaViS-userlists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user

AMaViS-FAQ:http://www.amav
is.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/ho
wto/

Bill,

> Okay, then is there any reason to have amavisd-new
break e-mail messages
> up for individual parts scanning?

Several virus scanners are not able to decode a MIME
structure,
or do a poor job at it, or can not decode certain types of
archives or encodings. Carefully decoding by amavisd can
also
protect virus scanners from mail bombs, e.g. recursive
archives.

MIME and archive decoding also provides information about
mail structure and its components to banning rules (file
names,
file types, mime types), and to a bad MIME-header check.

> Would it make sense to disable parts scanning and just
have
> amavisd-new only pass the entire raw message to clamd
for scanning?

If you trust your virus scanner, and don't need extra
information for banning rules, then sure, you may disable
decodings by amavisd.

Either disable decoders/dearchivers indvidually by
adjusting decoders list, or turn them off altogether
by setting  $bypass_decode_parts=1;

Now with 2.5.1-pre1, the $bypass_decode_parts=1 also
disables MIME decoding by MIME::Parser, and implicitly
enables passing of a complete mail to virus scanners,
which is what you are asking for.

  Mark

------------------------------------------------------------
-------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and
take
control of your XML. No limits. Just data. Click to get it
now.
http://sourcefor
ge.net/powerbar/db2/
_______________________________________________
AMaViS-user mailing list
AMaViS-userlists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user

AMaViS-FAQ:http://www.amav
is.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/ho
wto/

Bill,

> I'm running 2.5.1-pre1 and for testing I have set the
following relevant
> parts in my amavisd.config:
>
> $bypass_decode_parts = 1;
>
> $keep_decoded_original_re = 0;
> #keep_decoded_original_maps =
($keep_decoded_original_re);
> #decoders = ( ... );

If you have $bypass_decode_parts=1, there is no need to
also
disable decoders or to worry about keep_decoded_original_maps,
the $bypass_decode_parts overrules decoders, and the
keep_decoded_original_maps becomes irrelevant ....
... but it doesn't hurt to disable them if you like.

> I appears to be working as planned.
...
> Thanks Mark, I'll let you know if I see anything
strange since making
> the upgrade and amavisd.config changes.

Thanks for trying it out, feedback is welcome!

  Mark

------------------------------------------------------------
-------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and
take
control of your XML. No limits. Just data. Click to get it
now.
http://sourcefor
ge.net/powerbar/db2/
_______________________________________________
AMaViS-user mailing list
AMaViS-userlists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user

AMaViS-FAQ:http://www.amav
is.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/ho
wto/