Fighting bogus news spam

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi everyone

I've been using Bogofilter for some time now and it has been
working
great, only occasionally letting through one out of couple
of hundred
spam mails I get per day.

Recently however I've been getting around 20 mails per day
that contain
a short bogus news item in the subject and body. These beat
bogofilter
with ease since they do not contain any of the usual spam
keywords. Also
every mail seems to be different, so they also resist new
bogofilter
training.

See below for some examples of such mails.

I've been wondering if anyone has found a reliable way of
fighting this
new annoyance.

Thanks
Toma¾ ©olc


Examples (each mail contains one such statement in subject
and one in
body plus an URL). They are plain-text mails with random
user agents.

Hurricane strikes Lousiana, thousands dead
US issues official threat to China over trade surplus
"I Won't Raise Taxes," Says Schwarzenegger,
"except For The Indians."
Thousands of MSN passwords stolen, security compromised
...
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org


iD8DBQFIiz57sAlAlRhL9q8RAiMaAJ9iEROTkTM5/zll/FhO3hjyiXjIHgCg
mZ19
2aABH359vbhhFsD5GU06nzE=
=ukCa
-----END PGP SIGNATURE-----
_______________________________________________
Bogofilter mailing list
Bogofilterbogofilter.org
http://www.bogofilter.org/mailman/listinfo/bogofilter

On Sat, 26 Jul 2008 17:10:51 +0200 Tomaž Šolc
<tomaz.solctablix.org>
wrote:

> Recently however I've been getting around 20 mails per
day that
> contain a short bogus news item in the subject and
body. These beat
> bogofilter with ease since they do not contain any of
the usual spam
> keywords. Also every mail seems to be different, so
they also resist
> new bogofilter training.
> 
> See below for some examples of such mails.
> 
> I've been wondering if anyone has found a reliable way
of fighting
> this new annoyance.


I think the thing to do is to patiently keep training.
Bogofilter trains
on the whole message, including headers, not just the
message body, and
eventually it will pick up on originating IP address and the
like.
Unless every single one of these emails comes from a
different PC,
never duplicated, Bogofilter will get them in time,

-- 
 All the best,
 John
_______________________________________________
Bogofilter mailing list
Bogofilterbogofilter.org
http://www.bogofilter.org/mailman/listinfo/bogofilter

On Sat, Jul 26, 2008 at 05:10:51PM +0200, Toma¸ ¦olc wrote:

Hi,

> I've been using Bogofilter for some time now and it has
been working
> great, only occasionally letting through one out of
couple of hundred
> spam mails I get per day.
> 
> Recently however I've been getting around 20 mails per
day that contain
> a short bogus news item in the subject and body. These
beat bogofilter
> with ease since they do not contain any of the usual
spam keywords. Also
> every mail seems to be different, so they also resist
new bogofilter
> training.

I currently see some of them in my unsure folder with a 0.9x
rating. Until
now I just continue to feed them as usual but if those mail
will continue
to be propagated on larger scale one might thing about it
again.

AFAIR there were some waves with quotes from classical
literature to poison
statitical filters in the past but so far bogofilter was
able to cope with
it for me. At least I don't receive daily newsletters with
such stuff.

Maybe backup your DB file and just feed the stuff to find
out if it works
or not.

Sven
-- 
If God passed a mic to me to speak
I'd say stay in bed, world
Sleep in peace
   [The Cardigans - 03:45: No sleep]
_______________________________________________
Bogofilter mailing list
Bogofilterbogofilter.org
http://www.bogofilter.org/mailman/listinfo/bogofilter

On Sat, 26 Jul 2008 21:45:36 +0200 "Sven Hoexter"
<sventimegate.de>
wrote:

> I currently see some of them in my unsure folder with a
0.9x rating.

What?????

I count anything with a spamicity of over 0.6 as spam. Why
have you got
your spam-cutoff set so high?

I never get false positives, if that's what you're worried
about,

-- 
 All the best,
 John
_______________________________________________
Bogofilter mailing list
Bogofilterbogofilter.org
http://www.bogofilter.org/mailman/listinfo/bogofilter

I have never gotten bogofilter to work in my Postfix-based
setup, but what
has been so far 100% effective is greylisting. Assuming
spammers get no
smarter, it should work just as well for some time to come.

It's quite simple and like many of the martial arts, it uses
the
attacker's own style of attack against them. All email is
matched against
a database of known good senders (this is the training
aspect, but it
happens continually and without user action): mismatches get
returned to
the sending server with a request to retry. On the retry and
on all
subsequent deliveries the mail is passed without delay.
Since spammers
don't hang around for those kinds of niceties, all their
junk drops on the
floor unseen.

I'd like to engage bogofilter as well, but this will do
until it gets
easier or I get quicker in the uptake.
_______________________________________________
Bogofilter mailing list
Bogofilterbogofilter.org
http://www.bogofilter.org/mailman/listinfo/bogofilter

I have never gotten bogofilter to work in my Postfix-based
setup, but what
has been so far 100% effective is greylisting. Assuming
spammers get no
smarter, it should work just as well for some time to come.

It's quite simple and like many of the martial arts, it uses
the
attacker's own style of attack against them. All email is
matched against
a database of known good senders (this is the training
aspect, but it
happens continually and without user action): mismatches get
returned to
the sending server with a request to retry. On the retry and
on all
subsequent deliveries the mail is passed without delay.
Since spammers
don't hang around for those kinds of niceties, all their
junk drops on the
floor unseen.

I'd like to engage bogofilter as well, but this will do
until it gets
easier or I get quicker in the uptake.
_______________________________________________
Bogofilter mailing list
Bogofilterbogofilter.org
http://www.bogofilter.org/mailman/listinfo/bogofilter

On Sat, Jul 26, 2008 at 09:00:43PM +0100, John G Walker
wrote:
> 
> 
> On Sat, 26 Jul 2008 21:45:36 +0200 "Sven
Hoexter" <sventimegate.de>
> wrote:
> 
> > I currently see some of them in my unsure folder
with a 0.9x rating.
> 
> What?????
> 
> I count anything with a spamicity of over 0.6 as spam.
Why have you got
> your spam-cutoff set so high?

I delete everything else (0.98 and up) without a backup. And
yes sometimes
people are able to end up in my unsure folder with a very
high rating.
For me it works. If you backup spam it might be reasonable
for you to 
set a low cutoff value. All in all I think that's something
of personal
preference and this is my private, works at home, setup.


Sven
-- 
If God passed a mic to me to speak
I'd say stay in bed, world
Sleep in peace
   [The Cardigans - 03:45: No sleep]
_______________________________________________
Bogofilter mailing list
Bogofilterbogofilter.org
http://www.bogofilter.org/mailman/listinfo/bogofilter

On Sat, Jul 26, 2008 at 03:19:54PM -0700, paulpaulbeard.org wrote:

> I have never gotten bogofilter to work in my
Postfix-based setup, but what
> has been so far 100% effective is greylisting. Assuming
spammers get no
> smarter, it should work just as well for some time to
come.

Well bogofilter wasn't developed to work at the SMTP level
like greylisting.
The approach is slightly different. 

Beside that I remember a paper from a few years ago where
two guys build
some smtp level filtering with qmail and bogofilter. I don't
know if someone
ever tried it but something like this should be also
possible with a postfix
policy service or through the milter interface.


> I'd like to engage bogofilter as well, but this will do
until it gets
> easier or I get quicker in the uptake.

It works like a charm and isn't that hard. Maybe you should
open another
thread and describe how you'd like to deploy bogofilter and
where you
failed so far. Just because the list is mostly silent that
doesn't mean
you're not allowed to disturb the silence. 

Sven
-- 
If God passed a mic to me to speak
I'd say stay in bed, world
Sleep in peace
   [The Cardigans - 03:45: No sleep]
_______________________________________________
Bogofilter mailing list
Bogofilterbogofilter.org
http://www.bogofilter.org/mailman/listinfo/bogofilter

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi

> I currently see some of them in my unsure folder with a
0.9x rating. 
> Until now I just continue to feed them as usual but if
those mail 
> will continue to be propagated on larger scale one
might thing about 
> it again.

I've just checked the spamicity for the mails I'm getting
and they range
from 0.5 to 0.9. I use two-state filtering with default
cut-off value
(0.99).

I don't like the idea of lowering the cut-off. With the
amount of spam
I'm getting it's impossible to go through the spam box
manually and
check for false positives. I do know I get some occasionally
even with
this setting because two or three times in the last year I
found a mail
I was expecting in the spam box.

By the way, I'm using bogofilter with procmail and constant
training (-u
option).

Maybe it's time I switch to three-state filtering? I didn't
set this up
in the first place because I didn't saw a particular benefit
in this.
You have to read through both inbox and unsure folders
anyway, so I
don't see why this is better than just having everything in
inbox.

> AFAIR there were some waves with quotes from classical
literature to 
> poison statitical filters in the past but so far
bogofilter was able 
> to cope with it for me. At least I don't receive daily
newsletters 
> with such stuff.

I had pretty much the same experience with those.

Best regards
Toma¸
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org


iD8DBQFIjF7ksAlAlRhL9q8RAkRHAJ4klwzYD4IShdVD9XVbj+/enFiBkwCg
2XYe
PrpQ0TEiuosOZwVJKXm6YN0=
=/i26
-----END PGP SIGNATURE-----
_______________________________________________
Bogofilter mailing list
Bogofilterbogofilter.org
http://www.bogofilter.org/mailman/listinfo/bogofilter

On Sun, 27 Jul 2008 13:41:24 +0200
Tomaž Šolc wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Hi
> 
> > I currently see some of them in my unsure folder
with a 0.9x
> > rating. Until now I just continue to feed them as
usual but if
> > those mail will continue to be propagated on
larger scale one might
> > thing about it again.
> 
> I've just checked the spamicity for the mails I'm
getting and they
> range from 0.5 to 0.9. I use two-state filtering with
default cut-off
> value (0.99).
> 
> I don't like the idea of lowering the cut-off. With the
amount of spam
> I'm getting it's impossible to go through the spam box
manually and
> check for false positives. I do know I get some
occasionally even with
> this setting because two or three times in the last
year I found a
> mail I was expecting in the spam box.
> 
> By the way, I'm using bogofilter with procmail and
constant training
> (-u option).
> 
> Maybe it's time I switch to three-state filtering? I
didn't set this
> up in the first place because I didn't saw a particular
benefit in
> this. You have to read through both inbox and unsure
folders anyway,
> so I don't see why this is better than just having
everything in
> inbox.
> 
> > AFAIR there were some waves with quotes from
classical literature
> > to poison statitical filters in the past but so
far bogofilter was
> > able to cope with it for me. At least I don't
receive daily
> > newsletters with such stuff.
> 
> I had pretty much the same experience with those.
> 
> Best regards
> Tomaž

Greetings Tomaž,

I've been using "-u" with 3-state filtering since
bogofilter became
able to do that.  It works nicely so long as you train
bogofilter with
_every_ error (both false positives and false negatives) and
with
unsures.  However it will _not_ prevent the occasional
error.

I do get a false positive every once in a while.  Usually
it's when
I've first subscribed to a new mailing list or made
reservations at an
airline or a hotel I've not used before.  Over the years
bogofilter has
learned that most of the incoming html email is spam and has
to be told
that airlines and hotels are good, not bad.

This month my mailserver has received approx 45,000 spam and
has had 10
false negatives and 45 unsures (with 22 being ham and 23
being spam).
I'm not aware of any false positives, though some _could_ be
hidden
among the 45,000 spam :-<

Regards,

David
_______________________________________________
Bogofilter mailing list
Bogofilterbogofilter.org
http://www.bogofilter.org/mailman/listinfo/bogofilter

On Sun, Jul 27, 2008 at 01:42:55PM +0200, Toma¸ ¦olc wrote:

> I just found this article about how to set up something
similar on Exim4
> I'm using. So I might try this out.
> 
> Have you seen any side effects? Like people getting
bounced mail back?
> 
> htt
p://www.debian-administration.org/articles/167

You will have the effect of having a lot of mails delayed.
If that's ok for
you it will help you to fight spam. It's not the final
solution and in my
experience more and more spammers retry to fight greylisting
but it still has
its share in the race.

Beside that a lot of people expect mail to be an instant
communication channel
without delays. If you've some of them as customers they
might get angry when
they find out about the greylisting stuff. ;)

Sven
-- 
If God passed a mic to me to speak
I'd say stay in bed, world
Sleep in peace
   [The Cardigans - 03:45: No sleep]
_______________________________________________
Bogofilter mailing list
Bogofilterbogofilter.org
http://www.bogofilter.org/mailman/listinfo/bogofilter

On Sun, Jul 27, 2008 at 01:41:24PM +0200, Toma¸ ¦olc wrote:

Hi,

> I don't like the idea of lowering the cut-off. With the
amount of spam
> I'm getting it's impossible to go through the spam box
manually and
> check for false positives. I do know I get some
occasionally even with
> this setting because two or three times in the last
year I found a mail
> I was expecting in the spam box.
> 
> By the way, I'm using bogofilter with procmail and
constant training (-u
> option).
> 
> Maybe it's time I switch to three-state filtering? I
didn't set this up
> in the first place because I didn't saw a particular
benefit in this.
> You have to read through both inbox and unsure folders
anyway, so I
> don't see why this is better than just having
everything in inbox.

Well who knows what I 've missed in the spams I skiped but
so far I'm not
aware of having missed something.

The adavantage of the three-state filtering I see is that I
can read through
my inbox and mailinglist folders without getting distracted
by spam I've to
sort out. Once a day or when I miss something I'll take a
look at the content
in the unsure folder and sort them for training.
I'm not using the automatic training so far and I'm happy
with the result.

Maybe the results would be even better with the automatic
training? Hm should
try that one day.


> > AFAIR there were some waves with quotes from
classical literature to 
> > poison statitical filters in the past but so far
bogofilter was able 
> > to cope with it for me. At least I don't receive
daily newsletters 
> > with such stuff.
> 
> I had pretty much the same experience with those.

Strange but it's kinda hard to compare your mail traffic and
your database
with mine for such an individual system.

Sven
-- 
If God passed a mic to me to speak
I'd say stay in bed, world
Sleep in peace
   [The Cardigans - 03:45: No sleep]
_______________________________________________
Bogofilter mailing list
Bogofilterbogofilter.org
http://www.bogofilter.org/mailman/listinfo/bogofilter

On Sun, July 27, 2008 12:43 am, Sven Hoexter wrote:

> It works like a charm and isn't that hard. Maybe you
should open another
> thread and describe how you'd like to deploy bogofilter
and where you
> failed so far. Just because the list is mostly silent
that doesn't mean
> you're not allowed to disturb the silence. 
>

I have tried to find out how to do it, but I think everyone
else here is
sufficiently more advanced that I can't keep up.
_______________________________________________
Bogofilter mailing list
Bogofilterbogofilter.org
http://www.bogofilter.org/mailman/listinfo/bogofilter

On Sun, 2008-07-27 at 14:41 +0200, Sven Hoexter wrote:
> The adavantage of the three-state filtering I see is
that I can read through
> my inbox and mailinglist folders without getting
distracted by spam I've to
> sort out. Once a day or when I miss something I'll take
a look at the content
> in the unsure folder and sort them for training.

I've taken to doing more than tri-state.  I use the built-in
tri-state
filtering, but then in my mail client, I also separate the
spam by
spamicity.  Anything higher than 0.94 is considered high
probability and
I rarely if ever check it beyond very cursory subject line
scan before
deleting it.  Anything higher than 0.98 is automatically
dropped without
ever seeing it by bogofilter-milter (sending server will get
a bounce
message at SMTP time).  Those spams less than 0.94 I only
read sender
and subject lines before deleting them.  Unsures I usually
briefly page
through the content.  Rarely does ham ever end up in the
unsures, but it
does happen from time to time.  I never get false positives
in the spam
though.

> I'm not using the automatic training so far and I'm
happy with the result.
> Maybe the results would be even better with the
automatic training? Hm should
> try that one day.

I'd imagine it would.  It's served me well.

Tom


_______________________________________________
Bogofilter mailing list
Bogofilterbogofilter.org
http://www.bogofilter.org/mailman/listinfo/bogofilter