Criminals, The Network, and You

Re-sending due to Merit's minor outage.

- ferg


---------- Forwarded Message ----------


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

- -- Robert Blayzor <rblayzorinoc.net> wrote:

>The fact that they're rejecting on a 5xx error based on
no DNS PTR is a=

bit harsh.  While I'm all for requiring all hosts to have
valid PTR
records, there are times when transient or problem servers
can cause a
DNS lookup failure or miss, etc.  If anything they should be
returning a=

4xx to have the remote host"try again later".
>

Oh, wait till you realize that some of the HTTP returns are
bogus
altogether -- and actually still serve malware.

It's pretty rampant right now. :-/

- - ferg

-----BEGIN PGP SIGNATURE-----
Version: PGP Desktop 9.6.2 (Build 2014)

wj8DBQFGxR1lq1pz9mNUZTMRApQRAKCEOLpuu69A1+B4vCHQTZs+hHLKaACc
D1Ak
9JNwl2i1mL08WNUQSlXBYGM=3D
=3DffuN
-----END PGP SIGNATURE-----


--
"Fergie", a.k.a. Paul Ferguson
 Engineering Architecture for the Internet
 fergdawg(at)netzero.net
 ferg's tech blog: http://fergdawg.blogspo
t.com/

On Wed, Sep 12, 2007 at 03:43:22PM -0600, Jason J. W.
Williams wrote:
> In my opinion, the first and fourth statements are not
necessarily in
> conflict. A reputation system based purely on reverses
is pretty broken.

Actually, it's amazingly reliable, when the patterns
(literal string
matches and regular expressions) are carefully compiled and
cross-checked.

One example of the thousands of patterns I block outright is
*.fios.verizon.net.
I do this (a) because I'm seeing lots of spam from
generically-named hosts
in that subdomain, e.g., within the last hour attempts have
been noted from:

	relay=pool-71-164-205-139.dllstx.fios.verizon.net
[71.164.205.139]
	relay=pool-71-170-104-162.dllstx.fios.verizon.net
[71.170.104.162]
	relay=pool-71-170-70-195.dllstx.fios.verizon.net
[71.170.70.195]
	relay=pool-71-172-239-30.nwrknj.fios.verizon.net
[71.172.239.30]
	relay=pool-71-189-201-76.lsanca.fios.verizon.net
[71.189.201.76]
	relay=pool-71-190-246-40.nycmny.fios.verizon.net
[71.190.246.40]
	relay=pool-72-73-220-84.cmdnnj.fios.verizon.net
[72.73.220.84]
	relay=pool-72-76-240-99.nwrknj.fios.verizon.net
[72.76.240.99]
	relay=pool-72-84-85-199.nrflva.fios.verizon.net
[72.84.85.199]
	relay=pool-72-89-97-187.bflony.fios.verizon.net
[72.89.97.187]
	relay=static-71-183-52-18.nycmny.fios.verizon.net
[71.183.52.18]

(b) because I've yet to receive a single report of a false
positive from
this pattern after using it for a considerable period of
time and (c) because
Verizon seems disinclined to do anything about the problem
other than
have their paid professional spokesliars repeat the usual
B.S., viz.:

	http://www.theregister.co.uk/2
007/09/10/isps_ignore_strorm_worm_and_other_malware/
	ISPs turn blind eye to million-machine malware monster
	By Dan Goodin in San Francisco
	Published Monday 10th September 2007 06:02 GMT

	[...]
	Verizon turned down requests for an interview with a
security
	engineer, but a spokeswoman said officials are aware of
the
	rankings and are working to put new measures in place by
the end
	of the year to curb the spam flowing out of its network.
"We are
	concerned about it," the spokeswoman, Bobbi Hensen,
said. "We
	don't like spam. We are aggressively working on
that."
	[...]

Uh-huh.  This problem was reported to Verizon roughly FIVE
YEARS ago,
and should have, of course, been competently addressed
withing a matter
of days.  It hasn't been.  There is no sign that it will be.
 It's therefore
been necessary for those of us enduring torrential
quantities of
Verizon-originated abuse to take appropriate defensive
actions.

Like rejecting all SMTP traffic from *.fios.verizon.net.

And Verizon is merely one of the offenders -- the article
cited above lists
several others.  I just happened to single them out for use
as an example
here because I found the contrast between their years-long
history of utter
negligence and their officially-stated position to be
particularly striking.
Comcast, Charter, SBCGlobal, Ameritech, Level3, SWBell,
Nextgentel, Pacbell,
and Qwest, just to name a few off the cuff, are equally
culpable.

Anyway: the use of generic rDNS patterns for outright
rejection turns out
to be quite effective with a very low FP rate.

> Regarding the second, you're absolutely right. It's not
your
> responsibility if a 3rd party doesn't have a rDNS entry
(at all or
> non-generic), however the reality is you're going to
have to deal with
> it anyway. If your customers allow you to tell the
senders to buzz off
> and fix it, that's terrific.  However, you're in a more
authoritarian
> (IT-wise) environment than most I would suspect. Also,
you risk hurting
> your customers. As an example, it's not a suitable
answer to our law
> firm customers who are critically-dependent on
receiving e-mail from
> hopelessly broken senders.

Any firm that is critically dependent on email (beyond an
intranet
environment) is being naive and foolish by relying on a
known-unreliable
communications medium.  Connections fail.  DNS breaks. 
Servers croak.
Disks fill.  Poor software is deployed.  And the entire
Internet-wide
infrastructure for mail is under constant stress from spam,
DoS attacks,
misconfiguration, and outright stupidity (e.g. SAV).

As to hopelessly-broken senders, we do not do them any
favors by
accomodating their brokeness.  It is better in the long term
for all
of us to educate them about the not just the de jure, but
the de facto
minimum requirements for mail servers -- which in 2007
include not
only functional rDNS, but rDNS pointing to non-generic
hostnames.

---Rsk

On Tue, 18 Sep 2007, Rich Kulawiec wrote:
> here because I found the contrast between their
years-long history of utter
> negligence and their officially-stated position to be
particularly striking.
> Comcast, Charter, SBCGlobal, Ameritech, Level3, SWBell,
Nextgentel, Pacbell,
> and Qwest, just to name a few off the cuff, are equally
culpable.

They all suck isn't very useful information.  Although
collectively 
they've probably fixed hundreds of thousands of customer
computers over
the years, like bad Boston drivers, there is always more.

Instead of they suck, it might be more useful to highlight
providers of
similar scale which you think do a good job which others
could emulate.


> Anyway: the use of generic rDNS patterns for outright
rejection turns out
> to be quite effective with a very low FP rate.

Some people think that users on dynamic addresses should be
read-only, and
not allowed to operate servers or send messages. Like most
forms of 
red-lining, it tends to become self-fulling.  Websites that
only support 
Internet Explorer probably get very few false positives
because people 
affected are used to working around that or just ignore
them.  Networks
that don't update their Bogon lists probably get very few
false positives
because people learn to work around them or ignore them. 
And so on.

Unlesss accept all those messages from those addresses and
check them, you 
really don't know the false positive rate.  You only know
the 
self-reported complaint rate; which is usually a fraction of
the actual 
false positive rate.

> Instead of they suck, it might be more useful to
highlight 
> providers of similar scale which you think do a good
job 
> which others could emulate.

How about some smarter statistics. Instead of counting the
spam emails
from Network X, count the spam sources and divide that by
the number of
end user customers (or hosts) in Network X. By doing this
you get a
clearer picture of who is cleaning their house, and who is
letting it
slide.

Think of a messy house. You say that there were 8 dirty
plates in the
living room, on the floor, the sofa the coffee table.
Horrible, right?
Not if there are 8 people living in the house. In that case
it
represents one evening of laziness, going to bed without
cleaning up
first. But if only one person lives in the house and there
are no
guests, then the 8 dirty plates represent a big mess.

Whenever you scale up anything, small nits also grow in
absolute
magnitude. The small scale operator who ignores the nits is
following
the same practices as the large scale operator who ignores
the nits. If
there are lots of nits, I want to know if the large scale
operator
should be criticised for not adjusting their processes to
deal with
scaling up, or whether somebody really is being incompetent.
There are
different remedies to the two situations. Scaling issues can
be solved
by paying attention, education, installing
tools/products/services. But
incompetence generally requires replacing people, especially
management
who allow the incompetence to thrive.

> Unlesss accept all those messages from those addresses
and 
> check them, you really don't know the false positive
rate.  
> You only know the self-reported complaint rate; which
is 
> usually a fraction of the actual false positive rate.

Yes. It is tempting to take numbers at their face value, but
I find that
whenever somebody has an axe to grind, their numbers are
based on flawed
reasoning or measuring the wrong things.

--Michael Dillon

On Tue, Sep 18, 2007 at 02:42:43PM -0400, Sean Donelan
wrote:
> Instead of they suck, it might be more useful to
highlight providers of
> similar scale which you think do a good job which
others could emulate.

A first-order examination of the data suggests that Cox may
be
doing something pro-active.  The number of IP addresses on
their
network engaged in recent spam delivery attempts is
considerably
less than that from others, e.g.:

	Comcast:   1240
	Verizon:   1594
	Cox:	25

Of course, that could be due to the relative sizes of the
networks
involved, but a second-order examination of the same data
suggests
something more: the abuse-emitting IP addresses on Cox's
network do not
appear to show up repeatedly over long periods of time. 
Contrast with
those on Comcast and Verizon, where it's quite common to see
the same ones
in the logs for days/weeks/months.  This suggests to me that
Cox
is actually paying attention to abuse outbound from their
network
and is either disconnecting or quarantining hosts which emit
it.

> Some people think that users on dynamic addresses
should be read-only, and
> not allowed to operate servers or send messages. Like
most forms of 
> red-lining, it tends to become self-fulling.  

Of course, I've never suggested any such thing.  Users of
such ISPs
should normally be using their own ISPs outbound mail
server(s),
a solution which is perfectly adequate for the overwhelming
majority
of users.  And it should be no problem for them to use other
mail
servers, using SMTP AUTH (with TLS or SSL) or via HTTP. 
But
the days when direct-to-MX traffic is acceptable from all
addresses
are as gone as those when operation of an open SMTP relay
was acceptable.

Moreover, those who wish to operate servers (and whose ISPs
permit
that operation per their TOS) should have no difficulty in
having
the ISP assign them non-generic DNS/rDNS, and/or assign them
static
address space which is reserved for such users.

> Unlesss accept all those messages from those addresses
and check them, you 
> really don't know the false positive rate.  You only
know the 
> self-reported complaint rate; which is usually a
fraction of the actual 
> false positive rate.

Actually, I know quite a bit more than that.  But since this
is NANOG
and not an anti-spam discussion list, I elected not to delve
into the
rather lengthy details of the methodology used to ascertain
the FP rate.
But *briefly* and glossing heavily, when a particular IP
address (with
generic rDNS, let's say):

	- fails to wait for the SMTP greeting
	- fails to send a QUIT
	- fails to retry when given a deferral response
	- retries immediately when given a deferral response
	- attempts delivery to an MX that hasn't been an MX for
years
	- attempts delivery to a domain whose MX record hasn't
		been pointed to this MX for years
	- HELOs as ebay.com (or similar)
	- HELOs as a non-existent domain
	- HELOs as a very-recently-registered domain
	- HELOs as something different each time it connects
	- HELOs as *my* MX or a domain handle by it
	- attempts to send mail with a putative sender paypal.com (or similar)
	- attempts delivery repeatedly with different putative
senders/different
		putative sender domains
	- attempts to send mail with putative senders whose domain
does
		not exist or does not resolve
	- attempts to send mail with a putative senders whose
domain has
		A or MX records which resolve to invalid network space
	- attempts to send mail with a putative sender whose
domain
		has very suspicious DNS records [1]
	- attempts to send mail from a putative sender using
		a known-spammer-owned domain
	- attempts to send mail from a putative sender using
		a domain which resolves to DROP-listed space. [2]
	- attempts to send mail from a putative sender using
		a very-recently-registered domain
	- attempts to send mail from a putative sender using
		*my* domain
	- attempts delivery to spamtraps
	- attempts delivery to never-existed addresses
	- attempts delivery to one-off addresses available only in
SMTP
		reject notices
	- attempts delivery of messages whose headers contain
known
		spamware signatures
	- attempts to send mail whose body-part contains URIs
which
		match well-maintained lists of spammer-controlled URIs
	- has already been noted by other independent observers as
		attempting spam delivery to their MX's
	- passive-OS-fingerprints as running Windows
	- is listed in the A or NS records of a known spammer
domain
		(see [1] again)
	- has been noticed carrying out other attacks, e.g.,
attempting
		exploitation via HTTP, SSH, etc.
	- and so on
	
or multiple of the above (as is the case most of the time),
then it's
very, very unlikely that refusal of the traffic constitutes
a FP.

---Rsk


[1] A recent example: segron.com, queried yesterday (a
query
today will likely return different values, BTW):

	segron.com      a       62.214.228.21
	segron.com      a       75.47.248.183
	segron.com      a       79.113.34.148
	segron.com      a       79.120.16.19
	segron.com      a       80.133.250.133
	segron.com      a       81.198.35.132
	segron.com      a       85.179.60.176
	segron.com      a       89.110.23.181
	segron.com      a       89.178.60.48
	segron.com      a       89.212.0.195
	segron.com      a       121.137.164.24
	segron.com      a       121.200.140.244
	segron.com      a       218.254.186.186
	segron.com      a       221.126.244.121
	segron.com      a       221.127.158.128

which resolve to:

	i3ED6E415.versanet.de
	adsl-75-47-248-183.dsl.lsan03.sbcglobal.net
	79-113-34-148.rdsnet.ro
	p5085FA85.dip.t-dialin.net
	e179060176.adsl.alicedsl.de
	pppoe-181.23.110.89-adsl.spbnit.ru
	89-178-60-48.broadband.corbina.ru
	89-212-0-195.dynamic.dsl.t-2.net
	244.140.200.121.megaegg.ne.jp
	cm218-254-186-186.hkcable.com.hk
	(5 addresses yield NXDOMAIN)

Clearly, segron.com's "web site" is being hosted
on hijacked systems.

[2] See http://www.spamhaus.org
/DROP/ for details, but the gist is
that this short and very carefully maintained list
enumerates network
space which is either hijacked or 100% spammer-controlled or
both.

On Wed, 19 Sep 2007, Rich Kulawiec wrote:
> in the logs for days/weeks/months.  This suggests to me
that Cox
> is actually paying attention to abuse outbound from
their network
> and is either disconnecting or quarantining hosts which
emit it.

Its nice to see Cox getting some praise for a change.  Last
month people 
were castigating it for being too agressive at trying to
block Bots.
It seems like half the net is always criticizing ISPs for
doing
too little and half the net is always criticizing ISPs for
doing
too much.

Cox blocks a lot of ports on its network (25, 80, 135-139,
445, 1900,
1433, 1434, 1900, subseven ports); blackholes networks and
DNS names;
firewall software that blocked sites with bad TCP software
stacks such
as Craigslist; and so on.  Some people think Cox is being
pro-active
on the security front; other people think Cox is violating a
sacred
trust.  ISPs are pretty much just damned.

Why should an network user have to petition his or her ISP
to authorize
their use of a valid network protocol?  Shouldn't
application
authorization occur at the application level instead of
relying on
the equivalent of .rlogin network-level checks.

Companies like DynDNS show there is user demand to operate
their own
servers (including P2P servers, mail servers, web servers,
dns servers, 
etc) on dynamic IP addresses without needing a special
"static" IP address 
or different in-addr.arpa name.

With Fast-Flux, it looks like the next network port that
should be 
blocked on broadband/dialup connections is DNS tcp/udp 53.

> or multiple of the above (as is the case most of the
time), then it's
> very, very unlikely that refusal of the traffic
constitutes a FP.

Until a false positive happens.  I see 1-2 false positives a
year
using checks for "generic-looking" in-addr.arpa
names; and a few more 
false positives for IP addresses without in-addr.arpa names.

Nevertheless I still continue to use those checks because
the false 
positive rate is below my pain threshold.  But I don't
pretend it never 
happens or may not be a concern to someone else.

I also almost never get a valid e-mail to my postmaster
account, just
spam; but some people still think every mail server should
accept mail
to the postmaster account anyway no matter how rarely it
gets legitimate
email.  They even set up RBLs of mail servers without
postmaster accounts. 
Maybe we need a RBL of mail servers that don't accept mail
from generic 
in-addr.arpa or dynamic IP addresses.

On Thu, Sep 20, 2007 at 01:31:41PM -0400, Sean Donelan
wrote:
> Why should an network user have to petition his or her
ISP to authorize
> their use of a valid network protocol? 

Because many (most?) ISPs have done such a poor job of
controlling SMTP
abuse outbound from their networks over the past decade that
it's now
a best practice to consider all mail from generic
hostnames/dynamic
IP space highly suspect -- at best.

Those ISPs have repeatedly proven over many years that they
aren't capable
of detecting and squelching SMTP abuse sources on their own
networks; [1] this
leaves everyone else with a choice: either (a) put up with
it or (b) devise
measures to stuff a sock in it.  And (a) simply isn't
tenable for mail
servers receiving abuse in torrential quantities.

If any of those ISPs are unhappy with the choice of tactics
encompassed
by (b) then perhaps they should have anticipated that
unhappiness years
ago when they were first alerted to this problem.  Had they
taken even
rudimentary steps to solve it (instead of merely having
their spokesdroids
repeat the bare-faced lie that they "take the spam
problem seriously")
then perhaps it would not have been necessary for others to
devise
methods to deal with their failures.

If any network user is unhappy (and I can easily see why
they would be),
then he or she should take that up with their ISP, since
it's quite
likely that their own ISP has been a contributor to the
problem.

> Companies like DynDNS show there is user demand to
operate their own
> servers (including P2P servers, mail servers, web
servers, dns servers, 
> etc) on dynamic IP addresses without needing a special
"static" IP address 
> or different in-addr.arpa name.

That model is no longer viable, unfortunately.  I wish that
weren't the
case, but the combination of ISP and end-user negligence
along with mass
hijacking of end-user systems has rendered it so.

> They even set up RBLs of mail servers without
postmaster accounts. 
> Maybe we need a RBL of mail servers that don't accept
mail from generic 
> in-addr.arpa or dynamic IP addresses.

You are certainly free to set up a DNSBL or RHSBL using any
listing
criteria you wish, but please be aware that if you set up
one using
that particular criteria, anyone using it will likely be
refusing a LOT
of valid mail, including that of some very large
organizations, since
(as I said above) blocking such traffic has long since been
established
as a best practice.  There are multiple DNSBLs, RHSBLs, and
static
lists which enumerate such hosts; for example, consider the
Spamhaus PBL:

	http://www.sp
amhaus.org/pbl/index.lasso

which relies in part on input from the ISPs themselves, and
is one
of the zones included in the comprehensive "zen"
DNSBL zone published
by Spamhaus.

---Rsk

[1] I still adhere to the quaint/outdated/antique concept
that everyone
is responsible for making sure that their networks are not
an operational
hazard to everyone else's networks, and that they should
plan, budget,
staff, build, operate and train accordingly.