RE: botnets: web servers, end-systems and Vint Cerf

> I've concluded three things (by doing experiements like

> that).  (a) Where
> there are Windows boxes, there are zombies. 
"Securing 
> Microsoft operating
> systems adequately for use on the Internet" is not
a solved problem in
> computing.  

I disagree. Since 1994 I have been in the habit of setting
up MS Windows
boxes with Win98 and up, by installing from CD, connecting
to the net
and installing various patches and updates from the Windows
Update
service. I've never had a virus infection, a bot, a root kit
or
whatever. The secret is simple. These machines never
connected directly
to the Internet but went through a NAT box. Way back when it
was a
FreeBSD machine running TIS Firewalls Toolkit. These days it
is an
off-the-shelf Ethernet switch with DSL modem and NAT
built-in.

Therefore, I assert that securing systems adequately for use
on the
Internet is indeed a SOLVED PROBLEM in computing. However,
it isn't yet
solved in a social or business sense. On the business side,
I wonder why
PC's don't come with a built-in firewall/NAT device. It is
cheap enough
to do these days. This means that a computer would have no
Ethernet
ports on it. Instead, an internal Ethernet port would be
directly
connected to a NAT/firewall device on the same circuit board
(or via
PCI/PCMCIA/etc.). The external Ethernet port would belong to
the
firewall/NAT device. On the social side, people don't
realize that such
a solution is possible and therefore they aren't demanding
computer
vendors to build it in. The box vendors only build what the
OS vendors
want and the OS vendors are not interested in a piece of
hardware that
runs its own OS, most likely FreeBSD or Linux.

In the UK, companies who sell TV services (cable and
satellite) give
there customers a box to connect with. Why can't ISPs also
sell their
services with a proper box included? By proper, I mean a
NAT/firewall,
not a USB-connected DSL modem. 

> (c) Amusingly, it's possible
> to detect new end-user allocations and service rollouts
by noting when
> spam starts to arrive from them.  (e.g. the Verizon
FIOS 
> deployment, if I
> may use hostnames of the form *.fios.verizon.net as a
guide, is going
> well in NYC, Dallas, DC, Tampa, Philly, LA, Boston and

> Newark, but lags
> behind in Seattle, Pittsburgh, Buffalo and Syracuse.)

I wonder if Verizon is violating any SEC rules by not
reporting this
information publicly? This is a good example of something
that would not
be revealed if they provided a NAT/firewall box to every
customer who
didn't already have one.

Has anyone implemented a tool that ISPs could use to detect
whether or
not a NAT/firewall device is present? Perhaps based on OS
fingerprinting? Or even based on an agent that must be
installed by the
customer? If such tools are available then an ISP could
offer customers
a discount for being compliant with a NAT/firewall rule in
their
contract.

--Michael Dillon

* michael.dillonbt.com (michael.dillonbt.com)
[Fri 16 Feb 2007, 17:31 CET]:
[..]
>Therefore, I assert that securing systems adequately for
use on the 
>Internet is indeed a SOLVED PROBLEM in computing.

A HUNDRED MILLION machines beg to differ.


	-- Niels.

-- 

On Fri, 2007-02-16 at 16:27 +0000, michael.dillonbt.com
wrote:
> 

> Has anyone implemented a tool that ISPs could use to
detect whether or
> not a NAT/firewall device is present? Perhaps based on
OS
> fingerprinting?

Yes, p0f detects when hosts are behind a nat gateway.

-- 
Daniel J McDonald, CCIE # 2495, CISSP # 78281, CNX
Austin Energy
http://www.austinenergy.c
om

I really don't want to get into an OS debate here, but this
does
have major operational impact, so I will anyway but will be
as
brief as possible.  Please see second (whitespace-separated)
section
for some sample hijacked system statistics which may or may
not
reflect overall network population.

On Fri, Feb 16, 2007 at 04:27:55PM -0000, michael.dillonbt.com
wrote:
> I disagree. [...]
> 
> Therefore, I assert that securing systems adequately
for use on the
> Internet is indeed a SOLVED PROBLEM in computing.
However, it isn't yet
> solved in a social or business sense. 

I think I understand your point about the social and
business sense of the
problem; if so, then we're probably in at least rough
agreement on that.
People do stupid things with computers (like reading email
with a web
browser, or replying to spam) and it's proven to be very
difficult to
convince them to stop doing those things.

I'm reminded of Ranum's point (from
http://www.ranum.com/security/computer_security/
editorials/dumb/ ) about
how if user education was going to work...it would have
worked by now.
I think the ongoing success of phishing operations,
including those run
by illiterate amateurs, in face of massive publicity via
nearly every
communications channel society has to offer, illustrates it
nicely.

But, and this may be where we disagree, it's not solved
where Microsoft
operating systems are concerned -- and I don't accept the
notion that
just putting such systems behind a firewall/NAT box is
adequate.
(I'll also argue that any OS which *requires* an external
firewall
to survive more than a few minutes' exposure is unsuitable
for use
on the Internet.  *Not good enough*.)

But suppose you put such a firewall in place.  You'll need
to
configure the firewall properly -- paying as much attention
to
outbound rules as inbound.  (And how many people ever do
that?  Even
on corporate networks, there are still people stunningly
incompetent
enough to use default-permit policies on outbound traffic. 
And
controlling outbound traffic from these systems is arguably
more
important than controlling inbound -- inbound likely only
abuses
the owner, outbound abuses the entire Internet.)

You'll need to add anti-virus software.  And anti-spyware
software.
Then you need to make sure the "signature"
databases for both of those
are updated early and often, keeping in mind that you have
now elected
to play a game that you will inevitably lose the first time
that new
malware propagates faster than the keepers of those
databases can develop
and distribute signatures.  Vegas lives for suckers like
this.

And you'll need to de-install IE and Outlook, since
everything else you've done will be defeated as soon as the
next
IE/Outlook-remotely-exploitable-and-leading-directly-to-
full-system-compromise-here's-a-working-demo is published
on
full-disclosure, which should be, oh, about three hours from
now.

And this is before we even get to the licensing and DRM
backdoors
*designed into* Vista.

Something which requires this much work just to make it
through its
first day online, while being used by J. Random Person, is
hopelessly
inadequate.  Which is why systems like this are routinely
compromised in
huge numbers.  Which is why we have a large-scale problem on
our hands.




Which brings me to the second point, and that is skepticism
over the
100M ballpark figure that's been bandied about.  Personally,
I wouldn't
even blink if someone produced convincing proof that the
real number
was 300M.  I think that's completely plausible --
"plausible" but still,
I very much hope, unrealistically high.  So from my point of
view, this
100M stuff is old news -- i.e., I'm telling you the ocean is
wet.

A tiny example: some data (summarized below) from a small
experiment last
month using a single test mail server.  I threw away all the
data blocked
outright by the firewall in front of it.  I threw away all
data that didn't
involve connections directed at port 25.  I threw away all
the data for
connecting hosts without rDNS.  I threw away all the data
for connecting hosts
with rDNS that looked even vaguely server-like.  I threw
away repeat visits.
All of which means that my sampling method is akin to waving
a thimble in
a hurricane and will thus provide a gross (and likely
skewed) underestimate.

This left me with >1.5M observed hosts seen in a month. 
They're all sending
spam.  (How do I know?  Because 100% of the mail traffic
sent to that
server is spam.)  And they're all running Windows, except
for a handful
which aren't or which were indeterminate.  Note that rDNS
lookups were
from local long-lived cache, so rDNS may be well out-of-date
in some cases.

Some random examples:

	41.241.32.87	dsl-241-32-87.telkomadsl.co.za
	89.28.3.133	89-28-3-133.starnet.md
	190.49.152.243	190-49-152-243.speedy.com.ar
	218.178.50.40	softbank218178050040.bbtec.net
	200.171.123.83	200-171-123-83.dsl.telesp.net.br
	74.132.179.31	74-132-179-31.dhcp.insightbb.com
	61.246.79.101	dsl-del-dynamic-101.79.246.61.airtelbroadband
.in
	71.177.11.75	pool-71-177-11-75.lsanca.fios.verizon.net
	82.53.186.23	host23-186.pool8253.interbusiness.it
	200.181.9.48	200-181-9-48.gnace702.dsl.brasiltelecom.net.br

	190.64.51.225	r190-64-51-225.dialup.adsl.anteldata.net.uy
	91.124.188.160	160-188-124-91.pool.ukrtel.net
	201.212.228.70	201-212-228-70.cab.prima.net.ar
	81.192.49.178	adsl-178-49-192-81.adsl.iam.net.ma
	72.74.126.140	pool-72-74-126-140.bstnma.east.verizon.net
	83.110.220.148	dxb-b17260.alshamil.net.ae
	62.42.65.225	62.42.65.225.dyn.user.ono.com
	90.5.6.84	apoitiers-156-1-103-84.w90-5.abo.wanadoo.fr
	70.113.76.57	cpe-70-113-76-57.austin.res.rr.com
	70.224.195.25	ppp-70-224-195-25.dsl.applwi.ameritech.net
	24.17.158.50	c-24-17-158-50.hsd1.mn.comcast.net
	58.69.28.69	58.69.28.69.pldt.net
	61.230.54.51	61-230-54-51.dynamic.hinet.net
	190.76.27.171	190-76-27-171.dyn.movilnet.com.ve
	200.78.237.196	na-200-78-237-196.na.avantel.net.mx
	201.143.64.242	red-corp-201.143.64.242.telnor.net
	87.167.4.103	p57a70467.dip0.t-ipconnect.de
	24.158.153.152	24-158-153-152.dhcp.jcsn.tn.charter.com
	84.158.211.24	p549ed318.dip.t-dialin.net
	86.143.38.135	host86-143-38-135.range86-143.btcentralplus.c
om
	88.153.92.54	bzq-88-153-92-54.red.bezeqint.net
	83.27.13.15	auf15.neoplus.adsl.tpnet.pl
	64.234.17.88	host-64-234-17-88.nctv.com
	70.156.1.90	adsl-156-1-90.bna.bellsouth.net
	83.152.201.100	dyn-83-152-201-100.ppp.tiscali.fr
	189.136.208.229	dsl-189-136-208-229.prod-infinitum.com.mx
	12.210.197.239	12-210-197-239.client.mchsi.com
	195.14.207.192	xdsl-195-14-207-192.netcologne.de
	201.19.177.223	20119177223.user.veloxzone.com.br
	68.184.147.110	68-184-147-110.dhcp.stbr.ga.charter.com
	77.183.169.130	dsbg-4db7a982.pool.einsundeins.de
	83.40.159.62	62.red-83-40-159.dynamicip.rima-tde.net
	124.255.100.4	pppa509.e12.eacc.dti.ne.jp
	75.87.103.180	cpe-75-87-103-180.kc.res.rr.com
	200.247.145.36	145036.fln.virtua.com.br
	75.41.225.22	adsl-75-41-225-22.dsl.chcgil.sbcglobal.net
	24.71.79.23	s01060050ba8b5c7c.ok.shawcable.net

Some totals by north American ISP:

 81828 Comcast
 68794 Verizon
 60716 Roadrunner
 23165 Charter
 23099 Pacbell
 17981 Ameritech
 15855 SWBell
 14801 ATTBI
 13212 Shaw
  9833 Adelphia
  9769 QWest
  7353 Bellsouth

By other ISP:

  73944 rima-tde.net
  60220 wanadoo.fr
  49730 t-dialin.net
  45285 tpnet.pl
  38587 proxad.net
  32800 hinet.net
  32169 telesp.net.br
  26630 telecomitalia.it
  24665 veloxzone.com.br
  20157 t-ipconnect.de
  19547 arcor-ip.net
  19504 interbusiness.it
  18363 bezeqint.net
  17071 ttnet.net.tr
  15330 prod-infinitum.com.mx
  15136 blueyonder.co.uk

By TLD:

 729481 net
 217643 com
 120017 fr
  93806 br
  75782 pl
  75183 it
  61156 de
  42124 jp
  39376 ar
  34422 il

(.edu checks in with only 424, #59 on the list, by the
way.)

Consider what a larger, distributed effort would reveal.
*has* revealed.
And there is no reason to think the numbers are going down. 
There are
a lot of reasons to think the numbers are going up.

Pop quiz: how many of these operations tell their own
customers
"we take the spam problem seriously" while at the
same time
running networks that double as massive spam generation
engines?

	Note: spam is just *one* of the many things that these
systems
	are busily engaged in, and it's by far not the nastiest. 
It just
	happens to be one of the easier things to observe -- a
tell-tale,
	if you will.

Pop quiz, bonus round: how much does it cost Comcast to
defend its
mail servers from Verizon's spam, and vice versa?  Heck, how
much
does it cost Comcast to defend its mail servers from its own
spam?

Pop quiz, extra super special round: can any of these defend
their
networks from a concerted, clueful DDoS attack launched from
thousands
of hosts that are *on their network*?

---Rsk

On Mon, 19 Feb 2007, Rich Kulawiec wrote:
> Pop quiz, bonus round: how much does it cost Comcast to
defend its
> mail servers from Verizon's spam, and vice versa? 
Heck, how much
> does it cost Comcast to defend its mail servers from
its own spam?

How much do they spend on abuse/customer security?  Is it
more or less
than they spend on their mail servers?  Even if they shifted
all the money 
they spent on the mail departments (opex and capex) to their

abuse/customer security departments would it make a
significant 
difference?

Getting money usually isn't that much of a problem, within
reason. 
Figuring out what to spend it on that actually makes a
difference is the 
problem.