SendGate: Sendmail Multiple Vulnerabilities (Race Condition DoS, Memory Jumps, Integer Overflow)

Tech details:
Sendmail vulnerabilities were released yesterday. No real
public
announcements to speak of to the security community.

SecuriTeam released some data:
"Improper timeout calculation, usage of memory jumps
and integer
overflows allow attackers to perfom a race condition DoS on
sendmail, and
may also execute arbitrary code."
More here: h
ttp://www.securiteam.com/unixfocus/5RP0L0UI0S.html

ISS only reported the Race Condition (DoS?). The Sendmail
Advisory
reported the Race Condition DoS, the Memory Jumps and a
"theoretical" Integer Overflow.

To begin with, anyone noticed the memory leak they
(Sendmail) silently
patched?
I wonder how many other unreported silently-patched
vulnerabilities are out there?

Second, the Integer Overflow is practical, not theoretical.

ISS reported the Race Condition last mounth. There is NO
data available on
when the other vulnerabilities were discovered. Any guesses?

They also patched many non-security related bugs, added
checks and more
informative error messages, etc.

Sendmail is, as we know, the most used daemon for SMTP in
the world. This
is an International Infrastructure vulnerability and should
have been
treated that way. It wasn't. It was handled not only
poorly, but
irresponsibly.

Here's what ISS releasing the Race Condition vulnerability
has to say:
http://xfo
rce.iss.net/xforce/alerts/id/216
They say it's a remote code execution. They say it's a
race condition. No
real data available to speak of. I can't see how it's
remotely
exploitable, but well, no details, remember? From what we
can see it seems
like a DoS.

Bottom line
-----------
What they did behind the smoke-screen is replace a lot of
setjmp() and
longjmp() functions (not very secure ones at that) with
goto's
(interesting choice).
They changed the logic of the code, replaced everything that
calculated
timeout. Anything that calculated something and returned a
value now
returns a boolean result, when previously they just returned
void. They
used to look at the content rather than success.

The int overflow is possibly exploitable, not very sure
about the
jumps. No idea why ISS says the Race Condition is, would
love insight.

Public announcement
-------------------
FreeBSD were the only ones who released a public
announcement of a patch
and emailed it to bugtraq so far.

The patches
-----------
The FreeBSD patch much like the sendmail.org patch is very
long,
complicated and obscure. The release was made along with a
ton of other
patches for FreeBSD. Go figure what's in there.

Sendmail.com's patch is so big they may as well have
re-released the whole
program.

There are also patches available for other *nix systems, no
distributions
released updates yet.

Sendmail's announcement
-----------------------
Obscure. Not worth any other comments other than the ones
above.

CVE information
---------------
CVE-2006-0058 (reserved)

Commentary
----------
One could say ISS and Sendmail did good, obscuring the
information so that
the vulnerability-to-exploit time will be longer. That
proved wrong,
useless and pointless. They failed.

After looking at the available data for 30 minutes (more or
less), we know
exactly what the vulnerabilities are. Exploiting them may
not be that
trivial if indeed possible,  but there are most likely
already exploits
out there if it is. When will the first public POC be
released? Your guess
is as good as mine.
Not to mention the silently patched memory leak.

SMTP and Sendmail by extension are critical for the Internet
as an
International Infrastructure. If this ends up being
exploitable (no
details, remember?) both ISS and Sendmail should look good
and hard at the
coming massive exploitation of Sendmail servers.

With issues relating to the Internet Infrastructure I'd be
willing to go
even with the evil of non-disclosure, as long as something
gets done and
then reported publically when it finally scaled down in a
roll-back after
a couple of years.
If not, and you are going to make it public, make the effort
and fix it as
soon as you can, and give information to help the process of
healing. Don't do it a mounth late and obscure data.

It took Sendmail a mounth to fix this. A mounth.

A mounth!

With such Vendor Responsibility, perhaps it is indeed a Good
Thing to go
Full Disclosure. It seems like history is repeating itself
and Full
Disclosure is once again not only a choice, but necessary to
make vendors
become responsible.

I wish we could somehow avoid all the guys who will
inevitably shout in
the press "end of the world". The Internet is,
was and will stay
havoc. There will be exploitation. Those who care about
security will be
patched, those that don't will hopefully finally learn a
lesson. The
Internet won't die because of this, although email may
suffer ? but we are
used to that by now, even when losing money.

I am so very angry the details are obscure and hidden in the
way they are,
especially as that is useless in this case. Why did they do
it, to claim
they are ?responsible?? Too late.

"The avalanche has already started. It is too late for
the pebbles to
vote." - Kosh, Babylon 5.

How are they to show open source is reliable if this is how
they act? They
hurt the cause. If they don't know how to handle something
like this, they
should ask for help.

What, if it's not reported to Microsoft, there is no reason
to be
?responsible??

It's like annoying "fake porn" on TV. Either
show the nudity and rate the
program accordingly or stay suitable for normal viewing.
There is no
eating the cake and leaving it whole.

"Hey mom, what's my root password? I forgot"
"Dunno, just use the new sendmail
vulnerability!"

They should learn from Apache. With such a critical
vulnerability I know
the Apache guys would not have slept until it is patched!

We will update on the situation if required on http://blogs.securiteam.c
om

This text can be found
here: ht
tp://blogs.securiteam.com/index.php/archives/363

	Gadi Evron.

On Thu, 23 Mar 2006 03:41:52 CST, Gadi Evron said:

(I feel obligated to mention that there's 16 mentions of my
name in the Sendmail
release notes, and zero of Gadi's.  This of course
influences my opinions and
commentary, and possibly Gadi's as well...)

> ISS only reported the Race Condition (DoS?). The
Sendmail Advisory
> reported the Race Condition DoS, the Memory Jumps and a
> "theoretical" Integer Overflow.

Well, it *is* mostly a theoretical overflow - for it to
work, a site would have to:

a) Override the default value of "O
MaxHeadersLength=32768" to something in the
billions *and*

b) Be running on a box with enough RAM/swap to allow a call
to malloc(2**31) or
so to succeed.

Then the attacker actually has to (c) *send* some 2 gig of
headers.

Well, OK.  If you can find somebody still running something
pre-8.10.0 (2000/03/01),
you can bypass (a) - still need (b) and (c) though...

> To begin with, anyone noticed the memory leak they
(Sendmail) silently
> patched?

I suspect it's not as silently as you think.  If you
actually bothered to ask
Claus, he'll probably be able to tell you which
RELEASE_NOTES entry the leak
was attached to, or why it wasn't listed (see below).

And it's hard for them to be *too* silent, given that
anybody and their pet
llama Ralphie can get the 8.13.5 and 8.13.6 sources and diff
them.

There's also the *practical* aspect.  Well over a quarter
of the actual code
changes were due to the API change of *one* function.  You
start change-logging
all this in the release notes, important things become
easier to overlook...

> I wonder how many other unreported silently-patched
> vulnerabilities are out there?

It depends how you define "silently patched
vulnerabilities".  I've found my
share of Sendmail bugs - and 100% of the ones I've reported
that ever shipped
in a non-beta version are in the RELEASE_NOTES file.  (Those
bugs others and I
swatted in new, as-yet-unshipped code in alpha and beta
versions are *not* in
there - 8.12's multiple queue support didn't just spring
forth essentially
bug-free, for instance.  Also, bugs found and fixed by
Sendmail Inc employees
aren't listed - although if reported from outside, they
will get a "problem
noted by" entry).

A bigger problem is that quite often, what gets fixed is a
simple *bug* (for
instance, two I found in 8.12.2).  Are they vulnerabilities?
 Good question.
Some of the "bugs" I've found could possibly be
extended into a DoS (for instance,
what I *reported* in 8.12.2 was spontaneous queue-runner
hangs due to losing a
timer event - but that *could* be crafted into a DoS if an
attacker found a way
to control when things happened to force a timer event
loss).

So was that a "silently patched vulnerability"?
Or just a plain bug?

> Second, the Integer Overflow is practical, not
theoretical.

You have an actual exploit that works on a config that
isn't especially set
up to allow it to work (i.e. out of the box defaults, and a
non-absurd amount
of network traffic)?  Or can you at least point out enough
details of what
the exploit would have to do to convince me it's not just
hand-waving?

In particular, how do you get past the MaxHeadersLength
check?

> ISS reported the Race Condition last mounth. There is
NO data available on
> when the other vulnerabilities were discovered. Any
guesses?

The code drop for the fix of your "practical integer
overflow" happened between
the Alpha0 on Dec 23 and Alpha1 on Feb 12.  You'll have to
ask either Phil
Brass or Claus when that one got reported.

> Sendmail.com's patch is so big they may as well have
re-released the whole
> program.

I don't know what shipped to sendmail.com's paying
customers, but in fact,
sendmail.org *did* "re-release the whole
program", as a quick check of
ftp://ftp.sendmail.org/pub/sendmail/ would have revealed....

And in fact, if you actually *diff* the two trees to create
a patch, it's not
that large at all:

(numbers fudged very slightly to remove the diff of 2
versions of a Postscript
documentation file (doc/op/op.ps), which by itself was well
over half the total
changes):

% diff -ur sendmail-8.13.[56] | wc -l
5253
% diff -ur sendmail-8.13.? | diffstat | grep change
58 files changed, 1411 insertions(+), 886 deletions(-)
% wc sendmail-8.13.6/*/*.[ch] | grep total
 122092  394315 2780756 total

Yeah. Whatever. Hell of a lot of code churn there.  Almost
2%.  A very large
chunk of it looking like:

 -4546,10
+4569,12  
        {       
                if (bitset(MCIF_INHEADER,
mci->mci_flags))
                {
-                       putline("", mci);
+                       if (!putline("", mci))
+                               goto writeerr;
                        mci->mci_flags &=
~MCIF_INHEADER;
                }
-               putline("<<< No Message
Collected >>>", mci);
+               if (!putline("<<< No Message
Collected >>>", mci))
+                       goto writeerr;
                goto endofmessage;
        }

(and another 104 similar fixes of putline alone - that *by
itself* is about 30%
of the code changes).

But of course, I'm writing this having actually *looked* at
the diff, and having
bothered to ask Claus and Greg what the policy was for
listing fixes in the release
notes...

Also, it would help if instead of FUD-mongering, you
actually went to Claus
(or asked somebody else to) with *specific suggestions* of
how to improve the
process.  He may be stubborn about the way he does things,
but if you include
specific changes, and show how those changes make *visible*
improvements to
the product, he'll listen.

(visible improvements - saying "It Would Be Nice
If" doesn't cut it.  On the other
hand, replacing the IWBNI with "If X were done, then
the security community would
be able to do Y, and the network operations community could
do Z, with benefits
A, B, and C" - now *that* might get some traction...)

On March 23, 2006 01:41 am, Gadi Evron wrote:
> Here's what ISS releasing the Race Condition
vulnerability has to say:
> http://xfo
rce.iss.net/xforce/alerts/id/216
> They say it's a remote code execution. They say it's
a race condition. No
> real data available to speak of. I can't see how it's
remotely
> exploitable, but well, no details, remember? From what
we can see it seems
> like a DoS.

ISS's Mark Dowd is very clever guy. And if duke says it's
exploitable
I would believe him .  It's an
interesting new vector anyway.

But like all timing related attacks, the question is
reliability.
Though gossip has it, this one is repeatable with sub-100
attempts
and you get infinite shots at it because even if the process

does die it's a child of the parent listener. (So it is not
really
a DoS per se in any case.)

>
> Bottom line
> -----------
> What they did behind the smoke-screen is replace a lot
of setjmp() and
> longjmp() functions (not very secure ones at that) with
goto's
> (interesting choice).

Smoke screen seems like unfarily loaded terminology to use.

OpenBSD fixed (removed) many setjmp/longjmp functions in
their
tree a long time ago as a class of bugs. (Though this
sendmail 
exploitable collecttimeout() longjmp one is new and they
patched
it yesterday with everyone else, because as you noted,
replacing
it was kinda hairy...)

I don't think its fair to bitch about people fixing bugs
and then not
having the time to send out advisories for every little
tweak.
The important thing is to fix the bug. And often times the 
developer won't understand the real impact of fixing a bug 
until someone clever like Mark comes up with some innovative
way to exploit an "unexploitable" bug like this
one.

What will be interesting to see when the PoC exploits are 
finally released, is if any of the memory/stack protection
schemes
mitigate it.

<humor>
Besides, there is only one true mailer to mail them all,
and its name is Postfix.
</humor>

Now if we could only convince Mr. Venema to switch 
to a BSD license _everyone_ would switch to Postfix
and everything would be much better. If it weren't for
that "poison pill" clause in its license, I'm
sure most
OSes and commercial systems would have swapped 
out Sendmail for Postfix long ago.

cheers,
--dr

-- 
World Security Pros. Cutting Edge Training, Tools, and
Techniques
Vancouver, Canada    April 3-7 2006     http://cansecwest.com
pgpkey http://dragos.com/
kyxpgp

On Thu, 23 Mar 2006 Valdis.Kletnieksvt.edu wrote:
> Also, it would help if instead of FUD-mongering, you
actually went to Claus
> (or asked somebody else to) with *specific suggestions*
of how to improve the
> process.  He may be stubborn about the way he does
things, but if you include
> specific changes, and show how those changes make
*visible* improvements to
> the product, he'll listen.
> 
> (visible improvements - saying "It Would Be Nice
If" doesn't cut it.  On the other
> hand, replacing the IWBNI with "If X were done,
then the security community would
> be able to do Y, and the network operations community
could do Z, with benefits
> A, B, and C" - now *that* might get some
traction...)


No offense Valdis, you know I both like you and consider you
a friend,
but if you (sendmail) can't take the heat and/or stand up
to the task of
being International Infrastructure, step down.

This isn't about processes, it's about something that has
been around for
a while, many reply on and keeps ******* up. Where it simply
can't.

	Gadi.

On Thu, 23 Mar 2006 03:41:52 -0600 (CST), Gadi Evron
<gelinuxbox.org>
wrote:


> It took Sendmail a mounth to fix this. A mounth.
> 
> A mounth!
> 
> With such Vendor Responsibility, perhaps it is indeed a
Good Thing to go
> Full Disclosure. It seems like history is repeating
itself and Full
> Disclosure is once again not only a choice, but
necessary to make vendors
> become responsible.
> 

Given the scope of the changes you describe -- you wrote
"Sendmail.com's
patch is so big they may as well have re-released the whole
program."
-- I can't get upset at taking a month to fix it.  You're
dealing with
asynchronous events, which are really hard to start with.  I
suspect
that they spent some time deciding how to fix it -- you
don't appear
thrilled with their choice, but I don't know what other
options they
considered -- and then actually tested the new code.  Given
how many of
our security problems are due to buggy and
inadequately-tested code, I
suspect that taking a month was actually being quite
responsible.

		--Steven M. Bellovin, http://www.cs.columbi
a.edu/~smb

On Thu, 23 Mar 2006 19:28:16 CST, Gadi Evron said:
> No offense Valdis, you know I both like you and
consider you a friend,
> but if you (sendmail) can't take the heat and/or stand
up to the task of
> being International Infrastructure, step down.

Heat we can take.  Whining we can't.

Did you have a *specific* suggestion as to how it could be
done better, or
did you just feel the need to flame about how it was
handled?

Specific things that you did *not* consider, as far as I can
tell:

1) You don't, as far as I know, contribute anything to
paying the salaries
of the people actually doing the coding, and who know the
way the code works.
This means that a choice has to be made:

a) Handle it in a way that doesn't upset the people paying
the bills, even if
the people who aren't paying the bills don't like it.

b) Finding somebody else to pay the bills.

c) Dumping it and finding a project that *does* pay the
bills, and fix it
as a hobby rather than a full-time job.

d) Dump it, move on, and let somebody else who doesn't know
the code as well
do it, possibly worse.

So Gadi, what do you suggest we do, keeping in mind that the
guy who's actually
doing the work needs to pay his rent and buy groceries?

2) You're complaing about the huge size of the patch, *and*
the fact it took a
month to get it done.  If you want it faster, you can either
have less bugs
fixed, or less testing. Choose one. (Also, 2% code churn
between releases is
*nothing*. Take a look at the Linux kernel sometime - it has
a *far* higher
churn rate between releases that Linus is trying to keep on
a 3-month
schedule....)

Steve Bellovin is right - these asynchronous events are a
*pain* to debug,
because the human brain doesn't deal with race conditions
very well.  I
mentioned a timer event issue in a previous note - that one
took Claus and I a
good *six months* to debug and understand, and resulted in
all of 3 or 4 lines
of code.  It was one that was not reproducible on command,
only tripped a few
times a month, and by its nature never left any really
useful state wreckage
behind.  Trying to attach a debugger or adding debugging
instrumentation to the
code would change the timing, and as a result scare the bug
back into hiding...

Yes, it could potentially have been 

On March 23, 2006 06:08 pm, Steven M. Bellovin wrote:
> On Thu, 23 Mar 2006 03:41:52 -0600 (CST), Gadi Evron
<gelinuxbox.org>
>
> wrote:
> > It took Sendmail a mounth to fix this. A mounth.
> >
> > A mounth!
>
> Given the scope of the changes you describe -- you
wrote "Sendmail.com's
> patch is so big they may as well have re-released the
whole program."
> -- I can't get upset at taking a month to fix it. 
You're dealing with
> asynchronous events, which are really hard to start
with.  I suspect
> that they spent some time deciding how to fix it -- you
don't appear
> thrilled with their choice, but I don't know what
other options they
> considered -- and then actually tested the new code. 
Given how many of
> our security problems are due to buggy and
inadequately-tested code, I
> suspect that taking a month was actually being quite
responsible.

Hey I'm the first guy to line up to knock Sendmail a.k.a.
the pit of
infinite flaws... but lets not beat people up who don't
deserve it.

My understanding of it was they found out from ISS and fixed
it 
ASAP. (I'm sure they read this list, maybe they would care
to comment)
They took two weeks to update their customers and released
it to CERT
who was supposed to take two weeks - starting three days
ago... but something
happened (as it always does  and it was
sent out prematurely - widely.

Hence again some people got caught off guard.
I'm increasingly forming the opinion that waiting for
patches on disclosure
actually does harm. (Though I conceed that this is a
religious issue
likely never to be resolved.)

Responsible disclosure may in fact be immediate disclosure
so 
that people can measure impact without waiting for the
inevitable 
vague wordings. That way we can take any countermeasures
possible 
immediately - rather than being vulnerable in the silence
while 
"special" people know. Resulting in not doing
anything or being 
increasingly watchful as would be warranted. If you know
your 
mailserver is vulnerable but you can't fix it you can
always 
start to watch it like a hawk - if you know.

I guess it depends on your security posture. If you are
agressive 
on security you want information dispersed as widely as
possible.
If you are putting in minimal effort, then the less people
know 
the better it is for you.

Though only time will tell, I would also bet that this is
not the last 
sendmail signal handler issue we will see... at least until
the more 
Postfix-like commercial-only next version comes out... and
then we 
will have a brand new code base full of untested code to
deal with.
And since its taken a few decades to stabilize Sendmail up
to
_this_ point.... I'll just hug my Postfix code and resolve
to buy
Mr. Venema beer at the first available opportunity .

cheers,
--dr

P.s. I still maintain that the right solution is to convince
IBM to tweak
the Postfix license so those who absolutely need fully open
source 
can use it instead. I've personally audited a lot of that
code and know
how incredibly robust, nee paranoid it is. . Because
when Sendmail
X comes out the open source folks will have a big issue to
deal with.

-- 
World Security Pros. Cutting Edge Training, Tools, and
Techniques
Vancouver, Canada    April 3-7 2006     http://cansecwest.com
pgpkey http://dragos.com/
kyxpgp

> I wonder how many other unreported silently-patched
> vulnerabilities are out there?

You seem to be inferring that it is a bad thing to silently
patch bugs which may have security implications. The OpenBSD
team makes a habit of auditing software for flaws and fixing
them without waiting to find out whether they create actual
security vulnerabilities. They consider this to be a GOOD
thing.

I think that people who use software also consider it to
be good for software flaws to be fixed as quickly as
possible.
Inevitably, this means that if the DEVELOPERS discover a 
flaw, they will fix it before they tell anyone about it. The
reason that security researchers publish bulletins about
security flaws is because they are unable to fix them 
either due to lack of skill, or more commonly, they just 
don't have permission to commit changes to the source code.

Network operators are users of software and not developers,
therefore most network operators are happy when flaws are
fixed early and often.

--Michael Dillon

Steven M. Bellovin wrote:
> On Thu, 23 Mar 2006 03:41:52 -0600 (CST), Gadi Evron
<gelinuxbox.org>
> wrote:
> 
> 
> 
>>It took Sendmail a mounth to fix this. A mounth.
>>
>>A mounth!
>>
>>With such Vendor Responsibility, perhaps it is
indeed a Good Thing to go
>>Full Disclosure. It seems like history is repeating
itself and Full
>>Disclosure is once again not only a choice, but
necessary to make vendors
>>become responsible.
>>
> 
> 
> Given the scope of the changes you describe -- you
wrote "Sendmail.com's
> patch is so big they may as well have re-released the
whole program."
> -- I can't get upset at taking a month to fix it. 
You're dealing with
> asynchronous events, which are really hard to start
with.  I suspect
> that they spent some time deciding how to fix it -- you
don't appear
> thrilled with their choice, but I don't know what
other options they
> considered -- and then actually tested the new code. 
Given how many of
> our security problems are due to buggy and
inadequately-tested code, I
> suspect that taking a month was actually being quite
responsible.

I'd usually agree, compared to a year and a half with
Microsoft or 3 
years with Oracle.

The point here, though, if that the patch was released
almost with no 
notification _to_the_security_community_ (bugtraq, fd,
etc.). It was 
obfuscated (open source, funny notion) and released.
Exploits are 
already out there.

When you are critical infrastructure, you have higher
responsibility. 
You either practice non-disclosure and patch your users
over-time, then 
disclose, or simply disclose. It depends on needs and/or how
responsive 
the vendor is.

One can't have it both ways, unfortunately.

	Gadi.

Valdis.Kletnieksvt.edu wrote:
> Well, it *is* mostly a theoretical overflow - for it to
work, a site would have to:

Exploit is out there. How long did that take?

	Gadi.