send mmap timed out

Hello.

This is a bit of a long shot. I've done a bit of
Google-search, but
that only gives a small number of hits, most of which are
unanswered
questions on mailing lists or message boards.

Our web server occasionally gives the following 'errors' in
the error_log file:

[Mon Apr 23 12:41:40 2007] [info] [client 12.34.56.78] send
mmap timed out

and, more frequently:

[Mon Apr 23 12:41:40 2007] [info] [client 12.34.56.78]
(32)Broken
pipe: client stopped connection before send mmap completed

This usually isn't a problem and it never worried me (it is,
after
all, marked as "info"), until a number of the
former errors, followed
by a number of the latter errors (about 30 of each) caused
the server
to crash. The errors were all related to the same IP address
and, as
far as I can see, this belongs to a genuine user.

I did, however, find in the access log files that their
results
returned httpd status '206' (partial content) and the number
of bytes
on large files, was always a nice rounded number, like 32768
(which is
2^15). It suggests that the problem has to do with the user
downloading only parts of larger files (pdfs/gifs) and, I
would guess,
leaving an httpd process waiting for it to finish? At least
the number
of httpd processes had reached its maximum (which is why the
server
crashed).

But I don't know if my guesses are correct. And even if they
are, what
has really happened (for instance: is this really a problem
with the
client or with the server). And, of course, how to make sure
this
won't happen again.

Martijn

------------------------------------------------------------
---------
The official User-To-User support forum of the Apache HTTP
Server Project.
See <URL:http://htt
pd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribehttpd.apache.org
   "   from the digest: users-digest-unsubscribehttpd.apache.org
For additional commands, e-mail: users-helphttpd.apache.org

On 4/23/07, Martijn <sweetwatergeekgooglemail.com> wrote:

> [Mon Apr 23 12:41:40 2007] [info] [client 12.34.56.78]
send mmap timed out
>
> and, more frequently:
>
> [Mon Apr 23 12:41:40 2007] [info] [client 12.34.56.78]
(32)Broken
> pipe: client stopped connection before send mmap
completed
>
> This usually isn't a problem and it never worried me
(it is, after
> all, marked as "info"), until a number of the
former errors, followed
> by a number of the latter errors (about 30 of each)
caused the server
> to crash. The errors were all related to the same IP
address and, as
> far as I can see, this belongs to a genuine user.
>
> I did, however, find in the access log files that their
results
> returned httpd status '206' (partial content) and the
number of bytes
> on large files, was always a nice rounded number, like
32768 (which is
> 2^15). It suggests that the problem has to do with the
user
> downloading only parts of larger files (pdfs/gifs) and,
I would guess,
> leaving an httpd process waiting for it to finish? At
least the number
> of httpd processes had reached its maximum (which is
why the server
> crashed).

In all likelihood, those messages are only tangentially
related to
your problem. They simply indicate that the client end of
the
connection disappeared, which is a perfectly normal thing
for a busy
webserver. The 206 responses are probably caused by a very
impolite
download accelerator making multiple overlapping requests.
When it has
received what it wants, it drops all the requests, resulting
in the
error messages.

But then you say the server "crashed". What
exactly does that mean?
Did it dump core? Did it simply get very slow? It is not
normal for a
server to "crash" when it hits MaxClients. You
need to be more
detailed about what actually happened.

Joshua.

------------------------------------------------------------
---------
The official User-To-User support forum of the Apache HTTP
Server Project.
See <URL:http://htt
pd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribehttpd.apache.org
   "   from the digest: users-digest-unsubscribehttpd.apache.org
For additional commands, e-mail: users-helphttpd.apache.org

> In all likelihood, those messages are only tangentially
related to
> your problem. They simply indicate that the client end
of the
> connection disappeared, which is a perfectly normal
thing for a busy
> webserver. The 206 responses are probably caused by a
very impolite
> download accelerator making multiple overlapping
requests. When it has
> received what it wants, it drops all the requests,
resulting in the
> error messages.
>
> But then you say the server "crashed". What
exactly does that mean?
> Did it dump core? Did it simply get very slow? It is
not normal for a
> server to "crash" when it hits MaxClients.
You need to be more
> detailed about what actually happened.

Thank you for your reply.

It did not just reach MaxClients, but the number of httpd
processes
stayed at this maximum for hours, until someone (not me)
finally
restarted apache. (Of course, this problem happened on
Sunday night,
so I wasn't there to see what really happened. Or to sort
out things
myself.) Both the access_log file and the error_log file
weren't
written to from the time the aforementioned mmap-info
messages
appeared until apache was restarted hours later. There is,
indeed, no
sign that apache really 'crashed', so it probably, didn't
but the
effect was the same. People who were awake at that time told
me that
they couldn't reach the site, but the server did ping.

I hope this helps.

Cheers.

Martijn

------------------------------------------------------------
---------
The official User-To-User support forum of the Apache HTTP
Server Project.
See <URL:http://htt
pd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribehttpd.apache.org
   "   from the digest: users-digest-unsubscribehttpd.apache.org
For additional commands, e-mail: users-helphttpd.apache.org

On 4/23/07, Martijn <sweetwatergeekgooglemail.com> wrote:
> > In all likelihood, those messages are only
tangentially related to
> > your problem. They simply indicate that the client
end of the
> > connection disappeared, which is a perfectly
normal thing for a busy
> > webserver. The 206 responses are probably caused
by a very impolite
> > download accelerator making multiple overlapping
requests. When it has
> > received what it wants, it drops all the requests,
resulting in the
> > error messages.
> >
> > But then you say the server "crashed".
What exactly does that mean?
> > Did it dump core? Did it simply get very slow? It
is not normal for a
> > server to "crash" when it hits
MaxClients. You need to be more
> > detailed about what actually happened.
>
> Thank you for your reply.
>
> It did not just reach MaxClients, but the number of
httpd processes
> stayed at this maximum for hours, until someone (not
me) finally
> restarted apache. (Of course, this problem happened on
Sunday night,
> so I wasn't there to see what really happened. Or to
sort out things
> myself.) Both the access_log file and the error_log
file weren't
> written to from the time the aforementioned mmap-info
messages
> appeared until apache was restarted hours later. There
is, indeed, no
> sign that apache really 'crashed', so it probably,
didn't but the
> effect was the same. People who were awake at that time
told me that
> they couldn't reach the site, but the server did ping.

Was this just a denial-of-service attack? Did you use a tool
like
netstat to identify the client connections and track down
the origin?

If apache was really "frozen" (not serving a
connection and not
answering new connections, even after the Timeout has
expired), then
you should attach to some process to get a backtrace:

http://httpd.apache.org/dev/debugging.html#backtrace

Joshua.

------------------------------------------------------------
---------
The official User-To-User support forum of the Apache HTTP
Server Project.
See <URL:http://htt
pd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribehttpd.apache.org
   "   from the digest: users-digest-unsubscribehttpd.apache.org
For additional commands, e-mail: users-helphttpd.apache.org