Unclean restarts

We are running Apache 1.3.34.

On our Debian 4.0 system, each night cron.daily runs
logrotate.
Logrotate's postrotate script runs `/etc/init.d/apache
reload` (no
errors on stderr). When it is successful, top shows all
apache children
go <defunct>, then apache restarts. But more often
than not, all but 2
or 3 apache children die (without top ever showing
<defunct>), the pid
file is removed. Those 2 or 3 apache children continue to
live and
accept connections, but won't serve any requests. Since the
pid file has
been removed, we have to issue a `killall apache`, then
`/etc/init.d/apache start`

The net effect is pretty fatal. If we're not around when it
happens,
Apache remains effectively down until we come around to
clean it up.

Any help or guidance would be greatly appreciated.

Thanks!


-- 


Brian J. Cohen
Intercarve Networks, LLC
914.552.1274 [cell]

------------------------------------------------------------
---------
The official User-To-User support forum of the Apache HTTP
Server Project.
See <URL:http://htt
pd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribehttpd.apache.org
   "   from the digest: users-digest-unsubscribehttpd.apache.org
For additional commands, e-mail: users-helphttpd.apache.org

Some additional info -- when Apache is running normally (for
example,
during the middle of the day), we can issue a manual
'/etc/init.d/apache
reload' and it will work fine. The problems seem to arise
some
percentage of the time when it tries to run from
cron.daily's calling of
logrotate/postrotate.

Thanks--

Brian J. Cohen
Intercarve Networks, LLC
914.552.1274 [cell]


Brian Cohen wrote:
> We are running Apache 1.3.34.
> 
> On our Debian 4.0 system, each night cron.daily runs
logrotate.
> Logrotate's postrotate script runs `/etc/init.d/apache
reload` (no
> errors on stderr). When it is successful, top shows all
apache children
> go <defunct>, then apache restarts. But more
often than not, all but 2
> or 3 apache children die (without top ever showing
<defunct>), the pid
> file is removed. Those 2 or 3 apache children continue
to live and
> accept connections, but won't serve any requests. Since
the pid file has
> been removed, we have to issue a `killall apache`,
then
> `/etc/init.d/apache start`
> 
> The net effect is pretty fatal. If we're not around
when it happens,
> Apache remains effectively down until we come around to
clean it up.
> 
> Any help or guidance would be greatly appreciated.
> 
> Thanks!
> 
> 

------------------------------------------------------------
---------
The official User-To-User support forum of the Apache HTTP
Server Project.
See <URL:http://htt
pd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribehttpd.apache.org
   "   from the digest: users-digest-unsubscribehttpd.apache.org
For additional commands, e-mail: users-helphttpd.apache.org

On 8/28/07, Brian Cohen <brianintercarve.net> wrote:
> We are running Apache 1.3.34.
>
> On our Debian 4.0 system, each night cron.daily runs
logrotate.
> Logrotate's postrotate script runs `/etc/init.d/apache
reload` (no
> errors on stderr). When it is successful, top shows all
apache children
> go <defunct>, then apache restarts. But more
often than not, all but 2
> or 3 apache children die (without top ever showing
<defunct>), the pid
> file is removed. Those 2 or 3 apache children continue
to live and
> accept connections, but won't serve any requests. Since
the pid file has
> been removed, we have to issue a `killall apache`,
then
> `/etc/init.d/apache start`
>
> The net effect is pretty fatal. If we're not around
when it happens,
> Apache remains effectively down until we come around to
clean it up.
>
> Any help or guidance would be greatly appreciated.

Since this isn't something that I see frequently reported,
my first
guess would be a faulty third-party module. What are you
using other
than the standard apache modules?

Joshua.

------------------------------------------------------------
---------
The official User-To-User support forum of the Apache HTTP
Server Project.
See <URL:http://htt
pd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribehttpd.apache.org
   "   from the digest: users-digest-unsubscribehttpd.apache.org
For additional commands, e-mail: users-helphttpd.apache.org

Joshua Slive wrote:
> Since this isn't something that I see frequently
reported, my first
> guess would be a faulty third-party module. What are
you using other
> than the standard apache modules?
> 
> Joshua.
> 

Thanks for responding, Joshua. My full module list is as
follows:

AddModule mod_so.c
AddModule mod_macro.c
LoadModule security_module
/usr/lib/apache/1.3/mod_security.so
LoadModule config_log_module
/usr/lib/apache/1.3/mod_log_config.so
LoadModule referer_log_module
/usr/lib/apache/1.3/mod_log_referer.so
LoadModule mime_magic_module
/usr/lib/apache/1.3/mod_mime_magic.so
LoadModule mime_module /usr/lib/apache/1.3/mod_mime.so
LoadModule negotiation_module
/usr/lib/apache/1.3/mod_negotiation.so
LoadModule status_module /usr/lib/apache/1.3/mod_status.so
LoadModule includes_module
/usr/lib/apache/1.3/mod_include.so
LoadModule autoindex_module
/usr/lib/apache/1.3/mod_autoindex.so
LoadModule dir_module /usr/lib/apache/1.3/mod_dir.so
LoadModule cgi_module /usr/lib/apache/1.3/mod_cgi.so
LoadModule alias_module /usr/lib/apache/1.3/mod_alias.so
LoadModule rewrite_module
/usr/lib/apache/1.3/mod_rewrite.so
LoadModule access_module /usr/lib/apache/1.3/mod_access.so
LoadModule auth_module /usr/lib/apache/1.3/mod_auth.so
LoadModule expires_module
/usr/lib/apache/1.3/mod_expires.so
LoadModule headers_module
/usr/lib/apache/1.3/mod_headers.so
LoadModule usertrack_module
/usr/lib/apache/1.3/mod_usertrack.so
LoadModule setenvif_module
/usr/lib/apache/1.3/mod_setenvif.so
LoadModule dav_module /usr/lib/apache/1.3/libdav.so
LoadModule encoding_module
/usr/lib/apache/1.3/mod_encoding.so
LoadModule layout_module /usr/lib/apache/1.3/mod_layout.so
LoadModule perl_module /usr/lib/apache/1.3/mod_perl.so
LoadModule ssl_module /usr/lib/apache/1.3/mod_ssl.so
LoadModule auth_mysql_module
/usr/lib/apache/1.3/mod_auth_mysql.so
LoadModule php4_module /usr/lib/apache/1.3/libphp4.so
LoadModule gzip_module /usr/lib/apache/1.3/mod_gzip.so
LoadModule python_module /usr/lib/apache/1.3/mod_python.so


------------------------------------------------------------
---------
The official User-To-User support forum of the Apache HTTP
Server Project.
See <URL:http://htt
pd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribehttpd.apache.org
   "   from the digest: users-digest-unsubscribehttpd.apache.org
For additional commands, e-mail: users-helphttpd.apache.org

On 8/28/07, Brian Cohen <brianintercarve.net> wrote:
> Joshua Slive wrote:
> > Since this isn't something that I see frequently
reported, my first
> > guess would be a faulty third-party module. What
are you using other
> > than the standard apache modules?
> >
> > Joshua.
> >
>
> Thanks for responding, Joshua. My full module list is
as follows:
>

You have a TON of third-party modules. All of the following
are
non-standard in 1.3:
(although mod_ssl and mod_dav have been adopted into 2.x)

> AddModule mod_macro.c
> LoadModule security_module
/usr/lib/apache/1.3/mod_security.so
> LoadModule dav_module /usr/lib/apache/1.3/libdav.so
> LoadModule encoding_module
/usr/lib/apache/1.3/mod_encoding.so
> LoadModule layout_module
/usr/lib/apache/1.3/mod_layout.so
> LoadModule perl_module /usr/lib/apache/1.3/mod_perl.so
> LoadModule ssl_module /usr/lib/apache/1.3/mod_ssl.so
> LoadModule auth_mysql_module
/usr/lib/apache/1.3/mod_auth_mysql.so
> LoadModule php4_module /usr/lib/apache/1.3/libphp4.so
> LoadModule gzip_module /usr/lib/apache/1.3/mod_gzip.so
> LoadModule python_module
/usr/lib/apache/1.3/mod_python.so

Since that is a lot of stuff to go through by trial and
error, you
might alternatively try to attach to some of those
"stuck" processes
with a debugger and get a backtrace to see what module is
causing the
problems.

Joshua.

------------------------------------------------------------
---------
The official User-To-User support forum of the Apache HTTP
Server Project.
See <URL:http://htt
pd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribehttpd.apache.org
   "   from the digest: users-digest-unsubscribehttpd.apache.org
For additional commands, e-mail: users-helphttpd.apache.org

Hi
Not sure if it is related but we have seen problems before
that when you
restart apache very often under load it runs out of
semaphore. That
might worth looking into.

To clean up semaphores after stopping apache you can try the
following:
ipcs -s | grep www-data | awk '{print $2}'|xargs -n 1 ipcrm
sem

PLEASE dont run as is, because it might break things. Purely
meant to
get you started.

Regards 

> -----Original Message-----
> From: Brian Cohen [mailto:brianintercarve.net] 
> Sent: 28 August 2007 18:06
> To: usershttpd.apache.org
> Subject: Re: [usershttpd] Unclean restarts
> 
> Some additional info -- when Apache is running normally
(for 
> example, during the middle of the day), we can issue a
manual 
> '/etc/init.d/apache reload' and it will work fine. The

> problems seem to arise some percentage of the time when
it 
> tries to run from cron.daily's calling of
logrotate/postrotate.
> 
> Thanks--
> 
> Brian J. Cohen
> Intercarve Networks, LLC
> 914.552.1274 [cell]
> 
> 
> Brian Cohen wrote:
> > We are running Apache 1.3.34.
> > 
> > On our Debian 4.0 system, each night cron.daily
runs logrotate.
> > Logrotate's postrotate script runs
`/etc/init.d/apache reload` (no 
> > errors on stderr). When it is successful, top
shows all apache 
> > children go <defunct>, then apache restarts.
But more often 
> than not, 
> > all but 2 or 3 apache children die (without top
ever showing 
> > <defunct>), the pid file is removed. Those 2
or 3 apache children 
> > continue to live and accept connections, but won't
serve 
> any requests. 
> > Since the pid file has been removed, we have to
issue a `killall 
> > apache`, then `/etc/init.d/apache start`
> > 
> > The net effect is pretty fatal. If we're not
around when it 
> happens, 
> > Apache remains effectively down until we come
around to clean it up.
> > 
> > Any help or guidance would be greatly
appreciated.
> > 
> > Thanks!
> > 
> > 
> 
>
------------------------------------------------------------
---------
> The official User-To-User support forum of the Apache
HTTP 
> Server Project.
> See <URL:http://htt
pd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribehttpd.apache.org
>    "   from the digest:
users-digest-unsubscribehttpd.apache.org
> For additional commands, e-mail: users-helphttpd.apache.org
> 

____________________________________________________________
__________
This email has been scanned by the MessageLabs Email
Security System.
For more information please visit http://www.messagela
bs.com/email 
____________________________________________________________
__________

------------------------------------------------------------
---------
The official User-To-User support forum of the Apache HTTP
Server Project.
See <URL:http://htt
pd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribehttpd.apache.org
   "   from the digest: users-digest-unsubscribehttpd.apache.org
For additional commands, e-mail: users-helphttpd.apache.org

Are there any real security benefits to operating Virtual
Private  
Servers as opposed to properly configured Apache virtual
servers?

Bob

------------------------------------------------------------
---------
The official User-To-User support forum of the Apache HTTP
Server Project.
See <URL:http://htt
pd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribehttpd.apache.org
   "   from the digest: users-digest-unsubscribehttpd.apache.org
For additional commands, e-mail: users-helphttpd.apache.org