mongrel goes unresponsive after 30 minutes

Greetings!

I'm in the process of moving my app from a shared hosting
arrangement to a 
VPS and am seeing some mongrel behavior I haven't seen
before. 
Specifically, after about 30 minutes of inactivity mongrel
goes unresponsive 
and has to be restarted.  When I restart it writes the
messages below. 
Before that, there's nothing in the logs that shows any
problem at all.

In mongrel.log:

** USR2 signal received.
Thu Feb 08 08:43:51 EST 2007: Reaping 2 threads for slow
workers because of 
'shutdown'
Thread # is too old, killing.
Thu Feb 08 08:43:51 EST 2007: Error calling
Dispatcher.dispatch #Thread # is 
too old, killing.

In production.log:

Mongrel timed out this thread: shutdown

A set of trace messages follow this which I'm happy to post
if it would be 
helpful.

Can anybody help me understand what's going on and what I
need to do to fix 
it?

Thanks much,
Bill 



--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Deploying Rails" group.
To post to this group, send email to
rubyonrails-deploymentgooglegroups.com
To unsubscribe from this group, send email to
rubyonrails-deployment-unsubscribegooglegroups.com
For more options, visit this group at http://groups.google.com/group/rubyonrails-deployment
?hl=en
-~----------~----~----~----~------~----~------~--~---

>From Q: Mongrel stops working if it's left alone for a
long time.

If you find that Mongrel stops working after a long idle
time and
you're using MySQL then you're hitting a bug in the MySQL
driver that
doesn't properly timeout connections. What happens is the
MySQL server
side of the connection times out and closes, but the MySQL
client
doesn't detect this and just sits there.

What you have to do is set:

ActiveRecord::Base.verification_timeout = 14400

Or to any value that is lower than the MySQL server's
interactive_timeout setting. This will make sure that
ActiveRecord
checks the connection often enough to reset the connection.

Q: Mongrel stops working if it's left alone for a long
time.

If you find that Mongrel stops working after a long idle
time and
you're using MySQL then you're hitting a bug in the MySQL
driver that
doesn't properly timeout connections. What happens is the
MySQL server
side of the connection times out and closes, but the MySQL
client
doesn't detect this and just sits there.

What you have to do is set:

ActiveRecord::Base.verification_timeout = 14400

Or to any value that is lower than the MySQL server's
interactive_timeout setting. This will make sure that
ActiveRecord
checks the connection often enough to reset the connection.

--
Aníbal Rojas
http://www.rubycorner.com
http://www.hasmanyde
velopers.com

On Feb 8, 11:07 am, "Bill Walton"
<bill.wal...charter.net> wrote:
> Greetings!
>
> I'm in the process of moving my app from a shared
hosting arrangement to a
> VPS and am seeing some mongrel behavior I haven't seen
before.
> Specifically, after about 30 minutes of inactivity
mongrel goes unresponsive
> and has to be restarted.  When I restart it writes the
messages below.
> Before that, there's nothing in the logs that shows any
problem at all.
>
> In mongrel.log:
>
> ** USR2 signal received.
> Thu Feb 08 08:43:51 EST 2007: Reaping 2 threads for
slow workers because of
> 'shutdown'
> Thread # is too old, killing.
> Thu Feb 08 08:43:51 EST 2007: Error calling
Dispatcher.dispatch #Thread # is
> too old, killing.
>
> In production.log:
>
> Mongrel timed out this thread: shutdown
>
> A set of trace messages follow this which I'm happy to
post if it would be
> helpful.
>
> Can anybody help me understand what's going on and what
I need to do to fix
> it?
>
> Thanks much,
> Bill


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Deploying Rails" group.
To post to this group, send email to
rubyonrails-deploymentgooglegroups.com
To unsubscribe from this group, send email to
rubyonrails-deployment-unsubscribegooglegroups.com
For more options, visit this group at http://groups.google.com/group/rubyonrails-deployment
?hl=en
-~----------~----~----~----~------~----~------~--~---

The previous snippet is from:

http://mongrel.
rubyforge.org/faq.html

--
Aníbal Rojas
http://www.rubycorner.com
http://www.hasmanyde
velopers.com

On Feb 8, 4:54 pm, "Aníbal Rojas"
<anibalro...gmail.com> wrote:
> >From Q: Mongrel stops working if it's left alone
for a long time.
>
> If you find that Mongrel stops working after a long
idle time and
> you're using MySQL then you're hitting a bug in the
MySQL driver that
> doesn't properly timeout connections. What happens is
the MySQL server
> side of the connection times out and closes, but the
MySQL client
> doesn't detect this and just sits there.
>
> What you have to do is set:
>
> ActiveRecord::Base.verification_timeout = 14400
>
> Or to any value that is lower than the MySQL server's
> interactive_timeout setting. This will make sure that
ActiveRecord
> checks the connection often enough to reset the
connection.
>
> Q: Mongrel stops working if it's left alone for a long
time.
>
> If you find that Mongrel stops working after a long
idle time and
> you're using MySQL then you're hitting a bug in the
MySQL driver that
> doesn't properly timeout connections. What happens is
the MySQL server
> side of the connection times out and closes, but the
MySQL client
> doesn't detect this and just sits there.
>
> What you have to do is set:
>
> ActiveRecord::Base.verification_timeout = 14400
>
> Or to any value that is lower than the MySQL server's
> interactive_timeout setting. This will make sure that
ActiveRecord
> checks the connection often enough to reset the
connection.
>
> --
> Aníbal Rojashttp://www.rubycorner.comhttp://www.hasmanydevelopers.c
om
>
> On Feb 8, 11:07 am, "Bill Walton"
<bill.wal...charter.net> wrote:
>
> > Greetings!
>
> > I'm in the process of moving my app from a shared
hosting arrangement to a
> > VPS and am seeing some mongrel behavior I haven't
seen before.
> > Specifically, after about 30 minutes of inactivity
mongrel goes unresponsive
> > and has to be restarted.  When I restart it writes
the messages below.
> > Before that, there's nothing in the logs that
shows any problem at all.
>
> > In mongrel.log:
>
> > ** USR2 signal received.
> > Thu Feb 08 08:43:51 EST 2007: Reaping 2 threads
for slow workers because of
> > 'shutdown'
> > Thread # is too old, killing.
> > Thu Feb 08 08:43:51 EST 2007: Error calling
Dispatcher.dispatch #Thread # is
> > too old, killing.
>
> > In production.log:
>
> > Mongrel timed out this thread: shutdown
>
> > A set of trace messages follow this which I'm
happy to post if it would be
> > helpful.
>
> > Can anybody help me understand what's going on and
what I need to do to fix
> > it?
>
> > Thanks much,
> > Bill


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Deploying Rails" group.
To post to this group, send email to
rubyonrails-deploymentgooglegroups.com
To unsubscribe from this group, send email to
rubyonrails-deployment-unsubscribegooglegroups.com
For more options, visit this group at http://groups.google.com/group/rubyonrails-deployment
?hl=en
-~----------~----~----~----~------~----~------~--~---

Hi Anibal,

Thanks!  I've got the admin looking into now.  I really
appreciate the 
response.

Best regards,
Bill 



--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Deploying Rails" group.
To post to this group, send email to
rubyonrails-deploymentgooglegroups.com
To unsubscribe from this group, send email to
rubyonrails-deployment-unsubscribegooglegroups.com
For more options, visit this group at http://groups.google.com/group/rubyonrails-deployment
?hl=en
-~----------~----~----~----~------~----~------~--~---

You are welcome, good luck. Zed has a checklist in Mongrel's
site for
some other conditions that could cause this errors.

--
Aníbal Rojas
http://www.rubycorner.com
http://www.hasmanyde
velopers.com

On Feb 8, 6:00 pm, "Bill Walton"
<bill.wal...charter.net> wrote:
> Hi Anibal,
>
> Thanks!  I've got the admin looking into now.  I really
appreciate the
> response.
>
> Best regards,
> Bill


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Deploying Rails" group.
To post to this group, send email to
rubyonrails-deploymentgooglegroups.com
To unsubscribe from this group, send email to
rubyonrails-deployment-unsubscribegooglegroups.com
For more options, visit this group at http://groups.google.com/group/rubyonrails-deployment
?hl=en
-~----------~----~----~----~------~----~------~--~---

Bill Walton wrote:
> Greetings!
> 
> I'm in the process of moving my app from a shared
hosting arrangement to a 
> VPS and am seeing some mongrel behavior I haven't seen
before. 
> Specifically, after about 30 minutes of inactivity
mongrel goes unresponsive 
> and has to be restarted.  When I restart it writes the
messages below. 
> Before that, there's nothing in the logs that shows any
problem at all.
> 
> In mongrel.log:
> 
> ** USR2 signal received.
> Thu Feb 08 08:43:51 EST 2007: Reaping 2 threads for
slow workers because of 
> 'shutdown'
> Thread # is too old, killing.
> Thu Feb 08 08:43:51 EST 2007: Error calling
Dispatcher.dispatch #Thread # is 
> too old, killing.
> 
> In production.log:
> 
> Mongrel timed out this thread: shutdown
> 
> A set of trace messages follow this which I'm happy to
post if it would be 
> helpful.
> 
> Can anybody help me understand what's going on and what
I need to do to fix 
> it?

Do you have a cluster of mongrels running or just the one?

-Robby


-- 
Robby Russell
http://www.robbyonrails.
com/
http://www.planetargon.co
m/

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Deploying Rails" group.
To post to this group, send email to
rubyonrails-deploymentgooglegroups.com
To unsubscribe from this group, send email to
rubyonrails-deployment-unsubscribegooglegroups.com
For more options, visit this group at http://groups.google.com/group/rubyonrails-deployment
?hl=en
-~----------~----~----~----~------~----~------~--~---

Hi Robby,

Robby Russell wrote:
>
> Do you have a cluster of mongrels running or just the
one?

I'm not sure and asked similar question of my hosting
service today. I'll 
report back on that as soon as I find out.  For development
purposes I 
started with a shared hosting plan with the same provider
and the app I'm 
having problems with is running just fine on that site.  I
haven't even 
browsed to it in almost two weeks and when I did this
afternoon I found that 
it's just 'hummin along'.  It's not _exactly_the same app,
but there are 
very few differences.  My first instinct is to look for
differences in the 
code between the two but, IME, that's really only going to
be a productive 
approach if the underlying architecture is comparable.  Any
debugging advice 
/ experience is tremendously appreciated.

Thanks!
Bill 



--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Deploying Rails" group.
To post to this group, send email to
rubyonrails-deploymentgooglegroups.com
To unsubscribe from this group, send email to
rubyonrails-deployment-unsubscribegooglegroups.com
For more options, visit this group at http://groups.google.com/group/rubyonrails-deployment
?hl=en
-~----------~----~----~----~------~----~------~--~---

On Thu, 8 Feb 2007 09:07:47 -0600
"Bill Walton" <bill.waltoncharter.net> wrote:

> 
> Greetings!
> 
> I'm in the process of moving my app from a shared
hosting arrangement to a 
> VPS and am seeing some mongrel behavior I haven't seen
before. 
> Specifically, after about 30 minutes of inactivity
mongrel goes unresponsive 
> and has to be restarted.  When I restart it writes the
messages below. 
> Before that, there's nothing in the logs that shows any
problem at all.
> 

Restart (USR2) isn't that reliable because of how ruby/rails
needs to
reload.  You should use a full stop/start cycle instead.

Once you start doing that and you still get no response then
run strace
on the mongrel process to see what it's doing.  It's most
likely that
you have a library that's messing things up.

Also, don't use pstore, don't set Logger to rotate, don't do
file
locking, and don't use a crappy DNS.

-- 
Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu
http://www.zedshaw.com/
http:/
/www.awprofessional.com/title/0321483502 -- The Mongrel
Book
http://mongrel.rubyforg
e.org/
http://www.ling
r.com/room/3yXhqKbfPy8 -- Come get help.

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Deploying Rails" group.
To post to this group, send email to
rubyonrails-deploymentgooglegroups.com
To unsubscribe from this group, send email to
rubyonrails-deployment-unsubscribegooglegroups.com
For more options, visit this group at http://groups.google.com/group/rubyonrails-deployment
?hl=en
-~----------~----~----~----~------~----~------~--~---

Hi Zed,

Thanks very much for your reply.  I appreciate it very much.
 More below.

Zed A. Shaw wrote:

> Restart (USR2) isn't that reliable because of
> how ruby/rails needs to reload.  You should
> use a full stop/start cycle instead.

I've been doing that.  Should have said it that way.

> Once you start doing that and you still get no
> response then run strace on the mongrel process
> to see what it's doing.

The folks at the hosting service (a2hosting) have done that
I think and they 
tell me it hasn't given them any clues.

> It's most likely that you have a library that's
messing
> things up.

If by 'library' you mean gems / plugins, I do have a couple
installed: 
PDF::Writer and BackgroundRb.  But BackgroundRb is stopped
and there are no 
calls to PDF:Writer being used in the test scenerio I've
been using.  I 
thought Rails didn't load code before it needed it.  Anyway,
here's what I'm 
doing.

1) Start mongrel using mongrel_rails start -d -p 3002 -e
production < 
/dev/null >& /dev/null
2) Open a browser and browse to the index action of the
Rails app (the index 
action is empty and the view renders a page with a form on
it that has a 
group of 3 radio buttons and a submit button.)
3) Close the browser.
4) Wait 40 minutes, then do step 2 again.

The result of step 4 is a blank white screen, not an error
message.  Neither 
production.log nor mongel.log have anything written to them
at that point 
other than the initial startup messages in mongrel.log and
the initial 
request from step 2 in production.log.  When I stop mongrel,
it writes a 
message to both logs saying its killing a slow worker. 
That's the only clue 
I get.  Steps 1-4 can be repeated ad nauseum.

After their initial investigation the sys admins concluded
the problem had 
to be in my code (surprise, surprise  )  I had
a slightly different 
version that was running flawlessly on my shared account
with them, so I 
decided to 'dig in.'  I copied the entire Rails application
directory for 
both apps to my PC, then copied the one I was having
problems with on the 
VPS over to the shared account.  It ran without a problem
(and, a week 
later, is *still* running without a problem.)

Yesterday they set up a cron job outside my space to invoke
the index method 
every 10 minutes.  That's 'solved' the problem.  It doesn't
sit well with me 
though.  My experience leads me to fear that whatever the
real cause of the 
problem is will eventually manifest itself again.  At the
worst possible 
time.

If you (or anyone else reading this) have any interest in
understanding 
what's really going on (since mongrel's behavior is the only
clue from an 
end user perspective), please let me know.  I'll give you
any access you 
need.  If not, I understand completely.

Thanks again for your time.

Best regards,
Bill

> Also, don't use pstore, don't set Logger to rotate,
don't do file
> locking, and don't use a crappy DNS.
>
> -- 
> Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu
> http://www.zedshaw.com/
> http:/
/www.awprofessional.com/title/0321483502 -- The Mongrel
Book
> http://mongrel.rubyforg
e.org/
> http://www.ling
r.com/room/3yXhqKbfPy8 -- Come get help.
>
> >
> 



--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Deploying Rails" group.
To post to this group, send email to
rubyonrails-deploymentgooglegroups.com
To unsubscribe from this group, send email to
rubyonrails-deployment-unsubscribegooglegroups.com
For more options, visit this group at http://groups.google.com/group/rubyonrails-deployment
?hl=en
-~----------~----~----~----~------~----~------~--~---

On Fri, 16 Feb 2007 09:20:11 -0600
"Bill Walton" <bill.waltoncharter.net> wrote:

> Yesterday they set up a cron job outside my space to
invoke the index method 
> every 10 minutes.  That's 'solved' the problem.  It
doesn't sit well with me 
> though.  My experience leads me to fear that whatever
the real cause of the 
> problem is will eventually manifest itself again.  At
the worst possible 
> time.

Interesting, make sure you've got the compiled mysql gem and
you aren't
using the rails mysql.rb file.
 
> If you (or anyone else reading this) have any interest
in understanding 
> what's really going on (since mongrel's behavior is the
only clue from an 
> end user perspective), please let me know.  I'll give
you any access you 
> need.  If not, I understand completely.

Yeah, actually I'd be interested.  Email me off list to
coordinate it.

-- 
Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu
http://www.zedshaw.com/
http:/
/www.awprofessional.com/title/0321483502 -- The Mongrel
Book
http://mongrel.rubyforg
e.org/
http://www.ling
r.com/room/3yXhqKbfPy8 -- Come get help.

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Deploying Rails" group.
To post to this group, send email to
rubyonrails-deploymentgooglegroups.com
To unsubscribe from this group, send email to
rubyonrails-deployment-unsubscribegooglegroups.com
For more options, visit this group at http://groups.google.com/group/rubyonrails-deployment
?hl=en
-~----------~----~----~----~------~----~------~--~---