Re: policy change for correct handling of fatal program conditions

Thread: Re: policy change for correct handling of fatal program conditions

Search this list only Search all lists
Re: policy change for correct handling of fatal program conditions
	2007-01-23 06:15:47
> Has anyone considered the fact that malloc() never actually fails i.e. > memory starvation does not cause malloc() to return error. It always returns > true - when you actually access the memory one will get a SIGBUS (I think) > and the program will die. > > This behaviour (called overcommit) is to make the UNIX fork/exec model > efficient. For this reason one should use calloc() because at least then the > failure will occur close to the allocation call in the code. Do you refer to the Linux and FreeBSD implementation? http://baus.net/memory-management http://www.ussg.iu.edu/hypermail/linux/kernel/0311.3/0420.html http://homepage.mac.com/mguthaus/tips/2005/tip050903.html > There have been many discussions about this quirk of malloc. The man page > even describes it as a bug ... To which manual do you refer to? Is it a similar version like this one? http://sman.informatik.htw-dresden.de:6711/man?=malloc Other specifications: - http://opengroup.org/onlinepubs/009695399/functions/calloc.html - http://docs.sun.com/app/docs/doc/816-5168/6mbb3hrgq?a=view Regards, Markus _______________________________________________ --Bandwidth and Colocation provided by Easynews.com -- asterisk-dev mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-dev
Re: policy change for correct handling of fatal program conditions
	2007-01-23 09:33:42
On Tuesday 23 January 2007 14:15, SF Markus Elfring wrote: > > Has anyone considered the fact that malloc() never actually fails i.e. > > memory starvation does not cause malloc() to return error. It always > > returns true - when you actually access the memory one will get a SIGBUS > > (I think) and the program will die. > > > > This behaviour (called overcommit) is to make the UNIX fork/exec model > > efficient. For this reason one should use calloc() because at least then > > the failure will occur close to the allocation call in the code. > > Do you refer to the Linux and FreeBSD implementation? Hi Marcus Yes - although other UNIX's have had this feature in the past. I was in error in my comments above - when the kernel detects memory starvation, the dreaded OOM killer is called which attempts to kill processes that are memory hogs. The original OOM killer without fail would kill the X server if it was running. Later versions tried to be cleverer than this. There was kernel patch that allowed one to exempt ceratin processes from the OOM killer - I don't think it made it into the main tree however. asterisk would be a candidate for this. > http://baus.net/memory-management > http://www.ussg.iu.edu/hypermail/linux/kernel/0311.3/0420.html > http://homepage.mac.com/mguthaus/tips/2005/tip050903.html > > > There have been many discussions about this quirk of malloc. The man > > page even describes it as a bug ... > > To which manual do you refer to? Type 'man malloc' on the linux box.... See also http://developers.sun.com/solaris/articles/subprocess/subprocess.html#overcom There are some references at the end - the RedHat article incorrectly describes option 0 - the correct description is in /usr/src/linux/Documentation/vm/overcommit-accounting > Is it a similar version like this one? > http://sman.informatik.htw-dresden.de:6711/man?=malloc can't get that page > Linus has ranted about this strange quirk but eventually allowed the extra sysctl settings. (http://mail.python.org/pipermail/python-dev/1995-March/008945.html) Basically the problem is that the sbrk() call (on which malloc() is based) which gets memory pages from the kernel never returns them to the kernel. So once allocated to a process the only way to get the memory pages back into the kernel pool is to kill the process that allocated it. So if a memory hog process wants to spawn a low-memory process, it can do so without needing to duplicate all that memory (so-called copy-on-write) This means that free() only returns memory to a pool of bytes that are only available to the process for subsequent mallocs and not to any other processes. Some people write code that does all the allocations needed at startup. If there is not enough resources available, then the program will fail immediately instead of at some arbitrary later time - this minimises the surprise factor. Paul -- Paul Hewlett Technical Director Global Call Center Solutions Ltd, 2nd Floor, Milnerton Mall Cnr Loxton & Koeberg Roads, 7435 Milnerton paulgccs.co.za www.gccs.co.za Tel: +27 86 111 3433 Fax: +27 86 111 3520 Cel: +27 76 072 7906 Gizmo: 1 747 659 6171 _______________________________________________ --Bandwidth and Colocation provided by Easynews.com -- asterisk-dev mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-dev
Re: policy change for correct handling of fatal program conditions
	2007-01-23 12:54:09
On 23 Jan 2007, at 15:33, Paul Hewlett wrote: > On Tuesday 23 January 2007 14:15, SF Markus Elfring wrote: >>> Has anyone considered the fact that malloc() never actually >>> fails i.e. >>> memory starvation does not cause malloc() to return error. It always >>> returns true - when you actually access the memory one will get a >>> SIGBUS >>> (I think) and the program will die. >>> >>> This behaviour (called overcommit) is to make the UNIX fork/ >>> exec model >>> efficient. For this reason one should use calloc() because at >>> least then >>> the failure will occur close to the allocation call in the code. >> >> Do you refer to the Linux and FreeBSD implementation? > Hi Marcus > > Yes - although other UNIX's have had this feature in the past. Gosh, really ? MacOSX doesn't, solaris (hence system V) doesn't and OpenBSD doesn't. Linux seems to be the only one so far..... I sort of knew about the overcommit, but I never realized what a disaster it would be for a large long lived program. Tim Panton www.mexuar.net www.westhawk.co.uk/ _______________________________________________ --Bandwidth and Colocation provided by Easynews.com -- asterisk-dev mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-dev
Re: policy change for correct handling of fatal program conditions
	2007-01-23 13:32:05
On Tue, Jan 23, 2007 at 05:33:42PM +0200, Paul Hewlett wrote: > There was kernel patch that allowed one to exempt ceratin processes from the > OOM killer - I don't think it made it into the main tree however. asterisk > would be a candidate for this. I think it has: tzafrirboomtime:~$ ls -l /proc/self/oom_* -rw-r--r-- 1 tzafrir tzafrir 0 2007-01-23 21:30 /proc/self/oom_adj -r--r--r-- 1 tzafrir tzafrir 0 2007-01-23 21:30 /proc/self/oom_score tzafrirboomtime:~$ grep . /proc/self/oom_* /proc/self/oom_adj:0 /proc/self/oom_score:886 -- Tzafrir Cohen icq#16849755 jabber:tzafrirjabber.org +972-50-7952406 mailto:tzafrir.cohenxorcom.com http://www.xorcom.com iax:guestlocal.xorcom.com/tzafrir _______________________________________________ --Bandwidth and Colocation provided by Easynews.com -- asterisk-dev mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-dev
Re: policy change for correct handling of fatal program conditions
	2007-01-24 01:42:10
On Tuesday 23 January 2007 21:32, Tzafrir Cohen wrote: > On Tue, Jan 23, 2007 at 05:33:42PM +0200, Paul Hewlett wrote: > > There was kernel patch that allowed one to exempt ceratin processes from > > the OOM killer - I don't think it made it into the main tree however. > > asterisk would be a candidate for this. > > I think it has: > tzafrirboomtime:~$ ls -l /proc/self/oom_* > -rw-r--r-- 1 tzafrir tzafrir 0 2007-01-23 21:30 /proc/self/oom_adj > -r--r--r-- 1 tzafrir tzafrir 0 2007-01-23 21:30 /proc/self/oom_score > tzafrirboomtime:~$ grep . /proc/self/oom_* > /proc/self/oom_adj:0 > /proc/self/oom_score:886 Great - did not realise it was in. To bar asterisk from being killed by the oom_killer find the pid of asterisk PID=`cat /var/run/asterisk.pid` and set the oom_adj value to -17 echo "-17" > /proc/${PID}/oom_adj Paul -- Paul Hewlett Technical Director Global Call Center Solutions Ltd, 2nd Floor, Milnerton Mall Cnr Loxton & Koeberg Roads, 7435 Milnerton paulgccs.co.za www.gccs.co.za Tel: +27 86 111 3433 Fax: +27 86 111 3520 Cel: +27 76 072 7906 Gizmo: 1 747 659 6171 _______________________________________________ --Bandwidth and Colocation provided by Easynews.com -- asterisk-dev mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-dev
Re: policy change for correct handling of fatal program conditions
	2007-01-24 05:47:28
On Wednesday 24 January 2007 09:42, Paul Hewlett wrote: > Great - did not realise it was in. To bar asterisk from being killed by the > oom_killer find the pid of asterisk > > PID=`cat /var/run/asterisk.pid` > > and set the oom_adj value to -17 > A correction - try the following PIDS=`pidof asterisk` for i in $ do echo "-17" > /proc/$i/oom_adj done -- Paul Hewlett Technical Director Global Call Center Solutions Ltd, 2nd Floor, Milnerton Mall Cnr Loxton & Koeberg Roads, 7435 Milnerton paulgccs.co.za www.gccs.co.za Tel: +27 86 111 3433 Fax: +27 86 111 3520 Cel: +27 76 072 7906 Gizmo: 1 747 659 6171 _______________________________________________ --Bandwidth and Colocation provided by Easynews.com -- asterisk-dev mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-dev