seg fault in intelOSX+python2.4

Hi all,

I've been using PyTables (www.pytables.org) with python2.4
on intel  
Mac OSX 10.4.10
and I'm running into a seg. fault when generating a large
hdf5 file.

Almost certainly something to do with relationship between
OS and  
python.
Has anyone had a similar (memory leak?) experience?



_________________________________________________
experimental polymedia:	www.avatar.com.au
Sonic Communications Research Group,
University of Canberra:	 www.canberra.edu.au/vc-forum/scrg
vip=Verbal Interactivity Project



_______________________________________________
Pythonmac-SIG maillist  -  Pythonmac-SIGpython.org
http://mail.python.org/mailman/listinfo/pythonmac-sig

On 30 Aug, 2007, at 1:56, David Worrall wrote:

> Hi all,
>
> I've been using PyTables (www.pytables.org) with
python2.4 on intel
> Mac OSX 10.4.10
> and I'm running into a seg. fault when generating a
large hdf5 file.
>
> Almost certainly something to do with relationship
between OS and
> python.
> Has anyone had a similar (memory leak?) experience?

How large a file? It might be a memory management bug in
pytables as  
well.

If you have the developer tools (Xcode & friends)
installed you can  
find out where the crash occurs using gdb.

That is, 'gdb python', then on the prompt for gdb: 'r  
myscript.py' (adding arguments if needed). When the crash
occurs you  
can use 'where 20' to see the topmost 20 frames on the C
stack, or  
just 'where' to see the entire stack.

Ronald
>
>
>
>
> _________________________________________________
> experimental polymedia:	www.avatar.com.au
> Sonic Communications Research Group,
> University of Canberra:	
www.canberra.edu.au/vc-forum/scrg
> vip=Verbal Interactivity Project
>
>
>
> _______________________________________________
> Pythonmac-SIG maillist  -  Pythonmac-SIGpython.org
> http://mail.python.org/mailman/listinfo/pythonmac-sig

_______________________________________________
Pythonmac-SIG maillist  -  Pythonmac-SIGpython.org
http://mail.python.org/mailman/listinfo/pythonmac-sig

Thanks for your feedback Ronald.
The file is quite large: 4-5 GB
pytables buffers the data and controlling the buffer size  
occasionally stops or slows the rot.
The code has been tested under Linux without incident.
faltetcarabos.com, one of the developers of pytables
wrote:

> A Tuesday 21 August 2007, David Worrall escrigué:
>
>>   I lowered NODE_MAX_SLOTS from  256 to128 and that
slowed the leak -
>> enough to get some sort of DB happening.
>> It eventually seg faulted, however.
>> I've noticed that sometimes the seg fault causes
the (non-python)
>> heap to become corrupted, requiring a HW reboot
before pytables
>> becomes useful again.
>> Who knows what memory it may be writing over!
>>
>
> I don't think there is a leak in PyTables itself; it is
just that it
> takes a lot of memory to work with many nodes
simultaneously.  Perhaps
> reducing the NODE_MAX_SLOTS (to 64, for example) could
help a bit  
> more.
> Perhaps it would be a good idea to reduce the memory
requirements of
> PyTables nodes (for example, avoid the creation of
buffers when they
> are not needed), but this should take quite a few of
thought.
>
> Indeed it seems that there is a problem with the MacOSX
platform, but
> the fact that this is not reproducible on Linux is
unfortunate.
...
> I'm thinking that perhaps you can try with newer
versions of Python
> (2.5.1) and HDF5 (the 1.8.0 beta3 is out, and I know
that it wears a
> completely revamped cache system, so maybe it is worth
a try).
>

Anyway I'll try with gdb as you suggest and report back. I
need to  
put it on my task stack so it'll take some time.

thanks again,

David
On 30/08/2007, at 3:48 PM, Ronald Oussoren wrote:

>
> On 30 Aug, 2007, at 1:56, David Worrall wrote:
>
>> Hi all,
>>
>> I've been using PyTables (www.pytables.org) with
python2.4 on intel
>> Mac OSX 10.4.10
>> and I'm running into a seg. fault when generating a
large hdf5 file.
>>
>> Almost certainly something to do with relationship
between OS and
>> python.
>> Has anyone had a similar (memory leak?)
experience?
>
> How large a file? It might be a memory management bug
in pytables  
> as well.
>
> If you have the developer tools (Xcode & friends)
installed you can  
> find out where the crash occurs using gdb.
>
> That is, 'gdb python', then on the prompt for gdb: 'r 

> myscript.py' (adding arguments if needed). When the
crash occurs  
> you can use 'where 20' to see the topmost 20 frames on
the C stack,  
> or just 'where' to see the entire stack.
>
> Ronald
>>
>>
>>
>>
>> _________________________________________________
>> experimental polymedia:	www.avatar.com.au
>> Sonic Communications Research Group,
>> University of Canberra:	
www.canberra.edu.au/vc-forum/scrg
>> vip=Verbal Interactivity Project
>>
>>
>>
>> _______________________________________________
>> Pythonmac-SIG maillist  -  Pythonmac-SIGpython.org
>> http://mail.python.org/mailman/listinfo/pythonmac-sig
>

_________________________________________________
experimental polymedia:	www.avatar.com.au
Sonic Communications Research Group,
University of Canberra:	 www.canberra.edu.au/vc-forum/scrg
vip=Verbal Interactivity Project



_______________________________________________
Pythonmac-SIG maillist  -  Pythonmac-SIGpython.org
http://mail.python.org/mailman/listinfo/pythonmac-sig

David,

It might be stacksize related, you could try to to increase
that to see if that helps. If you use bash you can use the
following procedure to increase the stack size:

$ ulimit -Hs
<some number>
$ ulimit -s <some number>

This increases the stacksize limit to the hard limit for the
stack size. This wouldn't be real solution for the problem,
but would help pinpoint the problem.

Ronald
 
On Friday, August 31, 2007, at 11:20AM, "David
Worrall" <vipavatar.com.au> wrote:
>Thanks for your feedback Ronald.
>The file is quite large: 4-5 GB
>pytables buffers the data and controlling the buffer
size  
>occasionally stops or slows the rot.
>The code has been tested under Linux without incident.
>faltetcarabos.com, one of the developers of pytables
wrote:
>
>> A Tuesday 21 August 2007, David Worrall escrigué:
>>
>>>   I lowered NODE_MAX_SLOTS from  256 to128 and
that slowed the leak -
>>> enough to get some sort of DB happening.
>>> It eventually seg faulted, however.
>>> I've noticed that sometimes the seg fault
causes the (non-python)
>>> heap to become corrupted, requiring a HW reboot
before pytables
>>> becomes useful again.
>>> Who knows what memory it may be writing over!
>>>
>>
>> I don't think there is a leak in PyTables itself;
it is just that it
>> takes a lot of memory to work with many nodes
simultaneously.  Perhaps
>> reducing the NODE_MAX_SLOTS (to 64, for example)
could help a bit  
>> more.
>> Perhaps it would be a good idea to reduce the
memory requirements of
>> PyTables nodes (for example, avoid the creation of
buffers when they
>> are not needed), but this should take quite a few
of thought.
>>
>> Indeed it seems that there is a problem with the
MacOSX platform, but
>> the fact that this is not reproducible on Linux is
unfortunate.
>...
>> I'm thinking that perhaps you can try with newer
versions of Python
>> (2.5.1) and HDF5 (the 1.8.0 beta3 is out, and I
know that it wears a
>> completely revamped cache system, so maybe it is
worth a try).
>>
>
>Anyway I'll try with gdb as you suggest and report back.
I need to  
>put it on my task stack so it'll take some time.
>
>thanks again,
>
>David
>On 30/08/2007, at 3:48 PM, Ronald Oussoren wrote:
>
>>
>> On 30 Aug, 2007, at 1:56, David Worrall wrote:
>>
>>> Hi all,
>>>
>>> I've been using PyTables (www.pytables.org)
with python2.4 on intel
>>> Mac OSX 10.4.10
>>> and I'm running into a seg. fault when
generating a large hdf5 file.
>>>
>>> Almost certainly something to do with
relationship between OS and
>>> python.
>>> Has anyone had a similar (memory leak?)
experience?
>>
>> How large a file? It might be a memory management
bug in pytables  
>> as well.
>>
>> If you have the developer tools (Xcode &
friends) installed you can  
>> find out where the crash occurs using gdb.
>>
>> That is, 'gdb python', then on the prompt for gdb:
'r  
>> myscript.py' (adding arguments if needed). When the
crash occurs  
>> you can use 'where 20' to see the topmost 20 frames
on the C stack,  
>> or just 'where' to see the entire stack.
>>
>> Ronald
>>>
>>>
>>>
>>>
>>>
_________________________________________________
>>> experimental polymedia:	www.avatar.com.au
>>> Sonic Communications Research Group,
>>> University of Canberra:	
www.canberra.edu.au/vc-forum/scrg
>>> vip=Verbal Interactivity Project
>>>
>>>
>>>
>>>
_______________________________________________
>>> Pythonmac-SIG maillist  -  Pythonmac-SIGpython.org
>>> http://mail.python.org/mailman/listinfo/pythonmac-sig
>>
>
>_________________________________________________
>experimental polymedia:	www.avatar.com.au
>Sonic Communications Research Group,
>University of Canberra:	
www.canberra.edu.au/vc-forum/scrg
>vip=Verbal Interactivity Project
>
>
>
>_______________________________________________
>Pythonmac-SIG maillist  -  Pythonmac-SIGpython.org
>http://mail.python.org/mailman/listinfo/pythonmac-sig
>
>
_______________________________________________
Pythonmac-SIG maillist  -  Pythonmac-SIGpython.org
http://mail.python.org/mailman/listinfo/pythonmac-sig