Copying From NTFS Mount & oom-killer

Hi,

When copying large amounts of data (> 25GB) from an NTFS
mount, my kernel
panics.

escabot log # uname -a
Linux escabot 2.6.21-gentoo-r4 #3 SMP PREEMPT Thu Sep 13
14:44:12 EDT 2007
i686 AMD Opteron(tm) Processor 280 AuthenticAMD GNU/Linux

I watch top while copying the files and it just sucks up all
of the
memory. It gets to a point for a while where it seems to
copy fine leaving
just a few megabytes of memory free. If I copy smaller sets
of data, it
seems to work fine (but still sucks all the memory it can).
This machine
has 12GB of memory.

I have tried several times and I can pretty much get it to
always dump if
I copy enough stuff over. The last time I recompiled the
kernel with NTFS
debug support and enabled the debugging messages. I am
attaching a gzip'd
file of the kernel messages.

If attaching a file and sending it to the mailing list is
improper
etiquette, let me know and I can post it on a web page or do
something
else.

I'm not sure if this is a bug or something I'm doing wrong.

Thank you for your time.


--Marc

------------------------------------------------------------
-------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Linux-NTFS-User mailing list
Linux-NTFS-Userlists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-n
tfs-user

Hi,

On 13 Sep 2007, at 21:35, marc.smithesmail.mcc.edu wrote:
> When copying large amounts of data (> 25GB) from an
NTFS mount, my  
> kernel
> panics.

Ouch.  Would it be possible for you to write down and email
the  
actual panic text?  (Or capture it via serial console if you
don't  
want to type it by hand...)

> escabot log # uname -a
> Linux escabot 2.6.21-gentoo-r4 #3 SMP PREEMPT Thu Sep
13 14:44:12  
> EDT 2007
> i686 AMD Opteron(tm) Processor 280 AuthenticAMD
GNU/Linux

Out of curiosity.  Why are you running a 32-bit kernel on
such a  
large 64-bit machine?!?  Your machine would run a lot
quicker with a  
64-bit kernel (and 64-bit user space)...

> I watch top while copying the files and it just sucks
up all of the
> memory. It gets to a point for a while where it seems
to copy fine  
> leaving
> just a few megabytes of memory free. If I copy smaller
sets of  
> data, it
> seems to work fine (but still sucks all the memory it
can). This  
> machine
> has 12GB of memory.
>
> I have tried several times and I can pretty much get it
to always  
> dump if
> I copy enough stuff over. The last time I recompiled
the kernel  
> with NTFS
> debug support and enabled the debugging messages. I am
attaching a  
> gzip'd
> file of the kernel messages.

Great, thanks.

> If attaching a file and sending it to the mailing list
is improper
> etiquette, let me know and I can post it on a web page
or do something
> else.

No that is fine!

> I'm not sure if this is a bug or something I'm doing
wrong.
>
> Thank you for your time.

The kernel panic()ing is always a kernel bug!

The question is what are you copying to?

You said you are copying those large files from NTFS but
never said  
what file system is the destination...

To see whether it is NTFS at fault (I don't think it is, I
think it  
is either your destination file system or a bug in the
VFS/VM not a  
in a file system at all) try to copy to /dev/null, i.e. if
you are  
copying using "cp" command then do:

cp /path/to/ntfs/volume/huge26gbfile /dev/null

If you use "dd" then do:

dd if=/path/to/ntfs/volume/huge26gbfile of=/dev/null [insert
your  
usual dd options here]

Also out of interest what command are you exactly running to
do the  
copying?

That will effectively read in the whole file but throw what
it reads  
away instead of writing it to another file system.

Does this work or also panic?

My expectation is that it will work fine but there is only
one way to  
find out and that is for you to try it!  (-:

Best regards,

	Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at
with )
Unix Support, Computing Service, University of Cambridge,
CB2 3QH, UK
Linux NTFS maintainer, http://www.linux-ntfs.org/




------------------------------------------------------------
-------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Linux-NTFS-User mailing list
Linux-NTFS-Userlists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-n
tfs-user

> Hi,
>
> On 13 Sep 2007, at 21:35, marc.smithesmail.mcc.edu wrote:
>> When copying large amounts of data (> 25GB) from
an NTFS mount, my
>> kernel
>> panics.
>
> Ouch.  Would it be possible for you to write down and
email the
> actual panic text?  (Or capture it via serial console
if you don't
> want to type it by hand...)

I guess I may have stated that wrong, I don't think the ntfs
driver is
making the kernel panic directly, I think it is using up all
of the memory
and then when another process tries to use some memory, that
invokes
oom-killer. It doesn't seem like it always makes the kernel
panic, but I
think it is init that invokes oom-killer because there is no
more memory:
init invoked oom-killer

Is that correct?

Here is the message I get when the kernel panics:
Kernel panic - not syncing: Out of memory and no killable
processes.
..
Kernel panic - not syncing: Out of memory and no killable
processes.

>
>> escabot log # uname -a
>> Linux escabot 2.6.21-gentoo-r4 #3 SMP PREEMPT Thu
Sep 13 14:44:12
>> EDT 2007
>> i686 AMD Opteron(tm) Processor 280 AuthenticAMD
GNU/Linux
>
> Out of curiosity.  Why are you running a 32-bit kernel
on such a
> large 64-bit machine?!?  Your machine would run a lot
quicker with a
> 64-bit kernel (and 64-bit user space)...
>

I really don't have a good reason, I probably should be. To
be honest, I
kind of forgot this was an Opteron until you mentioned that.
=)

>> I watch top while copying the files and it just
sucks up all of the
>> memory. It gets to a point for a while where it
seems to copy fine
>> leaving
>> just a few megabytes of memory free. If I copy
smaller sets of
>> data, it
>> seems to work fine (but still sucks all the memory
it can). This
>> machine
>> has 12GB of memory.
>>
>> I have tried several times and I can pretty much
get it to always
>> dump if
>> I copy enough stuff over. The last time I
recompiled the kernel
>> with NTFS
>> debug support and enabled the debugging messages. I
am attaching a
>> gzip'd
>> file of the kernel messages.
>
> Great, thanks.
>
>> If attaching a file and sending it to the mailing
list is improper
>> etiquette, let me know and I can post it on a web
page or do something
>> else.
>
> No that is fine!
>
>> I'm not sure if this is a bug or something I'm
doing wrong.
>>
>> Thank you for your time.
>
> The kernel panic()ing is always a kernel bug!
>
> The question is what are you copying to?
>

I was copying to an ext3 file system.

> You said you are copying those large files from NTFS
but never said
> what file system is the destination...
>
> To see whether it is NTFS at fault (I don't think it
is, I think it
> is either your destination file system or a bug in the
VFS/VM not a
> in a file system at all) try to copy to /dev/null, i.e.
if you are
> copying using "cp" command then do:
>
> cp /path/to/ntfs/volume/huge26gbfile /dev/null
>
> If you use "dd" then do:
>
> dd if=/path/to/ntfs/volume/huge26gbfile of=/dev/null
[insert your
> usual dd options here]
>
> Also out of interest what command are you exactly
running to do the
> copying?
>

Before trying /dev/null just a: cp -R IMAGES /root

> That will effectively read in the whole file but throw
what it reads
> away instead of writing it to another file system.
>
> Does this work or also panic?
>

escabot CAD # ls -ltrh
total 49G
-r-------- 1 root root 2.0G Oct  7  2004 RTC26001.GHS
-r-------- 1 root root 2.0G Oct  7  2004 RTC26002.GHS
-r-------- 1 root root 2.0G Oct  7  2004 RTC2604.GHO
-r-------- 1 root root 620M Oct  7  2004 RTC26003.GHS
-r-------- 1 root root    4 Sep 14 09:32 asdf.txt
-r-------- 1 root root    8 Sep 14 09:33 rty.txt
-r-------- 1 root root  42G Sep 14 10:36 bigfile.gho
escabot CAD # cp bigfile.gho /dev/null


> My expectation is that it will work fine but there is
only one way to
> find out and that is for you to try it!  (-:
>
> Best regards,
>
> 	Anton
> --
> Anton Altaparmakov <aia21 at cam.ac.uk> (replace
at with )
> Unix Support, Computing Service, University of
Cambridge, CB2 3QH, UK
> Linux NTFS maintainer, http://www.linux-ntfs.org/

>

Thank you for your time and help. I look forward to hearing
from you soon.

--Marc

>



------------------------------------------------------------
-------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Linux-NTFS-User mailing list
Linux-NTFS-Userlists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-n
tfs-user

Hi,

On 14 Sep 2007, at 16:21, marc.smithesmail.mcc.edu wrote:
>> On 13 Sep 2007, at 21:35, marc.smithesmail.mcc.edu wrote:
>>> When copying large amounts of data (> 25GB)
from an NTFS mount, my
>>> kernel
>>> panics.
>>
>> Ouch.  Would it be possible for you to write down
and email the
>> actual panic text?  (Or capture it via serial
console if you don't
>> want to type it by hand...)
>
> I guess I may have stated that wrong, I don't think the
ntfs driver is
> making the kernel panic directly, I think it is using
up all of the  
> memory
> and then when another process tries to use some memory,
that invokes
> oom-killer. It doesn't seem like it always makes the
kernel panic,  
> but I
> think it is init that invokes oom-killer because there
is no more  
> memory:
> init invoked oom-killer
>
> Is that correct?

Not necessarily.  It may well be ext3 or some other bug in
the kernel  
that is causing it, not necessarily NTFS at all...

> Here is the message I get when the kernel panics:
> Kernel panic - not syncing: Out of memory and no
killable processes.
> ..
> Kernel panic - not syncing: Out of memory and no
killable processes.

Right, ok, thanks.

>>> escabot log # uname -a
>>> Linux escabot 2.6.21-gentoo-r4 #3 SMP PREEMPT
Thu Sep 13 14:44:12
>>> EDT 2007
>>> i686 AMD Opteron(tm) Processor 280 AuthenticAMD
GNU/Linux
>>
>> Out of curiosity.  Why are you running a 32-bit
kernel on such a
>> large 64-bit machine?!?  Your machine would run a
lot quicker with a
>> 64-bit kernel (and 64-bit user space)...
>
> I really don't have a good reason, I probably should
be. To be  
> honest, I
> kind of forgot this was an Opteron until you mentioned
that. =)

(-:

>>> I watch top while copying the files and it just
sucks up all of the
>>> memory. It gets to a point for a while where it
seems to copy fine
>>> leaving
>>> just a few megabytes of memory free. If I copy
smaller sets of
>>> data, it
>>> seems to work fine (but still sucks all the
memory it can). This
>>> machine
>>> has 12GB of memory.
>>>
>>> I have tried several times and I can pretty
much get it to always
>>> dump if
>>> I copy enough stuff over. The last time I
recompiled the kernel
>>> with NTFS
>>> debug support and enabled the debugging
messages. I am attaching a
>>> gzip'd
>>> file of the kernel messages.
>>
>> Great, thanks.
>>
>>> If attaching a file and sending it to the
mailing list is improper
>>> etiquette, let me know and I can post it on a
web page or do  
>>> something
>>> else.
>>
>> No that is fine!
>>
>>> I'm not sure if this is a bug or something I'm
doing wrong.
>>>
>>> Thank you for your time.
>>
>> The kernel panic()ing is always a kernel bug!
>>
>> The question is what are you copying to?
>
> I was copying to an ext3 file system.

Ok.  That may well be causing the problem given the size of
your  
memory and the fact that you are using a 32-bit kernel
because ext3  
only can use lowmem, i.e. well less than 1GiB of RAM for
buffers and  
that means it might fill up all of lowmem easily that way
when you  
have so much highmem available.  NTFS uses highmem for
everything so  
it should not be at fault.

>> You said you are copying those large files from
NTFS but never said
>> what file system is the destination...
>>
>> To see whether it is NTFS at fault (I don't think
it is, I think it
>> is either your destination file system or a bug in
the VFS/VM not a
>> in a file system at all) try to copy to /dev/null,
i.e. if you are
>> copying using "cp" command then do:
>>
>> cp /path/to/ntfs/volume/huge26gbfile /dev/null
>>
>> If you use "dd" then do:
>>
>> dd if=/path/to/ntfs/volume/huge26gbfile
of=/dev/null [insert your
>> usual dd options here]
>>
>> Also out of interest what command are you exactly
running to do the
>> copying?
>>
>
> Before trying /dev/null just a: cp -R IMAGES /root
>
>> That will effectively read in the whole file but
throw what it reads
>> away instead of writing it to another file system.
>>
>> Does this work or also panic?
>
> escabot CAD # ls -ltrh
> total 49G
> -r-------- 1 root root 2.0G Oct  7  2004 RTC26001.GHS
> -r-------- 1 root root 2.0G Oct  7  2004 RTC26002.GHS
> -r-------- 1 root root 2.0G Oct  7  2004 RTC2604.GHO
> -r-------- 1 root root 620M Oct  7  2004 RTC26003.GHS
> -r-------- 1 root root    4 Sep 14 09:32 asdf.txt
> -r-------- 1 root root    8 Sep 14 09:33 rty.txt
> -r-------- 1 root root  42G Sep 14 10:36 bigfile.gho
> escabot CAD # cp bigfile.gho /dev/null
>
>> My expectation is that it will work fine but there
is only one way to
>> find out and that is for you to try it!  (-:
>
> Thank you for your time and help. I look forward to
hearing from  
> you soon.

You forgot to say what happened!!!  Did it panic or did it
work when  
you did  the cp to /dev/null?

Best regards,

	Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at
with )
Unix Support, Computing Service, University of Cambridge,
CB2 3QH, UK
Linux NTFS maintainer, http://www.linux-ntfs.org/




------------------------------------------------------------
-------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Linux-NTFS-User mailing list
Linux-NTFS-Userlists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-n
tfs-user