Command failure: Too Many Files (rm, ls, tar)

I have a directory that has grown quite large (over 2500
files).

I regularly copy the files to a transfer directory, cd to
that
directory, and tar them and gzip them (tar cvzf xfer.tgz *)
to take them
to another site.

Last week I started getting errors indicating there were too
many files
for the command (rm *, ls *, tar cvzf xfer.tgz *) to
continue.

I was able to get the commands to work by reducing the file
count down
to about 1900.  Is there a magic number limit on those
commands?  Or am
I doing something wrong.  I'm currently running linux on an
Intel PC
with a Fedora 7 distribution.

Thanks.

-Tom

Thank you so much, Bob.  I knew that GNU programs aren't
supposed to
have an arbitrary limit.

And your examples and pointers are just what I need.

-Tom

Hi Bob,

* Bob Proulx wrote on Tue, Apr 15, 2008 at 06:55:35PM CEST:
> You are running into the kernel's ARG_MAX limitation. 
Please see this
> reference for more information.
> 
>   http://www.gnu.org/software/coreutils/faq/#Argu
ment-list-too-long

AFAIK this limitation has been lifted in Linux 2.6.23.  That
FAQ could
be updated for this.  On my 2.6.24, the non-limit isn't
shown:

$ getconf ARG_MAX
131072

but apparently it doesn't exist any more:

$ cd /usr/lib; /bin/echo * * * * * * * * * * * * * * * * * *
* * * * * * * | wc
      1   60950 1056000

Cheers,
Ralf

>-----Original Message-----
>From: Bob Proulx [mailto:bobproulx.com] 
>Sent: Tuesday, April 15, 2008 11:56
>To: Browder, Tom
>Cc: help-gnu-utilsgnu.org
>Subject: Re: Command failure: Too Many Files (rm, ls,
tar)
>Browder, Tom wrote:
>> Last week I started getting errors indicating there
were too many 
>> files for the command (rm *, ls *, tar cvzf
xfer.tgz *) to continue.
...
>You are running into the kernel's ARG_MAX limitation. 
Please 
>see this reference for more information.
...
>For your 'tar' case you can avoid the argument expansion

>entirely simply by giving '.' as the argument.
>
>  tar cvzf xfer.tgz .

Hm, when I try this the tgz archive gets sucked back into
itself.

-Tom

Browder, Tom wrote:
> Bob Proulx wrote:
> >For your 'tar' case you can avoid the argument
expansion 
> >entirely simply by giving '.' as the argument.
> >
> >  tar cvzf xfer.tgz .
> 
> Hm, when I try this the tgz archive gets sucked back
into itself.

Oops.  You are right.  I typed that in too quickly.  Of
course it is
possible to put the tar archive elsewhere.

  tar cvzf ../xfer.tgz .
  tar cvzf /tmp/xfer.tgz .

Also if you are simply sync'ing two directories then I
recommend using
'rsync' for this.  It is a great tool and designed for just
such
activity.

  rsync -av ./fromhere host1.example.com:/to/there/

  rsync -avz --progress ./fromhere
host1.example.com:/to/there/

Bob

Browder, Tom wrote:
> Last week I started getting errors indicating there
were too many files
> for the command (rm *, ls *, tar cvzf xfer.tgz *) to
continue.
> ...
> Is there a magic number limit on those commands?
> Or am I doing something wrong.

You are running into the kernel's ARG_MAX limitation. 
Please see this
reference for more information.

  http://www.gnu.org/software/coreutils/faq/#Argu
ment-list-too-long

> I regularly copy the files to a transfer directory, cd
to that
> directory, and tar them and gzip them (tar cvzf
xfer.tgz *) to take
> them to another site.

For your 'tar' case you can avoid the argument expansion
entirely
simply by giving '.' as the argument.

  tar cvzf xfer.tgz .

For 'rm *' I would normally either simply remove the entire
directory
with 'rm -rf ./dir' or use 'find' like this:

  find . -exec rm {} +

For 'ls *' there is no need of the '*'.  Just use ls. 
Actually using
'ls *' is inefficient because the shell expands the * to
match every
filename in the directory and ls then lists it.  But the
entire
purpose of ls is to list directories.  So 'ls *' does the
same work
twice while 'ls' does it once.  Better to simply use 'ls'.

  ls

Bob

Hi Ralf,

Ralf Wildenhues wrote:
> AFAIK this limitation has been lifted in Linux 2.6.23. 
That FAQ could
> be updated for this.

Do you know which distributions have shipped with this or
newer
kernel?  I was holding off mostly because I think it is
only
from-scratch builders that have that newest kernels that
will see this
new behavior.

> On my 2.6.24, the non-limit isn't shown:
> 
> $ getconf ARG_MAX
> 131072

I think this is a bug in glibc on kernels without a hard
limit.
(However it may be a lessor of two evils it if avoids
breakage of some
programs that always expect a limit.)  I wonder if libc
tries to
dynamically detect this or if there is a hard coded value
that needs
to be updated.  Hmm...

> but apparently it doesn't exist any more:
> 
> $ cd /usr/lib; /bin/echo * * * * * * * * * * * * * * *
* * * * * * * * * * | wc
>       1   60950 1056000

You say "apparently" but I am sure that it was you
posted a note
elsewhere noting this change in the upstream kernel.  I
followed your
reference to where it was committed into Linus' kernel tree
here:

  http://git.kernel.org/?p=linux/kernel/git/torvalds/linux
-2.6.git;a=commit;h=b6a2fea39318e43fee84fa7b0b90d68bed92d2ba


It is only a matter of time before this propagates.  But
time is also
what keeps everything from happening all at once and so we
will need
to wait a bit before this shows up on most machines in the
wild. 

Bob

* Bob Proulx wrote on Tue, Apr 15, 2008 at 09:56:08PM CEST:
> Ralf Wildenhues wrote:
> > AFAIK this limitation has been lifted in Linux
2.6.23.  That FAQ could
> > be updated for this.
> 
> Do you know which distributions have shipped with this
or newer
> kernel?  I was holding off mostly because I think it is
only
> from-scratch builders that have that newest kernels
that will see this
> new behavior.

Mine is from Debian testing or unstable.

> > $ getconf ARG_MAX
> > 131072
> 
> I think this is a bug in glibc on kernels without a
hard limit.
> (However it may be a lessor of two evils it if avoids
breakage of some
> programs that always expect a limit.)  I wonder if libc
tries to
> dynamically detect this or if there is a hard coded
value that needs
> to be updated.  Hmm...

Oh, I might not be running the latest glibc.  And I don't
consider this
a serious bug, either.  But it may cause libtool to run
slower than
necessary (as it believes getconf).

> > but apparently it doesn't exist any more:

> You say "apparently" but I am sure that it
was you posted a note
> elsewhere noting this change in the upstream kernel.

Yep, just checking whether you were paying attention.  

> It is only a matter of time before this propagates. 
But time is also
> what keeps everything from happening all at once and so
we will need
> to wait a bit before this shows up on most machines in
the wild. 

Sure.  But you can also spread the word now, maybe that will
cause it to
propagate faster!

Cheers,
Ralf

Ralf Wildenhues wrote:
> Mine is from Debian testing or unstable.

Of course Debian Testing at this time is in the queue for
release as
the upcoming stable Lenny.  But it hasn't released yet.  I
try to
avoid advocating normal users to use unreleased bits. 
Developers are
on their own and everyone has their own ideas of how to
manage systems
anyway and it isn't productive trying to get developers to
agree.  And
of course Debian unstable Sid is always development and
never
releases.  (I know you know this but am just saying it for
the archive
for readers who come later.)

> > It is only a matter of time before this
propagates.  But time is also
> > what keeps everything from happening all at once
and so we will need
> > to wait a bit before this shows up on most
machines in the wild. 
> 
> Sure.  But you can also spread the word now, maybe that
will cause it to
> propagate faster!

You have prodded me sufficiently.  I added a short blurb
about it at
the end of this entry:

  http://www.gnu.org/software/coreutils/faq/#Argu
ment-list-too-long

If you would like to see more there feel free to suggest
something! 

Bob