opening large data files in Perl

Hi

Is someone able to tell me if when using the open command to
read data
from a file, if the file was large, say 60mb or more, would
it load
much of the file into memory or would it only take a record
at a time?

To expand further, I currently use a perl script to open a
data file,
search each record for a match to a user input and then
store the
matched data in an array. I stop the search if there are
more than 200
results so the array never exceeds this, but as the data
file is so
large I wonder if there is any load on server memory,
especially if
numerous users were accessing it at once.

Thanks
pppe


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Perl Programming" group.
To post to this group, send email to perl-programminggooglegroups.com
To unsubscribe from this group, send email to
perl-programming-unsubscribegooglegroups.com
For more options, visit this group at http:
//groups.google.com/group/perl-programming
-~----------~----~----~----~------~----~------~--~---

Hmm, good question.

Re: first paragraph, my impression was that the open
function only
associates an internal filehandle with an external file
specification.
So only the operating system's information about the file
would be
stored in the memory.  But the actual content of the file
would not be
loaded until a record is read.  How much of the file gets
read into
memory at that time?  I can't really think of a reason why
it should be
any more than the size of the record read.

As I said though, that has only been my impression. 
Unfortunately I
was not successful in finding any concrete answers.

I'd encourage you to run a super search on http://www.perlmonks.org
and
post your question in the Seekers of Perl Wisdom forum if
you do not
find anything.

My request to you then is to please come back and post the
answer you
get here.

Cheers.


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Perl Programming" group.
To post to this group, send email to perl-programminggooglegroups.com
To unsubscribe from this group, send email to
perl-programming-unsubscribegooglegroups.com
For more options, visit this group at http:
//groups.google.com/group/perl-programming
-~----------~----~----~----~------~----~------~--~---

pppe wrote:

> Is someone able to tell me if when using the open
command to read data
> from a file, if the file was large, say 60mb or more,
would it load
> much of the file into memory or would it only take a
record at a time?

Using the open() function (it's not a
"command") does not load any of
the file's data into memory.  The file's data is loaded
into memory
when you actually *read* the file, using either read(),
readline(), or
the <> operator.

How much is read into memory depends on how you're reading
it.  If you
use read(), you specify exactly how many bites are read at
any one
time.  If you use readline(), you are reading one line into
memory at
any one time.  If you use <>, then it depends on the
context in which
you use the operator:

In scalar context, <> reads one line at a time.  In
list context, it
reads the entire file into memory:

while (my $line = <$fh>) {
   #$line contains one line.  When the loop begins the
   #next iteration, $line is destroyed, and the memory
released,
   #and a new $line is assigned to the next line of the file
}

my lines = <$fh>;
foreach my $line (lines) {
   #here, lines contains the *entire* file,
   #all stored in memory.  This is generally considered
   #a BAD thing.
}

foreach my $line (<$fh>) {
  #here, <$fh> is being evaluated in a list context
  #and so the entire file is read into memory, just
  #like the previous example that used an array.
}

In general, never use an array or a foreach to read from a
file using <
>.  Always use a while loop to iterate through the file. 
The sole exception is when you need to be able to process
more than one line at a time.  And even then, there are
often work arounds to reading the entire file into memory.

Paul Lalli


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Perl Programming" group.
To post to this group, send email to perl-programminggooglegroups.com
To unsubscribe from this group, send email to
perl-programming-unsubscribegooglegroups.com
For more options, visit this group at http:
//groups.google.com/group/perl-programming
-~----------~----~----~----~------~----~------~--~---