help with qx?

Hello all,

Sorry for not mailing to kick the crap out of
Perl5/6/MVC/Catalyst/ 
Maypole/Jifty/Mornington Crescent/other mongers..., but I
have a  
small technical question to ask and a room full of perl
sceptics  
(a.k.a C# developers) waiting for a script to cure their
build process!

The script has a system call (to the svn command-line
client), and it  
could possibly return a lot of lines back so I don't want to
store it  
in an array, but process it as it's coming through. Is there
a way to  
force qx or `` into list context, or something equally
helpful?

e.g

while( list qx($command) ) {
	do stuff...
}


Any help is much appreciated!

Iain

On Wed, Feb 06, 2008 at 12:39:20AM +0000, Iain Barnett
wrote:

> (a.k.a C# developers) waiting for a script to cure
their build process!

There would be some reason why they don't want to write this
in C#? 

> The script has a system call (to the svn command-line
client), and it  
> could possibly return a lot of lines back so I don't
want to store it  
> in an array, but process it as it's coming through. Is
there a way to  
> force qx or `` into list context, or something equally
helpful?
> 
> e.g
> 
> while( list qx($command) ) {
> 	do stuff...
> }
> 
> 
> Any help is much appreciated!

something like this?

open my $fh, '-|', $command or die $!;
while (<$fh>) {
	do stuff ...
}
close $fh or die $!;

(untested, and I'm not sure if $! is the most useful thing
for diagnostics.
But you want to use open)

Nicholas Clark

If you use it in a list context, you will get a list of
lines rather than one
big string. However, you won't be able to process it in an
"async" sort of
manner. The command will run to completion as part of
creating the list that
you then iterate over. So you'll still have used the same
amount of memory.

Randy
-- 
""""""""""
""""""""""
""""""""""
""""""""""
""""""""""
""""""""""
""""""""""
"""""""""
Randy J. Ray      Sunnyvale, CA      http://www.rjray.org  
rjrayblackperl.com

Silicon Valley Scale Modelers: http://www.svsm.org

On 6 Feb 2008, at 00:39, Iain Barnett wrote:

> Hello all,
>
> Sorry for not mailing to kick the crap out of
Perl5/6/MVC/Catalyst/ 
> Maypole/Jifty/Mornington Crescent/other mongers..., but
I have a  
> small technical question to ask and a room full of perl
sceptics  
> (a.k.a C# developers) waiting for a script to cure
their build  
> process!
>
> The script has a system call (to the svn command-line
client), and  
> it could possibly return a lot of lines back so I don't
want to  
> store it in an array, but process it as it's coming
through. Is  
> there a way to force qx or `` into list context, or
something  
> equally helpful?
>
> e.g
>
> while( list qx($command) ) {
> 	do stuff...
> }
>
>
> Any help is much appreciated!


my cmd = ( 'svn', 'up', '-r', $rev );
open my $ch, '-|', cmd or die "Oops";
while ( defined ( my $line = <$ch> ) ) {
     chomp $line; # or whatever
}
close $ch or die "Another oops";

The check on whether close succeeeded is important. Any
error may not  
be detected until then.

-- 
Andy Armstrong, Hexten

On Wed, Feb 06, 2008 at 12:39:20AM +0000, Iain Barnett
wrote:
> Hello all,
>
> Sorry for not mailing to kick the crap out of 
> Perl5/6/MVC/Catalyst/Maypole/Jifty/Mornington
Crescent/other mongers..., 
> but I have a small technical question to ask and a room
full of perl 
> sceptics (a.k.a C# developers) waiting for a script to
cure their build 
> process!
>
> The script has a system call (to the svn command-line
client), and it could 
> possibly return a lot of lines back so I don't want to
store it in an 
> array, but process it as it's coming through. Is there
a way to force qx or 
> `` into list context, or something equally helpful?

You might find IPC::Open3 / open3 useful here - it spawns
the command and gives
you back file handles for stdin, stderr.
See http://www.perl.com/doc/manual/html/lib/IPC/Open3.html


If you're just handling, say, stdout, then you might find it
simpler to just
use open with the | parameter, ie:

    my $cmd_output = IO::File->new('svn checkout foo |')
        or die("Error: $!");
    while (<$cmd_output>) {
        next unless /thing you're looking for/;
        warn "about stuffn";
    }

You can find discussion on these topics with 'perldoc
perlipc'.

cheers,
Toby

On Wed, Feb 06, 2008 at 12:39:20AM +0000, Iain Barnett
wrote:
>
> The script has a system call (to the svn command-line
client), and it  
> could possibly return a lot of lines back so I don't
want to store it in 
> an array, but process it as it's coming through. Is
there a way to force 
> qx or `` into list context, or something equally
helpful?
>
> e.g
>
> while( list qx($command) ) {
> 	do stuff...
> }
>

You can use the right half of the secret goaste operator to
force list
context:

    ()= qx($command);

but I doubt that will DWYM.

Using open() will work better:

    open my $fh, '-|', $command or die "Boff! Thwacke!!
Kapow!!!";
    while (<$fh>) {
        ...
    }

-- 
 Philippe Bruhat (BooK)

 Everyone acts their age... mental, if not physical.
                                    (Moral from Groo The
Wanderer #92 (Epic))

On 6 Feb 2008, at 12:51 am, Nicholas Clark wrote:

> On Wed, Feb 06, 2008 at 12:39:20AM +0000, Iain Barnett
wrote:
>
>> (a.k.a C# developers) waiting for a script to cure
their build  
>> process!
>
> There would be some reason why they don't want to write
this in  
> C#? 
>

a) they didn't want to write anything at all, as they're so
used to  
having to do everything through the gui because it's
Windoze, they  
got used to it taking 30 mins/n+ hours to do a build and
deploy for a  
small website, and for a lot of human intervention along the
way.  
They're used to it failing, I just think "why?".
b) if I write it in C# they can interfere more easily 
c) it's a good excuse to use more perl and try to make the
jump to  
'intermediate'! I'm tired of Windows development.


> But you want to use open)
>
> Nicholas Clark


Seems like open() with a pipe is the way. Thanks Nicholas
and Andy,  
and Randy for the insight about lists.

Iain

>>>>> "RJR" == Randy J Ray
<rjrayblackperl.com> writes:

  RJR> If you use it in a list context, you will get a
list of lines
  RJR> rather than one big string. However, you won't be
able to process
  RJR> it in an "async" sort of manner. The
command will run to
  RJR> completion as part of creating the list that you
then iterate
  RJR> over. So you'll still have used the same amount of
memory.

and it will be unlikely that his command will generate
enough output to
fill a stdio buffer in the qx or the open pipe. so looping
over the qx
lines in a list context or reading line by line from the
open pipe will
be about the same speed and storage. i would lean to the qx
as it is
slightly simpler to code up. the open pipe has the issue of
getting the
exit status from the close call which is easy to screw up or
forget.

uri

-- 
Uri Guttman  ------  uristemsystems.com  -------- 
http://www.sysarch.com
--
-----  Perl Architecture, Development, Training, Support,
Code Review  ------
-----------  Search or Offer Perl Jobs  ----- http://jobs.perl.org 
---------
---------  Gourmet Hot Cocoa Mix  ----  http://bestfriendscocoa.c
om ---------

>>>>> "RJR" == Randy J Ray
<rjrayblackperl.com> writes:

  >> and it will be unlikely that his command will
generate enough output to
  >> fill a stdio buffer in the qx or the open pipe.
so looping over the qx
  >> lines in a list context or reading line by line
from the open pipe will
  >> be about the same speed and storage.

  RJR> He did specifically say that he expected the
command to generate
  RJR> a considerable amount of output. I suppose that
depends on
  RJR> whether his concept of "considerable"
matches ours. Pulling large
  RJR> diffs or log history on a sizable project could
produce fairly
  RJR> large output.

if that is the case then open | will be more efficient. but
with today's
ram and buffer sizes, even 1Mb is small. we haven't been
told the output
size but if it is < 1MB (or even more given the ram
available) then
buffering will not matter much. so again, simpler coding
should win out
and qx is simpler than open |.

uri

-- 
Uri Guttman  ------  uristemsystems.com  -------- 
http://www.sysarch.com
--
-----  Perl Architecture, Development, Training, Support,
Code Review  ------
-----------  Search or Offer Perl Jobs  ----- http://jobs.perl.org 
---------
---------  Gourmet Hot Cocoa Mix  ----  http://bestfriendscocoa.c
om ---------

On 6 Feb 2008, at 7:11 am, Uri Guttman wrote:

>>>>>> "RJR" == Randy J Ray
<rjrayblackperl.com> writes:
>
>>> and it will be unlikely that his command will
generate enough  
>>> output to
>>> fill a stdio buffer in the qx or the open pipe.
so looping over  
>>> the qx
>>> lines in a list context or reading line by line
from the open  
>>> pipe will
>>> be about the same speed and storage.
>
>   RJR> He did specifically say that he expected the
command to  
> generate
>   RJR> a considerable amount of output. I suppose
that depends on
>   RJR> whether his concept of
"considerable" matches ours. Pulling  
> large
>   RJR> diffs or log history on a sizable project
could produce fairly
>   RJR> large output.
>
> if that is the case then open | will be more efficient.
but with  
> today's
> ram and buffer sizes, even 1Mb is small. we haven't
been told the  
> output
> size but if it is < 1MB (or even more given the ram
available) then
> buffering will not matter much. so again, simpler
coding should win  
> out
> and qx is simpler than open |.
>
> uri
>
> -- 
> Uri Guttman  ------  uristemsystems.com  -------- 
http:// 
> www.sysarch.com --
> -----  Perl Architecture, Development, Training,
Support, Code  
> Review  ------
> -----------  Search or Offer Perl Jobs  ----- http://jobs.perl.org   
> ---------
> ---------  Gourmet Hot Cocoa Mix  ----  http://bestfriendscocoa.c
om  
> ---------


I'm quite embarrassed to say it ended up being approx
250k... (gets  
coat)

In my defence, the buffer on the Windows command terminal
was so  
small that I couldn't tell how much there was - it looked
like a lot  
more than that!    Still,
could be a lot more output depending on  
what is checked out, but you're probably both right about
the power  
of the machines etc.

I ended up using a pipe because it still printed to STDOUT
with a  
filehandle. Once I got it working I just left it as is.

Iain