RE: bric_soap

> It depends on what you're doing. But yes, if you do a
bulk publish,  
> for example, you can easily crush your server. Use the
--chunks  
> option to break things up.

Oh yes, I always use chunks when republishing.

Here is a sample of what I was doing:

bric_soap story list_ids --search primary_uri=%/2006/12%
--search
element=story | bric_soap story export ->
stories_2006_12.xml --timeout
0

For this particular example, the apache process got up to
about 75% of
CPU usage. Prior to learning this, I was running bric_soap
command after
another without letting the apache process cycle down.

Chris

 
Is there a way to set the priority of these soap processes
when they're started?  Using 75% of the CPU is probably not
such a bad thing so long as the soap process resource
utilization is throttled back when higher priority processes
are running.

	-----Original Message----- 
	From: Chris Schults [mailto:cschultsgrist.org] 
	Sent: Wed 2/14/2007 12:54 PM 
	To: userslists.bricolage.cc 
	Cc: 
	Subject: RE: bric_soap
	
	

	> It depends on what you're doing. But yes, if you do a
bulk publish, 
	> for example, you can easily crush your server. Use the
--chunks 
	> option to break things up.
	
	Oh yes, I always use chunks when republishing.
	
	Here is a sample of what I was doing:
	
	bric_soap story list_ids --search primary_uri=%/2006/12%
--search
	element=story | bric_soap story export ->
stories_2006_12.xml --timeout
	0
	
	For this particular example, the apache process got up to
about 75% of
	CPU usage. Prior to learning this, I was running bric_soap
command after
	another without letting the apache process cycle down.
	
	Chris
	

> bric_soap story list_ids --search
primary_uri=%/2006/12% --search
> element=story | bric_soap story export ->
stories_2006_12.xml -- 
> timeout
> 0

A while back (maybe years by now) Scott posted a bash script
to take  
a list like that and export it as individual bric_soap jobs
instead  
of as a single server-destroying operation.  Here's the
modification  
I used for our recent migration:

for id in $(< bulk.txt)
do
	echo $id
	date
	bric_soap story export $id > $id.xml
	date
	sleep 0.1
done > bulk-story-export.log 2>&1

After creating the bulk.txt file with your bric_soap
list_ids  
operation, you run the above script, logout, and go home. 
The next  
morning you've got thousands of XML files with one story
each, and a  
logfile that looks like this:

story_1158
Mon Sep 11 11:40:19 EDT 2006
Mon Sep 11 11:40:20 EDT 2006
story_1159
Mon Sep 11 11:40:20 EDT 2006
Mon Sep 11 11:40:20 EDT 2006
story_1160
Mon Sep 11 11:40:20 EDT 2006
Mon Sep 11 11:40:21 EDT 2006
story_1161
Mon Sep 11 11:40:21 EDT 2006
Mon Sep 11 11:40:22 EDT 2006

If there's an error, it will show up between the timestamps.
 Since  
each export is a separate bric_soap call this won't kill the
whole  
operation, and you can search for the errors to deal with
them by hand.


--
Neal Sofge, IST External Web Program Manager
RAND Corporation <http://www.rand.org/>




--------------------

This email message is for the sole use of the intended
recipient(s) and
may contain privileged information. Any unauthorized review,
use,
disclosure or distribution is prohibited. If you are not the
intended
recipient, please contact the sender by reply email and
destroy all copies
of the original message.

On Feb 20, 2007, at 20:16, Neal Sofge wrote:

>> bric_soap story list_ids --search
primary_uri=%/2006/12% --search
>> element=story | bric_soap story export ->
stories_2006_12.xml -- 
>> timeout
>> 0
>
> A while back (maybe years by now) Scott posted a bash
script to  
> take a list like that and export it as individual
bric_soap jobs  
> instead of as a single server-destroying operation.

I recommend using the --use-cookie-file option to cut the
number of  
requests to the server in half. Just run some arbitrary
bric_soap  
search using --save-cookie-file, and then set up your script
to use  
that file for all subsequent requests.

Best,

David

On Wed, 21 Feb 2007, David E. Wheeler wrote:
> On Feb 20, 2007, at 20:16, Neal Sofge wrote:
>> A while back (maybe years by now) Scott posted a
bash script to take a list 
>> like that and export it as individual bric_soap
jobs instead of as a single 
>> server-destroying operation.
>
> I recommend using the --use-cookie-file option to cut
the number of requests 
> to the server in half. Just run some arbitrary
bric_soap search using 
> --save-cookie-file, and then set up your script to use
that file for all 
> subsequent requests.

Yep, I use:  bric_soap site --save-cookie-file cookie.txt
list_ids
because it's the shortest I think.

Thanks for the $(< foo.txt) bash trick.