Re: bric_soap publish timeout error

Hi Dawn,

Did you ever figure out the particular reason your stories
were taking so
long in soap and worked so quickly in the interface?

I'm working on a development copy of the CMS for
etonline.com, and I'm
having trouble publishing media items with soap.  I can find
these same
items in the interface and they publish within seconds. 
There is nothing
special about them, and they don't really even have
"templates" - not custom
templates written by me.  They're just media.

timeout=0 isn't working  for me... it just hangs - for ages.
 i tried
timeout 300 and nothing there.  am waiting still while a
timeout=0 publish
attempt sits there... doing nothing.

!!!!!!!!!!!!!!!!!! HOLD ON, IT JUST WORKED
!!!!!!!!!!!!!!!!!!!!

OK - so here's something for the wiki (maybe it's already
there someplace
and I didn't see it)....

I'm using bric 1.10.2 and just as I was waiting and typing
it went from
hanging like this:
rootadmin:~/bin# ./publish.sh
media list_ids: Authenticating to Bricolage...
workflow publish: Authenticating to Bricolage...
media list_ids: Login success.
media list_ids: Switched to media module.
media list_ids: Calling Bric::SOAP::Media->list_ids with
search:
(cover_date_start => 2007-08-07 00:00:00, element_type_id
=> 4,
cover_date_end => 2007-08-09 00:00:00)
workflow publish: Login success.
workflow publish: Switched to workflow module.
workflow publish: Reading ids from STDIN...
workflow publish: Calling Workflow->publish with ids:
media_id => 27306,
media_id => 27311, media_id => 27316, media_id =>
27320, media_id => 27322,
media_id => 27324, media_id => 27328, media_id =>
27335, media_id => 27337,
media_id => 27339, media_id => 27341, media_id =>
27346, media_id => 27351,
media_id => 27354, media_id => 27359, media_id =>
27362, media_id => 27365,
media_id => 27370, media_id => 27375, media_id =>
27381, media_id => 27386,
media_id => 27388, media_id => 27390, media_id =>
27393, media_id => 27395,
media_id => 27400, media_id => 27407, media_id =>
27412, media_id => 27416,
media_id => 27418, media_id => 27426, media_id =>
27428, media_id => 27430,
media_id => 27433, media_id => 27438, media_id =>
27441, media_id => 27443,
media_id => 27445, media_id => 27448, media_id =>
27453, media_id => 27455,
media_id => 27457, media_id => 27459, media_id =>
27462, media_id => 27467,
media_id => 27469, media_id => 27474, media_id =>
27481, media_id => 27486

..... (*me impatiently waiting here*)...

then suddenly:

media_27306
media_27311
media_27316
media_27320
media_27322
media_27324
media_27328
media_27335
media_27337
media_27339
media_27341
media_27346
media_27351
media_27354
media_27359
media_27362
media_27365
media_27370
media_27375
media_27381
media_27386
media_27388
media_27390
media_27393
media_27395
media_27400
media_27407
media_27412
media_27416
media_27418
media_27426
media_27428
media_27430
media_27433
media_27438
media_27441
media_27443
media_27445
media_27448
media_27453
media_27455
media_27457
media_27459
media_27462
media_27467
media_27469
media_27474
media_27481
media_27486


they all spit out at once.  So - bric_soap - in current
versions - with
media at least... schedules the jobs all at once and then
waits silently for
them ALL to publish before letting you know it was doing
them all.  And if
it doesn't get them all done, it fails with a read timeout
error with
nothing written to the logs...  So if you have very many
items to publish,
timeout=0 *does work* and is practically required.

jd


On 5/4/06, Dawn Buie <dawnsqueegeemedia.com>
wrote:
>
>
> On 4-May-06, at 12:40 AM, Scott Lanning wrote:
>
> > On Wed, 3 May 2006, Dawn wrote:
> >> I've been in republish purgatory trying to
figure out why
> >> bric_soap is no long able - so it seems - to
republish large
> >> numbers of stories.  I can republish from my
Web UI fine, but I
> >> need to republish all the element=story in the
whole site.
> >
> > You use bric_queued?
>
> No I'm using bric_dist_mon and Bricolage 1.8.8
(hopefully soon
> upgrading to 10.1.1 or .2???)
>
> > The bric_soap in your version of Bricolage (1.8.5
IIRC)
> > doesn't "schedule" publish jobs like the
queued does, it "burns" them
> > immediately (fixed 1.8.9). This could explain the
difference between
> > publishing through the UI and through bric_soap.
>
> Hmmm interesting.
>
> >   Depending on elements, data, and the
"quality" of your templates,
>
> yes I could streamline my templates a little more I
think.
>
> > publishing can take a long time indeed... I used
to put bulk publishes
> > in a for loop (you don't need to after 1.8.9 since
publish jobs are
> > only
> > scheduled), like
> >
> > for i in `bric_soap story --element_key_name=story
list_ids`;
> > do
> >     bric_soap workflow --timeout 222 publish $i;
> > done
> >
> > with extra crap around it like `date` and echoing
the ID in case
> > there's an error.
>
> thank you so much for this info - hopefully soon I'll
be able to
> upgrade and be done with this problem.
>
>
> BTW David's suggestion of leaving timeout 0 seems to
have done the
> trick for me so far at least with republishing the most
recent
> stories in small batches.
>
>
> > Also --use-cookie-file to avoid reauthenticating
> > each time.
>
> I hacked my bric_soap utility to use my username and
server by
> default. is that bad? (I did that after I had said slow
publishing
> problem though.)
>
> I'm planning on submitting my notes on how to use
bric_soap to the
> wiki soon - since it's taken me so long to figure out
how to use it
> based on other people's posts... and yet it is so
essential to
> managing a big site in bricolage.
>
> Dawn
>
>
>
>

ok and then I remembered I used --chunks 1 to get it to
report on its
progress one at a time.

feeling like a dog chasing tail today.

On 8/16/07, John Durkin <john.durkingmail.com> wrote:
>
> Hi Dawn,
>
> Did you ever figure out the particular reason your
stories were taking so
> long in soap and worked so quickly in the interface?
>
> I'm working on a development copy of the CMS for
etonline.com, and I'm
> having trouble publishing media items with soap.  I can
find these same
> items in the interface and they publish within seconds.
 There is nothing
> special about them, and they don't really even have
"templates" - not custom
> templates written by me.  They're just media.
>
> timeout=0 isn't working  for me... it just hangs - for
ages.  i tried
> timeout 300 and nothing there.  am waiting still while
a timeout=0 publish
> attempt sits there... doing nothing.
>
> !!!!!!!!!!!!!!!!!! HOLD ON, IT JUST WORKED
!!!!!!!!!!!!!!!!!!!!
>
> OK - so here's something for the wiki (maybe it's
already there someplace
> and I didn't see it)....
>
> I'm using bric 1.10.2 and just as I was waiting and
typing it went from
> hanging like this:
> rootadmin:~/bin# ./publish.sh
> media list_ids: Authenticating to Bricolage...
> workflow publish: Authenticating to Bricolage...
> media list_ids: Login success.
> media list_ids: Switched to media module.
> media list_ids: Calling Bric::SOAP::Media->list_ids
with search:
> (cover_date_start => 2007-08-07 00:00:00,
element_type_id => 4,
> cover_date_end => 2007-08-09 00:00:00)
> workflow publish: Login success.
> workflow publish: Switched to workflow module.
> workflow publish: Reading ids from STDIN...
> workflow publish: Calling Workflow->publish with
ids: media_id => 27306,
> media_id => 27311, media_id => 27316, media_id
=> 27320, media_id => 27322,
> media_id => 27324, media_id => 27328, media_id
=> 27335, media_id => 27337,
> media_id => 27339, media_id => 27341, media_id
=> 27346, media_id => 27351,
> media_id => 27354, media_id => 27359, media_id
=> 27362, media_id => 27365,
> media_id => 27370, media_id => 27375, media_id
=> 27381, media_id => 27386,
> media_id => 27388, media_id => 27390, media_id
=> 27393, media_id => 27395,
> media_id => 27400, media_id => 27407, media_id
=> 27412, media_id => 27416,
> media_id => 27418, media_id => 27426, media_id
=> 27428, media_id => 27430,
> media_id => 27433, media_id => 27438, media_id
=> 27441, media_id => 27443,
> media_id => 27445, media_id => 27448, media_id
=> 27453, media_id => 27455,
> media_id => 27457, media_id => 27459, media_id
=> 27462, media_id => 27467,
> media_id => 27469, media_id => 27474, media_id
=> 27481, media_id => 27486
>
> ..... (*me impatiently waiting here*)...
>
> then suddenly:
>
> media_27306
> media_27311
> media_27316
> media_27320
> media_27322
> media_27324
> media_27328
> media_27335
> media_27337
> media_27339
> media_27341
> media_27346
> media_27351
> media_27354
> media_27359
> media_27362
> media_27365
> media_27370
> media_27375
> media_27381
> media_27386
> media_27388
> media_27390
> media_27393
> media_27395
> media_27400
> media_27407
> media_27412
> media_27416
> media_27418
> media_27426
> media_27428
> media_27430
> media_27433
> media_27438
> media_27441
> media_27443
> media_27445
> media_27448
> media_27453
> media_27455
> media_27457
> media_27459
> media_27462
> media_27467
> media_27469
> media_27474
> media_27481
> media_27486
>
>
> they all spit out at once.  So - bric_soap - in current
versions - with
> media at least... schedules the jobs all at once and
then waits silently for
> them ALL to publish before letting you know it was
doing them all.  And if
> it doesn't get them all done, it fails with a read
timeout error with
> nothing written to the logs...  So if you have very
many items to publish,
> timeout=0 *does work* and is practically required.
>
> jd
>
>
> On 5/4/06, Dawn Buie <dawnsqueegeemedia.com>
wrote:
> >
> >
> > On 4-May-06, at 12:40 AM, Scott Lanning wrote:
> >
> > > On Wed, 3 May 2006, Dawn wrote:
> > >> I've been in republish purgatory trying
to figure out why
> > >> bric_soap is no long able - so it seems -
to republish large
> > >> numbers of stories.  I can republish from
my Web UI fine, but I
> > >> need to republish all the element=story
in the whole site.
> > >
> > > You use bric_queued?
> >
> > No I'm using bric_dist_mon and Bricolage 1.8.8
(hopefully soon
> > upgrading to 10.1.1 or .2???)
> >
> > > The bric_soap in your version of Bricolage
(1.8.5 IIRC)
> > > doesn't "schedule" publish jobs
like the queued does, it "burns" them
> > > immediately (fixed 1.8.9). This could explain
the difference between
> > > publishing through the UI and through
bric_soap.
> >
> > Hmmm interesting.
> >
> > >   Depending on elements, data, and the
"quality" of your templates,
> >
> > yes I could streamline my templates a little more
I think.
> >
> > > publishing can take a long time indeed... I
used to put bulk publishes
> > > in a for loop (you don't need to after 1.8.9
since publish jobs are
> > > only
> > > scheduled), like
> > >
> > > for i in `bric_soap story
--element_key_name=story list_ids`;
> > > do
> > >     bric_soap workflow --timeout 222 publish
$i;
> > > done
> > >
> > > with extra crap around it like `date` and
echoing the ID in case
> > > there's an error.
> >
> > thank you so much for this info - hopefully soon
I'll be able to
> > upgrade and be done with this problem.
> >
> >
> > BTW David's suggestion of leaving timeout 0 seems
to have done the
> > trick for me so far at least with republishing the
most recent
> > stories in small batches.
> >
> >
> > > Also --use-cookie-file to avoid
reauthenticating
> > > each time.
> >
> > I hacked my bric_soap utility to use my username
and server by
> > default. is that bad? (I did that after I had said
slow publishing
> > problem though.)
> >
> > I'm planning on submitting my notes on how to use
bric_soap to the
> > wiki soon - since it's taken me so long to figure
out how to use it
> > based on other people's posts... and yet it is so
essential to
> > managing a big site in bricolage.
> >
> > Dawn
> >
> >
> >
> >
>