Queue as a supplement to cron

I sometimes find myself building a queue in a hook_cron()
implementation. I decided to make a separate module for
this,
job_queue, http://drupal.or
g/project/job_queue. It provides an API
which takes a function name and array of arguments for later
execution
on cron. The only UI is a list of queued function calls.

Is this functionality something that people would like to
see in core?
Search module has a 'queue' of unindexed content, would it
benefit
from having an abstracted queue?

-- 
Neil Drumm
http://delocalizedham.com

On 3/12/07, Neil Drumm <drummdelocalizedham.com>
wrote:
> I sometimes find myself building a queue in a
hook_cron()
> implementation. I decided to make a separate module for
this,
> job_queue, http://drupal.or
g/project/job_queue. It provides an API
> which takes a function name and array of arguments for
later execution
> on cron. The only UI is a list of queued function
calls.
>
> Is this functionality something that people would like
to see in core?
> Search module has a 'queue' of unindexed content, would
it benefit
> from having an abstracted queue?

aggregator (and other, contrib-based aggregation) would
benefit from
this as well, I believe.


-- 
Boris Mann
Vancouver 778-896-2747
San Francisco 415-367-3595
Skype borismann
http://www.bryght.com

On 3/12/07, Boris Mann <borisbryght.com> wrote:
> On 3/12/07, Neil Drumm <drummdelocalizedham.com>
wrote:
> > I sometimes find myself building a queue in a
hook_cron()
> > implementation. I decided to make a separate
module for this,
> > job_queue, http://drupal.or
g/project/job_queue. It provides an API
> > which takes a function name and array of arguments
for later execution
> > on cron. The only UI is a list of queued function
calls.
> >
> > Is this functionality something that people would
like to see in core?
> > Search module has a 'queue' of unindexed content,
would it benefit
> > from having an abstracted queue?
>
> aggregator (and other, contrib-based aggregation) would
benefit from
> this as well, I believe.

Those happen regularly, so cron is best.

Ping module could use this since you want to queue those
HTTP requests
for later.

-- 
Neil Drumm
http://delocalizedham.com

On 3/12/07, Neil Drumm <drummdelocalizedham.com>
wrote:
> On 3/12/07, Boris Mann <borisbryght.com> wrote:

> > > Is this functionality something that people
would like to see in core?
> > > Search module has a 'queue' of unindexed
content, would it benefit
> > > from having an abstracted queue?
> >
> > aggregator (and other, contrib-based aggregation)
would benefit from
> > this as well, I believe.
>
> Those happen regularly, so cron is best.

Errr....except for mass numbers of aggregation where one
cron run
can't crank through everything. Queue might not be the right
solution,
but I wanted to point out that current cron architecture is
not up to
the task.

> Ping module could use this since you want to queue
those HTTP requests
> for later.



-- 
Boris Mann
Vancouver 778-896-2747
San Francisco 415-367-3595
Skype borismann
http://www.bryght.com

On 3/12/07, Boris Mann <borisbryght.com> wrote:
> On 3/12/07, Neil Drumm <drummdelocalizedham.com>
wrote:
> > On 3/12/07, Boris Mann <borisbryght.com> wrote:
>
> > > > Is this functionality something that
people would like to see in core?
> > > > Search module has a 'queue' of unindexed
content, would it benefit
> > > > from having an abstracted queue?
> > >
> > > aggregator (and other, contrib-based
aggregation) would benefit from
> > > this as well, I believe.
> >
> > Those happen regularly, so cron is best.
>
> Errr....except for mass numbers of aggregation where
one cron run
> can't crank through everything. Queue might not be the
right solution,
> but I wanted to point out that current cron
architecture is not up to
> the task.

Adding individual items to the queue on cron might be a good
pattern.
That is what I'll be doing to import the pile o' XML that
changes
every day, which I've got to deal with for a project.

I coded this to run queue items on cron until more than half
of the
PHP allowed execution time has been used.

-- 
Neil Drumm
http://delocalizedham.com

On 3/12/07, Boris Mann <borisbryght.com> wrote:
> On 3/12/07, Neil Drumm <drummdelocalizedham.com>
wrote:
> > On 3/12/07, Boris Mann <borisbryght.com> wrote:
>
> > > > Is this functionality something that
people would like to see in core?
> > > > Search module has a 'queue' of unindexed
content, would it benefit
> > > > from having an abstracted queue?
> > >
> > > aggregator (and other, contrib-based
aggregation) would benefit from
> > > this as well, I believe.
> >
> > Those happen regularly, so cron is best.
>
> Errr....except for mass numbers of aggregation where
one cron run
> can't crank through everything. Queue might not be the
right solution,
> but I wanted to point out that current cron
architecture is not up to
> the task.

Pathauto's bulk update has a similar need to "alias
{configurable
number of nodes} in a queued fasion."  I'm not sure
that it would be
best in cron, necessarily, but it is similar to the problem
Boris
described.

If the job_queue api stored the function and the arguments
to pass to
the function that would be nice.  A quick look at
htt
p://cvs.drupal.org/viewcvs/drupal/contributions/modules/job_
queue/job_queue.module?rev=1.1&view=markup
makes it seem like it does provide that.  Yay!

Regards,
Greg

On Monday 12 March 2007 20:45, Neil Drumm wrote:
> I sometimes find myself building a queue in a
hook_cron()
> implementation. I decided to make a separate module for
this,
> job_queue, http://drupal.or
g/project/job_queue. It provides an API
> which takes a function name and array of arguments for
later execution
> on cron. The only UI is a list of queued function
calls.
> 
> Is this functionality something that people would like
to see in core?
> Search module has a 'queue' of unindexed content, would
it benefit
> from having an abstracted queue?

If you decide to do this as a contrib module, you may want
to consider
making it part of my "cronplus" module.

Cronplus doesn't deal with the queueing, but rather creates
a set of API
hooks that are called at specific time intervals (daily,
weekly, etc.)
rather than every time cron.php executes. Of course the
timing isn't
exact because cronplus itself only triggers on cron runs,
but it gives
a reasonable approximation for batch jobs.

It would seem to me that your queue manager might be an
interesting
extension to cronplus. I won't be offended if you want to go
your own
way, but if you want to consider merging into cronplus I'd
certainly
be willing to let that happen.

Kind regards,

Scott (Syscrusher)

-- 
------------------------------------------------------------
-------------------
Syscrusher (Scott Courtney)          Drupal page:   http://drupal.org/user/91
84
syscrusher at 4th dot com            Home page:     http://4th.com/   

Neil Drumm wrote:
> I sometimes find myself building a queue in a
hook_cron()
> implementation. I decided to make a separate module for
this,
> job_queue, http://drupal.or
g/project/job_queue. It provides an API
> which takes a function name and array of arguments for
later execution
> on cron. The only UI is a list of queued function
calls.
> 
> Is this functionality something that people would like
to see in core?
> Search module has a 'queue' of unindexed content, would
it benefit
> from having an abstracted queue?

actions module could benefit from this GREATLY, since
dropping actions into job 
queue == easy scheduling.

Neil Drumm wrote:
> I sometimes find myself building a queue in a
hook_cron()
> implementation. I decided to make a separate module for
this,
> job_queue, http://drupal.or
g/project/job_queue. It provides an API
> which takes a function name and array of arguments for
later execution
> on cron. The only UI is a list of queued function
calls.
> 
> Is this functionality something that people would like
to see in core?
> Search module has a 'queue' of unindexed content, would
it benefit
> from having an abstracted queue?

Not only cron. A HTTP queue would be better in general  As we were

talking about recently. Update.php has a frontend facing job
queue, 
which is (apart from storing the queue in the session not
the database) 
a very much similar situation. Other frontend facing queues
would 
greatly benefit from this too. Like a long lasting data
import in an 
install profile (we are talking about PO file imports for
example).

Gabor