Re: Web Site Process Bus

Chris McDonough wrote:
> There are also non-webbish processes like postgres,
mysql, etc. that  
> need to be treated as "part of the
application".
> 
> I handle this currently by running all of the processes
related to a  
> specific project under a process controller (which
happens to be  
> implemented in Python, but that's besides the point,
see http:// 
> www.plope.com/software/supervisor2/).  The process
controller is  
> responsible for execing the child processes upon its
own startup.
> It is also responsible for restarting children if they
die,
> capturing their output (if any), and allowing
sufficiently
> privileged users to start and stop each one
independently.
> The only promise a subprocess must make to be managed
is that
> it must be possible to start the process "in the
foreground"
> (not under its down daemon manager).
> 
> If a "process bus" is implemented I suspect
it should be implemented  
> at this kind of level.

Ah, but there's the rub: we all have different ideas about
how to
*implement* IPC and control. Which is why the WSPBus I
outlined says
nothing about the mechanisms of message transport, RPC/IPC,
or process
or thread boundaries. Instead, it defines the messages
themselves, a set
of states for a given site, and a singleton message broker
and state
machine. That's it.

If we can get that one piece into place, I think it can be a
focal point
for interop between 1) application components, 2) HTTP
servers, and 3)
"process controllers" (great term, I think I'll
use it from now on). We
can achieve that without specifying how any process
controller is
implemented, I think. It's difficult to discuss and reason
about,
because CherryPy, Apache, Django, Zope, etcetera etcetera
all provide
all three. A common bus should make it easier to decouple,
say,
CherryPy's app+server from its process controller, allowing
more people
to try out supervisor2 more easily.

> "Actions" could be registered for a specific
subprocess types
> to send some input to a pipe file descriptor, send a
signal to
> the process, etc.

Yes, and a supervisor2-wspb package could provide a Bus
which does that,
then hand a reference to it to each child's components. The
"reference"
in this case would be a child-side proxy object, which knows
how to send
signals back to the parent process. But the children don't
need to know
those transport details--all they have to do is call
bus.subscribe,
bus.publish('log'), etc.

> It would also be possible to create some sort of
dependency
> map between processes in a configuration, that relate
the
> actions of one process to another (restart process A
if
> process B is restarted, send a signal S to process C
if
> signal T is sent to process D, etc).

Dependencies are a layer that can be built on top of the
basic bus.
Since it's the process controller that's calling bus.start,
stop and
restart, there's nothing about the WSPBus that stops
supervisor2 from
handling dependency graphs on its own. If process A has to
*know* that
process B has been restarted, that's a problem (which could
be addressed
via custom bus channels), but if only the process controller
has to
know, then there's no need to add that to the bus spec,
IMO.


Robert Brewer
System Architect
Amor Ministries
fumanchuamor.org
_______________________________________________
Web-SIG mailing list
Web-SIGpython.org
Web SIG: http://www.python.
org/sigs/web-sig
Unsubscribe: http://mail.python.org/mailman/options/web-sig/bo
nd%40yahoo.com

On Jun 26, 2007, at 2:39 PM, Robert Brewer wrote:

> Chris McDonough wrote:
>> There are also non-webbish processes like postgres,
mysql, etc. that
>> need to be treated as "part of the
application".
>>
>> I handle this currently by running all of the
processes related to a
>> specific project under a process controller (which
happens to be
>> implemented in Python, but that's besides the
point, see http://
>> www.plope.com/software/supervisor2/).  The process
controller is
>> responsible for execing the child processes upon
its own startup.
>> It is also responsible for restarting children if
they die,
>> capturing their output (if any), and allowing
sufficiently
>> privileged users to start and stop each one
independently.
>> The only promise a subprocess must make to be
managed is that
>> it must be possible to start the process "in
the foreground"
>> (not under its down daemon manager).
>>
>> If a "process bus" is implemented I
suspect it should be implemented
>> at this kind of level.
>
> Ah, but there's the rub: we all have different ideas
about how to
> *implement* IPC and control.

I'm confused by this in your earlier message, describing
example  
scenarios:

"""
If I'm primarily a Zope user instead, I might start my
website with
zdaemon. This would work exactly like the above, but the Bus
object
would be instantiated and started by the zdaemon package. If
I'm using
Graham's new mod_wsgi with Apache, I might expect it to
create and
control the Bus.
"""

I'm confused because zdaemon is a generic process
controller, it  
knows nothing in particular about the application running
under it  
except that it's a UNIX process.  It could start postgres
instead of  
Zope if you configured it to.  If zdaemon creates a Bus
object,  
nothing will be able to send messages to the bus except
zdaemon  
itself, and there can't be any useful listeners because it
doesn't  
share the same process space as its child.

I think I'm mostly confused by the name "process
bus" because it  
seems like the primary use case for something like this is
where all  
of the applications share the same process space and are all
written  
in Python.  Am I right?  If so, maybe a different name is in
order?   
"Application Bus"?  Or even "WSGI Bus",
if its presumed that all of  
the applications will be WSGI applications?

- C

_______________________________________________
Web-SIG mailing list
Web-SIGpython.org
Web SIG: http://www.python.
org/sigs/web-sig
Unsubscribe: http://mail.python.org/mailman/options/web-sig/bo
nd%40yahoo.com