Re: "packet" framing

On Sun, Oct 14, 2007 at 01:52:48PM -0700, Bill Kelly wrote:
> For what it's worth, I'm currently implementing
something similar,
> which I'll release open source (non-GPL.)
> 
> I'm not using DRb, because I need Ruby <--> C++
communication, not
> just Ruby <--> Ruby.
> 
> My protocol is focused on fast remote procedure call
(over TCP or
> UDP) between both Ruby and non-Ruby nodes.  However,
the data types
> understood by the protocol are deliberately
Ruby-oriented.

I had some thoughts along these lines a while ago, as I
needed a lightweight
IPC interface between Ruby and Perl. Actually the Perl part
of this has
evaporated now, so I haven't needed to develop this
further.

But for what it's worth, I have some suggestions if you're
looking to
implement a better-than-DRb for heterogenous applications.

(1) Make the basic mandatory data type the
"string". Allow the two ends to
negotiate additional encodings, such as binary integers,
IEEE floats,
arrays, Ruby Marshal dumps, Perl Storable, PHP serialize
etc; but if both
ends don't support anything else, fallback to string
representation.

A bitmap representing known types would be a simple way to
do this. Just AND
the sender and receiver capability bitmaps to get the
acceptable common
ones. Then for any particular object the sender can choose
their preferred
representation from among the common ones.

[Keeping the list of mandatory types small would make the
protocol very
useful in embedded devices]

(2) Like DRb, keep the low-level distinction between
"success" and "fail"
results. This means you can have a method which *returns* an
exception
object be distinct from a method which *raises* an exception
during its
execution. This might be done by having "execution
failure" be a basic type
of its own.

(3) Make "proxy object" be a basic type too (a la
DRbUndumped (*)). This
allows the receiver to invoke methods on objects held at the
caller. Making
this a basic type means that a Perl receiver could invoke a
Ruby proxy
object without having to understand how to unmarshal a Ruby
DRbObject.

(4) Include support for method calls to be sent in *both*
directions down
the same socket; that is, have separate 'request' and
'response' message
types which can be distinguished on the wire, and allow
"servers" to send
new requests back to "clients". This means that
callbacks can be made back
down the same socket as the incoming request came on, which
in turn means
that it will work properly through NAT firewalls - something
which DRb is
miserable at.

[This does rely on the sender keeping the connection nailed
up, however.
Perhaps each proxy object should also contain an
"alternative callback
address" to be used in the event that the original
connection has been
lost]

(5) Tag each request and response with an ID, so that you
can send multiple
overlapping requests and responses down the same socket.
This avoids what
DRb does, which is opening multiple transport-layer
connections to the same
target if concurrent requests to the same target are made
from separate
threads (see DRbConn)

A simple embedded client or server can always implement only
simple
request-response ordering if it wishes.

(5a) Dividing large requests into chunks of, say, 64KB might
make for better
interleaving in this case.

(6) Include support for running over Unix domain sockets
(surprisingly
awkward with DRb)

(7) Include support for running over stdin/stdout, e.g. one
process which
talks to another process that it started using popen().

Possibly also useful:

(8) Support for running over TLS (possibly upgrading the
connection after
the initial handshake)

(9) Simple [possibly mutual] authentication at start of the
connection

(10) Support for running over HTTP (and FastCGI), although
that would
enforce calls to be unidirectional.

(11) Like DRb, you'll probably need some concept of a
"head object" or
"default object", to which you can send requests
if you don't know the
object ID of any other object on the peer. For simple
servers you only want
to expose one object anyway. For more complex servers the
application writer
can provide a directory object; I don't think you need to
worry about this
within the protocol itself.

Regards,

Brian.

(*) If this doesn't mean anything to you, google for
DRbTutorial.
Unfortunately the RubyGarden wiki appears to be permanently
down these days,
but you may be able to find a cached copy.
_______________________________________________
Eventmachine-talk mailing list
Eventmachine-talkrubyforge.org
http://rubyforge.org/mailman/listinfo/eventmachine-talk

Re: "packet" framing

Thread: Re: "packet" framing