EM random questions

A few random questions for a newbie like me:

Is there a way to start a server on 'the next open port' instead of specifying one?

Does send_file send the entire file at once? i.e. is sending a 1G file unadvised?

Does eventmachine buffer all input to sockets, until Ruby gets the chance to process it, eventually? I.e. say it takes one incoming chunk of data 5s to process--will incoming data on other ports just fill network queues, or be accepted and buffered, during that time? ; That might be nice...

Do servers have a large backlog? Does that matter?

Is there a way to flush a socket?

More questions:
I would like to submit patches to help the documentation, add useful functions/features, etc.  How could I best contribute?  Where should I also post suggestions for new features I don't write?

And finally--why is eventmachine so cool?

Anyway that's it for now.  Happy to be here

Is there a way to start a server on 'the next open port' instead of specifying one?

No. What does "the next open port" mean in this context?

Do servers have a large backlog? Does that matter?

I assume you mean an accept backlog. It's 50 or 100 connections iirc, not sure which. Higher than the default on most platforms.

Is there a way to flush a socket?

Everything in EM is nonblocking. When you call #send_data, it returns immediately, and the data will be sent when the system gets to it. Why do you need an explicit flush? (Which usually means nothing more than copying data to the kernel buffers.) If necessary, that can be simulated now, but if you really needed it, we should add some syntactic sugar to make it easier.

If I understand correctly, either the TCP or IP layer caches data (in an attempt to create 'full' IP packets, instead of partial ones).   After a certain timeout it goes ahead and sends the partial packet.  Flushing it forces it to send any (currently) partial packets immediately.  What might be nice is if EM would just always flush after a send_data so that developers never had to worry about it, ever   I would totally dig that. ; Or had a send_data_and_flush_immediately command or what not.  Though I do understand your confusion, as it would seem that this is useless to the user, per se, as EM (currently) doesn't call any functions when data writing is done (see below for a sugg (suggestion) on that).
 

Post feature requests to this list. It's a lot more actively monitored than the lists at rubyforge. Docs: We need FAQs and use cases.

I will start a FAQ on the wiki, and add these questions to it   Is that  good place for it?  I think I will also make a list of all the 'user definable' functions and the order each is called (and when), and put it there.

Also I have edited the default 0.9.0 files slightly to make them more verbose and comprehensive.  Should I just submit patches?
 

Glad to have you.

On 10/15/07, Roger Pack < rogerpack2005gmail.com">rogerpack2005gmail.com> wrote:

Thank you for the quick response! Answer and questions below:

Is there a way to start a server on 'the next open port' instead of specifying one?

No. What does "the next open port" mean in this context?

The equivalent of what establishing a client does--just uses any random port. ; Usefulness being 'I don't want to choose a bad port!' [which, BTW I'm not even sure how to handle in EM--I should figure it out and write it up, as you suggested.]

Do servers have a large backlog? Does that matter?

I assume you mean an accept backlog. It's 50 or 100 connections iirc, not sure which. Higher than the default on most platforms.

I believe mongrel itself has the backlog at 1024. Might be worth considering.

Is there a way to flush a socket?

Everything in EM is nonblocking. When you call #send_data, it returns immediately, and the data will be sent when the system gets to it. Why do you need an explicit flush? (Which usually means nothing more than copying data to the kernel buffers.) If necessary, that can be simulated now, but if you really needed it, we should add some syntactic sugar to make it easier.

If I understand correctly, either the TCP or IP layer caches data (in an attempt to create 'full&#39; IP packets, instead of partial ones).   After a certain timeout it goes ahead and sends the partial packet.&nbsp; Flushing it forces it to send any (currently) partial packets immediately.  What might be nice is if EM would just always flush after a send_data so that developers never had to worry about it, ever   I would totally dig that. ; Or had a send_data_and_flush_immediately command or what not.  Though I do understand your confusion, as it would seem that this is useless to the user, per se, as EM (currently) doesn't call any functions when data writing is done (see below for a sugg (suggestion) on that).
&nbsp;

Post feature requests to this list. It's a lot more actively monitored than the lists at rubyforge. Docs: We need FAQs and use cases.

I will start a FAQ on the wiki, and add these questions to it   Is that  good place for it?  I think I will also make a list of all the 'user definable&#39; functions and the order each is called (and when), and put it there.

Also I have edited the default 0.9.0 files slightly to make them more verbose and comprehensive.  Should I just submit patches?
&nbsp;

Glad to have you.

Glad to be here.

In terms of suggestions, I do have a few.  The first I have is to execute any "associated connection block"; (meaning

EventMachine::start_server(&quot; 127.0.0.1", port, EchoServer) { |conn|
# this block right here
}
&nbsp;BEFORE post_init (or is there a different callback that is called after the block, but only once at init that I don't know of?)

as it then allows for more old school style code like
module EchoServer
 attr_accessor :your_special_number
 def post_init
  ; print "my number is #{your_special_number}&quot;
 ; end
end
&nbsp; 
25.times do { |n|
EventMachine::start_server(&quot;127.0.0.1", port+n, EchoServer) { |conn|
&nbsp; &nbsp; conn.your_special_number = n
}
&nbsp;
Which allows post_init to be more of an initializer for the instance, as it can then do things with 'passed in parameters' (from the block).&nbsp; Sorry if that didn't make much sense, but anyway it was quite useful for me.
It would be a trivial code fix, I could do it and commit it.


Another suggestion might be reusable class instances--say you have something that for some reason has a high churn and so is generating tons of instances--it might save memory or what not to be able to just reuse those (of course, this might be a very bad idea, too, in the case of people redefining modules for each instance or what not).

A question: is there a function "what is _my_ port (host port, host name)?&quot;

suggestion: I created some get_peername_ip and get_peername_host helper functions.  Is there a reason they don't exist? Should I commit them?


suggestion: the addition of a post_write function might be nice, as well (or does it exist?) i.e. if I wanted to write the numbers from 1-10000000 onto the wire for some reason, I couldn';t do that all in one loop (or could I?--even if I could I might not want to as it might starve the others).&nbsp; If I wanted to, a post_write function would be nice (maybe pass it the total number of bytes ever sent or something).  Then I could send it piece-wise.

Another suggestion might be the creation of a function send_partial (the equivalent of send in python) -- maybe it can report how much it was able to send 'instantaneously&#39;.  I wouldn';t actually need it for anything, but maybe somebody somewhere would like such a feature


Well that's about it for now!
EM rox!
-Roger

_______________________________________________
Eventmachine-talk mailing list
rubyforge.org">Eventmachine-talkrubyforge.org
http://rubyforge.org/mailman/listinfo/eventmachine-talk

Also, I'd like to put up some information on the website, mainly a few quick tutorials on the simple stuff I've done so far with EM. As I'm a huge fan of Trac, what would it take to install and run it on the site? Then we have a wiki, code browser, and bug tracker all in one.

Jason

The equivalent of what establishing a client does--just uses any random port. ; Usefulness being 'I don't want to choose a bad port!' [which, BTW I'm not even sure how to handle in EM--I should figure it out and write it up, as you suggested.]

Do servers have a large backlog? Does that matter?

I assume you mean an accept backlog. It's 50 or 100 connections iirc, not sure which. Higher than the default on most platforms.

I believe mongrel itself has the backlog at 1024. Might be worth considering.

Is there a way to flush a socket?

Everything in EM is nonblocking. When you call #send_data, it returns immediately, and the data will be sent when the system gets to it. Why do you need an explicit flush? (Which usually means nothing more than copying data to the kernel buffers.) If necessary, that can be simulated now, but if you really needed it, we should add some syntactic sugar to make it easier.

If I understand correctly, either the TCP or IP layer caches data (in an attempt to create 'full&#39; IP packets, instead of partial ones).   After a certain timeout it goes ahead and sends the partial packet.&nbsp; Flushing it forces it to send any (currently) partial packets immediately.  What might be nice is if EM would just always flush after a send_data so that developers never had to worry about it, ever   I would totally dig that. ; Or had a send_data_and_flush_immediately command or what not.  Though I do understand your confusion, as it would seem that this is useless to the user, per se, as EM (currently) doesn't call any functions when data writing is done (see below for a sugg (suggestion) on that).

Post feature requests to this list. It's a lot more actively monitored than the lists at rubyforge. Docs: We need FAQs and use cases.

I will start a FAQ on the wiki, and add these questions to it   Is that  good place for it?  I think I will also make a list of all the 'user definable&#39; functions and the order each is called (and when), and put it there.

Also I have edited the default 0.9.0 files slightly to make them more verbose and comprehensive.  Should I just submit patches?

In terms of suggestions, I do have a few.  The first I have is to execute any "associated connection block"; (meaning

EventMachine::start_server(&quot; 127.0.0.1", port, EchoServer) { |conn|
# this block right here
}
&nbsp;BEFORE post_init (or is there a different callback that is called after the block, but only once at init that I don't know of?)

as it then allows for more old school style code like
module EchoServer
 attr_accessor :your_special_number
 def post_init
  ; print "my number is #{your_special_number}&quot;
 ; end
end
&nbsp; 
25.times do { |n|
EventMachine::start_server(&quot;127.0.0.1", port+n, EchoServer) { |conn|
&nbsp; &nbsp; conn.your_special_number = n
}
&nbsp;
Which allows post_init to be more of an initializer for the instance, as it can then do things with 'passed in parameters' (from the block).&nbsp; Sorry if that didn't make much sense, but anyway it was quite useful for me.
It would be a trivial code fix, I could do it and commit it.

Another suggestion might be reusable class instances--say you have something that for some reason has a high churn and so is generating tons of instances--it might save memory or what not to be able to just reuse those (of course, this might be a very bad idea, too, in the case of people redefining modules for each instance or what not).

A question: is there a function "what is _my_ port (host port, host name)?&quot;

suggestion: I created some get_peername_ip and get_peername_host helper functions.  Is there a reason they don't exist? Should I commit them?

suggestion: the addition of a post_write function might be nice, as well (or does it exist?) i.e. if I wanted to write the numbers from 1-10000000 onto the wire for some reason, I couldn';t do that all in one loop (or could I?--even if I could I might not want to as it might starve the others).&nbsp; If I wanted to, a post_write function would be nice (maybe pass it the total number of bytes ever sent or something).  Then I could send it piece-wise.

Another suggestion might be the creation of a function send_partial (the equivalent of send in python) -- maybe it can report how much it was able to send 'instantaneously&#39;.  I wouldn';t actually need it for anything, but maybe somebody somewhere would like such a feature

I second the "who am I" queries, I've wanted such information recently and just had to do a very hacky counting scheme. A block that's run at Connection creation, as Roger suggested, seems like the right way to go.

And yes, some sort of helper for:

port, address = Socket.unpack_addr_in(EM.get_peername)

because that's just ugly. Maybe EM::get_connection_info?

Also, I'd like to put up some information on the website, mainly a few quick tutorials on the simple stuff I've done so far with EM. As I'm a huge fan of Trac, what would it take to install and run it on the site? Then we have a wiki, code browser, and bug tracker all in one.

The equivalent of what establishing a client does--just uses any random port. ; Usefulness being 'I don't want to choose a bad port!' [which, BTW I'm not even sure how to handle in EM--I should figure it out and write it up, as you suggested.]

Not really useful, because you have to then somehow tell your clients which ephemeral port you picked! In regard to "bad ports";: you can't pick an adress/port combination that is already in use, because EM will complain loudly. If you're concerned about not picking a port that is already "standard" for some usage or other, then look in /etc/services to see if the number you want is already in use.

Python does this for server acceptors, and it's not truly useful but more convenient than anything.&nbsp; Just a thought for a future version

Do servers have a large backlog? Does that matter?

I assume you mean an accept backlog. It's 50 or 100 connections iirc, not sure which. Higher than the default on most platforms.

I believe mongrel itself has the backlog at 1024. Might be worth considering.

Setting a large backlog actually consumes a lot of kernel resources. I don't see that it's worth doing unless socket acceptance proceeds slowly for some reason. (Remember, mongrel is multithreaded in Ruby, so it won't be as fast as EM.) I could be convinced otherwise.

Is there a way to flush a socket?

It sounds like you're talking about disabling the "slow-start"; or Nagle algorithm as it applies to TCP. Usually I always turn that off when I write a network app, so I checked in EM just to see. Lo and behold, it does *not* disable Nagling. That's worth a try, so thanks for pointing it out.

I had no idea it is called a nagle algorithm   Yeah turning it off would work or what not.  I know back in the day when I did C socket stuff we'd need to flush to ensure that what I sent was actually put on the wire. ; slow-start is impossible to turn off AFAIK
 

Post feature requests to this list. It's a lot more actively monitored than the lists at rubyforge. Docs: We need FAQs and use cases.

I will start a FAQ on the wiki, and add these questions to it   Is that  good place for it?  I think I will also make a list of all the 'user definable&#39; functions and the order each is called (and when), and put it there.

There9;s a web site at http://rubyeventmachine.org which has nothing but an under-construction page at the moment. That would be the ideal place to post new material. There's no TRAC instance up there yet, but I've wanted to do that for a while now. I'll post here when that's done. It would be great to have interested folks help getting this effort off the ground. I'd put up your FAQ on the Rubyforge Wiki, and expect that it will migrate to the actual EM site, hopefully soon. Just make sure you post announcements here as you add content.
 

Also I have edited the default 0.9.0 files slightly to make them more verbose and comprehensive.  Should I just submit patches?

Please do.
 

 

In terms of suggestions, I do have a few.  The first I have is to execute any "associated connection block"; (meaning

EventMachine::start_server(&quot; 127.0.0.1", port, EchoServer) { |conn|
# this block right here
}
&nbsp;BEFORE post_init (or is there a different callback that is called after the block, but only once at init that I don't know of?)

as it then allows for more old school style code like
module EchoServer
 attr_accessor :your_special_number
 def post_init
  ; print "my number is #{your_special_number}&quot;
 ; end
end
&nbsp; 
25.times do { |n|
EventMachine::start_server(&quot;127.0.0.1", port+n, EchoServer) { |conn|
&nbsp; &nbsp; conn.your_special_number = n
}
&nbsp;
Which allows post_init to be more of an initializer for the instance, as it can then do things with 'passed in parameters' (from the block).&nbsp; Sorry if that didn't make much sense, but anyway it was quite useful for me.
It would be a trivial code fix, I could do it and commit it.

There already is a specific and defined protocol for the associated block, and we shouldn9;t change it because there is probably a lot of code that depends on it. Take the following code:
class X < EM::Connection
  def initialize *args
&nbsp; &nbsp; super
&nbsp; &nbsp; STDERR.puts "initializer&quot;
  end
  def post_init
  ;  STDERR.puts "post init"
  end
end

EM.run {
  EM.connect ( some_host, some_port, X ) {|conn|
&nbsp; &nbsp; STDERR.puts "block"
&nbsp; }
}

When a connection is accepted, the subclassed constructor (initialize) is called FIRST. #post_init is called INSIDE of the base class constructor. (You MUST remember to call #super if you override #initialize.) Obviously you can execute your own code before and/or after the super call, but keep in mind the #post_init will execute and return before #super returns. Hope that was clear.

The block that you pass to #connect will be called AFTER both initialize and post_init have been called and returned.

Right so the main justification to not changing it is that it might break legacy code, n'est pas?  I think that what I was referring to is that we don't have direct access to the initialize call, but in retrospect I guess it shouldn9;t matter too much what order things go in--you just have to accomodate for it--for example if I want post_init to be called after the block, I can just name my own function called "post_block"; and call it at the end of my block
Thanks

A question: is there a function "what is _my_ port (my port, my hostname or ip)?"

suggestion: I created some get_peername_ip and get_peername_host helper functions.  Is there a reason they don't exist? Should I commit them?

I found that to be something of a pain because some connection types (like Unix-domain sockets and UDP sockets) don't implement these things the same way as TCP clients do. What's the best approach here? Provide helper functions for all the different connection types and just throw exceptions if you use one that doesn't make sense?

Another suggestion might be the creation of a function send_partial (the equivalent of send in Python) -- maybe it can report how much it was able to send 'instantaneously&#39;.  I wouldn';t actually need it for anything, but maybe somebody somewhere would like such a feature

I can see the point of this in a program that needs to explicitly manage the size of the outbound network buffers, but EM wraps all this up. I'm open to adding advanced functions that might expose something like this, but in general it's better not to rely on the fancy stuff unless really necessary.