ZLib Z_SYNC_FLUSH/Z_FULL_FLUSH

Hi!

I've seen that there is a ZLib implementation, but it
doesn't seem to
support giving deflate() Z_FULL_FLUSH or Z_SYNC_FLUSH which
seems to be
neccessary in most 'interactive' network applications.

I don't know how that would be best implemented as I'm not
familiar
enough with streams yet. But the use case would be: write
some data to
the deflate stream and then flush.
And some other process would wait for the nextHunk and would
write that
out to eg. the network.


Something like this:

   defl := DeflateStreama compressingTo: socket.
   defl nextPutAll: '[message]'.
   defl flush.

For flush parameters something like that would be nice:

   defl flush: Z_FULL_FLUSH.

(don't know how suchs constants would be handled in
smalltalk 

I hope the flush would also cause a flush on the socket
stream, or
at least the the compressed data until the flush would be
transmitted
to the socket. So that I can do "socket flush."
after "defl flush" (I
don't really know here how flushing is being done in
smalltalk 


Robin


_______________________________________________
help-smalltalk mailing list
help-smalltalkgnu.org

http://lists.gnu.org/mailman/listinfo/help-smalltalk

>    defl flush: Z_FULL_FLUSH.

Seems like a good idea.  One would have a #flush and a
#flushDictionary 
method for Z_SYNC_FLUSH and Z_FULL_FLUSH respectively, if I
understand 
correctly.

> I hope the flush would also cause a flush on the socket
stream, or
> at least the the compressed data until the flush would
be transmitted
> to the socket.

Yes, it's understandable that "defl flush" would
also flush the 
underlying stream.

Paolo


_______________________________________________
help-smalltalk mailing list
help-smalltalkgnu.org

http://lists.gnu.org/mailman/listinfo/help-smalltalk

Paolo Bonzini wrote:
>>    defl flush: Z_FULL_FLUSH.
> 
> Seems like a good idea.  One would have a #flush and a
#flushDictionary 
> method for Z_SYNC_FLUSH and Z_FULL_FLUSH respectively,
if I understand 
> correctly.

It turned out to be a little complicated because I was lazy
and I didn't 
implement the ZLibWriteStream, so to speak, directly. 
Instead I used an 
adaptor class that took the ReadStream decorator (the one
used for 
DeflateStream on: 'aaa') and transformed it into a
WriteStream decorator.

This was perfectly fine with the existing set of
functionality but now 
the problem, of course, is that the ReadStream decorator
cannot know the 
boundary at which flushing will occur.  The solution is to
write a real 
WriteStream decorator for zlib -- luckily, quite a lot of
common code 
can be factored into an abstract superclass used by both the
ReadStream 
decorator and WriteStream decorator.

The boring buffering code is already there FileStream.  It
should be 
copied over to the TCP buffering code as well, as it would
speed up 
#nextPutAll: on a socket consistently; and maybe the ZLib
bindings 
themselves could use the ReadBuffer and WriteBuffer classes
that sockets 
use.  Something for another day, though.

The C interface was almost ready; the only thing missing was
the ability 
to deflate/inflate an incomplete buffer.

Thanks for this request, it is indeed a useful one!

Paolo

_______________________________________________
help-smalltalk mailing list
help-smalltalkgnu.org

http://lists.gnu.org/mailman/listinfo/help-smalltalk

On Wed, Aug 22, 2007 at 10:54:11AM +0200, Paolo Bonzini
wrote:
> Paolo Bonzini wrote:
> >>   defl flush: Z_FULL_FLUSH.
> >
> >Seems like a good idea.  One would have a #flush
and a #flushDictionary 
> >method for Z_SYNC_FLUSH and Z_FULL_FLUSH
respectively, if I understand 
> >correctly.
[.snip.]
> Thanks for this request, it is indeed a useful one!

Yea, I'm currently writing some kind of network server where
I would
like to use Zlib to optimize the overhead a bit. I'm
currently trying
to figure out how streams are supposed to work at all and
wrote some
code some hours ago to init a compression handshake in the
protocol
I've implemented. Now all I need to figure out is how to
plug the
compression streams on that already existing TCP Stream 
(That blox doesn't work here slows down the browsing a bit

(see that bugreport I should've sent some days ago 


Thanks for the good work!


Robin


_______________________________________________
help-smalltalk mailing list
help-smalltalkgnu.org

http://lists.gnu.org/mailman/listinfo/help-smalltalk

> Yea, I'm currently writing some kind of network server
where I would
> like to use Zlib to optimize the overhead a bit. I'm
currently trying
> to figure out how streams are supposed to work at all
and wrote some
> code some hours ago to init a compression handshake in
the protocol
> I've implemented. Now all I need to figure out is how
to plug the
> compression streams on that already existing TCP Stream


The "testSyncFlush" test I introduced in
zlibtests.st can help.  The 
code would go something like this (just to give an idea):

initialize
     yourTCPSocket := blahblah.
     deflatedData := RawDeflateWriteStream on:
yourTCPSocket

nextPutPacket: uncompressedData compressedData:
compressedData
     yourTCPSocket nextPutAll: uncompressedData.
     "no need to flush yourTCPSocket, in the end
deflatedData
      just writes there too."
     deflatedData nextPutAll: compressedData.
     deflatedData syncFlush

nextPacket: uncompressedDataSize compressedData:
compressedDataSize
     | uncData cmpData |
     uncData := yourTCPSocket next: uncompressedDataSize.
     cmpData := yourTCPSocket next: compressedDataSize.
     cmpData := (RawInflateStream on: cmpData readStream)
contents.
     ^uncData, cmpData



Since the compressedDataSize most likely is somewhere in the

uncompressed data, you could do something like this
instead:

nextPacket: uncompressedDataSize compressedData:
compressedDataSizeBlock
     | uncData cmpData size |
     uncData := yourTCPSocket next: uncompressedDataSize.
     size := compressedDataSizeBlock value: uncData.
     cmpData := yourTCPSocket next: size.
     cmpData := (RawInflateStream on: cmpData readStream)
contents.
     ^uncData, cmpData

Paolo


_______________________________________________
help-smalltalk mailing list
help-smalltalkgnu.org

http://lists.gnu.org/mailman/listinfo/help-smalltalk

> That would be mostmy my usage: having the tcp stream
completly
> compressed and mostly replacing the tcpSocket by
inflate/deflate
> streams.
> 
> The actual read-loop in my program looks like this:
> 
>      [[inputStream atEnd not] whileTrue: [self
handleData: inputStream nextHunk]]
> 
> I would then after the compression-handshake replace
inputStream (which is a tcp socket
> before the handshake) with an inflate stream.

Yes, but don't expect nextHunk to retrieve a whole packet. 
It could 
return only part of it.  I would just do

   [inputStream atEnd not] whileTrue: [ self handleData:
inputStream ]

You can assume that the RawInflateStream will not request
more data from 
the socket when it sees a flush.  So, if #handleData: does
not try to 
read past the end of the packet, it should just work.

Paolo


_______________________________________________
help-smalltalk mailing list
help-smalltalkgnu.org

http://lists.gnu.org/mailman/listinfo/help-smalltalk