ZLib streams

Hi!

I'm sorry to disturb you further, but I found something
weird
while trying what we discussed:

   Eval [
      PackageLoader fileInPackage: #TCP.
      PackageLoader fileInPackage: #ZLib.
   ]
   Eval [
      |zr zw sock|
      sock := TCP.Socket remote: 'localhost' port: 12324.

      zw := RawDeflateStream on: sock.
      zr := RawInflateStream on: sock.

      zw nextPutAll: 'TESTABC'; syncFlush.

      [sock atEnd not] whileTrue: [
         | hunk |
         hunk := zr nextHunk.
         'hunk!' displayNl.
         hunk inspect.
      ].
   ]

I tested this with socat like this:

   socat TCP4-LISTEN:12324,reuseaddr PIPE

The problem is that this line blocks/hangs:

   zr := RawInflateStream on: sock.

It blocks because RawInflateStream>>#on: calls
ZlibReadStream>>#fillBuffer
a few method calls down the stack. And fillBuffer actually
does
call nextHunk to wait for data to arrive on the line.

If I rearrange the lines btw.:

      zw nextPutAll: 'TESTABC'; syncFlush.
      zr := RawInflateStream on: sock.

Then everything seems to be fine (because there is data for
nextHunk).

Would it be possible to prevent the call to fillBuffer to
not have
a blocking RawInflateStream creation and only have it block
when _I_
call RawInflateStream>>#nextHunk?


Thanks!

Robin


_______________________________________________
help-smalltalk mailing list
help-smalltalkgnu.org

http://lists.gnu.org/mailman/listinfo/help-smalltalk

> Would it be possible to prevent the call to fillBuffer
to not have
> a blocking RawInflateStream creation and only have it
block when _I_
> call RawInflateStream>>#nextHunk?

Seems easy...


--- orig/packages/zlib/ZLibReadStream.st
+++ mod/packages/zlib/ZLibReadStream.st
 -145,9
+145,9  position
  !ZlibReadStream methodsFor: 'private'!

  resetBuffer
+    ptr := 0.
      delta := 0.
-    endPtr := 0.
-    self fillBuffer!
+    endPtr := 0!

  initialize: aStream
      super initialize: aStream.


Paolo


_______________________________________________
help-smalltalk mailing list
help-smalltalkgnu.org

http://lists.gnu.org/mailman/listinfo/help-smalltalk

On Thu, Aug 23, 2007 at 09:55:30AM +0200, Paolo Bonzini
wrote:
> 
> >Would it be possible to prevent the call to
fillBuffer to not have
> >a blocking RawInflateStream creation and only have
it block when _I_
> >call RawInflateStream>>#nextHunk?
> 
> Seems easy...
> 
> 
> --- orig/packages/zlib/ZLibReadStream.st
> +++ mod/packages/zlib/ZLibReadStream.st
>  -145,9 +145,9  position
>  !ZlibReadStream methodsFor: 'private'!
> 
>  resetBuffer
> +    ptr := 0.
>      delta := 0.
> -    endPtr := 0.
> -    self fillBuffer!
> +    endPtr := 0!
> 
>  initialize: aStream
>      super initialize: aStream.

Ok, thats easy. But maybe a bit too easy 

That worked for my small example. But I've tried to
implement it in my program
now and ran into some other problem:

  [socket atEnd not] whileTrue: [self handleData:
inputStream nextHunk]

(Please note that I tried to use '[inputStream atEnd not]'
first, but that
seemed to run into the same problem I'm going to describe
below as a
call to ZlibReadStream>>#atEnd results in a call to
ZlibReadStream>>#fillBuffer)

nextHunk does not seem to return properly. I guess the
problem is in
 ZlibReadStream>>#fillBuffer:
       "Fill the output buffer, supplying data to zlib
until it can actually
        produce something."
       | flush |
       delta := delta + endPtr.
       ptr := 0.
       [
           inBytes isNil ifTrue: [
               inBytes := self stream atEnd
                   ifTrue: [ #[] ]
                   ifFalse: [ self stream nextHunk ] ].

           flush := self stream atEnd ifTrue: [ 4 ] ifFalse:
[ 0 ].
           endPtr := self processInput: flush size: inBytes
size.
           endPtr = 0 ] whileTrue.

       "End of data, or zlib error encountered."
       endPtr = -1 ifTrue: [ self checkError ]! !

The following line completes when it received the
syncFlushed data,
and an inspect revealed that there really was data.

           ifFalse: [ self stream nextHunk ] ].

However, I put in some displayNls and observed that the
call:

           flush := self stream atEnd ifTrue: [ 4 ] ifFalse:
[ 0 ].

Seemed to block. That wouldn't be a problem if there would
be further
data waiting now, but it isn't (due to the fact that the
first packet was a
'HELO' in my protocol and my software waits now for further
input which
wont come).

You may also note that the call to
ZlibReadStream>>#processInput:size: which
will result in a call to
RawInflateStream>>#processInput:size: will receive
the
wrong flag (0 Z_NO_FLUSH or 4 Z_FINISH, and not 2
Z_SYNC_FLUSH) when atEnd
finally returns.

I'm not familiar enough and currently don't have enough time
to try to fix it
or propose a solution to this.


Robin


_______________________________________________
help-smalltalk mailing list
help-smalltalkgnu.org

http://lists.gnu.org/mailman/listinfo/help-smalltalk

> That worked for my small example. But I've tried to
implement it in my program
> now and ran into some other problem:
> 
>   [socket atEnd not] whileTrue: [self handleData:
inputStream nextHunk]
> 
> Seemed to block.

You cannot know if the stream is atEnd unless a) it closes,
b) you have 
one more byte.  So yes, #atEnd blocks (by design).

I would do something like

    [
      socket atEnd ifTrue: [ ^self ]
      self handleData: inputStream nextHunk
    ] whileFalse.

with #handleData: returning true when it finishes reading a
packet.

> You may also note that the call to
ZlibReadStream>>#processInput:size: which
> will result in a call to
RawInflateStream>>#processInput:size: will receive
the
> wrong flag (0 Z_NO_FLUSH or 4 Z_FINISH, and not 2
Z_SYNC_FLUSH) when atEnd
> finally returns.

As far as I could see from the zlib source code,
Z_SYNC_FLUSH only 
matters for deflating.  On inflation, it only looks for
Z_NO_FLUSH, 
Z_FINISH, Z_BLOCK.

Paolo


_______________________________________________
help-smalltalk mailing list
help-smalltalkgnu.org

http://lists.gnu.org/mailman/listinfo/help-smalltalk

On Thu, Aug 23, 2007 at 03:14:20PM +0200, Paolo Bonzini
wrote:
> 
> >That worked for my small example. But I've tried to
implement it in my 
> >program
> >now and ran into some other problem:
> >
> >  [socket atEnd not] whileTrue: [self handleData:
inputStream nextHunk]
> >
> >Seemed to block.
> 
> You cannot know if the stream is atEnd unless a) it
closes, b) you have 
> one more byte.  So yes, #atEnd blocks (by design).

Thanks okay. But what I mean was that nextHunk blocked even
after it
received a SYNC_FLUSHed packet from the other end. See the
rest of the
mail I wrote there.

It's also fine in that loop when atEnd blocks, if it comes
back after data
arrived or the stream closed or whatever, but inputStream
(being (a
RawInflateStream (which I maybe missed to say, sorry)) needs
to give me the
packet it got (and it actually really got a _complete_
packet, which I found
out while debugging). The problem was that atEnd (on the
socket) which is
called in fillBuffer blocks _after_ the compressed data was
read from the socket.
Which resulted in nextHunk not to give me the finished
packet it got.

In fact nextHunk returned the packet when I shut down one
end of the TCP connection
(which resulted in the atEnd in fillBuffer to return).

> I would do something like
> 
>    [
>      socket atEnd ifTrue: [ ^self ]
>      self handleData: inputStream nextHunk
>    ] whileFalse.
> 
> with #handleData: returning true when it finishes
reading a packet.

handleData accumulates the data it got and maybe finished
parsing none,
one, two, ..., N protocol messages. And I don't see why I
would like to
end my read-loop when I got packets from the socket.

Maybe my approach of accumulating the bytes from the network
in a buffer
and reparse it when more bytes arrive to detect completed
packets is not
practical in smalltalk.

My logic was: "read all packets from the socket until
we get a disconnect":

    [inputStream "(or socket)" atEnd not]
        whileTrue: [self handleData: inputStream nextHunk]

> >You may also note that the call to
ZlibReadStream>>#processInput:size: 
> >which
> >will result in a call to
RawInflateStream>>#processInput:size: will 
> >receive the
> >wrong flag (0 Z_NO_FLUSH or 4 Z_FINISH, and not 2
Z_SYNC_FLUSH) when atEnd
> >finally returns.
> 
> As far as I could see from the zlib source code,
Z_SYNC_FLUSH only 
> matters for deflating.  On inflation, it only looks for
Z_NO_FLUSH, 
> Z_FINISH, Z_BLOCK.

I don't know about the zlib source code, but I read the
manual:
   http://www.zl
ib.net/manual.html#inflate

   If the parameter flush is set to Z_SYNC_FLUSH, inflate
flushes as much output
   as possible to the output buffer.
And:
   However if all decompression is to be performed in a
single step (a single
   call of inflate), the parameter flush should be set to
Z_FINISH. In this case
   all pending input is processed and all pending output is
flushed ;

The manual might be wrong, I don't know.


Robin


_______________________________________________
help-smalltalk mailing list
help-smalltalkgnu.org

http://lists.gnu.org/mailman/listinfo/help-smalltalk

> Thanks okay. But what I mean was that nextHunk blocked
even after it
> received a SYNC_FLUSHed packet from the other end. See
the rest of the
> mail I wrote there.

Ah, I see now.  Could you try this further patch?

Paolo

_______________________________________________
help-smalltalk mailing list
help-smalltalkgnu.org

http://lists.gnu.org/mailman/listinfo/help-smalltalk

On Thu, Aug 23, 2007 at 04:45:23PM +0200, Paolo Bonzini
wrote:
> 
> >Thanks okay. But what I mean was that nextHunk
blocked even after it
> >received a SYNC_FLUSHed packet from the other end.
See the rest of the
> >mail I wrote there.
> 
> Ah, I see now.  Could you try this further patch?

With pleasure!

Yes, it seems to work just fine now (together with the patch
for
resetBuffer). I'm however still a bit concerned about the
value
for 'flush', but if you think 4 and 0 are fine there I'll
trust
you 

Thanks!


Robin


_______________________________________________
help-smalltalk mailing list
help-smalltalkgnu.org

http://lists.gnu.org/mailman/listinfo/help-smalltalk

> Is the XMLParser (in SAX mode) able to parse directly
from a TCP.Socket
> in such kind of way that completed input leads to SAX
events? I've seen
> a XMLParser>>#on: method... would this work:
> 
>    socket := ...
>    XMLParser on: socket.
>    ...
> ?
> 
> (That kind of feature would be required for parsing
XMPP (the protocol
> that was formerly called 'Jabber')), but I'm not sure
I'm ever going to
> implement a XMPP lib in smalltalk (I've already done
that in Perl, and
> it was _not_ a pleasure )

I think so, yes.  You might have problems with lookahead
there, too, but 
apart from that it would be okay.

Paolo


_______________________________________________
help-smalltalk mailing list
help-smalltalkgnu.org

http://lists.gnu.org/mailman/listinfo/help-smalltalk