Proposed XMPP Extension: Roster Versioning

XMPP Extensions Editor wrote:
> The XMPP Extensions Editor has received a proposal for
a new XEP.
> 
> Title: Roster Versioning
> 
> Abstract: This specification proposes a modification to
the XMPP
> roster management protocol to support versioning of
rosters for more
> efficient downloading of the rost er information.
> 
> URL: http://www.xmpp.org/extensions/inbox/roster-versionin
g.html

This is the first of several proposals related to mobile
optimization
(and just general optimization) as discussed at the recent
devcon. More
to follow...

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

On Tue Mar  4 21:54:31 2008, XMPP Extensions Editor wrote:
> The XMPP Extensions Editor has received a proposal for
a new XEP.
> 
> Title: Roster Versioning
> 
> Abstract: This specification proposes a modification to
the XMPP  
> roster management protocol to support versioning of
rosters for  
> more efficient downloading of the rost er information.
> 
> URL: http://www.xmpp.org/extensions/inbox/roster-versionin
g.html

"Note: This document is provided for discussion
purposes in order to  
clarify the usage of SASL ANONYMOUS in XMPP systems. It is
not meant  
to supersede the text in RFC 3920, RFC 4422, or RFC 4505.
However,  
the recommendations in this document may be folded into
rfc3920bis."

Good to know that's cleared up, anyway. 

"The value of the 'version' attribute MUST start at
zero (0) and MUST  
be incremented by one (1) for each change to the
roster."

I would personally prefer that this read:

"The value of the 'version' attribute is an integer
value which is  
increased with any change to the roster, whether or not the
client  
supports this extension. In particular, the 'version'
attribute  
contained in roster pushes (see Section 2.5) MUST be
unique."

(Insert RFC 2119 language to suit).

My reasoning is:

a) Servers might need to increase it by more than 1 for some
changes,  
if they're not performed atomically.

b) Servers may need to increase it for changes which are not
 
user-visible, such as the from-pending state.

c) We might need to change it in the future for other
purposes.

Section 2.2:

Include some text saying that clients can set version to 0
if they  
have no roster cached with a version - otherwise there's no
bootstrap.

Sections 2.4/2.5 contain no closing </query>, and
don't show groups,  
etc. Both need to specify that the complete roster item is
used.

Section 2.4 needs to include a method for servers to
indicate whether  
this is an update, or a complete replacement.

Finally, I'd include an assertion for roster sets - so that
a client  
can perform actions such as a "set if not changed
since". Dead easy  
for servers, I think.

Dave.
-- 
Dave Cridland - mailto:davecridland.net - xmpp:dwdjabber.org
  -
acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Hi,

On Mar 4, 2008, at 10:27 PM, Dave Cridland wrote:
> On Tue Mar  4 21:54:31 2008, XMPP Extensions Editor
wrote:
>> The XMPP Extensions Editor has received a proposal
for a new XEP.
>> Title: Roster Versioning
>> Abstract: This specification proposes a
modification to the XMPP  
>> roster management protocol to support versioning of
rosters for  
>> more efficient downloading of the rost er
information.
>> URL: http://www.xmpp.org/extensions/inbox/roster-versionin
g.html
>
[...]
> "The value of the 'version' attribute MUST start
at zero (0) and  
> MUST be incremented by one (1) for each change to the
roster."
>
> I would personally prefer that this read:
>
> "The value of the 'version' attribute is an
integer value which is  
> increased with any change to the roster, whether or not
the client  
> supports this extension. In particular, the 'version'
attribute  
> contained in roster pushes (see Section 2.5) MUST be
unique."
>
> (Insert RFC 2119 language to suit).
>
> My reasoning is:
>
> a) Servers might need to increase it by more than 1 for
some  
> changes, if they're not performed atomically.
>
> b) Servers may need to increase it for changes which
are not user- 
> visible, such as the from-pending state.
>
> c) We might need to change it in the future for other
purposes.

+1. I can't see any reason for the spec to require more than
a  
increasing version number.


> Section 2.2:
>
> Include some text saying that clients can set version
to 0 if they  
> have no roster cached with a version - otherwise
there's no bootstrap.

Or omit  the version maybe?


> Sections 2.4/2.5 contain no closing </query>, and
don't show  
> groups, etc. Both need to specify that the complete
roster item is  
> used.

+1 for groups.


> Section 2.4 needs to include a method for servers to
indicate  
> whether this is an update, or a complete replacement.

Why? A complete replacement is just a full set of updates.
The final  
XML would be equivalent. From the client perspective, he
would  
replace all the cached local entries.


> Finally, I'd include an assertion for roster sets - so
that a  
> client can perform actions such as a "set if not
changed since".  
> Dead easy for servers, I think.

I don't follow what this is. Can you elaborate with a use
case?

Best regards,
-- 
Pedro Melo
Blog: http://www.simplic
idade.org/notes/
XMPP ID: melosimplicidade.org
Use XMPP!

Dave Cridland wrote:
> On Tue Mar  4 21:54:31 2008, XMPP Extensions Editor
wrote:
>> The XMPP Extensions Editor has received a proposal
for a new XEP.
>>
>> Title: Roster Versioning
>>
>> Abstract: This specification proposes a
modification to the XMPP
>> roster management protocol to support versioning of
rosters for more
>> efficient downloading of the rost er information.
>>
>> URL: http://www.xmpp.org/extensions/inbox/roster-versionin
g.html
> 
> "Note: This document is provided for discussion
purposes in order to
> clarify the usage of SASL ANONYMOUS in XMPP systems. It
is not meant to
> supersede the text in RFC 3920, RFC 4422, or RFC 4505.
However, the
> recommendations in this document may be folded into
rfc3920bis."
> 
> Good to know that's cleared up, anyway. 

Yeah yeah yeah, copy and paste error, always working too
fast...

> "The value of the 'version' attribute MUST start
at zero (0) and MUST be
> incremented by one (1) for each change to the
roster."
> 
> I would personally prefer that this read:
> 
> "The value of the 'version' attribute is an
integer value which is
> increased with any change to the roster, whether or not
the client
> supports this extension. In particular, the 'version'
attribute
> contained in roster pushes (see Section 2.5) MUST be
unique."
> 
> (Insert RFC 2119 language to suit).
> 
> My reasoning is:
> 
> a) Servers might need to increase it by more than 1 for
some changes, if
> they're not performed atomically.
> 
> b) Servers may need to increase it for changes which
are not
> user-visible, such as the from-pending state.
> 
> c) We might need to change it in the future for other
purposes.

Sure, that seems reasonable.

> Section 2.2:
> 
> Include some text saying that clients can set version
to 0 if they have
> no roster cached with a version - otherwise there's no
bootstrap.

Ah, good point.

> Sections 2.4/2.5 contain no closing </query>, and
don't show groups,
> etc. Both need to specify that the complete roster item
is used.

Right, I left out the groups because I'm lazy. I'll add a
few in.

> Section 2.4 needs to include a method for servers to
indicate whether
> this is an update, or a complete replacement.

I defined a boolean 'diff' attribute for this purpose.

> Finally, I'd include an assertion for roster sets - so
that a client can
> perform actions such as a "set if not changed
since". Dead easy for
> servers, I think.

Is that really necessary for the intended purpose
(optimization of
roster gets)? Or is it just a cool feature that piggybacks
on the
optimization use case?

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

On Wed Mar  5 07:07:20 2008, Alexander Gnauck wrote:
> Richard Dobson schrieb:
>>> +1. I can't see any reason for the spec to
require more than a  
>>> increasing version number.
>> 
>> I would prefer if it were just an opaque string,
certainly as far  
>> as the client in concerned there is no need for it
to do anything  
>> other than store the most recent version identifier
it has  
>> received and then return that to the server when
required (i.e. at  
>> login), only the server needs to know what to do
with it and this  
>> should certainly only be RECOMMENDED or
SUGGESTED/MAYBE and not  
>> MUST. What would happen in cases were the version
number needs to  
>> be reset to 0 because someone has such a busy
roster that over  
>> time they exhast the maximum integer value?, or if
maybe in future  
>> server implementors want to compress it somehow
into hex or  
>> something similar, it would be far far better to
leave this  
>> flexible and upto the server implementor on how
they format the  
>> version identifier.
> 
> yes I think we should recommend a an increasing
integer. But should  
> allow any string. So if some server vendor prefers hash
codes or  
> GUIDs for the versioning then this is fine for me too.

The only reason this scares me is that strictly increasing
numeric  
sequences have proved useful in the IMAP world, because
clients can  
spot when things go wrong much more easily.

Plus, nobody can get it wrong.

There's no way that even a 32-bit unsigned integer is going
to  
overflow - if you did an update every second, it'd take 136
years -  
but if that still unnerves you (in case PSA turns into the
undead, or  
something), use a 64-bit unsigned integer.

Dave.
-- 
Dave Cridland - mailto:davecridland.net - xmpp:dwdjabber.org
  -
acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

On Tue Mar  4 23:48:53 2008, Pedro Melo wrote:
> On Mar 4, 2008, at 10:27 PM, Dave Cridland wrote:
>> "The value of the 'version' attribute is an
integer value which is  
>>  increased with any change to the roster, whether
or not the  
>> client  supports this extension. In particular, the
'version'  
>> attribute  contained in roster pushes (see Section
2.5) MUST be  
>> unique."
> 
> +1. I can't see any reason for the spec to require more
than a   
> increasing version number.
> 
> 
The key phrase "strictly increasing sequence
number" needs to be  
present somewhere, to keep mathematicians happy.
("Monotonically  
increasing" varies in its meaning in different parts of
the world,  
bizarrely, so there's a phrase to avoid).



>> Section 2.2:
>> 
>> Include some text saying that clients can set
version to 0 if they  
>>  have no roster cached with a version - otherwise
there's no  
>> bootstrap.
> 
> Or omit  the version maybe?
> 
> 
No, omitting the version means "I'd like the full
roster and I do not  
understand versions".

>> Section 2.4 needs to include a method for servers
to indicate   
>> whether this is an update, or a complete
replacement.
> 
> Why? A complete replacement is just a full set of
updates. The  
> final  XML would be equivalent. From the client
perspective, he  
> would  replace all the cached local entries.
> 
> 
No, it isn't. There's that "removed" value for
subscription. In the  
case where a full update is needed, it's quite likely that
one of the  
reasons is because the server can't remember all (or any)
deletions.

If a full update looks the same as a diff, then a client
cannot know  
to remove otherwise undead entries.



>> Finally, I'd include an assertion for roster sets -
so that a   
>> client can perform actions such as a "set if
not changed since".   
>> Dead easy for servers, I think.
> 
> I don't follow what this is. Can you elaborate with a
use case?

It is, as PSA suggested, just a convenient place to put a
potentially  
cool feature. I'm far from wed to it, but it'd be nice to at
least  
keep it in mind.

But, let's say that you had a bot which processed all
entries in it's  
roster. Let's further complicate things by having multiple
instances  
of the bot, because it's a CPU heavy bot. Now, it can do
work by:

1) Finding all entries not in the "Processed"
group.
2) Doing a conditional modify of the roster, placing an
entry into a  
"Processing" group.
3) If that succeeds, process, then place it
(unconditionally) into  
the "Processed" group.
4) GOTO 1 (Gotta love that BASIC).

There's far simpler use-cases, such as adding a group. (The
client  
cannot add a group, it can merely change groups, so to add
one  
without potentially removing a newly added one, it must
ensure  
nothing else has changed the roster since it last
operated.)

For both these use-cases, it would imply that we could
specify this  
slightly tighter, and insist that each roster entry has a
sequence  
value visible to the client. I can't think of a sane server 

implementation that wouldn't do this anyway.

Dave.
-- 
Dave Cridland - mailto:davecridland.net - xmpp:dwdjabber.org
  -
acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

On Wed Mar  5 11:00:34 2008, Richard Dobson wrote:
>> Sometimes flexibility comes back to bite you. I'd
prefer to keep  
>> things
>> simple if we can. What does the extra flexibility
really do for  
>> us, and
>> is it worth the cost?
> Well I think it is better to be flexible in this case
otherwise its  
> forcing server implementors to implement this in a
particular way  
> when there is no real need to, what if in a particular 

> implementation its more efficient to implement it as
GUIDs for  
> example as Alex suggested because of how its clustered
or how the  
> database is implemented, in my case id want to
implement this as a  
> compressed timestamp rather than as an increasing
integer, and as  
> far as the spec is currently written and how it would
work for  
> clients it wouldn't make any difference what the
version identifier  
> is as it isn't (and IMO MUST NOT) take any meaning from
the value  
> of the version identifier so things stay nice and
simple, as as  
> soon as clients start taking any meaning of what the
version  
> identifier means you are introducing a whole raft of
potential  
> interoperability issues and bugs that need not exist.
> 
> 
You can't use timestamps - they're not strictly increasing,
for  
various reasons.

Firstly, two roster changes could happen at precisely the
same  
moment. To be fair, by introducing cluster node identifiers,
and  
having a strict strong ordering of them, you could avoid
this.

Secondly, the clock on a computer can, and surprisingly
often does,  
go backwards. That's a much harder problem to solve.

Thirdly, in a clustering situation, you'd have to ensure
that the  
time on each cluster node was perfectly synchronized.

So the closest you can do would be a modified timestamp that
had  
additional logic during generation to ensure it never went
backwards,  
in which case you don't need the cluster identifier anymore,
and  
that's effectively the same as having a strictly increasing
integer  
sequence anyway, so it's easier to just do that. But even if
you did  
want to use timestamps, just representing them as an integer
is  
pretty trivial. Look at the definition of
"modtime" in ACAP (RFC  
2244), which defines a strictly increasing modified
timestamp  
represented using digits.



>> The only reason this scares me is that strictly
increasing numeric  
>> sequences have proved useful in the IMAP world,
because clients  
>> can spot when things go wrong much more easily.
> I think this its a very very bad idea for the clients
to take any  
> meaning from the version identifier as explained above,
its just  
> opening a pandora's box of potential bugs and issues,
far better  
> for it to just be a opaque string as far as the client
in  
> concerned, which also helps to keep things as simple as
possible.
> 
> 
It's useful for clients to be able to determine the ordering
locally,  
on occasion. If we removed this, we'd also have to ensure
that roster  
pushes were sent to the client in-order, which currently we
don't  
mandate. (Making this a SHOULD is sane, but in the cluster
case, it's  
quite hard).



>> Plus, nobody can get it wrong.
> How exactly are they going to get it wrong if its an
identifier  
> that only the server is interpreting the meaning of?
> 
> 
It's the server I'm worried about. 



>> There's no way that even a 32-bit unsigned integer
is going to  
>> overflow - if you did an update every second, it'd
take 136 years  
>> - but if that still unnerves you (in case PSA turns
into the  
>> undead, or something), use a 64-bit unsigned
integer.
> Its possible even if unlikely, what if several updates
were made in  
> one second because of some kind of bulk update for
example, at the  
> very least the spec MUST define what should happen if
the number is  
> going to overflow, i.e. reseting to 0, even though it
is unlikely.

You just use a 128-bit unsigned integer. There is no upper
limit here  
- in particular, there is no upper limit specified anywhere
in this  
document - XSD merely states that a xs:nonNegativeInteger is
a  
sequence of digits, and has "countably infinite"
cardinality.

If you really and truly believe that practical limits of
64-bit  
unsigned integers can cause problems in the real world, I
honestly  
don't know what to say except show you the figures - you
could have  
thousands of updates every millisecond, and still last over
half a  
million years - 574,542 roughly, assuming a fixed year
length of  
365.25 days.

I'm all for designing for the future, but you have to draw
the line  
somewhere, and besides, I figure we'll be on something
bigger than  
64-bit well before then - a jump to 128-bit gains us 10^25
years of  
breathing space, and I'd like to imagine we can think up a
solution  
within that time, assuming that's prior to the heat death of
the  
universe.

Dave.
-- 
Dave Cridland - mailto:davecridland.net - xmpp:dwdjabber.org
  -
acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

On Wednesday 05 March 2008 5:33 pm, Richard Dobson wrote:
> Peter Saint-Andre wrote:
> > Alexander Gnauck wrote:
> >> Peter Saint-Andre schrieb:
> >>> Here is revised text:
> >>>
> >>>    The value of the 'version' attribute
SHOULD be a non-negative
> >>>    integer representing a strictly
increasing sequence number that
> >>>    is increased with any change to the
roster (whether or not the
> >>>    client supports this extension) but MAY
be a unique identifer that
> >>>    is opaque to the client but understood
by the server. In any case,
> >>>    the 'version' attribute contained in
roster pushes MUST be unique.
> >>
> >> great, +1
> >
> > The Council didn't like the change from MUST to
SHOULD as discussed here:
> >
> > http://logs.jabber.org
/councilconference.jabber.org/2008-03-05.html#13:4
> >0:54
> >
> > Discussion can continue, naturally. 
>
> Oh well thats a shame, just have leave it as my own
extension.
> Implemented timestamps this evening to try it out and
its working fine
> thus far.

I, too, don't understand why this value needs to be
specified as being a 
strictly-increasing integer, if the intent is that clients
should treat it as 
an opaque cookie thing anyway.

The argument is that allowing the value to be some free form
string is 
unnecessarily flexible.  Sure.  However, I'd argue that
requiring it to be a 
strictly-increasing integer amounts to overspecification,
especially when we 
have an established trend of using free form strings all
over XMPP.  I'd feel 
the same way if Dialback mandated a particular key format. 
We don't 
need "nanny specs".

There's a big difference between avoiding overspecification
(this) and being 
stupidly flexible (the joke in the council logs: "a
stanza of type 
<message /> SHOULD be used, but another arbitrary
string MAY be used, such as 
<thing />").

-Justin

On Wednesday 05 March 2008 8:27 pm, Peter Saint-Andre
wrote:
> Justin Karneges wrote:
> > I, too, don't understand why this value needs to
be specified as being a
> > strictly-increasing integer, if the intent is that
clients should treat
> > it as an opaque cookie thing anyway.
>
> I'm not convinced that that's the intent. It seems that
it might be
> useful in some situations for the client to know where
it stands.

That changes things then.  I saw some mention of that in the
council logs, but 
I must have missed that on the mailing list so I didn't
consider it.

> > The argument is that allowing the value to be some
free form string is
> > unnecessarily flexible.  Sure.  However, I'd argue
that requiring it to
> > be a strictly-increasing integer amounts to
overspecification, especially
> > when we have an established trend of using free
form strings all over
> > XMPP.  I'd feel the same way if Dialback mandated
a particular key
> > format.  We don't need "nanny specs".
>
> We do need technologies that won't break.

Indeed.  To clarify the point I was trying to make, imagine
there are three 
categories for specifications: overly flexible, neutral, and
restrictive.

Opaque string: neutral.
Strictly-increasing integer: restrictive.

Compare this to the joke about stanzas:

<message/>-only: neutral.
<message/> or <thing/>: overly flexible.

What counts as what is a matter of what the usual design
practices and trends 
are for related specifications.  My assessment, from what I
read in the 
mailing list arguments, is that opaque was in the 'neutral'
category.  Thus 
it should not be Richard that needs to argue for opaque, but
rather it should 
be Dave that needs to argue for integer.  IMO, I guess.

Okay, enough of that nonsense. 

> > There's a big difference between avoiding
overspecification (this)
>
> I'll let Dave Cridland explain why he thinks this is
not
> overspecification. I think his experience with IETF
email and data
> storage protocols is relevant here. In particular,
during the Council
> meeting [1] he said:
>
[snip]

To be honest, all of this still reads like implementation
notes to me, except:

> "One interesting point is that with an int,
there's always something a
> client can use to get the entire roster, with
versioning turned on."
>
> "I'm not quite sure what the client ought to send
if everything's
> completely opaque - it'd need more syntax."

Maybe these could be explained here on the list.  The
implication in these 
statements is that the revision value has meaning to the
client, and that the 
client can even create these values.

-Justin