Iris Roadmap

Folks,

I hope to resume development on Iris very soon, and so I
figured I'd jot down 
some plans.  First, I should say that I've been looking
forward to this for a 
*long* time.  Many of the ideas below have been in my notes
for years.

There are 3 major areas of Iris:
  1) networking (dns, tcp, etc)
  2) low-level xmpp (c2s, stanzas, etc)
  3) high-level xmpp (rosters, pep, etc)

I intend to focus on areas 1 & 2 for the time being.  I
might eventually work 
on area 3, but that mostly depends on how well the other Psi
developers 
maintain that area.  If it looks like things get stale or
there's no 
innovation, then I may step in.  Otherwise, area 3 has seen
a lot of action 
lately with 0.11 so I'll stay out of it.

Networking (irisnet)
-----
The goal for this area is to create a toolbox that makes
implementing network 
protocols easier.  In some ways it is
"cutestuff/network next-gen".  Classes 
in here won't be specific to XMPP.  Lots of code and
concepts that were once 
the domain of iris/xmpp-core (e.g. auth, sasl, proxying,
certificate 
handling, parsing) will probably be moved into here for
reuse by all of the 
protocols we support.  Also notable will be expanded support
for DNS 
(including DNS-SD) and UDP.  I'd like to also add SOCKS4,
since we seem to 
get a request about that every so often.  Finally, I'll have
"legacy" 
wrappers to assist in porting.  For example, there's already
a drop-in 
replacement for NDns which calls NameResolver internally.

Low-level XMPP
-----
XMPP::Stream will finally have subclasses besides
ClientStream.  I'm not sure 
about the class naming yet, but we essentially need
subclasses for c2s, s2s, 
xmpp-over-bosh, and c2c (link-local).  I'd then like to
extend the Router 
concept used in Ambrosia (and many other XMPP server
implementations) to 
clients.  Instead of writing a stanza to a specific stream
(currently, 
stanzas are written to a specific XMPP::Client instance
which is associated 
with a single XMPP::ClientStream instance), stanzas will be
sent to a Router 
object, which will then send the stanza out on the
appropriate stream, 
possibly even creating the stream on demand (s2s, c2c). 
XMPP::Stream objects 
may become invalid, but Router objects will never become
invalid.  Router 
objects will always be operational, regardless of the
connection state 
(similar to XMPP::Client), and if there is no stream to
write to (or none can 
be made on demand, if appropriate) then the Router would
bounce the stanza.  
It should even be conceivable to use a single Router to
operate multiple 
jabber accounts, although I don't expect Psi to be written
this way.  The 
Router concept will make the most sense with Link-local
chat, where you 
really don't want to be thinking in terms of streams.

Phase 1
-----
The first phase of my Iris work will target the lowest
levels of area 1:
  - dns
    - lookups, unicast and multicast (seems to be done)
    - service resolver abstraction (blends dns-sd and
internet srv)
    - jdns (done)
    - mDNSResponder plugin (need this for dns-sd support on
mac platform)
    - libidn/punycode/nameprep?  (Qt has some stuff, need to
investigate)
  - network interface detection (somewhat done)
  - abstractions: ByteStream (for tcp-ish) and PacketStream
(for udp-ish)
  - tcp
    - client and server
    - ack tracking for operating systems that support it
  - udp
    - client and server, 1-to-1 only
  - legacy wrappers

If anyone has a suggestion of something else that should be
put at this level, 
let me know.  A couple that I'm unsure about are operating
system proxy 
server detection and UPnP, but I'm thinking maybe these
should be a layer 
higher.

-Justin
_______________________________________________
Delta mailing list
Deltalists.affinix.com
http://lists.affinix.com/listinfo.cgi/delta-affinix.com

Yes, I figured splitting the bosh part out into irisnet
would be wise.  Do you 
know if there are plans to use bosh for anything besides
xmpp?

-Justin

On Sunday 16 September 2007 11:02 am, Norman Rasmussen
wrote:
> does this make sense:
>
> xep-0124 (bosh) is level 1
> xep-0206 (xmpp via bosh) is level 2
>
> that way anyone wanting to pick up bosh for comms would
use irisnet and not
> have to worry about any xmpp stuff
>
> On 9/15/07, Justin Karneges <justin-psi2affinix.com> wrote:
> > Folks,
> >
> > I hope to resume development on Iris very soon,
and so I figured I'd jot
> > down
> > some plans.  First, I should say that I've been
looking forward to this
> > for a
> > *long* time.  Many of the ideas below have been in
my notes for years.
> >
> > There are 3 major areas of Iris:
> >   1) networking (dns, tcp, etc)
> >   2) low-level xmpp (c2s, stanzas, etc)
> >   3) high-level xmpp (rosters, pep, etc)
> >
> > I intend to focus on areas 1 & 2 for the time
being.  I might eventually
> > work
> > on area 3, but that mostly depends on how well the
other Psi developers
> > maintain that area.  If it looks like things get
stale or there's no
> > innovation, then I may step in.  Otherwise, area 3
has seen a lot of
> > action
> > lately with 0.11 so I'll stay out of it.
> >
> > Networking (irisnet)
> > -----
> > The goal for this area is to create a toolbox that
makes implementing
> > network
> > protocols easier.  In some ways it is
"cutestuff/network
> > next-gen".  Classes
> > in here won't be specific to XMPP.  Lots of code
and concepts that were
> > once
> > the domain of iris/xmpp-core (e.g. auth, sasl,
proxying, certificate
> > handling, parsing) will probably be moved into
here for reuse by all of
> > the
> > protocols we support.  Also notable will be
expanded support for DNS
> > (including DNS-SD) and UDP.  I'd like to also add
SOCKS4, since we seem
> > to get a request about that every so often. 
Finally, I'll have "legacy"
> > wrappers to assist in porting.  For example,
there's already a drop-in
> > replacement for NDns which calls NameResolver
internally.
> >
> > Low-level XMPP
> > -----
> > XMPP::Stream will finally have subclasses besides
ClientStream.  I'm not
> > sure
> > about the class naming yet, but we essentially
need subclasses for c2s,
> > s2s,
> > xmpp-over-bosh, and c2c (link-local).  I'd then
like to extend the Router
> > concept used in Ambrosia (and many other XMPP
server implementations) to
> > clients.  Instead of writing a stanza to a
specific stream (currently,
> > stanzas are written to a specific XMPP::Client
instance which is
> > associated
> > with a single XMPP::ClientStream instance),
stanzas will be sent to a
> > Router
> > object, which will then send the stanza out on the
appropriate stream,
> > possibly even creating the stream on demand (s2s,
c2c).  XMPP::Stream
> > objects
> > may become invalid, but Router objects will never
become invalid.  Router
> > objects will always be operational, regardless of
the connection state
> > (similar to XMPP::Client), and if there is no
stream to write to (or none
> > can
> > be made on demand, if appropriate) then the Router
would bounce the
> > stanza.
> > It should even be conceivable to use a single
Router to operate multiple
> > jabber accounts, although I don't expect Psi to be
written this way.  The
> > Router concept will make the most sense with
Link-local chat, where you
> > really don't want to be thinking in terms of
streams.
> >
> > Phase 1
> > -----
> > The first phase of my Iris work will target the
lowest levels of area 1:
> >   - dns
> >     - lookups, unicast and multicast (seems to be
done)
> >     - service resolver abstraction (blends dns-sd
and internet srv)
> >     - jdns (done)
> >     - mDNSResponder plugin (need this for dns-sd
support on mac platform)
> >     - libidn/punycode/nameprep?  (Qt has some
stuff, need to investigate)
> >   - network interface detection (somewhat done)
> >   - abstractions: ByteStream (for tcp-ish) and
PacketStream (for udp-ish)
> >   - tcp
> >     - client and server
> >     - ack tracking for operating systems that
support it
> >   - udp
> >     - client and server, 1-to-1 only
> >   - legacy wrappers
> >
> > If anyone has a suggestion of something else that
should be put at this
> > level,
> > let me know.  A couple that I'm unsure about are
operating system proxy
> > server detection and UPnP, but I'm thinking maybe
these should be a layer
> > higher.
_______________________________________________
Delta mailing list
Deltalists.affinix.com
http://lists.affinix.com/listinfo.cgi/delta-affinix.com

On that note, I know Peter was considering splitting out the
"XML streaming" 
part of the XMPP-Core RFC such that the concept could stand
independent of 
XMPP.  Of course, other than XMPP I'm not sure what we'd
ever use XML 
streaming for (or even if we'd want to promote such a
thing!), but there is 
something attractive about having a hierarchical design. 
Iris already has 
XML streaming implemented independently of XMPP, and I plan
to move that part 
into irisnet.

-Justin

On Sunday 16 September 2007 1:23 pm, Norman Rasmussen
wrote:
> I'm not sure, does anyone else do any sort of serious
(partial) xml
> streaming (I looked on google but didn't find
_anything_ besides xmpp or
> course)
>
> On 9/16/07, Justin Karneges <justin-psi2affinix.com> wrote:
> > Yes, I figured splitting the bosh part out into
irisnet would be wise. 
> > Do you
> > know if there are plans to use bosh for anything
besides xmpp?
> >
> > -Justin
> >
> > On Sunday 16 September 2007 11:02 am, Norman
Rasmussen wrote:
> > > does this make sense:
> > >
> > > xep-0124 (bosh) is level 1
> > > xep-0206 (xmpp via bosh) is level 2
> > >
> > > that way anyone wanting to pick up bosh for
comms would use irisnet and
> >
> > not
> >
> > > have to worry about any xmpp stuff
_______________________________________________
Delta mailing list
Deltalists.affinix.com
http://lists.affinix.com/listinfo.cgi/delta-affinix.com

On Saturday 15 September 2007 14:40:11 Justin Karneges
wrote:
> I hope to resume development on Iris very soon

So, almost a year later...

> Phase 1
> -----
> The first phase of my Iris work will target the lowest
levels of area 1:
>   - dns
>     - lookups, unicast and multicast (seems to be
done)
>     - service resolver abstraction (blends dns-sd and
internet srv)
>     - jdns (done)
>     - mDNSResponder plugin (need this for dns-sd
support on mac platform)
>     - libidn/punycode/nameprep?  (Qt has some stuff,
need to investigate)
>   - network interface detection (somewhat done)
>   - abstractions: ByteStream (for tcp-ish) and
PacketStream (for udp-ish)
>   - tcp
>     - client and server
>     - ack tracking for operating systems that support
it
>   - udp
>     - client and server, 1-to-1 only
>   - legacy wrappers

I guess I did get some Iris work done here and there, the
most significant 
being the appledns plugin (corresponds to
"mDNSResponder plugin" in the above 
list from September).

Today I added a new class, NetAvailability, for determining
whether or not you 
have a usable network connection.  Previously, I was
planning for this 
information to be calculated from NetInterface, but it might
be better to 
allow NetAvailability to stand on its own and use native
system calls 
underneath if possible (for example, Windows has an event
specifically for 
network availability, and there's also NetworkManager on
Linux).

I need to do something about zlib and libidn.  I may want to
require these to 
exist as separate libraries, but then that might just annoy
everyone.  
Interestingly, Qt requires zlib and will bundle it if
necessary, and Qt even 
has its own stringprep code, but neither is public so I
don't think we can 
leverage them...

-Justin
_______________________________________________
Delta mailing list
Deltalists.affinix.com
http://lists.affinix.com/listinfo.cgi/delta-affinix.com

2008/8/20 Justin Karneges <justin-psi2affinix.com>:
> I need to do something about zlib and libidn.  I may
want to require these to
> exist as separate libraries, but then that might just
annoy everyone.
> Interestingly, Qt requires zlib and will bundle it if
necessary, and Qt even
> has its own stringprep code, but neither is public so I
don't think we can
> leverage them...
>

If you are going to bundle zlib and libidn (I guess to
simplify the
building of iris on mac and windows platform), please
provide an easy
way (e.g. configure option) to disable the bundled copies
and use the
system ones, with appropriate version checks of course.
Bundled
libraries are bad for linux distributions and package
mantainers.
Thank you.

Davide
_______________________________________________
Delta mailing list
Deltalists.affinix.com
http://lists.affinix.com/listinfo.cgi/delta-affinix.com

On Tuesday 19 August 2008 15:32:08 Justin Karneges wrote:
> I need to do something about zlib and libidn.  I may
want to require these
> to exist as separate libraries, but then that might
just annoy everyone.
> Interestingly, Qt requires zlib and will bundle it if
necessary, and Qt
> even has its own stringprep code, but neither is public
so I don't think we
> can leverage them...

After some thought, I think we should drop our copy of zlib.
 It is only used 
to make Windows compilation easier (which is not a bad
cause, but I sort of 
view it more as Windows being the hold-out to doing it
right).  Qt bundles 
zlib, and since Psi does also then that means Psi on Windows
generally 
contains two copies of zlib.  What we should start doing on
Windows is 
building a single copy of zlib that both Qt and Psi share.

I understand there are probably lots of Qt/Windows setups
that already use a 
bundled zlib.  Probably any of the binary packages of Qt are
this way.  
That's okay, you'd still be able to compile Psi by linking
to a second zlib 
that you obtain yourself.  This copy of zlib can even be
statically linked, 
if you don't want a DLL, and you'll end up with a Psi that
is no different 
than before.  But, at least we wouldn't be promoting an
ancient bundled zlib, 
and it would be possible to rebuild your Qt and Psi against
a shared zlib DLL 
if you so desired.

Kev/Remko: yea/nay?

-Justin
_______________________________________________
Delta mailing list
Deltalists.affinix.com
http://lists.affinix.com/listinfo.cgi/delta-affinix.com

> Kev/Remko: yea/nay?

How do you feel about the QCA way: bundling it by default
in
third-party/zlib, but having a --disable-bundled-zlib (and
--with-zlib-dir=) to allow you to link against an external
zlib if you
want to? I'ld like to keep the './configure && make
always work'
principle, because it makes life so much easier to start
developing on
platforms like windows etc. On the other hand, i'ld also
like our
binary to get as small as possible and not link in
unnecessary
external libs.

cheers,
Remko
_______________________________________________
Delta mailing list
Deltalists.affinix.com
http://lists.affinix.com/listinfo.cgi/delta-affinix.com

On Thursday 21 August 2008 23:09:06 Remko Tronçon wrote:
> > Kev/Remko: yea/nay?
>
> How do you feel about the QCA way: bundling it by
default in
> third-party/zlib, but having a --disable-bundled-zlib
(and
> --with-zlib-dir=) to allow you to link against an
external zlib if you
> want to? I'ld like to keep the './configure &&
make always work'
> principle, because it makes life so much easier to
start developing on
> platforms like windows etc. On the other hand, i'ld
also like our
> binary to get as small as possible and not link in
unnecessary
> external libs.

Psi already requires an external zlib on unix.  The
configure script already 
has --with-zlib-* stuff, and './configure && make'
already does work.  
Externalizing zlib for Windows wouldn't change that.

I guess if we made a source-based zlib bundle then that
would open the door to 
allowing bundling on unix platforms which we could tweak our
configure script 
for.  I don't think we want to promote that though.  Psi has
gone this long 
requiring an external zlib on unix without issue.

In any case, that weird Windows binary zlib has got to go. 
If we don't drop 
our zlib copy completely then let's at least use a source
bundle.

-Justin
_______________________________________________
Delta mailing list
Deltalists.affinix.com
http://lists.affinix.com/listinfo.cgi/delta-affinix.com

> In any case, that weird Windows binary zlib has got to
go.  If we don't drop
> our zlib copy completely then let's at least use a
source bundle.

Definitely.

Remko
_______________________________________________
Delta mailing list
Deltalists.affinix.com
http://lists.affinix.com/listinfo.cgi/delta-affinix.com

On Friday 22 August 2008 00:09:19 Remko Tronçon wrote:
> > In any case, that weird Windows binary zlib has
got to go.  If we don't
> > drop our zlib copy completely then let's at least
use a source bundle.
>
> Definitely.

Iris now builds stand-alone using an external zlib.  Psi can
of course still 
bundle zlib, but I consider this issue to be out of scope
for Iris.  Moving 
the thread to psi-devel.

-Justin
_______________________________________________
Delta mailing list
Deltalists.affinix.com
http://lists.affinix.com/listinfo.cgi/delta-affinix.com