hibernate prototype

Brian,

I've just about finished a hibernate persistence layer
prototype.  This 
layer consists of an object model (based on org.cosmo.model)
new dao 
interfaces, and hibernate implementations of the daos.  In
addition to a 
new User and Ticket dao, there is a Content (for general
content/files) 
and Calendar (for calendar objects) dao.  These two daos
should give you 
all the methods necessary for supporting webdav and caldav. 
I have done 
some initial tests on large datasets and the results are
very promising 
(sub second time-range/property queries).  There is still a
lot of 
optimizing to be done, but it is a good start.  I have
tested using 
mysql and derby.

I've been developing this outside of the cosmo environment
as a 
stand-alone layer(to speed things up), and its baked enough
to where I 
can start integrating it into the cosmo source via the
sandbox.  Is the 
sandbox up to date?  Also, I'm new to maven so I'm trying
to figure out 
how to add in dependency support.  What do I do if I I have
a dependency 
that isn't found on one of the maven repositories?

My goal is to have a first-pass checked in by the end of the
week so you 
can start re-factoring jcr-server.  I forsee a bunch of
changes as you 
find out what works/what doesn't.  I just want to get
something in place 
so we can move forward.

-Randy
_______________________________________________
cosmo-dev mailing list
cosmo-devlists.osafoundation.org
http://lists.osafoundation.org/mailman/listinfo/cosmo-d
ev

On 8/15/06, rletnesssimdeskcorp.com <rletnesssimdeskcorp.com> wrote:

> I've just about finished a hibernate persistence layer
prototype.  This
> layer consists of an object model (based on
org.cosmo.model) new dao
> interfaces, and hibernate implementations of the daos. 
In addition to a
> new User and Ticket dao, there is a Content (for
general content/files)
> and Calendar (for calendar objects) dao.  These two
daos should give you
> all the methods necessary for supporting webdav and
caldav.  I have done
> some initial tests on large datasets and the results
are very promising
> (sub second time-range/property queries).  There is
still a lot of
> optimizing to be done, but it is a good start.  I have
tested using
> mysql and derby.

great news! have you done any work with recurring events or
freebusy info?

> I've been developing this outside of the cosmo
environment as a
> stand-alone layer(to speed things up), and its baked
enough to where I
> can start integrating it into the cosmo source via the
sandbox.  Is the
> sandbox up to date?  Also, I'm new to maven so I'm
trying to figure out
> how to add in dependency support.  What do I do if I I
have a dependency
> that isn't found on one of the maven repositories?

the sandbox was copied from the trunk a few weeks ago. the
only work
that's happened on the trunk subsequent to the copy is the
scooby
merge, which is more or less independent of the hibernate
work.

our current plan is to release 0.5 with the merged scooby ui
and with
a few additonal sharing-related features. i will elaborate
on this
plan in a few days as we begin more in depth discussions.

after 0.5, assuming the hibernate and jcr-server code is
stable and we
want to move forward with it, we'll assess the viability of
merging it
from the sandbox back into the trunk and making it the
centerpiece of
0.6.

re maven - we host a repository at osaf where we can put
jars that
aren't available in the main repository. just let me know
what
specifically you need to be in there and i'll get that
done. i wonder
if you're thinking of the jta jar and others that aren't
specifically
publicly distributable on their own?

> My goal is to have a first-pass checked in by the end
of the week so you
> can start re-factoring jcr-server.  I forsee a bunch of
changes as you
> find out what works/what doesn't.  I just want to get
something in place
> so we can move forward.

as a matter of fact, i've been working on the jcr-server
updates for
the last week and a half. i've got the code that uses jcr
isolated
into single package and have replaced it in the running
server with
new code that uses HomeDirectoryService to read and write
content.
this requires new service apis, since right now there's no
way through
HDS to actually save a resource, and so far i've been
implementing
those service apis with jcr daos. but once you've committed
your
hibernate code, i'll wire up HDS to use that instead, and
the last 20%
of the jcr-server work should be pretty straightforward.
this is
exciting!
_______________________________________________
cosmo-dev mailing list
cosmo-devlists.osafoundation.org
http://lists.osafoundation.org/mailman/listinfo/cosmo-d
ev

Brian Moseley wrote:

> On 8/15/06, rletnesssimdeskcorp.com
<rletnesssimdeskcorp.com> wrote:
>
>> I've just about finished a hibernate persistence
layer prototype.  This
>> layer consists of an object model (based on
org.cosmo.model) new dao
>> interfaces, and hibernate implementations of the
daos.  In addition to a
>> new User and Ticket dao, there is a Content (for
general content/files)
>> and Calendar (for calendar objects) dao.  These two
daos should give you
>> all the methods necessary for supporting webdav and
caldav.  I have done
>> some initial tests on large datasets and the
results are very promising
>> (sub second time-range/property queries).  There is
still a lot of
>> optimizing to be done, but it is a good start.  I
have tested using
>> mysql and derby.
>
>
> great news! have you done any work with recurring
events or freebusy 
> info?

Well, I implemented a calendar query method that takes in a
calendar 
filter that
represents the caldav:filter element and supports time 
ranges/comp/prop/param filters. 
This was all based off of the way calendar queries are done
in cosmo 
today.  
I finally figured out some of what was going on.  Every time
an event is 
created,
it is run through the CalendarFlattener which generates
timeranges and 
stores them
as node properties.  I adapted the code to do the same
thing, except 
instead of
storing the timeranges as properties, I store them in a
separate 
calendar index
table, which is joined to the item table.  Then instead of
generating a
timerange xpath query, I generate a sql query based on the
same algorithm. 
So it should work as cosmo does today.

I looked at the way the  freebusy  query is done today and
it looks like 
you are
just generating a calendar query using a calendar filter, so
that should 
be supported
as well.

> the sandbox was copied from the trunk a few weeks ago.
the only work
> that's happened on the trunk subsequent to the copy is
the scooby
> merge, which is more or less independent of the
hibernate work.
>
Ok, cool.  I'm going to start merging stuff in soon.  To
prevent 
breaking things in the
sandbox, I'll probably just merge stuff in as separate
packages so as to
not conflict with the current model and dao definitions
(think 
org.osaf.cosmo.model.new).
Otherwise I see a lot of things breaking right away.  Then I
want to get 
some
unit tests checked in and working.

> re maven - we host a repository at osaf where we can
put jars that
> aren't available in the main repository. just let me
know what
> specifically you need to be in there and i'll get that
done. i wonder
> if you're thinking of the jta jar and others that
aren't specifically
> publicly distributable on their own?

Yeah, jta.jar was one of them.  There was also a library I
was using
to generate uuids, jug.

> as a matter of fact, i've been working on the
jcr-server updates for
> the last week and a half. i've got the code that uses
jcr isolated
> into single package and have replaced it in the running
server with
> new code that uses HomeDirectoryService to read and
write content.
> this requires new service apis, since right now
there's no way through
> HDS to actually save a resource, and so far i've been
implementing
> those service apis with jcr daos. but once you've
committed your
> hibernate code, i'll wire up HDS to use that instead,
and the last 20%
> of the jcr-server work should be pretty
straightforward. this is
> exciting!

Sweet, sounds good.  I'll probably have more questions as I
try to
merge stuff into the sandbox and get unit tests integrated
into the
build too.

-Randy
_______________________________________________
cosmo-dev mailing list
cosmo-devlists.osafoundation.org
http://lists.osafoundation.org/mailman/listinfo/cosmo-d
ev

On 8/15/06, rletnesssimdeskcorp.com <rletnesssimdeskcorp.com> wrote:
> I adapted the code to do the same thing, except
> instead of
> storing the timeranges as properties, I store them in a
separate
> calendar index
> table, which is joined to the item table.  Then instead
of generating a
> timerange xpath query, I generate a sql query based on
the same algorithm.
> So it should work as cosmo does today.

did you notice that we're only expanding recurring events
out to 2008?

before that we were going to i think 2050, and that was
causing
PUTs of these events to take several seconds.

something to consider for later, assuming we adopt the
prototype, is
how to handle infinitely recurring events (or even ones that
recur
past 2008). clearly it's tempting to index everything at
write time,
but at least with jcr, that had severe performance
implications. we
considered strategies like expanding two years at write time
and then
doing some sort of "just in time" expansion when
servicing queries for
time ranges that are outside that two year segment. love to
hear any
ideas you have.

> Ok, cool.  I'm going to start merging stuff in soon. 
To prevent
> breaking things in the
> sandbox, I'll probably just merge stuff in as separate
packages so as to
> not conflict with the current model and dao definitions
(think
> org.osaf.cosmo.model.new).
> Otherwise I see a lot of things breaking right away. 
Then I want to get
> some
> unit tests checked in and working.

i don't mind things breaking. to me, breakage is a clear
and constant
reminder of the work that's still to be done. i'm the only
person
other than you working in the sandbox, and i won't be back
to coding
for probably a week or more. so my vote would be to replace
the
existing code rather than using new packages. but it's
really up to
you.

> Yeah, jta.jar was one of them.  There was also a
library I was using
> to generate uuids, jug.

have you seen commons-id? we're using it to generate uuids
for scooby.
maybe you can replace jug with it? i've never heard of jug,
do you
have a link?
_______________________________________________
cosmo-dev mailing list
cosmo-devlists.osafoundation.org
http://lists.osafoundation.org/mailman/listinfo/cosmo-d
ev

Brian Moseley wrote:

>
> did you notice that we're only expanding recurring
events out to 2008?
> 
before that we were going to i think 2050, and that was
causing
> PUTs of these events to take several seconds.
>
> something to consider for later, assuming we adopt the
prototype, is
> how to handle infinitely recurring events (or even ones
that recur
> past 2008). clearly it's tempting to index everything
at write time,
> but at least with jcr, that had severe performance
implications. we
> considered strategies like expanding two years at write
time and then
> doing some sort of "just in time" expansion
when servicing queries for
> time ranges that are outside that two year segment.
love to hear any
> ideas you have.
>
Yeah I saw that.  One way to solve this is to just not allow
recurring 
events
that go past a certain number of recurrences.  What
percentage of events
are infinitely recurring??  It has to be very small.  The
other idea I 
had was
instead of indexing time ranges, just compute a start and
end date (could be
infinity) based on the recurrence rules.  This range would
be computed and
indexed at creation time.  This means a database query gets
a superset of
the events you are interested in.  Recurring events returned
as part of a
time-range query would be passed through some business logic
to filter out
events that don't  have a recurrence in the timerange. 
This works for 
simple
time ranges,  but when you have nested timeranges, like you
can in a caldav
calendar filter, it gets complicated. 

Because infinitely recurring events are such a small
percentage of events
it would suck to have to implement such complicated logic
just for that.

> have you seen commons-id? we're using it to generate
uuids for scooby.
> maybe you can replace jug with it? i've never heard of
jug, do you
> have a link?

I'll check it out...and probably just swich to using
commons-id to make 
it easy.
Here is a link to jug:  http://jug.safehaus.org/

-Randy
_______________________________________________
cosmo-dev mailing list
cosmo-devlists.osafoundation.org
http://lists.osafoundation.org/mailman/listinfo/cosmo-d
ev

On 8/15/06, rletnesssimdeskcorp.com <rletnesssimdeskcorp.com> wrote:

> Yeah I saw that.  One way to solve this is to just not
allow recurring
> events
> that go past a certain number of recurrences.  What
percentage of events
> are infinitely recurring??  It has to be very small.

well, looking at my personal calendar, i've got:

 * two events that repeat every other thursday with no end
date
 * one event that repeats every m,w,th,f,sun with no end
date
 * one event that repeats every m,t,th,f that ends after six
weeks
 * one event that repeats every t,th with no end date

so i don't think it's as small as you might hope 

> The other idea I
> had was
> instead of indexing time ranges, just compute a start
and end date (could be
> infinity) based on the recurrence rules.  This range
would be computed and
> indexed at creation time.  This means a database query
gets a superset of
> the events you are interested in.  Recurring events
returned as part of a
> time-range query would be passed through some business
logic to filter out
> events that don't  have a recurrence in the timerange.
 This works for
> simple
> time ranges,  but when you have nested timeranges, like
you can in a caldav
> calendar filter, it gets complicated.

yeah, this is essentially re-expanding the event on every
query. my
instinct is that it's the wrong approach, but i'm willing
to be proven
wrong.

> Because infinitely recurring events are such a small
percentage of events
> it would suck to have to implement such complicated
logic just for that.

unfortunately i think it's a pretty common use case :/
_______________________________________________
cosmo-dev mailing list
cosmo-devlists.osafoundation.org
http://lists.osafoundation.org/mailman/listinfo/cosmo-d
ev

Brian Moseley wrote:

> well, looking at my personal calendar, i've got:
>
> * two events that repeat every other thursday with no
end date
> * one event that repeats every m,w,th,f,sun with no end
date
> * one event that repeats every m,t,th,f that ends after
six weeks
> * one event that repeats every t,th with no end date
>
> so i don't think it's as small as you might hope 
>
Doh!  I don't have any like this but I guess its a valid
use case. 
I still think we could force the user to select a reasonable
end date by
denying infinte recurring events, but that might piss people
off 

>
> yeah, this is essentially re-expanding the event on
every query. my
> instinct is that it's the wrong approach, but i'm
willing to be proven
> wrong.
>
I'm sure a better approach would be a hybrid of indexing
for a certain
period (year) an then expanding if a timerange request comes
in outside of
that period.   There could even be a backround process to
re-index 
to keep the indexed periods up to date.

-Randy
_______________________________________________
cosmo-dev mailing list
cosmo-devlists.osafoundation.org
http://lists.osafoundation.org/mailman/listinfo/cosmo-d
ev