prefetch implemented - perf test results updated

Hi all,

I've implemented property prefetching in nxcore.
This means you can control which properties are used to
initialize a new 
DocumentModel.
This avoid to load entire schema data which may be a costly
operation 
(especially if the schema is big and have complex types)

When accessing a property not prefetched the entire parent
schema will 
be loaded. This usually happens when entering the page
that display/edit the DocumentModel.
So the prefetching is especially useful for listing children
documents 
that needs only a small subset of properties to be loaded

Here are the new results of the same test - see my previous
email 
(calling getChildren() for 17818 GEIDE documents)
I run the test 4 times for each of the following
configurations:

1. prefetching only dc:title:
<prefetch>dc:title</prefetch>

1st run: 53.474000000002405 sec.
2nd run: 44.83799999999909 sec.
3rd run: 44.83799999999909 sec.
4th run: 45.51399999999991sec.

~ 47.17 sec => 0.0026 sec per iteration

2. prefetching 4 properties (the properties used by the web
client to 
display document lists)
<prefetch>common.icon, dc:title, dc:modified,
dc:contributors</prefetch>

1st run: 59.17800000000126 sec
2nd run: 50.47499999999744 sec
3rd run: 50.25299999999888 sec
4th run:  2.40400000000286 sec

~ 55.58 sec => 0.0031sec per iteration

3. prefetching the entrie dublincore schema
<prefetch>dublincore</prefetch>

1st run: 151.58500000001374 sec.
2nd run: 94.64499999999938 sec.
3rd run: 90.21400000000061 sec.
4th run: 90.1530000000008 sec.

~106.65 sec => 0.006 sec per iteration

4. prefetching all schemas

1st run: 481.0899999999764 sec.
2nd run: 760.1829999999819 sec.
3rd run: 751.6479999999821 sec.
4th run: 757.9399999999843 sec.

~687.71 sec => 0.038 sec per iteration


So by prefetching only the needed properties we get an
iteration time of 
0.0031sec instead of 0.038 for the full set of schemas (~12
times faster 
 )
This means we can fetch 20 documents in 0.06 seconds, 100
documents in 
0.3 seconds and ~333 docs in 1 second.
So even having a pagination of 100 docs per page on the core
side we 
have an acceptable time of 0.3 seconds

Bogdan


_______________________________________________
ECM mailing list
ECMlists.nuxeo.com
http://li
sts.nuxeo.com/mailman/listinfo/ecm

Bogdan Stefanescu wrote:
> I've implemented property prefetching in nxcore.
> This means you can control which properties are used to
initialize a new
> DocumentModel.
> This avoid to load entire schema data which may be a
costly operation
> (especially if the schema is big and have complex
types)
> 
> When accessing a property not prefetched the entire
parent schema will
> be loaded. This usually happens when entering the page
> that display/edit the DocumentModel.
> So the prefetching is especially useful for listing
children documents
> that needs only a small subset of properties to be
loaded
> 
> Here are the new results of the same test - see my
previous email
> (calling getChildren() for 17818 GEIDE documents)
> I run the test 4 times for each of the following
configurations:
> 
> 1. prefetching only dc:title:
> <prefetch>dc:title</prefetch>
> 
> 1st run: 53.474000000002405 sec.
> 2nd run: 44.83799999999909 sec.
> 3rd run: 44.83799999999909 sec.
> 4th run: 45.51399999999991sec.
> 
> ~ 47.17 sec => 0.0026 sec per iteration
> 
> 2. prefetching 4 properties (the properties used by the
web client to
> display document lists)
> <prefetch>common.icon, dc:title, dc:modified,
dc:contributors</prefetch>
> 
> 1st run: 59.17800000000126 sec
> 2nd run: 50.47499999999744 sec
> 3rd run: 50.25299999999888 sec
> 4th run:  2.40400000000286 sec
> 
> ~ 55.58 sec => 0.0031sec per iteration
> 
> 3. prefetching the entrie dublincore schema
> <prefetch>dublincore</prefetch>
> 
> 1st run: 151.58500000001374 sec.
> 2nd run: 94.64499999999938 sec.
> 3rd run: 90.21400000000061 sec.
> 4th run: 90.1530000000008 sec.
> 
> ~106.65 sec => 0.006 sec per iteration
> 
> 4. prefetching all schemas
> 
> 1st run: 481.0899999999764 sec.
> 2nd run: 760.1829999999819 sec.
> 3rd run: 751.6479999999821 sec.
> 4th run: 757.9399999999843 sec.
> 
> ~687.71 sec => 0.038 sec per iteration
> 
> 
> So by prefetching only the needed properties we get an
iteration time of
> 0.0031sec instead of 0.038 for the full set of schemas
(~12 times faster
> 
)
> This means we can fetch 20 documents in 0.06 seconds,
100 documents in
> 0.3 seconds and ~333 docs in 1 second.
> So even having a pagination of 100 docs per page on the
core side we
> have an acceptable time of 0.3 seconds

Nice 

Did you think of our lazy fetch problem which used to invoke
back the
client and could let to errors if the document model was
transmitted on
the JMS channel and grabbed by an MDB for instance : in this
case the
client session was lost ?

	J.

-- 
Julien Anguenot | Nuxeo R&D (Paris, France)
Open Source ECM - http://www.nuxeo.com
Nuxeo 5 : http://www.nuxeo.org
Mobile: +33 (0) 6 72 57 57 66


_______________________________________________
ECM mailing list
ECMlists.nuxeo.com
http://li
sts.nuxeo.com/mailman/listinfo/ecm