Baselining Performance on Cosmo

One of the things that we would like to do in the short term
is to  
put in place a small set of performance tests that measure
basic  
operations that clients would do against Cosmo.

Once we have these tests in place, we would perform baseline
 
measurements in a controlled setting. With each release, we
would run  
these tests again to make sure that our performance in the
basic set  
of operations hasn't slipped.

To get things started, Brian Moseley and I brainstormed last
week  
about the "Top 5" operations that we should
consider testing. Here's  
that list:

- FoxMarks usage
   - GET a regular file (WebDAV)
   - PUT a regular file (WebDAV)
   - signup via CMP

- Chandler usage
   - MKCALENDAR (CalDAV)
   - MKTICKET on that calendar (CalDAV)
   - PUT events into calendar (CalDAV)
   - Syncing
      - PROPFIND on calendar (both large and small calendar)
(CalDAV)
      - GET on resource
      - PUT on same resource

I don't have basic Cosmo usages for Scooby. Bobby, can you
help me  
out with that? Does anyone have anything else we should be
measuring  
regularly.

The plan is that once we have a final reviewed list, our QA
team can  
go to work creating these tests and measuring these
operations with  
each RC candidate that we give them. We should also be able
to run  
these tests in our development environments as well.

I am posting this now and planning to do a Last Call on it
on Friday  
morning. I hope to have a final list we can hand over to QA
on Monday.

Thanks,
--> towns

_______________________________________________
scooby-dev mailing list
scooby-devlists.osafoundation.org
http://lists.osafoundation.org/cgi-bin/mailman
/listinfo/scooby-dev

I have a bunch of suggestions for feedback and additional
tests in  
line below.

-Mikeal

On Jun 14, 2006, at 1:42 PM, John Townsend wrote:

> One of the things that we would like to do in the short
term is to  
> put in place a small set of performance tests that
measure basic  
> operations that clients would do against Cosmo.
>
> Once we have these tests in place, we would perform
baseline  
> measurements in a controlled setting. With each
release, we would  
> run these tests again to make sure that our performance
in the  
> basic set of operations hasn't slipped.
>
> To get things started, Brian Moseley and I brainstormed
last week  
> about the "Top 5" operations that we should
consider testing.  
> Here's that list:
>
> - FoxMarks usage
>   - GET a regular file (WebDAV)
>   - PUT a regular file (WebDAV)
>   - signup via CMP
>
> - Chandler usage
>   - MKCALENDAR (CalDAV)
>   - MKTICKET on that calendar (CalDAV)

Do we want to MKTICKET on an empty calendar?

>   - PUT events into calendar (CalDAV)

How many? what type (recurring?), over what time frame &
how often in  
that timeframe?
Multiple ics files for each event?, one large ics file for
all events?

>   - Syncing
>      - PROPFIND on calendar (both large and small
calendar) (CalDAV)

How large? How small? What kind of events, see above for
examples.

>      - GET on resource

What is in the resource?

>      - PUT on same resource

How large is the resource already.

REPORT?
Depth?

>
> I don't have basic Cosmo usages for Scooby. Bobby, can
you help me  
> out with that? Does anyone have anything else we should
be  
> measuring regularly.
>
> The plan is that once we have a final reviewed list,
our QA team  
> can go to work creating these tests and measuring these
operations  
> with each RC candidate that we give them. We should
also be able to  
> run these tests in our development environments as
well.

What hardware? What jvm environment?

How many requests per second? How long should PUT's take
after x  
amount of events added?

How many users are we sending simultaneous requests to?


>
> I am posting this now and planning to do a Last Call on
it on  
> Friday morning. I hope to have a final list we can hand
over to QA  
> on Monday.
>
> Thanks,
> --> towns
>
> _______________________________________________
> scooby-dev mailing list
> scooby-devlists.osafoundation.org
> http://lists.osafoundation.org/cgi-bin/mailman
/listinfo/scooby-dev

_______________________________________________
scooby-dev mailing list
scooby-devlists.osafoundation.org
http://lists.osafoundation.org/cgi-bin/mailman
/listinfo/scooby-dev

I have a bunch of suggestions for feedback and additional
tests in  
line below.

-Mikeal

On Jun 14, 2006, at 1:42 PM, John Townsend wrote:

> One of the things that we would like to do in the short
term is to  
> put in place a small set of performance tests that
measure basic  
> operations that clients would do against Cosmo.
>
> Once we have these tests in place, we would perform
baseline  
> measurements in a controlled setting. With each
release, we would  
> run these tests again to make sure that our performance
in the  
> basic set of operations hasn't slipped.
>
> To get things started, Brian Moseley and I brainstormed
last week  
> about the "Top 5" operations that we should
consider testing.  
> Here's that list:
>
> - FoxMarks usage
>   - GET a regular file (WebDAV)
>   - PUT a regular file (WebDAV)
>   - signup via CMP
>
> - Chandler usage
>   - MKCALENDAR (CalDAV)
>   - MKTICKET on that calendar (CalDAV)

Do we want to MKTICKET on an empty calendar?

>   - PUT events into calendar (CalDAV)

How many? what type (recurring?), over what time frame &
how often in  
that timeframe?
Multiple ics files for each event?, one large ics file for
all events?

>   - Syncing
>      - PROPFIND on calendar (both large and small
calendar) (CalDAV)

How large? How small? What kind of events, see above for
examples.

>      - GET on resource

What is in the resource?

>      - PUT on same resource

How large is the resource already.

REPORT?
Depth?

>
> I don't have basic Cosmo usages for Scooby. Bobby, can
you help me  
> out with that? Does anyone have anything else we should
be  
> measuring regularly.
>
> The plan is that once we have a final reviewed list,
our QA team  
> can go to work creating these tests and measuring these
operations  
> with each RC candidate that we give them. We should
also be able to  
> run these tests in our development environments as
well.

What hardware? What jvm environment?

How many requests per second? How long should PUT's take
after x  
amount of events added?

How many users are we sending simultaneous requests to?


>
> I am posting this now and planning to do a Last Call on
it on  
> Friday morning. I hope to have a final list we can hand
over to QA  
> on Monday.
>
> Thanks,
> --> towns
>
> _______________________________________________
> scooby-dev mailing list
> scooby-devlists.osafoundation.org
> http://lists.osafoundation.org/cgi-bin/mailman
/listinfo/scooby-dev

_______________________________________________
scooby-dev mailing list
scooby-devlists.osafoundation.org
http://lists.osafoundation.org/cgi-bin/mailman
/listinfo/scooby-dev