JBossCache 2.0 Beta2 test code coverage analysis

Hi all,

I have run clover once again. This time against the head
branch. I did
2 data points. One was against right before 2.0 Beta1 and
the newer one
was done yesterday just before 2.0 Beta2. I used the ant
target in the
build all-functionaltests to instrument and capture the
code. The first
data point was done on the test suite where the second data
point was
performed on the test suite and the source code to be able
to see how
much of the source code is actually exercised by the test
suite. The
first data point resulted in 91.8% code coverage where as
the second
data point resulted in 79.4%.

I also did a gap analysis on the second data point to
identify what
test cases and what source files need to be exercised better
to improve
the overall coverage. I'm targeting a jump from 79.4 to 90 %
before
2.0GA.

I invite you to  take a look at the following 2  wiki pages
for both
data points and see what classes and test cases need to
provide more
coverage: 
Data point 1 (pre beta1) :
http://wiki.jboss.org/wiki/Wiki.jsp?p
age=JBossCacheHeadCodeCoverageAnalysis
Data Point 2 (pre beta2) :
http://wiki.jboss.org/wiki/Wiki.js
p?page=JBossCache20Beta2CodeCoverageAnalysis


On the wiki pages, you'll see the log of my command line
test run
(worth noting that I had 14 test cases reported one or more
failure or
error). Also, you'll only see the overall summary graph
presented. If
you'd like to see the graph details, you can download the
detailed HTML
report attached to the wiki page (*_html.zip) and unzip to
drill down to
a line level of the test cases or classes.

Finally, I have worked a nice ant build targets for clover
that I'd
like to add to the build script if everyone is OK with that.
Those build
targets provide clover instrumentation and various reporting
formats.
They require adding clover.jar and clover.license to
ant-dist directory.
If you don't prefer adding the files for legal reason, I can
just add
the targets and they're manual targets anyway and should not
interfere
with the automated build but they will be there for future
data coverage
reporting.

Please let me know you comments and thoughts.

Cheers,

Hany





Hany M. Mesha
Sr. Software Engineer, Consultant
Novell Identity Management Engineering
Toronto, Canada
hmeshanovell.com
Mobile: 416-456-6945
skype: hanymesha

Novell, Inc.
SUSEŽ Linux Enterprise 10
Your Linux is ready
http://www.novell.com/lin
ux 



_______________________________________________
jbosscache-dev mailing list
jbosscache-devlists.jboss.org
https://lists.jboss.org/mailman/listinfo/jbosscache-dev

Hany Mesha wrote:
> Hi all,
> 
> I have run clover once again. This time against the
head branch. I did
> 2 data points. One was against right before 2.0 Beta1
and the newer one
> was done yesterday just before 2.0 Beta2. I used the
ant target in the
> build all-functionaltests to instrument and capture the
code. The first
> data point was done on the test suite where the second
data point was
> performed on the test suite and the source code to be
able to see how
> much of the source code is actually exercised by the
test suite. The
> first data point resulted in 91.8% code coverage where
as the second
> data point resulted in 79.4%.
> 
> I also did a gap analysis on the second data point to
identify what
> test cases and what source files need to be exercised
better to improve
> the overall coverage. I'm targeting a jump from 79.4 to
90 % before
> 2.0GA.
> 
> I invite you to  take a look at the following 2  wiki
pages for both
> data points and see what classes and test cases need to
provide more
> coverage: 
> Data point 1 (pre beta1) :
> http://wiki.jboss.org/wiki/Wiki.jsp?p
age=JBossCacheHeadCodeCoverageAnalysis
> Data Point 2 (pre beta2) :
> http://wiki.jboss.org/wiki/Wiki.js
p?page=JBossCache20Beta2CodeCoverageAnalysis
> 
> 
> On the wiki pages, you'll see the log of my command
line test run
> (worth noting that I had 14 test cases reported one or
more failure or
> error). Also, you'll only see the overall summary graph
presented. If
> you'd like to see the graph details, you can download
the detailed HTML
> report attached to the wiki page (*_html.zip) and unzip
to drill down to
> a line level of the test cases or classes.
> 
> Finally, I have worked a nice ant build targets for
clover that I'd
> like to add to the build script if everyone is OK with
that. Those build
> targets provide clover instrumentation and various
reporting formats.
> They require adding clover.jar and clover.license to
ant-dist directory.
> If you don't prefer adding the files for legal reason,
I can just add
> the targets and they're manual targets anyway and
should not interfere
> with the automated build but they will be there for
future data coverage
> reporting.
> 
> Please let me know you comments and thoughts.
> 
> Cheers,
> 
> Hany
> 
> 
> 
> 
> 
> Hany M. Mesha
> Sr. Software Engineer, Consultant
> Novell Identity Management Engineering
> Toronto, Canada
> hmeshanovell.com
> Mobile: 416-456-6945
> skype: hanymesha
> 
> Novell, Inc.
> SUSEŽ Linux Enterprise 10
> Your Linux is ready
> http://www.novell.com/lin
ux 
> 
> 
> 
> _______________________________________________
> jbosscache-dev mailing list
> jbosscache-devlists.jboss.org
> https://lists.jboss.org/mailman/listinfo/jbosscache-dev

Thanks for this, Hany.  This is a useful analysis (insofar
as code 
coverage can be considered a useful thing).  It does clearly
show areas 
where we do need better coverage (rmi cache loaders, JMX,
some pojo 
cache features).  Some of the "red" on the report
can be misleading 
though - such as o.j.c.marshall.data (which contain just
dummy classes 
for unit tests).

In terms of targets, I think we can only feasibly consider
improving the 
test suite after BETA2, so it would make good sense to run
the code 
coverage analysis again just before releasing CR1.

Cheers,
-- 
Manik Surtani
Project Lead, JBoss Cache

JBoss, a division of Red Hat

http://labs.jboss.co
m/jbosscache

Phone: +44 7786 702 706
Email: manikjboss.com
Skype/Yahoo/AOL: maniksurtani
IRC: manik
MSN: maniksurtani.org
--
_______________________________________________
jbosscache-dev mailing list
jbosscache-devlists.jboss.org
https://lists.jboss.org/mailman/listinfo/jbosscache-dev

Manik Surtani wrote:
> Hany Mesha wrote:
>> Hi all,
>>
>> I have run clover once again. This time against the
head branch. I did
>> 2 data points. One was against right before 2.0
Beta1 and the newer one
>> was done yesterday just before 2.0 Beta2. 
> 
> Thanks for this, Hany.  This is a useful analysis
(insofar as code 
> coverage can be considered a useful thing).  It does
clearly show areas 
> where we do need better coverage (rmi cache loaders,
JMX, some pojo 
> cache features).  Some of the "red" on the
report can be misleading 
> though - such as o.j.c.marshall.data (which contain
just dummy classes 
> for unit tests).
> 
> In terms of targets, I think we can only feasibly
consider improving the 
> test suite after BETA2, so it would make good sense to
run the code 
> coverage analysis again just before releasing CR1.
> 

+10 on the thanks for this, Hany.  I like the idea of having
targets in 
the build file too.

I apologize; I'm pretty sure we discussed this before, but I
can't find 
the thread.  Is it possible to execute these with only the
contents of 
src being included in the analysis?  In general I'm really
not 
interested in coverage of code in the tests directory.
Besides some 
spurious 'red', on balance it's likely including tests
overstates the 
'green' % since presumably in most cases a very high % of
test code gets 
executed.


-- 
Brian Stansberry
Lead, AS Clustering
JBoss, a division of Red Hat
brian.stansberryredhat.com

_______________________________________________
jbosscache-dev mailing list
jbosscache-devlists.jboss.org
https://lists.jboss.org/mailman/listinfo/jbosscache-dev