AS Integration tests in JBC test suites

I'd like to propose adding a package structure in the core
cache and 
pojo cache testsuites for adding tests of compliance with
the behavior 
the AS and Hibernate expect from JBC. Basically, in these
packages the 
AS developers would write tests that simulate the AS and
Hibernate usage 
of JBC. Simulations would *not* add any new dependencies. 
Goal over 
time is to 99.5% eliminate the chances of finding JBC-caused
regressions 
in the AS/Hibernate testsuites by ensuring they are caught
in JBC itself.

Right now a lot of issues slip through because the JBC test
suites have 
a hard time covering all of its permutations of features. 
Something 
gets changed and a test gets added, but it doesn't cover the

configuration set AS/Hibernate uses, so we get a regression
at the AS level.

Downside to this is theoretically everything we add in these
new 
packages should be redundant. But in practice that will be
hard to achieve.

Comments?

-- 
Brian Stansberry
Lead, AS Clustering
JBoss, a division of Red Hat
brian.stansberryredhat.com
_______________________________________________
jbosscache-dev mailing list
jbosscache-devlists.jboss.org
https://lists.jboss.org/mailman/listinfo/jbosscache-dev

+1.

How do you foresee this, as a separate suite of tests? 
E.g., src/ 
integration-test/java?  And how about integration with
Hudson?  Run on  
every build?  Every release?

On 18 Mar 2008, at 17:26, Brian Stansberry wrote:
> I'd like to propose adding a package structure in the
core cache and  
> pojo cache testsuites for adding tests of compliance
with the  
> behavior the AS and Hibernate expect from JBC.
Basically, in these  
> packages the AS developers would write tests that
simulate the AS  
> and Hibernate usage of JBC. Simulations would *not* add
any new  
> dependencies.  Goal over time is to 99.5% eliminate the
chances of  
> finding JBC-caused regressions in the AS/Hibernate
testsuites by  
> ensuring they are caught in JBC itself.
>
> Right now a lot of issues slip through because the JBC
test suites  
> have a hard time covering all of its permutations of
features.   
> Something gets changed and a test gets added, but it
doesn't cover  
> the configuration set AS/Hibernate uses, so we get a
regression at  
> the AS level.
>
> Downside to this is theoretically everything we add in
these new  
> packages should be redundant. But in practice that will
be hard to  
> achieve.
>

When you say redundant, do you mean duplication between the
AS/ 
Hibernate test suite and JBC's test suite (+1 in this case),
or  
duplication between JBC's unit test suite and integration
test suite  
(-1 in this case)?

Cheers
Manik
--
Manik Surtani
Lead, JBoss Cache
manikjboss.org






_______________________________________________
jbosscache-dev mailing list
jbosscache-devlists.jboss.org
https://lists.jboss.org/mailman/listinfo/jbosscache-dev

Manik Surtani wrote:
> +1.
> 
> How do you foresee this, as a separate suite of tests? 
E.g., 
> src/integration-test/java?  And how about integration
with Hudson?  Run 
> on every build?  Every release?
> 

Hadn't really thought that far. 

I *was* thinking in terms of just a separate package (with
subpackages) 
under src/test/java. Say org.jboss.cache.integration or
something. But, 
yeah, there's no need to run these every build, just before
release, so 
lets organize in whatever way makes it easy to control when
they get run.

Re: hudson, since we don't want them run on every build, 
IMO it would 
be good to have a separate, unscheduled, hudson job to run
w/ them 
included.  They need to run before release, but we should
also make it 
easy for a dev to kick off a run that includes them so
problems can be 
caught early.

> On 18 Mar 2008, at 17:26, Brian Stansberry wrote:
>> I'd like to propose adding a package structure in
the core cache and 
>> pojo cache testsuites for adding tests of
compliance with the behavior 
>> the AS and Hibernate expect from JBC. Basically, in
these packages the 
>> AS developers would write tests that simulate the
AS and Hibernate 
>> usage of JBC. Simulations would *not* add any new
dependencies.  Goal 
>> over time is to 99.5% eliminate the chances of
finding JBC-caused 
>> regressions in the AS/Hibernate testsuites by
ensuring they are caught 
>> in JBC itself.
>>
>> Right now a lot of issues slip through because the
JBC test suites 
>> have a hard time covering all of its permutations
of features.  
>> Something gets changed and a test gets added, but
it doesn't cover the 
>> configuration set AS/Hibernate uses, so we get a
regression at the AS 
>> level.
>>
>> Downside to this is theoretically everything we add
in these new 
>> packages should be redundant. But in practice that
will be hard to 
>> achieve.
>>
> 
> When you say redundant, do you mean duplication between
the AS/Hibernate 
> test suite and JBC's test suite (+1 in this case), or
duplication 
> between JBC's unit test suite and integration test
suite (-1 in this case)?
> 

Meant both, although I was really talking about the latter.
The idea 
here is these integration tests set up caches configured
that match what 
AS/Hib uses and then exercise the call patterns that AS/Hib
makes.  In a 
perfect world, those call patterns are exercised with those

configuration options somewhere else in the JBC testsuite. 
But the 
world isn't perfect; there are too many permutations.

Part of the tradeoff here is Paul and I will be working to
expand the 
JBC suite and covering code paths not otherwise covered. But
when we 
make this effort we just think "here's the config,
here's the call 
pattern."  We don't spend time digging through all the
hundreds of tests 
in the main testsuite looking for redundancy.

Of course, if one of these tests surfaces a bug, a test in
the regular 
testsuite showing the bug should be added.

-- 
Brian Stansberry
Lead, AS Clustering
JBoss, a division of Red Hat
brian.stansberryredhat.com
_______________________________________________
jbosscache-dev mailing list
jbosscache-devlists.jboss.org
https://lists.jboss.org/mailman/listinfo/jbosscache-dev