Re: AS Integration tests in JBC test suites

Ok, let's do this in the main test set for now, and use
TestNG's  
grouping to separate.

E.g., put the integration tests in src/test/java, in a new
package  
(org.jboss.cache.integration ?) and annotate tests as being
in the  
"integration" group.  They can be run by passing
in the necessary  
parameters to mvn according to README-Maven.txt [1] (see
section 2.3)  
and we can set up a separate Hudson run to run these
integration tests.

- Manik

[1] http://anonsvn.jboss.org/repos/jbos
scache/core/tags/2.1.0.GA/README-Maven.txt

On 24 Mar 2008, at 18:51, Brian Stansberry wrote:
> Mircea Markus wrote:
>> Brian Stansberry wrote:
>>> Manik Surtani wrote:
>>>> +1.
>>>>
>>>> How do you foresee this, as a separate
suite of tests?  E.g., src/ 
>>>> integration-test/java?  And how about
integration with Hudson?   
>>>> Run on every build?  Every release?
>>>>
>>>
>>> Hadn't really thought that far. 
>>>
>>> I *was* thinking in terms of just a separate
package (with  
>>> subpackages) under src/test/java. Say
org.jboss.cache.integration  
>>> or something. But, yeah, there's no need to run
these every build,  
>>> just before release, so lets organize in
whatever way makes it  
>>> easy to control when they get run.
>>>
>> Do we expect these tests to perform slow?
>
> Maybe a bit. The general AS use cases involve things
like failover  
> after  async repl, so you need to pause a bit to be
sure the repl  
> had time to happen. Also passivation, where there is
some pausing to  
> allow passivation timeouts to happen. Stuff like that.
>
>> If not, I guess running them together with the
common test suite  
>> would only bring an additional safety net... I also
think that it  
>> worths having them in an different package as they
will mainly be  
>> written by the teams that needs integration (Brian
I think  )
>
> For sure we want a distinct package.   Gotta
ensure we cover all  
> the use cases; if tests are spread in random places
it's too hard to  
> track whether we did that or not.
>
>>> Re: hudson, since we don't want them run on
every build,  IMO it  
>>> would be good to have a separate, unscheduled,
hudson job to run  
>>> w/ them included.  They need to run before
release, but we should  
>>> also make it easy for a dev to kick off a run
that includes them  
>>> so problems can be caught early.
>>>
>>>> On 18 Mar 2008, at 17:26, Brian Stansberry
wrote:
>>>>> I'd like to propose adding a package
structure in the core cache  
>>>>> and pojo cache testsuites for adding
tests of compliance with  
>>>>> the behavior the AS and Hibernate
expect from JBC. Basically, in  
>>>>> these packages the AS developers would
write tests that simulate  
>>>>> the AS and Hibernate usage of JBC.
Simulations would *not* add  
>>>>> any new dependencies.  Goal over time
is to 99.5% eliminate the  
>>>>> chances of finding JBC-caused
regressions in the AS/Hibernate  
>>>>> testsuites by ensuring they are caught
in JBC itself.
>>>>>
>>>>> Right now a lot of issues slip through
because the JBC test  
>>>>> suites have a hard time covering all of
its permutations of  
>>>>> features.  Something gets changed and a
test gets added, but it  
>>>>> doesn't cover the configuration set
AS/Hibernate uses, so we get  
>>>>> a regression at the AS level.
>>>>>
>>>>> Downside to this is theoretically
everything we add in these new  
>>>>> packages should be redundant. But in
practice that will be hard  
>>>>> to achieve.
>>>>>
>>>>
>>>> When you say redundant, do you mean
duplication between the AS/ 
>>>> Hibernate test suite and JBC's test suite
(+1 in this case), or  
>>>> duplication between JBC's unit test suite
and integration test  
>>>> suite (-1 in this case)?
>>>>
>>>
>>> Meant both, although I was really talking about
the latter. The  
>>> idea here is these integration tests set up
caches configured that  
>>> match what AS/Hib uses and then exercise the
call patterns that AS/ 
>>> Hib makes.  In a perfect world, those call
patterns are exercised  
>>> with those configuration options somewhere else
in the JBC  
>>> testsuite.  But the world isn't perfect; there
are too many  
>>> permutations.
>>>
>>> Part of the tradeoff here is Paul and I will be
working to expand  
>>> the JBC suite and covering code paths not
otherwise covered. But  
>>> when we make this effort we just think
"here's the config, here's  
>>> the call pattern."  We don't spend time
digging through all the  
>>> hundreds of tests in the main testsuite looking
for redundancy.
>>>
>>> Of course, if one of these tests surfaces a
bug, a test in the  
>>> regular testsuite showing the bug should be
added.
>>>
>
> -- 
> Brian Stansberry
> Lead, AS Clustering
> JBoss, a division of Red Hat
> brian.stansberryredhat.com

--
Manik Surtani
Lead, JBoss Cache
manikjboss.org






_______________________________________________
jbosscache-dev mailing list
jbosscache-devlists.jboss.org
https://lists.jboss.org/mailman/listinfo/jbosscache-dev