Re: Status report of testsuite on trunk @ r24830

Mark Phippard wrote:
> Lieven,
> 
> What would you think about adding some small stats
output to the
> existing suite?  Maybe change the output to something
like this:
> 
> Running all tests in compat-test [1/50]...success
[14/0/1/0/2]
> 
> Where the numbers are [PASS/FAIL/SKIPPED/XPASS/XFAIL]

No problem of adding this, but what is the added value?

We have XFAIL and XPASS so that we can explicitly mark tests
as failing
and even learn when they are unknowingly solved, it's not
like we accept
tests to fail for more than a few days.

AFAIC the tests markers should be matched exactly by the
actual results
at all times, so for statistical reasons all there is to
know can be
gathered by listing the tests without running them, which
was basically
what I did with a semi-automated python script.

> We are trying to get our QA team at CollabNet to use
the test suite
> and they are wanting information like this from the
output.  It does
> not necessarily matter how it comes out or is
formatted, the above was
> just a suggestion.

Is your QA team currently keeping a separate list of issues
or
testscripts? Are they interested in knowing the status of
certain
issues? Or rather the progress of feature implementation?
I'm interested to know what problem these extra stats are
going to solve.

Personally I'd like to have two more types of info from the
testsuite:
1. which tests are blocking the next release: This can be
solved by
adding a new marker BLOCK, but I'd rather keep a TODO-1.5
file somewhere
on trunk to maintain this list. Or we could use the issue
tracker, but
not all issues and features are reported there.

2. the code coverage % we reach when running the full test
suite,
including the lines of code which aren't covered by any
test.

Lieven

------------------------------------------------------------
---------
To unsubscribe, e-mail: dev-unsubscribesubversion.tigris.org
For additional commands, e-mail: dev-helpsubversion.tigris.org

On 4/30/07, Lieven Govaerts <svnlgomobsol.be> wrote:
> Mark Phippard wrote:
> > Lieven,
> >
> > What would you think about adding some small stats
output to the
> > existing suite?  Maybe change the output to
something like this:
> >
> > Running all tests in compat-test [1/50]...success
[14/0/1/0/2]
> >
> > Where the numbers are
[PASS/FAIL/SKIPPED/XPASS/XFAIL]
>
> No problem of adding this, but what is the added
value?

To be honest, I asked the same question and never felt I
understood
their reasons.  It seemed criticial to them to be offer an
accurate
accounting of the number of tests they ran though.  I was
trying to
say well if you know there are 500 tests and 2 failed and 4
skipped
you know that 494 passed.

> We have XFAIL and XPASS so that we can explicitly mark
tests as failing
> and even learn when they are unknowingly solved, it's
not like we accept
> tests to fail for more than a few days.

I was under the impression that XFAIL tests are occasionally
added
when a problem is recognied and will live in the code until
a fix is
made, which could be a long time.  I am less concerned with
listing
these other than to keep an accurate accounting.

> AFAIC the tests markers should be matched exactly by
the actual results
> at all times, so for statistical reasons all there is
to know can be
> gathered by listing the tests without running them,
which was basically
> what I did with a semi-automated python script.

How do you do this?  I was not aware there were methods
other than
running the tests.  Feel free to point me at some docs.  I
just know
make check.

> > We are trying to get our QA team at CollabNet to
use the test suite
> > and they are wanting information like this from
the output.  It does
> > not necessarily matter how it comes out or is
formatted, the above was
> > just a suggestion.
>
> Is your QA team currently keeping a separate list of
issues or
> testscripts? Are they interested in knowing the status
of certain
> issues? Or rather the progress of feature
implementation?
> I'm interested to know what problem these extra stats
are going to solve.

My team is not involved in the development process
specifically.  We
are certifying release builds that we are releasing on
openCollabNet.
So we would generally expect tests to pass unless there was
some
problem in our build process.  These sound like cool ideas
and
features though, especially the code coverage.

-- 
Thanks

Mark Phippard
http://markphip.blogspo
t.com/

------------------------------------------------------------
---------
To unsubscribe, e-mail: dev-unsubscribesubversion.tigris.org
For additional commands, e-mail: dev-helpsubversion.tigris.org