On 4/18/07, Steve Hay <steve.hay uk.radan.com> wrote:
> demerphq wrote:
> > On 4/18/07, Steve Hay <steve.hay uk.radan.com> wrote:
> >> demerphq wrote:
> >> > On 4/17/07, Steve Hay <steve.hay uk.radan.com> wrote:
> >> >> Jerry D. Hedden wrote:
> >> >> > Jerry D. Hedden wrote:
> >> >> >> Also, should the test be
skipped/TODOed under Win32 until
> >> >> >> the problem is fixed?
> >> >> >
> >> >> > Yves wrote::
> >> >> >> We should todo it, although
maybe only under smoke testing.
> >> >> >>
> >> >> >> Use an environment variable.
Something like
> >> >> >> WIN32_SMOKE_SKIP_THREADS or
something. People who need it
> >> >> >> can set it up in a wrapper
script.
> >> >> >
> >> >> > Patch attached. If
$ENV{'WIN32_SMOKE'} is true, then the
> >> >> > offending tests will be set
TODO.
> >> >>
> >> >> I'm not sure that it makes sense to
only mark a test as TODO when run
> >> >> under Test-Smoke.
> >> >>
> >> >> Surely any test that is known to fail
should be marked as TODO
> >> >> regardless of how the test is being
run.
> >> >
> >> > My feeling is that we should assertively
test as much as we can, and
> >> > since the test is known not to fail from
the command line on Win32 I
> >> > dont think it should be TODO in that
case.
> >>
> >> I agree with your comments about handling
tests that fail under
> >> Test-Smoke but not interactively, but this is
not such a case for me (at
> >> least): I just tried manually running the test
half a dozen times and
> >> here's the results:
> >>
> >> Failed 29-50
> >> OK
> >> Failed 38-50
> >> OK
> >> Failed 36-50
> >> Failed 36-50
> >>
> >> With this level of failure, even
interactively, I think the test should
> >> be marked TODO.
> >>
> >> Are you saying that you don't see any failures
if you repeatedly run the
> >> test interactively?
> >
> > I have never seen the stress test fail from a
command line. Never.
> >
> > BUT, I havent tried running it over and over
either.
>
> Weird. I only ran it over and over to see with what
sort of frequency it
> fails, but I've often seen it fail when running the
whole "nmake test"
> test suite interactively.
>
> So we're agreed that we don't want FAIL reports from
Test-Smoke over
> this issue, but we're not sure what's best for
interactive users: either
> some users are going to get TODO tests unexpectedly
passing, or else
> other users are going to get normal tests unexpectedly
failing.
>
> I would say that an unexpected pass is less worrying
that an unexpected
> failure, so I'd still choose to make the test TODO, but
I admit I'm
> biased
That you see errors on the command line invalidates my
original
argument, so I agree, it should be TODO always.
Yves
--
perl -Mre=debug -e "/just|another|perl|hacker/"
|