help needed to tag (x)failing tests with bug numbers

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

help needed to tag (x)failing tests with bug numbers

John W. Eaton
Administrator
I checked in the following changeset yesterday:

   http://hg.savannah.gnu.org/hgweb/octave/rev/3473246a824e

This change allows %!assert, %!fail, %!test, %!testif, and %!xtest
blocks to be tagged with a "<MESSAGE>".  If MESSAGE is just an integer
it is recognized as an Octave bug report number.  Then failing tests
that are tagged this way will display the following info in the test log:

   !!!!! known bug: http://octave.org/testfailure/?NNNNN

Otherwise, if a test is tagged with <MESSAGE>, the output will be

   !!!!! known bug: MESSAGE

So it makes sense for MESSAGE to be a URL if it is not an Octave bug number.

Either way, I hope that this will prevent people from submitting
duplicate reports about these problems.

I tagged a few of the failing tests with bug numbers with this
additional changeset:

   http://hg.savannah.gnu.org/hgweb/octave/rev/c0f446d657bf

Note that %!test with a message behaves the same as %!xtest, so I
converted the %!xtest instances to %!test.

Any tests that were originally using %!xtest but that are now fixed
should be switched to using %!test now as well.  My thought is that any
failure we know about should link to a bug report.  Otherwise, the test
should be expected to succeed.

Now I need help to tag the rest or to create bug reports about these
failures if there isn't anything about them in the tracker.

Thanks,

jwe

Reply | Threaded
Open this post in threaded view
|

Re: help needed to tag (x)failing tests with bug numbers

Colin Macdonald-2
On 25/08/16 05:39, John W. Eaton wrote:
> Note that %!test with a message behaves the same as %!xtest, so I
> converted the %!xtest instances to %!test.

I like the extra information in that "x".  (The pattern "%!test
<BUG_ID>" does not inform me that is expected to fail b/c for example
your second change adds a bug id to at least one test which is currently
passing.)

In principle, keeping the "x" would allow us to modify our tooling so
that passing an xtest is fatal.  This is how the Doctest pkg behaves.

Anyway, I assume packages can keep using xtest.

best,
Colin

Reply | Threaded
Open this post in threaded view
|

Re: help needed to tag (x)failing tests with bug numbers

John W. Eaton
Administrator
On 08/25/2016 01:16 PM, Colin Macdonald wrote:
> On 25/08/16 05:39, John W. Eaton wrote:
>> Note that %!test with a message behaves the same as %!xtest, so I
>> converted the %!xtest instances to %!test.
>
> I like the extra information in that "x".  (The pattern "%!test
> <BUG_ID>" does not inform me that is expected to fail b/c for example
> your second change adds a bug id to at least one test which is currently
> passing.)

Calling these tests "expected failures" was probably the wrong choice.
These tests are not really "expected to fail".  The intent was more
like, "this test is known to fail but we don't want to alarm you by
saying that there is a failure here".  But there IS a failure and it
should be fixed.  Unfortunately, what happens when you call it an
"expected failure" instead of a "known bug" is that people ignore it and
don't even think that it is something that should be examined or fixed.

When running "make check" the tests with bug ids attached to them will
still be reported as XFAIL.  So there is no change there.

It's fine to have bug ids for tests that pass.  That just means that the
bug is fixed and the bug id can be ignored.  But if the test fails
again, the old bug id will be reported.  The failure might not be due to
exactly the same problem, but maybe it will help.  Existing tests that
begin to fail are usually noticed fairly quickly so I'm not too worried
that the messages point to old bug reports.  These bug report ids can
also give some clues about why a test was added.  It probably wouldn't
be a bad idea to add the bug id to new tests that are added because of a
bug report.

> In principle, keeping the "x" would allow us to modify our tooling so
> that passing an xtest is fatal.  This is how the Doctest pkg behaves.

For that case maybe you really want to use "%!fail"?

> Anyway, I assume packages can keep using xtest.

For now it's not going away.  I'd just like to discourage its use.

jwe



Reply | Threaded
Open this post in threaded view
|

Re: help needed to tag (x)failing tests with bug numbers

Olaf Till-2
On Thu, Aug 25, 2016 at 01:45:34PM -0400, John W. Eaton wrote:
> Existing tests that begin to fail are
> usually noticed fairly quickly so I'm not too worried that the messages
> point to old bug reports.

But you potentially converge different information into one flagging
mechanism (original bug report and 'there is a current bug report')
... If I see the message 'statelessly', I can't say if the bug report
it points to is current or stale. But, as I understand it, it was your
intention that the message makes immediately clear that there already
is a current bug report. Keeping the 'x'test could perhaps resolve
this ambiguity, if it is only used for tests with a _current_ bug
report? This would mean, however, that flagged tests without 'x'
should print the message in a different way (may be 'original bug
report was: ...').

Olaf

--
public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net

signature.asc (836 bytes) Download Attachment