Marking bugs as fixed in the test suite

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Marking bugs as fixed in the test suite

John W. Eaton
Administrator
About a month ago, I made the following change so that tests for
reported bugs could be tagged as fixed in the test suite by prefixing
the bug number with "*".

   http://hg.savannah.gnu.org/hgweb/octave/rev/b6144e6dda9e

With this change, any failing tests that are marked this way are shown
in the test suite summary as regressions.  Previously they were just
noted as "expected failures" or "reported bugs" so it was easy to miss
that a change actually reintroduced some bug that had been fixed.

I also added a script and Makefile target ("update-bug-status") that
attempts to automatically tag tests with bug numbers as fixed:

   http://hg.savannah.gnu.org/hgweb/octave/rev/0703d31e1ec0

Around the same time, I added bug numbers for as many xtests as I could.
  There are just a few left that don't have bug numbers.  Any help with
adding bug bug numbers for those would be much appreciated.  If bug
reports don't already exist, they should be created.  I would have done
it, but I don't really understand the purpose of the remaining xtests.
Maybe someone else could write better reports and I could.

Finally, there are now 4 tests that are always failing and that are
marked as regressions.  This is happening because the bug report is
closed as fixed, but there are multiple parts to the report:

   http://octave.org/testfailure/?50893

The tests that fail are all items that have been marked as "won't fix".
So, what should we do?  Open a separate bug report that won't be closed
as fixed?  Add another marker for the tests to indicate "won't fix"?

jwe


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Marking bugs as fixed in the test suite

Mike Miller-4
On Fri, Jul 07, 2017 at 14:27:39 -0400, John W. Eaton wrote:
> With this change, any failing tests that are marked this way are shown in
> the test suite summary as regressions.  Previously they were just noted as
> "expected failures" or "reported bugs" so it was easy to miss that a change
> actually reintroduced some bug that had been fixed.

I think it would be helpful to make these labels a little "louder".

With the current labeling they are still somewhat lost in the noise of
known failures, while they should be much more important. Instead of

Summary:

  PASS                            15422
  FAIL                                0
  XFAIL (expected failure)            5
  XFAIL (reported bug)               31
  XFAIL (regression)                  4
  SKIPPED (feature)                  50

maybe something like this instead

Summary:

  PASS                            15422
  FAIL                                0
  REGRESSION                          4
  XFAIL (expected failure)            5
  XFAIL (reported bug)               31
  SKIPPED (feature)                  50

> Finally, there are now 4 tests that are always failing and that are marked
> as regressions.  This is happening because the bug report is closed as
> fixed, but there are multiple parts to the report:
>
>   http://octave.org/testfailure/?50893
>
> The tests that fail are all items that have been marked as "won't fix". So,
> what should we do?  Open a separate bug report that won't be closed as
> fixed?  Add another marker for the tests to indicate "won't fix"?

I agree here. I went back to revision fd7a16594614 which resolved that
bug report, and the tests that were added did not pass then either. So
they were added at the time as cases that Octave did not pass. Maybe the
second asserts for each of these four tests were just typos?

This may not have been the case, but I don't think we want to add xtests
for things that we are intentionally not fixing. A better way would be
to test for what the value that Octave produces is, rather than what
Matlab does that we don't match. And maybe a FIXME or other comment
indicating why this test tests for something that is not compatible with
Matlab.

--
mike

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Marking bugs as fixed in the test suite

John W. Eaton
Administrator
On 07/07/2017 03:27 PM, Mike Miller wrote:

> I think it would be helpful to make these labels a little "louder".

That seems like a good idea to me.  I checked in the following change:

http://hg.savannah.gnu.org/hgweb/octave/rev/afe45e9ff6b0

> This may not have been the case, but I don't think we want to add xtests
> for things that we are intentionally not fixing.

I'm not sure what's best, but I agree we shouldn't add tests that we
expect will always fail.

jwe


Loading...