Removing XFAILs from test suite summary

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Removing XFAILs from test suite summary

Rik-4
When I run the test suite, I currently get

Summary:

  PASS                            14855
  FAIL                                0
  XFAIL (reported bug)               31
  XFAIL (expected failure)            5
  SKIP (missing feature)            138
  SKIP (run-time condition)          12

There are 5 reported XFAILs.  XFAILs are supposed to be known failures,
i.e., tests that the programmer has put in temporarily while writing code
and which are expected to pass as soon as the code is actually viable.  It
really is more of a development feature, and I don't think we need to ship
code that has known failures like this.

My suggestion would be that we file a bug report for each remaining XFAIL
and then associate that bug report with the BIST tests.  This would at
least give us a place to track these things.  As it is, nobody ever really
reviews these.  At least with a bug report filed on Savannah there would be
a record of it.

--Rik
Reply | Threaded
Open this post in threaded view
|

Re: Removing XFAILs from test suite summary

John W. Eaton
Administrator
On 04/11/2018 01:54 PM, Rik wrote:

> When I run the test suite, I currently get
>
> Summary:
>
>    PASS                            14855
>    FAIL                                0
>    XFAIL (reported bug)               31
>    XFAIL (expected failure)            5
>    SKIP (missing feature)            138
>    SKIP (run-time condition)          12
>
> There are 5 reported XFAILs.  XFAILs are supposed to be known failures,
> i.e., tests that the programmer has put in temporarily while writing code
> and which are expected to pass as soon as the code is actually viable.  It
> really is more of a development feature, and I don't think we need to ship
> code that has known failures like this.
>
> My suggestion would be that we file a bug report for each remaining XFAIL
> and then associate that bug report with the BIST tests.  This would at
> least give us a place to track these things.  As it is, nobody ever really
> reviews these.  At least with a bug report filed on Savannah there would be
> a record of it.

Yes, I agree that all XFAILS should have bug report numbers associated
with them.

Is it possible that those five already have bug reports but the numbers
were never recorded?  If not, then yes, we should create reports for them.

jwe


Reply | Threaded
Open this post in threaded view
|

Re: Removing XFAILs from test suite summary

Mike Miller-4
On Wed, Apr 11, 2018 at 14:03:12 -0400, John W. Eaton wrote:
> On 04/11/2018 01:54 PM, Rik wrote:
> > My suggestion would be that we file a bug report for each remaining XFAIL
> > and then associate that bug report with the BIST tests.  This would at
> > least give us a place to track these things.  As it is, nobody ever really
> > reviews these.  At least with a bug report filed on Savannah there would be
> > a record of it.
>
> Yes, I agree that all XFAILS should have bug report numbers associated with
> them.

Sounds good to me too.

However, there will always be one xtest without a bug number, because
the test function itself tests the xtest keyword.

> Is it possible that those five already have bug reports but the numbers were
> never recorded?  If not, then yes, we should create reports for them.

On my system,

  1 in gammainc.m
  1 in test.m
  1 in classdef.tst
  2 in jit.tst

There are no open bug reports about gammainc. I haven't looked into the
specifics of the classdef and jit test failures.

With grep, I also see several other %!xtests without bug numbers that
aren't failing on my system, but maybe fail on others, someone might
want to look into bug numbers for those as well.

--
mike

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Removing XFAILs from test suite summary

John W. Eaton
Administrator
On 04/11/2018 02:11 PM, Mike Miller wrote:

> However, there will always be one xtest without a bug number, because
> the test function itself tests the xtest keyword.

Ha, I wonder whether that test is really necessary.

> With grep, I also see several other %!xtests without bug numbers that
> aren't failing on my system, but maybe fail on others, someone might
> want to look into bug numbers for those as well.

If it's easy to find reports for those, then I'd say add the numbers.
Otherwise, we should probably just convert them to plain %!test blocks.
Those may have been tests that failed for a significant period of time
so we made them xtests so it would be easy to see whether there were any
new failing tests.  I'm guessing those would have been from a time
before we handled bug numbers in the test suite.

jwe



Reply | Threaded
Open this post in threaded view
|

Re: Removing XFAILs from test suite summary

Rik-4
On 04/11/2018 11:20 AM, John W. Eaton wrote:
> On 04/11/2018 02:11 PM, Mike Miller wrote:
>
>> However, there will always be one xtest without a bug number, because
>> the test function itself tests the xtest keyword.
>
> Ha, I wonder whether that test is really necessary.

Yes.  We need to test that a %!xtest block does not emit an error when the
test code passes, AND that it does emit an error when the test code fails. 
In other words, both sides of the coin.  I just created a bug report to
document this (https://savannah.gnu.org/bugs/index.php?53613) and then
promptly closed it.  Now I have a bug number that I can use to silence the
reporting.

>
>> With grep, I also see several other %!xtests without bug numbers that
>> aren't failing on my system, but maybe fail on others, someone might
>> want to look into bug numbers for those as well.
>
> If it's easy to find reports for those, then I'd say add the numbers.
> Otherwise, we should probably just convert them to plain %!test blocks.
> Those may have been tests that failed for a significant period of time so
> we made them xtests so it would be easy to see whether there were any new
> failing tests.  I'm guessing those would have been from a time before we
> handled bug numbers in the test suite.
>

Probably a good idea.  I took care of creating bug IDs and inserting them
into the xtests for the 5 failing xtests on my system
(http://hg.savannah.gnu.org/hgweb/octave/rev/b72972ab83f1).

--Rik

Reply | Threaded
Open this post in threaded view
|

Re: Removing XFAILs from test suite summary

Mike Miller-4
In reply to this post by John W. Eaton
On Wed, Apr 11, 2018 at 14:20:02 -0400, John W. Eaton wrote:
> If it's easy to find reports for those, then I'd say add the numbers.
> Otherwise, we should probably just convert them to plain %!test blocks.

There are six remaining bare xtests, not including the one in test.m
itself.

The only tests in tar.m and zip.m are xtest, presumably because the host
system may or may not have the tar, zip, or unzip programs installed.
These could probably be changed into testif with a runtime condition to
look for tar{,.exe}, unzip{,.exe}, and zip{,.exe}.

There is a xtest in fminsearch.m with a long comment about how the test
may fail. It was added as xtest

  https://hg.savannah.gnu.org/hgweb/octave/rev/9241a0fa7873#l1.376

There is a xtest in clf.m having to do with papertype, maybe Rik
remembers why this was added as xtest

  https://hg.savannah.gnu.org/hgweb/octave/rev/498b9f62199a

There are two xtests in speed.m that say they are known to fail on
systems with low resolution timer functions such as MinGW. The comment
suggests adding more work to make the test run longer so it won't fail.
Can someone test these on a current mxe-octave build to see if this is
still a problem?

--
mike

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Removing XFAILs from test suite summary

Rik-4
On 04/12/2018 11:13 AM, Mike Miller wrote:
> On Wed, Apr 11, 2018 at 14:20:02 -0400, John W. Eaton wrote:
>> If it's easy to find reports for those, then I'd say add the numbers.
>> Otherwise, we should probably just convert them to plain %!test blocks.
> There are six remaining bare xtests, not including the one in test.m
> itself.

I took care of that in 25216:b72972ab83f1.

>
> The only tests in tar.m and zip.m are xtest, presumably because the host
> system may or may not have the tar, zip, or unzip programs installed.
> These could probably be changed into testif with a runtime condition to
> look for tar{,.exe}, unzip{,.exe}, and zip{,.exe}.
>
> There is a xtest in fminsearch.m with a long comment about how the test
> may fail. It was added as xtest
>
>   https://hg.savannah.gnu.org/hgweb/octave/rev/9241a0fa7873#l1.376
>
> There is a xtest in clf.m having to do with papertype, maybe Rik
> remembers why this was added as xtest
>
>   https://hg.savannah.gnu.org/hgweb/octave/rev/498b9f62199a

Alas, no memory of this.  I just changed this to an ordinary %!test and ran

for i = 1:1000
  bm(i) = test ('clf');
end
sum (bm)
ans = 1000

Why don't you just go for it and change it over to an ordinary %!test and
we will see if it fails on other platforms.

--Rik
> There are two xtests in speed.m that say they are known to fail on
> systems with low resolution timer functions such as MinGW. The comment
> suggests adding more work to make the test run longer so it won't fail.
> Can someone test these on a current mxe-octave build to see if this is
> still a problem?
>


Reply | Threaded
Open this post in threaded view
|

Re: Removing XFAILs from test suite summary

John W. Eaton
Administrator
In reply to this post by Mike Miller-4
On 04/12/2018 02:13 PM, Mike Miller wrote:

> There are two xtests in speed.m that say they are known to fail on
> systems with low resolution timer functions such as MinGW. The comment
> suggests adding more work to make the test run longer so it won't fail.
> Can someone test these on a current mxe-octave build to see if this is
> still a problem?

I ran __run_test_suite__ for the Windows build of 4.3.90 this morning on
a Windows 10 system running on fairly fast hardware.  It passed 4/4 tests.

jwe

Reply | Threaded
Open this post in threaded view
|

Re: Removing XFAILs from test suite summary

Mike Miller-4
In reply to this post by Rik-4
On Thu, Apr 12, 2018 at 11:39:24 -0700, Rik wrote:
> Why don't you just go for it and change it over to an ordinary %!test and
> we will see if it fails on other platforms.

Ok, I changed all of these to %!test on stable with some comments in
case of failures

  https://hg.savannah.gnu.org/hgweb/octave/rev/e1024058833c

--
mike

signature.asc (849 bytes) Download Attachment