run BISTs for all installed packages

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

run BISTs for all installed packages

mmuetzel
I frequently cross-compile for Windows with all packages. But I very rarely actually test whether they work as expected.
Is there an easy way to run all BISTs of all installed packages?
Something that would be more streamlined than loading the packages one after the other and manually running "test" for all files.

Markus

Reply | Threaded
Open this post in threaded view
|

Re: run BISTs for all installed packages

apjanke-floss


On 1/21/19 11:44 AM, "Markus Mützel" wrote:
> I frequently cross-compile for Windows with all packages. But I very rarely actually test whether they work as expected.
> Is there an easy way to run all BISTs of all installed packages?
> Something that would be more streamlined than loading the packages one after the other and manually running "test" for all files.
>
> Markus
>

Here's a function to do this.

__test_pkgs__     ## test all installed packages
__test_pkgs__ control   ## test a single package
__test_pkgs__ ({'control','io'})    ## test just a list of specified
packages

I'm using this in my (new, experimental) jsonstuff package:
https://github.com/apjanke/octave-jsonstuff. You can have a look there
to see how to call it on multiple OSes in a Travis CI environment.
(Windows isn't working there yet, but I think it's close.) You can also
just run it interactively; it has no dependencies except for Octave itself.

The Windows Octave installers seem to come with a bunch of packages
preinstalled, so a plain __test_pkgs__ will end up testing a lot of
stuff there.

Currently, when I run it in Windows 10 version 1809 for all packages on
Octave 4.4.1, Octave crashes a few packages in. Also crashes on macOS
10.13.6 if I enable the "torture test" mode that tests each package
multiple times.

Cheers,
Andrew

__test_pkgs__.m (12K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: run BISTs for all installed packages

Ian McCallion
On Monday, 28 January 2019, Andrew Janke <[hidden email]> wrote:


On 1/21/19 11:44 AM, "Markus Mützel" wrote:
I frequently cross-compile for Windows with all packages. But I very rarely actually test whether they work as expected.
Is there an easy way to run all BISTs of all installed packages?
Something that would be more streamlined than loading the packages one after the other and manually running "test" for all files.

Markus


Here's a function to do this.

__test_pkgs__     ## test all installed packages
__test_pkgs__ control   ## test a single package
__test_pkgs__ ({'control','io'})    ## test just a list of specified packages

I'm using this in my (new, experimental) jsonstuff package: https://github.com/apjanke/octave-jsonstuff. You can have a look there to see how to call it on multiple OSes in a Travis CI environment. (Windows isn't working there yet, but I think it's close.) You can also just run it interactively; it has no dependencies except for Octave itself.

The Windows Octave installers seem to come with a bunch of packages preinstalled, so a plain __test_pkgs__ will end up testing a lot of stuff there.

Currently, when I run it in Windows 10 version 1809 for all packages on Octave 4.4.1, Octave crashes a few packages in. Also crashes on macOS 10.13.6 if I enable the "torture test" mode that tests each package multiple times.

I had a problem that no console output was appearing  with Octave when 1809 was installed on my laptop.  I switched to the legacy console and the problem went away.

Cheers... Ian
Reply | Threaded
Open this post in threaded view
|

Re: run BISTs for all installed packages

mmuetzel
In reply to this post by apjanke-floss
Am 28. Januar 2019 um 04:11 Uhr schrieb "Andrew Janke"
> On 1/21/19 11:44 AM, "Markus Mützel" wrote:
> > Is there an easy way to run all BISTs of all installed packages?
>
> Here's a function to do this.
>
> __test_pkgs__     ## test all installed packages
> __test_pkgs__ control   ## test a single package
> __test_pkgs__ ({'control','io'})    ## test just a list of specified
> packages

Thanks! IMHO that is pretty useful.
Do you think you could provide a patch that adds this to default Octave? Maybe by calling runtests directly.

> Currently, when I run it in Windows 10 version 1809 for all packages on
> Octave 4.4.1, Octave crashes a few packages in. Also crashes on macOS
> 10.13.6 if I enable the "torture test" mode that tests each package
> multiple times.

I've also seen this crash initially. But after 2 or 3 attempts, I can no longer reproduce the crash (surprisingly).

Markus

Reply | Threaded
Open this post in threaded view
|

Re: run BISTs for all installed packages

PhilipNienhuis
In reply to this post by mmuetzel
mmuetzel wrote
> I frequently cross-compile for Windows with all packages. But I very
> rarely actually test whether they work as expected.
> Is there an easy way to run all BISTs of all installed packages?
> Something that would be more streamlined than loading the packages one
> after the other and manually running "test" for all files.
>
> Markus

In the io package there's a "test_spsh" script + two helper functions that
checks all interfaces one by one, depending on presence of Java libs added
to the javaclasspath (+ on Windows, loaded windows package + MS-Office).

It's made that way to have consistent checks for all "interfaces", among
others. I suppose it should be possible to morph test-spsh.m into tests that
conform to Octave's regular test format but currently it serves my purpose
and I fear it would become quite messy to rewrite it.

Philip




--
Sent from: http://octave.1599824.n4.nabble.com/Octave-Maintainers-f1638794.html

Reply | Threaded
Open this post in threaded view
|

Re: run BISTs for all installed packages

jbect
In reply to this post by mmuetzel
Le 28/01/2019 à 20:04, "Markus Mützel" a écrit :
Am 28. Januar 2019 um 04:11 Uhr schrieb "Andrew Janke"
On 1/21/19 11:44 AM, "Markus Mützel" wrote:
Is there an easy way to run all BISTs of all installed packages?
Here's a function to do this.

__test_pkgs__     ## test all installed packages
__test_pkgs__ control   ## test a single package
__test_pkgs__ ({'control','io'})    ## test just a list of specified 
packages
Thanks! IMHO that is pretty useful.
Do you think you could provide a patch that adds this to default Octave? Maybe by calling runtests directly.

Some related work can be found on this ticket : http://savannah.gnu.org/bugs/?41215.

@++
Julien
Reply | Threaded
Open this post in threaded view
|

Re: run BISTs for all installed packages

apjanke-floss
In reply to this post by mmuetzel


On 1/28/19 2:04 PM, "Markus Mützel" wrote:

> Am 28. Januar 2019 um 04:11 Uhr schrieb "Andrew Janke"
>> On 1/21/19 11:44 AM, "Markus Mützel" wrote:
>>> Is there an easy way to run all BISTs of all installed packages?
>>
>> Here's a function to do this.
>>
>> __test_pkgs__     ## test all installed packages
>> __test_pkgs__ control   ## test a single package
>> __test_pkgs__ ({'control','io'})    ## test just a list of specified
>> packages
>
> Thanks! IMHO that is pretty useful.
> Do you think you could provide a patch that adds this to default Octave? Maybe by calling runtests directly.

Thanks!

It's still a work in progress. Once it's settled, I'll look at writing a
patch for it to go with the related patch at
https://savannah.gnu.org/bugs/?55522. runtests() would have to be
modified to return more info: the my_runtests() local function in
__test_pkgs__() is just a copy-and-paste of runtests(), modified to
return a failure count.

Cheers,
Andrew

Reply | Threaded
Open this post in threaded view
|

Re: run BISTs for all installed packages

mmuetzel
In reply to this post by PhilipNienhuis
Am 28. Januar 2019 um 20:20 Uhr schrieb "PhilipNienhuis"
> In the io package there's a "test_spsh" script + two helper functions that
> checks all interfaces one by one, depending on presence of Java libs added
> to the javaclasspath (+ on Windows, loaded windows package + MS-Office).
>
> It's made that way to have consistent checks for all "interfaces", among
> others. I suppose it should be possible to morph test-spsh.m into tests that
> conform to Octave's regular test format but currently it serves my purpose
> and I fear it would become quite messy to rewrite it.
>

I see what you mean.
Maybe the function could return false if the nice table at the end of the test contained a "-" at any position.
Then one way to "transform" this into a BIST without too much hassle could be to just add this line to the end of the file:
%!assert (test_spsh)

Reply | Threaded
Open this post in threaded view
|

Re: run BISTs for all installed packages

apjanke-floss
In reply to this post by mmuetzel


On 1/28/19 2:04 PM, "Markus Mützel" wrote:

> Am 28. Januar 2019 um 04:11 Uhr schrieb "Andrew Janke"
>> On 1/21/19 11:44 AM, "Markus Mützel" wrote:
>>> Is there an easy way to run all BISTs of all installed packages?
>>
>> Here's a function to do this.
>>
>> __test_pkgs__     ## test all installed packages
>> __test_pkgs__ control   ## test a single package
>> __test_pkgs__ ({'control','io'})    ## test just a list of specified
>> packages
>
> Thanks! IMHO that is pretty useful.
> Do you think you could provide a patch that adds this to default Octave? Maybe by calling runtests directly.

Since this is still a work in progress, instead of a patch, I've wrapped
this up as a pkg package and posted it:
https://github.com/apjanke/octave-testify. This "testify" package also
includes the "failure summary in test run results" functionality from
https://savannah.gnu.org/bugs/index.php?55522.

The code in the package modifies runtests() so that it can be called
directly instead of using a copy-and-paste variant.

Cheers,
Andrew

Reply | Threaded
Open this post in threaded view
|

Re: run BISTs for all installed packages

Carnë Draug
In reply to this post by apjanke-floss
On Mon, 28 Jan 2019 at 03:12, Andrew Janke <[hidden email]> wrote:

> On 1/21/19 11:44 AM, "Markus Mützel" wrote:
> > I frequently cross-compile for Windows with all packages. But I very rarely actually test whether they work as expected.
> > Is there an easy way to run all BISTs of all installed packages?
> > Something that would be more streamlined than loading the packages one after the other and manually running "test" for all files.
> >
> > Markus
> >
>
> Here's a function to do this.
>
> __test_pkgs__     ## test all installed packages
> __test_pkgs__ control   ## test a single package
> __test_pkgs__ ({'control','io'})    ## test just a list of specified
> packages
>
> I'm using this in my (new, experimental) jsonstuff package:
> https://github.com/apjanke/octave-jsonstuff. You can have a look there
> to see how to call it on multiple OSes in a Travis CI environment.
> (Windows isn't working there yet, but I think it's close.) You can also
> just run it interactively; it has no dependencies except for Octave itself.
>
> The Windows Octave installers seem to come with a bunch of packages
> preinstalled, so a plain __test_pkgs__ will end up testing a lot of
> stuff there.
>

Beware that not all package tests are in .m files.  There will be
tests in .cc files which get compiled into .oct files.  Such tests
will get discarded together with the source files.

Reply | Threaded
Open this post in threaded view
|

Re: run BISTs for all installed packages

apjanke-floss


On 2/5/19 11:01 AM, Carnë Draug wrote:

> On Mon, 28 Jan 2019 at 03:12, Andrew Janke <[hidden email]> wrote:
>> On 1/21/19 11:44 AM, "Markus Mützel" wrote:
>>> I frequently cross-compile for Windows with all packages. But I very rarely actually test whether they work as expected.
>>> Is there an easy way to run all BISTs of all installed packages?
>>> Something that would be more streamlined than loading the packages one after the other and manually running "test" for all files.
>>>
>>> Markus
>>>
>>
>> Here's a function to do this.
>>
>> __test_pkgs__     ## test all installed packages
>> __test_pkgs__ control   ## test a single package
>> __test_pkgs__ ({'control','io'})    ## test just a list of specified
>> packages
>>
>> I'm using this in my (new, experimental) jsonstuff package:
>> https://github.com/apjanke/octave-jsonstuff. You can have a look there
>> to see how to call it on multiple OSes in a Travis CI environment.
>> (Windows isn't working there yet, but I think it's close.) You can also
>> just run it interactively; it has no dependencies except for Octave itself.
>>
>> The Windows Octave installers seem to come with a bunch of packages
>> preinstalled, so a plain __test_pkgs__ will end up testing a lot of
>> stuff there.
>>
>
> Beware that not all package tests are in .m files.  There will be
> tests in .cc files which get compiled into .oct files.  Such tests
> will get discarded together with the source files.
>

Hmm.

In core Octave, are those the tests that end up in
matlabroot()/share/octave/4.4.1/etc/tests ans *.cc-tst or in the Octave
source tree as *.in.yy-tst files? Maybe we could add a mechanism to
packages to similarly extract them to *-tst files?

Reply | Threaded
Open this post in threaded view
|

Re: run BISTs for all installed packages

Mike Miller-4
On Tue, Feb 05, 2019 at 11:09:43 -0500, Andrew Janke wrote:
> In core Octave, are those the tests that end up in
> matlabroot()/share/octave/4.4.1/etc/tests ans *.cc-tst or in the Octave
> source tree as *.in.yy-tst files? Maybe we could add a mechanism to packages
> to similarly extract them to *-tst files?

Yes: https://savannah.gnu.org/bugs/?41298

It looks like you are interested in helping to improve and expand the
features of Octave's test suite functions, which is great. Please do
have a look at prior bug reports, discussions, ideas, and prior art
because there is a lot of it out there.

There have been discussions on a lot of this already, including a "pkg
test" subfunction, installing "-tst" files for compiled code in pkgs,
refactoring and deduplicating code between "runtests" and
"__run_test_suite__".

You might also want to look at the Debian dh-octave package, which
implements its own method for running all tests in a source package at
build time and in a CI environment.

--
mike

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: run BISTs for all installed packages

apjanke-floss


On 2/5/19 11:41 AM, Mike Miller wrote:

> On Tue, Feb 05, 2019 at 11:09:43 -0500, Andrew Janke wrote:
>> In core Octave, are those the tests that end up in
>> matlabroot()/share/octave/4.4.1/etc/tests ans *.cc-tst or in the Octave
>> source tree as *.in.yy-tst files? Maybe we could add a mechanism to packages
>> to similarly extract them to *-tst files?
>
> Yes: https://savannah.gnu.org/bugs/?41298
>
> It looks like you are interested in helping to improve and expand the
> features of Octave's test suite functions, which is great. Please do
> have a look at prior bug reports, discussions, ideas, and prior art
> because there is a lot of it out there.
>
> There have been discussions on a lot of this already, including a "pkg
> test" subfunction, installing "-tst" files for compiled code in pkgs,
> refactoring and deduplicating code between "runtests" and
> "__run_test_suite__".
>
> You might also want to look at the Debian dh-octave package, which
> implements its own method for running all tests in a source package at
> build time and in a CI environment.
>

Thanks; I hadn't seen that bug and wasn't aware of the dh-octave
package. I'll have another pass through the issue tracker to make sure
I'm up to date.

Cheers,
Andrew