run BISTs for all installed packages

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

run BISTs for all installed packages

mmuetzel
I frequently cross-compile for Windows with all packages. But I very rarely actually test whether they work as expected.
Is there an easy way to run all BISTs of all installed packages?
Something that would be more streamlined than loading the packages one after the other and manually running "test" for all files.

Markus

Reply | Threaded
Open this post in threaded view
|

Re: run BISTs for all installed packages

apjanke-floss


On 1/21/19 11:44 AM, "Markus Mützel" wrote:
> I frequently cross-compile for Windows with all packages. But I very rarely actually test whether they work as expected.
> Is there an easy way to run all BISTs of all installed packages?
> Something that would be more streamlined than loading the packages one after the other and manually running "test" for all files.
>
> Markus
>

Here's a function to do this.

__test_pkgs__     ## test all installed packages
__test_pkgs__ control   ## test a single package
__test_pkgs__ ({'control','io'})    ## test just a list of specified
packages

I'm using this in my (new, experimental) jsonstuff package:
https://github.com/apjanke/octave-jsonstuff. You can have a look there
to see how to call it on multiple OSes in a Travis CI environment.
(Windows isn't working there yet, but I think it's close.) You can also
just run it interactively; it has no dependencies except for Octave itself.

The Windows Octave installers seem to come with a bunch of packages
preinstalled, so a plain __test_pkgs__ will end up testing a lot of
stuff there.

Currently, when I run it in Windows 10 version 1809 for all packages on
Octave 4.4.1, Octave crashes a few packages in. Also crashes on macOS
10.13.6 if I enable the "torture test" mode that tests each package
multiple times.

Cheers,
Andrew

__test_pkgs__.m (12K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: run BISTs for all installed packages

Ian McCallion
On Monday, 28 January 2019, Andrew Janke <[hidden email]> wrote:


On 1/21/19 11:44 AM, "Markus Mützel" wrote:
I frequently cross-compile for Windows with all packages. But I very rarely actually test whether they work as expected.
Is there an easy way to run all BISTs of all installed packages?
Something that would be more streamlined than loading the packages one after the other and manually running "test" for all files.

Markus


Here's a function to do this.

__test_pkgs__     ## test all installed packages
__test_pkgs__ control   ## test a single package
__test_pkgs__ ({'control','io'})    ## test just a list of specified packages

I'm using this in my (new, experimental) jsonstuff package: https://github.com/apjanke/octave-jsonstuff. You can have a look there to see how to call it on multiple OSes in a Travis CI environment. (Windows isn't working there yet, but I think it's close.) You can also just run it interactively; it has no dependencies except for Octave itself.

The Windows Octave installers seem to come with a bunch of packages preinstalled, so a plain __test_pkgs__ will end up testing a lot of stuff there.

Currently, when I run it in Windows 10 version 1809 for all packages on Octave 4.4.1, Octave crashes a few packages in. Also crashes on macOS 10.13.6 if I enable the "torture test" mode that tests each package multiple times.

I had a problem that no console output was appearing  with Octave when 1809 was installed on my laptop.  I switched to the legacy console and the problem went away.

Cheers... Ian
Reply | Threaded
Open this post in threaded view
|

Re: run BISTs for all installed packages

mmuetzel
In reply to this post by apjanke-floss
Am 28. Januar 2019 um 04:11 Uhr schrieb "Andrew Janke"
> On 1/21/19 11:44 AM, "Markus Mützel" wrote:
> > Is there an easy way to run all BISTs of all installed packages?
>
> Here's a function to do this.
>
> __test_pkgs__     ## test all installed packages
> __test_pkgs__ control   ## test a single package
> __test_pkgs__ ({'control','io'})    ## test just a list of specified
> packages

Thanks! IMHO that is pretty useful.
Do you think you could provide a patch that adds this to default Octave? Maybe by calling runtests directly.

> Currently, when I run it in Windows 10 version 1809 for all packages on
> Octave 4.4.1, Octave crashes a few packages in. Also crashes on macOS
> 10.13.6 if I enable the "torture test" mode that tests each package
> multiple times.

I've also seen this crash initially. But after 2 or 3 attempts, I can no longer reproduce the crash (surprisingly).

Markus

Reply | Threaded
Open this post in threaded view
|

Re: run BISTs for all installed packages

PhilipNienhuis
In reply to this post by mmuetzel
mmuetzel wrote
> I frequently cross-compile for Windows with all packages. But I very
> rarely actually test whether they work as expected.
> Is there an easy way to run all BISTs of all installed packages?
> Something that would be more streamlined than loading the packages one
> after the other and manually running "test" for all files.
>
> Markus

In the io package there's a "test_spsh" script + two helper functions that
checks all interfaces one by one, depending on presence of Java libs added
to the javaclasspath (+ on Windows, loaded windows package + MS-Office).

It's made that way to have consistent checks for all "interfaces", among
others. I suppose it should be possible to morph test-spsh.m into tests that
conform to Octave's regular test format but currently it serves my purpose
and I fear it would become quite messy to rewrite it.

Philip




--
Sent from: http://octave.1599824.n4.nabble.com/Octave-Maintainers-f1638794.html

Reply | Threaded
Open this post in threaded view
|

Re: run BISTs for all installed packages

jbect
In reply to this post by mmuetzel
Le 28/01/2019 à 20:04, "Markus Mützel" a écrit :
Am 28. Januar 2019 um 04:11 Uhr schrieb "Andrew Janke"
On 1/21/19 11:44 AM, "Markus Mützel" wrote:
Is there an easy way to run all BISTs of all installed packages?
Here's a function to do this.

__test_pkgs__     ## test all installed packages
__test_pkgs__ control   ## test a single package
__test_pkgs__ ({'control','io'})    ## test just a list of specified 
packages
Thanks! IMHO that is pretty useful.
Do you think you could provide a patch that adds this to default Octave? Maybe by calling runtests directly.

Some related work can be found on this ticket : http://savannah.gnu.org/bugs/?41215.

@++
Julien
Reply | Threaded
Open this post in threaded view
|

Re: run BISTs for all installed packages

apjanke-floss
In reply to this post by mmuetzel


On 1/28/19 2:04 PM, "Markus Mützel" wrote:

> Am 28. Januar 2019 um 04:11 Uhr schrieb "Andrew Janke"
>> On 1/21/19 11:44 AM, "Markus Mützel" wrote:
>>> Is there an easy way to run all BISTs of all installed packages?
>>
>> Here's a function to do this.
>>
>> __test_pkgs__     ## test all installed packages
>> __test_pkgs__ control   ## test a single package
>> __test_pkgs__ ({'control','io'})    ## test just a list of specified
>> packages
>
> Thanks! IMHO that is pretty useful.
> Do you think you could provide a patch that adds this to default Octave? Maybe by calling runtests directly.

Thanks!

It's still a work in progress. Once it's settled, I'll look at writing a
patch for it to go with the related patch at
https://savannah.gnu.org/bugs/?55522. runtests() would have to be
modified to return more info: the my_runtests() local function in
__test_pkgs__() is just a copy-and-paste of runtests(), modified to
return a failure count.

Cheers,
Andrew

Reply | Threaded
Open this post in threaded view
|

Re: run BISTs for all installed packages

mmuetzel
In reply to this post by PhilipNienhuis
Am 28. Januar 2019 um 20:20 Uhr schrieb "PhilipNienhuis"
> In the io package there's a "test_spsh" script + two helper functions that
> checks all interfaces one by one, depending on presence of Java libs added
> to the javaclasspath (+ on Windows, loaded windows package + MS-Office).
>
> It's made that way to have consistent checks for all "interfaces", among
> others. I suppose it should be possible to morph test-spsh.m into tests that
> conform to Octave's regular test format but currently it serves my purpose
> and I fear it would become quite messy to rewrite it.
>

I see what you mean.
Maybe the function could return false if the nice table at the end of the test contained a "-" at any position.
Then one way to "transform" this into a BIST without too much hassle could be to just add this line to the end of the file:
%!assert (test_spsh)

Reply | Threaded
Open this post in threaded view
|

Re: run BISTs for all installed packages

apjanke-floss
In reply to this post by mmuetzel


On 1/28/19 2:04 PM, "Markus Mützel" wrote:

> Am 28. Januar 2019 um 04:11 Uhr schrieb "Andrew Janke"
>> On 1/21/19 11:44 AM, "Markus Mützel" wrote:
>>> Is there an easy way to run all BISTs of all installed packages?
>>
>> Here's a function to do this.
>>
>> __test_pkgs__     ## test all installed packages
>> __test_pkgs__ control   ## test a single package
>> __test_pkgs__ ({'control','io'})    ## test just a list of specified
>> packages
>
> Thanks! IMHO that is pretty useful.
> Do you think you could provide a patch that adds this to default Octave? Maybe by calling runtests directly.

Since this is still a work in progress, instead of a patch, I've wrapped
this up as a pkg package and posted it:
https://github.com/apjanke/octave-testify. This "testify" package also
includes the "failure summary in test run results" functionality from
https://savannah.gnu.org/bugs/index.php?55522.

The code in the package modifies runtests() so that it can be called
directly instead of using a copy-and-paste variant.

Cheers,
Andrew

Reply | Threaded
Open this post in threaded view
|

Re: run BISTs for all installed packages

Carnë Draug
In reply to this post by apjanke-floss
On Mon, 28 Jan 2019 at 03:12, Andrew Janke <[hidden email]> wrote:

> On 1/21/19 11:44 AM, "Markus Mützel" wrote:
> > I frequently cross-compile for Windows with all packages. But I very rarely actually test whether they work as expected.
> > Is there an easy way to run all BISTs of all installed packages?
> > Something that would be more streamlined than loading the packages one after the other and manually running "test" for all files.
> >
> > Markus
> >
>
> Here's a function to do this.
>
> __test_pkgs__     ## test all installed packages
> __test_pkgs__ control   ## test a single package
> __test_pkgs__ ({'control','io'})    ## test just a list of specified
> packages
>
> I'm using this in my (new, experimental) jsonstuff package:
> https://github.com/apjanke/octave-jsonstuff. You can have a look there
> to see how to call it on multiple OSes in a Travis CI environment.
> (Windows isn't working there yet, but I think it's close.) You can also
> just run it interactively; it has no dependencies except for Octave itself.
>
> The Windows Octave installers seem to come with a bunch of packages
> preinstalled, so a plain __test_pkgs__ will end up testing a lot of
> stuff there.
>

Beware that not all package tests are in .m files.  There will be
tests in .cc files which get compiled into .oct files.  Such tests
will get discarded together with the source files.

Reply | Threaded
Open this post in threaded view
|

Re: run BISTs for all installed packages

apjanke-floss


On 2/5/19 11:01 AM, Carnë Draug wrote:

> On Mon, 28 Jan 2019 at 03:12, Andrew Janke <[hidden email]> wrote:
>> On 1/21/19 11:44 AM, "Markus Mützel" wrote:
>>> I frequently cross-compile for Windows with all packages. But I very rarely actually test whether they work as expected.
>>> Is there an easy way to run all BISTs of all installed packages?
>>> Something that would be more streamlined than loading the packages one after the other and manually running "test" for all files.
>>>
>>> Markus
>>>
>>
>> Here's a function to do this.
>>
>> __test_pkgs__     ## test all installed packages
>> __test_pkgs__ control   ## test a single package
>> __test_pkgs__ ({'control','io'})    ## test just a list of specified
>> packages
>>
>> I'm using this in my (new, experimental) jsonstuff package:
>> https://github.com/apjanke/octave-jsonstuff. You can have a look there
>> to see how to call it on multiple OSes in a Travis CI environment.
>> (Windows isn't working there yet, but I think it's close.) You can also
>> just run it interactively; it has no dependencies except for Octave itself.
>>
>> The Windows Octave installers seem to come with a bunch of packages
>> preinstalled, so a plain __test_pkgs__ will end up testing a lot of
>> stuff there.
>>
>
> Beware that not all package tests are in .m files.  There will be
> tests in .cc files which get compiled into .oct files.  Such tests
> will get discarded together with the source files.
>

Hmm.

In core Octave, are those the tests that end up in
matlabroot()/share/octave/4.4.1/etc/tests ans *.cc-tst or in the Octave
source tree as *.in.yy-tst files? Maybe we could add a mechanism to
packages to similarly extract them to *-tst files?

Reply | Threaded
Open this post in threaded view
|

Re: run BISTs for all installed packages

Mike Miller-4
On Tue, Feb 05, 2019 at 11:09:43 -0500, Andrew Janke wrote:
> In core Octave, are those the tests that end up in
> matlabroot()/share/octave/4.4.1/etc/tests ans *.cc-tst or in the Octave
> source tree as *.in.yy-tst files? Maybe we could add a mechanism to packages
> to similarly extract them to *-tst files?

Yes: https://savannah.gnu.org/bugs/?41298

It looks like you are interested in helping to improve and expand the
features of Octave's test suite functions, which is great. Please do
have a look at prior bug reports, discussions, ideas, and prior art
because there is a lot of it out there.

There have been discussions on a lot of this already, including a "pkg
test" subfunction, installing "-tst" files for compiled code in pkgs,
refactoring and deduplicating code between "runtests" and
"__run_test_suite__".

You might also want to look at the Debian dh-octave package, which
implements its own method for running all tests in a source package at
build time and in a CI environment.

--
mike

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: run BISTs for all installed packages

apjanke-floss


On 2/5/19 11:41 AM, Mike Miller wrote:

> On Tue, Feb 05, 2019 at 11:09:43 -0500, Andrew Janke wrote:
>> In core Octave, are those the tests that end up in
>> matlabroot()/share/octave/4.4.1/etc/tests ans *.cc-tst or in the Octave
>> source tree as *.in.yy-tst files? Maybe we could add a mechanism to packages
>> to similarly extract them to *-tst files?
>
> Yes: https://savannah.gnu.org/bugs/?41298
>
> It looks like you are interested in helping to improve and expand the
> features of Octave's test suite functions, which is great. Please do
> have a look at prior bug reports, discussions, ideas, and prior art
> because there is a lot of it out there.
>
> There have been discussions on a lot of this already, including a "pkg
> test" subfunction, installing "-tst" files for compiled code in pkgs,
> refactoring and deduplicating code between "runtests" and
> "__run_test_suite__".
>
> You might also want to look at the Debian dh-octave package, which
> implements its own method for running all tests in a source package at
> build time and in a CI environment.
>

Thanks; I hadn't seen that bug and wasn't aware of the dh-octave
package. I'll have another pass through the issue tracker to make sure
I'm up to date.

Cheers,
Andrew

Reply | Threaded
Open this post in threaded view
|

Re: run BISTs for all installed packages

apjanke-floss


On 2/7/19 2:41 PM, Andrew Janke wrote:

>
>
> On 2/5/19 11:41 AM, Mike Miller wrote:
>> On Tue, Feb 05, 2019 at 11:09:43 -0500, Andrew Janke wrote:
>>> In core Octave, are those the tests that end up in
>>> matlabroot()/share/octave/4.4.1/etc/tests ans *.cc-tst or in the Octave
>>> source tree as *.in.yy-tst files? Maybe we could add a mechanism to
>>> packages
>>> to similarly extract them to *-tst files?
>>
>> Yes: https://savannah.gnu.org/bugs/?41298
>>
>> It looks like you are interested in helping to improve and expand the
>> features of Octave's test suite functions, which is great. Please do
>> have a look at prior bug reports, discussions, ideas, and prior art
>> because there is a lot of it out there.
>>
>> There have been discussions on a lot of this already, including a "pkg
>> test" subfunction, installing "-tst" files for compiled code in pkgs,
>> refactoring and deduplicating code between "runtests" and
>> "__run_test_suite__".
>>
>> You might also want to look at the Debian dh-octave package, which
>> implements its own method for running all tests in a source package at
>> build time and in a CI environment.
>>
>
> Thanks; I hadn't seen that bug and wasn't aware of the dh-octave
> package. I'll have another pass through the issue tracker to make sure
> I'm up to date.
>
> Cheers,
> Andrew

I've put together a list of references to bugs and Octave maintainers
mailing list threads related to enhancing the test suite:
https://github.com/apjanke/octave-testify/blob/e6e9cd4117c6d6e601a8c00487dc75769aa1699f/doc-project/Developer-Notes.md

Any major ones I'm missing?

Cheers,
Andrew

Reply | Threaded
Open this post in threaded view
|

Re: run BISTs for all installed packages

Mike Miller-4
On Tue, Mar 05, 2019 at 10:59:01 -0500, Andrew Janke wrote:
> I've put together a list of references to bugs and Octave maintainers
> mailing list threads related to enhancing the test suite: https://github.com/apjanke/octave-testify/blob/e6e9cd4117c6d6e601a8c00487dc75769aa1699f/doc-project/Developer-Notes.md
>
> Any major ones I'm missing?

Looks good, thank you for collecting all of these topics together.

I agree with your project goals. I also hope you'll preserve some of the
current test suite features I think are important, including being able
to run the test suite pre-install or post-install, read-only, optionally
print full test failures in auto build or CI environments.

If I have my own list of annoyances / wishlist things I'd like to see
improved, would you rather see them reported on savannah or on your
project?

Are you aware of the 'doctest' package [1] to run tests from Octave help
strings? I think it's mostly orthogonal to your goals, just wanted to
make sure you knew it was out there.

I also maintain a separate shell-based test suite [2] to test Octave's
command interface, command line options, exit status, and so on, also
unrelated to your project goals.

But at some point I'd like to see these two supported by Octave to some
extent.

[1]: https://github.com/catch22/octave-doctest
[2]: https://gitlab.com/mtmiller/octave-test-suite

--
mike

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: run BISTs for all installed packages

apjanke-floss


On 3/5/19 7:18 PM, Mike Miller wrote:
> On Tue, Mar 05, 2019 at 10:59:01 -0500, Andrew Janke wrote:
>> I've put together a list of references to bugs and Octave maintainers
>> mailing list threads related to enhancing the test suite: https://github.com/apjanke/octave-testify/blob/e6e9cd4117c6d6e601a8c00487dc75769aa1699f/doc-project/Developer-Notes.md
>>
>> Any major ones I'm missing?
>
> Looks good, thank you for collecting all of these topics together.
>
> I agree with your project goals.

Thanks!

> I also hope you'll preserve some of the
> current test suite features I think are important, including being able
> to run the test suite pre-install or post-install, read-only, optionally
> print full test failures in auto build or CI environments.

I'm gonna preserve ALL of these features, *ALL* of them, ha ha ha!

Though not necessarily with the same invocation syntax. :)

More seriously, my intention for Testify is to be a:
a) more-or-less strict functional superset of existing BIST
functionality, with
b) nicer output and more convenient calling forms, and
c) maybe more structured control over how the BISTs are invoked and
interact with each other and are collected together in summary objects

This Testify project isn't some big idea to make a whole new testing
layer for Octave[1]. I mean, originally all I wanted to do was provide a
concise summary output of failed tests at the end of the
__run_test_suite__() output so you didn't have to scroll back and copy &
paste multiple times to get a list of your failed tests
(https://savannah.gnu.org/bugs/index.php?55522). But all this code is
intertwingled, and one thing led to another, so...

[1] ...although, maybe, I'll throw in a port of the xUnit testing
framework, given some spare time and user interest...

> If I have my own list of annoyances / wishlist things I'd like to see
> improved, would you rather see them reported on savannah or on your
> project?

I'd prefer them reported on my project (one issue report per
annoyance/wishlist item, please, and no annoyance too small), because a)
GitHub has nicer issue/task-tracking tools than Savannah IMHO, and it'll
be easier to grind them down in to separate pieces of work there, b) I'd
probably consider these in-scope for this particular project, and could
iterate on them in GitHub independent of the Octave project management
because Testify is an independent project that's not in Octave Forge
yet, and c) I think in terms of tooling, it's actually easier for us to
work on an independent octave project using git/GitHub and `pkg install`
tooling? Once you get to multiple commits, forks and branches are easier
to manage than Hg changesets stored behind bug attachment URLs.

You should view Testify as "Andrew's wacky idea of where Octave BIST
support should go in the next release or two", and if you submit a
gripe/wishlist item there, I'm likely to pick it up and have an Idea and
do something about it. And then hopefully eventually we can get back
together with core Octave developers, find a way that a compromise
version of Testify can be merged back in to Octave core, do that, and
everybody benefits.

> Are you aware of the 'doctest' package [1] to run tests from Octave help
> strings? I think it's mostly orthogonal to your goals, just wanted to
> make sure you knew it was out there.
 >
 > [1]: https://github.com/catch22/octave-doctest

Uhh, no! Thanks for the heads-up. I'll look in to it.

At first glance, this actually doesn't look *that* different from BISTs,
it just has fancy ways of specifying the strings it's using as
"expected" values in its assert() logic...

> I also maintain a separate shell-based test suite [2] to test Octave's
> command interface, command line options, exit status, and so on, also
> unrelated to your project goals.

This is interesting. Looks like you're mostly concerned with the
interface between the `octave` command or runtime and the external
system. This is probably something I should learn about some day. :)

>
> But at some point I'd like to see these two supported by Octave to some
> extent.
>
> [2]: https://gitlab.com/mtmiller/octave-test-suite
>

Reply | Threaded
Open this post in threaded view
|

Re: run BISTs for all installed packages

apjanke-floss
In reply to this post by Mike Miller-4


On 3/5/19 7:18 PM, Mike Miller wrote:
> On Tue, Mar 05, 2019 at 10:59:01 -0500, Andrew Janke wrote:
>> I've put together a list of references to bugs and Octave maintainers
>> mailing list threads related to enhancing the test suite: https://github.com/apjanke/octave-testify/blob/e6e9cd4117c6d6e601a8c00487dc75769aa1699f/doc-project/Developer-Notes.md
[...snip...]
> Are you aware of the 'doctest' package [1] to run tests from Octave help
> strings? I think it's mostly orthogonal to your goals, just wanted to
> make sure you knew it was out there.

Can you give me examples of a couple Octave Forge packages that have
doctest-compatible tests to practice on?

Cheers,
Andrew