Release 5.0 Next Tasks

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Release 5.0 Next Tasks

Rik-4
Today we completed the language translations (task #12) and the lint checking of the code base (task #8).

Next up is tasks #9 and #10.

9) Verify 'make check' is passing on all buildbot combinations of OS and compilers
  • Start discussion on octave-maintainers list about which failing tests must be fixed
  • Identify and fix any tests determined critical in step above

10) Compile and run Octave test suite with --enable-address-sanitizer-flags to check for memory leaks 

  • Results posted to bug report: #55415
  • Memory leak in graphics subsystem #55287

I've looked through the memory leaks in bug #55415 and I don't think they are gating on the release.  However, the leak in bug #55287 should be addressed for 5.0.

This isn't a whole lot so I imagine that we could produce the first release candidate over the weekend.  That will undoubtedly flush out a few more bugs, but the process does feel like it is converging.

--Rik


Reply | Threaded
Open this post in threaded view
|

Re: Release 5.0 Next Tasks

Dmitri A. Sergatskov


On Thu, Jan 17, 2019 at 11:10 AM Rik <[hidden email]> wrote:
Today we completed the language translations (task #12) and the lint checking of the code base (task #8).

Next up is tasks #9 and #10.

9) Verify 'make check' is passing on all buildbot combinations of OS and compilers
  • Start discussion on octave-maintainers list about which failing tests must be fixed
  • Identify and fix any tests determined critical in step above

10) Compile and run Octave test suite with --enable-address-sanitizer-flags to check for memory leaks 

  • Results posted to bug report: #55415
  • Memory leak in graphics subsystem #55287

I've looked through the memory leaks in bug #55415 and I don't think they are gating on the release.  However, the leak in bug #55287 should be addressed for 5.0.

This isn't a whole lot so I imagine that we could produce the first release candidate over the weekend.  That will undoubtedly flush out a few more bugs, but the process does feel like it is converging.


What about SUNDIALS problem?

 

--Rik



Dmitri.
--
Reply | Threaded
Open this post in threaded view
|

Re: Release 5.0 Next Tasks

Rik-4
On 01/17/2019 07:12 PM, Dmitri A. Sergatskov wrote:


On Thu, Jan 17, 2019 at 11:10 AM Rik <[hidden email]> wrote:
Today we completed the language translations (task #12) and the lint checking of the code base (task #8).

Next up is tasks #9 and #10.

9) Verify 'make check' is passing on all buildbot combinations of OS and compilers
  • Start discussion on octave-maintainers list about which failing tests must be fixed
  • Identify and fix any tests determined critical in step above

10) Compile and run Octave test suite with --enable-address-sanitizer-flags to check for memory leaks 

  • Results posted to bug report: #55415
  • Memory leak in graphics subsystem #55287

I've looked through the memory leaks in bug #55415 and I don't think they are gating on the release.  However, the leak in bug #55287 should be addressed for 5.0.

This isn't a whole lot so I imagine that we could produce the first release candidate over the weekend.  That will undoubtedly flush out a few more bugs, but the process does feel like it is converging.


What about SUNDIALS problem?


Haven't been following this.  Is there a bug report about it?

--Rik
Reply | Threaded
Open this post in threaded view
|

Re: Release 5.0 Next Tasks

Dmitri A. Sergatskov


On Thu, Jan 17, 2019 at 9:51 PM Rik <[hidden email]> wrote:


What about SUNDIALS problem?


Haven't been following this.  Is there a bug report about it?

 

--Rik
Reply | Threaded
Open this post in threaded view
|

Re: Release 5.0 Next Tasks

apjanke-floss
In reply to this post by Rik-4


On 1/17/19 12:10 PM, Rik wrote:

> Today we completed the language translations (task #12) and the lint
> checking of the code base (task #8).
>
> Next up is tasks #9 and #10.
>
> 9) Verify 'make check' is passing on all buildbot combinations of OS and
> compilers
>
>   * Start discussion on octave-maintainers list about which failing
>     tests must be fixed
>   * Identify and fix any tests determined critical in step above
>
> 10) Compile and run Octave test suite with
> --enable-address-sanitizer-flags to check for memory leaks
>
>   * Results posted to bug report: #55415
>     <https://savannah.gnu.org/bugs/?55415>
>   * Memory leak in graphics subsystem #55287
>     <https://savannah.gnu.org/bugs/?55287>
>
> I've looked through the memory leaks in bug #55415 and I don't think
> they are gating on the release.  However, the leak in bug #55287 should
> be addressed for 5.0.
>
> This isn't a whole lot so I imagine that we could produce the first
> release candidate over the weekend.  That will undoubtedly flush out a
> few more bugs, but the process does feel like it is converging.
>
> --Rik
>
There are still some tests in the test suite failing on macOS for me.

Run on 1/120/2019 against about commit 2ae2dcf0f4d5.

Summary:

   PASS                            15453
   FAIL                               10
   XFAIL (reported bug)               43
   SKIP (missing feature)             48
   SKIP (run-time condition)          29

   Failed tests:
      libinterp/corefcn/pr-output.cc-tst
      libinterp/parse-tree/oct-parse.yy-tst
      general/logspace.m
      miscellaneous/tar.m
      plot/util/copyobj.m
      plot/util/hgsave.m
      sparse/eigs.m

Is the goal to have a clean test suite baseline for the release?

Cheers,
Andrew



octave 5.0 failing test log - macOS 10.13 - 2019-01-20.txt (13K) Download Attachment
runsometests.m (1K) Download Attachment
run_failing_5_0_tests.m (460 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Release 5.0 Next Tasks

Marco Caliari-4
On Sun, 20 Jan 2019 20:45:04 -0500
Andrew Janke <[hidden email]> wrote:

> On 1/17/19 12:10 PM, Rik wrote:
> > Today we completed the language translations (task #12) and the lint
> > checking of the code base (task #8).
> >
> > Next up is tasks #9 and #10.
> >
> > 9) Verify 'make check' is passing on all buildbot combinations of OS and
> > compilers
> >
> >   * Start discussion on octave-maintainers list about which failing
> >     tests must be fixed
> >   * Identify and fix any tests determined critical in step above
> >
> > 10) Compile and run Octave test suite with
> > --enable-address-sanitizer-flags to check for memory leaks
> >
> >   * Results posted to bug report: #55415
> >     <https://savannah.gnu.org/bugs/?55415>
> >   * Memory leak in graphics subsystem #55287
> >     <https://savannah.gnu.org/bugs/?55287>
> >
> > I've looked through the memory leaks in bug #55415 and I don't think
> > they are gating on the release.  However, the leak in bug #55287 should
> > be addressed for 5.0.
> >
> > This isn't a whole lot so I imagine that we could produce the first
> > release candidate over the weekend.  That will undoubtedly flush out a
> > few more bugs, but the process does feel like it is converging.
> >
> > --Rik
> >  
>
> There are still some tests in the test suite failing on macOS for me.
>
> Run on 1/120/2019 against about commit 2ae2dcf0f4d5.
>
> Summary:
>
>    PASS                            15453
>    FAIL                               10
>    XFAIL (reported bug)               43
>    SKIP (missing feature)             48
>    SKIP (run-time condition)          29
>
>    Failed tests:
>       libinterp/corefcn/pr-output.cc-tst
>       libinterp/parse-tree/oct-parse.yy-tst
>       general/logspace.m
>       miscellaneous/tar.m
>       plot/util/copyobj.m
>       plot/util/hgsave.m
>       sparse/eigs.m
>
> Is the goal to have a clean test suite baseline for the release?
>
> Cheers,
> Andrew

The problem in eigs is strange, the matrix and the function versions should
do exactly the same thing. Therefore, I implemented the assert
instructions with 0 tolerance. Anyway, the workaround is to relax a bit the
tolerance, 1e-14 should work.

Marco
Reply | Threaded
Open this post in threaded view
|

Re: Release 5.0 Next Tasks

Dmitri A. Sergatskov


On Mon, Jan 21, 2019 at 3:48 AM Marco Caliari <[hidden email]> wrote:

>
> Run on 1/120/2019 against about commit 2ae2dcf0f4d5.
>
> Summary:
>
>    PASS                            15453
>    FAIL                               10
>    XFAIL (reported bug)               43
>    SKIP (missing feature)             48
>    SKIP (run-time condition)          29
>
>    Failed tests:
>       libinterp/corefcn/pr-output.cc-tst
>       libinterp/parse-tree/oct-parse.yy-tst
>       general/logspace.m
>       miscellaneous/tar.m
>       plot/util/copyobj.m
>       plot/util/hgsave.m
>       sparse/eigs.m
>
> Is the goal to have a clean test suite baseline for the release?
>
> Cheers,
> Andrew

The problem in eigs is strange, the matrix and the function versions should
do exactly the same thing. Therefore, I implemented the assert
instructions with 0 tolerance. Anyway, the workaround is to relax a bit the
tolerance, 1e-14 should work.

The eigs issue is due to veclib, works fine with openblas (see:

Logspace issue is due to bugs in clang/libc++, so perhaps the test should
be skipped on macos (like e.g. tests for mappers.cc-tst).

Dmitri.
--



 

Marco
Reply | Threaded
Open this post in threaded view
|

Re: Release 5.0 Next Tasks

John W. Eaton
Administrator
On 1/21/19 11:01 AM, Dmitri A. Sergatskov wrote:

> The eigs issue is due to veclib, works fine with openblas (see:
> https://savannah.gnu.org/bugs/?55534 ).
>
> Logspace issue is due to bugs in clang/libc++, so perhaps the test should
> be skipped on macos (like e.g. tests for mappers.cc-tst).

Couldn't we just give them bug numbers so they will show up as XFAIL
instead of FAIL?

I do understand that the problem with XFAIL is that we don't seem to be
very effective at actually fixing some of these long standing known
bugs, but it seems better to me that we acknowledge rather than skip them.

jwe

Reply | Threaded
Open this post in threaded view
|

Re: Release 5.0 Next Tasks

apjanke-floss
In reply to this post by Marco Caliari-4


On 1/21/19 4:46 AM, Marco Caliari wrote:

> On Sun, 20 Jan 2019 20:45:04 -0500
> Andrew Janke <[hidden email]> wrote:
>
>>>    
>>
>> There are still some tests in the test suite failing on macOS for me.
>>
>> Run on 1/120/2019 against about commit 2ae2dcf0f4d5.
>>
>> Summary:
>>
>>     PASS                            15453
>>     FAIL                               10
>>     XFAIL (reported bug)               43
>>     SKIP (missing feature)             48
>>     SKIP (run-time condition)          29
>>
>>     Failed tests:
>>        libinterp/corefcn/pr-output.cc-tst
>>        libinterp/parse-tree/oct-parse.yy-tst
>>        general/logspace.m
>>        miscellaneous/tar.m
>>        plot/util/copyobj.m
>>        plot/util/hgsave.m
>>        sparse/eigs.m
>>
>> Is the goal to have a clean test suite baseline for the release?
>>
>> Cheers,
>> Andrew
>
> The problem in eigs is strange, the matrix and the function versions should
> do exactly the same thing. Therefore, I implemented the assert
> instructions with 0 tolerance. Anyway, the workaround is to relax a bit the
> tolerance, 1e-14 should work.
>
> Marco
>

Relaxing the tolerance may not work, or may not be the right thing to do
here. The errors in eigs are larger than 1e-14 in some spots:

    (2,1)        14.655       14.655      Abs err 1.1969e-11 exceeds tol
0 by 1e-11
    (3,1)       12.7518      12.7518      Abs err 2.3062e-11 exceeds tol
0 by 2e-11
    (4,1)       -46.9129     -46.9129     Abs err 8.7681e-11 exceeds tol
0 by 9e-11

I don't know enough about BLAS stuff to know how significant that is.

I've filed a bug report on it and discussion is going on there:
https://savannah.gnu.org/bugs/?55534

Andrew

Reply | Threaded
Open this post in threaded view
|

Re: Release 5.0 Next Tasks

Mike Miller-4
In reply to this post by John W. Eaton
On Mon, Jan 21, 2019 at 11:25:58 -0500, John W. Eaton wrote:

> On 1/21/19 11:01 AM, Dmitri A. Sergatskov wrote:
>
> > The eigs issue is due to veclib, works fine with openblas (see:
> > https://savannah.gnu.org/bugs/?55534 ).
> >
> > Logspace issue is due to bugs in clang/libc++, so perhaps the test should
> > be skipped on macos (like e.g. tests for mappers.cc-tst).
>
> Couldn't we just give them bug numbers so they will show up as XFAIL instead
> of FAIL?
This is now reported as https://savannah.gnu.org/bugs/?55538

> I do understand that the problem with XFAIL is that we don't seem to be very
> effective at actually fixing some of these long standing known bugs, but it
> seems better to me that we acknowledge rather than skip them.

Another problem is that a test may be tagged with one bug number that
was fixed, but then may fail on a particular system for a completely
different reason.

And yet another problem is that this bug may be closed as "won't fix" on
savannah because it's an LLVM bug. Someone may later tag it as <*55538>
because the bug report is closed, but the "*" is supposed to mark only
bugs that have been fixed.

--
mike

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Release 5.0 Next Tasks

apjanke-floss
In reply to this post by apjanke-floss


On 1/20/19 8:45 PM, Andrew Janke wrote:

>
> Summary:
>
>    PASS                            15453
>    FAIL                               10
>    XFAIL (reported bug)               43
>    SKIP (missing feature)             48
>    SKIP (run-time condition)          29
>
>    Failed tests:
>       libinterp/corefcn/pr-output.cc-tst
>       libinterp/parse-tree/oct-parse.yy-tst
>       general/logspace.m
>       miscellaneous/tar.m
>       plot/util/copyobj.m
>       plot/util/hgsave.m
>       sparse/eigs.m
>
> Is the goal to have a clean test suite baseline for the release?
>
> Cheers,
> Andrew
>
>

An update on these:

  libinterp/corefcn/pr-output.cc-tst
  libinterp/parse-tree/oct-parse.yy-tst
    - Setting "format long" before running the test suite breaks these:
https://savannah.gnu.org/bugs/index.php?55539

  general/logspace.m
    - Probable c++ library bug: https://savannah.gnu.org/bugs/?55538

  miscellaneous/tar.m
    - This is lack of BSD tar support.
https://savannah.gnu.org/bugs/index.php?53695. Patch submitted.

  plot/util/copyobj.m
  plot/util/hgsave.m
    - These were due to a bad Ghostscript installation on my machine.
Not an Octave issue.

  sparse/eigs.m
    - Reported: https://savannah.gnu.org/bugs/index.php?55534. Possible
Apple Accelerate veclib bug. Running under OpenBLAS passes.

Cheers,
Andrew