pending interval-3.0.0 release

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
19 messages Options
Reply | Threaded
Open this post in threaded view
|

pending interval-3.0.0 release

Olaf Till-2
Oliver,

trying to rebuild the release I get stuck with that:

git clone https://github.com/oheim/ITF1788.git "build/ITF1788"
Cloning into 'build/ITF1788'...
remote: Counting objects: 1769, done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 1769 (delta 0), reused 4 (delta 0), pack-reused 1761
Receiving objects: 100% (1769/1769), 394.30 KiB | 0 bytes/s, done.
Resolving deltas: 100% (881/881), done.
Checking connectivity... done.
Compiling src/test/abs_rev.itl ...
Traceback (most recent call last):
  File "/usr/lib/python3.4/runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.4/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/olaf/devel/octave-forge/package-repositories/all/octave-forge/interval/build/ITF1788/itf1788/__main__.py", line 27, in <module>
    from . import dslparser
  File "/home/olaf/devel/octave-forge/package-repositories/all/octave-forge/interval/build/ITF1788/itf1788/dslparser.py", line 29, in <module>
    import ply.lex as lex
ImportError: No module named 'ply'


Probably a typo in the current revision of this external repo.

I admit I'm somehow not happy with requiring to clone a 'non standard'
external repo for package building. Is there no other solution? What
is this python code supposed to do in building the interval package?

Olaf

--
public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: pending interval-3.0.0 release

Oliver Heimlich
On 19.08.2017 18:32, Olaf Till wrote:
> Oliver,
>
> trying to rebuild the release I get stuck with that:
>
> git clone https://github.com/oheim/ITF1788.git "build/ITF1788"
> Cloning into 'build/ITF1788'...
...

> Compiling src/test/abs_rev.itl ...
> Traceback (most recent call last):
>   File "/usr/lib/python3.4/runpy.py", line 170, in _run_module_as_main
>     "__main__", mod_spec)
>   File "/usr/lib/python3.4/runpy.py", line 85, in _run_code
>     exec(code, run_globals)
>   File "/home/olaf/devel/octave-forge/package-repositories/all/octave-forge/interval/build/ITF1788/itf1788/__main__.py", line 27, in <module>
>     from . import dslparser
>   File "/home/olaf/devel/octave-forge/package-repositories/all/octave-forge/interval/build/ITF1788/itf1788/dslparser.py", line 29, in <module>
>     import ply.lex as lex
> ImportError: No module named 'ply'

you will need to install the python modules listed at the end of this file.
https://github.com/oheim/ITF1788/blob/master/setup.py

> I admit I'm somehow not happy with requiring to clone a 'non standard'
> external repo for package building. Is there no other solution? What
> is this python code supposed to do in building the interval package?

I am also not satisfied with the current solution and am open to better
alternatives.  Would it be better to copy the python code into a
bootstrap folder of the Octave package repository?

The python code is used to convert the src/test/*.itl files into Octave
code, which will then be run and produce the file inst/test/itl.mat,
which contains the test data from the itl files.

The code generation requires the python code, which is developed at
Github, and depends on python and some python modules.

If you want to skip that part of the reproduction, you could use the
itl.mat from the tarball, which I have prepared for the release tracker.

Oliver

Reply | Threaded
Open this post in threaded view
|

Re: pending interval-3.0.0 release

Oliver Heimlich
On 19.08.2017 18:48, Oliver Heimlich wrote:

> On 19.08.2017 18:32, Olaf Till wrote:
>> Oliver,
>>
>> trying to rebuild the release I get stuck with that:
>>
>> git clone https://github.com/oheim/ITF1788.git "build/ITF1788"
>> Cloning into 'build/ITF1788'...
> ...
>> Compiling src/test/abs_rev.itl ...
>> Traceback (most recent call last):
>>   File "/usr/lib/python3.4/runpy.py", line 170, in _run_module_as_main
>>     "__main__", mod_spec)
>>   File "/usr/lib/python3.4/runpy.py", line 85, in _run_code
>>     exec(code, run_globals)
>>   File "/home/olaf/devel/octave-forge/package-repositories/all/octave-forge/interval/build/ITF1788/itf1788/__main__.py", line 27, in <module>
>>     from . import dslparser
>>   File "/home/olaf/devel/octave-forge/package-repositories/all/octave-forge/interval/build/ITF1788/itf1788/dslparser.py", line 29, in <module>
>>     import ply.lex as lex
>> ImportError: No module named 'ply'
>
> you will need to install the python modules listed at the end of this file.
> https://github.com/oheim/ITF1788/blob/master/setup.py
>
>> I admit I'm somehow not happy with requiring to clone a 'non standard'
>> external repo for package building. Is there no other solution? What
>> is this python code supposed to do in building the interval package?
>
> I am also not satisfied with the current solution and am open to better
> alternatives.  Would it be better to copy the python code into a
> bootstrap folder of the Octave package repository?
>
> The python code is used to convert the src/test/*.itl files into Octave
> code, which will then be run and produce the file inst/test/itl.mat,
> which contains the test data from the itl files.
>
> The code generation requires the python code, which is developed at
> Github, and depends on python and some python modules.
>
> If you want to skip that part of the reproduction, you could use the
> itl.mat from the tarball, which I have prepared for the release tracker.

I have now put the generated test data under version control and it is
possible to run the package from source or make a release tarball
without these special dependencies.

https://sourceforge.net/p/octave/interval/ci/43284e87a6eb8e80c9a8cbc2f0cb3b07136380af/

Oliver


Reply | Threaded
Open this post in threaded view
|

Re: pending interval-3.0.0 release

Olaf Till-2
On Sun, Aug 20, 2017 at 11:00:41AM +0200, Oliver Heimlich wrote:
> I have now put the generated test data under version control and it is
> possible to run the package from source or make a release tarball
> without these special dependencies.

I know I'm delaying the release, but I don't think that putting a
generated file additionally to the sources under version control is a
good idea.

The issue I saw was not so much with the difficulty (installing some
tools) to reproduce the release.

Rather, we should assure that generating the distributed data from
source is always possible. Since we distribute the binary file
'itl.mat', we must distribute its 'preferred form' of editable
sources.

These sources could be the corresponding .tst files, which currently
are only temporary files during the build. But though they are
editable, it's their .itl sources which are meant to be edited. We
distribute these .itl sources. The problem with this is that the tool
for converting .itl to .tst is not in standard distributions (?), but
only in an external repo. I think if we distribute the .itl files, we
also must distribute the conversion tool (and not rely on its
availability in an external repo). I am not a lawyer, but as long as
we distribute the package with no commercial intent, it could suffice
to clone the external repo at Octave Forge and to put a hint into the
distributed package where the tool is available at Octave Forge.

BTW it could be good to checkout a certain version of the cloned repo
during building (if I havn't overlooked that this is already done).

Olaf

--
public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: pending interval-3.0.0 release

Oliver Heimlich
On 20.08.2017 20:47, Olaf Till wrote:

> On Sun, Aug 20, 2017 at 11:00:41AM +0200, Oliver Heimlich wrote:
>> I have now put the generated test data under version control and it is
>> possible to run the package from source or make a release tarball
>> without these special dependencies.
>
> I know I'm delaying the release, but I don't think that putting a
> generated file additionally to the sources under version control is a
> good idea.
>
> The issue I saw was not so much with the difficulty (installing some
> tools) to reproduce the release.

There have been other reasons to put it under version control as well
(see the commit message), but I understand that it doesn't solve the
general problem.

> Rather, we should assure that generating the distributed data from
> source is always possible. Since we distribute the binary file
> 'itl.mat', we must distribute its 'preferred form' of editable
> sources.
>
> These sources could be the corresponding .tst files, which currently
> are only temporary files during the build. But though they are
> editable, it's their .itl sources which are meant to be edited. We
> distribute these .itl sources. The problem with this is that the tool
> for converting .itl to .tst is not in standard distributions (?), but
> only in an external repo. I think if we distribute the .itl files, we
> also must distribute the conversion tool (and not rely on its
> availability in an external repo). I am not a lawyer, but as long as
> we distribute the package with no commercial intent, it could suffice
> to clone the external repo at Octave Forge and to put a hint into the
> distributed package where the tool is available at Octave Forge.

The preferred form for editing is in the .itl files.  The intermediate
.tst-files are barely human readable.  On the other hand, the itl.mat
file can also be edited easily, since it is just a data file, which can
be processed with Octave.

The main difference at this point is that the .itl files are portable
(not Octave specific) and the .mat file can only be used with Octave.

That external tool (and the .itl files) is meant to be used by interval
arithmetic library developers to share the test data.  It is a very
specific use case, which is unique in the Octave community and maybe
affects only a few dozen users worldwide.  I guess you won't find that
tool largerly redistributed.

Instead of hosting it on OF we could as well bundle it in the interval
repository.  Another possibility could be to publish it in some official
Python repository, so you can pull it from there instead of Github.
Would that end your worries?

> BTW it could be good to checkout a certain version of the cloned repo
> during building (if I havn't overlooked that this is already done).

Yes, the Makefile should clone a particular revision.  I have extended
the Makefile accordingly.

Oliver

Reply | Threaded
Open this post in threaded view
|

Re: pending interval-3.0.0 release

Oliver Heimlich
On 20.08.2017 21:27, Oliver Heimlich wrote:
> On 20.08.2017 20:47, Olaf Till wrote:
>> On Sun, Aug 20, 2017 at 11:00:41AM +0200, Oliver Heimlich wrote:
>>> I have now put the generated test data under version control and it is
>>> possible to run the package from source or make a release tarball
>>> without these special dependencies.

>> ... These sources could be the corresponding .tst files, which currently
>> are only temporary files during the build. But though they are
>> editable, it's their .itl sources which are meant to be edited. We
>> distribute these .itl sources. The problem with this is that the tool
>> for converting .itl to .tst is not in standard distributions (?), but
>> only in an external repo. I think if we distribute the .itl files, we
>> also must distribute the conversion tool (and not rely on its
>> availability in an external repo). I am not a lawyer, but as long as
>> we distribute the package with no commercial intent, it could suffice
>> to clone the external repo at Octave Forge and to put a hint into the
>> distributed package where the tool is available at Octave Forge.

> That external tool (and the .itl files) is meant to be used by interval
> arithmetic library developers to share the test data.  It is a very
> specific use case, which is unique in the Octave community and maybe
> affects only a few dozen users worldwide.  I guess you won't find that
> tool largerly redistributed.

The GPL FAQ is an interesting read on that topic.  According to the FAQ
it suffices to put links to the external repository, see
https://www.gnu.org/licenses/gpl-faq.html#SourceAndBinaryOnDifferentSites
and https://www.gnu.org/licenses/gpl-faq.html#SourceInCVS

The remaining questions are:
 - Do we believe that Github will be available while we distribute the
software on Octave Forge?  If Github closes its service, we may still
clone that repository.
 - Do we believe that the repository might be deleted and vanish some
day?  Then we would want to clone it to have a backup.

Oliver

Reply | Threaded
Open this post in threaded view
|

Re: pending interval-3.0.0 release

Olaf Till-2
On Sun, Aug 20, 2017 at 10:27:08PM +0200, Oliver Heimlich wrote:
> The GPL FAQ is an interesting read on that topic.  According to the FAQ
> it suffices to put links to the external repository, see
> https://www.gnu.org/licenses/gpl-faq.html#SourceAndBinaryOnDifferentSites
> and https://www.gnu.org/licenses/gpl-faq.html#SourceInCVS

But we don't distribute a binary, but something which is meant to be a
source package. If the latter contains ('external') binary files
without their sources (or with uncertain means to convert the sources
to the binaries), this may not be noticed by the user, leaving him to
believe that he already has everything to build from source.

Treating this in the regular way would mean not to include the
'external' binary files into our source package, but to have them as a
build dependency. So the user has to install them, and thus can find
out, if he wants to, if he is able to build them from source.

If, for convenience, we include external files into the source
package, we may 'mask' difficulties in getting their source or in
building them from source. So I think we should include them in the
form of their source and, if necessary, provide access to their build
mechanism.

Package releases can, additionally to these sources, contain files
pre-built from them, thus avoiding a complicated build procedure.

Olaf

--
public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: pending interval-3.0.0 release

Olaf Till-2
In reply to this post by Oliver Heimlich
On Sun, Aug 20, 2017 at 09:27:38PM +0200, Oliver Heimlich wrote:
> On the other hand, the itl.mat
> file can also be edited easily, since it is just a data file, which can
> be processed with Octave.

This would possibly solve the problem, but I'm not sure that a data
file is really so easily edited.

> Instead of hosting it on OF we could as well bundle it in the interval
> repository.

Yes, if you think you can do this...

> Another possibility could be to publish it in some official
> Python repository, so you can pull it from there instead of Github.
> Would that end your worries?

Yes, this would also end this worry.

Unfortunately, there is a similar one: the external crlibm.mat is
distributed without sources.

See also my reply to your next post in this thread.

Olaf

--
public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: pending interval-3.0.0 release

Oliver Heimlich
On 21.08.2017 04:20, Olaf Till wrote:
> On Sun, Aug 20, 2017 at 09:27:38PM +0200, Oliver Heimlich wrote:
>> On the other hand, the itl.mat
>> file can also be edited easily, since it is just a data file, which can
>> be processed with Octave.
>
> This would possibly solve the problem, but I'm not sure that a data
> file is really so easily edited.

It is a file with numeric data.  Editing it with Octave (for the use in
this particular Octave package) is easier that editing the original
file, since you have more options:  You can add data by concatenating
the vectors and structs.  You can convert it into other formats using
octave's output functions.  You can query it with indexing expressions,
which is not even possible with a text editor on the original source.

For the Octave user, the itl.mat is everything he will ever need.  I
only want to give him the .itl files to document the license details of
itl.mat (and to advertise that external software).  We could as well not
redistribute the .itl source files at all and document the copyright in
COPYING explicitly.


>> Instead of hosting it on OF we could as well bundle it in the interval
>> repository.
>
> Yes, if you think you can do this...

It is just a matter of putting a particular version of the python
scripts into a folder src/ITF1788.  Then, the user will receive that
software together with the Octave packages' release.  I am hesitating to
do this, because (1) I find this not important for the Octave package
user (see above).  The external software is only important to produce
test data in a format different from itl.mat for use in an unrelated
interval library.  And (2), I want to keep that external software
separate from the interval package to advertise its use in other
interval libraries.  Having its source in the interval package's
repository could leave the impression that it is part of the interval
package and thus GPL'd and then some potential users would refrain from
using it, see for example

https://github.com/JuliaIntervals/ValidatedNumerics.jl/issues/65


>> Another possibility could be to publish it in some official
>> Python repository, so you can pull it from there instead of Github.
>> Would that end your worries?
>
> Yes, this would also end this worry.

I'd prefer this compared to bundling the software.  However, I have
little Python experience and first have to find out about publication.
IIRC, the setup.py is important for that.  Last time I checked, there
were at least two competing methods to do this (it's Python!).

Please look at the COPYING file in the tarball.  Currently, it tells the
user that if he wanted to compile the test data for another interval
library (or recompile the test data), he can find the compiler at a
Github repository.  I find the case of recompilation rather unimportant
(see above).  Should we really insist on having that generator available
this way?  If we develop that idea further, does it mean that I also
have to include the maintainer Makefile in the release tarball?  Even if
you have that external dependency, it is not trivial to use it.  So, one
could consider the Makefile rules to be part of the “missing source”.


> Unfortunately, there is a similar one: the external crlibm.mat is
> distributed without sources.

This is a different matter.  The crlibm.mat has been converted manually
(by me).  If you edited the original files, from which I have derived
this work, there is no way to apply that change to the crlibm.mat.  So,
I don't consider the original files as source files (=preferred form of
makeing changes) anymore.  That's why they are not part of the release
tarball.

Oliver

Reply | Threaded
Open this post in threaded view
|

Re: pending interval-3.0.0 release

Olaf Till-2
On Mon, Aug 21, 2017 at 06:07:43AM +0200, Oliver Heimlich wrote:

> On 21.08.2017 04:20, Olaf Till wrote:
> > On Sun, Aug 20, 2017 at 09:27:38PM +0200, Oliver Heimlich wrote:
> >> On the other hand, the itl.mat
> >> file can also be edited easily, since it is just a data file, which can
> >> be processed with Octave.
> >
> > This would possibly solve the problem, but I'm not sure that a data
> > file is really so easily edited.
>
> It is a file with numeric data.  Editing it with Octave (for the use in
> this particular Octave package) is easier that editing the original
> file, since you have more options:  You can add data by concatenating
> the vectors and structs.  You can convert it into other formats using
> octave's output functions.  You can query it with indexing expressions,
> which is not even possible with a text editor on the original source.
>
> For the Octave user, the itl.mat is everything he will ever need.  I
> only want to give him the .itl files to document the license details of
> itl.mat (and to advertise that external software).  We could as well not
> redistribute the .itl source files at all and document the copyright in
> COPYING explicitly.
Having read the GPL3 section 1 again, I really think we have to
provide the preferred form of making modifications (i.e the .itl
files) and all non-standard scripts (i.e. the python code) to generate
the result.

> >> Instead of hosting it on OF we could as well bundle it in the interval
> >> repository.
> >
> > Yes, if you think you can do this...
>
> It is just a matter of putting a particular version of the python
> scripts into a folder src/ITF1788.  Then, the user will receive that
> software together with the Octave packages' release.  I am hesitating to
> do this, because (1) I find this not important for the Octave package
> user (see above).  The external software is only important to produce
> test data in a format different from itl.mat for use in an unrelated
> interval library.
But it is your preferred format to make modifications for this
package, too.

> And (2), I want to keep that external software
> separate from the interval package to advertise its use in other
> interval libraries.  Having its source in the interval package's
> repository could leave the impression that it is part of the interval
> package and thus GPL'd and then some potential users would refrain from
> using it, see for example
>
> https://github.com/JuliaIntervals/ValidatedNumerics.jl/issues/65

If you fear that, you could put a note at the head of "COPYING",
saying that some code of the ITF1788 module is included into the
package, but the module is normally got from elsewhere, and is not
under the GPL3 but the Apache2 license...

I think now including the code is better than the possibility below.

> >> Another possibility could be to publish it in some official
> >> Python repository, so you can pull it from there instead of Github.
> >> Would that end your worries?
> >
> > Yes, this would also end this worry.
>
> I'd prefer this compared to bundling the software.  However, I have
> little Python experience and first have to find out about publication.
> IIRC, the setup.py is important for that.  Last time I checked, there
> were at least two competing methods to do this (it's Python!).
>
> Please look at the COPYING file in the tarball.  Currently, it tells the
> user that if he wanted to compile the test data for another interval
> library (or recompile the test data), he can find the compiler at a
> Github repository.  I find the case of recompilation rather unimportant
> (see above).  Should we really insist on having that generator available
> this way?  If we develop that idea further, does it mean that I also
> have to include the maintainer Makefile in the release tarball?  Even if
> you have that external dependency, it is not trivial to use it.  So, one
> could consider the Makefile rules to be part of the “missing source”.
So they are, if they are non-trivial... the problem, I think, is that
these rules should then actually go into the 'real' package Makefile
(src/Makefile). They can go under a target which is pre-built for
releases.

> > Unfortunately, there is a similar one: the external crlibm.mat is
> > distributed without sources.
>
> This is a different matter.  The crlibm.mat has been converted manually
> (by me).  If you edited the original files, from which I have derived
> this work, there is no way to apply that change to the crlibm.mat.  So,
> I don't consider the original files as source files (=preferred form of
> makeing changes) anymore.  That's why they are not part of the release
> tarball.

I don't know how you converted it manually. But I think what we are
required to do in this case is to provide the original sources,
necessary scripts (if they are not 'generally available'), and a
script of yours which does what you had done manually.

I agree that the whole thing is exceptionally complicated. An
alternative would be to have all the external matter as build
dependencies, instead of distributing it in the package (but the
disadvantage is obvious, too).

Olaf

--
public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: pending interval-3.0.0 release

Oliver Heimlich
On 21.08.2017 13:08, Olaf Till wrote:

> On Mon, Aug 21, 2017 at 06:07:43AM +0200, Oliver Heimlich wrote:
>> On 21.08.2017 04:20, Olaf Till wrote:
>>> On Sun, Aug 20, 2017 at 09:27:38PM +0200, Oliver Heimlich wrote:
>>>> On the other hand, the itl.mat
>>>> file can also be edited easily, since it is just a data file, which can
>>>> be processed with Octave.
>>>
>>> This would possibly solve the problem, but I'm not sure that a data
>>> file is really so easily edited.
>>
>> It is a file with numeric data.  Editing it with Octave (for the use in
>> this particular Octave package) is easier that editing the original
>> file, since you have more options:  You can add data by concatenating
>> the vectors and structs.  You can convert it into other formats using
>> octave's output functions.  You can query it with indexing expressions,
>> which is not even possible with a text editor on the original source.
>>
>> For the Octave user, the itl.mat is everything he will ever need.  I
>> only want to give him the .itl files to document the license details of
>> itl.mat (and to advertise that external software).  We could as well not
>> redistribute the .itl source files at all and document the copyright in
>> COPYING explicitly.
>
> Having read the GPL3 section 1 again, I really think we have to
> provide the preferred form of making modifications (i.e the .itl
> files) and all non-standard scripts (i.e. the python code) to generate
> the result.

No, I disagree.  They are not the preferred form of making
modifications.  See below.

Even if this would be the case, in COPYING the URL is mentioned, where
the source code can be downloaded, which would also be sufficient.
However, it is not needed.

Also, I own the sole copyright holder for most of these files.  The only
exceptions being some files that have been originally licensed under the
Apache License 2.0, so I don't even have the obligation to provide any
original code form for them if I don't want.  I could just delete any
.itl files from Octave Forge and only put the itl.mat test data there.

>>>> Instead of hosting it on OF we could as well bundle it in the interval
>>>> repository.
>>>
>>> Yes, if you think you can do this...
>>
>> It is just a matter of putting a particular version of the python
>> scripts into a folder src/ITF1788.  Then, the user will receive that
>> software together with the Octave packages' release.  I am hesitating to
>> do this, because (1) I find this not important for the Octave package
>> user (see above).  The external software is only important to produce
>> test data in a format different from itl.mat for use in an unrelated
>> interval library.
>
> But it is your preferred format to make modifications for this
> package, too.

The .itl files are the preferred format for me and only me, because I
want to make sure that the very same test data can be used in any other
interval library as well, not necessarily in Octave.

For everybody else (= licensee), the itl.mat file is perfectly fine to
be edited to be used with this package.

This has been different with earlier releases, where there was no
itl.mat and only a bunch of generated .tst files in the tarball, which
nobody would want to edit.  Starting with interval 3.0.0 this changes.

>> And (2), I want to keep that external software
>> separate from the interval package to advertise its use in other
>> interval libraries.  Having its source in the interval package's
>> repository could leave the impression that it is part of the interval
>> package and thus GPL'd and then some potential users would refrain from
>> using it, see for example
>>
>> https://github.com/JuliaIntervals/ValidatedNumerics.jl/issues/65
...
> I think now including the code is better than the possibility below.

>>>> Another possibility could be to publish it in some official
>>>> Python repository, so you can pull it from there instead of Github.
>>>> Would that end your worries?
>>>
>>> Yes, this would also end this worry.
>>
>> ...  If we develop that idea further, does it mean that I also
>> have to include the maintainer Makefile in the release tarball?  Even if
>> you have that external dependency, it is not trivial to use it.  So, one
>> could consider the Makefile rules to be part of the “missing source”.
>
> So they are, if they are non-trivial... the problem, I think, is that
> these rules should then actually go into the 'real' package Makefile
> (src/Makefile). They can go under a target which is pre-built for
> releases.

Yes, this could be done.  However, I think that it is not needed and
should not be done.

I would rather remove any connection to the python scripts and .itl
files from Octave Forge and produce the test data somewhere else to
import it into the package repository when I get updates from other
interval researchers.

>>> Unfortunately, there is a similar one: the external crlibm.mat is
>>> distributed without sources.
>>
>> This is a different matter.  The crlibm.mat has been converted manually
>> (by me).  If you edited the original files, from which I have derived
>> this work, there is no way to apply that change to the crlibm.mat.  So,
>> I don't consider the original files as source files (=preferred form of
>> makeing changes) anymore.  That's why they are not part of the release
>> tarball.
>
> I don't know how you converted it manually. But I think what we are
> required to do in this case is to provide the original sources,

Why should I have to redistribute the original work when I distribute a
derivative work?

> necessary scripts (if they are not 'generally available'), and a
> script of yours which does what you had done manually.

There is no script and never has been.  Why should I make a script?
This doesn't make sense.

> I agree that the whole thing is exceptionally complicated. An
> alternative would be to have all the external matter as build
> dependencies, instead of distributing it in the package (but the
> disadvantage is obvious, too).

Yes it is complicated and at the beginning I haven't been sure about
what to do.  But after this discussion my opinion is pretty firm that
the current release arrangement is good and I don't see a need to change
anything regarding the test data files.

If you still have objections regarding the .itl files I can offer to
remove them entirely from the repository and only put the static test
data (itl.mat) in the repository.  Then there will be no confusion about
which rest data files to edit in the package release tarball.

Oliver

Reply | Threaded
Open this post in threaded view
|

Re: pending interval-3.0.0 release

Olaf Till-2
On Mon, Aug 21, 2017 at 04:42:07PM +0200, Oliver Heimlich wrote:

> <snip>
> Also, I own the sole copyright holder for most of these files.  The only
> exceptions being some files that have been originally licensed under the
> Apache License 2.0, so I don't even have the obligation to provide any
> original code form for them if I don't want.  I could just delete any
> .itl files from Octave Forge and only put the itl.mat test data there.
> <snip>
> The .itl files are the preferred format for me and only me, because I
> want to make sure that the very same test data can be used in any other
> interval library as well, not necessarily in Octave.
>
> For everybody else (= licensee), the itl.mat file is perfectly fine to
> be edited to be used with this package.
> <snip>
> I would rather remove any connection to the python scripts and .itl
> files from Octave Forge and produce the test data somewhere else to
> import it into the package repository when I get updates from other
> interval researchers.
> <snip>
> > I don't know how you converted it manually. But I think what we are
> > required to do in this case is to provide the original sources,
>
> Why should I have to redistribute the original work when I distribute a
> derivative work?
>
> > necessary scripts (if they are not 'generally available'), and a
> > script of yours which does what you had done manually.
>
> There is no script and never has been.  Why should I make a script?
> This doesn't make sense.
(I left those parts of your post above which seem to be related to my
next remarks.)

I think the argumentation with 'only your' preferred source and 'for
the user it's sufficient...' is a bit adventurous. If we provide GPLed
code, even if you have the sole copyright, we should provide the code
in a way in which this license makes sense.

I believe that the data files are not sufficient as sources, and that
some script to generate them should be provided (this in particular
relates to your last two remarks above -- the script would need to
process the original sources, I think).

> <snip>
> Yes it is complicated and at the beginning I haven't been sure about
> what to do.  But after this discussion my opinion is pretty firm that
> the current release arrangement is good and I don't see a need to change
> anything regarding the test data files.

This means that we can't come to an agreement over this.

But since this is a controversy between two administrators, a solution
could be that you publish the package despite my disapproving.

Olaf

--
public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: pending interval-3.0.0 release

Oliver Heimlich
On 21.08.2017 17:32, Olaf Till wrote:

> On Mon, Aug 21, 2017 at 04:42:07PM +0200, Oliver Heimlich wrote:
>> <snip>
>> Also, I own the sole copyright holder for most of these files.  The only
>> exceptions being some files that have been originally licensed under the
>> Apache License 2.0, so I don't even have the obligation to provide any
>> original code form for them if I don't want.  I could just delete any
>> .itl files from Octave Forge and only put the itl.mat test data there.
>> <snip>
>> The .itl files are the preferred format for me and only me, because I
>> want to make sure that the very same test data can be used in any other
>> interval library as well, not necessarily in Octave.
>>
>> For everybody else (= licensee), the itl.mat file is perfectly fine to
>> be edited to be used with this package.
>> <snip>
>> I would rather remove any connection to the python scripts and .itl
>> files from Octave Forge and produce the test data somewhere else to
>> import it into the package repository when I get updates from other
>> interval researchers.
>> <snip>
>>> I don't know how you converted it manually. But I think what we are
>>> required to do in this case is to provide the original sources,
>>
>> Why should I have to redistribute the original work when I distribute a
>> derivative work?
>>
>>> necessary scripts (if they are not 'generally available'), and a
>>> script of yours which does what you had done manually.
>>
>> There is no script and never has been.  Why should I make a script?
>> This doesn't make sense.
>
> (I left those parts of your post above which seem to be related to my
> next remarks.)
>
> I think the argumentation with 'only your' preferred source and 'for
> the user it's sufficient...' is a bit adventurous. If we provide GPLed
> code, even if you have the sole copyright, we should provide the code
> in a way in which this license makes sense.
>
> I believe that the data files are not sufficient as sources, and that
> some script to generate them should be provided (this in particular
> relates to your last two remarks above -- the script would need to
> process the original sources, I think).

I have made a derivative work and want to distribute it, which is
perfectly fine.  You want to be able to reproduce my work with a script.

It might actually be possible technically.  In one case I have to bundle
a lot of software, which I have no interest in (it could just as well be
downloaded from Github).  In the other case you request me to automate
what I have done afterwards.

We are not even talking about software.  It's static data.  We are
talking about the data format which you don't like.  A data format which
is perfect for an Octave user (.mat).

>> <snip>
>> Yes it is complicated and at the beginning I haven't been sure about
>> what to do.  But after this discussion my opinion is pretty firm that
>> the current release arrangement is good and I don't see a need to change
>> anything regarding the test data files.
>
> This means that we can't come to an agreement over this.
>
> But since this is a controversy between two administrators, a solution
> could be that you publish the package despite my disapproving.

Yes it seems so, sorry for becoming a little bit emotional above...

I'd rather consider myself a package maintainer in this case and would
not want to take the role of an admin due to conflicting interests.

Maybe Julien can decide this issue unless no other community members
want to throw in further arguments.

Oliver

Reply | Threaded
Open this post in threaded view
|

Re: pending interval-3.0.0 release

Mike Miller-4
On Mon, Aug 21, 2017 at 17:57:30 +0200, Oliver Heimlich wrote:
> We are not even talking about software.  It's static data.  We are
> talking about the data format which you don't like.  A data format which
> is perfect for an Octave user (.mat).

Does the data change over time or is it pretty much a constant table of
data that might as well have been copied from a book?

FWIW, in previous projects where I have had to create or build large
test vectors, where the test data doesn't actually change at all or
often, I have usually written a script or set of comments to describe
how the data was generated, but commit the test vectors themselves to
version control to avoid having to regenerate the constant data on every
build. Also to avoid adding build dependencies on unrelated tools.

IMHO there is a legitimate trade-off between convenience and ability to
build something from its ultimate source.

--
mike

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: pending interval-3.0.0 release

Oliver Heimlich
On 21.08.2017 18:37, Mike Miller wrote:
> On Mon, Aug 21, 2017 at 17:57:30 +0200, Oliver Heimlich wrote:
>> We are not even talking about software.  It's static data.  We are
>> talking about the data format which you don't like.  A data format which
>> is perfect for an Octave user (.mat).
>
> Does the data change over time or is it pretty much a constant table of
> data that might as well have been copied from a book?

The existing data mainly comes from free software projects, which are no
longer actively maintained.

  * https://scm.gforge.inria.fr/anonscm/git/metalibm/crlibm.git/
  * https://github.com/nehmeier/libieeep1788
  * https://gforge.inria.fr/frs/?group_id=157

I don't expect to get any updates from them.  Also, I haven't yet
successfully convinced others to contribute test data.  Probably the
existing test data is quite comprehensive already.

So if I want to update / extend the test data, I am pretty much on my own.

> FWIW, in previous projects where I have had to create or build large
> test vectors, where the test data doesn't actually change at all or
> often, I have usually written a script or set of comments to describe
> how the data was generated, but commit the test vectors themselves to
> version control to avoid having to regenerate the constant data on every
> build. Also to avoid adding build dependencies on unrelated tools.

That's pretty much the same reason why I decided to put the test data
under version control (plus the build dependencies would be very exotic
in this case).

Oliver

Reply | Threaded
Open this post in threaded view
|

Re: pending interval-3.0.0 release

jbect
In reply to this post by Oliver Heimlich
Le 21/08/2017 à 17:57, Oliver Heimlich a écrit :

> On 21.08.2017 17:32, Olaf Till wrote:
>> This means that we can't come to an agreement over this.
>>
>> But since this is a controversy between two administrators, a solution
>> could be that you publish the package despite my disapproving.
> Yes it seems so, sorry for becoming a little bit emotional above...
>
> I'd rather consider myself a package maintainer in this case and would
> not want to take the role of an admin due to conflicting interests.
>
> Maybe Julien can decide this issue unless no other community members
> want to throw in further arguments.

Hi Oliver,

As you know (but I should have announced it on the list, so now is a
good opportunity) I have stopped being a member of the "OF admin team"
for some time now. I don't even have admin rights anymore.

Moreover, concerning the particular issue at hand, I admit that I don't
fully understand it. Others on this list will probably be more qualified
to help you guys settle this.

So, all in all, I don't feel more legitimate than any other OF dev to
make a decision regarding this issue.

IANAL: If you, as the main developer of the package, consider this
"itl.mat" file as the preferred form for making (future) modifications
to it, then I think that you are allowed to consider it as "source",
even if it was (originally) produced form some other data and/or code.  
In this case, I think (more or less as Mike said) that you should at
least describe (roughly) how and (precisely) from which sources this
file was produced, especially if the file is considered a derivative
work of some GPL'd project.

HTH,

Julien.

Reply | Threaded
Open this post in threaded view
|

Re: pending interval-3.0.0 release

Oliver Heimlich
On 21.08.2017 23:11, Julien Bect wrote:
> Le 21/08/2017 à 17:57, Oliver Heimlich a écrit :
>> On 21.08.2017 17:32, Olaf Till wrote:
>>> This means that we can't come to an agreement over this.
>>>
>>> But since this is a controversy between two administrators, a solution
>>> could be that you publish the package despite my disapproving.

> IANAL: If you, as the main developer of the package, consider this
> "itl.mat" file as the preferred form for making (future) modifications
> to it, then I think that you are allowed to consider it as "source",
> even if it was (originally) produced form some other data and/or code.

In a nutshell, I have obtained data (from various sources) with LGPL and
Apache 2.0 licenses.  The original form has been either a proprietary
raw data format (see [1] or [2]), or data that is interleaved in unit
test code (see [3]).

In either case, I have spent nights to extract the data (vectors of
floating point numbers).  This mainly involved a lot of copy-pasting and
search-replacing and read-parsing.  It is not something that I have
automated or would find useful to automate.  It has been a one-off
manual work that needed to be done to get the data into accessible form.

Since the data stayed the same during that process, but eventually got
stored in a different format, I consider my work product a derivative
work of the original files.  It's like migrating a program from one
language into another language without using a compiler.

In one case I have saved my work product in the crlibm.mat file.  In
other cases I have used an intermediate format to save my work product
(domain specific language, see [4] for an example) and use exotic python
scripts to eventually produce the itl.mat file for use in the interval
package.

To sum up, in either case there is no way to automate conversion from
the very original source into the .mat files currently.  Also, I don't
find it worthwhile to implement one.  The work to make the data
accessible has already been done.

I don't see a reason to include any original sources for the derivated
test data in the tarball of the interval package.  Also, I don't have to
provide any of the tools or methods that I have used for conversion.

 * The original authors have been attributed in the package manual [5].
 * In the package's COPYGING file [6] the copyright details of
   the .mat files are explained (since the .mat files don't contain
   copyright meta data on their own).

Currently, the release tarball also contains the intermediate format
(.itl files) and rough instructions plus a reference to the python code
on Github.  This is meant for the case that anybody would find the
intermediate format equally useful and would want to use the .itl files
for something else (i. e. not for this package).

In 2015, I have decided to put the intermediate .itl format into the
tarball [7], because back then the tests have been a bulk of generated
.tst code (which clearly is not the preferred form to edit anything for
anybody).  Back then, it would have made sense to also include the
python scripts.  Now, with interval 3.0.0, there is no more generated
.tst code, only a itl.mat file with data and the generated external
tests have been replaced with proper BISTs.  My point is, that the user
doesn't need the .itl files anymore (and thus not the python scripts).

This leads me to the conclusion that I would eventually want to remove
the .itl files from this project entirely and probably publish them
together with the python scripts somewhere else.  However, this can
still be done after this release.  I don't see this as a blocking issue.
 To put it in a positive way: the situation has already been improved
compared to the previous release.

I hope that this explanation suffices for now.  I will publish the
release and we can maybe come to an agreement later and fix any open
issues in the next version.  I would have waited and discussed this
longer, but the release contains the results of a GSoC project (this is
GSoC's final week) and I don't want to see them getting delayed further.

Oliver



[1]
https://sourceforge.net/p/octave/interval/ci/default/tree/src/crlibm/tests/acos.testdata
[2] https://scm.gforge.inria.fr/anonscm/svn/mpfi/trunk/mpfi/tests/acos.dat
[3]
https://github.com/nehmeier/libieeep1788/blob/1f10b896ff532e95818856614ab3073189e81199/test/p1788/infsup/test_integration_mpfr_bin_ieee754_flavor_elem.cpp#L401
[4]
https://sourceforge.net/p/octave/interval/ci/default/tree/src/test/mpfi.itl
[5] https://octave.sourceforge.io/interval/package_doc/Acknowledgments.html
[6]
https://sourceforge.net/p/octave/interval/ci/default/tree/doc/COPYING.texinfo
[7]
http://lists.gnu.org/archive/html/octave-maintainers/2015-03/msg00174.html

Reply | Threaded
Open this post in threaded view
|

Re: pending interval-3.0.0 release

Mike Miller-4
On Tue, Aug 22, 2017 at 01:21:37 +0200, Oliver Heimlich wrote:
> This leads me to the conclusion that I would eventually want to remove
> the .itl files from this project entirely and probably publish them
> together with the python scripts somewhere else.  However, this can
> still be done after this release.  I don't see this as a blocking issue.
>  To put it in a positive way: the situation has already been improved
> compared to the previous release.

Thanks for explaining the history and context. For my part I think your
approach looks good, and I don't see any reason to not distribute the
test vectors as a mat file the way you are doing, and publish the
intermediate files and scripts separately.

--
mike

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: pending interval-3.0.0 release

Olaf Till-2
On Mon, Aug 21, 2017 at 04:51:07PM -0700, Mike Miller wrote:

> On Tue, Aug 22, 2017 at 01:21:37 +0200, Oliver Heimlich wrote:
> > This leads me to the conclusion that I would eventually want to remove
> > the .itl files from this project entirely and probably publish them
> > together with the python scripts somewhere else.  However, this can
> > still be done after this release.  I don't see this as a blocking issue.
> >  To put it in a positive way: the situation has already been improved
> > compared to the previous release.
>
> Thanks for explaining the history and context. For my part I think your
> approach looks good, and I don't see any reason to not distribute the
> test vectors as a mat file the way you are doing, and publish the
> intermediate files and scripts separately.
The explanation has convinced me also that the release is ok.

Olaf

--
public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net

signature.asc (836 bytes) Download Attachment