OctConf 2018, second day -- a lot of discussions and some work done

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

OctConf 2018, second day -- a lot of discussions and some work done

Carnë Draug
Hi everyone

For those that couldn't come today, we continue to keep you informed
of what's going on OctConf.  Here's a short report of today's meeting.

We got some work done in the morning, of special interest the merging
of two GSoC projects: Michele Ginesi work in special functions, and
Cristiano Dorigo on improving iterative methods.

In the afternoon we had plenty of discussion.  Notes were added to the
wiki as discussions happened [1] but for a short report in email, the
main things are that:

  1) future changes in Octave C++ codebase will have to be paired with
  corresponding doxygen documentation;

  2) core will start to include a subset of packages, the first of
  which will be for statistics.

There was also plenty of discussion about better supporting external
Octave packages.  We added the option to install packages from
external URI via pkg so things like this will now work:

    pkg install https://bitbucket.org/KaKiLa/gpml/downloads/gpml-4.0.0.tar.gz

[1] https://wiki.octave.org/OctConf_2018_Notes#Tuesday
[2] http://hg.savannah.gnu.org/hgweb/octave/rev/ca43264971ea


-----------------------------------------
Join us March 12-15 at CERN near Geneva
Switzerland for OctConf 2018.  More info:
https://wiki.octave.org/OctConf_2018
-----------------------------------------
Reply | Threaded
Open this post in threaded view
|

Re: OctConf 2018, second day -- a lot of discussions and some work done

Olaf Till-2
On Tue, Mar 13, 2018 at 11:51:53PM +0100, Carnë Draug wrote:
>   2) core will start to include a subset of packages, the first of
>   which will be for statistics.

A short time ago Rik requested the statistics package to provide
statistics functions which will be removed from core Octave. This has
been done, by providing each of them in case they are detected to be
lacking in core Octave. Including the statistics package into core
Octave seems to go in the opposite direction. Or don't I see what in
detail is meant by 'include'? And what is the rationale?

Olaf

--
public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: OctConf 2018, second day -- a lot of discussions and some work done

Oliver Heimlich
Am 14. März 2018 20:11:47 MEZ schrieb Olaf Till <[hidden email]>:

>On Tue, Mar 13, 2018 at 11:51:53PM +0100, Carnë Draug wrote:
>>   2) core will start to include a subset of packages, the first of
>>   which will be for statistics.
>
>A short time ago Rik requested the statistics package to provide
>statistics functions which will be removed from core Octave. This has
>been done, by providing each of them in case they are detected to be
>lacking in core Octave. Including the statistics package into core
>Octave seems to go in the opposite direction. Or don't I see what in
>detail is meant by 'include'? And what is the rationale?
>
>Olaf
The rough idea is to eventually have statistics released together with core. It will still be a loadable package (you must use pkg load). It will no longer be released on OF.

The main difference will be on the developer side, e.g. which repository will be used (which primarily affects how much attention it gets from core devs). Also the package no longer will need to support multiple versions of Octave core.

Another plan is to target Matlab compatibility. So there might be additional work with moving incompatible functions out of statistics into a new(?) package.

Many details still have to be decided.

Oliver


signature.asc (515 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: OctConf 2018, second day -- a lot of discussions and some work done

Carnë Draug
In reply to this post by Olaf Till-2
On 14 March 2018 at 20:11, Olaf Till <[hidden email]> wrote:
> On Tue, Mar 13, 2018 at 11:51:53PM +0100, Carnë Draug wrote:
>>   2) core will start to include a subset of packages, the first of
>>   which will be for statistics.
>
> A short time ago Rik requested the statistics package to provide
> statistics functions which will be removed from core Octave. This has
> been done, by providing each of them in case they are detected to be
> lacking in core Octave. Including the statistics package into core
> Octave seems to go in the opposite direction.

Not really.  That move was about moving the functions from the core
functions into a statistics package and they will remain in a package.
It's just that this will be a core package that is included in the
Octave source releases.

> Or don't I see what in
> detail is meant by 'include'? And what is the rationale?

There were multiple reasons raised, the one I agree the most is the
recognition that this packages are too important and are the most
used.  So much so, that they should be distributed with Octave as core
packages (like python core packages for example).  Despite their
importance, their development does not get as much attention as core.
By having them as core packages, they will get the testing from the
more involved users which routinely test octave development versions.

For my personal case, if it ever gets around to be done for the image
package it would simplify maintenance since having to support so many
octave versions makes things a lot more complex.

Carnë

Reply | Threaded
Open this post in threaded view
|

Re: OctConf 2018, second day -- a lot of discussions and some work done

Juan Pablo Carbajal-2
There is a nuance that was not yet mentioned here and that is still unresolved.

There are functions in statistics (and other potential pkgs for core)
that are not in matlab and are very domain specific.
Will this be included in core as well? During the discussions it
seemed to me that the implicit answer is NO. This means that there
will be two statistics (or other) packages.
The one that goes to core, which the user will be aware of only when
they does pkg load statistics.
The other one that keeps all the "extended functionality" which the
user will need to be more aware of because they needs to download,
install, and load it.

How do we solve name clashing? Do we rename packages, like
statistics_extra or whatever? Do we some tricks with the paths?

I guess answers to these questions will start emerging while doing,
but it is worth having them in mind.

Reply | Threaded
Open this post in threaded view
|

Re: OctConf 2018, second day -- a lot of discussions and some work done

Olaf Till-2
A few further thoughts:

- Even with the current state, with a package outside the Octave
  distribution, supporting only the newest stable Octave version is
  possible. Supporting more Octave versions is done for the benefit of
  users. With the package in core, this additional benefit is
  lost. (Meaning: If a user wants the newest package version, he has
  to install the newest Octave version -- probably from source, except
  on Windows.)

- Maintaining a package in Octave core excludes (more or less) those
  without write access to core.

The solution of having corresponding packages inside and outside of
the Octave distribution seems impractial to me.

Does the mere fact that a package is hosted at OF instead of core
Octave hinder the involvement of core Octave developers? I don't think
so -- I think the main effect of the change will probably boil down to
spare the effort of supporting more than one Octave version. As I
said, this effort could be spared with the current state as well...

These are only thoughts, no determined opposition. I'm curious what
will turn out in the end.

Olaf

--
public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

maintaining some packages at core Octave (was: Re: OctConf 2018, second day -- a lot of discussions and some work done)

Olaf Till-2
Having thought a bit more about this, it makes me uneasy...

I think with such a change we'd need to reconsider the policy of
Octave Forge: Currently, the 'community' group of packages is meant
for coordinating development with Octave, at least potentially. The
proposed change would mean that Octave developers prefer a
coordination in a different way.

And I've got the feeling that it may be de-motivating for Octave Forge
maintainers to know that they could be left out of the business at any
moment by a shift of maintanance of their package to Octave core...

Olaf

--
public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: maintaining some packages at core Octave (was: Re: OctConf 2018, second day -- a lot of discussions and some work done)

Juan Pablo Carbajal-2
> I think with such a change we'd need to reconsider the policy of
> Octave Forge: Currently, the 'community' group of packages is meant
> for coordinating development with Octave, at least potentially. The
> proposed change would mean that Octave developers prefer a
> coordination in a different way.

I am with Olaf in this. Indeed it seems to me that at some point we
wanted to bring OF closer to core to then "move functions to core" (as
in the scripts folder).
But now we actually want to have a caste system of packages where we
have core and OF packages.

Also I tend to agree in the comment regarding developers attention.
Can't the core developers pay attention to OF?
We could have 'core','community' and 'external' packages as packages
categories in OF, in which 'core' has all the core development
standards (including a named branch "stable" used as in core). As an
extra benefit, delineating the criteria to classify as a 'core'
package in OF might go in the same direction as providing better
documentation for developers.

This latter solution still runs into the "naming" problem, but at
least this will be handle din one place, namely OF.

JPi

Reply | Threaded
Open this post in threaded view
|

Re: maintaining some packages at core Octave (was: Re: OctConf 2018, second day -- a lot of discussions and some work done)

Doug Stewart-4


On Fri, Mar 16, 2018 at 8:27 AM, Juan Pablo Carbajal <[hidden email]> wrote:
> I think with such a change we'd need to reconsider the policy of
> Octave Forge: Currently, the 'community' group of packages is meant
> for coordinating development with Octave, at least potentially. The
> proposed change would mean that Octave developers prefer a
> coordination in a different way.

I am with Olaf in this. Indeed it seems to me that at some point we
wanted to bring OF closer to core to then "move functions to core" (as
in the scripts folder).
But now we actually want to have a caste system of packages where we
have core and OF packages.

Also I tend to agree in the comment regarding developers attention.
Can't the core developers pay attention to OF?
We could have 'core','community' and 'external' packages as packages
categories in OF, in which 'core' has all the core development
standards (including a named branch "stable" used as in core). As an
extra benefit, delineating the criteria to classify as a 'core'
package in OF might go in the same direction as providing better
documentation for developers.
I agree with JPi on the three categories. 

This latter solution still runs into the "naming" problem, but at
least this will be handle din one place, namely OF.

JPi


I also like the idea of packaging these "core" packages with the released version
of octave, but still have the ability to use forge and update to newer pkgs
as they come available.

So with this setup anyone on windows, Linux or Apple would always get the "core" packages when they install octave.
But can upgrade from forge when they want to.




--
DASCertificate for 206392

Reply | Threaded
Open this post in threaded view
|

Re: maintaining some packages at core Octave (was: Re: OctConf 2018, second day -- a lot of discussions and some work done)

Olaf Till-2
On Fri, Mar 16, 2018 at 08:40:32AM -0400, Doug Stewart wrote:

> On Fri, Mar 16, 2018 at 8:27 AM, Juan Pablo Carbajal <[hidden email]>
> wrote:
> > ...
> > We could have 'core','community' and 'external' packages as packages
> > categories in OF, in which 'core' has all the core development
> > standards (including a named branch "stable" used as in core). As an
> > extra benefit, delineating the criteria to classify as a 'core'
> > package in OF might go in the same direction as providing better
> > documentation for developers.
> >
> I agree with JPi on the three categories.
> ...
> I also like the idea of packaging these "core" packages with the released
> version
> of octave, but still have the ability to use forge and update to newer pkgs
> as they come available.
For the suggested core packages, apart from the question of the
location (OF or core Octave), the following changes seem to be
proposed:

1. Include only selected functions (maybe ML-compatible functions),

2. clearly designate these packages as core (it could be considered to
  (also) mark the selected functions (see 1.) as core),

3. only support current Octave version.

AFAICS, 1. and 2. are hoped to draw more attention to these packages
(functions). (Have you an indication that this will indeed happen?
Maybe from discussions at OctConf?)

Point 3. seems to be meant to spare our resources. A valid point. But
is it really necessary to 'forbid' supporting elder Octave versions?

Anyway, in the case of still hosting these packages at OF, the
following could be an applicable scheme:

- We mark such packages as 'core'.

- Within a package:

-- we select functions which should get more attention as 'core
   package functions'; to mark these, only these will be in inst/ and
   src/ .

-- all other functions go under inst-extra/ and src-extra/ and are
   pulled in by src/Makefile, so they are still available.

- We only support the current Octave version (?). I.e. a stable branch
  supports only the latest major release and a development branch
  supports the development version of Octave.

Olaf

--
public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: maintaining some packages at core Octave (was: Re: OctConf 2018, second day -- a lot of discussions and some work done)

Juan Pablo Carbajal-2
> Anyway, in the case of still hosting these packages at OF, the
> following could be an applicable scheme:
>
> - We mark such packages as 'core'.
>
> - Within a package:
>
> -- we select functions which should get more attention as 'core
>    package functions'; to mark these, only these will be in inst/ and
>    src/ .
>
> -- all other functions go under inst-extra/ and src-extra/ and are
>    pulled in by src/Makefile, so they are still available.
>
> - We only support the current Octave version (?). I.e. a stable branch
>   supports only the latest major release and a development branch
>   supports the development version of Octave.

I only have a comment regarding the scheme. In OF I would try to keep
the process of building package as simple as possible, if we still
want to offer a place for a community which might not be composed only
of developers that enjoy reading and handling complicated Makefiles.
It seems to me that scheme will indeed increase the complexity of the Makefiles.

Are we going to re-discuss the role of OF? (without an Agora-like site
OF is the only place we have to develop a community of domain specific
contributors, i.e. what made matlab so popular, i.e. not developers)
If not then lets keep the complexity of developing, distributing, and
maintaining as low as possible, avoiding intricacies.

Alternatively we will have to maintain templates for dev-like OF
contributors (to my dislike this is what it is currently offered) and
for domain specify contributors. Do we want that (re: resources)?

Reply | Threaded
Open this post in threaded view
|

Re: maintaining some packages at core Octave (was: Re: OctConf 2018, second day -- a lot of discussions and some work done)

Olaf Till-2
On Fri, Mar 23, 2018 at 10:04:57AM +0100, Juan Pablo Carbajal wrote:
> On Fri, Mar 23, 2018 at 09:23:19AM +0100, Olaf Till wrote:
> > - Within a package:
> >
> > -- we select functions which should get more attention as 'core
> >    package functions'; to mark these, only these will be in inst/ and
> >    src/ .
> >
> > -- all other functions go under inst-extra/ and src-extra/ and are
> >    pulled in by src/Makefile, so they are still available.

To be clear: this partitioning was only proposed for packages
(proposed to be) marked as 'core'.

> I only have a comment regarding the scheme. In OF I would try to
> keep the process of building package as simple as possible, if we
> still want to offer a place for a community which might not be
> composed only of developers that enjoy reading and handling
> complicated Makefiles.  It seems to me that scheme will indeed
> increase the complexity of the Makefiles.

Would you have you preferences for an alternative, regarding what
should be done with existing 'extra' functions in a 'core' package?
Just deleting them?  Putting them into an extra package (you seem to
have suggested this in a previous mail)? Not distinguishing them from
'core' functions?

Olaf

--
public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: maintaining some packages at core Octave (was: Re: OctConf 2018, second day -- a lot of discussions and some work done)

Juan Pablo Carbajal-2
> Would you have you preferences for an alternative, regarding what
> should be done with existing 'extra' functions in a 'core' package?
> Just deleting them?  Putting them into an extra package (you seem to
> have suggested this in a previous mail)? Not distinguishing them from
> 'core' functions?

Yes, I would always try to erase distinction between core and OF
functions, simpler for us, simpler for the users who come into Octave
(what are core, what are extra, what are community what are
external?).
there is a difference between one's workflow and what we expose as the
workflow of the community. Let ma give an example, maybe not the most
relevant but I hope to make myself clear: I do not see the point of
exposing Makefiles that detect whether one is using git or mercurial.
Once the Makefile is there it is either one of them. If it is not,
then it is up the the person doing the "nicety" to solve his workflow
and not contaminate the workflow of the whole.
Another example is the unneeded use of bash commands when octave's
commands can do the same thing. Devs are definitely expected to know
octave's language, they should not be expected to be proficient at
bash, sed, perl or any other language. Use externals only when is
necessary.

Given the new layer of complexity added, I would say forget about this
change, put back those functions back into core.
I was very happy when Oliver, Mike and Carne managed to come to the
kind of makefiles we see in signal, geometry and interval (this one is
going crazy, though). They improved the workflow a lot with minimum
cost in learning new tools (a few Makefile commands, which they
commented clearly). That kind of makefile was actually encouraging my
students to make their own package. The template makefile offered
currently at OF and Makefiles like the current one in statistics
already managed to scare two of my students (environmental engineers),
and force me to intervene to put their motivation back in track
(basically I told them: forget about this, just look into signal).

The issue here, of course, is that nobody can be wrong or right. We,
as the Of community, never managed to set the mission of the site. I
was always convinced this was the place were we capture users, and
potentially motivate them to be maintainers (core or OF, I do not
care). Under my perspective, and the role I see to OF, I see most of
current improvements counter-productive.
I said improvements because I have no doubt they were done with that
intention, the issue here is the lack of alignment among us.

I hope I answered something!