[pkg.m] Load order of dependencies

classic Classic list List threaded Threaded
29 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[pkg.m] Load order of dependencies

JuanPi
Hi all,

Is there any reason to make pkg.m load first the package and then the
dependencies?

Taking the development version of geometry as an example:

> pkg load geometry
GEOMETRY
matgeom loaded 0
MATGEOM
geometry loaded 1

For me it would make only sense that at the time the package folder is
added to the path the dependencies are already loaded.

This would simplify enormously checking for functionality in the
dependencies and manipulating the loadpath at loading time.

Also, if we allow packages to shadow octave functions, then packages
should also be allowed to shadow other dependencies functions, that is
the package must load at the end of all dependencies.

Thank you for your answer

--
JuanPi Carbajal
https://goo.gl/ayiJzi

-----
“An article about computational result is advertising, not
scholarship. The actual scholarship is the full software environment,
code and data, that produced  the  result.”
- Buckheit and Donoho

Reply | Threaded
Open this post in threaded view
|

Re: [pkg.m] Load order of dependencies

PhilipNienhuis
JuanPi wrote
> Hi all,
>
> Is there any reason to make pkg.m load first the package and then the
> dependencies?

I had more or less similar questions about pkg.m, motivated by matgeom /
geometry as well, but some years ago also by rad2deg & deg2rad that happened
to live in two different OF packages and later on in core Octave as well.

The logic is in scripts/pkg/private/load_packages_and_dependencies.m
As far as I could follow that code there is no need for a specific order of
loading of packages. AFAICS the order in which package and dependency
directories are added to the path is just a corollary of the way the code
happened to be written.
But I can be wrong ....

Lately I did some work on pkg.m so I got some feeling for how it sticks
together. I'll look into this but only later this week or next weekend.

Philip




--
Sent from: https://octave.1599824.n4.nabble.com/Octave-Maintainers-f1638794.html

Reply | Threaded
Open this post in threaded view
|

Re: [pkg.m] Load order of dependencies

apjanke-floss


On 12/9/19 3:39 PM, PhilipNienhuis wrote:> JuanPi wrote
>> Hi all,
>>
>> Is there any reason to make pkg.m load first the package and then the
>> dependencies?
>
> I had more or less similar questions about pkg.m, motivated by matgeom /
> geometry as well, but some years ago also by rad2deg & deg2rad that
happened
> to live in two different OF packages and later on in core Octave as well.
>
> The logic is in scripts/pkg/private/load_packages_and_dependencies.m
> As far as I could follow that code there is no need for a specific
order of
> loading of packages. AFAICS the order in which package and dependency
> directories are added to the path is just a corollary of the way the code
> happened to be written.
> But I can be wrong ....
>
> Lately I did some work on pkg.m so I got some feeling for how it sticks
> together. I'll look into this but only later this week or next weekend.
>
> Philip

I would think that the dependencies of package A should be loaded before
A, because A's startup/teardown code (in PKG_ADD or PKG_DEL) might use
functions/classes provided by its dependencies?

Also, anybody who cares about this, come on by Packajoozle and give
suggestions if you want. https://github.com/apjanke/octave-packajoozle
Packajoozle loads its packages in dependency order (dependents before
dependers) for this reason.

Cheers,
Andrew


Reply | Threaded
Open this post in threaded view
|

Re: [pkg.m] Load order of dependencies

PhilipNienhuis
apjanke-floss wrote

> On 12/9/19 3:39 PM, PhilipNienhuis wrote:> JuanPi wrote
>>> Hi all,
>>>
>>> Is there any reason to make pkg.m load first the package and then the
>>> dependencies?
>>
>> I had more or less similar questions about pkg.m, motivated by matgeom /
>> geometry as well, but some years ago also by rad2deg & deg2rad that
> happened
>> to live in two different OF packages and later on in core Octave as well.
>>
>> The logic is in scripts/pkg/private/load_packages_and_dependencies.m
>> As far as I could follow that code there is no need for a specific
> order of
>> loading of packages. AFAICS the order in which package and dependency
>> directories are added to the path is just a corollary of the way the code
>> happened to be written.
>> But I can be wrong ....
>>
>> Lately I did some work on pkg.m so I got some feeling for how it sticks
>> together. I'll look into this but only later this week or next weekend.
>>
>> Philip
>
> I would think that the dependencies of package A should be loaded before
> A, because A's startup/teardown code (in PKG_ADD or PKG_DEL) might use
> functions/classes provided by its dependencies?

Sure, that feels like the obvious. But pkg.m as it stands just doesn't work
that way.

Moreover, dependencies aren't always so "linear".
ISTR there were (or are?) circular dependencies. Or mutual exclusive
dependencies (example: package A depends on release "X" of package B, but
package C won't work with release "X" and depends on another release of B).
And maybe someone loaded package D that also depends on package C and thus
implicitly loaded it, and now wants to also use package A. What should pkg.m
do then?
In short, there could be an almost infinite number of scenario's for
interdependencies that all need to be catered for (if only by informing
users of incompatibilities) in a proper package management system.
IMO the "-nodeps" flag is a rough but indispensable escape to somehow deal
with this.

Anyway, from what I learned recently when I dived in pkg.m, it is quite a
cleverly written piece of code, but apparently it never got completely
polished. Several small gotchas and corner case issues are still hidden here
and there. My guess is that the cause lies partly in the complexity I
touched upon above.

As to the problem at hand:
AFAICS (but correct me if I'm wrong) "loading" a package involves nothing
more than just adding its code directories (.m, .oct/.mex) to the path.
PKG_ADD code automatically takes care of the package init scripts.
So, this whole issue of JuanPi it is really just about the order in which
package directories *happen* to be added to the path.
Now, load_packages_and_directories.m first adds the directory of a package
to that list and only then dives in the dependencies and adds them one by
one, if needed recursively if dependencies themselves also happen to have
dependencies, and so on.  So *I think* merely reversing the final list
before handing it to the code that adds the package dirs to the path would
solve it in one fell swoop, so to say.
Again, I could be wrong - it just needs to be tried and I only have time for
that later this week or weekend.

Philip




--
Sent from: https://octave.1599824.n4.nabble.com/Octave-Maintainers-f1638794.html

Reply | Threaded
Open this post in threaded view
|

Re: [pkg.m] Load order of dependencies

Andrew Janke-2


On 12/10/19 11:57 AM, PhilipNienhuis wrote:

> apjanke-floss wrote
>> On 12/9/19 3:39 PM, PhilipNienhuis wrote:> JuanPi wrote
>>> Philip
>>
>> I would think that the dependencies of package A should be loaded before
>> A, because A's startup/teardown code (in PKG_ADD or PKG_DEL) might use
>> functions/classes provided by its dependencies?
>
> Sure, that feels like the obvious. But pkg.m as it stands just doesn't work
> that way.
>
> Moreover, dependencies aren't always so "linear".
> ISTR there were (or are?) circular dependencies. Or mutual exclusive
> dependencies (example: package A depends on release "X" of package B, but
> package C won't work with release "X" and depends on another release of B).
> And maybe someone loaded package D that also depends on package C and thus
> implicitly loaded it, and now wants to also use package A. What should pkg.m
> do then?
> In short, there could be an almost infinite number of scenario's for
> interdependencies that all need to be catered for (if only by informing
> users of incompatibilities) in a proper package management system.

Circular dependencies, ouch. I've done a fair amount of work with
package managers, and my view is that circular dependencies are
fundamentally an insoluble problem. Build-time circular dependencies
probably need to be prohibited. Run-time circular dependencies, just
load them in the order you find them doing a depth-first search through
the dependency graph, pruning the search and issuing a warning when you
get to a circularity.

> IMO the "-nodeps" flag is a rough but indispensable escape to somehow deal
> with this.

Agreed. You need to have a back door for cases like this.

> Anyway, from what I learned recently when I dived in pkg.m, it is quite a
> cleverly written piece of code, but apparently it never got completely
> polished. Several small gotchas and corner case issues are still hidden here
> and there. My guess is that the cause lies partly in the complexity I
> touched upon above.
>
> As to the problem at hand:
> AFAICS (but correct me if I'm wrong) "loading" a package involves nothing
> more than just adding its code directories (.m, .oct/.mex) to the path.
> PKG_ADD code automatically takes care of the package init scripts.
> So, this whole issue of JuanPi it is really just about the order in which
> package directories *happen* to be added to the path.

This is correct AFAIK, too.

> Now, load_packages_and_directories.m first adds the directory of a package
> to that list and only then dives in the dependencies and adds them one by
> one, if needed recursively if dependencies themselves also happen to have
> dependencies, and so on.  So *I think* merely reversing the final list
> before handing it to the code that adds the package dirs to the path would
> solve it in one fell swoop, so to say.
> Again, I could be wrong - it just needs to be tried and I only have time for
> that later this week or weekend.

This doesn't quite work, I think. The list you get now is a "pre-order"
traversal of the dependency graph. Reversing it gives you _a_
"post-order" solution, but it doesn't guarantee the correct ordering of
indirect dependencies which themselves might have dependencies on each
other. To get a proper "all dependencies first" load order, you need to
do a traversal through the dependency graph *with backtracking* to
handle dependencies between indirect dependencies. It's non-trivial.

One (incomplete) approach looks like this:
https://github.com/apjanke/octave-packajoozle/blob/efa4ac816bfcc51a674a0e426efb1417d4133896/inst/%2Bpackajoozle/%2Binternal/DependencyResolver.m#L75

Cheers,
Andrew



Reply | Threaded
Open this post in threaded view
|

Re: [pkg.m] Load order of dependencies

apjanke-floss
In reply to this post by PhilipNienhuis
Sorry if this is a re-send; my mail client screwed up when I was
originally writing this.

On 12/10/19 11:57 AM, PhilipNienhuis wrote:

> apjanke-floss wrote
>> On 12/9/19 3:39 PM, PhilipNienhuis wrote:> JuanPi wrote
>>
>> I would think that the dependencies of package A should be loaded before
>> A, because A's startup/teardown code (in PKG_ADD or PKG_DEL) might use
>> functions/classes provided by its dependencies?
>
> Sure, that feels like the obvious. But pkg.m as it stands just doesn't work
> that way.
>
> Moreover, dependencies aren't always so "linear".
> ISTR there were (or are?) circular dependencies. Or mutual exclusive
> dependencies (example: package A depends on release "X" of package B, but
> package C won't work with release "X" and depends on another release of B).
> And maybe someone loaded package D that also depends on package C and thus
> implicitly loaded it, and now wants to also use package A. What should pkg.m
> do then?

Circular dependencies; ouch. I've done some work on package managers,
and my view is that they're an insoluble problem. Probably, build-time
circular dependencies need to be prohibited, and run-time circulars,
well, just pick an arbitrary order based on what you find during the
dependency graph traversal, prune the traversal when you hit a cycle,
and hope it works.

> IMO the "-nodeps" flag is a rough but indispensable escape to somehow deal
> with this.

Agreed. You need that back door for cases like this.

> Anyway, from what I learned recently when I dived in pkg.m, it is quite a
> cleverly written piece of code, but apparently it never got completely
> polished. Several small gotchas and corner case issues are still hidden here
> and there. My guess is that the cause lies partly in the complexity I
> touched upon above.
>
> As to the problem at hand:
> AFAICS (but correct me if I'm wrong) "loading" a package involves nothing
> more than just adding its code directories (.m, .oct/.mex) to the path.
> PKG_ADD code automatically takes care of the package init scripts.
> So, this whole issue of JuanPi it is really just about the order in which
> package directories *happen* to be added to the path.

This is correct.

> Now, load_packages_and_directories.m first adds the directory of a package
> to that list and only then dives in the dependencies and adds them one by
> one, if needed recursively if dependencies themselves also happen to have
> dependencies, and so on.  So *I think* merely reversing the final list
> before handing it to the code that adds the package dirs to the path would
> solve it in one fell swoop, so to say.
> Again, I could be wrong - it just needs to be tried and I only have time for
> that later this week or weekend.

I don't think this is correct. The order you get now from the recursive
loading is a "pre-order" traversal of the dependency graph. Reversing
that list gives you _a_ "post-order" solution. But it doesn't
necessarily handle the case of indirect dependencies which might have
complex dependencies between them. You need to do a modified post-order
traversal *with backtracking* to handle the case of dependencies between
indirect dependencies. It's non-trivial.

One (incomplete) approach is here:
https://github.com/apjanke/octave-packajoozle/blob/efa4ac816bfcc51a674a0e426efb1417d4133896/inst/%2Bpackajoozle/%2Binternal/DependencyResolver.m#L75

Cheers,
Andrew

Reply | Threaded
Open this post in threaded view
|

Re: [pkg.m] Load order of dependencies

Juan Pablo Carbajal-2
Hi all,

This issue has been solved many times (I guess in different
idiosyncratic ways). I would just check how

* PyPI https://twitter.com/di_codes/status/1193980331004743680
https://github.com/pypa/pip/issues/988

* Julia https://julialang.github.io/Pkg.jl/dev/

does it

I particularly like the Julia way of handling user contributed packages...

My näive approach to the undocumented jungle of pkg.m  was that a DAG
is build by analyzing dependency statements of packages, and then the
tree is traversed from leafs to root, removing nodes as we go...
But the pile of nested for loops got me pinned down.

Reply | Threaded
Open this post in threaded view
|

Re: [pkg.m] Load order of dependencies

PhilipNienhuis
In reply to this post by JuanPi
Hi JuanPi:


JuanPi wrote

> Hi all,
>
> Is there any reason to make pkg.m load first the package and then the
> dependencies?
>
> Taking the development version of geometry as an example:
>
>> pkg load geometry
> GEOMETRY
> matgeom loaded 0
> MATGEOM
> geometry loaded 1
>
> For me it would make only sense that at the time the package folder is
> added to the path the dependencies are already loaded.
>
> This would simplify enormously checking for functionality in the
> dependencies and manipulating the loadpath at loading time.
>
> Also, if we allow packages to shadow octave functions, then packages
> should also be allowed to shadow other dependencies functions, that is
> the package must load at the end of all dependencies.

OK, I looked into it and my conclusions are as follows:

1. There is a subtle but deceiving difference between "loading" a package
and the effects of adding its directories to the path.
The earlier a package is loaded, the lower its directories show up in the
path. IOW, packages "loaded" later are searched earlier in the path.

2. Taking into account the contents of other posts here, pkg.m is actually
doing the exact right thing, i.e., when loading a package it first adds the
desired package directories to the path, and only then the directories of
the package dependencies, AFAICS all in the proper order. But see 4., below.

3. But what you want to accomplish is exactly the opposite. You want that to
make sure that functions in some package can shadow similarly named
functions in its dependencies.

4. Dependency directories are at the top of the path and searched first, as
one normally would expect and desire, because of the thinking that
dependencies may be needed during initialization of the desired package.
However, the weak point in this reasoning is that it is valid for *all*
packages. Some, if not most, packages don't need dependencies for
initialization, they just need dependencies to supply functionality for them
at run time, not initialization time.

Looking at how pkg.m works, getting together point 3. above would be
difficult because of the following;

* Process:
If we adapt pkg.m, it would very probably only apply to Octave-6.1.x, and
the packages matgeom, geometry (and mapping) (the actual packages concerned
here) would depend hard on the latest version of Octave and on the latest
version of each other. I think that would be an undesirable situation.
Packages should be backward-compatible with a few older octave releases.

* Technically:
Adapting pkg.m to first load dependencies and only then the desired package,
if so desired for some package, is surely possible.
But the story is more convoluted: just think of the possibility that one of
the dependencies *does* need another dependency for its initialization so
needs that dependency earlier in the path (= loaded later).
The implication is that the whole thing is going to be relying on the needs
of individual packages and cannot be catched in e.g., another flag for
pkg.m.
The most reliable way to get it together would be to add another field to
the package DESCRIPTION file or amend the "Depends" field to indicate how
some package wants to deal with each of its individual dependencies. That
has to be done in such a way that all currently existing packages aren't
affected.
I fear that is going to be a big headache at this moment.

Now as regards matgeom and geometry:
* matgeom is a dependency of geometry
* matgeom has been split from geometry for several reasons, mostly for ease
of upgrading geometry and matgeom separately, (IIRC) license stuff,
perceptions regarding external or community packages.
* nevertheless geometry and matgeom are heavily interdependent.
I suppose matgeom and geometry won't be used much separately.
Taking this into account, for the time being I'd advise to simply remove the
offending functions in matgeom (or rename them).
Maybe that can be done in some init script in the geometry package that
checks if matgeom is installed (should be as it's a dependency of geometry)
and if so, processes the pertinent matgeom function files. Hopefully that
won't give trouble, as this will rename a file actually in the search path
(cached?).

In view of this it's a bit of a pity that the post_install.m file in package
root dirs has been dropped. It would have offered an ideal option to deal
with these sort of issues.

Philip




--
Sent from: https://octave.1599824.n4.nabble.com/Octave-Maintainers-f1638794.html

Reply | Threaded
Open this post in threaded view
|

Re: [pkg.m] Load order of dependencies

Juan Pablo Carbajal-2
Thanks Philip for looking into this.
As said many times pkg.m should be replaced: burn to the ground, and
reborn like the fenix.
I am not convinced pkj does a much better job, but it at least enjoys
a more maintainable structure.

> 2. Taking into account the contents of other posts here, pkg.m is actually
> doing the exact right thing, i.e., when loading a package it first adds the
> desired package directories to the path, and only then the directories of
> the package dependencies, AFAICS all in the proper order. But see 4., below.

As far as I understand it , this is actually all wrong. The
dependencies should be added first and then the desired package.
Only this way the packages functions are found first.
Only then you get consistent behavior with what happens if the package
shadows an octave function (I hope we all agree octave is a dependency
of the package).
That is, you insert the latest loaded package on the top of the path.
So if you start by loading dependencies from the leaf nodes, this is
properly resolved.
But of course there might be other problems waiting hidden in that
implementation.

>
> 3. But what you want to accomplish is exactly the opposite. You want that to
> make sure that functions in some package can shadow similarly named
> functions in its dependencies.

Exactly, which is consistent to what happens if you shadow a core function.
And as I see it, the desired and expected behavior (principle of no
surprise: why are paths to core/m treated differently than paths to
packages?).

>
> 4. Dependency directories are at the top of the path and searched first, as
> one normally would expect and desire, because of the thinking that
> dependencies may be needed during initialization of the desired package.
> However, the weak point in this reasoning is that it is valid for *all*
> packages. Some, if not most, packages don't need dependencies for
> initialization, they just need dependencies to supply functionality for them
> at run time, not initialization time.

I guess it depends on the definitions here, but as of now, I do not
see any troubles in accepting that:

1. During "initialization" dependencies are available, but the package
itself isn't (call it pre_load if you want).

2. After initialization the package is actually loaded and its folders
added to the top of the search path. At this point dependencies are
shadowed if the package defines homonymous functions.

>
> Looking at how pkg.m works, getting together point 3. above would be
> difficult because of the following;
>
> * Process:
> If we adapt pkg.m, it would very probably only apply to Octave-6.1.x, and
> the packages matgeom, geometry (and mapping) (the actual packages concerned
> here) would depend hard on the latest version of Octave and on the latest
> version of each other. I think that would be an undesirable situation.
> Packages should be backward-compatible with a few older octave releases.
>
I do not generally agree with this, since it is only some
functionality that is lost when using older version of octave. Most of
the package still works.
For the user case at hand: for octave < 6.1 geometry::clipPolygon is
not implemented.

> * Technically:
> Adapting pkg.m to first load dependencies and only then the desired package,
> if so desired for some package, is surely possible.
> But the story is more convoluted: just think of the possibility that one of
> the dependencies *does* need another dependency for its initialization so
> needs that dependency earlier in the path (= loaded later).

As described before, afai understand it, this is a no problem, as the
dependencies were already resolved during pre_load of the "another
package"

> The implication is that the whole thing is going to be relying on the needs
> of individual packages and cannot be catched in e.g., another flag for
> pkg.m.
> The most reliable way to get it together would be to add another field to
> the package DESCRIPTION file or amend the "Depends" field to indicate how
> some package wants to deal with each of its individual dependencies. That
> has to be done in such a way that all currently existing packages aren't
> affected.
> I fear that is going to be a big headache at this moment.

This is a no-go. Specially when all other dependency manager I
mentioned (of course, with proper namespaces!) handle this correctly
and do not require this extra information.

>
> Now as regards matgeom and geometry:
> * matgeom is a dependency of geometry
> * matgeom has been split from geometry for several reasons, mostly for ease
> of upgrading geometry and matgeom separately, (IIRC) license stuff,
> perceptions regarding external or community packages.
> * nevertheless geometry and matgeom are heavily interdependent.
> I suppose matgeom and geometry won't be used much separately.

matgeom should be completely oblivious to geometry. Matgoem is a
stand-alone package, it does not depend on geometry.

> Taking this into account, for the time being I'd advise to simply remove the
> offending functions in matgeom (or rename them).
> Maybe that can be done in some init script in the geometry package that
> checks if matgeom is installed (should be as it's a dependency of geometry)
> and if so, processes the pertinent matgeom function files. Hopefully that
> won't give trouble, as this will rename a file actually in the search path
> (cached?).

Nah, this is a no-go either.
If this is the only solution, then I would rename geometry's functions
to avoid the whole problem.

Reply | Threaded
Open this post in threaded view
|

Re: [pkg.m] Load order of dependencies

Juan Pablo Carbajal-2
Here is one nasty solution, but it works

We use a register function (that is, a function that is called with
PKG_ADD directive, e.g. see [1]) to force the correct loading order.
I explain (call PKG the package and DEP the dependency)

1. When the register function is called PKG is already in the path,
but not DEP (bug!)
2. We unload PKG and load DEP
3. we load PKG

Because at runtime of 3. DEP is already loaded, luckily pkg.m is not
forcing a reload of it when it load PKG.

The code looks like this (user case PKG=geometry, DEP=matgeom)

## PKG_ADD: __geometry_package_register__ (1);
## PKG_DEL: __geometry_package_register__ (-1);

function __geometry_package_register__ (loading = 0)

  if (loading > 0)
    ## Check if matgeom is loaded
    if (pkg('list','matgeom'){1}.loaded == 0)
      ## Shadow matgeom functions
      ## i.e. first load matgeom then geometry
      pkg unload geometry  # because pkg.m has already loaded it,
before its dependencies!
      pkg load matgeom     # force first load of dependency to shadow
      pkg load geometry    # reload the package
    endif
  elseif (loading < 0)
    # we do not care about unloading
  endif

endfunction

Because this solves the problem at hand, I am convinced that the
current behavior of pkg.m is buggy

[1]: https://sourceforge.net/p/octave/matgeom/ci/master/tree/octave_pkg/inst/__matgeom_package_register__.m

On Thu, Dec 12, 2019 at 2:26 AM Juan Pablo Carbajal
<[hidden email]> wrote:

>
> Thanks Philip for looking into this.
> As said many times pkg.m should be replaced: burn to the ground, and
> reborn like the fenix.
> I am not convinced pkj does a much better job, but it at least enjoys
> a more maintainable structure.
>
> > 2. Taking into account the contents of other posts here, pkg.m is actually
> > doing the exact right thing, i.e., when loading a package it first adds the
> > desired package directories to the path, and only then the directories of
> > the package dependencies, AFAICS all in the proper order. But see 4., below.
>
> As far as I understand it , this is actually all wrong. The
> dependencies should be added first and then the desired package.
> Only this way the packages functions are found first.
> Only then you get consistent behavior with what happens if the package
> shadows an octave function (I hope we all agree octave is a dependency
> of the package).
> That is, you insert the latest loaded package on the top of the path.
> So if you start by loading dependencies from the leaf nodes, this is
> properly resolved.
> But of course there might be other problems waiting hidden in that
> implementation.
>
> >
> > 3. But what you want to accomplish is exactly the opposite. You want that to
> > make sure that functions in some package can shadow similarly named
> > functions in its dependencies.
>
> Exactly, which is consistent to what happens if you shadow a core function.
> And as I see it, the desired and expected behavior (principle of no
> surprise: why are paths to core/m treated differently than paths to
> packages?).
>
> >
> > 4. Dependency directories are at the top of the path and searched first, as
> > one normally would expect and desire, because of the thinking that
> > dependencies may be needed during initialization of the desired package.
> > However, the weak point in this reasoning is that it is valid for *all*
> > packages. Some, if not most, packages don't need dependencies for
> > initialization, they just need dependencies to supply functionality for them
> > at run time, not initialization time.
>
> I guess it depends on the definitions here, but as of now, I do not
> see any troubles in accepting that:
>
> 1. During "initialization" dependencies are available, but the package
> itself isn't (call it pre_load if you want).
>
> 2. After initialization the package is actually loaded and its folders
> added to the top of the search path. At this point dependencies are
> shadowed if the package defines homonymous functions.
>
> >
> > Looking at how pkg.m works, getting together point 3. above would be
> > difficult because of the following;
> >
> > * Process:
> > If we adapt pkg.m, it would very probably only apply to Octave-6.1.x, and
> > the packages matgeom, geometry (and mapping) (the actual packages concerned
> > here) would depend hard on the latest version of Octave and on the latest
> > version of each other. I think that would be an undesirable situation.
> > Packages should be backward-compatible with a few older octave releases.
> >
> I do not generally agree with this, since it is only some
> functionality that is lost when using older version of octave. Most of
> the package still works.
> For the user case at hand: for octave < 6.1 geometry::clipPolygon is
> not implemented.
>
> > * Technically:
> > Adapting pkg.m to first load dependencies and only then the desired package,
> > if so desired for some package, is surely possible.
> > But the story is more convoluted: just think of the possibility that one of
> > the dependencies *does* need another dependency for its initialization so
> > needs that dependency earlier in the path (= loaded later).
>
> As described before, afai understand it, this is a no problem, as the
> dependencies were already resolved during pre_load of the "another
> package"
>
> > The implication is that the whole thing is going to be relying on the needs
> > of individual packages and cannot be catched in e.g., another flag for
> > pkg.m.
> > The most reliable way to get it together would be to add another field to
> > the package DESCRIPTION file or amend the "Depends" field to indicate how
> > some package wants to deal with each of its individual dependencies. That
> > has to be done in such a way that all currently existing packages aren't
> > affected.
> > I fear that is going to be a big headache at this moment.
>
> This is a no-go. Specially when all other dependency manager I
> mentioned (of course, with proper namespaces!) handle this correctly
> and do not require this extra information.
>
> >
> > Now as regards matgeom and geometry:
> > * matgeom is a dependency of geometry
> > * matgeom has been split from geometry for several reasons, mostly for ease
> > of upgrading geometry and matgeom separately, (IIRC) license stuff,
> > perceptions regarding external or community packages.
> > * nevertheless geometry and matgeom are heavily interdependent.
> > I suppose matgeom and geometry won't be used much separately.
>
> matgeom should be completely oblivious to geometry. Matgoem is a
> stand-alone package, it does not depend on geometry.
>
> > Taking this into account, for the time being I'd advise to simply remove the
> > offending functions in matgeom (or rename them).
> > Maybe that can be done in some init script in the geometry package that
> > checks if matgeom is installed (should be as it's a dependency of geometry)
> > and if so, processes the pertinent matgeom function files. Hopefully that
> > won't give trouble, as this will rename a file actually in the search path
> > (cached?).
>
> Nah, this is a no-go either.
> If this is the only solution, then I would rename geometry's functions
> to avoid the whole problem.

Reply | Threaded
Open this post in threaded view
|

Re: [pkg.m] Load order of dependencies

siko1056
In reply to this post by Juan Pablo Carbajal-2
Dear all,

Since following the work on GNU Octave in 2013, I always notice a high
tension when it comes to the pkg implementation and the organization of
Octave Forge itself.  My humble perspective is, that the current
implementations do a good enough job so far and there is lack of
maintenance power in both Octave Forge and the pkg tool, which might
stress developers as well, who spend time and effort in improvements:

On 12/10/19 5:39 AM, PhilipNienhuis wrote:
> Lately I did some work on pkg.m so I got some feeling for how it
sticks together.

On 12/10/19 5:45 AM, Andrew Janke wrote:
> Also, anybody who cares about this, come on by Packajoozle and give
> suggestions if you want. https://github.com/apjanke/octave-packajoozle
> Packajoozle loads its packages in dependency order (dependents before
> dependers) for this reason.


The original question by Juan was for the scenario:

   A depends on {B depends on {D, E}, C}

if the loading order by pkg should be (and in pkj is)

   1. Load {D, E} undetermined order; functions to top of loadpath
   2. Load {B, C} undetermined order; functions to top of loadpath
   3. Load A                        ; functions to top of loadpath

but the exact opposite seems to be implemented in pkg, right?  To me the
stated order above seems more "natural" as well.

On 12/12/19 10:26 AM, Juan Pablo Carbajal wrote:
> As said many times pkg.m should be replaced [...] I am not convinced
> pkj does a much better job, but it at least enjoys a more maintainable
> structure.

@Juan, did you use pkj and can confirm that it cannot handle your
loading issue?  The dependency between geometry [1] and matgeom [2] will
occur in the upcoming release [3], right?


The next aspect that came as an argument for the current pkg implementation:

On 12/11/19 1:57 AM, PhilipNienhuis wrote:
> Moreover, dependencies aren't always so "linear". ISTR there were
> (or are?) circular dependencies.

Is this really somewhere the case, or theoretical consideration?
Personally, I agree with Andrew in this respect:

On 12/11/19 5:55 AM, Andrew Janke wrote:
> Circular dependencies; ouch.

Thus I would not stress this, if it is an exceptional/non-occuring case
in current OF packages.

@Philip, do you currently spend effort in improving the current pkg
function, or should pkj by Andrew be a favorable long term replacement?
 Agreeing on this might save time and effort.

@Andrew, what is the state of pkj?  Would you say it can be merged to
core Octave?

Best,
Kai

[1]: https://octave.sourceforge.io/geometry/index.html
[2]: https://github.com/mattools/matGeom
[3]: https://sourceforge.net/p/octave/geometry/ci/default/tree/DESCRIPTION

Reply | Threaded
Open this post in threaded view
|

Re: [pkg.m] Load order of dependencies

Andrew Janke-2
Thanks Kai!

On 12/11/19 9:15 PM, Kai Torben Ohlhus wrote:

> Dear all,
> On 12/10/19 5:45 AM, Andrew Janke wrote:
>> Also, anybody who cares about this, come on by Packajoozle and give
>> suggestions if you want. https://github.com/apjanke/octave-packajoozle
>> Packajoozle loads its packages in dependency order (dependents before
>> dependers) for this reason.
>
>
> The original question by Juan was for the scenario:
>
>    A depends on {B depends on {D, E}, C}
>
> if the loading order by pkg should be (and in pkj is)
>
>    1. Load {D, E} undetermined order; functions to top of loadpath
>    2. Load {B, C} undetermined order; functions to top of loadpath
>    3. Load A                        ; functions to top of loadpath
>
> but the exact opposite seems to be implemented in pkg, right?  To me the
> stated order above seems more "natural" as well.

One thing here: package load order and search path order are not locked
together. The addpath() function has both '-begin' and '-end' options,
so package directories could be either prependend or appended to the
path. That is, even if a dependency is loaded first, we could do so in a
manner that it ends up on the search path *after* the depender. Or vice
versa.

To me, though it seems like dependers should end up on the path before
dependencies: If package A depends on package B, that means that package
A knows about package B, and can determine which of B's functions it
should shadow. B doesn't necessarily know anything about package A.

So you could do a graph search to find a dependency-ordering solution,
load them in a dependencies-first order, but do so in a way that either
the dependers or the dependencies have priority in the path. Either as a
global policy, or on a per-package-pair basis, if that metadata were
added to the package definitions. (Which sounds like an unnecessary
complication to me, but maybe there are use cases for it?)

> On 12/12/19 10:26 AM, Juan Pablo Carbajal wrote:
>> As said many times pkg.m should be replaced [...] I am not convinced
>> pkj does a much better job, but it at least enjoys a more maintainable
>> structure.
>
> @Juan, did you use pkj and can confirm that it cannot handle your
> loading issue?  The dependency between geometry [1] and matgeom [2] will
> occur in the upcoming release [3], right?

If there's anything that pkj doesn't handle that y'all would like to see
supported, I'm happy to take feature requests! Just drop an issue on the
GitHub repo. My goal for pkj is to support Octave's current use cases in
a nicer manner, not to impose my view of package management on Octave.

> On 12/11/19 5:55 AM, Andrew Janke wrote:
>> Circular dependencies; ouch.
>
> Thus I would not stress this, if it is an exceptional/non-occuring case
> in current OF packages.

Well, if matgeom and geometry mutually depend on each other, then it's
unfortunately something we need to handle.

One note: I'm coming from the compiled-software world, where circular
dependencies are just a no-go. Things might be different for Octave.

> @Philip, do you currently spend effort in improving the current pkg
> function, or should pkj by Andrew be a favorable long term replacement?
>  Agreeing on this might save time and effort.
>
> @Andrew, what is the state of pkj?  Would you say it can be merged to
> core Octave?

Oh no. pkj/Packajoozle is still beta at best; it is a long way off from
being high-quality or stable enough to merge to core Octave. Most
importantly, we need more testers to try it and shake out bugs and tell
us what is missing! If you'd like to see pkj become a real thing, please
drop by and try it out.

Cheers,
Andrew

Reply | Threaded
Open this post in threaded view
|

All files attached to the bug or patch tracker of GNU Octave are inaccessible/lost?

siko1056
Dear all,

Does anyone observe the same problem as well?

   https://savannah.nongnu.org/support/?110158

I tried for other projects as well:

   https://savannah.gnu.org/bugs/?57047

Same here.

Kai

Reply | Threaded
Open this post in threaded view
|

Re: All files attached to the bug or patch tracker of GNU Octave are inaccessible/lost?

nrjank


On Thu, Dec 12, 2019, 12:29 AM Kai Torben Ohlhus <[hidden email]> wrote:
Dear all,

Does anyone observe the same problem as well?

I see the same. Looking at the latest comment on that bug report you filed it seems all files were moved, not lost. Hopefully it is fixed soon

Reply | Threaded
Open this post in threaded view
|

Re: [pkg.m] Load order of dependencies

PhilipNienhuis
In reply to this post by siko1056
Kai Torben Ohlhus wrote:
> Dear all,
>
> Since following the work on GNU Octave in 2013, I always notice a high
> tension when it comes to the pkg implementation and the organization of
> Octave Forge itself.  My humble perspective is, that the current
> implementations do a good enough job so far and there is lack of
> maintenance power in both Octave Forge and the pkg tool, which might
> stress developers as well, who spend time and effort in improvements:

Very true.

> On 12/10/19 5:39 AM, PhilipNienhuis wrote:
>> Lately I did some work on pkg.m so I got some feeling for how it
> sticks together.
>
> On 12/10/19 5:45 AM, Andrew Janke wrote:
>> Also, anybody who cares about this, come on by Packajoozle and give
>> suggestions if you want. https://github.com/apjanke/octave-packajoozle
>> Packajoozle loads its packages in dependency order (dependents before
>> dependers) for this reason.
>
>
> The original question by Juan was for the scenario:
>
>     A depends on {B depends on {D, E}, C}
>
> if the loading order by pkg should be (and in pkj is)
>
>     1. Load {D, E} undetermined order; functions to top of loadpath
>     2. Load {B, C} undetermined order; functions to top of loadpath
>     3. Load A                        ; functions to top of loadpath
>
> but the exact opposite seems to be implemented in pkg, right?  To me the
> stated order above seems more "natural" as well.

Sure. But as I wrote, pkg.m was written with initialization of packages
using dependencies in mind and somewhere in this thread it was mentioned
that then you need the other way round.
In L.46 of <pkg>/private/load_packages.m there is this comment:
   ## Load the packages, but take care of the ordering of dependencies.

I think in this thread it isn't very clear that the current Octave-Forge
setup never took deliberate shadowing into account.
Until now shadowing was mainly a nuisance when OF functions were
absorbed in core and the remaining OF package wasn't properly adapted to
the new situation.
Now JuanPi has the wish to explicitly shadow a function in another OF
package that is also a dependency. AFAIK that is new.

> On 12/12/19 10:26 AM, Juan Pablo Carbajal wrote:
>> As said many times pkg.m should be replaced [...] I am not convinced
>> pkj does a much better job, but it at least enjoys a more maintainable
>> structure.
>
> @Juan, did you use pkj and can confirm that it cannot handle your
> loading issue?  The dependency between geometry [1] and matgeom [2] will
> occur in the upcoming release [3], right?
>
>
> The next aspect that came as an argument for the current pkg implementation:
>
> On 12/11/19 1:57 AM, PhilipNienhuis wrote:
>> Moreover, dependencies aren't always so "linear". ISTR there were
>> (or are?) circular dependencies.
>  
> Is this really somewhere the case, or theoretical consideration?
> Personally, I agree with Andrew in this respect:

IIRC there were (years ago), like pkg A -> pkg B - > pkg C -> pkg A; but
I never checked back.

One of the weak points in our current package system is that there's no
"authority" overlooking dependency chains. Package maintainers have all
freedom and I'm not sure that every maintainer carefully checks existing
dependency trees before adding one to her/his package.
Equally weak is that a package maintainer can at her/his will remove
functions that other packages rely on, thus breaking a dependency. I'm
equally unsure that all package maintainers check for this, and AFAIK
this isn't checked by forge admins either.

> On 12/11/19 5:55 AM, Andrew Janke wrote:
>> Circular dependencies; ouch.
>
> Thus I would not stress this, if it is an exceptional/non-occuring case
> in current OF packages.

I think (hope) it was an exception.

My other argument is mutually exclusive dependencies. That may also be
an exception, but has happened as well.

I remember that at one time I thought I was in something looking very
much like Windows' "dll hell" - I tried to update or install some
package and next I know the rest of the evening I was occupied getting
other packages updated/installed. Of course my mistake was that I tried
to do it neatly but in hindsight I should have used the "nodeps" flag
right away.

The whole point is that -as outlined above- there's no sound or robust
dependency chain management whatsoever.

> @Philip, do you currently spend effort in improving the current pkg
> function, or should pkj by Andrew be a favorable long term replacement?
>   Agreeing on this might save time and effort.

I was just fixing bugs that were in the way and that were relatively
easy to solve.
Other than that I have no preference.

But: is pkj compatible with all existing packages? or do existing
packages need to adapt DESCRIPTION entries and/or other things?

Philip

Reply | Threaded
Open this post in threaded view
|

Re: [pkg.m] Load order of dependencies

PhilipNienhuis
In reply to this post by Juan Pablo Carbajal-2
Juan Pablo Carbajal wrote:

> Thanks Philip for looking into this.
> As said many times pkg.m should be replaced: burn to the ground, and
> reborn like the fenix.
> I am not convinced pkj does a much better job, but it at least enjoys
> a more maintainable structure.
>
>> 2. Taking into account the contents of other posts here, pkg.m is actually
>> doing the exact right thing, i.e., when loading a package it first adds the
>> desired package directories to the path, and only then the directories of
>> the package dependencies, AFAICS all in the proper order. But see 4., below.
>
> As far as I understand it , this is actually all wrong. The
> dependencies should be added first and then the desired package.
> Only this way the packages functions are found first.
> Only then you get consistent behavior with what happens if the package
> shadows an octave function (I hope we all agree octave is a dependency
> of the package).
> That is, you insert the latest loaded package on the top of the path.
> So if you start by loading dependencies from the leaf nodes, this is
> properly resolved.
> But of course there might be other problems waiting hidden in that
> implementation.

Need of dependencies for initialization of a package was mentioned.
But I don't know any package that needs dependencies for initialization.

<snip>

>> 4. Dependency directories are at the top of the path and searched first, as
>> one normally would expect and desire, because of the thinking that
>> dependencies may be needed during initialization of the desired package.
>> However, the weak point in this reasoning is that it is valid for *all*
>> packages. Some, if not most, packages don't need dependencies for
>> initialization, they just need dependencies to supply functionality for them
>> at run time, not initialization time.
>
> I guess it depends on the definitions here, but as of now, I do not
> see any troubles in accepting that:
>
> 1. During "initialization" dependencies are available, but the package
> itself isn't (call it pre_load if you want).
>
> 2. After initialization the package is actually loaded and its folders
> added to the top of the search path. At this point dependencies are
> shadowed if the package defines homonymous functions.

That would be ideal.

>> Looking at how pkg.m works, getting together point 3. above would be
>> difficult because of the following;
>>
>> * Process:
>> If we adapt pkg.m, it would very probably only apply to Octave-6.1.x, and
>> the packages matgeom, geometry (and mapping) (the actual packages concerned
>> here) would depend hard on the latest version of Octave and on the latest
>> version of each other. I think that would be an undesirable situation.
>> Packages should be backward-compatible with a few older octave releases.
>>
> I do not generally agree with this, since it is only some
> functionality that is lost when using older version of octave. Most of
> the package still works.

Maybe we should just await the bug reports coming in and see how bad the
perceptions are.

<snip>

> If this is the only solution, then I would rename geometry's functions
> to avoid the whole problem.

Maybe not the only solution, but it could be the best.

Philip

Reply | Threaded
Open this post in threaded view
|

Re: [pkg.m] Load order of dependencies

PhilipNienhuis
In reply to this post by JuanPi
JuanPi wrote

> Hi all,
>
> Is there any reason to make pkg.m load first the package and then the
> dependencies?
>
> Taking the development version of geometry as an example:
>
>> pkg load geometry
> GEOMETRY
> matgeom loaded 0
> MATGEOM
> geometry loaded 1
>
> For me it would make only sense that at the time the package folder is
> added to the path the dependencies are already loaded.

I looked a little more into it and found something I didn't know or realize:

>> addpath ("a", "b")
>> path
Octave's search path contains the following directories:

.
"a"
"b"
:
<rest of path>

so addpath() loads packages in its argument list from right to left; IMO
contrary to at least my expectations. Anyway this isn't documented in "help
addpath".

Maybe this behavior was different in the past when pkg.m was written?

Philip




--
Sent from: https://octave.1599824.n4.nabble.com/Octave-Maintainers-f1638794.html

Reply | Threaded
Open this post in threaded view
|

Re: [pkg.m] Load order of dependencies

PhilipNienhuis
PhilipNienhuis wrote

> JuanPi wrote
>> Hi all,
>>
>> Is there any reason to make pkg.m load first the package and then the
>> dependencies?
>>
>> Taking the development version of geometry as an example:
>>
>>> pkg load geometry
>> GEOMETRY
>> matgeom loaded 0
>> MATGEOM
>> geometry loaded 1
>>
>> For me it would make only sense that at the time the package folder is
>> added to the path the dependencies are already loaded.
>
> I looked a little more into it and found something I didn't know or
> realize:
>
>>> addpath ("a", "b")
>>> path
> Octave's search path contains the following directories:
>
> .
> "a"
> "b"
> :
> <rest of path>
> so addpath() loads packages in its argument list from right to left; IMO
> contrary to at least my expectations. Anyway this isn't documented in
> "help
> addpath".

The more I think about it, the more I'm getting convinced that it is
addpath's counter-intuitive behavior that is the culprit here.

I've filed a bug report + patch for pkg.m to fix the load order of packages
and dependencies and also supplied a doc fix for addpath(). (bug #57403).

Philip




--
Sent from: https://octave.1599824.n4.nabble.com/Octave-Maintainers-f1638794.html

Reply | Threaded
Open this post in threaded view
|

Re: [pkg.m] Load order of dependencies

Andrew Janke-2
In reply to this post by PhilipNienhuis


On 12/12/19 1:36 PM, Philip Nienhuis wrote:

> But: is pkj compatible with all existing packages? or do existing
> packages need to adapt DESCRIPTION entries and/or other things?

pkj is compatible with existing packages. No updates to DESCRIPTION or
other things are needed.

Cheers,
Andrew

Reply | Threaded
Open this post in threaded view
|

Re: [pkg.m] Load order of dependencies

Olaf Till-2
On Fri, Dec 13, 2019 at 09:31:54AM -0500, Andrew Janke wrote:

>
>
> On 12/12/19 1:36 PM, Philip Nienhuis wrote:
>
> > But: is pkj compatible with all existing packages? or do existing
> > packages need to adapt DESCRIPTION entries and/or other things?
>
> pkj is compatible with existing packages. No updates to DESCRIPTION or
> other things are needed.
>
> Cheers,
> Andrew
I had no time to thoroughly look over this thread and the related bug
report, but could this be the old problem dealt with here:

https://savannah.gnu.org/patch/?7976

Olaf

--
public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net

signature.asc (849 bytes) Download Attachment
12