Going to start Octave Package Management.

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Going to start Octave Package Management.

Hodor123456
Respected sir,
I have started working on Octave Package Management and here is my initial idea : https://docs.google.com/document/d/1Xd7dSBa5Bhr10RzjkdZiFujOb6s5Vao_snahXNR7y5Q/edit

I will be updating it time to time. Is there anything I am missing?
Please let me know. Is this project assigned to anyone else?


Reply | Threaded
Open this post in threaded view
|

Re: Going to start Octave Package Management.

Sergei Steshenko


On 1/31/19 5:57 PM, Dildar Sk wrote:
Respected sir,
I have started working on Octave Package Management and here is my initial idea : https://docs.google.com/document/d/1Xd7dSBa5Bhr10RzjkdZiFujOb6s5Vao_snahXNR7y5Q/edit

I will be updating it time to time. Is there anything I am missing?
Please let me know. Is this project assigned to anyone else?


    


It's not even a good start.


I strongly suggest to read "Code Loading" -  https://docs.julialang.org/en/v1/manual/code-loading/ . This chapter in detail describes approaches used in Julia language package management. The approaches address a lot of issues present in Octave package management.


Don't reinvent the wheel to begin with. I.e. analyze what's implemented and how; what are advantages and disadvantages and only then decide what you want to change (if anything) and why.


--Sergei.



Reply | Threaded
Open this post in threaded view
|

Re: Going to start Octave Package Management.

Dildar Sk
Thanks a lot, I am going through the GSoC page. And trying to solve the
problems referred there(Just idea phase). If I am sure about improvement I
would try to implement.



--
Sent from: http://octave.1599824.n4.nabble.com/Octave-General-f1599825.html


Reply | Threaded
Open this post in threaded view
|

Re: Going to start Octave Package Management.

Sergei Steshenko

On 1/31/19 9:29 PM, Dildar Sk wrote:

> Thanks a lot, I am going through the GSoC page. And trying to solve the
> problems referred there(Just idea phase). If I am sure about improvement I
> would try to implement.
>
>
>
> --
> Sent from: http://octave.1599824.n4.nabble.com/Octave-General-f1599825.html
>
>

Julia folks have not deeply enough addressed the issue of package
dependencies whose source is not in Julia language. Though I've stopped
monitoring Octave package development, I think Octave folks haven't
addressed the same issue deeply enough either.


Yesterday I was playing with Julia plotting, and my initial attempts
failed because a couple Qt libraries was missing. Installing the
libraries resolved the issue, but I needed to first conduct web search
on error messages I've encountered.


This issue is a big can of fat worms - regardless of language. This is
because to resolve the issue one would have to come up with a set of
methods that allow to establish presence of the needed dependencies, and
the methods would depend on OS and distro. I.e. even within Linux
different distros may have different versioning and different packaging
(by that I mean which files are included in which package) - let alone
Windows, *BSD and MacOs.


And in Linux we have DEB, RPM, ArchLinux, Gentoo, Pogo Linux (they used
to have and maybe still have a really nice packaging system with
separate tree per package) packaging systems - for each packaging system
we would have to implement a method to determine whether the needed
non-native (i.e. in case of Octave not in Octave language, in case of
Julia not in Julia language, etc) package is installed. And before that
we need to implement methods mapping files to packages. I.e. if an
Octave package needs libFoo.so, we'll need to implement methods mapping
libFoo.so to PackageContaininglibFoo.

What I mean is that in the end I want Octave, Julia, etc packaging
system to tell me: libFoo.so is missing, install PackageContaininglibFoo
to resolve the problem.


For that matter, one can truly appreciate the destructive force of GNU
Bazaar vs whatever Cathedral.


--Sergei.



Reply | Threaded
Open this post in threaded view
|

Re: Going to start Octave Package Management.

Dildar Sk
Thanks a lot for those insights.
"What I mean is that in the end I want Octave, Julia, etc packaging
system to tell me: libFoo.so is missing, install PackageContaininglibFoo
to resolve the problem. "

Already pkg helps by telling the package name in which the function is
available(if it' not available) and to download.



--
Sent from: http://octave.1599824.n4.nabble.com/Octave-General-f1599825.html


Reply | Threaded
Open this post in threaded view
|

Re: Going to start Octave Package Management.

Sergei Steshenko
In reply to this post by Sergei Steshenko

On 2/4/19 3:37 PM, Dildar Sk wrote:
> The answer would help me a lot. Thanks for your time.
>
>
>
> --
> Sent from: http://octave.1599824.n4.nabble.com/Octave-General-f1599825.html
>
>

I have a rough idea (which most easily can be implemented on Arch Linux
- though I've never run an Arch type distro) how to "marry" Octave (and
for that matter Julia) packaging with OS packaging. Gentoo is similar to
Arch in this respect.


The idea is to split package building and installation into two stages:

1) Octave/Julia script creates a source package. And the package
contains dependencies and its build method;

2) the source package is in the format supported by the OS.


As a result one first installs the Octave/Julia source package - which
means installation of its dependencies first if the OS package manager
is intelligent enough (nowadays they are).


Then as the source package is installed, the build method (typically a
script) is called, and the usable package is built. At this stage
Octave/Julia will be called to produce the corresponding files.


And a lot of disparate, but useful info can be found at
https://fusion809.github.io/package-development/ . Pay attention to OBS
there - maybe this infrastructure can be used to hide the GNU Bazaar
complexity.


--Sergei.



Reply | Threaded
Open this post in threaded view
|

Re: Going to start Octave Package Management.

Dildar Sk
If we do,
>> pkg update signal
error: 'installed_pkgs_list' undefined near line 530 column 29
error: called from
    pkg at line 530 column 26

I don't know if it is a bug or not. But in such cases there would be no
possible way to update a single intended package. In this case there will be
no package with multiple version available(If we do pkg update it would
update all packages including package_something-older,
package_something-moreOlder to latest version) Because there is no choice to
upate package_something-someSpecificVersion.



--
Sent from: http://octave.1599824.n4.nabble.com/Octave-General-f1599825.html


Reply | Threaded
Open this post in threaded view
|

Re: Going to start Octave Package Management.

Sergei Steshenko

On 2/4/19 5:47 PM, Dildar Sk wrote:

> If we do,
>>> pkg update signal
> error: 'installed_pkgs_list' undefined near line 530 column 29
> error: called from
>      pkg at line 530 column 26
>
> I don't know if it is a bug or not. But in such cases there would be no
> possible way to update a single intended package. In this case there will be
> no package with multiple version available(If we do pkg update it would
> update all packages including package_something-older,
> package_something-moreOlder to latest version) Because there is no choice to
> upate package_something-someSpecificVersion.
>
>
>
> --
> Sent from: http://octave.1599824.n4.nabble.com/Octave-General-f1599825.html
>
>
"I don't know if it is a bug or not" - forget about 'pkg.m' for a while.
It's a vivid example how SW should NOT be developed - because the whole
Octave packaging system to the best of my knowledge has never been
properly spec'ed.

--Sergei.



Reply | Threaded
Open this post in threaded view
|

Re: Going to start Octave Package Management.

nrjank

On Mon, Feb 4, 2019 at 10:58 AM Sergei Steshenko <[hidden email]> wrote:
> --
> Sent from: http://octave.1599824.n4.nabble.com/Octave-General-f1599825.html
>
>
"I don't know if it is a bug or not" - forget about 'pkg.m' for a while.

A good point.  a different line of thought might be - how do current package managers that can handle presence of multiple installed versions handle a general "update all" request?   I think users would generally expect a blanket 'pkg update' like function. Does python's pip allow it? what about OS package managers? 


Reply | Threaded
Open this post in threaded view
|

Re: Going to start Octave Package Management.

apjanke-floss


On 2/4/19 12:27 PM, Nicholas Jankowski wrote:

>
> On Mon, Feb 4, 2019 at 10:58 AM Sergei Steshenko <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>      > --
>      > Sent from:
>     http://octave.1599824.n4.nabble.com/Octave-General-f1599825.html
>      >
>      >
>     "I don't know if it is a bug or not" - forget about 'pkg.m' for a
>     while.
>
> A good point.  a different line of thought might be - how do current
> package managers that can handle presence of multiple installed versions
> handle a general "update all" request?   I think users would generally
> expect a blanket 'pkg update' like function. Does python's pip allow it?
> what about OS package managers?
>
>

Here's how a few different package managers I know handle this:

pip only installs a single version of each package at a time. "upgrade"
upgrades all packages to the latest version, removing the old version.
Multiple versions are handled through virtual environments.

conda only installs a single version of each package at a time.
"upgrade" upgrades all packages to the latest version, removing the old
version (unless you have pins, and within the version-range requirements
in your environment specification, in which case it will just keep the
old version and not install a new version). Multiple versions are
handled through virtual environments.

Ruby's gem can install multiple versions of a package at the same time
in a single environment. It also supports virtual environments through
both RVM and bundler. "gem update" will install the latest version of
each installed package, and will also leave all previously-installed
versions still installed, so it just adds new installed versions.

I don't know how NVM works.

Homebrew can install multiple versions side-by-side, but since it only
stores a definition for the latest version of a package, the only way to
do this is to have a pre-existing installation in place from an older
Homebrew version. "upgrade" installs the latest version of all packages,
and will leave older versions in place only if it detects that other
installed "kegs" have a dependency on them (at the native library
linkage level, by looking at .dylibs). It will automatically remove
older versions if it detects no lingering dependencies on them, at the
time of the upgrade. A separate "cleanup" operation will remove old
retained versions once they are no longer depended on (if they were
originally retained at upgrade time due to keg dependencies). But only
the latest version of the package is "linked" and active (visible) in
the system at large. No virtual environment support; if you want
scenarios with different versions, you install multiple complete
Homebrew installations at different paths and switch between them by
manipulating your $PATH.

There are also version-aware dependency managers like Maven to consider.
Maven doesn't have a global installation location; all installations are
local to a build. All dependencies are specified to an exact version,
and having multiple versions of a given package in a dependency graph is
an error, so there's no support for multiple versions of a package in a
given environment.

apt and all other Linux distro package managers that I'm aware of only
install a single version of a package at a time. (I think.) Side-by-side
installations of different versions of a program or library are handled
by making them distinct packages and incorporating part of the version
number in the package name. (Homebrew does this too.) "upgrade" upgrades
everything to the latest version, removing old versions. This is
probably a consequence of the fact that Linux packages install stuff
into global system locations instead of segregated per-package spaces.

Let's consider: if pkg supports multiple-versioning, does it want to
handle selection of those versions via a "virtual environment" mechanism
where you have scoped installations, or by selection of package versions
at "pkg load" time?

This stuff has actually been on my mind lately because I'm working on a
package manager for Matlab that will work similarly to Octave's pkg, and
will also have multiple-version support. I'm leaning toward doing
version resolution by having multiple versions of packages all installed
in a single scope, and selecting versions at "pkg load" time using
"package sets", which are possibly-named sets of (package, versionspec)
pairs, where versionspec may be an exact version or a set of "version
</<=/==/>=/>/!=" requirement specifiers. I prefer this to the "virtual
environment" approach because virtual environments (that I've seen)
require you to instantiate the installation of each particular
combination of library versions you want to run against; in a testing
scenario, I'd like to test against all the combinations of, say, the
last 3 versions of 5 libraries I'm coding against, and I want to just
load in each combination dynamically instead of having to manage 15
named environments.

Cheers,
Andrew


Reply | Threaded
Open this post in threaded view
|

Re: Going to start Octave Package Management.

Dildar Sk
I think pkg has many more problems. But I think just refactoring would be
enough. What you people say?
I saw this project got stuck many times refactoring. I am stuck between
thinking refactor and full rewrite. Thanks a lot, Expecting answer from
experienced.



--
Sent from: http://octave.1599824.n4.nabble.com/Octave-General-f1599825.html


Reply | Threaded
Open this post in threaded view
|

Re: Going to start Octave Package Management.

Sergei Steshenko

On 2/9/19 10:40 PM, Dildar Sk wrote:

> I think pkg has many more problems. But I think just refactoring would be
> enough. What you people say?
> I saw this project got stuck many times refactoring. I am stuck between
> thinking refactor and full rewrite. Thanks a lot, Expecting answer from
> experienced.
>
>
>
> --
> Sent from: http://octave.1599824.n4.nabble.com/Octave-General-f1599825.html
>
>

"I think just refactoring would be enough" - of course, not.


All in all, dealing with Octave package management, keep in mind what I
call the sardonic trinity:


1) you can't teach old dogs new tricks;

2) you can't weld rust;

3) you can't polish shit.


--Sergei.