Package system

classic Classic list List threaded Threaded
19 messages Options
Reply | Threaded
Open this post in threaded view
|

Package system

Søren Hauberg
Hi,
I just turned in a larger project for school, so now I have some time to
look at package management. Here's a summary of what's cooking in the
land of package systems, for details see
http://hauberg.org/octave-package/

== Good news ==
I think the package system is fully implemented, with support for the
following:
  *) Dependency checking at install and uninstall time.
  *) Multible installation roots, meaning it is possible to have
per-user packages on multi-user systems.
  *) Octave-forge packages 'signal' and 'image' was very easy to port,
so I assume the rest of Octave-forge will be too.

== Bad news ==
1) There's currently alot of code. It's all written in .m files and the
code should be pretty easiy to read once you get the structure of a
package. But still, there's alot of code.

2) Some of the code is not system-independent, and might not be easily
ported to platforms other then *nix. I haven't worked on another
platform then Linux for years, so I really don't know.

3) There's currently no other documentation.

/Søren

Reply | Threaded
Open this post in threaded view
|

Re: Package system

Stéfan van der Walt
Søren,

I checked out the packaging system yesterday (I was looking into
converting my FFMpeg-wrappers into an Octave-quaver or note or
whatever we call it.. python has eggs, ruby has gems!).

Question: what is the status of the system dependencies flag?  For
example, how would one go about requiring libgsl.so to be installed?

Regards
Stéfan

On Wed, Oct 19, 2005 at 07:44:47PM +0200, Søren Hauberg wrote:
> http://hauberg.org/octave-package/

Reply | Threaded
Open this post in threaded view
|

Re: Package system

Søren Hauberg
Stefan,
I uploaded the most recent code about the same time I wrote the last
message, so if you got code from before that I'd recommend
re-downloading (don't think there's been many changes, but still).

And your question, ...
Well, the basic idea is that you inform the user that FFMpeg is needed
(there's a field in DESCRIPTION called SystemRequirements for this), but
the package system doesn't check it. So to check if FFMpeg is installed
you need some autoconfig magic. This kinda sucks, but checking if a
library is installed is hard, so I don't think octave should try to do
configure's job.

/Søren

P.S. This is the same way R does it

ons, 19 10 2005 kl. 23:42 +0200, skrev Stefan van der Walt:

> Søren,
>
> I checked out the packaging system yesterday (I was looking into
> converting my FFMpeg-wrappers into an Octave-quaver or note or
> whatever we call it.. python has eggs, ruby has gems!).
>
> Question: what is the status of the system dependencies flag?  For
> example, how would one go about requiring libgsl.so to be installed?
>
> Regards
> Stéfan
>
> On Wed, Oct 19, 2005 at 07:44:47PM +0200, Søren Hauberg wrote:
> > http://hauberg.org/octave-package/
>

Reply | Threaded
Open this post in threaded view
|

Re: Package system

Stéfan van der Walt
On Thu, Oct 20, 2005 at 12:51:28AM +0200, Søren Hauberg wrote:
> And your question, ...
> Well, the basic idea is that you inform the user that FFMpeg is needed
> (there's a field in DESCRIPTION called SystemRequirements for this), but
> the package system doesn't check it. So to check if FFMpeg is installed
> you need some autoconfig magic. This kinda sucks, but checking if a
> library is installed is hard, so I don't think octave should try to do
> configure's job.

I agree.  Thanks for explaining exactly what SystemRequirements means.

Stéfan

Reply | Threaded
Open this post in threaded view
|

Re: Package system

Søren Hauberg
In reply to this post by Søren Hauberg
I just updated the code to include some documentation. You can get it
from http://hauberg.org/octave-package/package-0.2.1.tgz

/Søren

ons, 19 10 2005 kl. 19:44 +0200, skrev Søren Hauberg:

> Hi,
> I just turned in a larger project for school, so now I have some time to
> look at package management. Here's a summary of what's cooking in the
> land of package systems, for details see
> http://hauberg.org/octave-package/
>
> == Good news ==
> I think the package system is fully implemented, with support for the
> following:
>   *) Dependency checking at install and uninstall time.
>   *) Multible installation roots, meaning it is possible to have
> per-user packages on multi-user systems.
>   *) Octave-forge packages 'signal' and 'image' was very easy to port,
> so I assume the rest of Octave-forge will be too.
>
> == Bad news ==
> 1) There's currently alot of code. It's all written in .m files and the
> code should be pretty easiy to read once you get the structure of a
> package. But still, there's alot of code.
>
> 2) Some of the code is not system-independent, and might not be easily
> ported to platforms other then *nix. I haven't worked on another
> platform then Linux for years, so I really don't know.
>
> 3) There's currently no other documentation.
>
> /Søren
>

Reply | Threaded
Open this post in threaded view
|

Re: Package system

John W. Eaton-6
On 21-Oct-2005, Søren Hauberg wrote:

| I just updated the code to include some documentation. You can get it
| from http://hauberg.org/octave-package/package-0.2.1.tgz

This looks good so far.  Thanks for working on it.

I have a few comments:

  * I recently modified rmdir to accept an optional second argument
    ("s") which if present tells rmdir to work recursively (this
    choice was for compatibility; I guess the "s" means
    "subdirectories too"), so we shouldn't need the rm_rf.m file.

  * We probably need something different from the issuperuser
    function, which checks to see if the environment variable USER has
    a value of "root".  That's probably not a good test.  It's likely
    to fail on Windows systems (Cygwin or MinGW).  Even on Unixy
    systems it would miss the case of an additional user name with a
    UID of 0.

    Checking for root is maybe not the right thing anyway.  We just
    want to know whether we can install files in Octave's site
    directory.  That's a function of more than just being root.  Maybe
    we should do something like

      use_system_dir = false;
      fs = stat (system_dir);
      if (S_ISDIR (fs.mode))
        ## Directory exists.  Can we create a file there?
        tfile = strcat (system_dir, filesep, "foo");
        fid = fopen (tfile, "w");
        if (fid > 0)
          ## We can write files in existing directory, so we should be
          ## able to use it.
          use_system_dir = true;
          fid = fclose (tfile);
          unlink (tfile);
        endif
      else if (mkdir (system_dir))
        ## Directory did not exist, but we were able to create it, so
        ## we should be able to use it.
        use_system_dir = true;
      endif

    (This code will only work with recent CVS Octave due to changes in
    the way mkdir works and the use of S_ISDIR.)

  * We should make the untar function compatible with Matlab's untar
    function and include it with Octave.  (Maybe implement the tar,
    zip, unzip, gzip, and gunzip functions at the same time.)

  * The remaining functions currently have the following names

      installed_packages.m
      load_packages.m
      uninstall.m
      get_unsatisfied_deps.m
      install.m

    Maybe we should have just one command?

      pkg -list
      pkg -install -nodeps pkg-name ...
      pkg -uninstall nodeps pkg-name ...
      pkg -require pkg-name ...

    or multiple names, but all starting with pkg:

      pkg_list
      pkg_install
      pkg_uninstall
      pkg_require (or pkg_load)

    If the get_unsatisfied_deps is not intended to be called by
    users, then it should have a name like __pkg_uninstalled_deps__.

Thanks,

jwe

Reply | Threaded
Open this post in threaded view
|

Re: Package system

Bill Denney
On Fri, 21 Oct 2005, John W. Eaton wrote:

> I have a few comments:
>
>  * The remaining functions currently have the following names
>
>      installed_packages.m
>      load_packages.m
>      uninstall.m
>      get_unsatisfied_deps.m
>      install.m
>
>    Maybe we should have just one command?
>
>      pkg -list
>      pkg -install -nodeps pkg-name ...
>      pkg -uninstall nodeps pkg-name ...
>      pkg -require pkg-name ...
>
>    or multiple names, but all starting with pkg:
>
>      pkg_list
>      pkg_install
>      pkg_uninstall
>      pkg_require (or pkg_load)
>
>    If the get_unsatisfied_deps is not intended to be called by
>    users, then it should have a name like __pkg_uninstalled_deps__.

Perhaps for pkg_list, it could be implemented like matlab's ver command?

Bill

--
"You can tell whether a man is clever by his answers. You can tell
whether a man is wise by his questions."
   -- Naguib Mahfouz

Reply | Threaded
Open this post in threaded view
|

Re: Package system

John W. Eaton-6
On 21-Oct-2005, Bill Denney wrote:

| Perhaps for pkg_list, it could be implemented like matlab's ver command?

The installed_packags function (what I would call pkg_list) currently
returns a cell array of structures (I suppose it should maybe return a
struct array instead) that contains a lot more information than the
ver command provides.  So maybe we should keep pkg_list as it is and
use it to implement a mostly compatible ver function?

jwe

Reply | Threaded
Open this post in threaded view
|

Re: Package system

Søren Hauberg
In reply to this post by John W. Eaton-6
Hi,

fre, 21 10 2005 kl. 13:18 -0400, skrev John W. Eaton:

> On 21-Oct-2005, Søren Hauberg wrote:
>
> | I just updated the code to include some documentation. You can get it
> | from http://hauberg.org/octave-package/package-0.2.1.tgz
>
> This looks good so far.  Thanks for working on it.
>
> I have a few comments:
>
>   * I recently modified rmdir to accept an optional second argument
>     ("s") which if present tells rmdir to work recursively (this
>     choice was for compatibility; I guess the "s" means
>     "subdirectories too"), so we shouldn't need the rm_rf.m file.
Okay. I'm using 2.9.3 and would prefer not working on CVS version as a have
to do "real work" too. I'm guessing CVS is too unstable for that, right?

>   * We probably need something different from the issuperuser
>     function, which checks to see if the environment variable USER has
>     a value of "root".  That's probably not a good test.  It's likely
>     to fail on Windows systems (Cygwin or MinGW).  Even on Unixy
>     systems it would miss the case of an additional user name with a
>     UID of 0.
>
>     Checking for root is maybe not the right thing anyway.  We just
>     want to know whether we can install files in Octave's site
>     directory.  That's a function of more than just being root.  Maybe
>     we should do something like
>
>       use_system_dir = false;
>       fs = stat (system_dir);
>       if (S_ISDIR (fs.mode))
>         ## Directory exists.  Can we create a file there?
>         tfile = strcat (system_dir, filesep, "foo");
>         fid = fopen (tfile, "w");
>         if (fid > 0)
>           ## We can write files in existing directory, so we should be
>           ## able to use it.
>           use_system_dir = true;
>           fid = fclose (tfile);
>           unlink (tfile);
>         endif
>       else if (mkdir (system_dir))
>         ## Directory did not exist, but we were able to create it, so
>         ## we should be able to use it.
>         use_system_dir = true;
>       endif
>
>     (This code will only work with recent CVS Octave due to changes in
>     the way mkdir works and the use of S_ISDIR.)
As above.

>   * We should make the untar function compatible with Matlab's untar
>     function and include it with Octave.  (Maybe implement the tar,
>     zip, unzip, gzip, and gunzip functions at the same time.)
I'd be happy to this (writing untar was dead easy), but I just have two
questions:
1) Could we change split to return a cell array instead of an array?
This would IMHO be a good thing anyway, and would make untar work
without a lot of hassle. There's a patch for this somewhere on the bugs
list.
2) Is the approach taking in untar acceptable? It's basicly just a
system call, but that's not going to be portable (I'm thinking about
windows here).

>   * The remaining functions currently have the following names
>
>       installed_packages.m
>       load_packages.m
>       uninstall.m
>       get_unsatisfied_deps.m
>       install.m
>
>     Maybe we should have just one command?
>
>       pkg -list
>       pkg -install -nodeps pkg-name ...
>       pkg -uninstall nodeps pkg-name ...
>       pkg -require pkg-name ...
>
>     or multiple names, but all starting with pkg:
>
>       pkg_list
>       pkg_install
>       pkg_uninstall
>       pkg_require (or pkg_load)
>
>     If the get_unsatisfied_deps is not intended to be called by
>     users, then it should have a name like __pkg_uninstalled_deps__.
I have just changed the syntax to one function called pkg that accepts
-install, -uninstall, etc, as you described. Then get_unsatisfied_deps
can become a function inside the pkg file.

/Søren

> Thanks,
>
> jwe
>

Reply | Threaded
Open this post in threaded view
|

Re: Package system

John W. Eaton-6
On 21-Oct-2005, Søren Hauberg wrote:

| >   * I recently modified rmdir to accept an optional second argument
| >     ("s") which if present tells rmdir to work recursively (this
| >     choice was for compatibility; I guess the "s" means
| >     "subdirectories too"), so we shouldn't need the rm_rf.m file.
| Okay. I'm using 2.9.3 and would prefer not working on CVS version as a have
| to do "real work" too. I'm guessing CVS is too unstable for that, right?

I hope to make a new snapshot soon.

| I'd be happy to this (writing untar was dead easy), but I just have two
| questions:
| 1) Could we change split to return a cell array instead of an array?
| This would IMHO be a good thing anyway, and would make untar work
| without a lot of hassle.

Yes, I think it would be better to return a cell.  But how much code
might that break?  Should we just do it anyway?

| There's a patch for this somewhere on the bugs list.

I don't see the patch, so if you could find it that would help.

| 2) Is the approach taking in untar acceptable? It's basicly just a
| system call, but that's not going to be portable (I'm thinking about
| windows here).

It's not really a problem if Octave is running under Cygwin (one of
the big advantages Cygwin provides is a useful POSIX environment).
But yes, it would be a problem for Octave running on Windows without
Cygwin.  But I don't think it makes sense to try to duplicate the
functionality of things like tar, gzip, or zip inside of Octave.  So
using system() is probably the best solution.

| I have just changed the syntax to one function called pkg that accepts
| -install, -uninstall, etc, as you described. Then get_unsatisfied_deps
| can become a function inside the pkg file.

OK.

Since Octave allows

  x = pkg -list

I think we are OK with this change.  OTOH, that kind of statement
always looks a little odd to me.  OTTH, I suppose you can always write

  x = pkg ("-list")

I'd like to include the package code with Octave.  Is that OK with
you?

Thanks,

jwe

Reply | Threaded
Open this post in threaded view
|

Re: Package system

Søren Hauberg
fre, 21 10 2005 kl. 16:20 -0400, skrev John W. Eaton:

> On 21-Oct-2005, Søren Hauberg wrote:
>
> | >   * I recently modified rmdir to accept an optional second argument
> | >     ("s") which if present tells rmdir to work recursively (this
> | >     choice was for compatibility; I guess the "s" means
> | >     "subdirectories too"), so we shouldn't need the rm_rf.m file.
> | Okay. I'm using 2.9.3 and would prefer not working on CVS version as a have
> | to do "real work" too. I'm guessing CVS is too unstable for that, right?
>
> I hope to make a new snapshot soon.
Sounds good.

> | I'd be happy to this (writing untar was dead easy), but I just have two
> | questions:
> | 1) Could we change split to return a cell array instead of an array?
> | This would IMHO be a good thing anyway, and would make untar work
> | without a lot of hassle.
>
> Yes, I think it would be better to return a cell.  But how much code
> might that break?  Should we just do it anyway?
A quick grep told me that octave only uses split in very few functions,
so that should be easy to change. However, it's hard to tell how many people
are using the function. I think it's okay to break the API when doing
3.0 - that is the point of a major release. So I say, make the change.

> | There's a patch for this somewhere on the bugs list.
>
> I don't see the patch, so if you could find it that would help.
I'm sorry my memory was misleading me. The patch is on the maintainers list:
http://www.octave.org/octave-lists/archive/octave-maintainers.2005/msg00536.html
(although I don't know how to download the patch from that link).

> | 2) Is the approach taking in untar acceptable? It's basicly just a
> | system call, but that's not going to be portable (I'm thinking about
> | windows here).
>
> It's not really a problem if Octave is running under Cygwin (one of
> the big advantages Cygwin provides is a useful POSIX environment).
> But yes, it would be a problem for Octave running on Windows without
> Cygwin.  But I don't think it makes sense to try to duplicate the
> functionality of things like tar, gzip, or zip inside of Octave.  So
> using system() is probably the best solution.
Okay, will look into tar, zip, unzip, etc.

> | I have just changed the syntax to one function called pkg that accepts
> | -install, -uninstall, etc, as you described. Then get_unsatisfied_deps
> | can become a function inside the pkg file.
>
> OK.
>
> Since Octave allows
>
>   x = pkg -list
>
> I think we are OK with this change.  OTOH, that kind of statement
> always looks a little odd to me.  OTTH, I suppose you can always write
>
>   x = pkg ("-list")
Yeah, it looks strange to me too. The reason I went with this change is
that I throw all functions into one giant file.

> I'd like to include the package code with Octave.  Is that OK with
> you?
Of course. A package system isn't worth much if you have to install it
first :-)

/Søren

>
> Thanks,
>
> jwe
>

Reply | Threaded
Open this post in threaded view
|

RE: Package system

HALL, BENJAMIN            PW
In reply to this post by Søren Hauberg
>> | 1) Could we change split to return a cell array instead of an array?
>> | This would IMHO be a good thing anyway, and would make untar work
>> | without a lot of hassle.
>>
>> Yes, I think it would be better to return a cell.  But how much code
>> might that break?  Should we just do it anyway?
>A quick grep told me that octave only uses split in very few functions,
>so that should be easy to change. However, it's hard to tell how many
people
>are using the function. I think it's okay to break the API when doing
>3.0 - that is the point of a major release. So I say, make the change.

I agree, split returning a cell array would be a nice improvement.

Reply | Threaded
Open this post in threaded view
|

Re: Package system

Søren Hauberg
In reply to this post by John W. Eaton-6
fre, 21 10 2005 kl. 16:20 -0400, skrev John W. Eaton:
> I'd like to include the package code with Octave.  Is that OK with
> you?
As all code should be documented I've written a first draft of what I feel should go into
the manual. It's in texinfo and can be found at

http://hauberg.org/octave-package/package.texi

This isn't quite done, but the package system itself isn't quite done
either, so that shouldn't be a surprise.
Please remember that my first language isn't english so the above might
contain some varry badd engrish.

/Søren

> Thanks,
>
> jwe
>

Reply | Threaded
Open this post in threaded view
|

Re: Package system

Paul Kienzle
In reply to this post by John W. Eaton-6

On Oct 21, 2005, at 1:18 PM, John W. Eaton wrote:

>   * We should make the untar function compatible with Matlab's untar
>     function and include it with Octave.  (Maybe implement the tar,
>     zip, unzip, gzip, and gunzip functions at the same time.)

The name 'tar' collides with a function in octave-forge.  This
should be removed or renamed.  Note that it is used by functions
in vrml as well.

- Paul

Reply | Threaded
Open this post in threaded view
|

Re: Package system

Paul Kienzle
In reply to this post by John W. Eaton-6

On Oct 21, 2005, at 1:18 PM, John W. Eaton wrote:

>   * The remaining functions currently have the following names
>
>       installed_packages.m
>       load_packages.m
>       uninstall.m
>       get_unsatisfied_deps.m
>       install.m
>
>     Maybe we should have just one command?
>
>       pkg -list
>       pkg -install -nodeps pkg-name ...
>       pkg -uninstall nodeps pkg-name ...
>       pkg -require pkg-name ...
>
>     or multiple names, but all starting with pkg:
>
>       pkg_list
>       pkg_install
>       pkg_uninstall
>       pkg_require (or pkg_load)


Maintaining my debian system became a lot more pleasant once I started
using wajig.  Using a single prefix for all commands is a very good
thing.

Give that the pkg command is never called without a '-subcommand' I
would argue for pkg_subcommand.  Later when we have syntactic support
for namespaces then we can break it into pkg.subcommand (assuming a
namespace is a first class object like in python) or pkg::subcommand
(assuming a namespace is something syntactic like in C++).

At a minimum, use the syntax "pkg subcommand" rather than "pkg
-subcommand" since the subcommand is a requirement rather than an
option.

- Paul

Reply | Threaded
Open this post in threaded view
|

Re: Package system

Paul Kienzle
In reply to this post by Søren Hauberg

On Oct 21, 2005, at 11:56 AM, Søren Hauberg wrote:

> I just updated the code to include some documentation. You can get it
> from http://hauberg.org/octave-package/package-0.2.1.tgz
>

Some packaging issues that occur in octave-forge:

- main/audio creates a binary that should be installed in the helper
binary directory.  I put them into octave_config_info
("localverarchlibdir").

- main/audio/data has a sample wav file which I install into the data
path for the package.  In this case, I simply use the package path
itself and find from the script using file_in_loadpath('file.wav').

- main/comm/doc has texinfo documentation which should be built and
installed in the usual texinfo places.  Pure documentation packages
such as doc/coda are also possible.

- main/control is an example of a single function package intended to
extend an existing package.  Does it need its own directory?  Or can we
request that it be installed in the same directory as the control
package, while updating the list of functions reported by "help
control"?

- extra/engine has libraries and headers that are needed by other
programs.  They should be installed in /usr/include and /usr/lib or
/usr/local/include and /usr/local/lib.

- extra/mex and main/fixed have libraries and headers that need to be
available for compiling other oct-files.  They should be installed
along side the octave libraries and headers.

- extra/graceplot introduces alternatives. The package system should
support this in some way so that the user can toggle at run time
between graceplot and gnuplot.  extra/nan, which replaces some standard
octave commands with alternatives which ignore NaN, could also be
handled with alternatives.

- extra/perl and extra/soctcl are packages for other languages so they
are not our problem ;-)

- extra/MacOSX and extra/Windows are system dependent.  It's the user's
problem to decide whether or not to install them.

- configure.base contains configure information useful to a number of
packages.  Much of this is available in modern versions of octave,
through system commands such as octave-config and through the octave
command octave_config_info.  Other things such as the flags to the cp
command, and the details of txi->info conversion should be part of the
packaging system.

- anticipate translated help files for multiple languages as part of
the package.

- I didn't see where you created the PKG_ADD file by extracting PKG_ADD
lines from the individual m-files.


Some more general comments:

- package version should be available at run time.  Currently I have
OCTAVE_FORGE_VERSION.m for all the packages.  Maybe a version
subcommand on the pkg command is the best way to handle this?

- we should be able to build and test the packages before installing
them.

- we should note whenever a installed command shadows an existing
command.

- in octave forge most tests are embedded in the files themselves and
discovered with admin/mktests.sh.  The tests should be installed along
with the functions so that the user can see at run time what tests the
function is supposed to pass.  Also some tests are actually demo blocks
which are part of the documentation of the function.  Some tests are
embedded in C++ files.  These will need to be extracted before
installing.  In retrospect tests should perhaps be in separate .test
files in both cases, and maybe separate .demo files for the demos.

- for windows systems which may not have the compilers installed, we
may want to support binary packages.  This would be advised for OS X
systems as well, where tracking down and installing third party
libraries prior to installing the package would be a pain.

- the packaging system should be available and useful for matlab as
well.  That way authors of packages such as wavelab will have less work
to do to support octave.  For example, if you automatically generate
the contents.m from INDEX and admin/make_index the contents file will
be easier to maintain and it is a win for them.  Similarly, we should
Do the Right Thing with a usual matlab tarball which has contents.m
rather than COPYING DESCRIPTION and INDEX files.

Paul

Reply | Threaded
Open this post in threaded view
|

Re: Package system

Søren Hauberg
Hi,
Thanks for the info on octave-forge.

lør, 22 10 2005 kl. 16:59 -0400, skrev Paul Kienzle:
> - main/audio creates a binary that should be installed in the helper
> binary directory.  I put them into octave_config_info
> ("localverarchlibdir").
One problem with this is that not all users have write access to this directory.
Would it be possible to install this binary somewhere else, or would we
loose functionality then?

> - main/audio/data has a sample wav file which I install into the data
> path for the package.  In this case, I simply use the package path
> itself and find from the script using file_in_loadpath('file.wav').
I see you simply put the .wav file in the same directory as the .m files, so the easy
solution is simply put the .wav file in the 'inst' directory. However it might be a good
idea to provide some standard way to provide sample data with a package.

> - main/comm/doc has texinfo documentation which should be built and
> installed in the usual texinfo places.  Pure documentation packages
> such as doc/coda are also possible.
Yesterday, I submitted a patch to the bugs list (not a very good patch, but still) that makes
'help -i function' look for a file called doc.info in the directory
where 'function' lives, and if this file exist use it. If such
functionality gets integrated we would have a standard way of providing
documentation with packages.

> - main/control is an example of a single function package intended to
> extend an existing package.  Does it need its own directory?  Or can we
> request that it be installed in the same directory as the control
> package, while updating the list of functions reported by "help
> control"?
Hmmm, currently the answer is no. I need packages to go into separate directories in order
to make uninstallation easy. I'd really like not to keep track of each
file provided by a package, as I think this makes the system more
simple. It is however possible that it becomes to simple...

> - extra/engine has libraries and headers that are needed by other
> programs.  They should be installed in /usr/include and /usr/lib or
> /usr/local/include and /usr/local/lib.
>
> - extra/mex and main/fixed have libraries and headers that need to be
> available for compiling other oct-files.  They should be installed
> along side the octave libraries and headers.
The problem here is that not all users have the privileges to do this. We could of course
check this and tell the user if she doesn't have the needed privileges.
This should be possible in the current setup, by providing
'pre_install.m', 'post_install.m', and 'on_uninstall.m'. These could
also handle the installation of the files. However, this really isn't a
very pretty solution (it's really a hack). Personaly, I don't think the
package system should take extra care to make packages like this
possible as I'm guessing there won't be many packages like this.

> - extra/graceplot introduces alternatives. The package system should
> support this in some way so that the user can toggle at run time
> between graceplot and gnuplot.  extra/nan, which replaces some standard
> octave commands with alternatives which ignore NaN, could also be
> handled with alternatives.
Hmmm, I guess that 'pkg load' (or whatever name we choose) could add the
package to the beginning of the path, thereby overriding other
functions. The user can then choose which functionality to use by the
order in which packages gets loaded. Not very user-friendly, but it
should work.
As to the gnuplot/graceplot issue, would it be possible to provide
gnuplot as a package as well (removing gnuplot from core octave)? That
way the user could choose which system to use by loading the right
package. (This reminds me that we should have a 'pkg unload' (or
whatever naming we use))

> - extra/perl and extra/soctcl are packages for other languages so they
> are not our problem ;-)
I do love the use of a "Not My Problem Field" :-)

> - extra/MacOSX and extra/Windows are system dependent.  It's the user's
> problem to decide whether or not to install them.
Makes sense.

> - configure.base contains configure information useful to a number of
> packages.  Much of this is available in modern versions of octave,
> through system commands such as octave-config and through the octave
> command octave_config_info.  Other things such as the flags to the cp
> command, and the details of txi->info conversion should be part of the
> packaging system.
Couldn't this file be maintained by octave-forge and then copied to the packages that need
it? As to the txi->info issue, I think we need a standard way of providing extra
documentation (see above).

> - anticipate translated help files for multiple languages as part of
> the package.
I don't really know how this is going to be implemented. What is the
needs of the translation project?

> - I didn't see where you created the PKG_ADD file by extracting PKG_ADD
> lines from the individual m-files.
I don't, because I really don't know what that means :-)
What is PKG_ADD?

> Some more general comments:
>
> - package version should be available at run time.  Currently I have
> OCTAVE_FORGE_VERSION.m for all the packages.  Maybe a version
> subcommand on the pkg command is the best way to handle this?
Guess you're right. It was mentioned that matlab has a 'ver' command, which could be
implemented on top of 'pkg list'. Perhaps this would give the needed infomation.

> - we should be able to build and test the packages before installing
> them.
Yes! Octave-forge has scripts to handle this, right? Could these be integrated with octave?

> - we should note whenever a installed command shadows an existing
> command.
Good idea. Shouldn't be to hard to implement.

> - in octave forge most tests are embedded in the files themselves and
> discovered with admin/mktests.sh.  The tests should be installed along
> with the functions so that the user can see at run time what tests the
> function is supposed to pass.  Also some tests are actually demo blocks
> which are part of the documentation of the function.  Some tests are
> embedded in C++ files.  These will need to be extracted before
> installing.  In retrospect tests should perhaps be in separate .test
> files in both cases, and maybe separate .demo files for the demos.
So what you're suggesting is that octave provides two function
*) test('somefile.test')
*) demo('somefile.demo')
And that packages provide the .test and .demo files?

> - for windows systems which may not have the compilers installed, we
> may want to support binary packages.  This would be advised for OS X
> systems as well, where tracking down and installing third party
> libraries prior to installing the package would be a pain.
I choose not to call 'make install', but instead move files from the 'src' directory
to the 'inst' directory, to make creation of binary packages easy. While
not implemented, it should be very easy to create binary packages.
Simply stop the installation before installing any files.
So it's not implemented, but it's on the todo :-)

> - the packaging system should be available and useful for matlab as
> well.  That way authors of packages such as wavelab will have less work
> to do to support octave.  For example, if you automatically generate
> the contents.m from INDEX and admin/make_index the contents file will
> be easier to maintain and it is a win for them.  Similarly, we should
> Do the Right Thing with a usual matlab tarball which has contents.m
> rather than COPYING DESCRIPTION and INDEX files.
In general, I think we should provide scripts for creating different kinds of packages.
It seems like we need .rpm, .dep, and perhaps also matlab package
creator scripts. I don't know the first thing about any of these
formats, but we should have this functionality. Perhaps a 'Package
Creator' package should be available?

Once again, thanks for the input
Søren
> Paul


Reply | Threaded
Open this post in threaded view
|

Re: Package system

Paul Kienzle

On Oct 23, 2005, at 4:59 AM, Søren Hauberg wrote:

> Hi,
> Thanks for the info on octave-forge.
>
> lør, 22 10 2005 kl. 16:59 -0400, skrev Paul Kienzle:
>> - main/audio creates a binary that should be installed in the helper
>> binary directory.  I put them into octave_config_info
>> ("localverarchlibdir").
> One problem with this is that not all users have write access to this
> directory.
> Would it be possible to install this binary somewhere else, or would we
> loose functionality then?

The binary is called via popen from aurecord.m.  It doesn't matter where
it is so long as octave puts that directory on the executable path
prior to calling it.  How about putting it in .octave/bin/arch if the
package is being installed for a single user, and have octave add that
directory to the PATH environment variable on startup.  This is a
security
risk, but then so is installing any package you find on the net :-(

>> - main/audio/data has a sample wav file which I install into the data
>> path for the package.  In this case, I simply use the package path
>> itself and find from the script using file_in_loadpath('file.wav').
> I see you simply put the .wav file in the same directory as the .m
> files, so the easy
> solution is simply put the .wav file in the 'inst' directory. However
> it might be a good
> idea to provide some standard way to provide sample data with a
> package.

There is more than one instance in octave-forge:

$ find octave-forge -name "data" -print
./extra/pdb/data
./extra/testfun/data
./main/audio/data
./main/vrml/data

>> - main/comm/doc has texinfo documentation which should be built and
>> installed in the usual texinfo places.  Pure documentation packages
>> such as doc/coda are also possible.
> Yesterday, I submitted a patch to the bugs list (not a very good
> patch, but still) that makes
> 'help -i function' look for a file called doc.info in the directory
> where 'function' lives, and if this file exist use it. If such
> functionality gets integrated we would have a standard way of providing
> documentation with packages.

You don't want to store docs in the libexec directory, which is where
the oct-file lives.  Instead you will want to store them with the
m-files,
the data and other shared information for the package.

If we can look up the package containing a particular function and
the share directory for the package then we have the tools to
implement this.  Note that we will want this functionality anyway
for an automated 'see also' to find related commands based on the
contents of INDEX.

Can we use INDEX to gather texinfo strings from all functions into
the same info file?  Many packages (particularly those which are for
both matlab and octave) will have ascii help.  Can we automatically
wrap those for info with a raw text command?

Some packages may have html, doc or pdf help rather than texinfo help.

>> - main/control is an example of a single function package intended to
>> extend an existing package.  Does it need its own directory?  Or can
>> we
>> request that it be installed in the same directory as the control
>> package, while updating the list of functions reported by "help
>> control"?
> Hmmm, currently the answer is no. I need packages to go into separate
> directories in order
> to make uninstallation easy. I'd really like not to keep track of each
> file provided by a package, as I think this makes the system more
> simple. It is however possible that it becomes to simple...

Many people only have one or two scripts to contribute.  For now we
can use something like octave-forge to gather them into larger chunks.
This will be more convenient for the user anyway.

>
>> - extra/engine has libraries and headers that are needed by other
>> programs.  They should be installed in /usr/include and /usr/lib or
>> /usr/local/include and /usr/local/lib.
>>
>> - extra/mex and main/fixed have libraries and headers that need to be
>> available for compiling other oct-files.  They should be installed
>> along side the octave libraries and headers.
> The problem here is that not all users have the privileges to do this.
> We could of course
> check this and tell the user if she doesn't have the needed privileges.
> This should be possible in the current setup, by providing
> 'pre_install.m', 'post_install.m', and 'on_uninstall.m'. These could
> also handle the installation of the files. However, this really isn't a
> very pretty solution (it's really a hack). Personaly, I don't think the
> package system should take extra care to make packages like this
> possible as I'm guessing there won't be many packages like this.

These can be installed on a per-user basis.  With minor tweaks
to mkoctfile, other packages wouldn't have to know where these
are installed.  E.g.,

        .octave/{bin/arch,lib/arch,include,doc,scripts,...}

>
>> - extra/graceplot introduces alternatives. The package system should
>> support this in some way so that the user can toggle at run time
>> between graceplot and gnuplot.  extra/nan, which replaces some
>> standard
>> octave commands with alternatives which ignore NaN, could also be
>> handled with alternatives.
> Hmmm, I guess that 'pkg load' (or whatever name we choose) could add
> the
> package to the beginning of the path, thereby overriding other
> functions. The user can then choose which functionality to use by the
> order in which packages gets loaded. Not very user-friendly, but it
> should work.

See how it is done in extra/graceplot and in Debian.  I don't know
the details.

> As to the gnuplot/graceplot issue, would it be possible to provide
> gnuplot as a package as well (removing gnuplot from core octave)? That
> way the user could choose which system to use by loading the right
> package. (This reminds me that we should have a 'pkg unload' (or
> whatever naming we use))

With a package system John is free to move a number of things out
of the octave tree such as statistics and control systems and into
a less strictly controlled environment like octave-forge where domain
experts can have commit rights.

>> - configure.base contains configure information useful to a number of
>> packages.  Much of this is available in modern versions of octave,
>> through system commands such as octave-config and through the octave
>> command octave_config_info.  Other things such as the flags to the cp
>> command, and the details of txi->info conversion should be part of the
>> packaging system.
> Couldn't this file be maintained by octave-forge and then copied to
> the packages that need
> it? As to the txi->info issue, I think we need a standard way of
> providing extra
> documentation (see above).

The package system itself can have a configure script to discover
the frequently needed tools.  In particular, if you want to encourage
texinfo tools.  Octave already discovers these and the package system
will be part of octave, so it really is no trouble.

The thing to do is go through configure.base and Makeconf.base
and decide how they can be made available to the package makefile
conveniently through things such as mkoctfile and octave_config_info.
I believe most of them will already be covered.

>
>> - anticipate translated help files for multiple languages as part of
>> the package.
> I don't really know how this is going to be implemented. What is the
> needs of the translation project?

For each function there will be function.pt, function.es, function.fr,
etc. containing the translated help into language xx.  These files have
a one-line header giving the md5 sum and cvs revision number of the
original text, but other than that they are the usual help extracted
into DOCSTRINGS.

The main problem is that the translation is done by a third party
for each language under a separate source control system.  This is
to avoid having the package developer wait for all the translators to
finish prior to doing a release.

It would be convenient of the package system could query the translation
site to see if that particular package has been translated for a
particular language.  That way packages can be smaller and users don't
have download or install languages they are not interested in.

Alternatively, as part of the package build process, the current
snapshot of the translated text could be added to the package.  If each
translator rebuilds the package when they are finished with their
language and updates the central package repository.

A third alternative is to have pure translation packages which the
user would have to find and install along with the package.

Synchronizing translations with package versions is something I haven't
thought about.

>
>> - I didn't see where you created the PKG_ADD file by extracting
>> PKG_ADD
>> lines from the individual m-files.
> I don't, because I really don't know what that means :-)
> What is PKG_ADD?

Things like dispatch and mark_as_command which change the way the
interpreter interprets function calls are put in a PKG_ADD file
in each installed directory.  When the user sets the LOADPATH
variable, octave scans the directories added to the path and
executes all the PKG_ADD files it finds.

>
>> Some more general comments:
>>
>> - package version should be available at run time.  Currently I have
>> OCTAVE_FORGE_VERSION.m for all the packages.  Maybe a version
>> subcommand on the pkg command is the best way to handle this?
> Guess you're right. It was mentioned that matlab has a 'ver' command,
> which could be
> implemented on top of 'pkg list'. Perhaps this would give the needed
> infomation.

Yes.  I discovered this too.

>
>> - we should be able to build and test the packages before installing
>> them.
> Yes! Octave-forge has scripts to handle this, right? Could these be
> integrated with octave?

I think the testing routines have enough real world usage now that
we can incorporate them into octave, but the failure tests need
some work.

My point above is that we will need 'pkg build' and 'pkg test'
commands.

>
>> - we should note whenever a installed command shadows an existing
>> command.
> Good idea. Shouldn't be to hard to implement.

It gets hard when a script file contains multiple functions.  Then
you need to look for aliases in the PKG_ADD file.  This is new
syntax in the current CVS tree, and I don't remember the details.

>> - in octave forge most tests are embedded in the files themselves and
>> discovered with admin/mktests.sh.  The tests should be installed along
>> with the functions so that the user can see at run time what tests the
>> function is supposed to pass.  Also some tests are actually demo
>> blocks
>> which are part of the documentation of the function.  Some tests are
>> embedded in C++ files.  These will need to be extracted before
>> installing.  In retrospect tests should perhaps be in separate .test
>> files in both cases, and maybe separate .demo files for the demos.
> So what you're suggesting is that octave provides two function
> *) test('somefile.test')
> *) demo('somefile.demo')
> And that packages provide the .test and .demo files?

Octave forge looks for a file named 'fn' or 'fn.m' or 'fn.cc' on the
path.  If it finds one then it extracts the tests from there.  This
should be changed to look for 'fn.test' in addition.

Type 'help test' on octave-forge for details of the test syntax.

In addition to 'test' blocks there are also 'demo' blocks which
are extracted and displayed to the user using the example command
and run with the demo command.  I find I don't use the example
command, however it would be convenient if any examples could
be automatically extracted and included with the documentation for
the function.

>
>> - for windows systems which may not have the compilers installed, we
>> may want to support binary packages.  This would be advised for OS X
>> systems as well, where tracking down and installing third party
>> libraries prior to installing the package would be a pain.
> I choose not to call 'make install', but instead move files from the
> 'src' directory
> to the 'inst' directory, to make creation of binary packages easy.
> While
> not implemented, it should be very easy to create binary packages.
> Simply stop the installation before installing any files.
> So it's not implemented, but it's on the todo :-)

E.g., with a 'pkg build' command.

>> - the packaging system should be available and useful for matlab as
>> well.  That way authors of packages such as wavelab will have less
>> work
>> to do to support octave.  For example, if you automatically generate
>> the contents.m from INDEX and admin/make_index the contents file will
>> be easier to maintain and it is a win for them.  Similarly, we should
>> Do the Right Thing with a usual matlab tarball which has contents.m
>> rather than COPYING DESCRIPTION and INDEX files.
> In general, I think we should provide scripts for creating different
> kinds of packages.
> It seems like we need .rpm, .dep, and perhaps also matlab package
> creator scripts. I don't know the first thing about any of these
> formats, but we should have this functionality. Perhaps a 'Package
> Creator' package should be available?

The flip side is that it would be nice to be able to take a usual
matlab package file (whatever that is) and install it in octave,
similar to something like alien.

>
> Once again, thanks for the input
> Søren
>> Paul
>
>


Reply | Threaded
Open this post in threaded view
|

Re: Package system

David Bateman-3
In reply to this post by Søren Hauberg
Søren Hauberg wrote:

>>- extra/engine has libraries and headers that are needed by other
>>programs.  They should be installed in /usr/include and /usr/lib or
>>/usr/local/include and /usr/local/lib.
>>
>>- extra/mex and main/fixed have libraries and headers that need to be
>>available for compiling other oct-files.  They should be installed
>>along side the octave libraries and headers.
>>    
>>
>The problem here is that not all users have the privileges to do this. We could of course
>check this and tell the user if she doesn't have the needed privileges.
>This should be possible in the current setup, by providing
>'pre_install.m', 'post_install.m', and 'on_uninstall.m'. These could
>also handle the installation of the files. However, this really isn't a
>very pretty solution (it's really a hack). Personaly, I don't think the
>package system should take extra care to make packages like this
>possible as I'm guessing there won't be many packages like this.
>  
>
Hi Soren,

The reason the fixed toolbox is like this is that cygwin/mingw resolves
the symbols in a binary at link time. It is therefore impossible to have
a library under cygwin/mingw that has unresolved symbols. Now imagine I
have two oct-files that both rely on the same basic functionality, under
systems other than cygwin/mingw the basic functionality can exist in one
of these oct-files that will be the first to be called, and the other
can use it. Under cygwin/mingw this is not possible and you're left with
two choices. Either all of the oct-files can be built into a single file
with linking to this single file or a both oct-files can link against a
separate library containing the basic functionality. The first is
easier, except under mingw with the lack of symbolic links this means
that each oct-file is very large. However, the limitation of the first
scheme is that it isn't extensible..

In the case of the fixed point type, I wanted to be able to write new
oct-files for further functionality at will and so just for mingw/cygwin
this split was necessary. Note that the split only exists in this case
and not otherwise. Therefore the installation question is simplified to
can you install the necessary dll's under windows, not a more generic
question...

Regards
David


An oct-file is effectively a library and so if I won't to create an
oct-file that depended on the fixed type it needed to have access to the
a separate library

--
David Bateman                                [hidden email]
Motorola Labs - Paris                        +33 1 69 35 48 04 (Ph)
Parc Les Algorithmes, Commune de St Aubin    +33 1 69 35 77 01 (Fax)
91193 Gif-Sur-Yvette FRANCE

The information contained in this communication has been classified as:

[x] General Business Information
[ ] Motorola Internal Use Only
[ ] Motorola Confidential Proprietary