packaging system

classic Classic list List threaded Threaded
52 messages Options
123
Reply | Threaded
Open this post in threaded view
|

packaging system

Stéfan van der Walt
On Wed, Jun 15, 2005 at 02:08:06PM -0400, John W. Eaton wrote:
> On a related topic, it would be useful to have a packaging system for
> contributed Octave code.  That would allow people to independently
> distribute collections of functions.  Octave forge would still be
> useful as a place for collaborative deveoplment, but we would not be
> restricted to a monolithic Octave plus a monolithic Octave forge
> collection.  Given a functioning package system, there are a number of
> things that could be removed from Octave and placed in independent
> packages.

I suggest that we take a look at the Debian package format, and create
a similarly structured archive.  This archive will, for example,
contain the following:

pre-install.m
post-install.m
pre-remove.m
post-install.m
content.tar
INDEX

Then, some choice functions could be provided to aid the pre- and
post-install/remove scripts.  The INDEX format used in Octave Forge
works quite well -- and we already have a parser for it, written in
C++.

Regards
Stéfan

Reply | Threaded
Open this post in threaded view
|

Re: packaging system

Quentin Spencer
Stefan van der Walt wrote:

>On Wed, Jun 15, 2005 at 02:08:06PM -0400, John W. Eaton wrote:
>  
>
>>On a related topic, it would be useful to have a packaging system for
>>contributed Octave code.  That would allow people to independently
>>distribute collections of functions.  Octave forge would still be
>>useful as a place for collaborative deveoplment, but we would not be
>>restricted to a monolithic Octave plus a monolithic Octave forge
>>collection.  Given a functioning package system, there are a number of
>>things that could be removed from Octave and placed in independent
>>packages.
>>    
>>
>
>I suggest that we take a look at the Debian package format, and create
>a similarly structured archive.  This archive will, for example,
>contain the following:
>
>pre-install.m
>post-install.m
>pre-remove.m
>post-install.m
>content.tar
>INDEX
>
>Then, some choice functions could be provided to aid the pre- and
>post-install/remove scripts.  The INDEX format used in Octave Forge
>works quite well -- and we already have a parser for it, written in
>C++.
>  
>
Let me just point out one thing before we get too far into this
discussion. Past discussion of a possible packaging system has alluded
to TeX, R, and perl as examples. I'm not familiar with R, but both of
the other two do not require compilation for their packages. A packaging
approach will work for m files, but octave-forge has a fair amount of
.oct files, some because they are wrappers for other things, and some
because "it's faster that way". I really don't see this packaging
approach being particularly practical for compiled code--the sheer
number of possible OSs and architectures (especially with the increasing
popularity of x86_64-specific Linux distributions) becomes quickly
unmanageable.

-Quentin

Reply | Threaded
Open this post in threaded view
|

Re: packaging system

John W. Eaton-6
On 15-Jun-2005, Quentin Spencer wrote:

| I really don't see this packaging
| approach being particularly practical for compiled code--the sheer
| number of possible OSs and architectures (especially with the increasing
| popularity of x86_64-specific Linux distributions) becomes quickly
| unmanageable.

I have no plans for binary packages.  If a package contains the source
for .oct files, then you will need to have a system capable of running
mkoctfile.  After downloading, the package is compiled, then installed.
I think this is similar to what happens if you download a package from
CPAN that includes C (or other compiled-language) code.

jwe

Reply | Threaded
Open this post in threaded view
|

RE: packaging system

Søren Hauberg
In reply to this post by Stéfan van der Walt
Hi,
I'm really no expert on the subject of packaging systems, but it seems
to me that the largest problem with creating such systems, is to what
extend dependencies should be handled.

*) If package A depends on package B, should the package system check of
package B is installed, and handle the situation of it is not?
*) If the installed package A depends on the installed package B, and
the user chooses to uninstall package B, should the package system check
and handle this situation?
*) If package A depends on the external library L in order to run,
should the package system check that L is installed on the system?
*) ... (did I forget something?)

/Søren

Reply | Threaded
Open this post in threaded view
|

Re: packaging system

Przemek Klosowski
In reply to this post by Quentin Spencer
   Let me just point out one thing before we get too far into this
   discussion. Past discussion of a possible packaging system has alluded
   to TeX, R, and perl as examples. I'm not familiar with R, but both of
   the other two do not require compilation for their packages. A packaging

Actually, Perl's CPAN system does allow for compilation. From experience,
a large fraction of Perl packages are compiled extensions. The typical
process is like this:

        perl -MCPAN -e 'install SuperModule'

        <CPAN code gets the source, unpacks, compiles, runs tests, installs>

It is impressive (and sometimes frustrating) how strict they are with the
QA process. Any compilation or regression test error prevents installation
of the package. CPAN archive system is rather comprehensive: upon installation
(which is automatic when you first try to use CPAN) it goes through a list
of archive sites, and selects several closest ones. The archive is indexed,
allows for pre-installation searching, etc.etc.

       p

Reply | Threaded
Open this post in threaded view
|

Re: packaging system

Søren Hauberg
Just to get things started I've created a quick-and-dirty
implementation, of a *simple* packaging system. The system is basicly
stolen from the R project, but the implementation is in the Octave
language. A bit more details and the code is at
http://hauberg.org/octave-package/

/Søren

Reply | Threaded
Open this post in threaded view
|

Re: packaging system

John W. Eaton-6
On 17-Jun-2005, Søren Hauberg wrote:

| Just to get things started I've created a quick-and-dirty
| implementation, of a *simple* packaging system. The system is basicly
| stolen from the R project, but the implementation is in the Octave
| language. A bit more details and the code is at
| http://hauberg.org/octave-package/

I looked at your sample package and installation script and here are
some comments.

  * I think the tar file should unpack into a single directory with
    the name PACKAGE-VERSION.  I'm always annoyed when I unpack a tar
    file and it dumps a bunch of files directly into my current
    directory.  Usually I check to see what will happen first with
    "tar tzf" but I sometimes forget)

  * The file COPYING should be the full text of the license, not just
    the text that goes at the head of each source file.

  * Should we also provide scripts for creating packages and verifying
    that they are complete?

Thanks,

jwe

Reply | Threaded
Open this post in threaded view
|

RE: packaging system

John W. Eaton-6
In reply to this post by Søren Hauberg
On 15-Jun-2005, Søren Hauberg wrote:

| I'm really no expert on the subject of packaging systems, but it seems
| to me that the largest problem with creating such systems, is to what
| extend dependencies should be handled.

It would be good to have some dependency checking, but I'm not sure
that it really matters too much until we also have a way to
download and install packages from Octave (or using some other
script).

jwe


Reply | Threaded
Open this post in threaded view
|

Re: packaging system

Søren Hauberg
In reply to this post by John W. Eaton-6
John W. Eaton wrote:

> On 17-Jun-2005, Søren Hauberg wrote:
>
> | Just to get things started I've created a quick-and-dirty
> | implementation, of a *simple* packaging system. The system is basicly
> | stolen from the R project, but the implementation is in the Octave
> | language. A bit more details and the code is at
> | http://hauberg.org/octave-package/
>
> I looked at your sample package and installation script and here are
> some comments.
>
>   * I think the tar file should unpack into a single directory with
>     the name PACKAGE-VERSION.  I'm always annoyed when I unpack a tar
>     file and it dumps a bunch of files directly into my current
>     directory.  Usually I check to see what will happen first with
>     "tar tzf" but I sometimes forget)
Since the user shouldn't unpack the tgz (the install script does that),
this shouldn't be a big deal. The reason why I haven't done it is that
until the DESCRIPTION file is read PACKAGE and VERSION are unknown. It
should however be trivial to change. I'll look into it.

>   * The file COPYING should be the full text of the license, not just
>     the text that goes at the head of each source file.
Right! Since this is just a sample, I took the first "license-looking"
thing I could find.

>   * Should we also provide scripts for creating packages and verifying
>     that they are complete?
I'm thinking yes! It shouldn't be that hard. I guess there should be a
script to create a binary package from a source package as this would
make whings easier for some people (e.g. linux distributers).

/Søren
> Thanks,
>
> jwe
>

Reply | Threaded
Open this post in threaded view
|

Re: packaging system

Søren Hauberg
In reply to this post by John W. Eaton-6
John W. Eaton wrote:

> On 15-Jun-2005, Søren Hauberg wrote:
>
> | I'm really no expert on the subject of packaging systems, but it seems
> | to me that the largest problem with creating such systems, is to what
> | extend dependencies should be handled.
>
> It would be good to have some dependency checking, but I'm not sure
> that it really matters too much until we also have a way to
> download and install packages from Octave (or using some other
> script).
The implementation I linked to in another post earlier today does
depency checking at installation time. It will check if the needed
packages (but not external libraries) are installed. However, I don't
think dependency checking at uninstallation time will be implemented as
it seems a bit harder (more book-keeping is necesary).

/Søren
>
> jwe
>
>

Reply | Threaded
Open this post in threaded view
|

Re: packaging system

John W. Eaton-6
In reply to this post by Søren Hauberg
On 17-Jun-2005, Søren Hauberg wrote:

| Just to get things started I've created a quick-and-dirty
| implementation, of a *simple* packaging system.

I also noticed that your install script has

    ## Get the prefix of the installation directory
    # XXX: The user should be able to change this, through a variable
    # XXX: Should we check that this directory exist?
    prefix = "~/Dokumenter/Octave/package";
    prefix = tilde_expand(prefix);

I think we should be using octave_config_info to get the values of the
variables

  localapifcnfiledir
  localapioctfiledir
  localarchlibdir
  localfcnfiledir
  localoctfiledir
  localstartupfiledir
  localverarchlibdir
  localverfcnfiledir
  localveroctfiledir

to install package components?  These would work for privileged users
installing files to be shared by all users on a system.  Maybe we
should define another set of directories for installing files for a
particular user?  In that case, maybe installing the entire contents
of a package in a subdirectory of ~/octave is sufficient?  The name of
the top-level directory should be configurable, of course.  Perhaps
refuse to install if the user hasn't set it in some way?  I think this
is what the perl CPAN module does.

jwe

Reply | Threaded
Open this post in threaded view
|

Re: packaging system

Søren Hauberg
John W. Eaton wrote:
[snip]

> I think we should be using octave_config_info to get the values of the
> variables
>
>   localapifcnfiledir
>   localapioctfiledir
>   localarchlibdir
>   localfcnfiledir
>   localoctfiledir
>   localstartupfiledir
>   localverarchlibdir
>   localverfcnfiledir
>   localveroctfiledir
>
> to install package components?  These would work for privileged users
> installing files to be shared by all users on a system.
Makes sense. I would prefer to keep the file installed by package X in a
subdirectory called X or X-VERSION, since that would make uninstallation
trivial (we wouldn't have to keep track of which files belongs to which
packages). Of course this would mean the user would have change her path.

> Maybe we
> should define another set of directories for installing files for a
> particular user?  In that case, maybe installing the entire contents
> of a package in a subdirectory of ~/octave is sufficient?  The name of
> the top-level directory should be configurable, of course.  Perhaps
> refuse to install if the user hasn't set it in some way?  I think this
> is what the perl CPAN module does.
To me, this is very important, since I use Octave at school where I
don't have the priviliges to install software outside my home directory.
I don't think a default directory should be used if the user doesn't
supply one, but I don't really have any good arguments. We can just
introduce a variable called octave_package_prefix (or simular).
>
> jwe
/Søren

Reply | Threaded
Open this post in threaded view
|

Re: packaging system

Søren Hauberg
In reply to this post by John W. Eaton-6
Hi
I've updated the code (it's still at http://hauberg.org/octave-package),
with the following changes:
*) If the archive contains a directory, cd to that directory. This means
that package can keep it's files in a directory (e.g. PACKAGE-VERSION).
*) A bit more error checking. Not enough though
*) Support for uninstallation of packages. No dependency checking is
done during uninstallation.
*) The installation prefix is defined by the global variable
OCTAVE_PACKAGE_PREFIX. If this variable is not defined an error is
issued. Packages will be install in OCTAVE_VALUE_PREFIX/PACKAGE-VERSION

As long as the packages doesn't have to many errors the code is usable
in it's current state.

/Søren

Reply | Threaded
Open this post in threaded view
|

Re: packaging system

Stéfan van der Walt
I still think that something like a pre- and post-install script would
be useful.  Why stop at being able to modify the installation prefix?
Why not create a standard structure with all installation variables,
populated before installation starts.

The installing package then has an opportunity to modify this struct
in the pre-install script after which installation takes place.
Afterwards, the post-install script can do any additional
admin/cleanup needed while having access to that information still.

Your package has everything in place to do this already.  It would
just require some minor modifications.

Regards
Stefan

On Fri, Jun 17, 2005 at 11:01:35PM +0200, S?ren Hauberg wrote:

> Hi
> I've updated the code (it's still at http://hauberg.org/octave-package),
> with the following changes:
> *) If the archive contains a directory, cd to that directory. This means
> that package can keep it's files in a directory (e.g. PACKAGE-VERSION).
> *) A bit more error checking. Not enough though
> *) Support for uninstallation of packages. No dependency checking is
> done during uninstallation.
> *) The installation prefix is defined by the global variable
> OCTAVE_PACKAGE_PREFIX. If this variable is not defined an error is
> issued. Packages will be install in OCTAVE_VALUE_PREFIX/PACKAGE-VERSION
>
> As long as the packages doesn't have to many errors the code is usable
> in it's current state.
>
> /Søren
>

Reply | Threaded
Open this post in threaded view
|

Re: packaging system

Søren Hauberg
I'm going to need a little hand-holding here, as don't quite understand
your post.

You want the installation routine to (before installation) check if a
file called pre-install.m exist, and if it does run it from Octave,
right? If so then that's just a minor change that can be done in a
minute. You also want a similar post-install.m, to be executed at the
end of installation? (btw, should these be .m files or shell scripts?)

What I don't get is what these scripts should do. Could you elaborate a bit?

/Søren

Stefan van der Walt wrote:

> I still think that something like a pre- and post-install script would
> be useful.  Why stop at being able to modify the installation prefix?
> Why not create a standard structure with all installation variables,
> populated before installation starts.
>
> The installing package then has an opportunity to modify this struct
> in the pre-install script after which installation takes place.
> Afterwards, the post-install script can do any additional
> admin/cleanup needed while having access to that information still.
>
> Your package has everything in place to do this already.  It would
> just require some minor modifications.
>
> Regards
> Stefan
>
> On Fri, Jun 17, 2005 at 11:01:35PM +0200, S?ren Hauberg wrote:
>
>>Hi
>>I've updated the code (it's still at http://hauberg.org/octave-package),
>>with the following changes:
>>*) If the archive contains a directory, cd to that directory. This means
>>that package can keep it's files in a directory (e.g. PACKAGE-VERSION).
>>*) A bit more error checking. Not enough though
>>*) Support for uninstallation of packages. No dependency checking is
>>done during uninstallation.
>>*) The installation prefix is defined by the global variable
>>OCTAVE_PACKAGE_PREFIX. If this variable is not defined an error is
>>issued. Packages will be install in OCTAVE_VALUE_PREFIX/PACKAGE-VERSION
>>
>>As long as the packages doesn't have to many errors the code is usable
>>in it's current state.
>>
>>/Søren
>>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: packaging system

Stéfan van der Walt
Hi Soren

On Sat, Jun 18, 2005 at 11:51:09AM +0200, S?ren Hauberg wrote:
> I'm going to need a little hand-holding here, as don't quite understand
> your post.

Sorry, I'll try to be a bit clearer.

> You want the installation routine to (before installation) check if a
> file called pre-install.m exist, and if it does run it from Octave,
> right? If so then that's just a minor change that can be done in a
> minute. You also want a similar post-install.m, to be executed at the
> end of installation? (btw, should these be .m files or shell
> scripts?)

Yes, that's the idea.  Each function (pre-install.m and
post-install.m) should take one argument: the struct describing all
aspects of the installation -- e.g. installation directory etc. etc.

> What I don't get is what these scripts should do. Could you
  elaborate a bit?

That is for the package admin to decide.  It gives him the flexibility
to change the installation in any way he requires, by executing
commands before and after the installation, and also by modifying the
installation struct.

For example, say a package requires that a certain file is present on
the system before installation, that can be done in the pre-install.m
(in addition to the standard package dependencies)

Or, after an installation, some special linking of devices or files
might be requires, which post-install.m can handle.

It's just intended as a way to generalise the process a bit.  I think
it's easier to implement such functionality in Octave than it would be
in a Makefile.  We can add a few choice functions to aid in that
process.

I might have missed it, but how do we intend to keep an index of all
functions installed?  Something that can aid in listing the contents
of toolboxes etc.  Paul Kienzle proposed the INDEX format, currently
used my Octave Forge -- it might be worth taking a look at it, or
using something similar.

Regards
Stefan

>
> Stefan van der Walt wrote:
> >I still think that something like a pre- and post-install script would
> >be useful.  Why stop at being able to modify the installation prefix?
> >Why not create a standard structure with all installation variables,
> >populated before installation starts.
> >
> >The installing package then has an opportunity to modify this struct
> >in the pre-install script after which installation takes place.
> >Afterwards, the post-install script can do any additional
> >admin/cleanup needed while having access to that information still.
> >
> >Your package has everything in place to do this already.  It would
> >just require some minor modifications.
> >
> >Regards
> >Stefan
> >
> >On Fri, Jun 17, 2005 at 11:01:35PM +0200, S?ren Hauberg wrote:
> >
> >>Hi
> >>I've updated the code (it's still at http://hauberg.org/octave-package),
> >>with the following changes:
> >>*) If the archive contains a directory, cd to that directory. This means
> >>that package can keep it's files in a directory (e.g. PACKAGE-VERSION).
> >>*) A bit more error checking. Not enough though
> >>*) Support for uninstallation of packages. No dependency checking is
> >>done during uninstallation.
> >>*) The installation prefix is defined by the global variable
> >>OCTAVE_PACKAGE_PREFIX. If this variable is not defined an error is
> >>issued. Packages will be install in OCTAVE_VALUE_PREFIX/PACKAGE-VERSION
> >>
> >>As long as the packages doesn't have to many errors the code is usable
> >>in it's current state.
> >>
> >>/Søren
> >>
> >
> >
>

Reply | Threaded
Open this post in threaded view
|

Re: packaging system

Søren Hauberg
Stefan van der Walt wrote:
[snip]
> I might have missed it, but how do we intend to keep an index of all
> functions installed?  Something that can aid in listing the contents
> of toolboxes etc.  Paul Kienzle proposed the INDEX format, currently
> used my Octave Forge -- it might be worth taking a look at it, or
> using something similar.
The current implementation (which might be far from the right solution)
is desinged with simplicity, not power, in mind. A package is installed
into it's own directory called OCTAVE_PACKAGE_PREFIX/PACKAGE-VERSION, so
  there's no need to keep track on files, only directories. The
installation path is written to ~/.octave-packages. To uninstall you
simply delete the directory, and remove the corresponding line from
~/.octave-packages. This file is also used to handle dependencies at
install-time.

Perhaps this approach is to simple? How much power should a package
creator have?

/Søren

P.S. Where can I get some information on the INDEX format

>
> Regards
> Stefan
>
>
>>Stefan van der Walt wrote:
>>
>>>I still think that something like a pre- and post-install script would
>>>be useful.  Why stop at being able to modify the installation prefix?
>>>Why not create a standard structure with all installation variables,
>>>populated before installation starts.
>>>
>>>The installing package then has an opportunity to modify this struct
>>>in the pre-install script after which installation takes place.
>>>Afterwards, the post-install script can do any additional
>>>admin/cleanup needed while having access to that information still.
>>>
>>>Your package has everything in place to do this already.  It would
>>>just require some minor modifications.
>>>
>>>Regards
>>>Stefan
>>>
>>>On Fri, Jun 17, 2005 at 11:01:35PM +0200, S?ren Hauberg wrote:
>>>
>>>
>>>>Hi
>>>>I've updated the code (it's still at http://hauberg.org/octave-package),
>>>>with the following changes:
>>>>*) If the archive contains a directory, cd to that directory. This means
>>>>that package can keep it's files in a directory (e.g. PACKAGE-VERSION).
>>>>*) A bit more error checking. Not enough though
>>>>*) Support for uninstallation of packages. No dependency checking is
>>>>done during uninstallation.
>>>>*) The installation prefix is defined by the global variable
>>>>OCTAVE_PACKAGE_PREFIX. If this variable is not defined an error is
>>>>issued. Packages will be install in OCTAVE_VALUE_PREFIX/PACKAGE-VERSION
>>>>
>>>>As long as the packages doesn't have to many errors the code is usable
>>>>in it's current state.
>>>>
>>>>/Søren
>>>>
>>>
>>>
>

Reply | Threaded
Open this post in threaded view
|

Re: packaging system

Stéfan van der Walt
On Sat, Jun 18, 2005 at 12:46:14PM +0200, S?ren Hauberg wrote:
> Perhaps this approach is to simple? How much power should a package
> creator have?

It takes a lot of effort to geta system like this into place, so I
thought we might as well make it flexible.  That doesn't necessarily
mean a lot of additional effort.

> P.S. Where can I get some information on the INDEX format

It's a very simple format -- look at the files included with Octave
Forge.  We have C++, Octave and Python parsers for it already :)

Regards
Stefan

Reply | Threaded
Open this post in threaded view
|

Re: packaging system

Søren Hauberg
Stefan van der Walt wrote:
> On Sat, Jun 18, 2005 at 12:46:14PM +0200, S?ren Hauberg wrote:
>
>>Perhaps this approach is to simple? How much power should a package
>>creator have?
>
>
> It takes a lot of effort to geta system like this into place, so I
> thought we might as well make it flexible.  That doesn't necessarily
> mean a lot of additional effort.
Depends on how flexible it is :-)
You said in another mail the pre-install.m should be able to alter the
installation struct (a structure containing various details on the
installation, e.g. installation directory etc). This would give the
packager alot of power, but why should the packager be able to alter the
installation directory?
My concern is that if we give the packager to much power, the
installation script will have to perform alot of checking on what the
package does to make sure it doesn't screw things up. Of course, if the
packager doesn't have enough power the package system ends up unused.

>> P.S. Where can I get some information on the INDEX format
> It's a very simple format -- look at the files included with Octave
> Forge.  We have C++, Octave and Python parsers for it already :)
OK, so this is a file format to list which packages contains which
files, right? I don't see the advantage of using this fileformat instead
of putting the files from different packages in seperate directories. Am
I missing the obvious?

/Søren

> Regards
> Stefan
>

Reply | Threaded
Open this post in threaded view
|

Re: packaging system

Stéfan van der Walt
On Sat, Jun 18, 2005 at 01:25:31PM +0200, S?ren Hauberg wrote:
> >It takes a lot of effort to geta system like this into place, so I
> >thought we might as well make it flexible.  That doesn't necessarily
> >mean a lot of additional effort.
> Depends on how flexible it is :-)

True!

> You said in another mail the pre-install.m should be able to alter the
> installation struct (a structure containing various details on the
> installation, e.g. installation directory etc). This would give the
> packager alot of power, but why should the packager be able to alter the
> installation directory?

The idea was that you could send the structure through a couple of
processes, and at the end of it simply feed it to the installer.  I
remember, for example, that you said you might want to install to a
non-standard directory.  Now, all you need to do is create an
installer that sets that field in the struct.  But yes, if the package
maintainer can manipulate those fields it might give him too much
power.  Then again, so does allowing him to do system("...").

> My concern is that if we give the packager to much power, the
> installation script will have to perform alot of checking on what the
> package does to make sure it doesn't screw things up. Of course, if the
> packager doesn't have enough power the package system ends up unused.

Octave's security is (by default) weak.  Users are allowed to do
almost anything -- so any script you run, you are basically at the
coder's mercy.  Maybe we can anly allow pre- and post- install access
to a select subset of functions?  How would one do this?  We don't
have namespaces.  Maybe a special mode for Octave?  Like octave
--install scriptname.m.

> >>P.S. Where can I get some information on the INDEX format
> >It's a very simple format -- look at the files included with Octave
> >Forge.  We have C++, Octave and Python parsers for it already :)
> OK, so this is a file format to list which packages contains which
> files, right? I don't see the advantage of using this fileformat instead
> of putting the files from different packages in seperate directories. Am
> I missing the obvious?

INDEX files are specifically aimed at describing toolboxes and their
content.  You can have many packages contributing functions to a
single toolbox.

Regards
Stefan

123