URL for forge packages

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

URL for forge packages

Carnë Draug-2
Hi

the current method to get the URL for packages when installing with
the -forge flag is to parse the HTML of the package for the version
and use it to generate a URL to download it. I see two problems with
this:

1. if we are to change the format of the documentation page this will
start failing
2. if we move the packages to another host (as has been discussed
before), it will also fail

While such changes are not planned to happen on OF soon, when they do
it will mean that users using octave versions before the change won't
be able to use the -forge flag. I think to prevent this before becomes
an issie it's a good idea.  I would like to propose to have a simple
text file on octave.org with this info. 1 line with package names and
their URLs would suffice. This would also make the code to get
packages from forge simpler since the parsing would also be simpler.

Any comments?

Carnë
Reply | Threaded
Open this post in threaded view
|

Re: URL for forge packages

bpabbott
Administrator

On Aug 16, 2012, at 8:14 AM, Carnë Draug wrote:

> Hi
>
> the current method to get the URL for packages when installing with
> the -forge flag is to parse the HTML of the package for the version
> and use it to generate a URL to download it. I see two problems with
> this:
>
> 1. if we are to change the format of the documentation page this will
> start failing
> 2. if we move the packages to another host (as has been discussed
> before), it will also fail
>
> While such changes are not planned to happen on OF soon, when they do
> it will mean that users using octave versions before the change won't
> be able to use the -forge flag. I think to prevent this before becomes
> an issie it's a good idea.  I would like to propose to have a simple
> text file on octave.org with this info. 1 line with package names and
> their URLs would suffice. This would also make the code to get
> packages from forge simpler since the parsing would also be simpler.
>
> Any comments?
>
> Carnë

Sounds like a good idea to me.  Should there also be a way to locally over-ride the web-site to allow packages not on OF to be installed automatically?  Perhaps a local list of repositories/mirrors that can be successively searched for the desired package?

Ben

Reply | Threaded
Open this post in threaded view
|

Re: URL for forge packages

Carnë Draug-2
On 16 August 2012 13:18, Ben Abbott <[hidden email]> wrote:

> On Aug 16, 2012, at 8:14 AM, Carnë Draug wrote:
>> Hi
>>
>> the current method to get the URL for packages when installing with
>> the -forge flag is to parse the HTML of the package for the version
>> and use it to generate a URL to download it. I see two problems with
>> this:
>>
>> 1. if we are to change the format of the documentation page this will
>> start failing
>> 2. if we move the packages to another host (as has been discussed
>> before), it will also fail
>>
>> While such changes are not planned to happen on OF soon, when they do
>> it will mean that users using octave versions before the change won't
>> be able to use the -forge flag. I think to prevent this before becomes
>> an issie it's a good idea.  I would like to propose to have a simple
>> text file on octave.org with this info. 1 line with package names and
>> their URLs would suffice. This would also make the code to get
>> packages from forge simpler since the parsing would also be simpler.
>>
>> Any comments?
>>
>> Carnë
>
> Sounds like a good idea to me.  Should there also be a way to locally over-ride the web-site to allow packages not on OF to be installed automatically?  Perhaps a local list of repositories/mirrors that can be successively searched for the desired package?

The plan is to at least add an URL flag. Other suggestions are welcome.

Carnë
Reply | Threaded
Open this post in threaded view
|

Re: URL for forge packages

Carnë Draug-2
In reply to this post by Carnë Draug-2
On 16 August 2012 13:14, Carnë Draug <[hidden email]> wrote:

> Hi
>
> the current method to get the URL for packages when installing with
> the -forge flag is to parse the HTML of the package for the version
> and use it to generate a URL to download it. I see two problems with
> this:
>
> 1. if we are to change the format of the documentation page this will
> start failing
> 2. if we move the packages to another host (as has been discussed
> before), it will also fail
>
> While such changes are not planned to happen on OF soon, when they do
> it will mean that users using octave versions before the change won't
> be able to use the -forge flag. I think to prevent this before becomes
> an issie it's a good idea.  I would like to propose to have a simple
> text file on octave.org with this info. 1 line with package names and
> their URLs would suffice. This would also make the code to get
> packages from forge simpler since the parsing would also be simpler.
>
> Any comments?

I gave this some more thought and here's the solution I propose. The
idea will be to have a text file in octave.org that is automatically
updated with a file I'll set up in SF. The following syntax for such
file would allow for other projects similar to Octave Forge (if and
when they appear) to also have their -project flag in pkg.

=Forge http://path-text-file-with-list-of-forge-packages
package1-name package1-version package1-url
package2-name package2-version package2-url
package3-name package3-version package3-url
=ProjectX http://path-text-file-with-list-of-projectx-packages
package1-name package1-version package1-url

Basically, the word after the equal would be the project and flag used
to get packages, followed by a URL for a text file that is
responsibility of that project. That text file would have just the
project, version, and url lines. Should be pretty simple for a perl
script to routinely check those URL's for changes and edit the file on
octave.org.

I'd guess that XML would be better but having to parse them in Octave
is more complicated and I don't think we need that.

Any comments? Can I start writing such scripts?

Carnë
Reply | Threaded
Open this post in threaded view
|

Re: URL for forge packages

Søren Hauberg

On Aug 17, 2012, at 4:42 AM, Carnë Draug wrote:

> On 16 August 2012 13:14, Carnë Draug <[hidden email]> wrote:
>> Hi
>>
>> the current method to get the URL for packages when installing with
>> the -forge flag is to parse the HTML of the package for the version
>> and use it to generate a URL to download it. I see two problems with
>> this:
>>
>> 1. if we are to change the format of the documentation page this will
>> start failing
>> 2. if we move the packages to another host (as has been discussed
>> before), it will also fail
>>
>> While such changes are not planned to happen on OF soon, when they do
>> it will mean that users using octave versions before the change won't
>> be able to use the -forge flag. I think to prevent this before becomes
>> an issie it's a good idea.  I would like to propose to have a simple
>> text file on octave.org with this info. 1 line with package names and
>> their URLs would suffice. This would also make the code to get
>> packages from forge simpler since the parsing would also be simpler.
>>
>> Any comments?
>
> I gave this some more thought and here's the solution I propose. The
> idea will be to have a text file in octave.org that is automatically
> updated with a file I'll set up in SF. The following syntax for such
> file would allow for other projects similar to Octave Forge (if and
> when they appear) to also have their -project flag in pkg.
>
> =Forge http://path-text-file-with-list-of-forge-packages
> package1-name package1-version package1-url
> package2-name package2-version package2-url
> package3-name package3-version package3-url
> =ProjectX http://path-text-file-with-list-of-projectx-packages
> package1-name package1-version package1-url

Why not just use Octave's standard file format for loading and saving data, such that you do not have to parse anything (just 'load' the file) ?

Søren
Reply | Threaded
Open this post in threaded view
|

Re: URL for forge packages

rocketdude
In reply to this post by Carnë Draug-2
>
> On 16 August 2012 13:14, Carn? Draug <[hidden email]> wrote:
>> Hi
>>
>> the current method to get the URL for packages when installing with
>> the -forge flag is to parse the HTML of the package for the version
>> and use it to generate a URL to download it. I see two problems with
>> this:
>>
>> 1. if we are to change the format of the documentation page this will
>> start failing
>> 2. if we move the packages to another host (as has been discussed
>> before), it will also fail
>>
>> While such changes are not planned to happen on OF soon, when they do
>> it will mean that users using octave versions before the change won't
>> be able to use the -forge flag. I think to prevent this before becomes
>> an issie it's a good idea.  I would like to propose to have a simple
>> text file on octave.org with this info. 1 line with package names and
>> their URLs would suffice. This would also make the code to get
>> packages from forge simpler since the parsing would also be simpler.
>>
>> Any comments?
>
> I gave this some more thought and here's the solution I propose. The
> idea will be to have a text file in octave.org that is automatically
> updated with a file I'll set up in SF. The following syntax for such
> file would allow for other projects similar to Octave Forge (if and
> when they appear) to also have their -project flag in pkg.
>
> =Forge http://path-text-file-with-list-of-forge-packages
> package1-name package1-version package1-url
> package2-name package2-version package2-url
> package3-name package3-version package3-url
> =ProjectX http://path-text-file-with-list-of-projectx-packages
> package1-name package1-version package1-url
>
> Basically, the word after the equal would be the project and flag used
> to get packages, followed by a URL for a text file that is
> responsibility of that project. That text file would have just the
> project, version, and url lines. Should be pretty simple for a perl
> script to routinely check those URL's for changes and edit the file on
> octave.org.
>
> I'd guess that XML would be better but having to parse them in Octave
> is more complicated and I don't think we need that.
>
> Any comments? Can I start writing such scripts?
>

I think XML is overkill - a self explanatory text file is easier to implement and easier to read.

I would assume that one could then just change the url's to local file system url - like file://data/stuff/io-1.0.9.tgz?
It is important to be able to use a directory full of packages on a local file system. My enterprise setup does not have the cluster computers connected to the internet, only to the company LAN (for a lot of reasons). It's much easier to maintain all the Forge packages if I can download them all and do an install of them all with one command.

Tom
Reply | Threaded
Open this post in threaded view
|

Re: URL for forge packages

Juan Pablo Carbajal
On Fri, Aug 17, 2012 at 8:30 AM, Thomas Yengst <[hidden email]> wrote:

>>
>> On 16 August 2012 13:14, Carn? Draug <[hidden email]> wrote:
>>> Hi
>>>
>>> the current method to get the URL for packages when installing with
>>> the -forge flag is to parse the HTML of the package for the version
>>> and use it to generate a URL to download it. I see two problems with
>>> this:
>>>
>>> 1. if we are to change the format of the documentation page this will
>>> start failing
>>> 2. if we move the packages to another host (as has been discussed
>>> before), it will also fail
>>>
>>> While such changes are not planned to happen on OF soon, when they do
>>> it will mean that users using octave versions before the change won't
>>> be able to use the -forge flag. I think to prevent this before becomes
>>> an issie it's a good idea.  I would like to propose to have a simple
>>> text file on octave.org with this info. 1 line with package names and
>>> their URLs would suffice. This would also make the code to get
>>> packages from forge simpler since the parsing would also be simpler.
>>>
>>> Any comments?
>>
>> I gave this some more thought and here's the solution I propose. The
>> idea will be to have a text file in octave.org that is automatically
>> updated with a file I'll set up in SF. The following syntax for such
>> file would allow for other projects similar to Octave Forge (if and
>> when they appear) to also have their -project flag in pkg.
>>
>> =Forge http://path-text-file-with-list-of-forge-packages
>> package1-name package1-version package1-url
>> package2-name package2-version package2-url
>> package3-name package3-version package3-url
>> =ProjectX http://path-text-file-with-list-of-projectx-packages
>> package1-name package1-version package1-url
>>
>> Basically, the word after the equal would be the project and flag used
>> to get packages, followed by a URL for a text file that is
>> responsibility of that project. That text file would have just the
>> project, version, and url lines. Should be pretty simple for a perl
>> script to routinely check those URL's for changes and edit the file on
>> octave.org.
>>
>> I'd guess that XML would be better but having to parse them in Octave
>> is more complicated and I don't think we need that.
>>
>> Any comments? Can I start writing such scripts?
>>
>
> I think XML is overkill - a self explanatory text file is easier to implement and easier to read.
>
> I would assume that one could then just change the url's to local file system url - like file://data/stuff/io-1.0.9.tgz?
> It is important to be able to use a directory full of packages on a local file system. My enterprise setup does not have the cluster computers connected to the internet, only to the company LAN (for a lot of reasons). It's much easier to maintain all the Forge packages if I can download them all and do an install of them all with one command.
>
> Tom

Carne,

I agree with Soren, we should use internal format if possible.

Indeed, I would suggest to store a struct as it is done in
.octave_packages in the user folder. This will also ease the merging
of and update of that file.

m2c.


--
M. Sc. Juan Pablo Carbajal
-----
PhD Student
University of Zürich
http://ailab.ifi.uzh.ch/carbajal/
Reply | Threaded
Open this post in threaded view
|

Re: URL for forge packages

c.-2
In reply to this post by Carnë Draug-2

On 17 Aug 2012, at 07:24, [hidden email] wrote:

> Message: 5
> Date: Fri, 17 Aug 2012 03:42:18 +0100
> From: Carn? Draug <[hidden email]>
> To: octave maintainers mailing list <[hidden email]>
> Subject: Re: URL for forge packages
> Message-ID:
> <CAPOrs_1qVvFK09WJ3o2ML3ThmhLg+VvaDN=[hidden email]>
> Content-Type: text/plain; charset=ISO-8859-1
>
> On 16 August 2012 13:14, Carn? Draug <[hidden email]> wrote:
>> Hi
>>
>> the current method to get the URL for packages when installing with
>> the -forge flag is to parse the HTML of the package for the version
>> and use it to generate a URL to download it. I see two problems with
>> this:
>>
>> 1. if we are to change the format of the documentation page this will
>> start failing
>> 2. if we move the packages to another host (as has been discussed
>> before), it will also fail
>>
>> While such changes are not planned to happen on OF soon, when they do
>> it will mean that users using octave versions before the change won't
>> be able to use the -forge flag. I think to prevent this before becomes
>> an issie it's a good idea.  I would like to propose to have a simple
>> text file on octave.org with this info. 1 line with package names and
>> their URLs would suffice. This would also make the code to get
>> packages from forge simpler since the parsing would also be simpler.
>>
>> Any comments?
>
> I gave this some more thought and here's the solution I propose. The
> idea will be to have a text file in octave.org that is automatically
> updated with a file I'll set up in SF. The following syntax for such
> file would allow for other projects similar to Octave Forge (if and
> when they appear) to also have their -project flag in pkg.
>
> =Forge http://path-text-file-with-list-of-forge-packages
> package1-name package1-version package1-url
> package2-name package2-version package2-url
> package3-name package3-version package3-url
> =ProjectX http://path-text-file-with-list-of-projectx-packages
> package1-name package1-version package1-url
>
> Basically, the word after the equal would be the project and flag used
> to get packages, followed by a URL for a text file that is
> responsibility of that project. That text file would have just the
> project, version, and url lines. Should be pretty simple for a perl
> script to routinely check those URL's for changes and edit the file on
> octave.org.
>
> I'd guess that XML would be better but having to parse them in Octave
> is more complicated and I don't think we need that.
>
> Any comments? Can I start writing such scripts?
>
> Carn?

Carnë,

I like the basic idea of your package list file and it seems a very good strategy
to allow smooth transition if we are to move away the current HTML pages from Octave Forge.

On the other hand there is one nice feature in the current setup that would be nice to keep,
i.e., it is automatically generated rather than manually maintained.

So rather than setting up a format for a file to be maintained manually, I would suggest to modify
generate_package_html so that it generates a text file with the url of the package to be downloaded
and places it in the package docs directory, e.g. for package geometry there should be a file

http://octave.sourceforge.net/geometry/download_url.txt

this would keep the release procedure more automatic and impose less work on the site maintainer.
the root of the url could be set as a variable in Octave so that changing that could also be automated
by issuing a command like

forge_base_url ("http://www.new.forge.org/")

what do you think?
c.



Reply | Threaded
Open this post in threaded view
|

Re: URL for forge packages

Carnë Draug-2
On 18 August 2012 05:45, c. <[hidden email]> wrote:

> On 17 Aug 2012, at 07:24, Carnë Draug wrote:
>> On 16 August 2012 13:14, Carnë Draug <[hidden email]> wrote:
>>> Hi
>>>
>>> the current method to get the URL for packages when installing with
>>> the -forge flag is to parse the HTML of the package for the version
>>> and use it to generate a URL to download it. I see two problems with
>>> this:
>>>
>>> 1. if we are to change the format of the documentation page this will
>>> start failing
>>> 2. if we move the packages to another host (as has been discussed
>>> before), it will also fail
>>>
>>> While such changes are not planned to happen on OF soon, when they do
>>> it will mean that users using octave versions before the change won't
>>> be able to use the -forge flag. I think to prevent this before becomes
>>> an issie it's a good idea.  I would like to propose to have a simple
>>> text file on octave.org with this info. 1 line with package names and
>>> their URLs would suffice. This would also make the code to get
>>> packages from forge simpler since the parsing would also be simpler.
>>>
>>> Any comments?
>>
>> I gave this some more thought and here's the solution I propose. The
>> idea will be to have a text file in octave.org that is automatically
>> updated with a file I'll set up in SF. The following syntax for such
>> file would allow for other projects similar to Octave Forge (if and
>> when they appear) to also have their -project flag in pkg.
>>
>> =Forge http://path-text-file-with-list-of-forge-packages
>> package1-name package1-version package1-url
>> package2-name package2-version package2-url
>> package3-name package3-version package3-url
>> =ProjectX http://path-text-file-with-list-of-projectx-packages
>> package1-name package1-version package1-url
>>
>> Basically, the word after the equal would be the project and flag used
>> to get packages, followed by a URL for a text file that is
>> responsibility of that project. That text file would have just the
>> project, version, and url lines. Should be pretty simple for a perl
>> script to routinely check those URL's for changes and edit the file on
>> octave.org.
>>
>> I'd guess that XML would be better but having to parse them in Octave
>> is more complicated and I don't think we need that.
>>
>> Any comments? Can I start writing such scripts?
>>
>
> Carnë,
>
> I like the basic idea of your package list file and it seems a very good strategy
> to allow smooth transition if we are to move away the current HTML pages from Octave Forge.
>
> On the other hand there is one nice feature in the current setup that would be nice to keep,
> i.e., it is automatically generated rather than manually maintained.
>
> So rather than setting up a format for a file to be maintained manually, I would suggest to modify
> generate_package_html so that it generates a text file with the url of the package to be downloaded
> and places it in the package docs directory, e.g. for package geometry there should be a file
>
> http://octave.sourceforge.net/geometry/download_url.txt
>
> this would keep the release procedure more automatic and impose less work on the site maintainer.
> the root of the url could be set as a variable in Octave so that changing that could also be automated
> by issuing a command like
>
> forge_base_url ("http://www.new.forge.org/")
>
> what do you think?

Of course. Just because I was suggesting a simple text file, doesn't
mean I meant to edit it by hand. The idea is to yes, have a script to
handle all of that. As Søren mentioned, using an octave file will even
makes things much easier for us. jwe suggested on IRC that we could
even use Octave's text format, no need to be a binary.

The only complication I see is the one that Thomas Yengst mentioned
about systems that are connected only to a private network. I see two
solutions. Either pkg needs an option to set the value where to look
for this file/database of URLs (I think that's best) or we recommend
people to change the source of their pkg installation. Those other
systems will want to change the values of the URLs for something
internal so using an octave format will make it more difficult for
them to update it automatically unless they also have Octave
installed. I don't know, I'm not very familiar on how things are
organized on those places. Anyone?

Carnë