Octave-Forge: requirement for a maintainer Makefile for release

classic Classic list List threaded Threaded
23 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Octave-Forge: requirement for a maintainer Makefile for release

Carnë Draug
Hi everyone

As you will know, there is a requirement that Octave Forge packages to
have a clone of their repository in our project at sourceforge with,
at the very least, the state for the package releases.  However, this
sometimes doesn't happen and releases are requested with files that
are missing in the repository.

Many of the packages now have a Makefile at the root of the package
for maintainer tasks such as making a release.  I would propose to
make this a requirement to solve this issue --- since hg and git export
would not export files and changes not commited --- and to reduce the
work required for pushing a release --- since I would not have to check
this myself.

The plan would be that package maintainers would provide a revision
hash (or maybe a revision tag), and whoever is pushing the release
would build the actual release tarball.  The package maintainer would still
be responsible to upload the html documentation.

The only issue I see is with packages that may require a special tool
or a specific version of the tool.  Even different versions of autoconf
could generate slightly different configure scripts.  However, I am
guessing that in practice this will not happen and can be handled at
the time if they do.

Anyone has any more thoughts on this? Anyone opposes?

Carnë

Reply | Threaded
Open this post in threaded view
|

Re: Octave-Forge: requirement for a maintainer Makefile for release

Juan Pablo Carbajal-2
On Fri, Nov 11, 2016 at 8:59 PM, Carnë Draug <[hidden email]> wrote:

> Hi everyone
>
> As you will know, there is a requirement that Octave Forge packages to
> have a clone of their repository in our project at sourceforge with,
> at the very least, the state for the package releases.  However, this
> sometimes doesn't happen and releases are requested with files that
> are missing in the repository.
>
> Many of the packages now have a Makefile at the root of the package
> for maintainer tasks such as making a release.  I would propose to
> make this a requirement to solve this issue --- since hg and git export
> would not export files and changes not commited --- and to reduce the
> work required for pushing a release --- since I would not have to check
> this myself.
>
> The plan would be that package maintainers would provide a revision
> hash (or maybe a revision tag), and whoever is pushing the release
> would build the actual release tarball.  The package maintainer would still
> be responsible to upload the html documentation.
>
> The only issue I see is with packages that may require a special tool
> or a specific version of the tool.  Even different versions of autoconf
> could generate slightly different configure scripts.  However, I am
> guessing that in practice this will not happen and can be handled at
> the time if they do.
>
> Anyone has any more thoughts on this? Anyone opposes?
>
> Carnë
>
I support this, it also brings us closer to an url based installation procedure.

Reply | Threaded
Open this post in threaded view
|

Re: Octave-Forge: requirement for a maintainer Makefile for release

Philip Nienhuis
In reply to this post by Carnë Draug
Carnë Draug wrote
Hi everyone

As you will know, there is a requirement that Octave Forge packages to
have a clone of their repository in our project at sourceforge with,
at the very least, the state for the package releases.  However, this
sometimes doesn't happen and releases are requested with files that
are missing in the repository.

Many of the packages now have a Makefile at the root of the package
for maintainer tasks such as making a release.  I would propose to
make this a requirement to solve this issue --- since hg and git export
would not export files and changes not commited --- and to reduce the
work required for pushing a release --- since I would not have to check
this myself.

The plan would be that package maintainers would provide a revision
hash (or maybe a revision tag), and whoever is pushing the release
would build the actual release tarball.  The package maintainer would still
be responsible to upload the html documentation.

The only issue I see is with packages that may require a special tool
or a specific version of the tool.  Even different versions of autoconf
could generate slightly different configure scripts.  However, I am
guessing that in practice this will not happen and can be handled at
the time if they do.

Anyone has any more thoughts on this? Anyone opposes?
If I interpret your motive correctly, the things you're after is better quality control (QC) and shifting the burden of QC from you to the package maintainers.
The QC aspects solved by your suggestion comprise missing files, outdated files (not updated in the repo), autoconf & configure having been run and I might overlook a few things. In summary, the more "administrative" parts.
 
OK I agree that QC shouldn't be a primary task of the Octave-Forge admin; maybe a responsibility but if so a responsibility that can at least be delegated.

I doubt if effective QC can be done by the same person as the one who created the code & package in the first place (=package maintainer).
Yet automating boring parts of it, if only by requiring a Makefile with predescribed contents, may help catch the basic errors.
Can makefiles be used to check if an on-line repo is up-to-date?

As to separating the release tarball from the html tarball: my experience is that distributed responsibilities just don't work reliably.
Can a makefile be used to create (and upload) package html (by invoking Octave)? (I'd guess yes)

One of the things with the current Makefile setup is that it also generates a release tag.
I usually wait with a release tag until the release is finally on Octave-Forge as it sometimes happens that additional bugs are discovered and fixes are needed while the release waits on the release tracker; to avoid "gaps" in the release numbers. But maybe release version gaps isn't such a big deal.

Philip
Reply | Threaded
Open this post in threaded view
|

Re: Octave-Forge: requirement for a maintainer Makefile for release

Carnë Draug
On 11 November 2016 at 22:09, Philip Nienhuis <[hidden email]> wrote:

> Carnë Draug wrote
>> Hi everyone
>>
>> As you will know, there is a requirement that Octave Forge packages to
>> have a clone of their repository in our project at sourceforge with,
>> at the very least, the state for the package releases.  However, this
>> sometimes doesn't happen and releases are requested with files that
>> are missing in the repository.
>>
>> Many of the packages now have a Makefile at the root of the package
>> for maintainer tasks such as making a release.  I would propose to
>> make this a requirement to solve this issue --- since hg and git export
>> would not export files and changes not commited --- and to reduce the
>> work required for pushing a release --- since I would not have to check
>> this myself.
>>
>> The plan would be that package maintainers would provide a revision
>> hash (or maybe a revision tag), and whoever is pushing the release
>> would build the actual release tarball.  The package maintainer would
>> still
>> be responsible to upload the html documentation.
>>
>> The only issue I see is with packages that may require a special tool
>> or a specific version of the tool.  Even different versions of autoconf
>> could generate slightly different configure scripts.  However, I am
>> guessing that in practice this will not happen and can be handled at
>> the time if they do.
>>
>> Anyone has any more thoughts on this? Anyone opposes?
>
> If I interpret your motive correctly, the things you're after is better
> quality control (QC) and shifting the burden of QC from you to the package
> maintainers.
> The QC aspects solved by your suggestion comprise missing files, outdated
> files (not updated in the repo), autoconf & configure having been run and I
> might overlook a few things. In summary, the more "administrative" parts.
>
> OK I agree that QC shouldn't be a primary task of the Octave-Forge admin;
> maybe a responsibility but if so a responsibility that can at least be
> delegated.
>
> I doubt if effective QC can be done by the same person as the one who
> created the code & package in the first place (=package maintainer).
> Yet automating boring parts of it, if only by requiring a Makefile with
> predescribed contents, may help catch the basic errors.
> Can makefiles be used to check if an on-line repo is up-to-date?
>
> As to separating the release tarball from the html tarball: my experience is
> that distributed responsibilities just don't work reliably.
> Can a makefile be used to create (and upload) package html (by invoking
> Octave)? (I'd guess yes)

makefiles are just a list of commands.  If you can write it, then you can
have your Makefile do it.  All Makefiles at the moment also create the
html so that is already done.  In theory you could also have code to create
a new ticket on the release tracker and upload it.

Note that I am not asking for a standard Makefile across all repos.  All
I am proposing is that at least a "dist" target must exist to recreate
the release tarball.  I would say this is more about reproducibility than
quality control.

> One of the things with the current Makefile setup is that it also generates
> a release tag.
> I usually wait with a release tag until the release is finally on
> Octave-Forge as it sometimes happens that additional bugs are discovered and
> fixes are needed while the release waits on the release tracker; to avoid
> "gaps" in the release numbers. But maybe release version gaps isn't such a
> big deal.

I just checked all Makefiles in Octave forge and not a single one tags the
release.  Some tell you to tag the release but none actually does it.  Even
if they did tag the release, tags are just a line in file that you can change
afterwards if the release fails review for some reason.

Carnë

Reply | Threaded
Open this post in threaded view
|

Re: Octave-Forge: requirement for a maintainer Makefile for release

PhilipNienhuis
Carnë Draug wrote:

> On 11 November 2016 at 22:09, Philip Nienhuis <[hidden email]> wrote:
>> Carnë Draug wrote
>>> Hi everyone
>>>
>>> As you will know, there is a requirement that Octave Forge packages to
>>> have a clone of their repository in our project at sourceforge with,
>>> at the very least, the state for the package releases.  However, this
>>> sometimes doesn't happen and releases are requested with files that
>>> are missing in the repository.
>>>
>>> Many of the packages now have a Makefile at the root of the package
>>> for maintainer tasks such as making a release.  I would propose to
>>> make this a requirement to solve this issue --- since hg and git export
>>> would not export files and changes not commited --- and to reduce the
>>> work required for pushing a release --- since I would not have to check
>>> this myself.
>>>
>>> The plan would be that package maintainers would provide a revision
>>> hash (or maybe a revision tag), and whoever is pushing the release
>>> would build the actual release tarball.  The package maintainer would
>>> still
>>> be responsible to upload the html documentation.
>>>
>>> The only issue I see is with packages that may require a special tool
>>> or a specific version of the tool.  Even different versions of autoconf
>>> could generate slightly different configure scripts.  However, I am
>>> guessing that in practice this will not happen and can be handled at
>>> the time if they do.
>>>
>>> Anyone has any more thoughts on this? Anyone opposes?
>>
>> If I interpret your motive correctly, the things you're after is better
>> quality control (QC) and shifting the burden of QC from you to the package
>> maintainers.
>> The QC aspects solved by your suggestion comprise missing files, outdated
>> files (not updated in the repo), autoconf & configure having been run and I
>> might overlook a few things. In summary, the more "administrative" parts.
>>
>> OK I agree that QC shouldn't be a primary task of the Octave-Forge admin;
>> maybe a responsibility but if so a responsibility that can at least be
>> delegated.
>>
>> I doubt if effective QC can be done by the same person as the one who
>> created the code & package in the first place (=package maintainer).
>> Yet automating boring parts of it, if only by requiring a Makefile with
>> predescribed contents, may help catch the basic errors.
>> Can makefiles be used to check if an on-line repo is up-to-date?
>>
>> As to separating the release tarball from the html tarball: my experience is
>> that distributed responsibilities just don't work reliably.
>> Can a makefile be used to create (and upload) package html (by invoking
>> Octave)? (I'd guess yes)
>
> makefiles are just a list of commands.  If you can write it, then you can
> have your Makefile do it.  All Makefiles at the moment also create the
> html so that is already done.  In theory you could also have code to create
> a new ticket on the release tracker and upload it.
>
> Note that I am not asking for a standard Makefile across all repos.  All
> I am proposing is that at least a "dist" target must exist to recreate
> the release tarball.  I would say this is more about reproducibility than
> quality control.

Reproducibility is a form of QC.

>> One of the things with the current Makefile setup is that it also generates
>> a release tag.
>> I usually wait with a release tag until the release is finally on
>> Octave-Forge as it sometimes happens that additional bugs are discovered and
>> fixes are needed while the release waits on the release tracker; to avoid
>> "gaps" in the release numbers. But maybe release version gaps isn't such a
>> big deal.
>
> I just checked all Makefiles in Octave forge and not a single one tags the
> release.  Some tell you to tag the release but none actually does it.  Even
> if they did tag the release, tags are just a line in file that you can change
> afterwards if the release fails review for some reason.

You're right, I just checked in of-mapping. And it can be rolled back.
Sorry for the confusion.


Some things I do not understand about the first two paragraphs in your
original post:
AFAIU missing files are a primary motive, right?

Now, even if the OF site admin were to make a release based on some
revision of the on-line repo using a package makefile, there's no
guarantee that the on-line repo revision is "complete". The OF package
maintainer may still have forgotten to check in files into his local
repo before syncing it to the on-line repo. A makefile won't help there.

A more robust check would be if all OF packages would contain a test
suite that covers the entire package contents. Or something like lint to
check & follow all function calls.

Not that I am opposed to a makefile with a dist (+ html) target; it'll
help to run autoconf/bootstrap etc (I remember to have forgotten that
myself before uploading a package) and -most of all- automates a lot of
work.

Philip


Reply | Threaded
Open this post in threaded view
|

Re: Octave-Forge: requirement for a maintainer Makefile for release

Carnë Draug
On 13 November 2016 at 21:36, Philip Nienhuis <[hidden email]> wrote:

> [...]
> Some things I do not understand about the first two paragraphs in your
> original post:
> AFAIU missing files are a primary motive, right?
>
> Now, even if the OF site admin were to make a release based on some revision
> of the on-line repo using a package makefile, there's no guarantee that the
> on-line repo revision is "complete". The OF package maintainer may still
> have forgotten to check in files into his local repo before syncing it to
> the on-line repo. A makefile won't help there.

It might.  A package maintainer only needs to give a revision to make the
release.  However, it should still do the release himself via the Makefile
and test it.  If it does, then it might notice the missing files.

But the issue with missing files is that we are left unable to reproduce
the release from the repository.  A Makefile will fix this.  That's the
issue this is suppose to fix.

> A more robust check would be if all OF packages would contain a test suite
> that covers the entire package contents. Or something like lint to check &
> follow all function calls.

At the moment, the Makefiles have a check and a install target.  I guess a
"check-dist" target could be added to install and then check the installation.

> Not that I am opposed to a makefile with a dist (+ html) target; it'll help
> to run autoconf/bootstrap etc (I remember to have forgotten that myself
> before uploading a package) and -most of all- automates a lot of work.
>

Good.  Let's wait a few more days to see if anyone opposes.

Carnë

Reply | Threaded
Open this post in threaded view
|

Re: Octave-Forge: requirement for a maintainer Makefile for release

Alexander Barth-3
Hi all,

I am maintain currently 6 packages and I use a bash script to prepare
a release (do a clean check out, remove unnecessary files such as
.gitignore, make configure script, run tests, generate documentation
...)

I think it would be good to have all these tasks at a central place (a
bash script or a Makefile fragment) rather than to explicitly copy
this logic in all 6 Makefiles. Otherwise it would be quite difficult
to maintain the Makefiles in the long run.

Best regards,
Alex

Reply | Threaded
Open this post in threaded view
|

Re: Octave-Forge: requirement for a maintainer Makefile for release

Oliver Heimlich
In reply to this post by Carnë Draug
On 13.11.2016 23:34, Carnë Draug wrote:

> On 13 November 2016 at 21:36, Philip Nienhuis <[hidden email]> wrote:
>> Not that I am opposed to a makefile with a dist (+ html) target; it'll help
>> to run autoconf/bootstrap etc (I remember to have forgotten that myself
>> before uploading a package) and -most of all- automates a lot of work.
>>
>
> Good.  Let's wait a few more days to see if anyone opposes.
>
> Carnë
>

“Any sufficiently advanced Makefile is indistinguishable from magic.”

I have never written Makefiles before creating an Octave package.  It is
one of many things which I have learned from developing in the Octave
ecosystem.  I can clearly remember the time before that, when Makefiles
have been something that I tried to evade because of their syntax and
how they work, which I didn't understand at that time.

Looking back, Makefiles are a very simple tool. But you have to practice
a little bit before they lose scariness. :-)

So, if Makefiles become a requirement, please make sure that we offer as
much support as possible to new developers, such that this is not seen
as a burden.

* List a Makefile template in the wiki or at
  http://octave.sourceforge.net/developers.html
* Offer help to adopt the Makefile to someone's project
  via maintainers mailing list.

Oliver

Reply | Threaded
Open this post in threaded view
|

Re: Octave-Forge: requirement for a maintainer Makefile for release

Carnë Draug
On 20 November 2016 at 13:03, Oliver Heimlich <[hidden email]> wrote:

> On 13.11.2016 23:34, Carnë Draug wrote:
>> On 13 November 2016 at 21:36, Philip Nienhuis <[hidden email]> wrote:
>>> Not that I am opposed to a makefile with a dist (+ html) target; it'll help
>>> to run autoconf/bootstrap etc (I remember to have forgotten that myself
>>> before uploading a package) and -most of all- automates a lot of work.
>>>
>>
>> Good.  Let's wait a few more days to see if anyone opposes.
>>
>> Carnë
>>
>
> “Any sufficiently advanced Makefile is indistinguishable from magic.”
>
> I have never written Makefiles before creating an Octave package.  It is
> one of many things which I have learned from developing in the Octave
> ecosystem.  I can clearly remember the time before that, when Makefiles
> have been something that I tried to evade because of their syntax and
> how they work, which I didn't understand at that time.
>
> Looking back, Makefiles are a very simple tool. But you have to practice
> a little bit before they lose scariness. :-)
>
> So, if Makefiles become a requirement, please make sure that we offer as
> much support as possible to new developers, such that this is not seen
> as a burden.
>
> * List a Makefile template in the wiki or at
>   http://octave.sourceforge.net/developers.html

I have uploaded a template with several comments [1] which I hope it
explains most of the magic.  Can you take a look and maybe expand [2]
any magic that was not obvious to you at the start?

> * Offer help to adopt the Makefile to someone's project
>   via maintainers mailing list.

Of course.  There's also many other people in the mailing list that can
help with Makefiles but I have been helping shape Makefiles in packages.

Carnë

[1] http://octave.sourceforge.net/templates/Makefile
[2] http://hg.code.sf.net/p/octave/project-web/file/d2dd7f6a47d6/htdocs/templates/Makefile

Reply | Threaded
Open this post in threaded view
|

Re: Octave-Forge: requirement for a maintainer Makefile for release

John W. Eaton
Administrator
On 11/23/2016 06:18 PM, Carnë Draug wrote:

> I have uploaded a template with several comments [1] which I hope it
> explains most of the magic.  Can you take a look and maybe expand [2]
> any magic that was not obvious to you at the start?

I generally avoid inheriting values from the environment if possible, so
instead of

   OCTAVE ?= octave

I would just write

   OCTAVE := octave

You can still set the value on the command line, but you have to do it
with a variable assignment in the argument list that is passed to make:

   make OCTAVE=/some/other/octave

Doing it this way avoids the mysterious problems and confusion that can
happen if someone has set OCTAVE in their environment and then later
forgot about it.

jwe


Reply | Threaded
Open this post in threaded view
|

Re: Octave-Forge: requirement for a maintainer Makefile for release

Carnë Draug
In reply to this post by Alexander Barth-3
On 16 November 2016 at 14:47, Alexander Barth <[hidden email]> wrote:

> Hi all,
>
> I am maintain currently 6 packages and I use a bash script to prepare
> a release (do a clean check out, remove unnecessary files such as
> .gitignore, make configure script, run tests, generate documentation
> ...)
>
> I think it would be good to have all these tasks at a central place (a
> bash script or a Makefile fragment) rather than to explicitly copy
> this logic in all 6 Makefiles. Otherwise it would be quite difficult
> to maintain the Makefiles in the long run.
>

The problem with a central place with a unique script or Makefile for
releases is that it would impose a unique shape on all packages which
would be too restritive (or would have to be too complex to achieve
that flexibility).  For example, not all packages need to run autoconf
to create the configure script.  Some of those need also to run aclocal
before.  Some packages have directories for scripts and code snippets
that are removed from package releases.  The ltfat Octave package is
only part of a larger project.  I am sure there will be other cases.

You can still keep all the 6 Makefile on your packages the same, it's
just that you would have to sync them when a change is made.  You can
even a one line Makefile that only calls your script (but your existing
script would still have to be committed on the six repos).  I would
guess that they wouldn't change much overtime so it wouldn't be too
much of a burden.

Also, while it would be nice to also have targets on the Makefile to
run the tests and install development versions, I am only proposing
a target to make a release which can be this:

    PACKAGE := $(shell grep "^Name: " DESCRIPTION | cut -f2 -d" ")
    VERSION := $(shell grep "^Version: " DESCRIPTION | cut -f2 -d" ")

    TARGET_DIR := target/
    RELEASE_DIR     := $(TARGET_DIR)/$(PACKAGE)-$(VERSION)
    RELEASE_TARBALL := $(TARGET_DIR)/$(PACKAGE)-$(VERSION).tar.gz

    .PHONY: dist

    dist: $(RELEASE_TARBALL)

    %.tar.gz: %
        tar -c -f - --posix -C "$(TARGET_DIR)/" "$(notdir $<)" | gzip -9n > "$@"

    $(RELEASE_DIR): .git/index
        $(RM) -r "$@"
        git archive --format=tar --prefix="$@/" HEAD | tar -x
        $(RM) "$@/.gitignore"
        chmod -R a+rX,u+w,go-w "$@"

I have also written a template [1] for the developers instructions based on
the existing Makefiles now present on the forge packages (the example above
is adapted from it).  Could you give any feedback if that can help?

Carnë

[1] http://hg.code.sf.net/p/octave/project-web/file/d2dd7f6a47d6/htdocs/templates/Makefile

Reply | Threaded
Open this post in threaded view
|

Re: Octave-Forge: requirement for a maintainer Makefile for release

Juan Pablo Carbajal-2
On Fri, Nov 25, 2016 at 12:45 AM, Carnë Draug <[hidden email]> wrote:

> On 16 November 2016 at 14:47, Alexander Barth <[hidden email]> wrote:
>> Hi all,
>>
>> I am maintain currently 6 packages and I use a bash script to prepare
>> a release (do a clean check out, remove unnecessary files such as
>> .gitignore, make configure script, run tests, generate documentation
>> ...)
>>
>> I think it would be good to have all these tasks at a central place (a
>> bash script or a Makefile fragment) rather than to explicitly copy
>> this logic in all 6 Makefiles. Otherwise it would be quite difficult
>> to maintain the Makefiles in the long run.
>>
>
> The problem with a central place with a unique script or Makefile for
> releases is that it would impose a unique shape on all packages which
> would be too restritive (or would have to be too complex to achieve
> that flexibility).  For example, not all packages need to run autoconf
> to create the configure script.  Some of those need also to run aclocal
> before.  Some packages have directories for scripts and code snippets
> that are removed from package releases.  The ltfat Octave package is
> only part of a larger project.  I am sure there will be other cases.
>
> You can still keep all the 6 Makefile on your packages the same, it's
> just that you would have to sync them when a change is made.  You can
> even a one line Makefile that only calls your script (but your existing
> script would still have to be committed on the six repos).  I would
> guess that they wouldn't change much overtime so it wouldn't be too
> much of a burden.
>
> Also, while it would be nice to also have targets on the Makefile to
> run the tests and install development versions, I am only proposing
> a target to make a release which can be this:
>
>     PACKAGE := $(shell grep "^Name: " DESCRIPTION | cut -f2 -d" ")
>     VERSION := $(shell grep "^Version: " DESCRIPTION | cut -f2 -d" ")
>
>     TARGET_DIR := target/
>     RELEASE_DIR     := $(TARGET_DIR)/$(PACKAGE)-$(VERSION)
>     RELEASE_TARBALL := $(TARGET_DIR)/$(PACKAGE)-$(VERSION).tar.gz
>
>     .PHONY: dist
>
>     dist: $(RELEASE_TARBALL)
>
>     %.tar.gz: %
>         tar -c -f - --posix -C "$(TARGET_DIR)/" "$(notdir $<)" | gzip -9n > "$@"
>
>     $(RELEASE_DIR): .git/index
>         $(RM) -r "$@"
>         git archive --format=tar --prefix="$@/" HEAD | tar -x
>         $(RM) "$@/.gitignore"
>         chmod -R a+rX,u+w,go-w "$@"
>
> I have also written a template [1] for the developers instructions based on
> the existing Makefiles now present on the forge packages (the example above
> is adapted from it).  Could you give any feedback if that can help?
>
> Carnë
>
> [1] http://hg.code.sf.net/p/octave/project-web/file/d2dd7f6a47d6/htdocs/templates/Makefile
>

Carne,

The Makefile in templates will fail to make check with packages that
have dependencies. They need ot be loaded. Mike fixed this in signal,
then I improve it in geometry.

DEPENDS := $(shell $(SED) -n -e 's/^Depends[^,]*, \(.*\)/\1/p'
DESCRIPTION | $(SED) 's/ *([^()]*),*/ /g')

check: all
$(OCTAVE) --silent \
 --eval 'if(!isempty("$(DEPENDS)")); pkg load $(DEPENDS); endif;' \
 --eval 'addpath (fullfile ([pwd filesep "inst"]));' \
 --eval '$(PKG_ADD)'\
 --eval 'cellfun(@(x)runtests (x), __geometry_package_register__);'

Same goes for make run

Reply | Threaded
Open this post in threaded view
|

Re: Octave-Forge: requirement for a maintainer Makefile for release

Juan Pablo Carbajal-2
On Wed, Nov 30, 2016 at 12:40 AM, Juan Pablo Carbajal
<[hidden email]> wrote:

> On Fri, Nov 25, 2016 at 12:45 AM, Carnë Draug <[hidden email]> wrote:
>> On 16 November 2016 at 14:47, Alexander Barth <[hidden email]> wrote:
>>> Hi all,
>>>
>>> I am maintain currently 6 packages and I use a bash script to prepare
>>> a release (do a clean check out, remove unnecessary files such as
>>> .gitignore, make configure script, run tests, generate documentation
>>> ...)
>>>
>>> I think it would be good to have all these tasks at a central place (a
>>> bash script or a Makefile fragment) rather than to explicitly copy
>>> this logic in all 6 Makefiles. Otherwise it would be quite difficult
>>> to maintain the Makefiles in the long run.
>>>
>>
>> The problem with a central place with a unique script or Makefile for
>> releases is that it would impose a unique shape on all packages which
>> would be too restritive (or would have to be too complex to achieve
>> that flexibility).  For example, not all packages need to run autoconf
>> to create the configure script.  Some of those need also to run aclocal
>> before.  Some packages have directories for scripts and code snippets
>> that are removed from package releases.  The ltfat Octave package is
>> only part of a larger project.  I am sure there will be other cases.
>>
>> You can still keep all the 6 Makefile on your packages the same, it's
>> just that you would have to sync them when a change is made.  You can
>> even a one line Makefile that only calls your script (but your existing
>> script would still have to be committed on the six repos).  I would
>> guess that they wouldn't change much overtime so it wouldn't be too
>> much of a burden.
>>
>> Also, while it would be nice to also have targets on the Makefile to
>> run the tests and install development versions, I am only proposing
>> a target to make a release which can be this:
>>
>>     PACKAGE := $(shell grep "^Name: " DESCRIPTION | cut -f2 -d" ")
>>     VERSION := $(shell grep "^Version: " DESCRIPTION | cut -f2 -d" ")
>>
>>     TARGET_DIR := target/
>>     RELEASE_DIR     := $(TARGET_DIR)/$(PACKAGE)-$(VERSION)
>>     RELEASE_TARBALL := $(TARGET_DIR)/$(PACKAGE)-$(VERSION).tar.gz
>>
>>     .PHONY: dist
>>
>>     dist: $(RELEASE_TARBALL)
>>
>>     %.tar.gz: %
>>         tar -c -f - --posix -C "$(TARGET_DIR)/" "$(notdir $<)" | gzip -9n > "$@"
>>
>>     $(RELEASE_DIR): .git/index
>>         $(RM) -r "$@"
>>         git archive --format=tar --prefix="$@/" HEAD | tar -x
>>         $(RM) "$@/.gitignore"
>>         chmod -R a+rX,u+w,go-w "$@"
>>
>> I have also written a template [1] for the developers instructions based on
>> the existing Makefiles now present on the forge packages (the example above
>> is adapted from it).  Could you give any feedback if that can help?
>>
>> Carnë
>>
>> [1] http://hg.code.sf.net/p/octave/project-web/file/d2dd7f6a47d6/htdocs/templates/Makefile
>>
>
> Carne,
>
> The Makefile in templates will fail to make check with packages that
> have dependencies. They need ot be loaded. Mike fixed this in signal,
> then I improve it in geometry.
>
> DEPENDS := $(shell $(SED) -n -e 's/^Depends[^,]*, \(.*\)/\1/p'
> DESCRIPTION | $(SED) 's/ *([^()]*),*/ /g')
>
> check: all
> $(OCTAVE) --silent \
>  --eval 'if(!isempty("$(DEPENDS)")); pkg load $(DEPENDS); endif;' \
>  --eval 'addpath (fullfile ([pwd filesep "inst"]));' \
>  --eval '$(PKG_ADD)'\
>  --eval 'cellfun(@(x)runtests (x), __geometry_package_register__);'
>
> Same goes for make run

Hi,

I have merged what was nice form that makefile with the one in
geometry the result is in the repo [1].

make clean is currently unsafe, for example

make TARGET_DIR=/home/myuser all
make TARGET_DIR=/home/myuser clean

bye bye myuser folder!

instead of

$(RM) -r $(TARGET_DIR)

better use

$(RM) -r $(RELEASE_DIR) $(RELEASE_TARBALL) $(HTML_TARBALL) $(HTML_DIR)

[1]: https://sourceforge.net/p/octave/geometry/ci/default/tree/Makefile

Reply | Threaded
Open this post in threaded view
|

Re: Octave-Forge: requirement for a maintainer Makefile for release

Carnë Draug
On 29 November 2016 at 23:40, Juan Pablo Carbajal <[hidden email]> wrote:

>
> Carne,
>
> The Makefile in templates will fail to make check with packages that
> have dependencies. They need ot be loaded. Mike fixed this in signal,
> then I improve it in geometry.
>
> DEPENDS := $(shell $(SED) -n -e 's/^Depends[^,]*, \(.*\)/\1/p'
> DESCRIPTION | $(SED) 's/ *([^()]*),*/ /g')
>
> check: all
> $(OCTAVE) --silent \
>  --eval 'if(!isempty("$(DEPENDS)")); pkg load $(DEPENDS); endif;' \
>  --eval 'addpath (fullfile ([pwd filesep "inst"]));' \
>  --eval '$(PKG_ADD)'\
>  --eval 'cellfun(@(x)runtests (x), __geometry_package_register__);'
>
> Same goes for make run

Can you make the change and push it [1]?  And add some comments about
what it is for, and that may not be necessary for all packages?

On 30 November 2016 at 01:16, Juan Pablo Carbajal <[hidden email]> wrote:

> [...]
> make clean is currently unsafe, for example
>
> make TARGET_DIR=/home/myuser all
> make TARGET_DIR=/home/myuser clean
>
> bye bye myuser folder!
>
> instead of
>
> $(RM) -r $(TARGET_DIR)
>
> better use
>
> $(RM) -r $(RELEASE_DIR) $(RELEASE_TARBALL) $(HTML_TARBALL) $(HTML_DIR)

It would be unsafe if the package maintainer set up TARGET_DIR to his
own home directory but yeah I can imagine someone doing it by accident.
I still prefer the current code on the template because it will also
clean releases other than the current version (which I guess is more
frequent in packages with stable and default branche).

Can you make your proposed change on the clean target then and add a comment
that it will not clean all older builds and how to change?

Thank you
Carnë

[1] http://hg.code.sf.net/p/octave/project-web

Reply | Threaded
Open this post in threaded view
|

Re: Octave-Forge: requirement for a maintainer Makefile for release

Juan Pablo Carbajal-2
On Thu, Dec 1, 2016 at 10:04 PM, Carnë Draug <[hidden email]> wrote:

> On 29 November 2016 at 23:40, Juan Pablo Carbajal <[hidden email]> wrote:
>>
>> Carne,
>>
>> The Makefile in templates will fail to make check with packages that
>> have dependencies. They need ot be loaded. Mike fixed this in signal,
>> then I improve it in geometry.
>>
>> DEPENDS := $(shell $(SED) -n -e 's/^Depends[^,]*, \(.*\)/\1/p'
>> DESCRIPTION | $(SED) 's/ *([^()]*),*/ /g')
>>
>> check: all
>> $(OCTAVE) --silent \
>>  --eval 'if(!isempty("$(DEPENDS)")); pkg load $(DEPENDS); endif;' \
>>  --eval 'addpath (fullfile ([pwd filesep "inst"]));' \
>>  --eval '$(PKG_ADD)'\
>>  --eval 'cellfun(@(x)runtests (x), __geometry_package_register__);'
>>
>> Same goes for make run
>
> Can you make the change and push it [1]?  And add some comments about
> what it is for, and that may not be necessary for all packages?
>
> On 30 November 2016 at 01:16, Juan Pablo Carbajal <[hidden email]> wrote:
>> [...]
>> make clean is currently unsafe, for example
>>
>> make TARGET_DIR=/home/myuser all
>> make TARGET_DIR=/home/myuser clean
>>
>> bye bye myuser folder!
>>
>> instead of
>>
>> $(RM) -r $(TARGET_DIR)
>>
>> better use
>>
>> $(RM) -r $(RELEASE_DIR) $(RELEASE_TARBALL) $(HTML_TARBALL) $(HTML_DIR)
>
> It would be unsafe if the package maintainer set up TARGET_DIR to his
> own home directory but yeah I can imagine someone doing it by accident.
> I still prefer the current code on the template because it will also
> clean releases other than the current version (which I guess is more
> frequent in packages with stable and default branche).
>
> Can you make your proposed change on the clean target then and add a comment
> that it will not clean all older builds and how to change?
>
> Thank you
> Carnë
>
> [1] http://hg.code.sf.net/p/octave/project-web

Done,

I couldn't check it (no time!).
I guess you could improve the DEPENDS part to be like the others. I
did not check how you are using cut, but seems much simpler.

I also suggest using runtests instead of an undocumented function, if
the objective is to make the thing readable.

You are free to reject any of the changes.

Cheers

Reply | Threaded
Open this post in threaded view
|

Re: Octave-Forge: requirement for a maintainer Makefile for release

Alexander Barth-3
In reply to this post by Carnë Draug
"You can still keep all the 6 Makefile on your packages the same, it's
just that you would have to sync them when a change is made.  You can
even a one line Makefile that only calls your script (but your existing
script would still have to be committed on the six repos).  I would
guess that they wouldn't change much overtime so it wouldn't be too
much of a burden."

Yes, this is what I wanted to avoid: copying code in 6 different
places and maintaining all 6 copies.


Last time I checked, it was not necessary for R or Julia package to
provide a Makefile for package containing only scripts. Shouldn't we
rather make to process of submitting a package not as simple as
possible (at least for packages which a just a collection of scripts
without any C/C++ or Fortran code)? The number of octave package is
with 69 (+33 unmaintained) rather low compared to Julia (1128
including the unmaintained packages) or R (9624).

Reply | Threaded
Open this post in threaded view
|

Re: Octave-Forge: requirement for a maintainer Makefile for release

Juan Pablo Carbajal-2
On Fri, Dec 2, 2016 at 2:24 PM, Alexander Barth
<[hidden email]> wrote:

> "You can still keep all the 6 Makefile on your packages the same, it's
> just that you would have to sync them when a change is made.  You can
> even a one line Makefile that only calls your script (but your existing
> script would still have to be committed on the six repos).  I would
> guess that they wouldn't change much overtime so it wouldn't be too
> much of a burden."
>
> Yes, this is what I wanted to avoid: copying code in 6 different
> places and maintaining all 6 copies.
>
>
> Last time I checked, it was not necessary for R or Julia package to
> provide a Makefile for package containing only scripts. Shouldn't we
> rather make to process of submitting a package not as simple as
> possible (at least for packages which a just a collection of scripts
> without any C/C++ or Fortran code)? The number of octave package is
> with 69 (+33 unmaintained) rather low compared to Julia (1128
> including the unmaintained packages) or R (9624).
>

i support any effort that aims at reducing the hurdle to people to
contribute code. however, to keep Carnë's high standards in OF, I
would rather extend pkg to accept a url and install "unofficial"
packages (similar to what Julia does). Those packages do not need to
follow OF standars.
Last time I checked Carlo de Falco was evaluating the possibility of
setting up what Jluia has for their packages.

Reply | Threaded
Open this post in threaded view
|

Re: Octave-Forge: requirement for a maintainer Makefile for release

Carnë Draug
In reply to this post by Alexander Barth-3
On 2 December 2016 at 13:24, Alexander Barth <[hidden email]> wrote:

> "You can still keep all the 6 Makefile on your packages the same, it's
> just that you would have to sync them when a change is made.  You can
> even a one line Makefile that only calls your script (but your existing
> script would still have to be committed on the six repos).  I would
> guess that they wouldn't change much overtime so it wouldn't be too
> much of a burden."
>
> Yes, this is what I wanted to avoid: copying code in 6 different
> places and maintaining all 6 copies.
>
>
> Last time I checked, it was not necessary for R or Julia package to
> provide a Makefile for package containing only scripts. Shouldn't we
> rather make to process of submitting a package not as simple as
> possible (at least for packages which a just a collection of scripts
> without any C/C++ or Fortran code)? The number of octave package is
> with 69 (+33 unmaintained) rather low compared to Julia (1128
> including the unmaintained packages) or R (9624).

R and Julia packages do not need a Makefile and neither does Octave
which will happily install from a tarball.  The requirement is on
Octave Forge which is one site for Octave packages.

In Julia a package must be a git a repository and the structure in
the repository must be the structure of the "release" since the release
is just a tag on the repository [1].

In R I know of at least 3 package sites: CRAN, bioconductor, and r-forge.
I know little of those but at least bioconductor requires a fixed
structure on a github repository so that core language commands
("R CMD build") can build a package from a known commit in the
master branch [2].

They still require package maintainers to provide a commit hash (or tag)
to make the release just like we do at Octave Forge now.  They do not
require a Makefile but they require a fixed structure on the repository.

If a package in the repository has the same structure as the package
release, then making a release is as simple as exporting a specific
commit.  That will be a really simple Makefile and may looks pointless.
But requiring a Makefile instead of a fixed structure allows package
maintainers to develop in any structure they want.

Anyway, what the other projects do and don't do is not really important.
One of the requirements in Octave Forge is that we host a clone of the
repository from where releases can be made.  Until now we have been
trusting package maintainers but packages have been released with code
that was never committed.  A method to reproduce a release from the
repository is needed to prevent that and since a lot of packages already
have a maintainer Makefile, that seems like the best option.

Carnë

[1] http://docs.julialang.org/en/release-0.5/manual/packages/#package-development
[2] https://github.com/Bioconductor/Contributions/blob/master/CONTRIBUTING.md

Reply | Threaded
Open this post in threaded view
|

Re: Octave-Forge: requirement for a maintainer Makefile for release

Oliver Heimlich
On 10.12.2016 19:55, Carnë Draug wrote:
> If a package in the repository has the same structure as the package
> release, then making a release is as simple as exporting a specific
> commit.  That will be a really simple Makefile and may looks pointless.
> But requiring a Makefile instead of a fixed structure allows package
> maintainers to develop in any structure they want.

I can give an example from the interval package.  I, as a developer,
have chosen not to put any generated files under version control.  When
I started the package this seemed like a less important decision and I
have decided this way because it seemed like the cleanest solution.

As a consequence, making a release is not a simple “hg archive”.  I need
a maintainer Makefile and have to regenerate a lot of files and patch
them into the tarball created by “hg archive”.  This step involves a
bunch of more or less exotic programs which I do not expect to be
present at the user's computer (otherwise the files could have been
generated by src/Makefile during “pkg install”).  All of these steps are
very specific to this particular package and when I wrote the maintainer
Makefile I didn't really care about interoperability of the Makefile for
different systems.

At the moment it is not easy to make a release for the interval package
unless you have installed all aforementioned dependencies.

However, if we decide that it should be easier to make or verify a
package release by a third party, I could just put the generated files
under version control.  The maintainer Makefile would barely change,
except that the release target became much simpler after a fresh clone
of the repository.  The other side of the coin is that I would have to
be more careful to keep generated files in sync during commits.

> Anyway, what the other projects do and don't do is not really important.
> One of the requirements in Octave Forge is that we host a clone of the
> repository from where releases can be made.  Until now we have been
> trusting package maintainers but packages have been released with code
> that was never committed.  A method to reproduce a release from the
> repository is needed to prevent that and since a lot of packages already
> have a maintainer Makefile, that seems like the best option.

The only problem that I see here is that you can get into “dependency
hell” as mentioned above.  Putting all generated files under version
control could be a way out and this would also allow to simply “pkg
install” a development version directly from the source repository.

Best
Oliver

Reply | Threaded
Open this post in threaded view
|

Re: Octave-Forge: requirement for a maintainer Makefile for release

Colin Macdonald-2
On 11/12/16 02:45, Oliver Heimlich wrote:
> Putting all generated files under version
> control could be a way out and this would also allow to simply “pkg
> install” a development version directly from the source repository.

+1 for this last point.  For example, suppose a user reports bug with
current release; this lowers the bar for her to check with the current
development branch.

Colin

12