gnulib is no longer an hg subrepo

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

gnulib is no longer an hg subrepo

John W. Eaton
Administrator
After this change (on stable, merged with default):

   http://hg.savannah.gnu.org/hgweb/octave/rev/ece72b94486f

the gnulib sources are no longer managed within the Octave mercurial
archive as an hg subrepo.  Instead, we now have the bootstrap script
fetching a copy of gnulib using git.

After updating the Octave sources to get this change you will need to
remove the gnulib subdirectory in your source tree and run bootstrap to
pull a new copy of gnulib from upstream (commands executed at the
top-level of the Octave source tree):

   hg pull -u
   rm -rf gnulib
   ./bootstrap

The gnulib revision is now stored in the bootstrap.conf file instead of
.hgsubstate.  You may override this revision by setting GNULIB_REVISION
in the environment when you run the bootstrap script.  For example (bash
syntax):

   GNULIB_REVISION=8cb410df8b66d3bd0fe43ef580a9f912cc402a96 ./bootstrap

If the revision is empty, bootstrap will use whatever the current
revision is in your gnulib directory.

Note that the specified revision must be present in your gnulib source
tree.  If it is not, then bootstrap will fail to set the revision until
you update your gnulib sources from upstream so that it includes the
revision you are requesting.  This limitation is a feature imposed by
the bootstrap script that we inherit from gnulib.  If it turns out to be
inconvenient or confusing, we may attempt to make updates happen
automatically when the requested revision is not present.

You may also continue to use an external copy of gnulib by using the
--gnulib-srcdir option:

   ./bootstrap --gnulib-srcdir=/path/to/gnulib

See also the following bug report:

   https://savannah.gnu.org/bugs/?57044

jwe

Reply | Threaded
Open this post in threaded view
|

Re: gnulib is no longer an hg subrepo

mmuetzel
Am 24. Januar 2020 um 15:44 Uhr schrieb "John W. Eaton":
> After this change (on stable, merged with default):
>
>    http://hg.savannah.gnu.org/hgweb/octave/rev/ece72b94486f
>
> the gnulib sources are no longer managed within the Octave mercurial
> archive as an hg subrepo.  Instead, we now have the bootstrap script
> fetching a copy of gnulib using git.

Thank you very much for that change. That will probably make updating gnulib a lot easier in the future.

> After updating the Octave sources to get this change you will need to
> remove the gnulib subdirectory in your source tree and run bootstrap to
> pull a new copy of gnulib from upstream (commands executed at the
> top-level of the Octave source tree):
>
>    hg pull -u
>    rm -rf gnulib
>    ./bootstrap

The MXE Octave buildbots (native and Windows cross-builds) seem to be failing since that change.
Maybe the octave-hg-repo (or at least the gnulib subdirectory) needs to be cleared once on the workers, too?

Markus

Reply | Threaded
Open this post in threaded view
|

Re: gnulib is no longer an hg subrepo

John W. Eaton
Administrator
On 1/26/20 6:34 AM, "Markus Mützel" wrote:

> Am 24. Januar 2020 um 15:44 Uhr schrieb "John W. Eaton":
>> After this change (on stable, merged with default):
>>
>>     http://hg.savannah.gnu.org/hgweb/octave/rev/ece72b94486f
>>
>> the gnulib sources are no longer managed within the Octave mercurial
>> archive as an hg subrepo.  Instead, we now have the bootstrap script
>> fetching a copy of gnulib using git.
>
> Thank you very much for that change. That will probably make updating gnulib a lot easier in the future.
>
>> After updating the Octave sources to get this change you will need to
>> remove the gnulib subdirectory in your source tree and run bootstrap to
>> pull a new copy of gnulib from upstream (commands executed at the
>> top-level of the Octave source tree):
>>
>>     hg pull -u
>>     rm -rf gnulib
>>     ./bootstrap
>
> The MXE Octave buildbots (native and Windows cross-builds) seem to be failing since that change.
> Maybe the octave-hg-repo (or at least the gnulib subdirectory) needs to be cleared once on the workers, too?

I removed the gnulib directories from the source trees on the buildbot
systems that I manage.  I think they should work now.  I also restarted
the buildbot worker process on three of the four systems I manage.  I
think they were disabled after recent upgrades.  Things should be back
to normal soon.

To make updating gnulib easier for us, I think we will need to come up
with some solution in the bootstrap script so that it will automatically
fetch new changes for gnulib.  Otherwise, we will have to manually
update the gnulib sources when we change GNULIB_REVISION in
bootstrap.conf.  This will require some coordination with the gnulib
folks.  I looked at it but couldn't decide the right thing to do.  I
might be wrong, but it looks like the update will already happen if
gnulib is a git subrepo.  Otherwise, fetching new changes doesn't
happen.  Should that always happen, even if the GNULIB_SRCDIR is set?
Should a failed fetch be fatal by default?  Maybe you are not on the
network or don't have permission to write to a shared gnulib source
tree.  If it is fatal by default, then there should probably be an
option to allow skipping that step.  Too many twists and turns.  I gave
up.  Would someone like to take up this cause and try to make bootstrap
do the right thing for us?

jwe

Reply | Threaded
Open this post in threaded view
|

Re: gnulib is no longer an hg subrepo

mmuetzel
Am 26. Januar 2020 um 18:05 Uhr schrieb "John W. Eaton":
> I removed the gnulib directories from the source trees on the buildbot
> systems that I manage.  I think they should work now.  I also restarted
> the buildbot worker process on three of the four systems I manage.  I
> think they were disabled after recent upgrades.  Things should be back
> to normal soon.

Thanks!

> To make updating gnulib easier for us, I think we will need to come up
> with some solution in the bootstrap script so that it will automatically
> fetch new changes for gnulib.  Otherwise, we will have to manually
> update the gnulib sources when we change GNULIB_REVISION in
> bootstrap.conf.  This will require some coordination with the gnulib
> folks.  I looked at it but couldn't decide the right thing to do.  I
> might be wrong, but it looks like the update will already happen if
> gnulib is a git subrepo.  Otherwise, fetching new changes doesn't
> happen.  Should that always happen, even if the GNULIB_SRCDIR is set?
> Should a failed fetch be fatal by default?  Maybe you are not on the
> network or don't have permission to write to a shared gnulib source
> tree.  If it is fatal by default, then there should probably be an
> option to allow skipping that step.  Too many twists and turns.  I gave
> up.  Would someone like to take up this cause and try to make bootstrap
> do the right thing for us?

Would this change help?

# HG changeset patch
# User Markus Mützel <[hidden email]>
# Date 1580064128 -3600
#      Sun Jan 26 19:42:08 2020 +0100
# Node ID b08a376fb1c4cd1af470dcce467a7276813da234
# Parent  e6482c932d4bc097050f5fb2e4e5ffd98a5581df
bootstrap: Pull from remote git if $GNULIB_REVISION doesn't exist in local git.

diff -r e6482c932d4b -r b08a376fb1c4 bootstrap
--- a/bootstrap Sun Jan 26 17:00:05 2020 +0100
+++ b/bootstrap Sun Jan 26 19:42:08 2020 +0100
@@ -670,6 +670,9 @@
         || cleanup_gnulib
 
       trap - 1 2 13 15
+
+    elif ! git --git-dir="$gnulib_path"/.git cat-file commit "$GNULIB_REVISION"; then
+      git --git-dir="$gnulib_path"/.git pull
     fi
     GNULIB_SRCDIR=$gnulib_path
     ;;



This only executes if the package is not in a GIT repository (which is the case for Octave) and the "gnulib" sub-directory already exists.
It assumes that $gnulib_path (i.e. the "gnulib" sub-directory) is a GIT repository and pulls from remote if the specific commit cannot be found in the local repository.
This only happens if GNULIB_SRCDIR is not previously set.

Is this what we want?

Markus

Reply | Threaded
Open this post in threaded view
|

Re: gnulib is no longer an hg subrepo

mmuetzel
Am 26. Januar 2020 um 19:52 Uhr schrieb "Markus Mützel":

> > To make updating gnulib easier for us, I think we will need to come up
> > with some solution in the bootstrap script so that it will automatically
> > fetch new changes for gnulib.  Otherwise, we will have to manually
> > update the gnulib sources when we change GNULIB_REVISION in
> > bootstrap.conf.  This will require some coordination with the gnulib
> > folks.  I looked at it but couldn't decide the right thing to do.  I
> > might be wrong, but it looks like the update will already happen if
> > gnulib is a git subrepo.  Otherwise, fetching new changes doesn't
> > happen.  Should that always happen, even if the GNULIB_SRCDIR is set?
> > Should a failed fetch be fatal by default?  Maybe you are not on the
> > network or don't have permission to write to a shared gnulib source
> > tree.  If it is fatal by default, then there should probably be an
> > option to allow skipping that step.  Too many twists and turns.  I gave
> > up.  Would someone like to take up this cause and try to make bootstrap
> > do the right thing for us?
>
> Would this change help?
>
> # HG changeset patch
> # User Markus Mützel <[hidden email]>
> # Date 1580064128 -3600
> #      Sun Jan 26 19:42:08 2020 +0100
> # Node ID b08a376fb1c4cd1af470dcce467a7276813da234
> # Parent  e6482c932d4bc097050f5fb2e4e5ffd98a5581df
> bootstrap: Pull from remote git if $GNULIB_REVISION doesn't exist in local git.
>
> diff -r e6482c932d4b -r b08a376fb1c4 bootstrap
> --- a/bootstrap Sun Jan 26 17:00:05 2020 +0100
> +++ b/bootstrap Sun Jan 26 19:42:08 2020 +0100
> @@ -670,6 +670,9 @@
>          || cleanup_gnulib
>  
>        trap - 1 2 13 15
> +
> +    elif ! git --git-dir="$gnulib_path"/.git cat-file commit "$GNULIB_REVISION"; then
> +      git --git-dir="$gnulib_path"/.git pull
>      fi
>      GNULIB_SRCDIR=$gnulib_path
>      ;;
>
>
>
> This only executes if the package is not in a GIT repository (which is the case for Octave) and the "gnulib" sub-directory already exists.
> It assumes that $gnulib_path (i.e. the "gnulib" sub-directory) is a GIT repository and pulls from remote if the specific commit cannot be found in the local repository.
> This only happens if GNULIB_SRCDIR is not previously set.
>

A slight modification to skip the test if GNULIB_REVISION is set empty:
diff -r e6482c932d4b bootstrap
--- a/bootstrap Sun Jan 26 17:00:05 2020 +0100
+++ b/bootstrap Sun Jan 26 20:18:45 2020 +0100
@@ -670,6 +670,11 @@
         || cleanup_gnulib
 
       trap - 1 2 13 15
+
+    elif test -n "$GNULIB_REVISION" \
+         && ! git --git-dir="$gnulib_path"/.git cat-file \
+              commit "$GNULIB_REVISION"; then
+      git --git-dir="$gnulib_path"/.git pull
     fi
     GNULIB_SRCDIR=$gnulib_path
     ;;


That would allow to skip the update by calling:
GNULIB_REVISION= ./bootstrap

Markus

Reply | Threaded
Open this post in threaded view
|

Re: gnulib is no longer an hg subrepo

mmuetzel
In reply to this post by John W. Eaton
Am 26. Januar 2020 um 18:05 Uhr schrieb "John W. Eaton":
> I removed the gnulib directories from the source trees on the buildbot
> systems that I manage.  I think they should work now.  I also restarted
> the buildbot worker process on three of the four systems I manage.  I
> think they were disabled after recent upgrades.  Things should be back
> to normal soon.

The "mxe-native-on-debian" builder on "jwe-debian-x86_64-1" still seems to have an issue.

Markus

Reply | Threaded
Open this post in threaded view
|

Re: gnulib is no longer an hg subrepo

John W. Eaton
Administrator
On 1/26/20 2:28 PM, "Markus Mützel" wrote:

> The "mxe-native-on-debian" builder on "jwe-debian-x86_64-1" still seems to have an issue.

Thanks.  I thought I had deleted all the old gnulib directories but I
must have missed a few.  I think it is fixed now.  Let me know if you
spot any other problems.

jwe



Reply | Threaded
Open this post in threaded view
|

Re: gnulib is no longer an hg subrepo

John W. Eaton
Administrator
In reply to this post by mmuetzel
On 1/26/20 2:23 PM, "Markus Mützel" wrote:
> Am 26. Januar 2020 um 19:52 Uhr schrieb "Markus Mützel":

>> Would this change help?
>>
>> # HG changeset patch
>> # User Markus Mützel <[hidden email]>
>> # Date 1580064128 -3600
>> #      Sun Jan 26 19:42:08 2020 +0100
>> # Node ID b08a376fb1c4cd1af470dcce467a7276813da234
>> # Parent  e6482c932d4bc097050f5fb2e4e5ffd98a5581df
>> bootstrap: Pull from remote git if $GNULIB_REVISION doesn't exist in local git.
>>
>> diff -r e6482c932d4b -r b08a376fb1c4 bootstrap
>> --- a/bootstrap Sun Jan 26 17:00:05 2020 +0100
>> +++ b/bootstrap Sun Jan 26 19:42:08 2020 +0100
>> @@ -670,6 +670,9 @@
>>           || cleanup_gnulib
>>  
>>         trap - 1 2 13 15
>> +
>> +    elif ! git --git-dir="$gnulib_path"/.git cat-file commit "$GNULIB_REVISION"; then
>> +      git --git-dir="$gnulib_path"/.git pull
>>       fi
>>       GNULIB_SRCDIR=$gnulib_path
>>       ;;
>>
>>
>>
>> This only executes if the package is not in a GIT repository (which is the case for Octave) and the "gnulib" sub-directory already exists.
>> It assumes that $gnulib_path (i.e. the "gnulib" sub-directory) is a GIT repository and pulls from remote if the specific commit cannot be found in the local repository.
>> This only happens if GNULIB_SRCDIR is not previously set.
>>
>
> A slight modification to skip the test if GNULIB_REVISION is set empty:
> diff -r e6482c932d4b bootstrap
> --- a/bootstrap Sun Jan 26 17:00:05 2020 +0100
> +++ b/bootstrap Sun Jan 26 20:18:45 2020 +0100
> @@ -670,6 +670,11 @@
>           || cleanup_gnulib
>  
>         trap - 1 2 13 15
> +
> +    elif test -n "$GNULIB_REVISION" \
> +         && ! git --git-dir="$gnulib_path"/.git cat-file \
> +              commit "$GNULIB_REVISION"; then
> +      git --git-dir="$gnulib_path"/.git pull
>       fi
>       GNULIB_SRCDIR=$gnulib_path
>       ;;
>
>
> That would allow to skip the update by calling:
> GNULIB_REVISION= ./bootstrap

That seems good to me.  We could maintain our own patched version of the
bootstrap script but it seems to me that it would be better to have
something like this added to the version that is provided in gnulib.

jwe



Reply | Threaded
Open this post in threaded view
|

Re: gnulib is no longer an hg subrepo

mmuetzel
Am 27. Januar 2020 um 07:41 Uhr schrieb "John W. Eaton":

> On 1/26/20 2:23 PM, "Markus Mützel" wrote:
> > Am 26. Januar 2020 um 19:52 Uhr schrieb "Markus Mützel":
>
> >> Would this change help?
> >>
> >> # HG changeset patch
> >> # User Markus Mützel <[hidden email]>
> >> # Date 1580064128 -3600
> >> #      Sun Jan 26 19:42:08 2020 +0100
> >> # Node ID b08a376fb1c4cd1af470dcce467a7276813da234
> >> # Parent  e6482c932d4bc097050f5fb2e4e5ffd98a5581df
> >> bootstrap: Pull from remote git if $GNULIB_REVISION doesn't exist in local git.
> >>
> >> diff -r e6482c932d4b -r b08a376fb1c4 bootstrap
> >> --- a/bootstrap Sun Jan 26 17:00:05 2020 +0100
> >> +++ b/bootstrap Sun Jan 26 19:42:08 2020 +0100
> >> @@ -670,6 +670,9 @@
> >>           || cleanup_gnulib
> >>  
> >>         trap - 1 2 13 15
> >> +
> >> +    elif ! git --git-dir="$gnulib_path"/.git cat-file commit "$GNULIB_REVISION"; then
> >> +      git --git-dir="$gnulib_path"/.git pull
> >>       fi
> >>       GNULIB_SRCDIR=$gnulib_path
> >>       ;;
> >>
> >>
> >>
> >> This only executes if the package is not in a GIT repository (which is the case for Octave) and the "gnulib" sub-directory already exists.
> >> It assumes that $gnulib_path (i.e. the "gnulib" sub-directory) is a GIT repository and pulls from remote if the specific commit cannot be found in the local repository.
> >> This only happens if GNULIB_SRCDIR is not previously set.
> >>
> >
> > A slight modification to skip the test if GNULIB_REVISION is set empty:
> > diff -r e6482c932d4b bootstrap
> > --- a/bootstrap Sun Jan 26 17:00:05 2020 +0100
> > +++ b/bootstrap Sun Jan 26 20:18:45 2020 +0100
> > @@ -670,6 +670,11 @@
> >           || cleanup_gnulib
> >  
> >         trap - 1 2 13 15
> > +
> > +    elif test -n "$GNULIB_REVISION" \
> > +         && ! git --git-dir="$gnulib_path"/.git cat-file \
> > +              commit "$GNULIB_REVISION"; then
> > +      git --git-dir="$gnulib_path"/.git pull
> >       fi
> >       GNULIB_SRCDIR=$gnulib_path
> >       ;;
> >
> >
> > That would allow to skip the update by calling:
> > GNULIB_REVISION= ./bootstrap
>
> That seems good to me.  We could maintain our own patched version of the
> bootstrap script but it seems to me that it would be better to have
> something like this added to the version that is provided in gnulib.

Maybe we could patch our own version and see whether this works as expected across different systems, version of git, etc.
When (or if) we are confident that this is a good addition, we could donate the change upstream.

Markus


Reply | Threaded
Open this post in threaded view
|

Re: gnulib is no longer an hg subrepo

John W. Eaton
Administrator
On 1/27/20 3:29 AM, "Markus Mützel" wrote:
> Am 27. Januar 2020 um 07:41 Uhr schrieb "John W. Eaton":

>> That seems good to me.  We could maintain our own patched version of the
>> bootstrap script but it seems to me that it would be better to have
>> something like this added to the version that is provided in gnulib.
>
> Maybe we could patch our own version and see whether this works as expected across different systems, version of git, etc.
> When (or if) we are confident that this is a good addition, we could donate the change upstream.

OK, I pushed your changes on stable and merged with default.  Then I
updated gnulib on default (not sure whether we want to do that on stable
just before the release?) and then ran bootstrap --bootstrap-sync
followed by reapplying your changes to preserve them after the updates
to the bootstrap script from gnulib.  Updating the bootstrap script from
the gnulib version is now more difficult because we have to remember to
reapply our local changes.  Thankfully, the gnulib bootstrap script
doesn't appear to be changing much now.  But it would still be good to
get this modification accepted in gnulib if it works out for us.  Let's
see what happens with the buildbots when they get these changes.

jwe



Reply | Threaded
Open this post in threaded view
|

Re: gnulib is no longer an hg subrepo

mmuetzel
Am 27. Januar 2020 um 17:46 Uhr schrieb "John W. Eaton":

> On 1/27/20 3:29 AM, "Markus Mützel" wrote:
> > Am 27. Januar 2020 um 07:41 Uhr schrieb "John W. Eaton":
>
> >> That seems good to me.  We could maintain our own patched version of the
> >> bootstrap script but it seems to me that it would be better to have
> >> something like this added to the version that is provided in gnulib.
> >
> > Maybe we could patch our own version and see whether this works as expected across different systems, version of git, etc.
> > When (or if) we are confident that this is a good addition, we could donate the change upstream.
>
> OK, I pushed your changes on stable and merged with default.  Then I
> updated gnulib on default (not sure whether we want to do that on stable
> just before the release?) and then ran bootstrap --bootstrap-sync
> followed by reapplying your changes to preserve them after the updates
> to the bootstrap script from gnulib.  Updating the bootstrap script from
> the gnulib version is now more difficult because we have to remember to
> reapply our local changes.  Thankfully, the gnulib bootstrap script
> doesn't appear to be changing much now.  But it would still be good to
> get this modification accepted in gnulib if it works out for us.  Let's
> see what happens with the buildbots when they get these changes.

We are using these changes since a couple of months now. So far no one was complaining about them.
Should we submit it upstream now (together with Mike's follow up fix)? Or better wait until we actually moved to a newer gnulib revision and see if the logic does what we expect on different systems?

In any case, I am not sure where this should be submitted. Does gnulib have a tracker?

Markus

Reply | Threaded
Open this post in threaded view
|

Re: gnulib is no longer an hg subrepo

siko1056
On 3/31/20 10:47 PM, "Markus Mützel" wrote:

> Am 27. Januar 2020 um 17:46 Uhr schrieb "John W. Eaton":
>> On 1/27/20 3:29 AM, "Markus Mützel" wrote:
>>> Am 27. Januar 2020 um 07:41 Uhr schrieb "John W. Eaton":
>>
>>>> That seems good to me.  We could maintain our own patched version of the
>>>> bootstrap script but it seems to me that it would be better to have
>>>> something like this added to the version that is provided in gnulib.
>>>
>>> Maybe we could patch our own version and see whether this works as expected across different systems, version of git, etc.
>>> When (or if) we are confident that this is a good addition, we could donate the change upstream.
>>
>> OK, I pushed your changes on stable and merged with default.  Then I
>> updated gnulib on default (not sure whether we want to do that on stable
>> just before the release?) and then ran bootstrap --bootstrap-sync
>> followed by reapplying your changes to preserve them after the updates
>> to the bootstrap script from gnulib.  Updating the bootstrap script from
>> the gnulib version is now more difficult because we have to remember to
>> reapply our local changes.  Thankfully, the gnulib bootstrap script
>> doesn't appear to be changing much now.  But it would still be good to
>> get this modification accepted in gnulib if it works out for us.  Let's
>> see what happens with the buildbots when they get these changes.
>
> We are using these changes since a couple of months now. So far no one was complaining about them.
> Should we submit it upstream now (together with Mike's follow up fix)? Or better wait until we actually moved to a newer gnulib revision and see if the logic does what we expect on different systems?
>
> In any case, I am not sure where this should be submitted. Does gnulib have a tracker?
>
> Markus
>


Since that change I am really happy about Octave's repository, which is
much more manageable with local clones now.

According to the gnulib website [1], bugs should be reported to their
mailing list [2].

Kai

[1] https://www.gnu.org/software/gnulib/
[2] [hidden email]