octave.mk MXE file still necessary?

[Rik-4]

All,

I just checked the recipes for making in Octave in the MXE src/ directory
and I find stable-octave.mk (4.4.0), default-octave.mk (5.0.0), and
octave.mk (4.2.0-rc4).  Do we need the last recipe?  I think it is
confusing because without any other information I would think to build just
octave.mk, and that would get me a very out-of-date version.

--Rik

[John W. Eaton]

On 06/21/2018 12:06 PM, Rik wrote:
> All,
>
> I just checked the recipes for making in Octave in the MXE src/ directory
> and I find stable-octave.mk (4.4.0), default-octave.mk (5.0.0), and
> octave.mk (4.2.0-rc4).  Do we need the last recipe?  I think it is
> confusing because without any other information I would think to build just
> octave.mk, and that would get me a very out-of-date version.

I agree that it's a bit of a mess right now.

I'm not sure how best to manage this or what the names of the targets
should be, but my goals would be to allow the following four configurations:

   * building from the current default branch in the hg archive.  As we
do now, it's fine with me if this requires creating a tarball from the
current hg sources separately from mxe-octave.  This configuration
allows testing for the next major release.

   * building from the current stable branch in the hg archive.  This
can work the same way as for default, but just using the stable branch.
This configuration allows testing for the next bug-fixing stable release.

   * A way to build the last stable released version of Octave from
ftp.gnu.org but with the latest mxe-octave tools and dependencies.  This
configuration allows us to check whether the last stable release can
still be built with new versions of tools and dependencies.

   * A way to build the last stable released version of Octave from
ftp.gnu.org with the same versions of mxe-octave tools and dependency
libraries.  This configuration allows anyone to reproduce the builds
that we distribute.

   * A way to build the last stable released version of Octave from
ftp.gnu.org but with the latest mxe-octave tools and dependencies.  This
configuration allows us to check whether the last stable release can
still be built with new versions of tools and dependencies.


At least the first two configurations should be built and tested often
by the buildbots.  The others are less important but seem useful to have.

jwe

[Rik-4]

On 06/21/2018 10:35 AM, John W. Eaton wrote:

> On 06/21/2018 12:06 PM, Rik wrote:
>> All,
>>
>> I just checked the recipes for making in Octave in the MXE src/ directory
>> and I find stable-octave.mk (4.4.0), default-octave.mk (5.0.0), and
>> octave.mk (4.2.0-rc4).  Do we need the last recipe?  I think it is
>> confusing because without any other information I would think to build just
>> octave.mk, and that would get me a very out-of-date version.
>
> I agree that it's a bit of a mess right now.
>
> I'm not sure how best to manage this or what the names of the targets
> should be, but my goals would be to allow the following four configurations:
>
>   * building from the current default branch in the hg archive.  As we do
> now, it's fine with me if this requires creating a tarball from the
> current hg sources separately from mxe-octave.  This configuration allows
> testing for the next major release.
>
>   * building from the current stable branch in the hg archive.  This can
> work the same way as for default, but just using the stable branch. This
> configuration allows testing for the next bug-fixing stable release.
>
>   * A way to build the last stable released version of Octave from
> ftp.gnu.org but with the latest mxe-octave tools and dependencies.  This
> configuration allows us to check whether the last stable release can
> still be built with new versions of tools and dependencies.
>
>   * A way to build the last stable released version of Octave from
> ftp.gnu.org with the same versions of mxe-octave tools and dependency
> libraries.  This configuration allows anyone to reproduce the builds that
> we distribute.
>
>   * A way to build the last stable released version of Octave from
> ftp.gnu.org but with the latest mxe-octave tools and dependencies.  This
> configuration allows us to check whether the last stable release can
> still be built with new versions of tools and dependencies.
>
>
> At least the first two configurations should be built and tested often by
> the buildbots.  The others are less important but seem useful to have.
>
> jwe
>
>

Agree with all of this.  To avoid confusion, I deleted the obsolete recipe
octave.mk.  If we want specific recipes for some of these scenarios they
can be written with appropriately descriptive names.

--Rik

[mmuetzel]

Rik wrote:
> Agree with all of this.  To avoid confusion, I deleted the obsolete recipe
> octave.mk.
 
With that file deleted, I can no longer cross-build Octave. Build bots show the same error:
http://buildbot.octave.org:8011/#/builders/18/builds/49/steps/2/logs/stdio
 
Markus

[John W. Eaton]

On 06/26/2018 09:31 AM, "Markus Mützel" wrote:
> Rik wrote:
>> Agree with all of this.  To avoid confusion, I deleted the obsolete recipe
>> octave.mk.
>  
> With that file deleted, I can no longer cross-build Octave. Build bots show the same error:
> http://buildbot.octave.org:8011/#/builders/18/builds/49/steps/2/logs/stdio

Oops.  I backed it out.

jwe

[Rik-4]

On 06/26/2018 06:40 AM, John W. Eaton wrote:

> On 06/26/2018 09:31 AM, "Markus Mützel" wrote:
>> Rik wrote:
>>> Agree with all of this.  To avoid confusion, I deleted the obsolete recipe
>>> octave.mk.
>>   With that file deleted, I can no longer cross-build Octave. Build bots
>> show the same error:
>> http://buildbot.octave.org:8011/#/builders/18/builds/49/steps/2/logs/stdio
>
> Oops.  I backed it out.
>

I can't debug this.  The URL is empty for me, and the buildbot URL I know
is http://buildbot.octave.org:8010/waterfall.

Using 'grep -r octave.mk *' in the mxe directory I see that configure.ac,
index.html, and dist-files.mk mention that file.  It is easy enough to
remove octave.mk from dist-files.mk and index.html, and to update
configure.ac to use default-octave.mk.

I ran 'make clean' and 'autoconf' and am now testing whether I can do a
full build.

--Rik

> jwe
>
>
>
>

[mmuetzel]

Rik wrote:
> >> http://buildbot.octave.org:8011/#/builders/18/builds/49/steps/2/logs/stdio
> >
> > Oops.  I backed it out.
> >
>
> I can't debug this.  The URL is empty for me, and the buildbot URL I know
> is http://buildbot.octave.org:8010/waterfall[http://buildbot.octave.org:8010/waterfall][http://buildbot.octave.org:8010/waterfall[http://buildbot.octave.org:8010/waterfall]].

The link should open a page with the stdio of the configure step of the w64-64-on-debian builder. That is the following for me:

nice -n 19 ./configure --with-pkg-dir=../../mxe-pkg-src --with-ccache --enable-octave=default --enable-binary-packages --enable-qt5 --enable-devel-tools --enable-fortran-int64 --disable-system-opengl --enable-64

in dir /scratch/buildbot/workers/jwe-debian-x86_64-0/w64-64-on-debian/src (timeout 1200 secs)

watching logfiles {}

argv: [b'nice', b'-n', b'19', b'./configure', b'--with-pkg-dir=../../mxe-pkg-src', b'--with-ccache', b'--enable-octave=default', b'--enable-binary-packages', b'--enable-qt5', b'--enable-devel-tools', b'--enable-fortran-int64', b'--disable-system-opengl', b'--enable-64']

environment:

DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1004/bus

HOME=/var/lib/buildbot

LANG=en_US.UTF-8

LOGNAME=buildbot

MAIL=/var/mail/buildbot

PATH=/var/lib/buildbot/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games

PWD=/scratch/buildbot/workers/jwe-debian-x86_64-0/w64-64-on-debian/src

SHELL=/bin/sh

USER=buildbot

XDG_RUNTIME_DIR=/run/user/1004

XDG_SESSION_ID=c1

using PTY: False

configure: error: cannot find sources (src/octave.mk) in . or ..

program finished with exit code 1

elapsedTime=0.183402

 

If I understood correctly, on port 8010 there is an older version of buildbot. Most of the "workers" have been updated to buildbot 1.1 which is incompatible with buildbot 0.9.

 

Markus

 

[John W. Eaton]

On 06/26/2018 12:08 PM, Rik wrote:

> I can't debug this.  The URL is empty for me, and the buildbot URL I know
> is http://buildbot.octave.org:8010/waterfall.

For now, the new buildbot is at buildbot.octave.org:8011.  We still have
a few workers that haven't been updated.  Once that happens I'll
probably disable the older master and move the new one to respond at
port 8010.

jwe

[Rik-4]

On 06/26/2018 09:34 AM, John W. Eaton wrote:
> On 06/26/2018 12:08 PM, Rik wrote:
>
>> I can't debug this.  The URL is empty for me, and the buildbot URL I know
>> is http://buildbot.octave.org:8010/waterfall.
>
> For now, the new buildbot is at buildbot.octave.org:8011.  We still have
> a few workers that haven't been updated.  Once that happens I'll probably
> disable the older master and move the new one to respond at port 8010.

I feel lame, but I don't seem to be able to access any useful information
from the new buildbots.  This URL
(http://buildbot.octave.org:8011/#/waterfall) does retrieve a page with
build bars.  But when I click on a build I get an empty Build Summary
pop-up window.  The "Expand All Steps" button doesn't help either.  Is
there some minimum version of Chromium or Firefox required?

--Rik

[John W. Eaton]

On 06/26/2018 01:03 PM, Rik wrote:

> On 06/26/2018 09:34 AM, John W. Eaton wrote:
>> On 06/26/2018 12:08 PM, Rik wrote:
>>
>>> I can't debug this.  The URL is empty for me, and the buildbot URL I know
>>> is http://buildbot.octave.org:8010/waterfall.
>>
>> For now, the new buildbot is at buildbot.octave.org:8011.  We still have
>> a few workers that haven't been updated.  Once that happens I'll probably
>> disable the older master and move the new one to respond at port 8010.
>
> I feel lame, but I don't seem to be able to access any useful information
> from the new buildbots.  This URL
> (http://buildbot.octave.org:8011/#/waterfall) does retrieve a page with
> build bars.  But when I click on a build I get an empty Build Summary
> pop-up window.  The "Expand All Steps" button doesn't help either.  Is
> there some minimum version of Chromium or Firefox required?

I'm not sure.  If I click on a build (the text above one of the
waterfall display columns) I go to a URL like this:

   http://buildbot.octave.org:8011/#/builders/1

Does that work if you go directly to that URL?

jwe

[John W. Eaton]

On 06/26/2018 12:08 PM, Rik wrote:

> Using 'grep -r octave.mk *' in the mxe directory I see that configure.ac,
> index.html, and dist-files.mk mention that file.  It is easy enough to
> remove octave.mk from dist-files.mk and index.html, and to update
> configure.ac to use default-octave.mk.
>
> I ran 'make clean' and 'autoconf' and am now testing whether I can do a
> full build.

The mxe-octave makefile is really complicated and does lots of
transformations on variables so unfortunately grep won't find
everything.  And then there are some conventions that are not
immediately obvious.

I checked in the following change to remove the "octave" package:

   http://hg.octave.org/mxe-octave/rev/9942a9c37ffe

If I remember correctly, later versions of mxe (not mxe-octave) don't
rely on the index.html file to indicate which packages (and src/PKG.mk
files) are part of the collection.

jwe

[Rik-4]

On 06/26/2018 10:34 AM, John W. Eaton wrote:

> On 06/26/2018 12:08 PM, Rik wrote:
>
>> Using 'grep -r octave.mk *' in the mxe directory I see that configure.ac,
>> index.html, and dist-files.mk mention that file.  It is easy enough to
>> remove octave.mk from dist-files.mk and index.html, and to update
>> configure.ac to use default-octave.mk.
>>
>> I ran 'make clean' and 'autoconf' and am now testing whether I can do a
>> full build.
>
> The mxe-octave makefile is really complicated and does lots of
> transformations on variables so unfortunately grep won't find
> everything.  And then there are some conventions that are not immediately
> obvious.
>
> I checked in the following change to remove the "octave" package:
>
>   http://hg.octave.org/mxe-octave/rev/9942a9c37ffe
>
> If I remember correctly, later versions of mxe (not mxe-octave) don't
> rely on the index.html file to indicate which packages (and src/PKG.mk
> files) are part of the collection.

This looks right now.  I'm building, but it will be a while before I know
for sure.

--Rik

[John W. Eaton]

On 06/25/2018 05:02 PM, Rik wrote:

> On 06/21/2018 10:35 AM, John W. Eaton wrote:
>> On 06/21/2018 12:06 PM, Rik wrote:
>>> All,
>>>
>>> I just checked the recipes for making in Octave in the MXE src/ directory
>>> and I find stable-octave.mk (4.4.0), default-octave.mk (5.0.0), and
>>> octave.mk (4.2.0-rc4).  Do we need the last recipe?  I think it is
>>> confusing because without any other information I would think to build just
>>> octave.mk, and that would get me a very out-of-date version.
>>
>> I agree that it's a bit of a mess right now.
>>
>> I'm not sure how best to manage this or what the names of the targets
>> should be, but my goals would be to allow the following four configurations:
>>
>>    * building from the current default branch in the hg archive.  As we do
>> now, it's fine with me if this requires creating a tarball from the
>> current hg sources separately from mxe-octave.  This configuration allows
>> testing for the next major release.
>>
>>    * building from the current stable branch in the hg archive.  This can
>> work the same way as for default, but just using the stable branch. This
>> configuration allows testing for the next bug-fixing stable release.
>>
>>    * A way to build the last stable released version of Octave from
>> ftp.gnu.org but with the latest mxe-octave tools and dependencies.  This
>> configuration allows us to check whether the last stable release can
>> still be built with new versions of tools and dependencies.
>>
>>    * A way to build the last stable released version of Octave from
>> ftp.gnu.org with the same versions of mxe-octave tools and dependency
>> libraries.  This configuration allows anyone to reproduce the builds that
>> we distribute.
>>
>>    * A way to build the last stable released version of Octave from
>> ftp.gnu.org but with the latest mxe-octave tools and dependencies.  This
>> configuration allows us to check whether the last stable release can
>> still be built with new versions of tools and dependencies.
>>
>>
>> At least the first two configurations should be built and tested often by
>> the buildbots.  The others are less important but seem useful to have.
>>
>> jwe
>>
>>
>
> Agree with all of this.  To avoid confusion, I deleted the obsolete recipe
> octave.mk.  If we want specific recipes for some of these scenarios they
> can be written with appropriately descriptive names.

I decided on the following naming scheme:

   default:  current sources from the default development branch

   stable:  current sources from the stable development branch

   release:  current released version

For stable and default, you must build a tar.lz file separately and
place it where mxe-octave looks for package sources.  There is no
checksum in the mxe-octave makefile fragment for these, and the package
URL is set to an invalid location so it will take whatever tar.lz file
you have in the package source directory as long as it has the expected
version number (currently 5.0.0 for default and 4.0.0+ for stable).  The
package URL is also invalid.

For the "release-octave" package there is checksum and valid URL.

Note that the release-octave package builds the released sources with
the latest mxe-octave packages so it does not duplicate the binary
builds that we distribute.  The comments in the release-octave.mk file
explain how to do that:

## To reproduce the binary
## builds for Windows that are distributed on ftp.gnu.org, you must
## choose the revision of mxe-octave that matches the release you
## wish to reproduce.  Those should be tagged in the mxe-octave archive
## with tags like "octave-release-4.4.0".  The options used to build are
##
##   --enable-octave=release
##   --enable-binary-packages
##   --enable-devel-tools
##   --enable-qt5
##   --disable-system-opengl
##
## and one of the following
##
##   * 64-bit Windows build; 32-bit integers for Fortran
##     (including BLAS and LAPACK libraries) which is the typical
##     configuration for all Linux distributions:
##
##       --enable-windows-64 --enable-64 --disable-fortran-int64
##
##   * 64-bit Windows build; 64-bit integers for Fortran
##     (including BLAS and LAPACK libraries):
##
##       --enable-windows-64 --enable-64 --enable-fortran-int64
##
##   * 32-bit Windows build:
##
##       --disable-windows-64 --disable-64 --disable-fortran-int64

I've also updated the buildbots to build all nine of the possible
combinations: (default, stable, release) X (32-bit Windows, 64-bit
windows with 32-bit Fortran integers, and 64-bit Windows with 64-bit
Fortran integers).

jwe