GSoC project about binary packaging

classic Classic list List threaded Threaded
46 messages Options
123
Reply | Threaded
Open this post in threaded view
|

GSoC project about binary packaging

Michael Goffioul
Hi all,

I've a student who's supposed to work on improving/completing binary packaging of octave on Windows and OS X during the next 2 months. However, I can see that there are already 3 to 4 people actively working on MXE. MXE/octave seems to already work for MinGW cross and native, as well as native Linux. So I'm a bit concerned that there won't be much left to do for the student.

So I'd like you to help define clear goals for this project in order to avoid stepping on each other's toes. For instance, I had listed adding support for OpenBLAS as a possible goal for him, but I saw today it had been added to MXE yesterday.

Basically, what's left at the moment?
- writing a NSI installer that's a bit smarter than "dump everything"; it should also support smart reinstallation and package addition (as in: re-run the installer to install additional components, which were not selected previously)
- missing octave-forge packages + dependencies
- OS X
(feel free the extend the list)

For OS X, I'd like some input from OS X users to assess what's doable. For instance, is it possible to integrate installer building inside MXE (for instance using some native OS X compilation)? I also remember a recent mail from Ben mentioning some new efforts in MacPorts to get octave compiled on OS X.

At the end of the day, I hope there will be enough work to be done to make a GSoC project.

Thanks,
Michael.

Reply | Threaded
Open this post in threaded view
|

Re: GSoC project about binary packaging

Lukas Reichlin-4
On 21.06.2013, at 20:44, Michael Goffioul <[hidden email]> wrote:

> Hi all,
>
> I've a student who's supposed to work on improving/completing binary packaging of octave on Windows and OS X during the next 2 months. However, I can see that there are already 3 to 4 people actively working on MXE. MXE/octave seems to already work for MinGW cross and native, as well as native Linux. So I'm a bit concerned that there won't be much left to do for the student.
>
> So I'd like you to help define clear goals for this project in order to avoid stepping on each other's toes. For instance, I had listed adding support for OpenBLAS as a possible goal for him, but I saw today it had been added to MXE yesterday.
>
> Basically, what's left at the moment?
> - writing a NSI installer that's a bit smarter than "dump everything"; it should also support smart reinstallation and package addition (as in: re-run the installer to install additional components, which were not selected previously)
> - missing octave-forge packages + dependencies
> - OS X
> (feel free the extend the list)
>
> For OS X, I'd like some input from OS X users to assess what's doable. For instance, is it possible to integrate installer building inside MXE (for instance using some native OS X compilation)? I also remember a recent mail from Ben mentioning some new efforts in MacPorts to get octave compiled on OS X.
>
> At the end of the day, I hope there will be enough work to be done to make a GSoC project.
>
> Thanks,
> Michael.
>

Hi Michael,

A new version of a real app bundle [1] for OS X would be nice. App bundles can be installed by drag-and-drop without an installer. This could be done using Thomas Treichl's work [1] for Octave 3.2 or possibly the new MXE thing. I don't see how this could work with macports.

Regards,
Lukas


[1]
http://sourceforge.net/p/octave/code/HEAD/tree/trunk/octave-forge/admin/MacOSX/

Reply | Threaded
Open this post in threaded view
|

Re: GSoC project about binary packaging

PhilipNienhuis
In reply to this post by Michael Goffioul
Michael Goffioul wrote:
> Hi all,
>
> I've a student who's supposed to work on improving/completing binary
> packaging of octave on Windows and OS X during the next 2 months.
> However, I can see that there are already 3 to 4 people actively working
> on MXE. MXE/octave seems to already work for MinGW cross and native, as
> well as native Linux. So I'm a bit concerned that there won't be much
> left to do for the student.

Thanks for bringing this up, Michael. Good point.


My personal goal is not quite to make a binary Windows installer (I've
no involvement with OSX). I'm trying to just get Octave building
natively on MinGW, simply to be able to:

- ...avoid the
{[clone repo in Linux/bootstrap/]adapt-code/configure/build/make
dist/copy to MXE/adapt SHA1 chksum/build again now in
MXE/transplant-to-Windows/install/at last, try out the new fix}
   hassle associated with the "blind" (but otherwise extremely
efficient) MXE cross-build on Linux;

- ...run a "make check" in Windows (that's a thing that MXE cannot
offer), as I think that is a required quality check on any platform
before a binary can be distributed.
That would be a goal for GSOC as well but as it's such a small step once
a build has succeeded I think that doesn't matter;

- ...build a debug-enabled Octave. There are a few debugging targets
(old bugs specific for Windows) that I think need such a debug build.

Once Octave builds fine (and I think it does now, it just got past the
previously offending parser stumbling blocks here ) I will limit myself
to trying to fix/improve Octave itself on the Windows side, if only
because lack of time.

> So I'd like you to help define clear goals for this project in order to
> avoid stepping on each other's toes. For instance, I had listed adding
> support for OpenBLAS as a possible goal for him, but I saw today it had
> been added to MXE yesterday.

Documented empirical evidence with MinGW builds indicates that multiple
architecture-specific blas libs may be needed for a MinGW binary.
Selecting/suggesting the right one would be a good sub-task :-)

> Basically, what's left at the moment?
> - writing a NSI installer that's a bit smarter than "dump everything";
 > it should also support smart reinstallation and package addition (as in:
> re-run the installer to install additional components, which were not
> selected previously)

... or not available at the time (e.g., new packages)

- Deinstall packages and even Octave, plus proper cleaning up (.config
files in the profile, etc) would also be nice.

- How about an installer that could detect and then upgrade an existing
Octave MinGW or MSVC installation in place, rather than add a complete
tree next to the existing one?
I have about 7 or more independent MinGW/MSVC installations on my box,
each with the usual texinfo/makeinfo/ls/..... just to be able to test
the io package and do some basic bug triaging; that could be arranged
more effciently, if only to conserve disk space.

> - missing octave-forge packages + dependencies
> - OS X
> (feel free the extend the list)

- A thing that is on my list but that I'd gladly delegate is to get the
built-in core Java support activated in the MinGW builds (but maybe it
is too difficult for GSOC).
Until now java-enabled builds consistently break somewhere in liboctave
- I have reported it a few times in the ML list. Whether it is specific
to MXE I don't know.
I think Java support (optional or not) is required for a complete
binary, but depending on POV it might not fit the GSOC perspective.

- Documenting the build stuff for sub-guru levels, or even lower, so
that more Windows developers could be attracted, would also be a welcome
sub-task. I have a short text ready that could serve as a start.

> For OS X, I'd like some input from OS X users to assess what's doable.
> For instance, is it possible to integrate installer building inside MXE
> (for instance using some native OS X compilation)? I also remember a
> recent mail from Ben mentioning some new efforts in MacPorts to get
> octave compiled on OS X.
>
> At the end of the day, I hope there will be enough work to be done to
> make a GSoC project.

I doubt that there will be too little left.

Philip
Reply | Threaded
Open this post in threaded view
|

Re: GSoC project about binary packaging

PhilipNienhuis
In reply to this post by Michael Goffioul
Michael Goffioul wrote:

> Hi all,
>
> I've a student who's supposed to work on improving/completing binary
> packaging of octave on Windows and OS X during the next 2 months.
> However, I can see that there are already 3 to 4 people actively working
> on MXE. MXE/octave seems to already work for MinGW cross and native, as
> well as native Linux. So I'm a bit concerned that there won't be much
> left to do for the student.
>
> So I'd like you to help define clear goals for this project in order to
> avoid stepping on each other's toes. For instance, I had listed adding
> support for OpenBLAS as a possible goal for him, but I saw today it had
> been added to MXE yesterday.
>
> Basically, what's left at the moment?
> - writing a NSI installer that's a bit smarter than "dump everything";
> it should also support smart reinstallation and package addition (as in:
> re-run the installer to install additional components, which were not
> selected previously)
> - missing octave-forge packages + dependencies
> - OS X
> (feel free the extend the list)

Forgot this one:

- 64-bit Windows binary

Philip
Reply | Threaded
Open this post in threaded view
|

RE: GSoC project about binary packaging

John Donoghue-3
In reply to this post by Michael Goffioul

 

 

From: Michael Goffioul [mailto:[hidden email]]
Sent: Friday, June 21, 2013 2:45 PM
To: Octave Maintainers List; Philip Nienhuis; John Donoghue; John W. Eaton; Anirudha Bose; Ben Abbott
Subject: GSoC project about binary packaging

 

Hi all,

 

I've a student who's supposed to work on improving/completing binary packaging of octave on Windows and OS X during the next 2 months. However, I can see that there are already 3 to 4 people actively working on MXE. MXE/octave seems to already work for MinGW cross and native, as well as native Linux. So I'm a bit concerned that there won't be much left to do for the student.

 

So I'd like you to help define clear goals for this project in order to avoid stepping on each other's toes. For instance, I had listed adding support for OpenBLAS as a possible goal for him, but I saw today it had been added to MXE yesterday.

 

Basically, what's left at the moment?

- writing a NSI installer that's a bit smarter than "dump everything"; it should also support smart reinstallation and package addition (as in: re-run the installer to install additional components, which were not selected previously)

- missing octave-forge packages + dependencies

- OS X

(feel free the extend the list)

 

For OS X, I'd like some input from OS X users to assess what's doable. For instance, is it possible to integrate installer building inside MXE (for instance using some native OS X compilation)? I also remember a recent mail from Ben mentioning some new efforts in MacPorts to get octave compiled on OS X.

 

At the end of the day, I hope there will be enough work to be done to make a GSoC project.

 

Thanks,

Michael.

 

 

Im not planning on doing anything with my installer from what I had already done – function was mainly as an easy way to package a test version together to be able install and test it easily. I did note (at least on my computer) if you try to use MUI parts of the installer in linux, that it crashed nsis – I didn’t look at that as it worked good enough for me.

OpenBLAS seems to compile ok for native mingw compile, however might be a little flaky on cross builds? I have one linux computer that compiled it fine, another that did not – I hadn’t looked into why. Also I haven’t looked at what speed differences/advantages there are in using openblas vs blas in octave – it might be good for someone to verify that.

OpenBLAS can also replace some of the lapack functions – I didn’t activate that part, if someone wants to take a look at it.

 

 

 

 

Reply | Threaded
Open this post in threaded view
|

Re: GSoC project about binary packaging

Jordi Gutiérrez Hermoso-2
On 21 June 2013 16:43, John D <[hidden email]> wrote:

> OpenBLAS seems to compile ok for native mingw compile, however might be a
> little flaky on cross builds? I have one linux computer that compiled it
> fine, another that did not – I hadn’t looked into why.

There's something I don't understand. Most BLASes try at compile time to
figure out what the optimal build flags are and how to produce the
best binary (e.g. which vector instructions the CPU supports, etc).
How can this work for distributing a compiled BLAS? Does the OpenBLAS
build know how to delegate these questions to runtime instead of
compile time?

> Also I haven’t looked at what speed differences/advantages there are
> in using openblas vs blas in octave

You mean vs ATLAS? We haven't shipped a reference BLAS with Octave for
a long time.

- Jordi G. H.
Reply | Threaded
Open this post in threaded view
|

RE: GSoC project about binary packaging

John Donoghue-3


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Jordi Gutiérrez Hermoso
Sent: Friday, June 21, 2013 5:03 PM
To: John D
Cc: Michael Goffioul; Octave Maintainers List; Philip Nienhuis; John W. Eaton; Anirudha Bose; Ben Abbott
Subject: Re: GSoC project about binary packaging

On 21 June 2013 16:43, John D <[hidden email]> wrote:

> OpenBLAS seems to compile ok for native mingw compile, however might
> be a little flaky on cross builds? I have one linux computer that
> compiled it fine, another that did not – I hadn’t looked into why.

There's something I don't understand. Most BLASes try at compile time to figure out what the optimal build flags are and how to produce the best binary (e.g. which vector instructions the CPU supports, etc).
How can this work for distributing a compiled BLAS? Does the OpenBLAS build know how to delegate these questions to runtime instead of compile time?

> Also I haven’t looked at what speed differences/advantages there are
> in using openblas vs blas in octave

You mean vs ATLAS? We haven't shipped a reference BLAS with Octave for a long time.

- Jordi G. H.

________________

I was meaning the blas we were compiling against in mingw if openblas isn’t used.

Reply | Threaded
Open this post in threaded view
|

Re: GSoC project about binary packaging

Jordi Gutiérrez Hermoso-2
On 21 June 2013 17:04, John D <[hidden email]> wrote:
> On Behalf Of Jordi Gutiérrez Hermoso

> > On 21 June 2013 16:43, John D <[hidden email]> wrote:
> >
> >
> >> Also I haven’t looked at what speed differences/advantages there
> >> are in using openblas vs blas in octave
> >
> > You mean vs ATLAS? We haven't shipped a reference BLAS with Octave
> > for a long time.
>
> I was meaning the blas we were compiling against in mingw if
> openblas isn’t used.

Right, that's usually ATLAS. Do you know anything about the following
part?

> >> OpenBLAS seems to compile ok for native mingw compile, however
> >> might be a little flaky on cross builds? I have one linux
> >> computer that compiled it fine, another that did not – I hadn’t
> >> looked into why.
> >
> > There's something I don't understand. Most BLASes try at compile
> > time to figure out what the optimal build flags are and how to
> > produce the best binary (e.g. which vector instructions the CPU
> > supports, etc). How can this work for distributing a compiled
> > BLAS? Does the OpenBLAS build know how to delegate these questions
> > to runtime instead of compile time?

- Jordi G. H.
Reply | Threaded
Open this post in threaded view
|

Re: GSoC project about binary packaging

Michael Goffioul
In reply to this post by Jordi Gutiérrez Hermoso-2
On Fri, Jun 21, 2013 at 5:02 PM, Jordi Gutiérrez Hermoso <[hidden email]> wrote:
On 21 June 2013 16:43, John D <[hidden email]> wrote:

> OpenBLAS seems to compile ok for native mingw compile, however might be a
> little flaky on cross builds? I have one linux computer that compiled it
> fine, another that did not – I hadn’t looked into why.

There's something I don't understand. Most BLASes try at compile time to
figure out what the optimal build flags are and how to produce the
best binary (e.g. which vector instructions the CPU supports, etc).
How can this work for distributing a compiled BLAS? Does the OpenBLAS
build know how to delegate these questions to runtime instead of
compile time?

OpenBLAS can be compiled in 2 modes: single-arch and dynamic-arch. In single-arch, you compile for a specific CPU family/chipset. There's no compile-time optimization involved like ATLAS, once a cpu/chipset is chosen, OpenBLAS is created out of a predefined set of highly optimized routines, mostly written in assembler. In dynamic-arch mode, all routines for all cpu/chipset are compiled and OpenBLAS then uses runtime CPU detection to select the set of routines to use. I suppose is a one-time detection at runtime.

In the past, I've compared multithreaded ATLAS and OpenBLAS under Windows on the architecture I had (Atom N470) and OpenBLAS outperformed ATLAS (not to mention that it compiles much faster than ATLAS).
 
> Also I haven’t looked at what speed differences/advantages there are
> in using openblas vs blas in octave

You mean vs ATLAS? We haven't shipped a reference BLAS with Octave for
a long time.

I have, in my MSVC binaries. The reference BLAS, though slow, is deemed to be more stable than OpenBLAS/ATLAS.

Michael.

Reply | Threaded
Open this post in threaded view
|

Re: GSoC project about binary packaging

bpabbott
Administrator
In reply to this post by Lukas Reichlin-4

On Jun 22, 2013, at 3:52 AM, Lukas Reichlin wrote:

> On 21.06.2013, at 20:44, Michael Goffioul <[hidden email]> wrote:
>
>> Hi all,
>>
>> I've a student who's supposed to work on improving/completing binary packaging of octave on Windows and OS X during the next 2 months. However, I can see that there are already 3 to 4 people actively working on MXE. MXE/octave seems to already work for MinGW cross and native, as well as native Linux. So I'm a bit concerned that there won't be much left to do for the student.
>>
>> So I'd like you to help define clear goals for this project in order to avoid stepping on each other's toes. For instance, I had listed adding support for OpenBLAS as a possible goal for him, but I saw today it had been added to MXE yesterday.
>>
>> Basically, what's left at the moment?
>> - writing a NSI installer that's a bit smarter than "dump everything"; it should also support smart reinstallation and package addition (as in: re-run the installer to install additional components, which were not selected previously)
>> - missing octave-forge packages + dependencies
>> - OS X
>> (feel free the extend the list)
>>
>> For OS X, I'd like some input from OS X users to assess what's doable. For instance, is it possible to integrate installer building inside MXE (for instance using some native OS X compilation)? I also remember a recent mail from Ben mentioning some new efforts in MacPorts to get octave compiled on OS X.
>>
>> At the end of the day, I hope there will be enough work to be done to make a GSoC project.
>>
>> Thanks,
>> Michael.
>>
>
> Hi Michael,
>
> A new version of a real app bundle [1] for OS X would be nice. App bundles can be installed by drag-and-drop without an installer. This could be done using Thomas Treichl's work [1] for Octave 3.2 or possibly the new MXE thing. I don't see how this could work with macports.
>
> Regards,
> Lukas

Regarding MacOS X, I'd also assumed we'd want to produce an app bundled.  If all dependencies are available (via Macports, MXE, or manually built) it isn't too hard to create a bundled app.  The tricky part will be getting mkoctfile to work.  Due to a lack of time, I'm delaying further effort on the Macport's approach, and am planning to wait until the MXE approach begins to take shape before starting again.

Ben

Reply | Threaded
Open this post in threaded view
|

Re: GSoC project about binary packaging

Michael Goffioul
Thanks to everybody for all the valuable inputs. Let me try now to make a more detailed plan for this GSoC project. In general, I think the high-level targets for the project should be as follows (pretty close to the original GSoC proposal anyway):

- first code session (until mid-term): MinGW native/cross build with installer generation
- second code session: OS X native build with app bundle

1) MinGW build with installer

This will be based on MXE and allow to make octave binaries using MinGW cross-compiler and native compiler. The build itself should contain as many features as possible. This means:
- OpenBLAS: I would initially go with a 32-bits multi-thread dynamic arch version of the library; because of possible problems on some architectures, I think it makes also sense to ship octave with a reference BLAS; what I used to do in my installers is to ship both, let user select which one to install and install the selected on as the "blas" DLL; in practice it means that octave is compiled against the reference BLAS, but the reference BLAS DLL can get overwritten by the OpenBLAS DLL (using the same name); this works as long as both DLL export the same symbols
- LLVM/JIT
- Java: octave should be compiled with java support; but the JRE should not be compiled or bundled into the installer; the octave/java bridge looks the the JRE at run-time using information from the registry
- GUI: obviously we want the GUI by default compiled into octave
- octave-forge packages: here I'm not sure it makes sense to bundle *all* packages, some of them are "exotic" and not wanted by most users; I would suggest to make a list of must-have core packages, and let the others package be provided as addons with their own installer

As mentioned previously, it would be nice to have a smarter installer script able to do more than simply dumping everything into a directory. In particular, the features I have in mind are:
- component selection: things that I think make sense to be (de)selectable: development files (headers, library files, compiler...), GUI, documentation/manuals, octave-forge packages, MSYS (a few elements are still needed by code octave, for the pager and formatting help text)
- if we ship a reference BLAS and OpenBLAS, it's nice to be able to select which one to install
- default graphics toolkit selection
- selection of octave forge packages that the user wants autoloaded
- possibility to upgrade an existing installation or install side-by-side; in case of upgrading, I think the only sensible thing to do is to uninstall the previous version and install the new one
- possibility to re-run the installer to install components that weren't install the first time; though I'm not sure how easy this can be done with NSIS, I never looked into that
- possibility to package octave-forge packages into their own installer that is deployed on top an existing installation; the approach I've taken in my installers as to generate a binary archive with "pkg build" then wrap that archive into a NSIS installer

Another thing that Philip has brought on the table, which I find interesting, is to have binaries produced with mingw64. I think having a full 64-bits version of octave would be happily welcome by a number of users. It's an open question, but IMO it can make sense to keep the installer simpler, save time on the implementation, and use that time to be able to produce 64-bits binaries. What do others think?

2) OS X build

Here it's still not clear what's feasible. So, Anirudha, during the first coding session, I'd like you also to think about OS X binary and app bundle generation.

Michael.

Reply | Threaded
Open this post in threaded view
|

Re: GSoC project about binary packaging

PhilipNienhuis
Michael Goffioul wrote
Thanks to everybody for all the valuable inputs. Let me try now to make a
more detailed plan for this GSoC project. In general, I think the
high-level targets for the project should be as follows (pretty close to
the original GSoC proposal anyway):

- first code session (until mid-term): MinGW native/cross build with
installer generation
- second code session: OS X native build with app bundle

1) MinGW build with installer

This will be based on MXE and allow to make octave binaries using MinGW
cross-compiler and native compiler. The build itself should contain as many
features as possible. This means:
:
<snip>
- octave-forge packages: here I'm not sure it makes sense to bundle *all* packages, some of them are "exotic" and not wanted by most users; I would suggest to make a list of must-have core packages, and let the others package be provided as addons with their own installer
Or we could have the user rely on "pkg install -forge ....".
Only some basic development tools are needed for that with (at least for MinGW) modest disk space requirements.
The extremes are all OF packages pre-built (and individually selectable), and none but then including build tools. Your suggestion above is somewhere in between. I wonder which option would require the lowest maintenance burden.
Skipping the preparation (and required maintenance) of an arbitrary set of OF core packages will save precious time for the GSOC student, its mentors and (later on) maintainers.

I don't know what the best option would be.

:
<snip>
- possibility to upgrade an existing installation or install side-by-side;
in case of upgrading, I think the only sensible thing to do is to uninstall
the previous version and install the new one
Is that really true? (I honestly don't know)

My suggestion to merge new MinGW installations into existing ones is based on the fact that the MinGW directory tree closely mimics the one in Linux systems. Which leads me to suspect that an in-situ upgrade should be possible just like on Linux.

The most relevant stumbling block is how to handle the dependencies (.dll files) in /bin. But as long as those are compatible it might just work to either leave them alone, or upgrade them as well. Or maybe they can be put in some /bin/i686-pc-mingw32-api-v<version>+ subdir and added to octave-version-specific PATHs (wild guess).

I suggested this because for every Octave invocation on my system(s) I have to copy over all my preferences (octaverc, paths to some binaries, fig2dev and pstoedit, favorite editor, etc). It would be nice if octaverc could be stored somewhere in my Windows user profile (is that possible?) and automatically be picked up by new Octave installations.

:
<snip>
Another thing that Philip has brought on the table, which I find
interesting, is to have binaries produced with mingw64. I think having a
full 64-bits version of octave would be happily welcome by a number of
users. It's an open question, but IMO it can make sense to keep the
installer simpler, save time on the implementation, and use that time to be
able to produce 64-bits binaries. What do others think?
Given the available bits and pieces I suppose creating the binary MinGW (MSVC) installer itself shouldn't take much time anyway.
OTOH fixing the rough edges in mxe-octave may need more time.

After I posted that 64-bit suggestion, I was reading up on various MinGW sites and I got the impression that 64-bit MinGW maybe isn't so mature yet. There are some parallel MinGW-based projects that claim better 64-bit support.  Or am I mistaken?
(Does MSVC allow 64-bit builds?)

Do we have a clear view of the pros and cons of 64-bit Octave? Can it be mixed with 32-bit packages? Would it have a smooth binary interface to 32-bit SW?
Since a few months I work with a mix of 32-bit and 64-bit Windows systems (at home and at work) and every week I find smaller or bigger incompatibilities with 64 & 32-bit SW. One of those is that the OF windows package (32 bit) won't work with 64-bit MS-Office.

Nevertheless I think at some point we'll have to make a start with 64-bit Octave on Windows anyway. If this project can give us a headstart I'm all in favor, even if 64-bit doesn't get completely finished. I do think the primary aim is to get a reliable, easily maintainable 32-bit binary installer for a feature complete Octave.

<rest snipped>
Philip
Reply | Threaded
Open this post in threaded view
|

Re: GSoC project about binary packaging

Anirudha Bose
In reply to this post by Michael Goffioul
Sorry for being out of contact for a few days. I read your detailed proposal and now I will be able to work according to the ideas you have proposed. I am a little behind schedule so I will focus on the first coding session goals, as of now. I will blog about the new goals of this project once I have got a clear idea of the deliverables.

On Mon, Jun 24, 2013 at 1:41 AM, Michael Goffioul <[hidden email]> wrote:
Thanks to everybody for all the valuable inputs. Let me try now to make a more detailed plan for this GSoC project. In general, I think the high-level targets for the project should be as follows (pretty close to the original GSoC proposal anyway):

- first code session (until mid-term): MinGW native/cross build with installer generation
- second code session: OS X native build with app bundle

1) MinGW build with installer

This will be based on MXE and allow to make octave binaries using MinGW cross-compiler and native compiler. The build itself should contain as many features as possible. This means:
- OpenBLAS: I would initially go with a 32-bits multi-thread dynamic arch version of the library; because of possible problems on some architectures, I think it makes also sense to ship octave with a reference BLAS; what I used to do in my installers is to ship both, let user select which one to install and install the selected on as the "blas" DLL; in practice it means that octave is compiled against the reference BLAS, but the reference BLAS DLL can get overwritten by the OpenBLAS DLL (using the same name); this works as long as both DLL export the same symbols
- LLVM/JIT
- Java: octave should be compiled with java support; but the JRE should not be compiled or bundled into the installer; the octave/java bridge looks the the JRE at run-time using information from the registry
- GUI: obviously we want the GUI by default compiled into octave
- octave-forge packages: here I'm not sure it makes sense to bundle *all* packages, some of them are "exotic" and not wanted by most users; I would suggest to make a list of must-have core packages, and let the others package be provided as addons with their own installer
 
Instead of including all the octave-forge packages in the installer, can we let the users select the desired packages and download them? If we do include all the packages in the installer itself and let the users choose which ones to install during the installation, the size of the installer will get very large. This is something which is undesirable for someone who wants the basic installation packed in a small setup file.


As mentioned previously, it would be nice to have a smarter installer script able to do more than simply dumping everything into a directory. In particular, the features I have in mind are:
- component selection: things that I think make sense to be (de)selectable: development files (headers, library files, compiler...), GUI, documentation/manuals, octave-forge packages, MSYS (a few elements are still needed by code octave, for the pager and formatting help text)
- if we ship a reference BLAS and OpenBLAS, it's nice to be able to select which one to install

I think it might be confusing for some users who don't know which BLAS they should install. Are you proposing that we build Octave for Windows without OpenBLAS, and just add the DLL file of OpenBLAS if the users wants? In that case what should be the substitute here 
    --with-blas=openblas
in src/octave.mk? Will the --enable-openblas option in configure still be applicable? Is Octave compiled with reference BLAS by default if OpenBLAS is disabled?

- default graphics toolkit selection
- selection of octave forge packages that the user wants autoloaded
- possibility to upgrade an existing installation or install side-by-side; in case of upgrading, I think the only sensible thing to do is to uninstall the previous version and install the new one
- possibility to re-run the installer to install components that weren't install the first time; though I'm not sure how easy this can be done with NSIS, I never looked into that

I will investigate into this and let you know if I find a solution. From what I have read about NSIS, I am not sure how to implement this feature yet.

- possibility to package octave-forge packages into their own installer that is deployed on top an existing installation; the approach I've taken in my installers as to generate a binary archive with "pkg build" then wrap that archive into a NSIS installer

Another thing that Philip has brought on the table, which I find interesting, is to have binaries produced with mingw64. I think having a full 64-bits version of octave would be happily welcome by a number of users. It's an open question, but IMO it can make sense to keep the installer simpler, save time on the implementation, and use that time to be able to produce 64-bits binaries. What do others think?

2) OS X build

Here it's still not clear what's feasible. So, Anirudha, during the first coding session, I'd like you also to think about OS X binary and app bundle generation.

Michael.


Reply | Threaded
Open this post in threaded view
|

Re: GSoC project about binary packaging

Michael Goffioul
On Mon, Jun 24, 2013 at 3:29 AM, Anirudha Bose <[hidden email]> wrote:
Sorry for being out of contact for a few days. I read your detailed proposal and now I will be able to work according to the ideas you have proposed. I am a little behind schedule so I will focus on the first coding session goals, as of now. I will blog about the new goals of this project once I have got a clear idea of the deliverables.

You goal for this week is to get octave builds in native and cross MinGW, with all features enabled (LLVM/JIT, OpenBLAS, Java, GUI...). Those builds should pass the test suite (bar the expected failures under MinGW). If there are things not working in MXE, you'll have to fix them along the way. Hopefully, it'll go smoothly and you can then start looking at the installer script before the end of the week.
  
Instead of including all the octave-forge packages in the installer, can we let the users select the desired packages and download them? If we do include all the packages in the installer itself and let the users choose which ones to install during the installation, the size of the installer will get very large. This is something which is undesirable for someone who wants the basic installation packed in a small setup file.

To be honest, I think the size of the packages will not be that big compared to the rest (octave, dependencies, development files, compiler, MSYS...). Anyway, there are 2 trends here:
1) No package in included in the main installer; each package is provided into its own installer, which also contains its own dependencies. I still think it makes sense to provide pre-compiled version of the packages.
2) Include all packages into the main installer.

I'm fine with both options.
 
I think it might be confusing for some users who don't know which BLAS they should install.

The default should be OpenBLAS. The reference BLAS is only there in cases OpenBLAS crashes, as a backup for the user.
 
Are you proposing that we build Octave for Windows without OpenBLAS, and just add the DLL file of OpenBLAS if the users wants? In that case what should be the substitute here 
    --with-blas=openblas
in src/octave.mk? Will the --enable-openblas option in configure still be applicable? Is Octave compiled with reference BLAS by default if OpenBLAS is disabled?

Octave doesn't really care about BLAS or OpenBLAS, it's just looking for an API. You can compile octave against BLAS and then later on make it run with OpenBLAS, as long as OpenBLAS provides the same API/ABI. So the idea is to compile octave against the reference BLAS. You include both DLL in the installer, let's say libblas.dll and libopenblas.dll. At installation time, depending on the user selection, you install libblas.dll or libopenblas.dll, but whatever choice, the DLL is installed with the name "libblas.dll".

Another possibility would be to install both and have a tool to let the user choose what DLL to use at runtime. For instance installing the 2 DLL: libblasref.dll and libopenblas.dll; then the tool would copy the wanted BLAS as libblas.dll.

Michael.

Reply | Threaded
Open this post in threaded view
|

Re: GSoC project about binary packaging

PhilipNienhuis
Michael Goffioul wrote
On Mon, Jun 24, 2013 at 3:29 AM, Anirudha Bose <[hidden email]> wrote:

> Sorry for being out of contact for a few days. I read your detailed
> proposal and now I will be able to work according to the ideas you have
> proposed. I am a little behind schedule so I will focus on the first coding
> session goals, as of now. I will blog about the new goals of this project
> once I have got a clear idea of the deliverables.
>

You goal for this week is to get octave builds in native and cross MinGW,
with all features enabled (LLVM/JIT, OpenBLAS, Java, GUI...). Those builds
should pass the test suite (bar the expected failures under MinGW). If
there are things not working in MXE, you'll have to fix them along the way.
Hopefully, it'll go smoothly and you can then start looking at the
installer script before the end of the week.
I'll attach my write-up for compiling on Windows natively here. Not to help Anirudha Bose with cheating, but hopefully it'll help speed things up. (The spirit of OSS it to build on top of other people's work after all.)

MXE-octave_buildinstr.txt 

Good luck,

Philip
Reply | Threaded
Open this post in threaded view
|

Re: GSoC project about binary packaging

Anirudha Bose
On Mon, Jun 24, 2013 at 11:44 PM, PhilipNienhuis <[hidden email]> wrote:

I'll attach my write-up for compiling on Windows natively here. Not to help
Anirudha Bose with cheating, but hopefully it'll help speed things up. (The
spirit of OSS it to build on top of other people's work after all.)

MXE-octave_buildinstr.txt
<http://octave.1599824.n4.nabble.com/file/n4654744/MXE-octave_buildinstr.txt>

Thanks for the providing this! While trying to build Octave natively on Windows, I get this message (when I do configure --enable-native-gcc --enable-native-build):
    configure: WARNING: unrecognized options: --enable-native-gcc

My guess would be that there are problems with the PATH. After I did 
    echo $PATH > .profile
I navigated to this location C:\MinGW\msys\1.0\home\ani where I found that file. Before making changes to it, the file contents were:
export $PATH=.:/usr/local/bin:/mingw/bin:/bin:/c/Program Files (x86)/Intel/iCLS Client/:/c/Program Files/Intel/iCLS Client/:/c/Windows/system32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0/:/c/Program Files (x86)/Windows Live/Shared:/c/Program Files (x86)/Intel/OpenCL SDK/2.0/bin/x86:/c/Program Files (x86)/Intel/OpenCL SDK/2.0/bin/x64:/c/Program Files/Intel/Intel(R) Management Engine Components/DAL:/c/Program Files/Intel/Intel(R) Management Engine Components/IPT:/c/Program Files (x86)/Intel/Intel(R) Management Engine Components/DAL:/c/Program Files (x86)/Intel/Intel(R) Management Engine Components/IPT:/c/Program Files/TortoiseHg/

I changed the above contents and these are the contents now:
export $PATH=.:/usr/local/bin:/mingw/bin:/bin:/c/"Program Files"/TortoiseHg/:/c/"Program Files"/gs/gs9.07
export PKG_CONFIG_PATH=/home/ani/mxe-octave/usr/i686-pc-mingw32/lib/pkgconfig

Even after the above changes my MinGW shell is showing errors when I restart it. Am I doing something wrong? Please help.


Good luck,

Philip



--
View this message in context: http://octave.1599824.n4.nabble.com/GSoC-project-about-binary-packaging-tp4654636p4654744.html
Sent from the Octave - Maintainers mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: GSoC project about binary packaging

Anirudha Bose
On Wed, Jun 26, 2013 at 10:40 AM, Anirudha Bose <[hidden email]> wrote:
On Mon, Jun 24, 2013 at 11:44 PM, PhilipNienhuis <[hidden email]> wrote:

I'll attach my write-up for compiling on Windows natively here. Not to help
Anirudha Bose with cheating, but hopefully it'll help speed things up. (The
spirit of OSS it to build on top of other people's work after all.)

MXE-octave_buildinstr.txt
<http://octave.1599824.n4.nabble.com/file/n4654744/MXE-octave_buildinstr.txt>

Thanks for the providing this! While trying to build Octave natively on Windows, I get this message (when I do configure --enable-native-gcc --enable-native-build):
    configure: WARNING: unrecognized options: --enable-native-gcc

My guess would be that there are problems with the PATH. After I did 
    echo $PATH > .profile
I navigated to this location C:\MinGW\msys\1.0\home\ani where I found that file. Before making changes to it, the file contents were:
export $PATH=.:/usr/local/bin:/mingw/bin:/bin:/c/Program Files (x86)/Intel/iCLS Client/:/c/Program Files/Intel/iCLS Client/:/c/Windows/system32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0/:/c/Program Files (x86)/Windows Live/Shared:/c/Program Files (x86)/Intel/OpenCL SDK/2.0/bin/x86:/c/Program Files (x86)/Intel/OpenCL SDK/2.0/bin/x64:/c/Program Files/Intel/Intel(R) Management Engine Components/DAL:/c/Program Files/Intel/Intel(R) Management Engine Components/IPT:/c/Program Files (x86)/Intel/Intel(R) Management Engine Components/DAL:/c/Program Files (x86)/Intel/Intel(R) Management Engine Components/IPT:/c/Program Files/TortoiseHg/

Sorry, the export $PATH= was not present in the original file. So initially the .profile file was something like this:
.:/usr/local/bin:/mingw/bin:/bin:/c/Program Files (x86)/Intel/iCLS Client/:/c/Program Files/Intel/iCLS Client/:/c/Windows/system32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0/:/c/Program Files (x86)/Windows Live/Shared:/c/Program Files (x86)/Intel/OpenCL SDK/2.0/bin/x86:/c/Program Files (x86)/Intel/OpenCL SDK/2.0/bin/x64:/c/Program Files/Intel/Intel(R) Management Engine Components/DAL:/c/Program Files/Intel/Intel(R) Management Engine Components/IPT:/c/Program Files (x86)/Intel/Intel(R) Management Engine Components/DAL:/c/Program Files (x86)/Intel/Intel(R) Management Engine Components/IPT:/c/Program Files/TortoiseHg/


I changed the above contents and these are the contents now:
export $PATH=.:/usr/local/bin:/mingw/bin:/bin:/c/"Program Files"/TortoiseHg/:/c/"Program Files"/gs/gs9.07
export PKG_CONFIG_PATH=/home/ani/mxe-octave/usr/i686-pc-mingw32/lib/pkgconfig

Even after the above changes my MinGW shell is showing errors when I restart it. Am I doing something wrong? Please help.


Good luck,

Philip



--
View this message in context: http://octave.1599824.n4.nabble.com/GSoC-project-about-binary-packaging-tp4654636p4654744.html
Sent from the Octave - Maintainers mailing list archive at Nabble.com.


Reply | Threaded
Open this post in threaded view
|

Re: GSoC project about binary packaging

PhilipNienhuis
Hi Anirudha (is that your first name?)

Anirudha Bose wrote
On Wed, Jun 26, 2013 at 10:40 AM, Anirudha Bose <[hidden email]> wrote:

> On Mon, Jun 24, 2013 at 11:44 PM, PhilipNienhuis <[hidden email]>wrote:
>
>>
>> I'll attach my write-up for compiling on Windows natively here. Not to
>> help
>> Anirudha Bose with cheating, but hopefully it'll help speed things up.
>> (The
>> spirit of OSS it to build on top of other people's work after all.)
>>
>> MXE-octave_buildinstr.txt
>> <
>> http://octave.1599824.n4.nabble.com/file/n4654744/MXE-octave_buildinstr.txt
>> >
>>
>
> Thanks for the providing this! While trying to build Octave natively on
> Windows, I get this message (when I do configure --enable-native-gcc
> --enable-native-build):
>     configure: WARNING: unrecognized options: --enable-native-gcc
Please run "configure --help" from octave's source dir to see what options you need to enter exactly how.

> My guess would be that there are problems with the PATH. After I did
>     echo $PATH > .profile
> I navigated to this location C:\MinGW\msys\1.0\home\ani where I found that
> file. Before making changes to it, the file contents were:
> export $PATH=.:/usr/local/bin:/mingw/bin:/bin:/c/Program Files
> (x86)/Intel/iCLS Client/:/c/Program Files/Intel/iCLS
> Client/:/c/Windows/system32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0/:/c/Program
> Files (x86)/Windows Live/Shared:/c/Program Files (x86)/Intel/OpenCL
> SDK/2.0/bin/x86:/c/Program Files (x86)/Intel/OpenCL
> SDK/2.0/bin/x64:/c/Program Files/Intel/Intel(R) Management Engine
> Components/DAL:/c/Program Files/Intel/Intel(R) Management Engine
> Components/IPT:/c/Program Files (x86)/Intel/Intel(R) Management Engine
> Components/DAL:/c/Program Files (x86)/Intel/Intel(R) Management Engine
> Components/IPT:/c/Program Files/TortoiseHg/
>

Sorry, the export $PATH= was not present in the original file. So initially
the .profile file was something like this:
.:/usr/local/bin:/mingw/bin:/bin:/c/Program Files (x86)/Intel/iCLS
Client/:/c/Program Files/Intel/iCLS
Client/:/c/Windows/system32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0/:/c/Program
Files (x86)/Windows Live/Shared:/c/Program Files (x86)/Intel/OpenCL
SDK/2.0/bin/x86:/c/Program Files (x86)/Intel/OpenCL
SDK/2.0/bin/x64:/c/Program Files/Intel/Intel(R) Management Engine
Components/DAL:/c/Program Files/Intel/Intel(R) Management Engine
Components/IPT:/c/Program Files (x86)/Intel/Intel(R) Management Engine
Components/DAL:/c/Program Files (x86)/Intel/Intel(R) Management Engine
Components/IPT:/c/Program Files/TortoiseHg/


> I changed the above contents and these are the contents now:
> export $PATH=.:/usr/local/bin:/mingw/bin:/bin:/c/"Program
> Files"/TortoiseHg/:/c/"Program Files"/gs/gs9.07
> export
> PKG_CONFIG_PATH=/home/ani/mxe-octave/usr/i686-pc-mingw32/lib/pkgconfig
> export $PATH=.:/usr/local/bin:/mingw/bin:/bin:/c/"Program
> Files"/TortoiseHg/:/c/"Program Files"/gs/gs9.07

The "Program Files" thing isn't going to work like you entered it. You will HAVE to install Mercurial and ghostview in a path w/o spaces, period, otherwise it won't be picked up properly by the various build scripts.
(Once Octave is built and runs the spaces in the PATH don't matter; it is at build time that I found things to go wrong)
Furthermore you'd need to include not the main program subdirs for hg and gs but the bin/ dirs inside.

Please reread the build instructions more carefully.

> Even after the above changes my MinGW shell is showing errors when I
> restart it. Am I doing something wrong? Please help.
I'm sorry, I have limited time this week (and my own jobs to do). If you really get stuck try to contact the mentors as well. I'll follow this discussion anyway, don't worry

Philip
Reply | Threaded
Open this post in threaded view
|

Re: GSoC project about binary packaging

Patrick Noffke
In reply to this post by Michael Goffioul
On Sun, Jun 23, 2013 at 3:11 PM, Michael Goffioul
<[hidden email]> wrote:
> 1) MinGW build with installer
>

Sorry to come into this discussion late.  Am I correct that the
Windows installer file can only be created on a Windows machine (i.e.
you have to run NSIS on Windows)?  I see nsis is cross-compiled, but
I'm not sure what good that does if you want to make a full installer
from Linux.

Anyway, I just wanted to throw this out there.  It is possible to
build a Windows Installer (.msi) file from Linux.  I've done this
using the msi-tools[1] package on Fedora 18.  If you want, you can
also use WiX natively on Windows.  Both are similar, though the Linux
version implements a subset of the WiX features.  Either way involves
writing a .wxs file[2] to describe your installer, then run wixl (or
equivalent (candle.exe and light.exe) on Windows) to build the .msi
file.

You can create fancy UI screens for user-selectable options if you
want.  I've done this before (on Windows -- I'd have to test again
with msi-tools), and could help if this is useful.  I'd have to dig up
some of my past .wxs files, but you can also check out the WiX
tutorial here:  http://wix.tramontana.co.hu/tutorial

For easy importing of a whole bunch of DLLs, you can use the wixl-heat
(I think just heat.exe on Windows) to generate fragment files.  Then
you just include the fragment files as input files along with your
main .wxs file when running wixl.  Here is a snippet from a makefile
that runs on Linux, which imports all the necessary cross-compiled
DLLs from the mingw64-distribution (includes Qt5 and some others), and
builds the final .msi file.

        ls $(MINGW_SYSROOT)/mingw/bin/Qt5*.dll | \
          wixl-heat --component-group Qt5DLLs -p $(MINGW_SYSROOT)/mingw/bin/ > \
          $(PRODUCT_BUILD_DIR)/Qt5DLLs.wxs.in
        cat $(PRODUCT_BUILD_DIR)/Qt5DLLs.wxs.in | \
          sed s,"TARGETDIR","INSTALLDIR", | \
          sed s,"SourceDir","$(MINGW_SYSROOT)/mingw/bin", \
            > $(PRODUCT_BUILD_DIR)/Qt5DLLs.wxs
        find $(MINGW_SYSROOT)/mingw/lib/qt5/plugins -name \*.dll | \
          wixl-heat --component-group Qt5Plugins -p
$(MINGW_SYSROOT)/mingw/lib/qt5/plugins/ > \
          $(PRODUCT_BUILD_DIR)/Qt5Plugins.wxs.in
        cat $(PRODUCT_BUILD_DIR)/Qt5Plugins.wxs.in | \
          sed s,"TARGETDIR","INSTALLDIR", | \
          sed s,"SourceDir","$(MINGW_SYSROOT)/mingw/lib/qt5/plugins", \
            > $(PRODUCT_BUILD_DIR)/Qt5Plugins.wxs
        ls $(MINGW_SYSROOT)/mingw/bin/lib*.dll \
           $(MINGW_SYSROOT)/mingw/bin/zlib1.dll \
           $(MINGW_SYSROOT)/mingw/bin/qwt*.dll \
           $(MINGW_SYSROOT)/mingw/bin/iconv.dll \
           $(MINGW_SYSROOT)/mingw/bin/pthread*.dll \
          | \
          wixl-heat --component-group MinGWOtherLibs -p
$(MINGW_SYSROOT)/mingw/bin/ > \
          $(PRODUCT_BUILD_DIR)/MinGWOtherLibs.wxs.in
        cat $(PRODUCT_BUILD_DIR)/MinGWOtherLibs.wxs.in | \
          sed s,"TARGETDIR","INSTALLDIR", | \
          sed s,"SourceDir","$(MINGW_SYSROOT)/mingw/bin", \
            > $(PRODUCT_BUILD_DIR)/MinGWOtherLibs.wxs
        wixl \
           $(PRODUCT_BUILD_DIR)/Qt5DLLs.wxs \
           $(PRODUCT_BUILD_DIR)/Qt5Plugins.wxs \
           $(PRODUCT_BUILD_DIR)/MinGWOtherLibs.wxs \
           $(PRODUCT_BUILD_DIR)/UpdateUtility/UpdateUtility.wxs \
          -o $(PRODUCT_BUILD_DIR)/UpdateUtility/UpdateUtility.msi

The name of my application is "UpdateUtility."  MINGW_SYSROOT was set using:

MINGW_SYSROOT=$(shell x86_64-w64-mingw32-gcc -print-sysroot)

This should obviously change to correspond to the cross-compiler being used.

[1] https://wiki.gnome.org/msitools
[2] http://wix.sourceforge.net/manual-wix2/wxs.htm

Patrick
Reply | Threaded
Open this post in threaded view
|

Re: GSoC project about binary packaging

Jordi Gutiérrez Hermoso-2
On 2 July 2013 15:06, Patrick Noffke <[hidden email]> wrote:
>
> Sorry to come into this discussion late.  Am I correct that the
> Windows installer file can only be created on a Windows machine (i.e.
> you have to run NSIS on Windows)?

No, this seems to be false, e.g.:

    http://packages.debian.org/wheezy/nsis

- Jordi G. H.
123