Further on MEX

classic Classic list List threaded Threaded
30 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: Further on MEX

David Bateman-2
Aravindh Krishnamoorthy wrote:

> Dear David,
>
>  
>> Or are you saying that if that m-file depends on a function that is only in
>> Octave, and so can only run in Octave then its distribution is covered by
>> the GPL? That position is perhaps more valid, but this hasn't been what I'd
>> understood as being the licensing position on m-files in the past, and makes
>> thing a real minefield for development with Octave for any one that works in
>> Industry.
>>    
>
> GNU GPL FAQ on interpreted files:
> http://www.gnu.org/licenses/gpl-faq.html#IfInterpreterIsGPL
> Also there's a note on using M-files that are distributed under GPL in
> another M-file (last two paragraphs) -- which was quite a surprise to
> me!
>  
I don't see why that is a surprise. If an m-file is released under the
GPL then if it is used in another m-file, necessarily the other is
GPLed.. But the Octave syntax even for the majority of the Octave-forge
packages duplicates the same functionality and API of the equivalent
matlab function, so the user can  legitimately say their code targeted
matlab and just incidentally worked in Octave, which effectively
circumvents this. This however is not true of functions that  are only
in octave or octave-forge under the GPL.

>> We clearly can't take a position that an oct-file doesn't fall under the GPL
>> as that exposes the entirety of Octave's internal outside the context of the
>> protection of the GPL. I totally agree than an oct-file is and always will
>> be covered by the GPL. What I'd like to see is that the MEX ABI is
>> considered like the m-file API as being part of the Octave Octave language
>> rather than the program and being outside the GPL.
>>    
>
> IMHO, this isn't possible at the moment. As Octave is "Copyright (C)
> 2007 John W. Eaton and others." [_others_] and since John mentioned
> (in the other thread) that he stopped collecting copyright related
> information from contributors, I'm not sure even if an excessive audit
> can separate the contributed code and re-license Octave under the
> modified license.
> http://www.gnu.org/licenses/gpl-faq.html#LinkingOverControlledInterface
>  
I'm not sure that is needed in the case of the MEX ABI however, which is
why I've recently suggest uses that as the means of giving proprietary
code users of Octave at least some chance of distributing binary mex-files.

>
> One of the options that remains, however, is to convert MEX code to a
> separate self-contained program (separate process) which communicates
> with Octave for data exchange.
>
> [One could stick with border-line plug-in cases but that would be
> tricky and with re-definition of mxArray + will continue to be a
> border-line case even then.]
>
> If one were pressed to use their non-free MEX C/C++ code with code, he
> will undoubtedly come out with a solution. You must be knowing the
> many Linux embedded system developers that hide their proprietary
> device driver code by using signals, mmap and writing their whole
> device driver as a user-mode module.
>
> This method [non-free linux dev drivers] is of course sick, and slow,
> and so will be the options that allow MEX interaction with Octave.
> Thing is -- I volunteered to write this code, but I won't because
> there's opposition from John and other Octave developers. [I am
> wanting to do some good development for Octave -- can't make enemies
> here :-)]
>  

Yes sick and slow... Better to go for a fully matlab compatible MEX ABI
and fall under the same argument as distribution of MEX code as source
code.. That is of course if  we can't just consider that the current
Octave MEX ABI isn't already  separate enough from Octave and closer to
matlab that it doesn't already fall under the same argument.

Hey I consider myself an Octave developer and suspect others here do to,
and I've advocated this position for quite a while.

D.


--
David Bateman                                [hidden email]
35 rue Gambetta                              +33 1 46 04 02 18 (Home)
92100 Boulogne-Billancourt FRANCE            +33 6 72 01 06 33 (Mob)

Reply | Threaded
Open this post in threaded view
|

Re: Further on MEX

John W. Eaton
Administrator
In reply to this post by David Bateman-2

Before I respond to your points, I want to say that my reason for
improving Paul Kienzle's original MEX interface code and including it
in Octave was to allow Octave to run free software that uses MEX files
(my particular goal was to run SundialsTB in Octave).  The point was
to liberate that software from Matlab and increase the amount of free
software available to Octave users, not to enable people to write
proprietary code for Octave.  Regardless of what we decide for MEX
files, I still would not encourage anyone to write proprietary
additions to Octave.

On  7-Jan-2009, David Bateman wrote:

| But Octave, as well as being a program, is a numerical language. Your
| stand comes down to saying that any idea that is expressed in the Octave
| language is covered by the GPL... Thats not a reasonable position
| basically because if a person writes an m-file, that m-file might be run
| in Octave or Matlab and so clearly doesn't fall under the GPL.
|
| Or are you saying that if that m-file depends on a function that is only
| in Octave, and so can only run in Octave then its distribution is
| covered by the GPL?

No.  The GPL FAQ covers this topic in the question:

  Q:  If a programming language interpreter is released under the GPL,
  does that mean programs written to be interpreted by it must be
  under GPL-compatible licenses?

  A:  When the interpreter just interprets a language, the answer is
  no. The interpreted program, to the interpreter, is just data; a
  free software license like the GPL, based on copyright law, cannot
  limit what data you use the interpreter on. You can run it on any
  data (interpreted program), any way you like, and there are no
  requirements about licensing that data to anyone.

The answer for this question goes on to discuss what happens when
linking with software that is not merely interpreted:

  However, when the interpreter is extended to provide ?bindings? to
  other facilities (often, but not necessarily, libraries), the
  interpreted program is effectively linked to the facilities it uses
  through these bindings. So if these facilities are released under the
  GPL, the interpreted program that uses them must be released in a
  GPL-compatible way. The JNI or Java Native Interface is an example of
  such a binding mechanism; libraries that are accessed in this way are
  linked dynamically with the Java programs that call them. These
  libraries are also linked with the interpreter. If the interpreter is
  linked statically with these libraries, or if it is designed to link
  dynamically with these specific libraries, then it too needs to be
  released in a GPL-compatible way.

However, as you point out, the details of the MEX interface raise some
interesting issues.

  * The MEX interface is not unique to Octave (it's not even our
    interface).

  * It is easy enough to write and compile MEX files that do not even
    include mex.h or link to any Octave- or Matlab-specific libraries
    and that can be loaded and run by either program.  You don't even
    need Matlab or Octave on a computer in order to build a
    functioning MEX file.

  * Except for a small number of functions (mexCallMATLAB,
    mexEvalString, mexGetVariable, mexPutVariable, etc.) all a MEX
    file can do is

      - accept some data in mxArray format (not unique to Octave)
      - manipulate it via the mxFOO functions
      - call mexFOO functions
      - return data in the mxArray format.

  * Even the mexFOO functions (mexCallMATLAB, mexEvalString, etc.)
    only permit doing things that could be done in the scripting
    language itself (call functions, evaluation scripting language
    code, get/set variable values, etc.).

  * The MEX interface does not allow access to arbitrary internal
    functions of MATLAB (or Octave, in the Octave implementation).

OK.  Hmm.  It does seem to me that claiming a MEX file as a
derivative work of Octave is a bit of a stretch given the specifics of
this case.  But before we decide, I'd like to know what the FSF
position is for this case (I'll ask).

| The API of mexFunction is different between v4 and v5.. That is  
| "mxArray *prhs[]" is marked as const in V5 and later. So we can make the
| assumption that the RHS is const and use that to avoid a copy.

OK.  Changing this would be OK for me.  Should we even bother to try
to support code that uses the older interface?  Should mexproto.h
include a prototype for mexFunction so that we enforce the const
interface?

jwe
Reply | Threaded
Open this post in threaded view
|

Re: Further on MEX

John W. Eaton
Administrator
In reply to this post by David Bateman-2
On  7-Jan-2009, David Bateman wrote:

| Better to go for a fully matlab compatible MEX ABI
| and fall under the same argument as distribution of MEX code as source
| code.. That is of course if  we can't just consider that the current
| Octave MEX ABI isn't already  separate enough from Octave and closer to
| matlab that it doesn't already fall under the same argument.

What do you mean by ABI?  Do you mean that we should change Octave so
that a MEX file built with Matlab can run in Octave?

FYI, here is a MEX file that can be compiled on a system that does not
have either Octave or Matlab and that can be run in either program:

  typedef void mxArray;

  extern void mexPrintf (const char *fmt, ...);
  extern mxArray *mxDuplicateArray (const mxArray *v);

  void
  mexFunction (int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[])
  {
    int i;

    if (nlhs == nrhs)
      {
        mexPrintf ("copying inputs to outputs\n");
        for (i = 0; i < nrhs; i++)
          plhs[i] = mxDuplicateArray (prhs[i]);
      }
  }

Just compile with gcc -shared -fPIC -o foo.mex foo.c (you may need to
change the extension to mexglx so that Matlab can load it).

Since mxArray is an opaque type (all you ever need to use is a
pointer to it, and you never access contents directly) you don't need
to know its actual definition.  That's why the void typedef works and
you don't have to include mex.h for this struct definition.  You can
get the prototypes for the most of the interface from the
documentation.  The only sticking points are a few typedefs like
mxLogical, mxChar, mwSize, and mwIndex.  For those, you can probably
figure out what they are by trial and error.  Oh, and the only other
thing is you should avoid character data since Octave uses 8 bit
characters internally, but I think Matlab uses something wider.  So
direct character manipulation will probably crash either Matlab or
Octave (depending on what actual data type you choose for mxChar) or
I expect you will at least get some jumbled character strings.

jwe
Reply | Threaded
Open this post in threaded view
|

Re: Further on MEX

David Bateman-2
John W. Eaton wrote:

> On  7-Jan-2009, David Bateman wrote:
>
> | Better to go for a fully matlab compatible MEX ABI
> | and fall under the same argument as distribution of MEX code as source
> | code.. That is of course if  we can't just consider that the current
> | Octave MEX ABI isn't already  separate enough from Octave and closer to
> | matlab that it doesn't already fall under the same argument.
>
> What do you mean by ABI?  Do you mean that we should change Octave so
> that a MEX file built with Matlab can run in Octave?
>  

ABI = Application Binary Interface

> FYI, here is a MEX file that can be compiled on a system that does not
> have either Octave or Matlab and that can be run in either program:
>
>   typedef void mxArray;
>
>   extern void mexPrintf (const char *fmt, ...);
>   extern mxArray *mxDuplicateArray (const mxArray *v);
>
>   void
>   mexFunction (int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[])
>   {
>     int i;
>
>     if (nlhs == nrhs)
>       {
> mexPrintf ("copying inputs to outputs\n");
> for (i = 0; i < nrhs; i++)
>  plhs[i] = mxDuplicateArray (prhs[i]);
>       }
>   }
>
> Just compile with gcc -shared -fPIC -o foo.mex foo.c (you may need to
> change the extension to mexglx so that Matlab can load it).
>
> Since mxArray is an opaque type (all you ever need to use is a
> pointer to it, and you never access contents directly) you don't need
> to know its actual definition.  That's why the void typedef works and
> you don't have to include mex.h for this struct definition.  You can
> get the prototypes for the most of the interface from the
> documentation.  The only sticking points are a few typedefs like
> mxLogical, mxChar, mwSize, and mwIndex.  For those, you can probably
> figure out what they are by trial and error.  Oh, and the only other
> thing is you should avoid character data since Octave uses 8 bit
> characters internally, but I think Matlab uses something wider.  So
> direct character manipulation will probably crash either Matlab or
> Octave (depending on what actual data type you choose for mxChar) or
> I expect you will at least get some jumbled character strings.
>
>  
I'd thought that matlab mex-function that are pre-compiled couldn't be
used, and the above is a reasonable explanation of why. I had tried in
the past with a couple of compiled mex-files without success. I'd be all
for getting those typedefs right, even with workarounds for 8-bit
character strings coded as UTF8, and that would all ow even pre-built
mex-files to be used with Octave. We'd also then need to use the same
mex extensions as matlab itself does though.

D.

--
David Bateman                                [hidden email]
35 rue Gambetta                              +33 1 46 04 02 18 (Home)
92100 Boulogne-Billancourt FRANCE            +33 6 72 01 06 33 (Mob)

Reply | Threaded
Open this post in threaded view
|

Re: Further on MEX

John W. Eaton
Administrator
On  7-Jan-2009, David Bateman wrote:

| We'd also then need to use the same mex extensions as matlab itself
| does though.

Why?  Can't you just rename the files to change the extension when you
install them?

The only reason for having the different extensions is so that MEX
files for multiple platforms can be installed in the saem directory.
Doing that causes trouble for people packaging Octave, doesn't it?
The assumption is that binary files go in specific directories
(/usr/lib or /usr/libexec instead of /usr/share).  But we still have
problems because of the way the loadpath works (the assumption being
that you can add a package in Matlab by adding a directory to the
path).

jwe
Reply | Threaded
Open this post in threaded view
|

Re: Further on MEX

John W. Eaton
Administrator
In reply to this post by David Bateman-2
On  7-Jan-2009, David Bateman wrote:

| John W. Eaton wrote:
| > On  7-Jan-2009, David Bateman wrote:
| >
| > | Better to go for a fully matlab compatible MEX ABI
| > | and fall under the same argument as distribution of MEX code as source
| > | code.. That is of course if  we can't just consider that the current
| > | Octave MEX ABI isn't already  separate enough from Octave and closer to
| > | matlab that it doesn't already fall under the same argument.
| >
| > What do you mean by ABI?  Do you mean that we should change Octave so
| > that a MEX file built with Matlab can run in Octave?
| >  
|
| ABI = Application Binary Interface

Yes, I know, but I wanted to know what you specifically meant with
regard to Octave.  At what level of detail?  Do you expect Octave to
be able to be able to run a MEX file compiled with (any version of)
Matlab?  Do you expect Matlab to be able to run a MEX file compiled
with Octave?  I'm not sure that is practical.  That seems like one of
the areas where compatibility is just too much of a PITA.

jwe
Reply | Threaded
Open this post in threaded view
|

Re: Further on MEX

David Bateman-2
In reply to this post by John W. Eaton
John W. Eaton wrote:
> On  7-Jan-2009, David Bateman wrote:
>
> | We'd also then need to use the same mex extensions as matlab itself
> | does though.
>
> Why?  Can't you just rename the files to change the extension when you
> install them?
>  
Yes, you can. However, that is an additional step that a novice users
has to do to get it to work.
> The only reason for having the different extensions is so that MEX
> files for multiple platforms can be installed in the saem directory.
> Doing that causes trouble for people packaging Octave, doesn't it?
> The assumption is that binary files go in specific directories
> (/usr/lib or /usr/libexec instead of /usr/share).  But we still have
> problems because of the way the loadpath works (the assumption being
> that you can add a package in Matlab by adding a directory to the
> path).
>  
I thought the LSB specified architecture dependent and independent
directories and so the mex files should always be in different
directories for different platforms if packaged correctly.

D.

> jwe
>
>  


--
David Bateman                                [hidden email]
35 rue Gambetta                              +33 1 46 04 02 18 (Home)
92100 Boulogne-Billancourt FRANCE            +33 6 72 01 06 33 (Mob)

Reply | Threaded
Open this post in threaded view
|

Re: Further on MEX

David Bateman-2
In reply to this post by John W. Eaton
John W. Eaton wrote:

> On  7-Jan-2009, David Bateman wrote:
>
> | John W. Eaton wrote:
> | > On  7-Jan-2009, David Bateman wrote:
> | >
> | > | Better to go for a fully matlab compatible MEX ABI
> | > | and fall under the same argument as distribution of MEX code as source
> | > | code.. That is of course if  we can't just consider that the current
> | > | Octave MEX ABI isn't already  separate enough from Octave and closer to
> | > | matlab that it doesn't already fall under the same argument.
> | >
> | > What do you mean by ABI?  Do you mean that we should change Octave so
> | > that a MEX file built with Matlab can run in Octave?
> | >  
> |
> | ABI = Application Binary Interface
>
> Yes, I know, but I wanted to know what you specifically meant with
> regard to Octave.  At what level of detail?  Do you expect Octave to
> be able to be able to run a MEX file compiled with (any version of)
> Matlab?  Do you expect Matlab to be able to run a MEX file compiled
> with Octave?  I'm not sure that is practical.  That seems like one of
> the areas where compatibility is just too much of a PITA.
>  
That would be a nice goal and the ultimate proof that the MEX ABI can't
be under the GPL. I suppose we should try for it, but I'd say its a
where is becomes too much of a PITA we don't bother..

D.


--
David Bateman                                [hidden email]
35 rue Gambetta                              +33 1 46 04 02 18 (Home)
92100 Boulogne-Billancourt FRANCE            +33 6 72 01 06 33 (Mob)

Reply | Threaded
Open this post in threaded view
|

Re: Further on MEX

Thomas Weber-8
In reply to this post by David Bateman-2
Am Mittwoch, den 07.01.2009, 22:20 +0100 schrieb David Bateman:

> John W. Eaton wrote:
> > On  7-Jan-2009, David Bateman wrote:
> >
> > | We'd also then need to use the same mex extensions as matlab itself
> > | does though.
> >
> > Why?  Can't you just rename the files to change the extension when you
> > install them?
> >  
> Yes, you can. However, that is an additional step that a novice users
> has to do to get it to work.
> > The only reason for having the different extensions is so that MEX
> > files for multiple platforms can be installed in the saem directory.
> > Doing that causes trouble for people packaging Octave, doesn't it?
> > The assumption is that binary files go in specific directories
> > (/usr/lib or /usr/libexec instead of /usr/share).  But we still have
> > problems because of the way the loadpath works (the assumption being
> > that you can add a package in Matlab by adding a directory to the
> > path).
> >  
> I thought the LSB specified architecture dependent and independent
> directories

Yes[1].

> and so the mex files should always be in different
> directories for different platforms if packaged correctly.

No. The platform-specific files go into /usr/lib, platform-independent
files into /usr/share.
So, you will have the .mex file in a directory under /usr/lib, but that
directory name is the same for every architecture.

[1] Actually, it's the FHS, but the LSB has the FHS as a mandatory part.

Reply | Threaded
Open this post in threaded view
|

Re: Further on MEX

John W. Eaton
Administrator
On  8-Jan-2009, Thomas Weber wrote:

| Am Mittwoch, den 07.01.2009, 22:20 +0100 schrieb David Bateman:
| > John W. Eaton wrote:
| > I thought the LSB specified architecture dependent and independent
| > directories
|
| Yes[1].
|
| > and so the mex files should always be in different
| > directories for different platforms if packaged correctly.
|
| No. The platform-specific files go into /usr/lib, platform-independent
| files into /usr/share.
| So, you will have the .mex file in a directory under /usr/lib, but that
| directory name is the same for every architecture.

But aren't they installed in directories with names like

  /usr/lib/octave/3.0.3/oct/x86_64-pc-linux-gnu/qr.oct

so these files are installed in directories with names derived from
the architecture?

Am I missing something?

jwe
12