Default PATH for executing external programs

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Default PATH for executing external programs

John W. Eaton
Administrator
Currently, Octave sets the PATH for searching for executable programs
to be the user's PATH followed by Other directories under OCTAVE_HOME
that may also contain programs to execute.  By setting things this
way, we may end up executing the wrong version of mkoctfile (for
example).  Octave's mkoctfile function takes care of this by
explicitly executing the copy of the external mkoctfile program found
in Octave's $bindir, but that is not done when executing the external
mkoctfile program using "pkg install" because "pkg install" just
executes Make, and there is no way to know how the package Makefiles
will execute mkoctfile.

There was a bug report about this issue for Windows:

   https://savannah.gnu.org/bugs/?func=detailitem&item_id=43164

The solution was to prepend Octave's $bindir to the user's PATH before
starting Octave.  That works, but shouldn't Octave be doing this
itself?

Or perhaps it should not be doing this at all, because it doesn't
allow the user to easily override that choice.

Another option is to have Octave export MKOCTFILE (with a default
setting, but not overriding the value found in the environment, if
any) and require package Makefiles to use this variable if they want
to behave properly.

jwe

Reply | Threaded
Open this post in threaded view
|

Re: Default PATH for executing external programs

Philip Nienhuis
John W. Eaton wrote
Currently, Octave sets the PATH for searching for executable programs
to be the user's PATH followed by Other directories under OCTAVE_HOME
that may also contain programs to execute.  By setting things this
way, we may end up executing the wrong version of mkoctfile (for
example).  Octave's mkoctfile function takes care of this by
explicitly executing the copy of the external mkoctfile program found
in Octave's $bindir, but that is not done when executing the external
mkoctfile program using "pkg install" because "pkg install" just
executes Make, and there is no way to know how the package Makefiles
will execute mkoctfile.

There was a bug report about this issue for Windows:

   https://savannah.gnu.org/bugs/?func=detailitem&item_id=43164

The solution was to prepend Octave's $bindir to the user's PATH before
starting Octave.  That works, but shouldn't Octave be doing this
itself?
(Actually, you stirr up two separate but related issues: polluted PATHs and "pkg" command robustness)

At least on Windows, I think Octave rather shouldn't put its bin subdir after the current PATH. On my local 3.8.0 (MinGW) installation, started with this shortcut:

X:\Octave\Octave-3.8.0\bin\octave-gui.exe

...Octave gets to work in an environment with this PATH:

>> strsplit (getenv ("PATH"), pathsep)
ans =
{
  [1,1] = C:\Programs\Python33\
  [1,2] = C:\WINDOWS\system32
  [1,3] = C:\WINDOWS
  [1,4] = C:\WINDOWS\System32\Wbem
  [1,5] = C:\Programs\TortoiseSVN\bin
  [1,6] = C:\Programs\gs\gs9.04\bin
  [1,7] = C:\Programs\QuickTime\QTSystem\
  [1,8] = C:\Programs\Mercurial
  [1,9] = C:\Programs\VMware\VMware View\Client\bin
  [1,10] = X:\Octave\Octave-3.8.0\bin
  [1,11] = X:\Octave\Octave-3.8.0\notepad++
  [1,12] = X:\Octave\Octave-3.8.0\libexec\octave\3.8.0\site\exec\i686-pc-mingw32
  [1,13] = X:\Octave\Octave-3.8.0\libexec\octave\api-v49+\site\exec\i686-pc-mingw32
  [1,14] = X:\Octave\Octave-3.8.0\libexec\octave\site\exec\i686-pc-mingw32
  [1,15] = X:\Octave\Octave-3.8.0\libexec\octave\3.8.0\exec\i686-pc-mingw32
  [1,16] = X:\Octave\Octave-3.8.0\bin
}
>>

from where one can see that Octave's bindir has been added twice to the PATH.

Now I'm usually quite careful on my own systems, but especially unsuspecting if not ignorant Windows users can have PATHs badly polluted by sloppily written Windows applications.
("sloppily written" = my judgement of programs that silently put their own path-to-executables into the "static" Windows system PATH without further ado).
On the one hand, luckily those PATH entries are usually but not always appended rather than prepended; on the other hand, from the example PATH above one can see that -say- "trustworthy" and even well-known programs like Python, Mercurial, Quicktime and SVN behave this way as well. Some even leave a trailing backslash in their PATH entry (whose only harm is witnessing of some lack of attention to detail of their developers :-) ).

For example, Microsoft's own MS-Office doesn't need system PATH entries, so clearly there are (IMO superior) alternatives.

Octave started with a .bat file that puts Octave's bindir in front of all other PATH entries has this PATH:

>> strsplit (getenv ("PATH"), pathsep)
ans =
{
  [1,1] = X:\Octave\Octave-3.8.2\bin\
  [1,2] = C:\WINDOWS\system32
  [1,3] = C:\WINDOWS
  [1,4] = C:\WINDOWS\system32\wbem
  [1,5] = X:\Octave\Octave-3.8.2\bin
  [1,6] = X:\Octave\Octave-3.8.2\notepad++
  [1,7] = X:\Octave\Octave-3.8.2\libexec\octave\3.8.2\site\exec\i686-w64-mingw32
  [1,8] = X:\Octave\Octave-3.8.2\libexec\octave\api-v49+\site\exec\i686-w64-mingw32
  [1,9] = X:\Octave\Octave-3.8.2\libexec\octave\site\exec\i686-w64-mingw32
  [1,10] = X:\Octave\Octave-3.8.2\libexec\octave\3.8.2\exec\i686-w64-mingw32
  [1,11] = X:\Octave\Octave-3.8.2\bin
}
>>

...featuring Octave's bin dir even three times.

Or perhaps it should not be doing this at all, because it doesn't
allow the user to easily override that choice.

Another option is to have Octave export MKOCTFILE (with a default
setting, but not overriding the value found in the environment, if
any) and require package Makefiles to use this variable if they want
to behave properly.
First off, I think the whole issue is primarily a Windows one. I have never encountered it on my Linux systems.
Furthermore, mkoctfile is just one example; but Octave's bindir on MinGW also contains many MSYS utilities (less, gzip, etc.) that probably work better with the Octave binary they came with, than other MSYS utilities that may happen to be elsewhere on the system but appear earlier in the PATH. Not to mention equally named executables that serve different purposes; not all Windows developers know about *nix utilities.

I think that it doesn't harm at all to have the bin dir of a program appearing first in the PATH search order;  at least on Windows (but I wouldn't know why it wouldn't work the same way on other OS-es).
After all, it is reasonable to expect that a program (say, Octave) would need its own dependencies (associated executables, its own libraries) much more often than other executables and libraries elsewhere on the host system.
AFAIK programs generally run in their own "bubble" (RAM slot, "local" copy of environment, "local" state of system driver data, etc.) where other programs have no access to. So it can be argued that other program's PATH entries have no use in a program's state space unless those other programs are invoked by the first program (in wich case they inherit their own environment settings incl. PATH from the first program).

A batch file as shown in bug #3164 may not be "superior", but it is effective and IMO even somewhat elegant.
The only side-effect I can imagine is that it creates another "state space" with environment settings (i.e., the one owned by the cmd32.exe invocation that executes the batch file) but as Octave is "started" (= detached) that "state space" is soon left and erased by the final "exit" command in the batch file while Octave is starting.

So yes, I think it would be reasonable for Octave to prepend its own $bindir to the PATH.
But of course that doesn't preclude making the "pkg" command more robust.

But note, I'm not very familiar with the nitty-gritty of how executables etc. are searched, loaded in memory and launched in today's operating systems. My own experience dates back from the late 1980-ies when I did my CS minor and things have become much more complicated since.

Philip
Reply | Threaded
Open this post in threaded view
|

Re: Default PATH for executing external programs

John W. Eaton
Administrator
On 11/07/2014 06:31 AM, Philip Nienhuis wrote:

> First off, I think the whole issue is primarily a Windows one. I have never
> encountered it on my Linux systems.

It is a problem if you have multiple versions of Octave installed and at
least one version is in your PATH.  Then if you execute one of the
others (not first in the PATH) the pkg command will execute Make and any
package Makefile that uses "mkoctfile" will get the mkoctfile that
corresponds to the copy of Octave that is first in the PATH instead of
the one that corresponds to the one currently running.  That will cause
trouble because mkoctfile has things like the location of Octave header
files and libraries embedded in it.  Those things are obviously specific
to a particular version.

jwe


Reply | Threaded
Open this post in threaded view
|

Re: Default PATH for executing external programs

Carnë Draug
In reply to this post by John W. Eaton
On 6 November 2014 18:51, John W. Eaton <[hidden email]> wrote:

> Currently, Octave sets the PATH for searching for executable programs
> to be the user's PATH followed by Other directories under OCTAVE_HOME
> that may also contain programs to execute.  By setting things this
> way, we may end up executing the wrong version of mkoctfile (for
> example).  Octave's mkoctfile function takes care of this by
> explicitly executing the copy of the external mkoctfile program found
> in Octave's $bindir, but that is not done when executing the external
> mkoctfile program using "pkg install" because "pkg install" just
> executes Make, and there is no way to know how the package Makefiles
> will execute mkoctfile.
>
> There was a bug report about this issue for Windows:
>
>   https://savannah.gnu.org/bugs/?func=detailitem&item_id=43164
>
> The solution was to prepend Octave's $bindir to the user's PATH before
> starting Octave.  That works, but shouldn't Octave be doing this
> itself?
>
> Or perhaps it should not be doing this at all, because it doesn't
> allow the user to easily override that choice.
>
> Another option is to have Octave export MKOCTFILE (with a default
> setting, but not overriding the value found in the environment, if
> any) and require package Makefiles to use this variable if they want
> to behave properly.

Many OF packages already check if there is a MKOCTFILE environment
variable and will use it instead of searching the path. This seems like a
reasonable option. I wonder if it makes sense to have it stored in
octave_config_info as is the path for most directories and mkoctfile flags.

Carnë

Reply | Threaded
Open this post in threaded view
|

Re: Default PATH for executing external programs

John W. Eaton
Administrator
On 11/07/2014 04:17 PM, Carnë Draug wrote:

> Many OF packages already check if there is a MKOCTFILE environment
> variable and will use it instead of searching the path. This seems like a
> reasonable option. I wonder if it makes sense to have it stored in
> octave_config_info as is the path for most directories and mkoctfile flags.

I suppose we could do that to avoid duplicating the same logic in
multiple places (mkoctfile.m, pkg.m, somewhere else later?).  Then the
value just needs to be exported to the environment when Octave starts?
Or should it only be exported as needed?

jwe



Reply | Threaded
Open this post in threaded view
|

Re: Default PATH for executing external programs

Mike Miller
On Fri, Nov 7, 2014 at 16:21:55 -0500, John W. Eaton wrote:
> I suppose we could do that to avoid duplicating the same logic in multiple
> places (mkoctfile.m, pkg.m, somewhere else later?).  Then the value just
> needs to be exported to the environment when Octave starts? Or should it
> only be exported as needed?

Hmm, I think we already consider it a bug when OF packages do not use

  MKOCTFILE ?= mkoctfile

or equivalent in their Makefile. I know I test with multiple Octave
versions in PATH at the same time and would consider it a bug if a
package built against the wrong Octave.

I'm a little confused as to what change is being proposed, as we
already do export MKOCTFILE, OCTAVE_CONFIG, and OCTAVE when building
packages [1]. In your original message, you claim that "pkg install"
may call the wrong version of mkoctfile, but I think that sounds
rather like a buggy package not relying on the MKOCTFILE variable that
Octave is already providing. Or am I missing something?

In fact, I don't think Octave's $bindir should even need to be on PATH
for Octave's own programs to work from inside the interpreter. Unless
it's for the use of other helper programs like gnuplot or gs that
happen to be in the same bin directory and bundled with the Octave
install on Windows for example.

Can you clarify or show an example of a package that doesn't install properly?

[1] http://hg.savannah.gnu.org/hgweb/octave/file/39a69f54417e/scripts/pkg/private/configure_make.m#l50

--
mike

Reply | Threaded
Open this post in threaded view
|

Re: Default PATH for executing external programs

PhilipNienhuis
In reply to this post by John W. Eaton
John W. Eaton wrote:

> On 11/07/2014 06:31 AM, Philip Nienhuis wrote:
>
>> First off, I think the whole issue is primarily a Windows one. I have
>> never
>> encountered it on my Linux systems.
>
> It is a problem if you have multiple versions of Octave installed and at
> least one version is in your PATH.  Then if you execute one of the
> others (not first in the PATH) the pkg command will execute Make and any
> package Makefile that uses "mkoctfile" will get the mkoctfile that
> corresponds to the copy of Octave that is first in the PATH instead of
> the one that corresponds to the one currently running.  That will cause
> trouble because mkoctfile has things like the location of Octave header
> files and libraries embedded in it.  Those things are obviously specific
> to a particular version.

Thank you for this explanation.
As I understand I've just been lucky; the situation you describe is what
I have on my Linux systems. It must have worked because the Octave
libraries for my installed versions probably didn't differ that much.

Another thing here is that binary parts of OF packages installed in
different Octave versions end up in one and the same tree under
...local/lib/packages, contrary to the .oct files from Octave itself;
similar to the .m-file parts.
What I found is that I can use OF packages built with Octave-3.9.0+
without any problems with Octave-4.1.0+ [*] and some vice-versa; but not
with Octave-3.8.2. So I think it may be needed to separate the binary
parts of OF packages as well according to Octave version - if one wants
different Octave versions peacefully co-installed.

Philip

[*] Several OF packages (incl. some in mxe-octave) can't be built
anymore in 4.1.0+ as they depend on deprecated octave-map. But they can
still be built in 3.9.0+ and then run fine in 4.1.0+.
I regularly build combined 3.9.0+/4.1.0+ mxe versions for MingW, more or
less following patch #8469