Robustness against accidental shadowing of 'core' functions

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Robustness against accidental shadowing of 'core' functions

Tasos Papastylianou
Hi,

We managed to spark an interesting conversation on stack overflow over this question: https://stackoverflow.com/q/52174614/4183191
I thought it might be more fruitful to continue this in the mailing list.

My own opinion is that functions that are relied upon for 'core' functionality (such as 'glob' seemingly being used internally by 'pkg') should be called in a more carefully robust way (i.e. via 'builtin'), to prevent accidental breakage of core functionality in the event that an external package manages to shadow that function for whatever reason (good or bad).

I agree that ultimately the package developer is largely responsible for not breaking core functionality by inappropriate shadowing, but this does not negate the argument for built-in robustness against such errors. The example in the stackoverflow question is a nice one demonstrating this point, since the developer of Lightspeed was not negligent or acting in bad faith; they simply built a matlab package which works as intended on matlab, but happens to break octave in the manner stated above.

I hesitate to call this a bug, since, after all, we're talking about a package that was not written specifically with octave in mind. At the same time, however, it is entirely conceivable that had 'pkg' used the builtin 'glob' rather than whichever happened to be path-available at the time, then the package would work as expected.

Thoughts?

Tasos.



Reply | Threaded
Open this post in threaded view
|

Re: Robustness against accidental shadowing of 'core' functions

siko1056
On Wed, Sep 5, 2018 at 3:14 PM Tasos Papastylianou <[hidden email]> wrote:
Hi,

We managed to spark an interesting conversation on stack overflow over this question: https://stackoverflow.com/q/52174614/4183191
I thought it might be more fruitful to continue this in the mailing list.

My own opinion is that functions that are relied upon for 'core' functionality (such as 'glob' seemingly being used internally by 'pkg') should be called in a more carefully robust way (i.e. via 'builtin'), to prevent accidental breakage of core functionality in the event that an external package manages to shadow that function for whatever reason (good or bad).

There is just one spot where "glob" is used inside "pkg" to tell if a file exists [1].  If the developers of "Lightspeed" adapted their "glob" interface to Octave's, the "pkg" function might run perfectly with it.

I agree that ultimately the package developer is largely responsible for not breaking core functionality by inappropriate shadowing, but this does not negate the argument for built-in robustness against such errors.

The examples of accidental function shadowing are even worse, sometimes trivial, sometimes subtile.  See some examples from our mailing-list [2-4].  Just look for "shadow" and you'll find some.  That is the price to pay for this feature, like all languages offer features, that have to be used with care (bad memory access [off by one errors] in C, MS Office macro malware, ...).  You cannot simply defeat against/imagine as developer all of the mistakes of bad intentions of human beings.
 
The example in the stackoverflow question is a nice one demonstrating this point, since the developer of Lightspeed was not negligent or acting in bad faith; they simply built a matlab package which works as intended on matlab, but happens to break octave in the manner stated above.

If the package is so prominent, file a bug report [5] for Octave compatibility or fork the project (MIT licence) if the developers are not responsive (4 open bugs, last commit 10 months ago, 17 commits in total).

The problem described on stackoverflow is prettily easy solved: Do the Octave stuff first (pkg install...), then load the non-Octave stuff.
 
I hesitate to call this a bug, since, after all, we're talking about a package that was not written specifically with octave in mind. At the same time, however, it is entirely conceivable that had 'pkg' used the builtin 'glob' rather than whichever happened to be path-available at the time, then the package would work as expected.

Thoughts?

Tasos.

Sure this is no bug, it's a feature to defend against packages, that don't care about Octave ;-)  *joke* Honestly, we are talking about a non-Matlab function "glob", thus it is not a Matlab compatibility issue if it is not working.

Kai.




Reply | Threaded
Open this post in threaded view
|

Re: Robustness against accidental shadowing of 'core' functions

apjanke-floss
In reply to this post by Tasos Papastylianou


On 9/5/18 9:14 AM, Tasos Papastylianou wrote:
 >
 > My own opinion is that functions that are relied upon for 'core'
 > functionality (such as 'glob' seemingly being used internally by 'pkg')
 > should be called in a more carefully robust way (i.e. via 'builtin'), to
 > prevent accidental breakage of core functionality in the event that an
 > external package manages to shadow that function for whatever reason
 > (good or bad).
 >
 > [...]
 >
 > Thoughts?

Namespaces seem like a natural fit here, at least for some of the problem.

If you were designing Octave from scratch now, and wanted to make Matlab
compatibility a priority, you could have an "octave" namespace that
contained all the functions supplied by Octave that are not present in
Matlab. That way you only have one global name that's subject to
collision, and one that users are unlikely to use themselves. Similarly,
functions for Octave's internal use that shouldn't be called by user
code could go in "octave.internal". It would be "future-proof", too: you
wouldn't have to worry about future versions of Matlab introducing a
"glob()" function with incompatible semantics. Perhaps future
Octave-specific global functions and classes should go in a namespace,
to avoid this issue.

Matlab seems like it might be going this way, too, with a +matlab
namespace hierarchy.

Seems like it would be a lot of work (and code uglification) to try to
defend against this shadowing everywhere, though, with either namespaces
or "builtin".


On 9/5/18 4:57 PM, Kai Torben Ohlhus wrote:

 > Honestly, we are talking about a
 > non-Matlab function "glob", thus it is not a Matlab compatibility issue
 > if it is not working.

The user has a program that works under Matlab, but fails under Octave.
That sounds to me like the definition of a Matlab compatibility issue.
The space of global identifiers which are available for user functions
without interfering with the system's functioning is part of the
language/platform's behavior.

Cheers,
Andrew


Reply | Threaded
Open this post in threaded view
|

Re: Robustness against accidental shadowing of 'core' functions

Mike Miller-4
In reply to this post by siko1056
On Wed, Sep 05, 2018 at 22:57:20 +0200, Kai Torben Ohlhus wrote:
> The examples of accidental function shadowing are even worse, sometimes
> trivial, sometimes subtile.  See some examples from our mailing-list
> [2-4].  Just look for "shadow" and you'll find some.  That is the price to
> pay for this feature, like all languages offer features, that have to be
> used with care (bad memory access [off by one errors] in C, MS Office macro
> malware, ...).  You cannot simply defeat against/imagine as developer all
> of the mistakes of bad intentions of human beings.

I agree with Kai in general here. The ability to shadow core function
names is both a feature and a problem. There are going to continue to be
problems, and users and developers both have to be aware of it and deal
with it.

> If the package is so prominent, file a bug report [5] for Octave
> compatibility or fork the project (MIT licence) if the developers are not
> responsive (4 open bugs, last commit 10 months ago, 17 commits in total).

I agree, this looks like a typical Octave porting issue. The developer
of the project in question just needs to be aware of some small Octave
compatibility issues and work with them.

--
mike



signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Robustness against accidental shadowing of 'core' functions

Mike Miller-4
In reply to this post by apjanke-floss
On Wed, Sep 05, 2018 at 18:29:19 -0400, Andrew Janke wrote:

> Namespaces seem like a natural fit here, at least for some of the problem.
>
> If you were designing Octave from scratch now, and wanted to make Matlab
> compatibility a priority, you could have an "octave" namespace that
> contained all the functions supplied by Octave that are not present in
> Matlab. That way you only have one global name that's subject to collision,
> and one that users are unlikely to use themselves. Similarly, functions for
> Octave's internal use that shouldn't be called by user code could go in
> "octave.internal". It would be "future-proof", too: you wouldn't have to
> worry about future versions of Matlab introducing a "glob()" function with
> incompatible semantics. Perhaps future Octave-specific global functions and
> classes should go in a namespace, to avoid this issue.
We do that to some extent. For example, Matlab function names are not
allowed to start with underscores. Many private or internal Octave
functions start with double underscores. Also Matlab-provided functions
seem to avoid underscores in general, while many Octave-only functions
do include underscores.

I think that standard C library function names should be kept as is. For
example, the functions getgrent, getpwent, glob, and unlink are all
simple wrappers for the corresponding C functions. If namespaced, maybe
a "libc" namespace would be appropriate.

Another strategy for future-proofing might be to provide the full
implementation of a function, let's say glob, as a compiled function
__glob__. The the glob name could be a convenience wrapper for the user.
All Octave functions can safely call the underlying __glob__ function
with less danger of being shadowed.

> Seems like it would be a lot of work (and code uglification) to try to
> defend against this shadowing everywhere, though, with either namespaces or
> "builtin".

Agree.

> The user has a program that works under Matlab, but fails under Octave.

Not really. In this particular case, the error is happening when the
user tries to call the Octave pkg command after loading another
third-party module into their global namespace. If they don't use the
pkg command, there is no failure.

--
mike



signature.asc (849 bytes) Download Attachment