[OF geometry] un-syncable with upstreeam

[Juan Pablo Carbajal-2]

Hi Phillip, Of leaders, and maintainers,

I was trying to release geometry today. It is impossible! (well very hard).
The problem is that from the beginning I made the mistake of trying to
convert the package to Octave and Of standards... wrong!

So I am inclined to:
1. Reduce the current geometry package to contain only our extensions to matgeom
2. Put 3rd party geometry code into matgeom package (which will just
package what comes from matgeom developers)
3. mapping will have to depend on geometry, which in turns depends on matgeom

Now the BIG issue with this:
The reason I tried to convert matgeom to octave standard was to be
able to include the package in OF. This is currently a nightmare!
So by making an Octave package matgeom that can be easily synched with
upstream, I will have to abandon Octave and OF standards and matgeom
wont be hosted in OF anymore cause it will be not compliant.
This means installation of packages that depend on matgeom wont be
automatic, e.g.
pkg install -forge geometry
will always fail due to missing dependency matgeom.

What to do in this scenario?

One alternative I see is to fork only what our extensions actually
need from matgeom and put those in geometry in a OF complaint form
(definitely diverging from matgeom). Beware! This is not minor amount
of functions, and to do it in a reasonable time I will need help.

Is this scenario better?

I will really appreciate your inputs here!

Regards,

[Oliver Heimlich]

Am 14. August 2018 00:00:02 MESZ schrieb Juan Pablo Carbajal <[hidden email]>:

>Hi Phillip, Of leaders, and maintainers,
>
>I was trying to release geometry today. It is impossible! (well very
>hard).
>The problem is that from the beginning I made the mistake of trying to
>convert the package to Octave and Of standards... wrong!
>
>So I am inclined to:
>1. Reduce the current geometry package to contain only our extensions
>to matgeom
>2. Put 3rd party geometry code into matgeom package (which will just
>package what comes from matgeom developers)
>3. mapping will have to depend on geometry, which in turns depends on
>matgeom

Hi Juan,

thank you for trying to convert the matgeom package into our coding standards. Apparently, the lesson learned is, that this approach is not maintainable while upstream is still actively under development.

I agree that a third party matgeom package sounds like a good solution.


>Now the BIG issue with this:
>The reason I tried to convert matgeom to octave standard was to be
>able to include the package in OF. This is currently a nightmare!
>So by making an Octave package matgeom that can be easily synched with
>upstream, I will have to abandon Octave and OF standards and matgeom
>wont be hosted in OF anymore cause it will be not compliant.

Yes, the third party matgeom package, which mostly contains the upstream contents, would not be a candidate to be hosted on OF as an individual package.

>This means installation of packages that depend on matgeom wont be
>automatic, e.g.
>pkg install -forge geometry
>will always fail due to missing dependency matgeom.
>
>What to do in this scenario?

I can think of two possible solutions. You could check for the presence of matgeom during package installation of geometry. For example, in a configure script or Makefile. Or during runtime, where you have more options to query the user.

 1. If matgeom is missing, there could be an error message with instructions to download the third party dependency from elsewhere. Octave can 'pkg install' from external URLs since version 4.4.0, so it could be possible to offer automatic download and installation / let the user copy and paste the required command.

2. The third party dependency could be bundled in the release tarball of geometry to be installed if needed. We also bundle third party C libraries which don't match our coding style, so I see no problem in bundling a third party Octave package.

Both approaches overcome the installation problem for the user. However, you have to worry about more things to not produce new problems:

A) The matgeom package won't receive updates. How should a case be handled when an old matgeom version is already installed, but geometry needs a newer version?

B) Should a missing matgeom dependency be installed automatically? Should there be a query which the user can acknowledge?

C) Can/should we publish documentation for matgeom functions on OF as part of geometry's function reference? Probably not, if I understand correctly how you want to utilize matgeom.

>One alternative I see is to fork only what our extensions actually
>need from matgeom and put those in geometry in a OF complaint form
>(definitely diverging from matgeom). Beware! This is not minor amount
>of functions, and to do it in a reasonable time I will need help.
>
>Is this scenario better?

To me, this doesn't sound more maintainable than your initial approach. I'd recommend against it.

Oliver

[PhilipNienhuis]

hi JuanPi,

Juan Pablo Carbajal wrote:
> Hi Phillip, Of leaders, and maintainers,
>
> I was trying to release geometry today. It is impossible! (well very
hard).
> The problem is that from the beginning I made the mistake of trying to
> convert the package to Octave and Of standards... wrong!
>
> So I am inclined to:
> 1. Reduce the current geometry package to contain only our extensions
to matgeom
> 2. Put 3rd party geometry code into matgeom package (which will just
> package what comes from matgeom developers)
> 3. mapping will have to depend on geometry, which in turns depends on
matgeom

>
> Now the BIG issue with this:
> The reason I tried to convert matgeom to octave standard was to be
> able to include the package in OF. This is currently a nightmare!
> So by making an Octave package matgeom that can be easily synched with
> upstream, I will have to abandon Octave and OF standards and matgeom
> wont be hosted in OF anymore cause it will be not compliant.
> This means installation of packages that depend on matgeom wont be
> automatic, e.g.
> pkg install -forge geometry
> will always fail due to missing dependency matgeom.
>
> What to do in this scenario?
>
> One alternative I see is to fork only what our extensions actually
> need from matgeom and put those in geometry in a OF complaint form
> (definitely diverging from matgeom). Beware! This is not minor amount
> of functions, and to do it in a reasonable time I will need help.
>
> Is this scenario better?

AFAICS the underlying question is:
"Is it acceptable for an OF package to depend on an external package?",
right?
Because this is gonna happen either way as far as OF-mapping goes:
either mapping depends on an external geometry, or mapping depends on OF
geometry that depends on external matgeom.

At the very moment the decision was made (AFAIR with fairly good reason)
to have OF packages an external packages, I already figured there would
be corner cases; simply because every sort of "yes/no" or "in/out"
distinction inevitably brings those along.

My stance, from a practical POV, would be to just include matgeom as-is
in a subdir (probably with some hierarchy inside) of an OF geometry
package so that it can be somehow marked "external". As long as all
functions are documented and have copyright + license info I suppose
we're okay (that could be just some awk script or even sed).
There are already some wrapper functions linking to the matgeom
functions anyway.
Perhaps doumentation may need a bit of thinking over, but there already
is some, or even all docs.

TTBOMK there are existing examples: external gdalread.cc in
mapping/src/, slicot in control, clipperlib in geometry/src/, .... all
of which may or may not comply to formal OF standards.

HTH
Philip

[jbect]

Le 14/08/2018 à 00:00, Juan Pablo Carbajal a écrit :

> Hi Phillip, Of leaders, and maintainers,
>
> I was trying to release geometry today. It is impossible! (well very hard).
> The problem is that from the beginning I made the mistake of trying to
> convert the package to Octave and Of standards... wrong!
>
> So I am inclined to:
> 1. Reduce the current geometry package to contain only our extensions to matgeom
> 2. Put 3rd party geometry code into matgeom package (which will just
> package what comes from matgeom developers)
> 3. mapping will have to depend on geometry, which in turns depends on matgeom
>
> Now the BIG issue with this:
> The reason I tried to convert matgeom to octave standard was to be
> able to include the package in OF. This is currently a nightmare!
> So by making an Octave package matgeom that can be easily synched with
> upstream, I will have to abandon Octave and OF standards and matgeom
> wont be hosted in OF anymore cause it will be not compliant.

Hi,

What are the "Octave and OF standards" that you are trying to comply
with, and that are causing you troubles ?

Perhaps would it be feasible to have a set of scripts and/or a stack of
patches (manages, e.g, with quilt) that perform the conversion
automatically ?

@++
Julien

[Juan Pablo Carbajal-2]

Hi

Thanks all for the comments, specially Oliver, you understood my
problem pretty well.
To solve this issue as soon as possible i will go like this.

1. I do have a way to create a installable matgeom package, hence
geometry can depend on a given version of matgeom. I can add a PKG_ADD
directive in the geometry package to check the installed version of
matgeom.
2. I need to find a place to host the matgeom tar file. If we are
bundling non-OF-compliant code with distributions, can we not also
host non-OF-compliant needed packages, like this case?
  I see the following advantages: 1.Adds modularity to the package
development. 2. Packages with this "external dependencies" get
simpler, i.e. simpler makefile, configure scripts, etc... 3. Several
packages with same external dependency do not duplicate code (i.e.
smaller worst case tar files). What are the cons here?
  I can foresee resistance to follow this path due to the "how to
avoid these other boderline cases", but i can imagine, if we accept
bundled code, how this can be different. I guess we will also get
complaints about reproducibility of installation of older versions,
which is solved by keeping older tar of the external dep.
  If OF wont host the external dep matgeom (package, not repo), any
recommendations where to do it? SF? Bitbucket? Somewhere else?

Thanks
On Wed, Aug 15, 2018 at 9:41 PM Oliver Heimlich <[hidden email]> wrote:

>
>
>
> Am 14. August 2018 00:00:02 MESZ schrieb Juan Pablo Carbajal <[hidden email]>:
> >Hi Phillip, Of leaders, and maintainers,
> >
> >I was trying to release geometry today. It is impossible! (well very
> >hard).
> >The problem is that from the beginning I made the mistake of trying to
> >convert the package to Octave and Of standards... wrong!
> >
> >So I am inclined to:
> >1. Reduce the current geometry package to contain only our extensions
> >to matgeom
> >2. Put 3rd party geometry code into matgeom package (which will just
> >package what comes from matgeom developers)
> >3. mapping will have to depend on geometry, which in turns depends on
> >matgeom
>
> Hi Juan,
>
> thank you for trying to convert the matgeom package into our coding standards. Apparently, the lesson learned is, that this approach is not maintainable while upstream is still actively under development.
>
> I agree that a third party matgeom package sounds like a good solution.
>
>
> >Now the BIG issue with this:
> >The reason I tried to convert matgeom to octave standard was to be
> >able to include the package in OF. This is currently a nightmare!
> >So by making an Octave package matgeom that can be easily synched with
> >upstream, I will have to abandon Octave and OF standards and matgeom
> >wont be hosted in OF anymore cause it will be not compliant.
>
> Yes, the third party matgeom package, which mostly contains the upstream contents, would not be a candidate to be hosted on OF as an individual package.
>
> >This means installation of packages that depend on matgeom wont be
> >automatic, e.g.
> >pkg install -forge geometry
> >will always fail due to missing dependency matgeom.
> >
> >What to do in this scenario?
>
> I can think of two possible solutions. You could check for the presence of matgeom during package installation of geometry. For example, in a configure script or Makefile. Or during runtime, where you have more options to query the user.
>
>  1. If matgeom is missing, there could be an error message with instructions to download the third party dependency from elsewhere. Octave can 'pkg install' from external URLs since version 4.4.0, so it could be possible to offer automatic download and installation / let the user copy and paste the required command.
>
> 2. The third party dependency could be bundled in the release tarball of geometry to be installed if needed. We also bundle third party C libraries which don't match our coding style, so I see no problem in bundling a third party Octave package.
>
> Both approaches overcome the installation problem for the user. However, you have to worry about more things to not produce new problems:
>
> A) The matgeom package won't receive updates. How should a case be handled when an old matgeom version is already installed, but geometry needs a newer version?
>
> B) Should a missing matgeom dependency be installed automatically? Should there be a query which the user can acknowledge?
>
> C) Can/should we publish documentation for matgeom functions on OF as part of geometry's function reference? Probably not, if I understand correctly how you want to utilize matgeom.
>
> >One alternative I see is to fork only what our extensions actually
> >need from matgeom and put those in geometry in a OF complaint form
> >(definitely diverging from matgeom). Beware! This is not minor amount
> >of functions, and to do it in a reasonable time I will need help.
> >
> >Is this scenario better?
>
> To me, this doesn't sound more maintainable than your initial approach. I'd recommend against it.
>
> Oliver

[Juan Pablo Carbajal-2]

Hi all again.

Upstream matgeom has accepted my scripts to package to octave[1] and
they will host the files in their wiki[2].
The geometry package has been striped in the dev branch.

The only part missing for the next release is to decide whether
geometry should also install matgeom or not.
It would be easy to package the octave package matgeom inside the
geometry package and then add a line to install it when geometry is
installed. Is this recommended? (I understood that some packages are
already doing this)
Should I just give enough information for people to download or build
the matgeom package by themselves? can I trigger such info when "pkg
install -forge geometry" fails to install due to missing matgeom?

Regards,

[1]: https://github.com/mattools/matGeom/pull/63
[2]: http://matgeom.sourceforge.net/wiki/index.php/MatGeom
On Mon, Sep 10, 2018 at 4:00 PM Juan Pablo Carbajal
<[hidden email]> wrote:

>
> Hi
>
> Thanks all for the comments, specially Oliver, you understood my
> problem pretty well.
> To solve this issue as soon as possible i will go like this.
>
> 1. I do have a way to create a installable matgeom package, hence
> geometry can depend on a given version of matgeom. I can add a PKG_ADD
> directive in the geometry package to check the installed version of
> matgeom.
> 2. I need to find a place to host the matgeom tar file. If we are
> bundling non-OF-compliant code with distributions, can we not also
> host non-OF-compliant needed packages, like this case?
>   I see the following advantages: 1.Adds modularity to the package
> development. 2. Packages with this "external dependencies" get
> simpler, i.e. simpler makefile, configure scripts, etc... 3. Several
> packages with same external dependency do not duplicate code (i.e.
> smaller worst case tar files). What are the cons here?
>   I can foresee resistance to follow this path due to the "how to
> avoid these other boderline cases", but i can imagine, if we accept
> bundled code, how this can be different. I guess we will also get
> complaints about reproducibility of installation of older versions,
> which is solved by keeping older tar of the external dep.
>   If OF wont host the external dep matgeom (package, not repo), any
> recommendations where to do it? SF? Bitbucket? Somewhere else?
>
> Thanks
> On Wed, Aug 15, 2018 at 9:41 PM Oliver Heimlich <[hidden email]> wrote:
> >
> >
> >
> > Am 14. August 2018 00:00:02 MESZ schrieb Juan Pablo Carbajal <[hidden email]>:
> > >Hi Phillip, Of leaders, and maintainers,
> > >
> > >I was trying to release geometry today. It is impossible! (well very
> > >hard).
> > >The problem is that from the beginning I made the mistake of trying to
> > >convert the package to Octave and Of standards... wrong!
> > >
> > >So I am inclined to:
> > >1. Reduce the current geometry package to contain only our extensions
> > >to matgeom
> > >2. Put 3rd party geometry code into matgeom package (which will just
> > >package what comes from matgeom developers)
> > >3. mapping will have to depend on geometry, which in turns depends on
> > >matgeom
> >
> > Hi Juan,
> >
> > thank you for trying to convert the matgeom package into our coding standards. Apparently, the lesson learned is, that this approach is not maintainable while upstream is still actively under development.
> >
> > I agree that a third party matgeom package sounds like a good solution.
> >
> >
> > >Now the BIG issue with this:
> > >The reason I tried to convert matgeom to octave standard was to be
> > >able to include the package in OF. This is currently a nightmare!
> > >So by making an Octave package matgeom that can be easily synched with
> > >upstream, I will have to abandon Octave and OF standards and matgeom
> > >wont be hosted in OF anymore cause it will be not compliant.
> >
> > Yes, the third party matgeom package, which mostly contains the upstream contents, would not be a candidate to be hosted on OF as an individual package.
> >
> > >This means installation of packages that depend on matgeom wont be
> > >automatic, e.g.
> > >pkg install -forge geometry
> > >will always fail due to missing dependency matgeom.
> > >
> > >What to do in this scenario?
> >
> > I can think of two possible solutions. You could check for the presence of matgeom during package installation of geometry. For example, in a configure script or Makefile. Or during runtime, where you have more options to query the user.
> >
> >  1. If matgeom is missing, there could be an error message with instructions to download the third party dependency from elsewhere. Octave can 'pkg install' from external URLs since version 4.4.0, so it could be possible to offer automatic download and installation / let the user copy and paste the required command.
> >
> > 2. The third party dependency could be bundled in the release tarball of geometry to be installed if needed. We also bundle third party C libraries which don't match our coding style, so I see no problem in bundling a third party Octave package.
> >
> > Both approaches overcome the installation problem for the user. However, you have to worry about more things to not produce new problems:
> >
> > A) The matgeom package won't receive updates. How should a case be handled when an old matgeom version is already installed, but geometry needs a newer version?
> >
> > B) Should a missing matgeom dependency be installed automatically? Should there be a query which the user can acknowledge?
> >
> > C) Can/should we publish documentation for matgeom functions on OF as part of geometry's function reference? Probably not, if I understand correctly how you want to utilize matgeom.
> >
> > >One alternative I see is to fork only what our extensions actually
> > >need from matgeom and put those in geometry in a OF complaint form
> > >(definitely diverging from matgeom). Beware! This is not minor amount
> > >of functions, and to do it in a reasonable time I will need help.
> > >
> > >Is this scenario better?
> >
> > To me, this doesn't sound more maintainable than your initial approach. I'd recommend against it.
> >
> > Oliver

[jbect]

Le 28/09/2018 à 01:12, Juan Pablo Carbajal a écrit :

> Hi all again.
>
> Upstream matgeom has accepted my scripts to package to octave[1] and
> they will host the files in their wiki[2].
> The geometry package has been striped in the dev branch.
>
> The only part missing for the next release is to decide whether
> geometry should also install matgeom or not.
> It would be easy to package the octave package matgeom inside the
> geometry package and then add a line to install it when geometry is
> installed. Is this recommended? (I understood that some packages are
> already doing this)
> Should I just give enough information for people to download or build
> the matgeom package by themselves? can I trigger such info when "pkg
> install -forge geometry" fails to install due to missing matgeom?
>
> Regards,
>
> [1]: https://github.com/mattools/matGeom/pull/63
> [2]: http://matgeom.sourceforge.net/wiki/index.php/MatGeom

Hi JPi,

Why not host the octave package on OF as an external package ?

You would simply have a mercurial repo that is updated before each
release with the result of your script (using, e.g., "hg addremove").

This would make it possible for users to install matgeom using "pkg
install -forge" as any other OF package.

?

@++
Julien

[Juan Pablo Carbajal-2]

Hi Julien,

I am not sure why we have not contemplated that option, I guess I
forgot about it.

I will check the developers page to set the external repo.

Thanks

On Fri, Sep 28, 2018 at 9:47 AM Julien Bect
<[hidden email]> wrote:

>
> Le 28/09/2018 à 01:12, Juan Pablo Carbajal a écrit :
> > Hi all again.
> >
> > Upstream matgeom has accepted my scripts to package to octave[1] and
> > they will host the files in their wiki[2].
> > The geometry package has been striped in the dev branch.
> >
> > The only part missing for the next release is to decide whether
> > geometry should also install matgeom or not.
> > It would be easy to package the octave package matgeom inside the
> > geometry package and then add a line to install it when geometry is
> > installed. Is this recommended? (I understood that some packages are
> > already doing this)
> > Should I just give enough information for people to download or build
> > the matgeom package by themselves? can I trigger such info when "pkg
> > install -forge geometry" fails to install due to missing matgeom?
> >
> > Regards,
> >
> > [1]: https://github.com/mattools/matGeom/pull/63
> > [2]: http://matgeom.sourceforge.net/wiki/index.php/MatGeom
>
>
> Hi JPi,
>
> Why not host the octave package on OF as an external package ?
>
> You would simply have a mercurial repo that is updated before each
> release with the result of your script (using, e.g., "hg addremove").
>
> This would make it possible for users to install matgeom using "pkg
> install -forge" as any other OF package.
>
> ?
>
> @++
> Julien
>
>