OCL package 1.0.0 released

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

OCL package 1.0.0 released

Matthias W. Klein
Hi Octave folks,

a new release of the OCL package providing OpenCL support for
parallelized calculations (including GPGPU) is available [1], version
1.0.0 .

It is the first package release which is fully compatible with all
recently released octave versions.  (Thanks to the "communications"
package, I learned a crucial octave programming technique.)  A summary
of all important user-visible changes is available online [2].

Users and developers can choose to either download and install the
package tarball [3], or clone the repo [4] and use 'make dist' and other
Makefile targets for local testing.

Feedback is welcome, by e-mail or project ticket.

@Octave Forge maintainers: Can the package become officially
affiliated to Octave Forge, after your fair testing?

Best,
Matt

[1] https://sourceforge.net/p/octave-ocl/
[2] https://sourceforge.net/p/octave-ocl/code/ci/default/tree/NEWS
[3] https://sourceforge.net/projects/octave-ocl/files/latest/download
[4] hg clone http://hg.code.sf.net/p/octave-ocl/code octave-ocl-code

Reply | Threaded
Open this post in threaded view
|

Re: OCL package 1.0.0 released

Matthias W. Klein
> Dear Matthias,
>
> I have managed to install your package but am getting the error
> message below.
>
> I am using ubuntu linux 14.04LTS. I am not sure whether opencl
> is completely installed?
>
> I do not know what the correct name of the opencl package is for
> ubuntu 14.04 so that I can specify this on the synaptic package
> manager?
>
> How do I specify that your package (or opencl) should use the
> core i7 on my machine?
>
> My machine also has a radeon graphics card (but I do not know
> how to determine the exact model number and specifications of
> the radeon graphics card, etc).
>
> How do I specify that your package (or opencl) should rather use
> the radeon graphics card for the computations (as this may be
> faster)?
>
> Thank you very much for any assistance with regard to the above.
>
> Best wishes,
> Constantinos
>
>
> octave:24> ocl_tests
> Testing ocl_constant function...
> Testing ocl_lib function...
> Testing ocl_context function...
> ocl: calling OpenCL function 'clGetPlatformIDs'
>   returned error 'CL_PLATFORM_NOT_FOUND_KHR' (-1001).
>   Please check your OpenCL installation.
> error: ocl_context: OpenCL function call error
> error: called from:
> error:   /home/cfrangos/octave/ocl-1.0.0/ocl_tests.m at line 85, column 11
> octave:24>
>
>
>
>
Constantinos,

the OCL package comes with a README file, in the doc/ subdirectory.  You
can use the octave command 'pkg list' to find your directory where the
package is installed, in order to get to the file.  The README contains
comprehensive info on using the package, including a "troubleshooting"
section distinguishing OpenCL installation issues from OCL issues.

 From your messages, I estimate that your system contains an OpenCL
library (=the "ICD loader"), but no specific hardware device drivers (or
none correctly installed; see also your /etc/OpenCL/vendors/* in Linux).
  You can find lots of infos on OpenCL Linux driver installation in the web.

Once you have your drivers accessible, you can use the OCL function
'ocl_context' to list and choose your hardware to use - type 'help
ocl_context' to see its syntax help.

Best,
Matt


Reply | Threaded
Open this post in threaded view
|

Re: OCL package 1.0.0 released

Constantinos Frangos
Hi Matthias,

Thanks for the detailed response.

Any help with the following would be appreciated.

(1) The directory /etc/OpenCL/vendors/ contains only one file
called intel.icd and the contents is as follows.
/usr/lib/beignet/libcl.so

(2) I tried the command ocl_context ("get_resources") and the
output is given below. Not clear what this indicates?

(3) Please provide some information on the relevant opencl linux
hardware driver for 4th generation intel core i7 and I will try to
download this from the internet.

(The information I found on the internet was not clear? The easiest
would be to locate the name of the relevant ubuntu 14.04 package
on the synaptic package manager.)

Thank you.

Best regards,
Constantinos

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

octave:32> d = ocl_context ("get_resources")
d =

  scalar structure containing the fields:

    platforms =
    {
      [1,1] =

        scalar structure containing the fields:

          platform_index = 0
          name = Experiment Intel Gen OCL Driver
          version = OpenCL 1.1 beignet 0.3
          profile = FULL_PROFILE
          vendor = Intel
          extensions = cl_khr_global_int32_base_atomics
cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics
cl_khr_local_int32_extended_atomics cl_khr_byte_addressable_store
cl_khr_icd

    }

On 8/12/19, Matthias W. Klein <[hidden email]> wrote:

>> Dear Matthias,
>>
>> I have managed to install your package but am getting the error
>> message below.
>>
>> I am using ubuntu linux 14.04LTS. I am not sure whether opencl
>> is completely installed?
>>
>> I do not know what the correct name of the opencl package is for
>> ubuntu 14.04 so that I can specify this on the synaptic package
>> manager?
>>
>> How do I specify that your package (or opencl) should use the
>> core i7 on my machine?
>>
>> My machine also has a radeon graphics card (but I do not know
>> how to determine the exact model number and specifications of
>> the radeon graphics card, etc).
>>
>> How do I specify that your package (or opencl) should rather use
>> the radeon graphics card for the computations (as this may be
>> faster)?
>>
>> Thank you very much for any assistance with regard to the above.
>>
>> Best wishes,
>> Constantinos
>>
>>
>> octave:24> ocl_tests
>> Testing ocl_constant function...
>> Testing ocl_lib function...
>> Testing ocl_context function...
>> ocl: calling OpenCL function 'clGetPlatformIDs'
>>   returned error 'CL_PLATFORM_NOT_FOUND_KHR' (-1001).
>>   Please check your OpenCL installation.
>> error: ocl_context: OpenCL function call error
>> error: called from:
>> error:   /home/cfrangos/octave/ocl-1.0.0/ocl_tests.m at line 85, column
>> 11
>> octave:24>
>>
>>
>>
>>
> Constantinos,
>
> the OCL package comes with a README file, in the doc/ subdirectory.  You
> can use the octave command 'pkg list' to find your directory where the
> package is installed, in order to get to the file.  The README contains
> comprehensive info on using the package, including a "troubleshooting"
> section distinguishing OpenCL installation issues from OCL issues.
>
>  From your messages, I estimate that your system contains an OpenCL
> library (=the "ICD loader"), but no specific hardware device drivers (or
> none correctly installed; see also your /etc/OpenCL/vendors/* in Linux).
>   You can find lots of infos on OpenCL Linux driver installation in the
> web.
>
> Once you have your drivers accessible, you can use the OCL function
> 'ocl_context' to list and choose your hardware to use - type 'help
> ocl_context' to see its syntax help.
>
> Best,
> Matt
>
>

Reply | Threaded
Open this post in threaded view
|

Re: OCL package 1.0.0 released

PhilipNienhuis
In reply to this post by Matthias W. Klein
Matthias W. Klein wrote

> Hi Octave folks,
>
> a new release of the OCL package providing OpenCL support for
> parallelized calculations (including GPGPU) is available [1], version
> 1.0.0 .
>
> It is the first package release which is fully compatible with all
> recently released octave versions.  (Thanks to the "communications"
> package, I learned a crucial octave programming technique.)  A summary
> of all important user-visible changes is available online [2].
>
> Users and developers can choose to either download and install the
> package tarball [3], or clone the repo [4] and use 'make dist' and other
> Makefile targets for local testing.
>
> Feedback is welcome, by e-mail or project ticket.

I just tested ocl-1.0 on a cross-built dev Octave 6.0.0 on my Windows 7 box
with 2 graphics HW, i.e., built-in Intel plus an NVidia graphics card.
Results: ocl_tests works fine & gives no errors and the example in the
doc/README works fine compared to mimicking the commands in Octave.

While installing/building the package I got a LOT of errors along the lines
of:

:
C:/Programs/Octave/OCTAVE~1.0_2/mingw64/bin/mkoctfile-6.0.0.exe --verbose -c
ocl_ov_matrix.cc
g++ -c
-I/home/philip/devel/octdev/mxe/mxe_64b_20190808/usr/x86_64-w64-mingw32/include
-IC:\Progr
ams\Octave\OCTAVE~1.0_2\mingw64\include\octave-6.0.0\octave\..
-IC:\Programs\Octave\OCTAVE~1.0_2\m
ingw64\include\octave-6.0.0\octave
-IC:\Programs\Octave\OCTAVE~1.0_2\mingw64\include   -fopenmp -g
 -O2    ocl_ov_matrix.cc -o ocl_ov_matrix.o
ocl_ov_matrix.cc: In member function 'octave_base_ocl_matrix<AT>*
octave_base_ocl_matrix<AT>::ocl_
index_op(const octave_value_list&, bool)':
ocl_ov_matrix.cc:181:13: warning: 'error_state' is deprecated: [6]: this
variable is obsolete and
always has the value 0 [-Wdeprecated-declarations]
         if (error_state) break;
             ^~~~~~~~~~~
In file included from
c:\programs\octave\octave~1.0_2\mingw64\include\octave-6.0.0\octave\oct.h:33
:0,
                 from ocl_octave_versions.h:4,
                 from ocl_ov_matrix.h:37,
                 from ocl_ov_matrix.cc:32:
c:\programs\octave\octave~1.0_2\mingw64\include\octave-6.0.0\octave\error.h:551:26:
note: declared
 here
 extern OCTINTERP_API int error_state;
                          ^~~~~~~~~~~
ocl_ov_matrix.cc:190:11: warning: 'error_state' is deprecated: [6]: this
variable is obsolete and
always has the value 0 [-Wdeprecated-declarations]
:

so I'd suggest to update these statements so that ocl-1.0 isn't outdated the
moment that Octave-6.1.0 is released. Perhaps some configure option stuff is
needed for that.

Next try: on the virtualized HW of the VDI clients at work. Maybe maybe it
works.

Nice work, Matthias!

Philip




--
Sent from: https://octave.1599824.n4.nabble.com/Octave-Maintainers-f1638794.html

Reply | Threaded
Open this post in threaded view
|

Re: OCL package 1.0.0 released

Matthias W. Klein
On 20.08.2019 00:08, PhilipNienhuis wrote:

> Matthias W. Klein wrote
>> Hi Octave folks,
>>
>> a new release of the OCL package providing OpenCL support for
>> parallelized calculations (including GPGPU) is available [1], version
>> 1.0.0 .
>>
>> It is the first package release which is fully compatible with all
>> recently released octave versions.  (Thanks to the "communications"
>> package, I learned a crucial octave programming technique.)  A summary
>> of all important user-visible changes is available online [2].
>>
>> Users and developers can choose to either download and install the
>> package tarball [3], or clone the repo [4] and use 'make dist' and other
>> Makefile targets for local testing.
>>
>> Feedback is welcome, by e-mail or project ticket.
>
> I just tested ocl-1.0 on a cross-built dev Octave 6.0.0 on my Windows 7 box
> with 2 graphics HW, i.e., built-in Intel plus an NVidia graphics card.
> Results: ocl_tests works fine & gives no errors and the example in the
> doc/README works fine compared to mimicking the commands in Octave.
>
> While installing/building the package I got a LOT of errors along the lines
> of:
>
> :
> C:/Programs/Octave/OCTAVE~1.0_2/mingw64/bin/mkoctfile-6.0.0.exe --verbose -c
> ocl_ov_matrix.cc
> g++ -c
> -I/home/philip/devel/octdev/mxe/mxe_64b_20190808/usr/x86_64-w64-mingw32/include
> -IC:\Progr
> ams\Octave\OCTAVE~1.0_2\mingw64\include\octave-6.0.0\octave\..
> -IC:\Programs\Octave\OCTAVE~1.0_2\m
> ingw64\include\octave-6.0.0\octave
> -IC:\Programs\Octave\OCTAVE~1.0_2\mingw64\include   -fopenmp -g
>  -O2    ocl_ov_matrix.cc -o ocl_ov_matrix.o
> ocl_ov_matrix.cc: In member function 'octave_base_ocl_matrix<AT>*
> octave_base_ocl_matrix<AT>::ocl_
> index_op(const octave_value_list&, bool)':
> ocl_ov_matrix.cc:181:13: warning: 'error_state' is deprecated: [6]: this
> variable is obsolete and
> always has the value 0 [-Wdeprecated-declarations]
>          if (error_state) break;
>              ^~~~~~~~~~~
> In file included from
> c:\programs\octave\octave~1.0_2\mingw64\include\octave-6.0.0\octave\oct.h:33
> :0,
>                  from ocl_octave_versions.h:4,
>                  from ocl_ov_matrix.h:37,
>                  from ocl_ov_matrix.cc:32:
> c:\programs\octave\octave~1.0_2\mingw64\include\octave-6.0.0\octave\error.h:551:26:
> note: declared
>  here
>  extern OCTINTERP_API int error_state;
>                           ^~~~~~~~~~~
> ocl_ov_matrix.cc:190:11: warning: 'error_state' is deprecated: [6]: this
> variable is obsolete and
> always has the value 0 [-Wdeprecated-declarations]
> :
>
> so I'd suggest to update these statements so that ocl-1.0 isn't outdated the
> moment that Octave-6.1.0 is released. Perhaps some configure option stuff is
> needed for that.
>
> Next try: on the virtualized HW of the VDI clients at work. Maybe maybe it
> works.
>
> Nice work, Matthias!
>
> Philip
Philip,

thanks for your dedicated evaluation and motivating comment.

The last months, I worked on making the OCL project compatible with
released versions of octave as prio #1. Doing the same with the octave
dev has been on my mind - as one of several ideas in which direction OCL
should grow in the future, maybe as the most important of these. (I
will, however, need to setup a new dedicated environment for that, in
short, so don't expect it tomorrow.)

Your tests clearly show very promising results with the octave dev, very
appreciated. And also show only little work left, probably only a few
encapsulations with preprocessor directives using an extended
ocl_octave_versions.h

In order to clarify for me: What difference will it make to have a
dev-compatible ocl package, what aspects are relevant to you octave
maintainers in practice for the package? Where do you see it in relation
to the octave (forge) project?

Please do keep me posted on you further tests as announced.

Best, Matt

Reply | Threaded
Open this post in threaded view
|

Re: OCL package 1.0.0 released

PhilipNienhuis
Matthias W. Klein wrote:

> On 20.08.2019 00:08, PhilipNienhuis wrote:
>> Matthias W. Klein wrote
>>> Hi Octave folks,
>>>
>>> a new release of the OCL package providing OpenCL support for
>>> parallelized calculations (including GPGPU) is available [1], version
>>> 1.0.0 .
>>>
>>> It is the first package release which is fully compatible with all
>>> recently released octave versions.  (Thanks to the "communications"
>>> package, I learned a crucial octave programming technique.)  A summary
>>> of all important user-visible changes is available online [2].
>>>
>>> Users and developers can choose to either download and install the
>>> package tarball [3], or clone the repo [4] and use 'make dist' and other
>>> Makefile targets for local testing.
>>>
>>> Feedback is welcome, by e-mail or project ticket.
>>
>> I just tested ocl-1.0 on a cross-built dev Octave 6.0.0 on my Windows
>> 7 box
>> with 2 graphics HW, i.e., built-in Intel plus an NVidia graphics card.
>> Results: ocl_tests works fine & gives no errors and the example in the
>> doc/README works fine compared to mimicking the commands in Octave.
>>
>> While installing/building the package I got a LOT of errors along the
>> lines
>> of:
>>
>> :
>> C:/Programs/Octave/OCTAVE~1.0_2/mingw64/bin/mkoctfile-6.0.0.exe
>> --verbose -c
>> ocl_ov_matrix.cc
>> g++ -c
>> -I/home/philip/devel/octdev/mxe/mxe_64b_20190808/usr/x86_64-w64-mingw32/include
>>
>> -IC:\Progr
>> ams\Octave\OCTAVE~1.0_2\mingw64\include\octave-6.0.0\octave\..
>> -IC:\Programs\Octave\OCTAVE~1.0_2\m
>> ingw64\include\octave-6.0.0\octave
>> -IC:\Programs\Octave\OCTAVE~1.0_2\mingw64\include   -fopenmp -g
>>  -O2    ocl_ov_matrix.cc -o ocl_ov_matrix.o
>> ocl_ov_matrix.cc: In member function 'octave_base_ocl_matrix<AT>*
>> octave_base_ocl_matrix<AT>::ocl_
>> index_op(const octave_value_list&, bool)':
>> ocl_ov_matrix.cc:181:13: warning: 'error_state' is deprecated: [6]: this
>> variable is obsolete and
>> always has the value 0 [-Wdeprecated-declarations]
>>          if (error_state) break;
>>              ^~~~~~~~~~~
>> In file included from
>> c:\programs\octave\octave~1.0_2\mingw64\include\octave-6.0.0\octave\oct.h:33
>>
>> :0,
>>                  from ocl_octave_versions.h:4,
>>                  from ocl_ov_matrix.h:37,
>>                  from ocl_ov_matrix.cc:32:
>> c:\programs\octave\octave~1.0_2\mingw64\include\octave-6.0.0\octave\error.h:551:26:
>>
>> note: declared
>>  here
>>  extern OCTINTERP_API int error_state;
>>                           ^~~~~~~~~~~
>> ocl_ov_matrix.cc:190:11: warning: 'error_state' is deprecated: [6]: this
>> variable is obsolete and
>> always has the value 0 [-Wdeprecated-declarations]
>> :
>>
>> so I'd suggest to update these statements so that ocl-1.0 isn't
>> outdated the
>> moment that Octave-6.1.0 is released. Perhaps some configure option
>> stuff is
>> needed for that.
>>
>> Next try: on the virtualized HW of the VDI clients at work. Maybe
>> maybe it
>> works.
>>
>> Nice work, Matthias!
>>
>> Philip
> Philip,
>
> thanks for your dedicated evaluation and motivating comment.
>
> The last months, I worked on making the OCL project compatible with
> released versions of octave as prio #1. Doing the same with the octave
> dev has been on my mind - as one of several ideas in which direction OCL
> should grow in the future, maybe as the most important of these. (I
> will, however, need to setup a new dedicated environment for that, in
> short, so don't expect it tomorrow.)
>
> Your tests clearly show very promising results with the octave dev, very
> appreciated. And also show only little work left, probably only a few
> encapsulations with preprocessor directives using an extended
> ocl_octave_versions.h
>
> In order to clarify for me: What difference will it make to have a
> dev-compatible ocl package, what aspects are relevant to you octave
> maintainers in practice for the package? Where do you see it in relation
> to the octave (forge) project?
>
> Please do keep me posted on you further tests as announced.

A lot of questions, Matthias. I think of myself not so much as a core
dev but rather a contributor and OF package maintainer (io and mapping).

I merely wanted to test the package for you in its current state as you
asked on the ML. Well, it was fun in the limited time I had available.

To be compatible with dev Octave now is primarily a convenience. It will
make sure that you do not have to worry later if 5.2.0 is skipped and
6.1.0 is released instead. As for myself, I always use bleeding edge
Octave, usually not "older" than one or two weeks.
IMO it is always a good thing to try to be compatible with dev Octave,
because when a new Octave is released it usually takes a few months to
get OF packages updated. For end users that is not very motivating. But
admittedly dev Octave can be a moving target.

While testing I got the impression that ocl by itself just provides a
framework to build upon. To do useful things with it user code has to
incorporate its functions, possibly with try/catch around it to follow
another code path in case ocl isn't supported on the local HW. It's not
as if ocl is a mere drop-in replacement for user code, it requires a
little more.
Or did I get that wrong? I only briefly tested a few things.

Anyway, in my opinion ocl is a very useful and promising package. I'd
like to try really big matrices to test if ocl really makes a difference
but that'll have to wait a bit I'm afraid.

Philip







Reply | Threaded
Open this post in threaded view
|

Re: OCL package 1.0.0 released

Matthias W. Klein
On 20.08.2019 23:57, Philip Nienhuis wrote:

> Matthias W. Klein wrote:
>> On 20.08.2019 00:08, PhilipNienhuis wrote:
>>> Next try: on the virtualized HW of the VDI clients at work. Maybe
>>> maybe it
>>> works.
>>>
>>> Nice work, Matthias!
>>>
>>> Philip
>> Philip,
>>
>> thanks for your dedicated evaluation and motivating comment.
>>
>> The last months, I worked on making the OCL project compatible with
>> released versions of octave as prio #1. Doing the same with the octave
>> dev has been on my mind - as one of several ideas in which direction OCL
>> should grow in the future, maybe as the most important of these. (I
>> will, however, need to setup a new dedicated environment for that, in
>> short, so don't expect it tomorrow.)
>>
>> Your tests clearly show very promising results with the octave dev, very
>> appreciated. And also show only little work left, probably only a few
>> encapsulations with preprocessor directives using an extended
>> ocl_octave_versions.h
>>
>> In order to clarify for me: What difference will it make to have a
>> dev-compatible ocl package, what aspects are relevant to you octave
>> maintainers in practice for the package? Where do you see it in relation
>> to the octave (forge) project?
>>
>> Please do keep me posted on you further tests as announced.
>
> A lot of questions, Matthias. I think of myself not so much as a core
> dev but rather a contributor and OF package maintainer (io and mapping).
>
> I merely wanted to test the package for you in its current state as you
> asked on the ML. Well, it was fun in the limited time I had available.
>
> To be compatible with dev Octave now is primarily a convenience. It will
> make sure that you do not have to worry later if 5.2.0 is skipped and
> 6.1.0 is released instead. As for myself, I always use bleeding edge
> Octave, usually not "older" than one or two weeks.
> IMO it is always a good thing to try to be compatible with dev Octave,
> because when a new Octave is released it usually takes a few months to
> get OF packages updated. For end users that is not very motivating. But
> admittedly dev Octave can be a moving target.

Dear Philip and octave maintainers!

Your package testing with dev octave is highly valued, in past and
future, Philip!

My questions about the package compatibility with dev octave was aimed
at all maintainers - also giving you the opportunity to start the
discussion with your assessment (thank you for picking up the ball,
Philip). Your point towards convenience convinces me and fits to my
(previously vague) understanding. Agreed, compatibility is worthwhile
and attractive for the near future.

@Maintainers:
a) So is it that each package release should be tested against a recent
dev octave before release?
b) How is the other way usually handled, with all existing OF packages?
Is it each package maintainer's responsibility to re-achieve package
compatibility after octave release, or is some other way of
collaboration common?
I consider myself a newbie in these respects, thanks for letting me know.

 >
 > While testing I got the impression that ocl by itself just provides a
 > framework to build upon. To do useful things with it user code has to
 > incorporate its functions, possibly with try/catch around it to follow
 > another code path in case ocl isn't supported on the local HW. It's not
 > as if ocl is a mere drop-in replacement for user code, it requires a
 > little more.
 > Or did I get that wrong? I only briefly tested a few things.
 >
 > Anyway, in my opinion ocl is a very useful and promising package. I'd
 > like to try really big matrices to test if ocl really makes a difference
 > but that'll have to wait a bit I'm afraid.

Ocl as "just a framework": If you like to strongly simplify it, yes. As
is all of octave in the first place: The user has to put in code to get
meaningful results which are of value to him/her.

In analogy, at least to a certain extent, this is the purpose of the ocl
package: Give the user the ability to (hopefully) accelerate his/her
code using present hardware. Another analogy is Mathworks' 'gpuArray'
functionality [1], which, as far as documented, must have concerned
somewhat consequently rewriting "all" their core math functions to also
work with this GPU data type (in my interpretation from their doc; I
never had access to Mathworks' parallel computing package for testing).
Well, not fully: They say math functions which shall output complex
numbers must be fed with complex number data as input.

The ocl package implements many math functions and operators for
oclArray data (but surely not as much as the Mathworks package today).
Much octave code can simply be reused with ocl. But I have not tested
what octave functions beyond those in ocl_test.m actually run already today.

What I have done with ocl, for instance, is write a complete Monte Carlo
simulation (with 1e5 to 1e8 particles) in three different ways:
1) pure octave code of ~1000 summed lines,
2) the very same code with ~10 lines added at the beginning/end, to
completely work with ocl math functions and operators, and
3) translating the code to OpenCL C and using 'ocl_program'.
My results were:
a) The OpenCL C code ran almost a factor of 1000 faster than pure octave
code, using a good but old GPU.
b) The code using ocl math functions and operators was slower than
octave, even for large datasets, due to overhead.

This having said, I think the ocl package is very good at the following:

* Provide a detailed infrastructure between octave, the (correctly
installed) OpenCL software and the (chosen) OpenCL hardware.

* Provide best acceleration with user-written OpenCL C code, which is
handled by ocl interactively.

* Provide handy means (functions, operators, indexing) for inspection of
intermediate and end result data.

The package is currently not so efficient or good at:

* Provide efficient acceleration with standard functions and operators.

* Work with complex numbers.

I have had an active look into complex numbers as a future major
extension to ocl.

As to efficient standard functions and operators for re-using octave
code: I have had a few thoughts about more efficient workload
distribution models in ocl_array(_prog).cc, this might be a lightweight
improvement. The alternative would be to fall back onto an established
OpenCL-related math library, like clBLAS(T) [2,3] or MAGMA [4], but,
given the ocl structure, this would raise technical (library calls,
dependencies) and/or licensing questions.

In general, I think the ocl package will never cover all possible
standard functions and operators efficiently, the effort is simply too much.

But I do see, as you do, Philip, that the package can be of great use
for a broad audience of octave users, and have designed the package
correspondingly. I can well imagine ocl to become an OF package, but I
do not know what you maintainers or OF admins require in practice for
this step? (Yes, I have read [5], conform to most of it, and am willing
to discuss the rest. With whom?) Also, the decision community vs.
external package seems crucial. The package itself contains about 10000
lines of C++ code and a few points will have to be adapted slightly in a
concerted action as the octave interface for oct-files evolves. What
procedure is expected or common?

@Maintainers: Who can kindly assist me with the latter questions?

Thanks,
Matt

[1] https://www.mathworks.com/help/parallel-computing/gpuarray.html
[2] https://github.com/clMathLibraries/clBLAS
[3] https://github.com/CNugteren/CLBlast
[4] https://icl.cs.utk.edu/magma/
[5] https://octave.sourceforge.io/developers.php


Reply | Threaded
Open this post in threaded view
|

Re: OCL package 1.0.0 released

siko1056
Dear Matt,

Sorry for the more than late reply for your great effort.  Maybe I can
give you a few answers below:

On 8/21/19 10:54 PM, Matthias W. Klein wrote:

> [...]
>
> @Maintainers:
> a) So is it that each package release should be tested against a recent
> dev octave before release?
> b) How is the other way usually handled, with all existing OF packages?
> Is it each package maintainer's responsibility to re-achieve package
> compatibility after octave release, or is some other way of
> collaboration common?
> I consider myself a newbie in these respects, thanks for letting me know.


For the past major releases, say Octave 5, there was an announcement on
this mailing-list for feature freezes and a few release candidates.
From this moment on, each active OF maintainer had about a month to look
at his packages to make them work with that new release candidates.
Many working OF packages with binary code get bundled with the official
MS Windows release (with is the most crucial deadline for OF package
maintainers, I think).  If you don't care about that, you can fix
compatibility at any later time.


> [...] I can well imagine ocl to become an OF package, but I
> do not know what you maintainers or OF admins require in practice for
> this step? (Yes, I have read [5], conform to most of it, and am willing
> to discuss the rest. With whom?) Also, the decision community vs.
> external package seems crucial. The package itself contains about 10000
> lines of C++ code and a few points will have to be adapted slightly in a
> concerted action as the octave interface for oct-files evolves. What
> procedure is expected or common?
>
> @Maintainers: Who can kindly assist me with the latter questions?
>
> Thanks,
> Matt
>
> [...]
>

I also would love to see your package on OF, and was surprised it did
not happen yet?!

Today I learned from another OF package creation attempt [6] that a
ticket should be opened at SourceForge [7].  But I am neither admin nor
developer of Octave Forge, thus I cannot help you out here.  Olaf or
Juan, can you help out here to not be forgotten again?

The rest is really up to you.  If you want to host a community package,
the review process might be a little stricter, but in your case it seems
realistic to go with that.  Thus please provide for the ticket:

  Name: ocl ?
  SCM: mercurial/git/svn ?
  Type: community/external package ?

Hope that you are still interested and things don't get stuck again.

Best,
Kai


[5] https://octave.sourceforge.io/developers.php
[6]
https://lists.gnu.org/archive/html/octave-maintainers/2019-11/msg00085.html
[7] https://sourceforge.net/p/octave/package-releases/

Reply | Threaded
Open this post in threaded view
|

Re: OCL package 1.0.0 released

Juan Pablo Carbajal-2
> Today I learned from another OF package creation attempt [6] that a
> ticket should be opened at SourceForge [7].  But I am neither admin nor
> developer of Octave Forge, thus I cannot help you out here.  Olaf or
> Juan, can you help out here to not be forgotten again?
>

I do not think that the ticket method is actually official, but I
think is a good way to request a new repo.
I personally will suggest opencl as package name rather than ocl, the
3 extra letters make a huge difference in clarity.
Please let me know and I will open the ticket (can do it yourself?) in
the package release tracker [1].

[1]: https://sourceforge.net/p/octave/package-releases/

> The rest is really up to you.  If you want to host a community package,
> the review process might be a little stricter, but in your case it seems
> realistic to go with that.  Thus please provide for the ticket:
>
>   Name: ocl ?
>   SCM: mercurial/git/svn ?
>   Type: community/external package ?
>
> Hope that you are still interested and things don't get stuck again.
>
My vary personal suggestion is to keep it as external, because OF
maintainers cannot keep up with all the work that is coming towards
us, and you will suffer unnecessary delays.
You can also follow Mike's example [2] and not even post it in OF, but
announce the release in Admin and Help mailing list. This way you are
the only responsible for delay in releases and you can always motivate
people to join your team.
We need to find a better way to syndicate OF packages.

[2]: https://octave.1599824.n4.nabble.com/Pythonic-package-0-0-1-released-td4692932.html

> Best,
> Kai
>
>
> [5] https://octave.sourceforge.io/developers.php
> [6]
> https://lists.gnu.org/archive/html/octave-maintainers/2019-11/msg00085.html
> [7] https://sourceforge.net/p/octave/package-releases/
>

Reply | Threaded
Open this post in threaded view
|

Re: OCL package 1.0.0 released

Matthias W. Klein
Dear Kai, dear Juan,

thank you for giving a new jolt to the project's status.  BTW, I think
responding supportingly is never "too late" considering we all (or
almost all) do the developing besides our jobs and other private activities.

 > I personally will suggest opencl as package name rather than ocl, the
 > 3 extra letters make a huge difference in clarity.

As to the package name:  There are a number of packages both with long
and with short names at OF (openmpi_ext was even renamed mpi).  Clarity
is a point, but the package content is also clear from the one-line
title ("OpenCL support for GNU Octave").  I chose the name OCL over a
year ago before publishing.  I also hold a copyright disclaimer signed
by my employer referring to "ocl" as the package's name (I have
developed the package privately, but also use it at work).  Also, the
acronym ocl is already a prefix for the package's octave commands (in
order not to pollute the namespace in the octave symbol table).  Thus, I
have a strong interest in keeping "ocl" as the package name.

 >> The rest is really up to you.  If you want to host a community package,
 >> the review process might be a little stricter, but in your case it seems
 >> realistic to go with that.
 >>
 > My vary personal suggestion is to keep it as external, because OF
 > maintainers cannot keep up with all the work that is coming towards
 > us, and you will suffer unnecessary delays.

As to the decision for community vs. external package:  I feel it makes
a crucial difference, but I myself do not have a strong opinion, due to
lack of knowledge.  The OCL package is mainly C++ code depending on
octave core sources, which would favor a community package status (in
the long run).  On the other hand, OCL has found intermediate attention
so far (being advertized only on this list, it has been downloaded ~130
times).  Collaboration for development is also somewhat simpler with an
external package.  Hence, I can currently see OCL as an "external
package", as an important advance in status.

What I have in mind is a transition time for migrating the ocl project
from sourceforge [1] to octave forge, maintaining both repos for a
period of time and eventually disactivating (forwarding) the former
clone (while respecting consistency of names and releases at all times).

[1] https://sourceforge.net/p/octave-ocl/

 > I do not think that the ticket method is actually official, but I
 > think is a good way to request a new repo.
 > Please let me know and I will open the ticket (can do it yourself?) in
 > the package release tracker [...].

Juan, can you please open the ticket with the following information:

    Name: ocl
    SCM:  mercurial
    Type: external package

 > You can also follow Mike's example [...] and not even post it in OF, but
 > announce the release in Admin and Help mailing list. This way you are
 > the only responsible for delay in releases and you can always motivate
 > people to join your team.

This is how it has been up to now - my project has been hosted on [1].
The drawback is that 'pkg install -forge ocl' obviously did not work for
the general user.  Being hosted at OF, it will.  And it will draw more
attention, as intended, also by several expressed opinions on this list.
  Bundling it into distros will be an additional advantage of the OF
package.

Thank you very much for your support!

Best,
Matt


Reply | Threaded
Open this post in threaded view
|

Re: OCL package 1.0.0 released

Juan Pablo Carbajal-2
[...]
> On the other hand, OCL has found intermediate attention
> so far (being advertized only on this list, it has been downloaded ~130
> times).

I heva promoted your package here as well
https://flossforscience.com/podcast/season-2-episode-9
See section Links, GPU acceleration ...

> Juan, can you please open the ticket with the following information:
[...]
done!
https://sourceforge.net/p/octave/package-releases/395/

[...]
> The drawback is that 'pkg install -forge ocl' obviously did not work for
> the general user.

pkg was upgraded to solve this issue.
Have you tried the following?

dev (hash is the commit hash):
'pkg install https://sourceforge.net/code-snapshots/hg/o/oc/octave-ocl/code/octave-ocl-code-07aa8e087b8098aaf4123e833ae38397a9ae1479.zip'

release:
'pkg install https://sourceforge.net/projects/octave-ocl/files/ocl-1.0.0.tar.gz/download'

Reply | Threaded
Open this post in threaded view
|

Re: OCL package 1.0.0 released

Matthias W. Klein
> I heva promoted your package here as well
> https://flossforscience.com/podcast/season-2-episode-9
> See section Links, GPU acceleration ...
>
>> Juan, can you please open the ticket with the following information:
> [...]
> done!
> https://sourceforge.net/p/octave/package-releases/395/

Hola Juan,

thank you for the ticket!

@OF Admin: I am now awaiting the "ocl" repo to be opened.  Moreover, my
user name @ sourceforge is "mattwklein" [1]. Can you please add me to
the list of OF developers so I can start filling the "ocl" repo.

Also many thanks for supporting the advertisement!

Best,
Matt

[1] https://sourceforge.net/u/mattwklein/profile/


Reply | Threaded
Open this post in threaded view
|

Re: OCL package 1.0.0 released

Olaf Till-2
On Tue, Dec 03, 2019 at 09:25:03PM +0100, Matthias W. Klein wrote:

> > I heva promoted your package here as well
> > https://flossforscience.com/podcast/season-2-episode-9
> > See section Links, GPU acceleration ...
> >
> > > Juan, can you please open the ticket with the following information:
> > [...]
> > done!
> > https://sourceforge.net/p/octave/package-releases/395/
>
> Hola Juan,
>
> thank you for the ticket!
>
> @OF Admin: I am now awaiting the "ocl" repo to be opened.  Moreover, my
> user name @ sourceforge is "mattwklein" [1]. Can you please add me to
> the list of OF developers so I can start filling the "ocl" repo.
>
> Also many thanks for supporting the advertisement!
>
> Best,
> Matt
Hi Matt,

welcome on board, and sorry for the delay -- the subject of this
thread could have been a bit more to the point.

You have now OF maintainer rights and an empty mercurial repository
'ocl' has been created at OF, you should be able to clone it with:

hg clone ssh://[hidden email]/p/octave/ocl ocl

IMHO the package is a candidate for the 'community' group. In testing,
I explicitly request help of others (something has already been
done). Most testing is usually to be expected when you initiate a
release at OF.

A few notes:

- As long as there is no release at OF, it is still time to
  re-consider the name of the package at OF. Is the corresponding
  Matlab package named the same?

- Please study the directives for maintainers at our web page. You
  should add a Makefile at the root level of your package, with
  targets for releases and so on. This could point into the respective
  existing targets of your src/Makefile, but that's not usual. If you
  like, you can use (modifiy) the template root level Makefile from
  our web-page.

- I'm a bit astonished that a package heavily tied to Octave internals
  and external C-code (?) works without a configure script.

Olaf

--
public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: OCL package 1.0.0 released

Matthias W. Klein
> welcome on board, and sorry for the delay -- the subject of this
> thread could have been a bit more to the point.

Hi Olaf,

thanks for introducing me into the team. - I had issued a first message
on this ML with "New Octave package" in the subject line back in
April... Anyway, since then the project has picked up much interest and
supporting words which I am grateful of and has manifested, in short,
that the package is suitable for OF. No worries.

> You have now OF maintainer rights and an empty mercurial repository
> 'ocl' has been created at OF, you should be able to clone it with:
>
> hg clone ssh://[hidden email]/p/octave/ocl ocl

What I have in mind is a transition time for migrating my project
from sourceforge [1] to octave forge, maintaining both repos for a
period of time and eventually disactivating (forwarding) the former
clone (while respecting consistency of names and releases at all times).

[1] https://sourceforge.net/p/octave-ocl/

So I think the OF repo should, as a next step, become a clone of the
existing SF repo; I would do this by pushing my local repo to the empty
OF repo. Will this give the correct result? I am cautious on the emtpy
repo in order not to start off wrong.

> IMHO the package is a candidate for the 'community' group. In testing,
> I explicitly request help of others (something has already been
> done). Most testing is usually to be expected when you initiate a
> release at OF.

It's controversial. I can see arguments for both 'community' and
'external': The OCL package is mainly C++ code depending on
octave core sources, which would favor a community package status. On
the other hand, OCL - especially depending on third-party OpenCL drivers
- has shown to not always be plug-and-play, which would make an external
status suitable. Currently, I am fine with 'external', but I am open for
a regrouping whenever (several of) you highly experienced guys deem it
more suitable and worth the effort.

> - As long as there is no release at OF, it is still time to
>   re-consider the name of the package at OF. Is the corresponding
>   Matlab package named the same?

The 'parallel' toolbox of Matlab offers support only for CUDA using
NVidia hardware. 'parallel' for octave is taken, and given the different
dependencies I would NOT suggest merging.

I also hold a copyright disclaimer signed by my employer referring to
"ocl" as the package's name (I have developed the package privately, but
also use it at work). Changing the package name would require me to go
through this process again... Let me avoid this.

> - Please study the directives for maintainers at our web page. You
>   should add a Makefile at the root level of your package, with
>   targets for releases and so on. This could point into the respective
>   existing targets of your src/Makefile, but that's not usual. If you
>   like, you can use (modifiy) the template root level Makefile from
>   our web-page.

Many of the directives have been familiar to me for the project layout.
So far I have, nevertheless, chosen a somewhat uncommon directory
structure in the repo. This is mostly due to the fact that, for
the usual repeated incremental code developing and testing, I am using
'make' for partial rebuilts. However, I made sure that 'make dist'
creates the correct installable release tarball. Other aspects can be
discussed with interested collaborators.

> - I'm a bit astonished that a package heavily tied to Octave internals
>   and external C-code (?) works without a configure script.

This is due to the fact, frankly, that I am mostly ignorant of autoconf
tools in practice. I know how to read and write Makefiles, but not
configure scripts. The ocl project might benefit substantially from a
collaborator familiar with the autoconf toolchain!

Best,
Matt


Reply | Threaded
Open this post in threaded view
|

Re: OCL package 1.0.0 released

Olaf Till-2
On Wed, Dec 04, 2019 at 10:17:42PM +0100, Matthias W. Klein wrote:
> So I think the OF repo should, as a next step, become a clone of the
> existing SF repo; I would do this by pushing my local repo to the empty
> OF repo. Will this give the correct result?

Yes, it should.

> > IMHO the package is a candidate for the 'community' group. In testing,
> > I explicitly request help of others (something has already been
> > done). Most testing is usually to be expected when you initiate a
> > release at OF.
>
> It's controversial. I can see arguments for both 'community' and
> 'external': The OCL package is mainly C++ code depending on
> octave core sources, which would favor a community package status. On
> the other hand, OCL - especially depending on third-party OpenCL drivers
> - has shown to not always be plug-and-play, which would make an external
> status suitable.
As you can read at our 'Developers' page, not the issues you mention,
but others, are supposed to be a base for deciding between 'community'
and 'external'.

If you want to decide together with others what is the best way to
develop the package, and you are willing to sometimes follow a
decision although you think another would be better, choose
'community'. If you want to decide everything yourself, choose
'external'. But in the latter case, not all of us may be motivated to
help. Note also that an external package may cause a problem if it
provides a very desirable functionality and its owner stops
maintainance or develops the package into a direction not thought
ideal by the Octave community.

> > - Please study the directives for maintainers at our web page. You
> >   should add a Makefile at the root level of your package, with
> >   targets for releases and so on. This could point into the respective
> >   existing targets of your src/Makefile, but that's not usual. If you
> >   like, you can use (modifiy) the template root level Makefile from
> >   our web-page.
>
> Many of the directives have been familiar to me for the project layout.
> So far I have, nevertheless, chosen a somewhat uncommon directory
> structure in the repo. This is mostly due to the fact that, for
> the usual repeated incremental code developing and testing, I am using
> 'make' for partial rebuilts. However, I made sure that 'make dist'
> creates the correct installable release tarball. Other aspects can be
> discussed with interested collaborators.
I don't understand why this hinders you to add a root level Makefile
for the release-related stuff. src/Makefile should still be there, of
course, and you are free to use it as you like. A decent additional
root level Makefile makes things easier for the person who has to
check and publish the release. Indeed, it is a 'hard' requirement for
all OF packages.

Olaf

--
public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: OCL package 1.0.0 released

Matthias W. Klein
Dear Olaf,

 >> So I think the OF repo should, as a next step, become a clone of the
 >> existing SF repo; I would do this by pushing my local repo to the empty
 >> OF repo. Will this give the correct result?
 > Yes, it should.

Done. The OF repo content is a current clone of the SF repo.

Next question is whether we also replicate the existing releases, 4 so
far, and I am currently preparing a new one.

 >>> IMHO the package is a candidate for the 'community' group. In testing,
 >>> I explicitly request help of others (something has already been
 >>> done). Most testing is usually to be expected when you initiate a
 >>> release at OF.
 >> It's controversial. I can see arguments for both 'community' and
 >> 'external': The OCL package is mainly C++ code depending on
 >> octave core sources, which would favor a community package status. On
 >> the other hand, OCL - especially depending on third-party OpenCL drivers
 >> - has shown to not always be plug-and-play, which would make an external
 >> status suitable.
 > As you can read at our 'Developers' page, not the issues you mention,
 > but others, are supposed to be a base for deciding between 'community'
 > and 'external'.
 >
 > If you want to decide together with others what is the best way to
 > develop the package, and you are willing to sometimes follow a
 > decision although you think another would be better, choose
 > 'community'. If you want to decide everything yourself, choose
 > 'external'. But in the latter case, not all of us may be motivated to
 > help. Note also that an external package may cause a problem if it
 > provides a very desirable functionality and its owner stops
 > maintainance or develops the package into a direction not thought
 > ideal by the Octave community.

With 'controversial' I meant that the question has been discussed
controversially on the maintainer ML, regarding ocl.

I recently expressed that, given the tight dependency of ocl to octave
sources, I would favor a community package status at least in the long
run. It is perfectly acceptable for me to start this long run early.

Here is my suggestion: I am preparing the next release, still as a
one-man show, and we subsequently turn ocl into a community package.
Does this sound like a plan? It will be good to agree on such a course
of action in advance.

The suggestion would mean I can expect to release version 1.1.0 soon. We
should agree on whether we want peer review before/with release or not
(yet). For the following collaboration to start I see a bunch of
possible aspects how peers can contribute to raise the project to a more
elaborate level (you mentioned a few), but I am very open for further
suggestions.

 >>> - Please study the directives for maintainers at our web page. You
 >>>   should add a Makefile at the root level of your package, with
 >>>   targets for releases and so on. This could point into the respective
 >>>   existing targets of your src/Makefile, but that's not usual. If you
 >>>   like, you can use (modifiy) the template root level Makefile from
 >>>   our web-page.
 >> Many of the directives have been familiar to me for the project layout.
 >> So far I have, nevertheless, chosen a somewhat uncommon directory
 >> structure in the repo. This is mostly due to the fact that, for
 >> the usual repeated incremental code developing and testing, I am using
 >> 'make' for partial rebuilts. However, I made sure that 'make dist'
 >> creates the correct installable release tarball. Other aspects can be
 >> discussed with interested collaborators.
 > I don't understand why this hinders you to add a root level Makefile
 > for the release-related stuff. src/Makefile should still be there, of
 > course, and you are free to use it as you like. A decent additional
 > root level Makefile makes things easier for the person who has to
 > check and publish the release. Indeed, it is a 'hard' requirement for
 > all OF packages.

I was unclear, sorry for the misunderstanding. The ocl repo currently
has a flat directory "structure" and one Makefile (at this "root level")
which does everything I needed so far: without arguments, it compiles
the sources locally; with other targets it aids in development; and
'make dist' yields the release tarball (which contains a non-trivial
internal directory structure as needed, including a clone of the single
Makefile at src/Makefile).

I'll be happy to discuss changes, hopefully maintaining the local
incremental building feature which I use most of the times during
developments and which the OF template, to best of my knowledge, does
not offer so far.

Best,
Matt


Reply | Threaded
Open this post in threaded view
|

Re: OCL package 1.0.0 released

Olaf Till-2
On Sat, Dec 07, 2019 at 10:49:03PM +0100, Matthias W. Klein wrote:
> Done. The OF repo content is a current clone of the SF repo.

Ok, got 47 changesets.

> Next question is whether we also replicate the existing releases, 4 so
> far

I'd say no. They most likely would require changes, and with changes
it would be different releases.

> and I am currently preparing a new one.

Let's concentrate on the upcoming one, I'd say.

> With 'controversial' I meant that the question has been discussed
> controversially on the maintainer ML, regarding ocl.
>
> I recently expressed that, given the tight dependency of ocl to octave
> sources, I would favor a community package status at least in the long
> run. It is perfectly acceptable for me to start this long run early.
>
> Here is my suggestion: I am preparing the next release, still as a
> one-man show, and we subsequently turn ocl into a community package.
> Does this sound like a plan? It will be good to agree on such a course
> of action in advance.
Ok from my side. And to be honest: most of the work is likely to be on
your shoulders anyway, even with community status.

> The suggestion would mean I can expect to release version 1.1.0 soon. We
> should agree on whether we want peer review before/with release or not
> (yet).

Any release is reviewed, how much 'peer' depends on the situation. A
new package will be reviewed more thoroughly. The usual way is that
the maintainer (you) makes something ready of which he thinks it can
be released. With reviewing, changes may be applied. If you have
specific questions before, you can ask them at the maintainers list.

> I was unclear, sorry for the misunderstanding. The ocl repo currently
> has a flat directory "structure" and one Makefile (at this "root level")
> which does everything I needed so far: without arguments, it compiles
> the sources locally; with other targets it aids in development; and
> 'make dist' yields the release tarball (which contains a non-trivial
> internal directory structure as needed, including a clone of the single
> Makefile at src/Makefile).

We can sort this out time by time. In the download from your origingal
repository, the directory structure wasn't flat, there were inst/ and
src/ directories. These should be re-introduced now (i.e., be conform
with the docs of Octaves pkg() for new packages). You can leave your
Makefile under src/. Additionally, there should be a Makefile at root,
with targets 'dist' and related ones.

Olaf

--
public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net

signature.asc (849 bytes) Download Attachment