64-bit indexing -- Test machine available with 128GB ram 32 CPUs 12T scuzzy on Fedora31

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

64-bit indexing -- Test machine available with 128GB ram 32 CPUs 12T scuzzy on Fedora31

ndyer1
I saw mention of your 5.2 developer needing a test machine with greater than 32 GB ram.
I am motivated to get out the release (s) for linux similar to the 5.2 64-bit indexing offering for windows as of January.
I have a Hewlitt Packard  Z820 with 128 GB ram, 32 CPUs, 12T of fast scuzzy storage, generally suitable for big data.
On Fedora31.
And offer it to run Octave test scripts.

I'm at Univerity of Illinois, doing physics modeling since 2003.  Have written thousands of matlab functions.
This year I'm beginning the changeover to Octave.  So each function will be somehow tested.
Outside of octave/matlab i am only a requirements writer, not a coder.

If I can be of any help to you, glad to serve.

Kind regards,
Norm Dyer



Reply | Threaded
Open this post in threaded view
|

Re: 64-bit indexing -- Test machine available with 128GB ram 32 CPUs 12T scuzzy on Fedora31

nrjank

I'm at Univerity of Illinois, doing physics modeling since 2003.  Have written thousands of matlab functions.
This year I'm beginning the changeover to Octave.  So each function will be somehow tested.
Outside of octave/matlab i am only a requirements writer, not a coder.

If I can be of any help to you, glad to serve.

I can't comment on the 64bit linux dev, but I can definitely say you can be of great help 'without being a coder'. Providing a fresh look at things that don't work, pointing out bugs as you try to port over..But, since a large fraction of the code base is written in m-code, you actually probably are in a decent position to help on the code side as well. (I personally can't touch anything outside of m-code world, but have still been able to generate a few functions and bug patches)

Welcome aboard!
Reply | Threaded
Open this post in threaded view
|

Re: 64-bit indexing -- Test machine available with 128GB ram 32 CPUs 12T scuzzy on Fedora31

siko1056
On 2/9/20 9:36 AM, Nicholas Jankowski wrote:

>
>     I'm at Univerity of Illinois, doing physics modeling since 2003. 
>     Have written thousands of matlab functions.
>     This year I'm beginning the changeover to Octave.  So each function
>     will be somehow tested.
>     Outside of octave/matlab i am only a requirements writer, not a coder.
>
>     If I can be of any help to you, glad to serve.
>
> I can't comment on the 64bit linux dev, but I can definitely say you can
> be of great help 'without being a coder'. Providing a fresh look at
> things that don't work, pointing out bugs as you try to port over..But,
> since a large fraction of the code base is written in m-code, you
> actually probably are in a decent position to help on the code side as
> well. (I personally can't touch anything outside of m-code world, but
> have still been able to generate a few functions and bug patches)
>
> Welcome aboard!


Dear Norm Dyer,

Thanks for your interest.  If your system supports Singularity [1,2]
(free and open source software), you can try out my test image [3] and
run the example from the wiki [4] in it's memory extensive form (32 GB)
for testing:

   clear all;
   N = 2^31;
   ## The following lines requires about 32 GB of RAM!
   a = ones (N, 1);
   b = ones (N, 1);
   c = a' * b

I would be glad to receive comments on this approach.  Especially, if my
 test images "survive" your Matlab scripts.

Basically I want Linux user's to make use of Octave in several flavors
with only having to compile it from scratch.  But all still work in
progress.

Kai

[1] https://sylabs.io/singularity/
[2] https://github.com/sylabs/singularity
[3] https://github.com/siko1056/GNU-Octave-64-Singularity
[4]
https://wiki.octave.org/Enable_large_arrays:_Build_octave_such_that_it_can_use_arrays_larger_than_2Gb.

Reply | Threaded
Open this post in threaded view
|

Re: 64-bit indexing -- Test machine available with 128GB ram 32 CPUs 12T scuzzy on Fedora31

siko1056
On 2/11/20 8:12 AM, Dyer, Norman Jay wrote:

>
> [...]  If your system supports Singularity [1,2]
> (free and open source software), you can try out my test image [3] and
> run the example from the wiki [4] in it's memory extensive form (32 GB)
> for testing:
>
>    clear all;
>    N = 2^31;
>    ## The following lines requires about 32 GB of RAM!
>    a = ones (N, 1);
>    b = ones (N, 1);
>    c = a' * b
>
> [...]
> [1] https://sylabs.io/singularity/
> [2] https://github.com/sylabs/singularity
> [3] https://github.com/siko1056/GNU-Octave-64-Singularity
> [4]
> https://wiki.octave.org/Enable_large_arrays:_Build_octave_such_that_it_can_use_arrays_larger_than_2Gb.
>
>
> Yes, I have been running similar NxN arrays with random numbers
> increasing N until it breaks.
> (as predicted always breaks at 2G elements)
>
> GNU Octave, version 5.1.0
> Octave was configured for "x86_64-redhat-linux-gnu".
>
>>> clear all;
>>>
>>>    N = 2^31;
>>>
>>>    ## The following lines requires about 32 GB of RAM!
>>>
>>>    a = ones (N, 1);
> error: out of memory or dimension too large for Octave's index type
>>>
>>>    b = ones (N, 1);
> error: out of memory or dimension too large for Octave's index type
>>>
>>>    c = a' * b
>
> Obviously still stuck at 32-bit indexing.  Need a linux Octave 5.2  or
> 6.0 set up for 64-bit indexing, which evidently exists - but only
> Windows is posted at Octave.org ; awaiting distro managers to each
> release their own.
> I presume containerizing it would not alter this result; isn't that
> correct?
> kind regards,
> Norm
>

Now I am a bit confused what exactly was tested.  When using my
Singularity container [3] you should receive a ready baked Octave
compiled with all dependencies (OpenBLAS, SuiteSparse, ...) to enable
those 64-bit indices.

If you compile Octave on your own machine from scratch, with only
installing the dependent packages from your Linux distribution, it is
likely not to work.  You can check your configure logs.  If it looks
like the following, your Octave software stack cannot handle 64-bit indices:

   [...]
   checking BLAS library integer size... 4
   checking for cheev_... yes
   checking default size of Fortran INTEGER... 4
   [...]
   64-bit array dims and indexing:       yes
   64-bit BLAS array dims and indexing:  no
   [...]

HTH,
Kai

Reply | Threaded
Open this post in threaded view
|

Re: 64-bit indexing -- Test machine available with 128GB ram 32 CPUs 12T scuzzy on Fedora31

John W. Eaton
Administrator
On 2/11/20 9:58 PM, Kai Torben Ohlhus wrote:

> If you compile Octave on your own machine from scratch, with only
> installing the dependent packages from your Linux distribution, it is
> likely not to work.  You can check your configure logs.  If it looks
> like the following, your Octave software stack cannot handle 64-bit indices:
>
>     [...]
>     checking BLAS library integer size... 4
>     checking for cheev_... yes
>     checking default size of Fortran INTEGER... 4
>     [...]
>     64-bit array dims and indexing:       yes
>     64-bit BLAS array dims and indexing:  no

There are two separate issues.  First, the ability to create arrays with
more than 2^31 elements, and second, the ability to do linear algebra
and some other operations with large arrays.

With Octave 4.4 and later, if you have a 64-bit system and you have not
used the configure option --disable-64, then you should be able to
create arrays with more than 2^31 elements.  That is shown by the

     64-bit array dims and indexing:       yes

result in the configure script summary.

If you don't have LAPACK, BLAS, and other (typically Fortran-based)
libraries compiled to use 64-bit indexing as well, then you won't be
able to do linear algebra or the other operations provided by those
libraries using those large arrays that you can create.  That is shown
by the

     64-bit BLAS array dims and indexing:  no

result in the configure script summary.

You should be able to test this even on a system with only 3-4GB of
memory using something like

   n = 2_100_000_000;
   x = ones (n, 1, 'int8');

If this isn't working properly and you have Octave configured with
64-bit indexing enabled, then let's try to figure out why.

jwe