out of memory or dimension too large for Octave's index type

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

out of memory or dimension too large for Octave's index type

mtalberg
Hello

I am having an issue with the amount of memory Octave can use compared to the available memory on my system. I am running on Window's 7 64-bit, 16GB of RAM, and have GNU Octave version 3.8.2, configured for "i686-w64-mingw32".

I saw this error after trying to write a 9329664x2 matrix to a .wav file using the wavwrite command.

I monitor the memory usage of my computer as a whole, and have determined that Octave is only using roughly 500MB of RAM. Is there any way to boost the amount of RAM accessible by Octave? Any help would be greatly appreciated, thank you!
Reply | Threaded
Open this post in threaded view
|

Re: out of memory or dimension too large for Octave's index type

Philip Nienhuis
mtalberg wrote
Hello

I am having an issue with the amount of memory Octave can use compared to the available memory on my system. I am running on Window's 7 64-bit, 16GB of RAM, and have GNU Octave version 3.8.2, configured for "i686-w64-mingw32".

I saw this error after trying to write a 9329664x2 matrix to a .wav file using the wavwrite command.

I monitor the memory usage of my computer as a whole, and have determined that Octave is only using roughly 500MB of RAM. Is there any way to boost the amount of RAM accessible by Octave? Any help would be greatly appreciated, thank you!
A few months ago I replied to a similar question here in more detail, you can search the mailing list archive for that.
In short:
You're hit by memory fragmentation, and that is very "normal" on any OS.
Octave isn't even so bad in this respect. I tried both Octave and Matlab on the same PC (32bit XP w. 4 GB RAM) to find out how much memory was available. Matlab has a call for that, in Octave one needs trial and error.
Turned out the largest contiguous piece of RAM ML could assign amounted to only (IIRC) ~600 MB. Octave was marginally better, but that number had to be inferred (indirectly).

You'd have to wait for a 64-bit Octave. I heard rumours that that shouldn't take too long now.
Cross-building it on Linux is fairly easy these days. I have one here, Markus has one, but there needs to be some polishing done before it can be -unofficially-  "released".

Philip
Reply | Threaded
Open this post in threaded view
|

Re: out of memory or dimension too large for Octave's index type

marco atzeri-2
On 07/09/2014 13:50, Philip Nienhuis wrote:

> mtalberg wrote
>> Hello
>>
>> I am having an issue with the amount of memory Octave can use compared to
>> the available memory on my system. I am running on Window's 7 64-bit, 16GB
>> of RAM, and have GNU Octave version 3.8.2, configured for
>> "i686-w64-mingw32".
>>
>> I saw this error after trying to write a 9329664x2 matrix to a .wav file
>> using the wavwrite command.
>>
>
> You'd have to wait for a 64-bit Octave. I heard rumours that that shouldn't
> take too long now.

in the mean time you can try the octave built for cygwin 64 bit.

http://www.cygwin.com/ml/cygwin-announce/2014-08/msg00033.html

>
> Philip


Regards
Marco

_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave
Reply | Threaded
Open this post in threaded view
|

Re: out of memory or dimension too large for Octave's index type

tmacchant
In reply to this post by Philip Nienhuis
----- Original Message -----

> From: Philip Nienhuis 
> To: help-octave
> Cc:
> Date: 2014/9/7, Sun 20:50
> Subject: Re: out of memory or dimension too large for Octave's index type
>
> mtalberg wrote
>>  Hello
>>
>>  I am having an issue with the amount of memory Octave can use compared to
>>  the available memory on my system. I am running on Window's 7 64-bit,
> 16GB
>>  of RAM, and have GNU Octave version 3.8.2, configured for
>>  "i686-w64-mingw32".
>>
>>  I saw this error after trying to write a 9329664x2 matrix to a .wav file
>>  using the wavwrite command.
>>
>>  I monitor the memory usage of my computer as a whole, and have determined
>>  that Octave is only using roughly 500MB of RAM. Is there any way to boost
>>  the amount of RAM accessible by Octave? Any help would be greatly
>>  appreciated, thank you!
>
> A few months ago I replied to a similar question here in more detail, you
> can search the mailing list archive for that.
> In short:
> You're hit by memory fragmentation, and that is very "normal" on
> any OS.
> Octave isn't even so bad in this respect. I tried both Octave and Matlab on
> the same PC (32bit XP w. 4 GB RAM) to find out how much memory was
> available. Matlab has a call for that, in Octave one needs trial and error.
> Turned out the largest contiguous piece of RAM ML could assign amounted to
> only (IIRC) ~600 MB. Octave was marginally better, but that number had to be
> inferred (indirectly).
>
> You'd have to wait for a 64-bit Octave. I heard rumours that that
> shouldn't
> take too long now.
> Cross-building it on Linux is fairly easy these days. I have one here,
> Markus has one, but there needs to be some polishing done before it can be
> -unofficially-  "released".
>
> Philip
>  
Unofficial Large Indexing was build with enable-64 in 
mxeoctave.osuv.de.


http://mxeoctave.osuv.de/


Note that the binary are *experimental*!!!

Tatsuro


_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave
Reply | Threaded
Open this post in threaded view
|

Re: out of memory or dimension too large for Octave's index type

Markus Bergholz



On Sun, Sep 7, 2014 at 4:06 PM, Tatsuro MATSUOKA <[hidden email]> wrote:
----- Original Message -----

> From: Philip Nienhuis 
> To: help-octave
> Cc:
> Date: 2014/9/7, Sun 20:50
> Subject: Re: out of memory or dimension too large for Octave's index type
>
> mtalberg wrote
>>  Hello
>>
>>  I am having an issue with the amount of memory Octave can use compared to
>>  the available memory on my system. I am running on Window's 7 64-bit,
> 16GB
>>  of RAM, and have GNU Octave version 3.8.2, configured for
>>  "i686-w64-mingw32".
>>
>>  I saw this error after trying to write a 9329664x2 matrix to a .wav file
>>  using the wavwrite command.
>>
>>  I monitor the memory usage of my computer as a whole, and have determined
>>  that Octave is only using roughly 500MB of RAM. Is there any way to boost
>>  the amount of RAM accessible by Octave? Any help would be greatly
>>  appreciated, thank you!
>
> A few months ago I replied to a similar question here in more detail, you
> can search the mailing list archive for that.
> In short:
> You're hit by memory fragmentation, and that is very "normal" on
> any OS.
> Octave isn't even so bad in this respect. I tried both Octave and Matlab on
> the same PC (32bit XP w. 4 GB RAM) to find out how much memory was
> available. Matlab has a call for that, in Octave one needs trial and error.
> Turned out the largest contiguous piece of RAM ML could assign amounted to
> only (IIRC) ~600 MB. Octave was marginally better, but that number had to be
> inferred (indirectly).
>
> You'd have to wait for a 64-bit Octave. I heard rumours that that
> shouldn't
> take too long now.
> Cross-building it on Linux is fairly easy these days. I have one here,
> Markus has one, but there needs to be some polishing done before it can be
> -unofficially-  "released".
>
> Philip
>  
Unofficial Large Indexing was build with enable-64 in 
mxeoctave.osuv.de.


http://mxeoctave.osuv.de/


Note that the binary are *experimental*!!!

yeah, very very experumental!


Another question is, do you realy need *all* data at once in your workspace? or just 2-3 columns at once?

I've put an interpreter (lua) in the octave interpreter (simple mex function), so you can use lua
* to prevent bottlenecks
* database connections
* and much more

e.g. I've got here a 1.8GB csv file (1mio lines, 100 columns). If I just want to access column number 18, it takes ~20 seconds to read it into octave.


octave:14> tic, m=lua('csvaccess','big.csv','col',18,','); toc
all done, lua stack top 0
Elapsed time is 20.6918 seconds.
octave:15> system('du -csh big.csv')
1,8G    big.csv
1,8G    insgesamt
ans =                    0
octave:16> whos
Variables in the current scope:

   Attr Name        Size                     Bytes  Class
   ==== ====        ====                     =====  =====
        ans         1x1                          8  double
        m           1x1000000              8000000  double

Total is 1000001 elements using 8000008 bytes

octave:17> memory

 Memory used by Octave:   300.188 MB
 Physical Memory (RAM): 7929.52 MB




and of course that means, you have to write a lua function for your need... but you can still write a C/C++ mex/oct file for that too. But for me, lua is easier to write.
I can put my lua function later online, but it's still experimental too :D

 

Tatsuro


_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave



--
icq: 167498924
XMPP|Jabber: [hidden email]

_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave
Reply | Threaded
Open this post in threaded view
|

Re: out of memory or dimension too large for Octave's index type

Thomas Weber-3
In reply to this post by Philip Nienhuis
On Sun, Sep 07, 2014 at 04:50:40AM -0700, Philip Nienhuis wrote:
> You're hit by memory fragmentation, and that is very "normal" on any OS.
> Octave isn't even so bad in this respect. I tried both Octave and Matlab on
> the same PC (32bit XP w. 4 GB RAM) to find out how much memory was
> available. Matlab has a call for that, in Octave one needs trial and error.

$ cat /proc/buddyinfo
might be helpful,
https://www.kernel.org/doc/Documentation/filesystems/proc.txt

        Thomas

_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave