"out of memory or dimension too large for Octave's index type"

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

"out of memory or dimension too large for Octave's index type"

Yan Zhang

Dear Octave Community,

 

I got this "out of memory or dimension too large for Octave's index type" issue when I was trying to create a matrix of size 5184 by 3904 in an octave command line window. The screen shot when this issue happened is attached as ‘Capture2.png’.

 

I’m sure that octave can handle/create a matrix of such size, as I can successfully create such a matrix if I restart octave cli, as shown in attached ‘Capture3.png’.

 

The only thing is that the octave program has been running for a while. But I’ve cleared all the variables by running ‘clear all’ command and results of ‘whos’ command show no variable exists in local workspace.

 

The version of octave is 4.2.1 32 bit, OS is Windows 7 64bit. I understand that switching to octave 64bit might solve this issue, but it is not a viable solution for my current situation.

 

Could anyone help me with this problem?

 

Thanks a lot!

 

Best Regards,

 

Yan

 


_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave

Capture2.PNG (35K) Download Attachment
Capture3.PNG (37K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: "out of memory or dimension too large for Octave's index type"

Francesco Potortì
>I got this "out of memory or dimension too large for Octave's index
>type" issue when I was trying to create a matrix of size 5184 by 3904
...
>The only thing is that the octave program has been running for a
>while. But I've cleared all the variables by running 'clear all'
>command and results of 'whos' command show no variable exists in local
>workspace.
>
>The version of octave is 4.2.1 32 bit, OS is Windows 7 64bit. I
>understand that switching to octave 64bit might solve this issue, but
>it is not a viable solution for my current situation.

The size of the matrix is 160MB, nothing to do with 64 bit indexing.

As far as I can remember, this may have to do with memory fragmentation
under Windows.  I am sure Windows users can be more specific, and that
if you look for that keyword you should find earlier explanations, but I
suspect that rebooting the system will help at least for a while.

--
Francesco Potortì (ricercatore)        Voice:  +39.050.621.3058
ISTI - Area della ricerca CNR          Mobile: +39.348.8283.107
via G. Moruzzi 1, I-56124 Pisa         Skype:  wnlabisti
(entrance 20, 1st floor, room C71)     Web:    http://fly.isti.cnr.it


_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave
Reply | Threaded
Open this post in threaded view
|

Re: "out of memory or dimension too large for Octave's index type"

PhilipNienhuis
Francesco Potortì wrote

>>I got this "out of memory or dimension too large for Octave's index
>>type" issue when I was trying to create a matrix of size 5184 by 3904
> ...
>>The only thing is that the octave program has been running for a
>>while. But I've cleared all the variables by running 'clear all'
>>command and results of 'whos' command show no variable exists in local
>>workspace.
>>
>>The version of octave is 4.2.1 32 bit, OS is Windows 7 64bit. I
>>understand that switching to octave 64bit might solve this issue, but
>>it is not a viable solution for my current situation.
>
> The size of the matrix is 160MB, nothing to do with 64 bit indexing.
>
> As far as I can remember, this may have to do with memory fragmentation
> under Windows.  I am sure Windows users can be more specific, and that
> if you look for that keyword you should find earlier explanations, but I
> suspect that rebooting the system will help at least for a while.

I think Francesco summed it up quite nicely.

A thing you might check is the size of the JVM (Java virtual machine), in
far-fetched cases it may hold quite a bit of data. The octave function
javamem is your friend. But to be honest I don't expect much gain there.

Other options would be checking Windows' Task Manager for resource hogs
(usually there are ample of those around) and -if you are administrator-
disabling unneeded Windows services (also occupying resources).
Browser windows & tabs can also claim amazing amounts of RAM, as do Office
programs.

Indeed, 64-bit Octave would help a lot. 32-bit programs are getting outdated
these days.

Other than that, restarting Octave and in the end even restarting your
system is the only way to get further.

Philip




--
Sent from: http://octave.1599824.n4.nabble.com/Octave-General-f1599825.html

_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave
Reply | Threaded
Open this post in threaded view
|

RE: "out of memory or dimension too large for Octave's index type"

Yan Zhang
In reply to this post by Francesco Potortì
Hi Francesco and Philip,

Thank you very much for the help.

Restarting octave is not an option as the "out of memory" issue happens in the middle of a program. Although I didn't use any java interface functions in the program, I tried the "javamem" function and found no issues. I also checked the windows services and processes but found no abnormal usage of memory.

It also occurred to me that the issue might be caused by fragmentation before I posted my question, but I couldn't find a solution to it other than switching to 64 bit octave. I'm wondering if there is  a way to defragment memory used by octave (but not by restarting), or any other suggestions.

Thanks.

Best Regards,

Yan


-----Original Message-----
From: Francesco Potortì [mailto:[hidden email]]
Sent: Monday, January 15, 2018 11:21 AM
To: Yan Zhang
Cc: [hidden email]
Subject: Re: "out of memory or dimension too large for Octave's index type"

>I got this "out of memory or dimension too large for Octave's index
>type" issue when I was trying to create a matrix of size 5184 by 3904
...
>The only thing is that the octave program has been running for a while.
>But I've cleared all the variables by running 'clear all'
>command and results of 'whos' command show no variable exists in local
>workspace.
>
>The version of octave is 4.2.1 32 bit, OS is Windows 7 64bit. I
>understand that switching to octave 64bit might solve this issue, but
>it is not a viable solution for my current situation.

The size of the matrix is 160MB, nothing to do with 64 bit indexing.

As far as I can remember, this may have to do with memory fragmentation under Windows.  I am sure Windows users can be more specific, and that if you look for that keyword you should find earlier explanations, but I suspect that rebooting the system will help at least for a while.

--
Francesco Potortì (ricercatore)        Voice:  +39.050.621.3058
ISTI - Area della ricerca CNR          Mobile: +39.348.8283.107
via G. Moruzzi 1, I-56124 Pisa         Skype:  wnlabisti
(entrance 20, 1st floor, room C71)     Web:    http://fly.isti.cnr.it
_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave
Reply | Threaded
Open this post in threaded view
|

Re: "out of memory or dimension too large for Octave's index type"

Francesco Potortì
Yan Zhang:

>>I got this "out of memory or dimension too large for Octave's index
>>type" issue when I was trying to create a matrix of size 5184 by 3904
>...
>>The only thing is that the octave program has been running for a while.
>>But I've cleared all the variables by running 'clear all'
>>command and results of 'whos' command show no variable exists in local
>>workspace.
>>
>>The version of octave is 4.2.1 32 bit, OS is Windows 7 64bit. I
>>understand that switching to octave 64bit might solve this issue, but
>>it is not a viable solution for my current situation.

Francesco Potortì:
>The size of the matrix is 160MB, nothing to do with 64 bit indexing.
>
>As far as I can remember, this may have to do with memory fragmentation
>under Windows.  I am sure Windows users can be more specific, and that
>if you look for that keyword you should find earlier explanations, but
>I suspect that rebooting the system will help at least for a while.

Yan Zhang:
>Restarting octave is not an option as the "out of memory" issue happens
>in the middle of a program. Although I didn't use any java interface
>functions in the program, I tried the "javamem" function and found no
>issues. I also checked the windows services and processes but found no
>abnormal usage of memory.

As also Philip was suggesting, have you considered rebooting your system
just before launching Octave, and avoid launching any other programs
while Octave runs?  Fragmentation is a system issue, non Octave's.

>It also occurred to me that the issue might be caused by fragmentation
>before I posted my question, but I couldn't find a solution to it other
>than switching to 64 bit octave. I'm wondering if there is a way to
>defragment memory used by octave (but not by restarting), or any other
>suggestions.

As far as I understand, memory fragmentation is a system problem on
Windows for 32-bit programs, and you can do nothing apart from the
suggestion the Philips and me already gave.  Again, if you look in this
list archives for those keywords you may be able to find some more info.

--
Francesco Potortì (ricercatore)        Voice:  +39.050.621.3058
ISTI - Area della ricerca CNR          Mobile: +39.348.8283.107
via G. Moruzzi 1, I-56124 Pisa         Skype:  wnlabisti
(entrance 20, 1st floor, room C71)     Web:    http://fly.isti.cnr.it


_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave
Reply | Threaded
Open this post in threaded view
|

Re: "out of memory or dimension too large for Octave's index type"

Ian McCallion
In case it helps, I have had one or two unexpected occurrences of this message, but only since 4.2.0. I did not try to track down the source of the problem and worked around it. I'm afraid I associated it with an increased frequency of octave crashes that started at about the same time. I'm using win10 home.

Yan, can you try using Octave 4.0.x to see if that solves it?

Cheers... Ian

On 16 Jan 2018 2:34 pm, "Francesco Potortì" <[hidden email]> wrote:
Yan Zhang:
>>I got this "out of memory or dimension too large for Octave's index
>>type" issue when I was trying to create a matrix of size 5184 by 3904
>...
>>The only thing is that the octave program has been running for a while.
>>But I've cleared all the variables by running 'clear all'
>>command and results of 'whos' command show no variable exists in local
>>workspace.
>>
>>The version of octave is 4.2.1 32 bit, OS is Windows 7 64bit. I
>>understand that switching to octave 64bit might solve this issue, but
>>it is not a viable solution for my current situation.

Francesco Potortì:
>The size of the matrix is 160MB, nothing to do with 64 bit indexing.
>
>As far as I can remember, this may have to do with memory fragmentation
>under Windows.  I am sure Windows users can be more specific, and that
>if you look for that keyword you should find earlier explanations, but
>I suspect that rebooting the system will help at least for a while.

Yan Zhang:
>Restarting octave is not an option as the "out of memory" issue happens
>in the middle of a program. Although I didn't use any java interface
>functions in the program, I tried the "javamem" function and found no
>issues. I also checked the windows services and processes but found no
>abnormal usage of memory.

As also Philip was suggesting, have you considered rebooting your system
just before launching Octave, and avoid launching any other programs
while Octave runs?  Fragmentation is a system issue, non Octave's.

>It also occurred to me that the issue might be caused by fragmentation
>before I posted my question, but I couldn't find a solution to it other
>than switching to 64 bit octave. I'm wondering if there is a way to
>defragment memory used by octave (but not by restarting), or any other
>suggestions.

As far as I understand, memory fragmentation is a system problem on
Windows for 32-bit programs, and you can do nothing apart from the
suggestion the Philips and me already gave.  Again, if you look in this
list archives for those keywords you may be able to find some more info.

--
Francesco Potortì (ricercatore)        Voice:  +39.050.621.3058
ISTI - Area della ricerca CNR          Mobile: +39.348.8283.107
via G. Moruzzi 1, I-56124 Pisa         Skype:  wnlabisti
(entrance 20, 1st floor, room C71)     Web:    http://fly.isti.cnr.it


_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave


_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave
Reply | Threaded
Open this post in threaded view
|

RE: "out of memory or dimension too large for Octave's index type"

mmuetzel
In reply to this post by Yan Zhang
Hi Yan,

If you know the size of the matrices you need already at the start of your
program, you could try and "reserve" a continuous memory block for it. You
might be able to do something like:

function A = fragments_memory ()
A = zeros(5184,3904, typename);

... (the lines that fragment your memory)

A(:,:) = what_it_should_be;

endfunction

Best regards,
Markus




--
Sent from: http://octave.1599824.n4.nabble.com/Octave-General-f1599825.html

_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave
Reply | Threaded
Open this post in threaded view
|

RE: "out of memory or dimension too large for Octave's index type"

Yan Zhang
In reply to this post by Francesco Potortì
Hi Francesco,

Thank you again for the help.

Best Regards,

Yan

-----Original Message-----
From: Francesco Potortì [mailto:[hidden email]]
Sent: Tuesday, January 16, 2018 6:17 AM
To: Yan Zhang
Cc: [hidden email]; PhilipNienhuis
Subject: Re: "out of memory or dimension too large for Octave's index type"

Yan Zhang:

>>I got this "out of memory or dimension too large for Octave's index
>>type" issue when I was trying to create a matrix of size 5184 by 3904
>...
>>The only thing is that the octave program has been running for a while.
>>But I've cleared all the variables by running 'clear all'
>>command and results of 'whos' command show no variable exists in local
>>workspace.
>>
>>The version of octave is 4.2.1 32 bit, OS is Windows 7 64bit. I
>>understand that switching to octave 64bit might solve this issue, but
>>it is not a viable solution for my current situation.

Francesco Potortì:
>The size of the matrix is 160MB, nothing to do with 64 bit indexing.
>
>As far as I can remember, this may have to do with memory fragmentation
>under Windows.  I am sure Windows users can be more specific, and that
>if you look for that keyword you should find earlier explanations, but
>I suspect that rebooting the system will help at least for a while.

Yan Zhang:
>Restarting octave is not an option as the "out of memory" issue happens
>in the middle of a program. Although I didn't use any java interface
>functions in the program, I tried the "javamem" function and found no
>issues. I also checked the windows services and processes but found no
>abnormal usage of memory.

As also Philip was suggesting, have you considered rebooting your system just before launching Octave, and avoid launching any other programs while Octave runs?  Fragmentation is a system issue, non Octave's.

>It also occurred to me that the issue might be caused by fragmentation
>before I posted my question, but I couldn't find a solution to it other
>than switching to 64 bit octave. I'm wondering if there is a way to
>defragment memory used by octave (but not by restarting), or any other
>suggestions.

As far as I understand, memory fragmentation is a system problem on Windows for 32-bit programs, and you can do nothing apart from the suggestion the Philips and me already gave.  Again, if you look in this list archives for those keywords you may be able to find some more info.

--
Francesco Potortì (ricercatore)        Voice:  +39.050.621.3058
ISTI - Area della ricerca CNR          Mobile: +39.348.8283.107
via G. Moruzzi 1, I-56124 Pisa         Skype:  wnlabisti
(entrance 20, 1st floor, room C71)     Web:    http://fly.isti.cnr.it
_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave
Reply | Threaded
Open this post in threaded view
|

RE: "out of memory or dimension too large for Octave's index type"

Yan Zhang
In reply to this post by Ian McCallion

Hi Ian,

 

Thanks for the help. I ran into ‘out of memory’ issue even more when I was using octave 4.0.0.  Actually it’s why I upgraded to 4.2.0.

 

Best Regards,

 

Yan

 

From: Help-octave [mailto:help-octave-bounces+yan.zhang=[hidden email]] On Behalf Of Ian McCallion
Sent: Tuesday, January 16, 2018 7:04 AM
Cc: [hidden email]
Subject: Re: "out of memory or dimension too large for Octave's index type"

 

In case it helps, I have had one or two unexpected occurrences of this message, but only since 4.2.0. I did not try to track down the source of the problem and worked around it. I'm afraid I associated it with an increased frequency of octave crashes that started at about the same time. I'm using win10 home.

 

Yan, can you try using Octave 4.0.x to see if that solves it?

 

Cheers... Ian

 

On 16 Jan 2018 2:34 pm, "Francesco Potortì" <[hidden email]> wrote:

Yan Zhang:

>>I got this "out of memory or dimension too large for Octave's index
>>type" issue when I was trying to create a matrix of size 5184 by 3904
>...
>>The only thing is that the octave program has been running for a while.
>>But I've cleared all the variables by running 'clear all'
>>command and results of 'whos' command show no variable exists in local
>>workspace.
>>
>>The version of octave is 4.2.1 32 bit, OS is Windows 7 64bit. I
>>understand that switching to octave 64bit might solve this issue, but
>>it is not a viable solution for my current situation.

Francesco Potortì:

>The size of the matrix is 160MB, nothing to do with 64 bit indexing.
>
>As far as I can remember, this may have to do with memory fragmentation
>under Windows.  I am sure Windows users can be more specific, and that
>if you look for that keyword you should find earlier explanations, but
>I suspect that rebooting the system will help at least for a while.

Yan Zhang:

>Restarting octave is not an option as the "out of memory" issue happens
>in the middle of a program. Although I didn't use any java interface
>functions in the program, I tried the "javamem" function and found no
>issues. I also checked the windows services and processes but found no
>abnormal usage of memory.

As also Philip was suggesting, have you considered rebooting your system
just before launching Octave, and avoid launching any other programs
while Octave runs?  Fragmentation is a system issue, non Octave's.


>It also occurred to me that the issue might be caused by fragmentation
>before I posted my question, but I couldn't find a solution to it other
>than switching to 64 bit octave. I'm wondering if there is a way to
>defragment memory used by octave (but not by restarting), or any other
>suggestions.

As far as I understand, memory fragmentation is a system problem on
Windows for 32-bit programs, and you can do nothing apart from the
suggestion the Philips and me already gave.  Again, if you look in this
list archives for those keywords you may be able to find some more info.


--
Francesco Potortì (ricercatore)        Voice:  +39.050.621.3058
ISTI - Area della ricerca CNR          Mobile: +39.348.8283.107
via G. Moruzzi 1, I-56124 Pisa         Skype:  wnlabisti
(entrance 20, 1st floor, room C71)     Web:    http://fly.isti.cnr.it


_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave

 


_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave
Reply | Threaded
Open this post in threaded view
|

RE: "out of memory or dimension too large for Octave's index type"

Yan Zhang
In reply to this post by Ian McCallion

Hi Ian,

 

Thanks for the help. I ran into ‘out of memory’ issue even more when I was using octave 4.0.0.  Actually it’s why I upgraded to 4.2.0.

 

Best Regards,

 

Yan

 

From: Help-octave [mailto:help-octave-bounces+yan.zhang=[hidden email]] On Behalf Of Ian McCallion
Sent: Tuesday, January 16, 2018 7:04 AM
Cc: [hidden email]
Subject: Re: "out of memory or dimension too large for Octave's index type"

 

In case it helps, I have had one or two unexpected occurrences of this message, but only since 4.2.0. I did not try to track down the source of the problem and worked around it. I'm afraid I associated it with an increased frequency of octave crashes that started at about the same time. I'm using win10 home.

 

Yan, can you try using Octave 4.0.x to see if that solves it?

 

Cheers... Ian

 

On 16 Jan 2018 2:34 pm, "Francesco Potortì" <[hidden email]> wrote:

Yan Zhang:

>>I got this "out of memory or dimension too large for Octave's index
>>type" issue when I was trying to create a matrix of size 5184 by 3904
>...
>>The only thing is that the octave program has been running for a while.
>>But I've cleared all the variables by running 'clear all'
>>command and results of 'whos' command show no variable exists in local
>>workspace.
>>
>>The version of octave is 4.2.1 32 bit, OS is Windows 7 64bit. I
>>understand that switching to octave 64bit might solve this issue, but
>>it is not a viable solution for my current situation.

Francesco Potortì:

>The size of the matrix is 160MB, nothing to do with 64 bit indexing.
>
>As far as I can remember, this may have to do with memory fragmentation
>under Windows.  I am sure Windows users can be more specific, and that
>if you look for that keyword you should find earlier explanations, but
>I suspect that rebooting the system will help at least for a while.

Yan Zhang:

>Restarting octave is not an option as the "out of memory" issue happens
>in the middle of a program. Although I didn't use any java interface
>functions in the program, I tried the "javamem" function and found no
>issues. I also checked the windows services and processes but found no
>abnormal usage of memory.

As also Philip was suggesting, have you considered rebooting your system
just before launching Octave, and avoid launching any other programs
while Octave runs?  Fragmentation is a system issue, non Octave's.


>It also occurred to me that the issue might be caused by fragmentation
>before I posted my question, but I couldn't find a solution to it other
>than switching to 64 bit octave. I'm wondering if there is a way to
>defragment memory used by octave (but not by restarting), or any other
>suggestions.

As far as I understand, memory fragmentation is a system problem on
Windows for 32-bit programs, and you can do nothing apart from the
suggestion the Philips and me already gave.  Again, if you look in this
list archives for those keywords you may be able to find some more info.


--
Francesco Potortì (ricercatore)        Voice:  +39.050.621.3058
ISTI - Area della ricerca CNR          Mobile: +39.348.8283.107
via G. Moruzzi 1, I-56124 Pisa         Skype:  wnlabisti
(entrance 20, 1st floor, room C71)     Web:    http://fly.isti.cnr.it


_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave

 


_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave
Reply | Threaded
Open this post in threaded view
|

RE: "out of memory or dimension too large for Octave's index type"

Yan Zhang
In reply to this post by mmuetzel
Hi Markus,

That's a really smart idea. Thank you!

Best Regards,

Yan

-----Original Message-----
From: Help-octave [mailto:help-octave-bounces+yan.zhang=[hidden email]] On Behalf Of mmuetzel
Sent: Tuesday, January 16, 2018 7:59 AM
To: [hidden email]
Subject: RE: "out of memory or dimension too large for Octave's index type"

Hi Yan,

If you know the size of the matrices you need already at the start of your program, you could try and "reserve" a continuous memory block for it. You might be able to do something like:

function A = fragments_memory ()
A = zeros(5184,3904, typename);

... (the lines that fragment your memory)

A(:,:) = what_it_should_be;

endfunction

Best regards,
Markus




--
Sent from: http://octave.1599824.n4.nabble.com/Octave-General-f1599825.html

_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave

_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave
Reply | Threaded
Open this post in threaded view
|

Re: "out of memory or dimension too large for Octave's index type"

Francesco Potortì
mmuetzel:

>If you know the size of the matrices you need already at the start of
>your program, you could try and "reserve" a continuous memory block for
>it. You might be able to do something like:
>
>function A = fragments_memory ()
>A = zeros(5184,3904, typename);
>
>... (the lines that fragment your memory)
>
>A(:,:) = what_it_should_be;
>
>endfunction

Oh, right, I had taken that for granted, while I shouldn't have. What
Marku sugegsts is (probably the most useful of) a series of optimization
techniqus to care for when you program in Octave using not-so-small
data.

Have a look at chapter "Vectorization and Faster Code Execution" in the
Octave manual for more ideas.

--
Francesco Potortì (ricercatore)        Voice:  +39.050.621.3058
ISTI - Area della ricerca CNR          Mobile: +39.348.8283.107
via G. Moruzzi 1, I-56124 Pisa         Skype:  wnlabisti
(entrance 20, 1st floor, room C71)     Web:    http://fly.isti.cnr.it


_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave