possible memory wasting in array handling

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

possible memory wasting in array handling

Olaf Till-2
On the help list, a problem came up where a probable memory exhaustion
occured only if some large array operations, with indexing, took place
before (they took place within a called function). This reminded me of
the following possible problem with the current array handling.

The Array class maintains a pointer to (and a length of) a 'slice' of
data in ArrayRep. If a smaller array B is created by continuous
indexing of a larger one A:

A = rand (1, 1000000);

B = A(1:1000);

B references a 'slice' in the data of A, without allocating own memory
for the data. But if A is cleared (or destructed on return from a
function), all of its memory remains allocated, as long as B exists
(even though B needs only a small part of this memory). (Even on
writing into B, Array::make_unique() does not help with this if B is
the only reference to the data.)

Is this behavior considered problematic, or even a bug? It possibly
has caused the said problem on the help list. One possibility would be
to just remove this 'slice' optimization. A more difficult way would
involve to maintain, in ArrayRep, not only a refcount, but also a
notion of the greatest slice referenced by any Array.

Olaf

--
public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: possible memory wasting in array handling

Olaf Till-2
On Thu, Jan 19, 2017 at 03:16:44PM +0100, Olaf Till wrote:
> A more difficult way would
> involve to maintain, in ArrayRep, not only a refcount, but also a
> notion of the greatest slice referenced by any Array.

I'm more and more convinced that the problem caused by the described
behaviour is serious. Would you consider a patch?

(I have a test case, too. It's machine dependent and on GNU/Linux
involves switching to single user mode and turning off swapping.)

Olaf

--
public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: possible memory wasting in array handling

John W. Eaton
Administrator
On 01/20/2017 04:15 AM, Olaf Till wrote:

> On Thu, Jan 19, 2017 at 03:16:44PM +0100, Olaf Till wrote:
>> A more difficult way would
>> involve to maintain, in ArrayRep, not only a refcount, but also a
>> notion of the greatest slice referenced by any Array.
>
> I'm more and more convinced that the problem caused by the described
> behaviour is serious. Would you consider a patch?
>
> (I have a test case, too. It's machine dependent and on GNU/Linux
> involves switching to single user mode and turning off swapping.)

It should be fairly easy to make the array slice optimization optional.

If we make change things so that Octave chooses slices vs. copying based
on the slice to full array size, then that should probably also be
configurable in some way.

jwe


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: possible memory wasting in array handling

Olaf Till-2
On Tue, Jan 24, 2017 at 04:12:49PM -0500, John W. Eaton wrote:

> On 01/20/2017 04:15 AM, Olaf Till wrote:
> >On Thu, Jan 19, 2017 at 03:16:44PM +0100, Olaf Till wrote:
> >>A more difficult way would
> >>involve to maintain, in ArrayRep, not only a refcount, but also a
> >>notion of the greatest slice referenced by any Array.
> >
> >I'm more and more convinced that the problem caused by the described
> >behaviour is serious. Would you consider a patch?
> >
> >(I have a test case, too. It's machine dependent and on GNU/Linux
> >involves switching to single user mode and turning off swapping.)
>
> It should be fairly easy to make the array slice optimization optional.
>
> If we make change things so that Octave chooses slices vs. copying based on
> the slice to full array size, then that should probably also be configurable
> in some way.
Not yet using the patch tracker, since probably useless:

Testing the patch (single user mode, swapoff), unpatched gives:

octave:2> a = ones (500000000, 1);
octave:3> b = a(1:10);
octave:4> clear a
octave:5> a = ones (500000000, 1);
error: out of memory or dimension too large for Octave's index type
octave:5>

and in patched, as excepted, this can be continued further:

octave:1> a = ones (500000000, 1);
octave:2> b = a(1:10);
octave:3> clear a
octave:4> a = ones (500000000, 1);
octave:5> c = a(1:10);
octave:6> clear a
octave:7> a = ones (500000000, 1);
octave:8> ... and so on.

But if the original array is allocated in a function, even unpatched
has no problems:

octave:1> function ret = mkarray ()
> a = ones (500000000, 1);
> ret = a(1:10);
> endfunction
octave:2> b = mkarray;
octave:3> c = mkarray;
octave:4> d = mkarray;
octave:5> ... and so on.

Seems returning from function involves Array::make_unique(), which I
didn't realize.

Since the case with the function seems more typical than using 'clear'
in a script I'd say I'll dump my patch... (?)

Olaf

--
public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: possible memory wasting in array handling

Olaf Till-2
On Thu, Mar 02, 2017 at 01:38:16PM +0100, Olaf Till wrote:
> Since the case with the function seems more typical than using 'clear'
> in a script I'd say I'll dump my patch... (?)

Anyway, I've put it to the tracker:

https://savannah.gnu.org/patch/index.php?9276

Olaf

--
public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net

signature.asc (836 bytes) Download Attachment
Loading...