pinv() error message

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

pinv() error message

Ether Jones-2
Please, what settings do I need to change
so Octave can complete the following computation
without error. I have 3GB installed memory,
and Process Explorer says I have over 2GB
of physical memory available:

+ size (A)
   25362    3331

+ size (b)
   25362    1

+ X = pinv(A) * b;

error: memory exhausted or requested size too large
for range of Octave's index type
error: execution failed

_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave
Reply | Threaded
Open this post in threaded view
|

Re: pinv() error message

siko1056
Ether Jones-2 wrote

> Please, what settings do I need to changeso Octave can complete the
> following computation without error. I have 3GB installed memory, and
> Process Explorer says I have over 2GB of physical memory available:
>
> + size (A)
>    25362    3331
>
> + size (b)
>    25362    1
>
> + X = pinv(A) * b;
>
> error: memory exhausted or requested size too large
> for range of Octave's index type
> error: execution failed

The Matrix itself is about 0.7 GB (= 25362 * 3331 * 8 Byte).  Now explicitly
computing the pseudo-inverse is 2*0.7 GB (you see where the story goes),
plus some intermediate storage for the algorithm and your system is out of
memory (assuming, your operating system also requires some of the 3 GB).  On
my machine (16 GB of main memory) I was able to figure out the memory
requirements using `pinv` for a random dense matrix and tortured my system:

+ A = rand (25362, 3331);
+ b = rand (25362, 1);

# 2.2 GB (Octave only) for intermediate computations
+ tic; X1 = pinv(A) * b; toc
Elapsed time is 452.245 seconds.

# 1.3 GB (Octave only) for intermediate computations
+ tic; X2 = A \ b; toc
Elapsed time is 31.0354 seconds.

# Both solutions almost identical
+ norm (X1 - X2)
ans =    1.0875e-14
+ max (X1 - X2)
ans =    6.1409e-16

Thus `pinv` failed for your example due to some missing megabytes I assume.
Anyway there is help for you:

Use Octave's swiss army knife solver "left division" [1], a.k.a "mldivide"
[2], a.k.a "backslash": `x = A \ b` like in the example above for `X2`.

In your case `A \ b` is equivalent to `pinv (A) * b`, but "mldivide" uses a
much more intelligent algorithm: it does not compute the pseudo-inverse
explicitly (you don't need that pseudo-inverse either!) and computes the
least-squares solution more directly, thus saving you lots of intermediate
memory and computation time.

Depending on the structure of A, you can further optimize your example:

1) Is your matrix dense, or are there more 0 entries, than non-zero ones?
==> Sparse matrices
2) Is your matrix structured somehow? ==>  `A \ b` will find out, or
`matrix_type` [3]

HTH,
Kai

[1]: https://www.gnu.org/software/octave/doc/interpreter/Arithmetic-Ops.html
[2]: https://www.gnu.org/software/octave/doc/interpreter/XREFmldivide
[3]: https://www.gnu.org/software/octave/doc/interpreter/XREFmatrix_005ftype



--
Sent from: http://octave.1599824.n4.nabble.com/Octave-General-f1599825.html

_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave
Reply | Threaded
Open this post in threaded view
|

Re: pinv() error message

Ether Jones
siko1056 wrote
> # 1.3 GB (Octave only) for intermediate computations
> + tic; X2 = A \ b; toc
> Elapsed time is 31.0354 seconds.

Thank you, that was very helpful.

If I may ask, would you try this on your system and tell me if it gives the
same answer (to a reasonable accuracy?):

tic; X3 = (A'*A)\(A'*b); toc

.



--
Sent from: http://octave.1599824.n4.nabble.com/Octave-General-f1599825.html

_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave
Reply | Threaded
Open this post in threaded view
|

Re: pinv() error message

Doug Stewart-4


On Thu, Oct 12, 2017 at 11:07 AM, Ether Jones <[hidden email]> wrote:
siko1056 wrote
> # 1.3 GB (Octave only) for intermediate computations
> + tic; X2 = A \ b; toc
> Elapsed time is 31.0354 seconds.

Thank you, that was very helpful.

If I may ask, would you try this on your system and tell me if it gives the
same answer (to a reasonable accuracy?):

tic; X3 = (A'*A)\(A'*b); toc

.



I get this

>> tic; X3 = (A'*A)\(A'*b); toc
Elapsed time is 4.46032 seconds.
>>  tic; X2 = A \ b; toc
Elapsed time is 38.9548 seconds.
>> 
--
DAS


_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave
Reply | Threaded
Open this post in threaded view
|

Re: pinv() error message

Ether Jones
Doug Stewart-4 wrote
> I get this
>
>>> tic; X3 = (A'*A)\(A'*b); toc
> Elapsed time is 4.46032 seconds.
>>>  tic; X2 = A \ b; toc
> Elapsed time is 38.9548 seconds.
>>>

Thank you for replying; I was asking how closely the two methods agreed;
Could you do this please:

norm (X2 - X3)

max (X2 - X3)

.



--
Sent from: http://octave.1599824.n4.nabble.com/Octave-General-f1599825.html

_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave
Reply | Threaded
Open this post in threaded view
|

Re: pinv() error message

Doug Stewart-4


On Thu, Oct 12, 2017 at 11:50 AM, Ether Jones <[hidden email]> wrote:
Doug Stewart-4 wrote
> I get this
>
>>> tic; X3 = (A'*A)\(A'*b); toc
> Elapsed time is 4.46032 seconds.
>>>  tic; X2 = A \ b; toc
> Elapsed time is 38.9548 seconds.
>>>

Thank you for replying; I was asking how closely the two methods agreed;
Could you do this please:

norm (X2 - X3)

max (X2 - X3)

.

>> norm (X2 - X3)
ans =    9.5485e-14
>>
>> max (X2 - X3)
ans =    5.9640e-15
>>

--
DAS


_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave