Please, what settings do I need to change so Octave can complete the following computation without error. I have 3GB installed memory, and Process Explorer says I have over 2GB of physical memory available: + size (A) _______________________________________________ Help-octave mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/help-octave |
Ether Jones-2 wrote
> Please, what settings do I need to changeso Octave can complete the > following computation without error. I have 3GB installed memory, and > Process Explorer says I have over 2GB of physical memory available: > > + size (A) > 25362 3331 > > + size (b) > 25362 1 > > + X = pinv(A) * b; > > error: memory exhausted or requested size too large > for range of Octave's index type > error: execution failed The Matrix itself is about 0.7 GB (= 25362 * 3331 * 8 Byte). Now explicitly computing the pseudo-inverse is 2*0.7 GB (you see where the story goes), plus some intermediate storage for the algorithm and your system is out of memory (assuming, your operating system also requires some of the 3 GB). On my machine (16 GB of main memory) I was able to figure out the memory requirements using `pinv` for a random dense matrix and tortured my system: + A = rand (25362, 3331); + b = rand (25362, 1); # 2.2 GB (Octave only) for intermediate computations + tic; X1 = pinv(A) * b; toc Elapsed time is 452.245 seconds. # 1.3 GB (Octave only) for intermediate computations + tic; X2 = A \ b; toc Elapsed time is 31.0354 seconds. # Both solutions almost identical + norm (X1 - X2) ans = 1.0875e-14 + max (X1 - X2) ans = 6.1409e-16 Thus `pinv` failed for your example due to some missing megabytes I assume. Anyway there is help for you: Use Octave's swiss army knife solver "left division" [1], a.k.a "mldivide" [2], a.k.a "backslash": `x = A \ b` like in the example above for `X2`. In your case `A \ b` is equivalent to `pinv (A) * b`, but "mldivide" uses a much more intelligent algorithm: it does not compute the pseudo-inverse explicitly (you don't need that pseudo-inverse either!) and computes the least-squares solution more directly, thus saving you lots of intermediate memory and computation time. Depending on the structure of A, you can further optimize your example: 1) Is your matrix dense, or are there more 0 entries, than non-zero ones? ==> Sparse matrices 2) Is your matrix structured somehow? ==> `A \ b` will find out, or `matrix_type` [3] HTH, Kai [1]: https://www.gnu.org/software/octave/doc/interpreter/Arithmetic-Ops.html [2]: https://www.gnu.org/software/octave/doc/interpreter/XREFmldivide [3]: https://www.gnu.org/software/octave/doc/interpreter/XREFmatrix_005ftype -- Sent from: http://octave.1599824.n4.nabble.com/Octave-General-f1599825.html _______________________________________________ Help-octave mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/help-octave |
siko1056 wrote
> # 1.3 GB (Octave only) for intermediate computations > + tic; X2 = A \ b; toc > Elapsed time is 31.0354 seconds. Thank you, that was very helpful. If I may ask, would you try this on your system and tell me if it gives the same answer (to a reasonable accuracy?): tic; X3 = (A'*A)\(A'*b); toc . -- Sent from: http://octave.1599824.n4.nabble.com/Octave-General-f1599825.html _______________________________________________ Help-octave mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/help-octave |
On Thu, Oct 12, 2017 at 11:07 AM, Ether Jones <[hidden email]> wrote: -- siko1056 wrote I get this >> tic; X3 = (A'*A)\(A'*b); toc Elapsed time is 4.46032 seconds. >> tic; X2 = A \ b; toc Elapsed time is 38.9548 seconds. >> _______________________________________________ Help-octave mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/help-octave |
Doug Stewart-4 wrote
> I get this > >>> tic; X3 = (A'*A)\(A'*b); toc > Elapsed time is 4.46032 seconds. >>> tic; X2 = A \ b; toc > Elapsed time is 38.9548 seconds. >>> Thank you for replying; I was asking how closely the two methods agreed; Could you do this please: norm (X2 - X3) max (X2 - X3) . -- Sent from: http://octave.1599824.n4.nabble.com/Octave-General-f1599825.html _______________________________________________ Help-octave mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/help-octave |
On Thu, Oct 12, 2017 at 11:50 AM, Ether Jones <[hidden email]> wrote: Doug Stewart-4 wrote >> norm (X2 - X3) ans = 9.5485e-14 >> >> max (X2 - X3) ans = 5.9640e-15 >> _______________________________________________ Help-octave mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/help-octave |
Free forum by Nabble | Edit this page |