

If you really have to avoid the loop, you could try to use "arrayfun" with an
anonymous function (which kind of hides the loop).
I can't think of a straightforward "nonloopy" solution of your problem
from the top of my head. I would be interested to see one, too.
Markus

Sent from: https://octave.1599824.n4.nabble.com/OctaveGeneralf1599825.html


If you really have to avoid the loop, you could try to use "arrayfun" with an
anonymous function (which kind of hides the loop).
I can't think of a straightforward "nonloopy" solution of your problem
from the top of my head. I would be interested to see one, too.
Markus
I have a working solution, but is this a homework assignment? I'm hesitant to just post the full code.
The solution is almost always to spread the looped operation over an expanded array (here it can be kept to two dimensions. I've frequently made use of ndimensional operations, and broadcast projection for this. for large data sets you have to be careful, as you're trading time for memory)
you're working with you c matrix as a single column of data, and looping 4 times, once over each a (and b, but they're looped together, so it's only one extra dimension). you can make four parallel copies of the c matrix, and then apply the min function. a common way to do quick array duplication is:
c = c(:); cc = c .* ones(1,4);
or more generally
cc = c.* ones(1,numel(a));
then, you notice that min and max functions apply only in one dimension. so min(cc) will give you the min of each column.
the tricky part is that you aren't taking the min of the full column, just a portion of it. so you'll need to figure out how to sample only a portion of each column, as specified in a and b, maybe using logical indexing, and take the min over that. this is complicated by it being a different length portion each time.


Alternatively, you could use ACCUMARRAY (with subscripts created by REPELEM) to apply MIN to groups of values extracted from 'c'. The only slightly nontrivial bit is generating the
indices into 'c', but there are very standard CUMSUMbased ways of doing so.
Regards,
Bård Skaflestad
SINTEF Digital, Mathematics & Cybernetics
Computational Geosciences group
From: Helpoctave <helpoctavebounces+bard.skaflestad=[hidden email]>
On Behalf Of Nicholas Jankowski
Sent: Thursday, August 22, 2019 4:07 PM
To: mmuetzel <[hidden email]>; shivax <[hidden email]>; LucaLuca <[hidden email]>
Cc: Help GNU Octave <[hidden email]>
Subject: Re: a difficult homework
If you really have to avoid the loop, you could try to use "arrayfun" with an
anonymous function (which kind of hides the loop).
I can't think of a straightforward "nonloopy" solution of your problem
from the top of my head. I would be interested to see one, too.
Markus
I have a working solution, but is this a homework assignment? I'm hesitant to just post the full code.
The solution is almost always to spread the looped operation over an expanded array (here it can be kept to two dimensions. I've frequently made use of ndimensional operations, and broadcast projection for this. for large data sets you
have to be careful, as you're trading time for memory)
you're working with you c matrix as a single column of data, and looping 4 times, once over each a (and b, but they're looped together, so it's only one extra dimension). you can make four parallel copies of the c matrix, and then apply
the min function. a common way to do quick array duplication is:
cc = c.* ones(1,numel(a));
then, you notice that min and max functions apply only in one dimension. so min(cc) will give you the min of each column.
the tricky part is that you aren't taking the min of the full column, just a portion of it. so you'll need to figure out how to sample only a portion of each column, as specified in a and b, maybe using logical indexing, and take the min
over that. this is complicated by it being a different length portion each time.


thank you for answer
mmuetzel : can you give me an example to use arrayfun with mine?
nrjank : i can't use your method because i need to matrix so large...and i'm
afraid to finish memory
Bård Skaflestad: can you give me an example to use your method with mine?
thank for patience

Sent from: https://octave.1599824.n4.nabble.com/OctaveGeneralf1599825.html


On Thu, Aug 22, 2019 at 1:46 PM shivax via Help list for GNU Octave < [hidden email]> wrote: nrjank : i can't use your method because i need to matrix so large...and i'm
afraid to finish memory
how large does yours need to be? It would take exceptionally large amounts of data to hit memory limits for something you showed that can be done in 2D.
example: I had another problem where I was manipulating arrays of 2x2 matrices. I used broadcast projectionof the matrices along 'axes' for time, length, frequency, and eigen space, and then additional artificial axes to avoid loops, simplify calculations, etc. with arbitrary refinement for convergence, especially with poorly chosen initial base, I found out how to quickly exhaust memory on a 32bit system.
(a) Most of us are now working on 64bit systems with much more accessible memory, (b) estimating memory use is fairly simple and can usually be avoided. I was just lazy.


hi nrjank , thank you for answer
i found this solution :
(i try to use matrix this large matrix but the PC stops me for lack of
memory...i use pc portable intel i5 with 8 giga Ram )
a1=fix(rand(225).*100);
a=a1(:);
b=1:50624;
b=b';
c=(1:50624)+1;
c=c';
cc=a.*ones(1,numel(b));
%d=cc(b:end)
g=(1:rows(cc))'.*ones(1,numel(b));
g1=g>=b&g<=c;
e=min(~g1.*max(cc).+g1.*cc)

Sent from: https://octave.1599824.n4.nabble.com/OctaveGeneralf1599825.html


On Sun, Aug 25, 2019 at 12:27 AM shivax via Help list for GNU Octave < [hidden email]> wrote: hi nrjank , thank you for answer
i found this solution :
(i try to use matrix this large matrix but the PC stops me for lack of
memory...i use pc portable intel i5 with 8 giga Ram )
a1=fix(rand(225).*100);
a=a1(:);
b=1:50624;
b=b';
c=(1:50624)+1;
c=c';
cc=a.*ones(1,numel(b));
%d=cc(b:end)
g=(1:rows(cc))'.*ones(1,numel(b));
g1=g>=b&g<=c;
e=min(~g1.*max(cc).+g1.*cc)
Since you're only dealing with integers between 0 and 100, if you change the first line to:
a1 = int8(rand(225).*100);
you will significantly decrease the memory required. again, later, g=(uint16(1:rows(cc)))'.*ones(1,numel(b));
you can use the 'whos' command to see the variable memory usage. cc 50625x50624 2562840000 int8 g 50625x50624 5125680000 uint16
this gets to your next line which produces an error:
>> g1=g>=b&g<=c; error: mx_el_ge: nonconformant arguments (op1 is 50625x50624, op2 is 50624x1)
so you should reexamine what you're trying to do there.


sorry .. I made a mistake in writing the code
The correct version is:
(I recreated an exercise with the number of numbers of my work..this is a
test to show you that I can't run it for memory)
B=5000;
a1=fix(rand(B).*100);
a=a1(:);
b=1:(B*B1);
c=((1:(B*B1)))+1;
size(b)
size(c)
cc=uint16(a.*ones(1,numel(b)));
%d=cc(b:end)
g=uint16((1:rows(cc))'.*ones(1,numel(b)));
g1=g>=b&g<=c;
e=min(~g1.*max(cc).+g1.*cc)
error: out of memory or dimension too large for Octave's index type
error: called from
prova2 at line 8 column 4
i used unint16..but the result is the same..:(
if i try this code with B=50 it runny perfectly

Sent from: https://octave.1599824.n4.nabble.com/OctaveGeneralf1599825.html


On Tue, Aug 27, 2019 at 8:24 AM shivax via Help list for GNU Octave < [hidden email]> wrote: sorry .. I made a mistake in writing the code
The correct version is:
(I recreated an exercise with the number of numbers of my work..this is a
test to show you that I can't run it for memory)
B=5000;
a1=fix(rand(B).*100);
a=a1(:);
b=1:(B*B1);
c=((1:(B*B1)))+1;
size(b)
size(c)
cc=uint16(a.*ones(1,numel(b)));
%d=cc(b:end)
g=uint16((1:rows(cc))'.*ones(1,numel(b)));
g1=g>=b&g<=c;
e=min(~g1.*max(cc).+g1.*cc)
error: out of memory or dimension too large for Octave's index type
error: called from
prova2 at line 8 column 4
i used unint16..but the result is the same..:(
if i try this code with B=50 it runny perfectly
ok, since it runs for B=50 and you've avoided looping, it seems you've managed to solve the problem as you first presented it. there are numerous computer algorithms that will work perfectly well for certain data sizes but require completely different approaches for 'big data'.
Perhaps you could share the actual problem you're trying to solve. I feel like we could keep proposing an infinite number of suggestions that would lead you in the wrong direction if we don't understand what it is that the homework is trying to teach you. Is the goal to work on code vectorization, to manage large dataset problems, etc. While we don't want to solve the actual problem for you, we should be able to provide more useful guidance.


I simplified the problem with an example code .. the only difference is that
the vector b and the vector c are not as in the example (1: x) but they are
pointers to the position of the base matrix
Now if I have to bring back my original code I do a little effort .. and
it's very long .. and it's divided by modules so I find it hard to bring
back all the code executed
However I can tell you what it is for or the basic matrix is a financial
curve and what I want to find is the distance between different points of
the curve .. These points are given to me by another function that I built
that takes into account many factors
Excuse for my bad english (i use google translate)

Sent from: https://octave.1599824.n4.nabble.com/OctaveGeneralf1599825.html


On Tue, Aug 27, 2019 at 11:40 AM shivax via Help list for GNU Octave < [hidden email]> wrote: I simplified the problem with an example code .. the only difference is that
the vector b and the vector c are not as in the example (1: x) but they are
pointers to the position of the base matrix
ok, so if we look back at your original example:
you have a matrix, and you have two sets of points that refer to locations in that matrix. the distance between these locations can vary, and you want to perform some operations on the subset of the matrix between those locations.
Assuming you have other functions that the create matrix, c, and vectors of points, a and b, the real questions are 'what is that operation' (in your first example it was the min operation. is that not actually the case? in the last email you said 'calculate the distance'?) and 'are there any other factors/requirements/constraints?' (how large is the data set? how large are c, a, and b (separately) allowed to be? are they all int type? is b always larger than a? is there anything unique about c?)
Two useful things to read. (1) What we're getting into falls into XY problems. Where you are asking how to solve X when you're actual problem is Y, because you hink solving X will solve Y, and that isn't always true. see: http://xyproblem.info(2) please use 'bottom posting' on this help list. it makes it easier for others to follow the conversation. See http://www.idallen.com/topposting.html (if you are using Gmail, you have to click on the 'three dots' to show previous replies when you type your message.)


now i try to expian better
NEW EXAMPLE:
a= Matrix of all equity
10 4 13 20
25 6 10 15
13 10 8 17
8 8 15 22
b and c is a matrix position of a
c=
1 5 9 13
2 6 10 14
3 7 11 15
4 8 12 16
b=
1 5 9 13
2 5 9 13
2 6 9 14
3 7 11 15
c: is a matrix of "a" linear position
1 a(1,1) 5 a(1,2) 9 a(1,3) 13 a(1,4)
2 a(2,1) 6 a(2,2) 10 a(2,3) 14 a(2,4)
3 a(3,1) 7 a(3,2) 11 a(3,3) 15 a(3,4)
4 a(4,1) 8 a(4,2) 12 a(4,3) 16 a(4,4)
b: is a matrix of "a" linear position
1 a(1,1) 5 a(1,2) 9 a(1,3) 13 a(1,4)
2 a(2,1) 5 a(2,2) 9 a(2,3) 13 a(2,4)
2 a(3,1) 6 a(3,2) 9 a(3,3) 14 a(3,4)
3 a(4,1) 7 a(4,2) 11 a(4,3) 15 a(4,4)
I thought of creating the matrix c to analyze the minimum of the equity
between the 2 points
why do I need the minimums between the various points of the matrix a?
because a is the equity matrix..and I want to find the minimum peak between
the various surveys (positions) using matrix b and matrix c
 the elements of b are less than a
the element of b and c are egual/lower of length(a)
(because are position of a)
 a is a matrix so large..
the matrix a is fed from time therefore the dimension is variable.
However at present it occupies 5000 (row) * 88 (columns) size
I don't think it can exceed 10,000 * 500
that's all ... I think I was exhaustive
(
55/5000
I hope I didn't make mistakes in rebuilding the matrices)

Sent from: https://octave.1599824.n4.nabble.com/OctaveGeneralf1599825.html


In reply to this post by Octave  General mailing list
On Tue, Aug 27, 2019 at 2:42 PM shivax via Help list for GNU Octave < [hidden email]> wrote: now i try to expian better
NEW EXAMPLE:
ok. now what do you define as a successful solution? Your initial, very simple but effective script with loops works. it should work for very large arrays. It may not be the fastest on octave. Matlab is probably much faster because it has a working JIT (Just In Time) compiler that knows how to optimize loops.
In the script that I threw together, it removed the loops but only sped things up by about 30% for small cases. I suspect that might increase as the data size increases, but as you have seen, we're trading memory for speed. We can only hold so much data at one time. Your looping case holds very little. Projection operations to parallelize operation are not memory efficient. They multiply, and often duplicate, the data in memory to try to operate on it faster in parallel. For any solution and a particular data set you could tabulate speed and peak memory requirement to compare them and decide which is best.
What are your requirements for 'success' on the homework? You've created two working algorithms. Is vectorization or speed a requirement?

