Pararrayfun

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Pararrayfun

Clinton Winant-2
I want to use the parallel package to furnish a large 4 dimensional array.  The following:

clear;
function a=single_thread(ky);
  a(:,:,1)=ky*[1,2;4,5];
  a(:,:,2)=[1,2;4,5];
  a(:,:,3)=-ky*[1,2;4,5];
endfunction

b=pararrayfun(4,@single_thread,[1:4],"ChunksPerProc", 1)

works, giving b that has the results from each thread tacked onto the end. 

Changing the (1:4] to [1:5] in the invocation breaks the script with errors:
warning: parcellfun: unhandled error in subprocess 2
warning: parcellfun: unhandled error in subprocess 1
warning: called from
    parcellfun at line 291 column 9
warning: called from
    chunk_parcellfun at line 47 column 23
    parcellfun at line 291 column 9
    parcellfun at line 137 column 26
    chunk_parcellfun at line 47 column 23
    pararrayfun at line 85 column 26
    parcellfun at line 137 column 26
    playparallel at line 8 column 2
    pararrayfun at line 85 column 26
    playparallel at line 8 column 2

I can fix the last version by increasing ChunksPer Proc to 2. 
I know this means I don't understand the features of the parallel package, but it is way too wonderful to not take advantage of.  Can anyone shed some light, particularly on how the output value (b in the case here) is constructed if it is multidimensional?

Many thanks

_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave
Reply | Threaded
Open this post in threaded view
|

Re: Pararrayfun

Olaf Till-2
On Sat, Mar 11, 2017 at 04:06:26PM -0800, Clinton Winant wrote:

> I want to use the parallel package to furnish a large 4 dimensional array.
> The following:
>
> clear;
> function a=single_thread(ky);
>   a(:,:,1)=ky*[1,2;4,5];
>   a(:,:,2)=[1,2;4,5];
>   a(:,:,3)=-ky*[1,2;4,5];
> endfunction
>
> b=pararrayfun(4,@single_thread,[1:4],"ChunksPerProc", 1)
>
> works
by some chance. As arrayfun, pararrayfun with "vectorized" not set to
true by default expects the user function to return a single
element. With "vectorized" set to true, pararrayfun expects the user
function an array of the same dimensions as the user functions input
argument(s).

You can set "UniformOutput" to false and concatenate the the results
in the returned cell array manually.

If you want to go in for some hassle to optimize your processing, read on.

Currently pararrayfun is written in a way making the following work:

octave:1> function a = fmultdim (arg)
> a = cat (4, cat (3, [1; 2], [3; 4]), cat (3, arg * [1; 2], arg * [3; 4]), cat (3, [1; 2], [3; 4]));
> endfunction

octave:2> b = pararrayfun (4, @fmultdim, [1:5])
parcellfun: 5/5 jobs done
b =

ans(:,:,1,1) =

   1   1   1   1   1
   2   2   2   2   2

...

But if you give "ChunksPerProc" with "vectorized" false, cellfun is
called internally, and doesn't accept this input. If you set
"vectorized" to true, it should work if you change the user function
to accept row vectors and return the corresponding number of slices
along dimension 2.

Olaf

--
public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net

_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Pararrayfun

Clinton Winant-2
Many thanks for your helpful reply.  I understand that with "Vectorized", the output can be a 2d matrix, but I am not quite there yet.  I modified your script as:

clear all;
nproc=4;

function N = single_thread (arg)
  Array2d=arg*[1,i;-i,1];
  N=Array2d(:);
endfunction

b = pararrayfun (nproc, @single_thread, [1:4],'Vectorized',true,'ChunksPerProc',1)

b = pararrayfun (nproc, @single_thread, [1:5],'Vectorized',true,'ChunksPerProc',2)

b = pararrayfun (nproc, @single_thread, [1:5],'Vectorized',true,'ChunksPerProc',1)

the output:

octave:1> test_pararray
parcellfun: 4/4 jobs done
b =

   1 + 0i   2 + 0i   3 + 0i   4 + 0i
  -0 - 1i  -0 - 2i  -0 - 3i  -0 - 4i
   0 + 1i   0 + 2i   0 + 3i   0 + 4i
   1 + 0i   2 + 0i   3 + 0i   4 + 0i

parcellfun: 5/5 jobs done
b =

   1 + 0i   2 + 0i   3 + 0i   4 + 0i   5 + 0i
  -0 - 1i  -0 - 2i  -0 - 3i  -0 - 4i  -0 - 5i
   0 + 1i   0 + 2i   0 + 3i   0 + 4i   0 + 5i
   1 + 0i   2 + 0i   3 + 0i   4 + 0i   5 + 0i

parcellfun: 2/4 jobs done
warning: parcellfun: unhandled error in subprocess 1
warning: called from
    parcellfun at line 291 column 9
    chunk_parcellfun at line 47 column 23
    pararrayfun at line 73 column 26
    test_pararray at line 13 column 3
parcellfun: 3/4 jobs done
error: reshape: can't reshape 12x1 array to 1x5 array
error: called from
    chunk_parcellfun>@<anonymous> at line 55 column 33
    chunk_parcellfun at line 55 column 15
    pararrayfun at line 73 column 26
    test_pararray at line 13 column 3

The first two calls work fine, but the third does not, leading me to conclude I dont undestand what ChunksPerProc means.  For instance if the input vector had 256 values, and nproc=16, the should ChunksPerProc need to be 16?  Clearly not, if it was that simple, the program would it compute it by itself.

Sorry to be so dense.  The possibilty of breaking large calculations into "chunks" and using multiple processors is such a giant step forward, Perhaps I could contribute in some way? 


On Sun, Mar 12, 2017 at 10:21 AM, Olaf Till <[hidden email]> wrote:
On Sat, Mar 11, 2017 at 04:06:26PM -0800, Clinton Winant wrote:
> I want to use the parallel package to furnish a large 4 dimensional array.
> The following:
>
> clear;
> function a=single_thread(ky);
>   a(:,:,1)=ky*[1,2;4,5];
>   a(:,:,2)=[1,2;4,5];
>   a(:,:,3)=-ky*[1,2;4,5];
> endfunction
>
> b=pararrayfun(4,@single_thread,[1:4],"ChunksPerProc", 1)
>
> works

by some chance. As arrayfun, pararrayfun with "vectorized" not set to
true by default expects the user function to return a single
element. With "vectorized" set to true, pararrayfun expects the user
function an array of the same dimensions as the user functions input
argument(s).

You can set "UniformOutput" to false and concatenate the the results
in the returned cell array manually.

If you want to go in for some hassle to optimize your processing, read on.

Currently pararrayfun is written in a way making the following work:

octave:1> function a = fmultdim (arg)
> a = cat (4, cat (3, [1; 2], [3; 4]), cat (3, arg * [1; 2], arg * [3; 4]), cat (3, [1; 2], [3; 4]));
> endfunction

octave:2> b = pararrayfun (4, @fmultdim, [1:5])
parcellfun: 5/5 jobs done
b =

ans(:,:,1,1) =

   1   1   1   1   1
   2   2   2   2   2

...

But if you give "ChunksPerProc" with "vectorized" false, cellfun is
called internally, and doesn't accept this input. If you set
"vectorized" to true, it should work if you change the user function
to accept row vectors and return the corresponding number of slices
along dimension 2.

Olaf

--
public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net


_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave
Reply | Threaded
Open this post in threaded view
|

Re: Pararrayfun

Olaf Till-2
On Sun, Mar 12, 2017 at 03:26:43PM -0700, Clinton Winant wrote:

> Many thanks for your helpful reply.  I understand that with "Vectorized",
> the output can be a 2d matrix, but I am not quite there yet.  I modified
> your script as:
>
> clear all;
> nproc=4;
>
> function N = single_thread (arg)
>   Array2d=arg*[1,i;-i,1];
>   N=Array2d(:);
> endfunction
This function doesn't handle vectors as accepted, so you can't specify
"Vectorized". But if the function returns more elements than it gets
as argument, without "UniformOutput" set to false, we rely on
unspecified behavior, and it works only in special cases.

I think we should stick with the specified behavior. Your function
should return as many elements as given in "arg". If it can handle
only a scalar "arg", don't specify "Vectorized". If it returns a
matrix for a scalar "arg", as it does currently, don't specify
"Vectorized", but specify ..., "UniformOutput", false, ... and
concatenate the result manually e.g. with

b = pararrayfun (nproc, @single_thread, [1:5], "UniformOutput", false);

result = cat (1, b{:});

Olaf

--
public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net

_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Pararrayfun

Clinton Winant-2
Many thanks again. I completely agree to stick with the specified behavior. I have modified my script as follows:

clear all;
nproc=4;
function N = single_thread (arg)
  Array2d=arg*[1,i;-i,1];
  N=Array2d(:);
endfunction
b = pararrayfun (nproc, @single_thread, [1:5], "UniformOutput", false);

which works (you already knew that), size(b) returns 1 5, even though  for instance b[1,1] has four elements.  If b was an array, it would be 4 by 5, so it must be  a cell. It must be clear to you by now I dont understand cells.

I will learn about cells and use what you have given me for my real problem.

I notice that on http://wiki.octave.org/Parallel_package  it doesnt say much about the three options (Chunks Per Proc, UniformOutput and Vectorized), although the examples give some info?  Would you like me to expand the documentation (the answers would have to come from you), or is there any other way I could contribute?  This seems such an important (and unique) feature of octave.








On Sun, Mar 12, 2017 at 10:54 PM, Olaf Till <[hidden email]> wrote:
On Sun, Mar 12, 2017 at 03:26:43PM -0700, Clinton Winant wrote:
> Many thanks for your helpful reply.  I understand that with "Vectorized",
> the output can be a 2d matrix, but I am not quite there yet.  I modified
> your script as:
>
> clear all;
> nproc=4;
>
> function N = single_thread (arg)
>   Array2d=arg*[1,i;-i,1];
>   N=Array2d(:);
> endfunction

This function doesn't handle vectors as accepted, so you can't specify
"Vectorized". But if the function returns more elements than it gets
as argument, without "UniformOutput" set to false, we rely on
unspecified behavior, and it works only in special cases.

I think we should stick with the specified behavior. Your function
should return as many elements as given in "arg". If it can handle
only a scalar "arg", don't specify "Vectorized". If it returns a
matrix for a scalar "arg", as it does currently, don't specify
"Vectorized", but specify ..., "UniformOutput", false, ... and
concatenate the result manually e.g. with

b = pararrayfun (nproc, @single_thread, [1:5], "UniformOutput", false);

result = cat (1, b{:});

Olaf

--
public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net


_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave
Reply | Threaded
Open this post in threaded view
|

Re: Pararrayfun

Olaf Till-2
On Mon, Mar 13, 2017 at 09:42:08AM -0700, Clinton Winant wrote:
> I notice that on http://wiki.octave.org/Parallel_package  it doesnt say
> much about the three options (Chunks Per Proc, UniformOutput and
> Vectorized), although the examples give some info?  Would you like me to
> expand the documentation (the answers would have to come from you), or is
> there any other way I could contribute?  This seems such an important (and
> unique) feature of octave.

It's not under my control what is on the wiki for parallel. The
decision what to put there is yours.

Olaf

--
public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net

_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave

signature.asc (836 bytes) Download Attachment