I want to use the parallel package to furnish a large 4 dimensional array. The following: clear; function a=single_thread(ky); a(:,:,1)=ky*[1,2;4,5]; a(:,:,2)=[1,2;4,5]; a(:,:,3)=-ky*[1,2;4,5]; endfunction b=pararrayfun(4,@single_thread,[1:4],"ChunksPerProc", 1) works, giving b that has the results from each thread tacked onto the end. Changing the (1:4] to [1:5] in the invocation breaks the script with errors: warning: parcellfun: unhandled error in subprocess 2 warning: parcellfun: unhandled error in subprocess 1 warning: called from parcellfun at line 291 column 9 warning: called from chunk_parcellfun at line 47 column 23 parcellfun at line 291 column 9 parcellfun at line 137 column 26 chunk_parcellfun at line 47 column 23 pararrayfun at line 85 column 26 parcellfun at line 137 column 26 playparallel at line 8 column 2 pararrayfun at line 85 column 26 playparallel at line 8 column 2 I can fix the last version by increasing ChunksPer Proc to 2. I know this means I don't understand the features of the parallel package, but it is way too wonderful to not take advantage of. Can anyone shed some light, particularly on how the output value (b in the case here) is constructed if it is multidimensional? Many thanks _______________________________________________ Help-octave mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/help-octave |
On Sat, Mar 11, 2017 at 04:06:26PM -0800, Clinton Winant wrote:
> I want to use the parallel package to furnish a large 4 dimensional array. > The following: > > clear; > function a=single_thread(ky); > a(:,:,1)=ky*[1,2;4,5]; > a(:,:,2)=[1,2;4,5]; > a(:,:,3)=-ky*[1,2;4,5]; > endfunction > > b=pararrayfun(4,@single_thread,[1:4],"ChunksPerProc", 1) > > works true by default expects the user function to return a single element. With "vectorized" set to true, pararrayfun expects the user function an array of the same dimensions as the user functions input argument(s). You can set "UniformOutput" to false and concatenate the the results in the returned cell array manually. If you want to go in for some hassle to optimize your processing, read on. Currently pararrayfun is written in a way making the following work: octave:1> function a = fmultdim (arg) > a = cat (4, cat (3, [1; 2], [3; 4]), cat (3, arg * [1; 2], arg * [3; 4]), cat (3, [1; 2], [3; 4])); > endfunction octave:2> b = pararrayfun (4, @fmultdim, [1:5]) parcellfun: 5/5 jobs done b = ans(:,:,1,1) = 1 1 1 1 1 2 2 2 2 2 ... But if you give "ChunksPerProc" with "vectorized" false, cellfun is called internally, and doesn't accept this input. If you set "vectorized" to true, it should work if you change the user function to accept row vectors and return the corresponding number of slices along dimension 2. Olaf -- public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net _______________________________________________ Help-octave mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/help-octave signature.asc (836 bytes) Download Attachment |
Many thanks for your helpful reply. I understand that with "Vectorized", the output can be a 2d matrix, but I am not quite there yet. I modified your script as: clear all; nproc=4; function N = single_thread (arg) Array2d=arg*[1,i;-i,1]; N=Array2d(:); endfunction b = pararrayfun (nproc, @single_thread, [1:4],'Vectorized',true,'ChunksPerProc',1) b = pararrayfun (nproc, @single_thread, [1:5],'Vectorized',true,'ChunksPerProc',2) b = pararrayfun (nproc, @single_thread, [1:5],'Vectorized',true,'ChunksPerProc',1) the output: octave:1> test_pararray parcellfun: 4/4 jobs done b = 1 + 0i 2 + 0i 3 + 0i 4 + 0i -0 - 1i -0 - 2i -0 - 3i -0 - 4i 0 + 1i 0 + 2i 0 + 3i 0 + 4i 1 + 0i 2 + 0i 3 + 0i 4 + 0i parcellfun: 5/5 jobs done b = 1 + 0i 2 + 0i 3 + 0i 4 + 0i 5 + 0i -0 - 1i -0 - 2i -0 - 3i -0 - 4i -0 - 5i 0 + 1i 0 + 2i 0 + 3i 0 + 4i 0 + 5i 1 + 0i 2 + 0i 3 + 0i 4 + 0i 5 + 0i parcellfun: 2/4 jobs done warning: parcellfun: unhandled error in subprocess 1 warning: called from parcellfun at line 291 column 9 chunk_parcellfun at line 47 column 23 pararrayfun at line 73 column 26 test_pararray at line 13 column 3 parcellfun: 3/4 jobs done error: reshape: can't reshape 12x1 array to 1x5 array error: called from chunk_parcellfun>@<anonymous> at line 55 column 33 chunk_parcellfun at line 55 column 15 pararrayfun at line 73 column 26 test_pararray at line 13 column 3 The first two calls work fine, but the third does not, leading me to conclude I dont undestand what ChunksPerProc means. For instance if the input vector had 256 values, and nproc=16, the should ChunksPerProc need to be 16? Clearly not, if it was that simple, the program would it compute it by itself. Sorry to be so dense. The possibilty of breaking large calculations into "chunks" and using multiple processors is such a giant step forward, Perhaps I could contribute in some way? On Sun, Mar 12, 2017 at 10:21 AM, Olaf Till <[hidden email]> wrote: On Sat, Mar 11, 2017 at 04:06:26PM -0800, Clinton Winant wrote: _______________________________________________ Help-octave mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/help-octave |
On Sun, Mar 12, 2017 at 03:26:43PM -0700, Clinton Winant wrote:
> Many thanks for your helpful reply. I understand that with "Vectorized", > the output can be a 2d matrix, but I am not quite there yet. I modified > your script as: > > clear all; > nproc=4; > > function N = single_thread (arg) > Array2d=arg*[1,i;-i,1]; > N=Array2d(:); > endfunction "Vectorized". But if the function returns more elements than it gets as argument, without "UniformOutput" set to false, we rely on unspecified behavior, and it works only in special cases. I think we should stick with the specified behavior. Your function should return as many elements as given in "arg". If it can handle only a scalar "arg", don't specify "Vectorized". If it returns a matrix for a scalar "arg", as it does currently, don't specify "Vectorized", but specify ..., "UniformOutput", false, ... and concatenate the result manually e.g. with b = pararrayfun (nproc, @single_thread, [1:5], "UniformOutput", false); result = cat (1, b{:}); Olaf -- public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net _______________________________________________ Help-octave mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/help-octave signature.asc (836 bytes) Download Attachment |
Many thanks again. I completely agree to stick with the specified behavior. I have modified my script as follows: clear all; nproc=4; function N = single_thread (arg) Array2d=arg*[1,i;-i,1]; N=Array2d(:); endfunction b = pararrayfun (nproc, @single_thread, [1:5], "UniformOutput", false); which works (you already knew that), size(b) returns 1 5, even though for instance b[1,1] has four elements. If b was an array, it would be 4 by 5, so it must be a cell. It must be clear to you by now I dont understand cells. I will learn about cells and use what you have given me for my real problem. I notice that on http://wiki.octave.org/Parallel_package it doesnt say much about the three options (Chunks Per Proc, UniformOutput and Vectorized), although the examples give some info? Would you like me to expand the documentation (the answers would have to come from you), or is there any other way I could contribute? This seems such an important (and unique) feature of octave. On Sun, Mar 12, 2017 at 10:54 PM, Olaf Till <[hidden email]> wrote: On Sun, Mar 12, 2017 at 03:26:43PM -0700, Clinton Winant wrote: _______________________________________________ Help-octave mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/help-octave |
On Mon, Mar 13, 2017 at 09:42:08AM -0700, Clinton Winant wrote:
> I notice that on http://wiki.octave.org/Parallel_package it doesnt say > much about the three options (Chunks Per Proc, UniformOutput and > Vectorized), although the examples give some info? Would you like me to > expand the documentation (the answers would have to come from you), or is > there any other way I could contribute? This seems such an important (and > unique) feature of octave. It's not under my control what is on the wiki for parallel. The decision what to put there is yours. Olaf -- public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net _______________________________________________ Help-octave mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/help-octave signature.asc (836 bytes) Download Attachment |
Free forum by Nabble | Edit this page |