save/load with pipes

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

save/load with pipes

Daniel Kraft
Hi!

In one of my codes, I'm using fork to split processing of my data set
onto multiple processes.  The subprocesses send their results back via
temporary files and the save/load mechanism for simplicity.  It would be
much cleaner to use pipes instead.  But since the pipe() function
returns a pair of file descriptors, I wonder whether one can use
save/load (or a similar high-level function which serialises the data
for me) also with file descriptors and not just with file names.  Is
there a way to do that?

(My code is basically a custom implementation of parcellfun, so maybe
I'll switch to parcellfun in the future anyway.  But I still believe
this is something interesting to know in case I need a communication
tool like this in the future for something else.)

Yours,
Daniel

--
http://www.domob.eu/
OpenPGP: 901C 5216 0537 1D2A F071  5A0E 4D94 6EED 04F7 CF52
Namecoin: id/domob -> https://nameid.org/?name=domob
--
Done:  Arc-Bar-Cav-Hea-Kni-Ran-Rog-Sam-Tou-Val-Wiz
To go: Mon-Pri
_______________________________________________
Help-octave mailing list
[hidden email]
https://mailman.cae.wisc.edu/listinfo/help-octave
Reply | Threaded
Open this post in threaded view
|

Re: save/load with pipes

Olaf Till-2
On Mon, Mar 03, 2014 at 10:00:15AM +0100, Daniel Kraft wrote:

> Hi!
>
> In one of my codes, I'm using fork to split processing of my data set
> onto multiple processes.  The subprocesses send their results back via
> temporary files and the save/load mechanism for simplicity.  It would be
> much cleaner to use pipes instead.  But since the pipe() function
> returns a pair of file descriptors, I wonder whether one can use
> save/load (or a similar high-level function which serialises the data
> for me) also with file descriptors and not just with file names.  Is
> there a way to do that?
Yes, in the current version of the 'parallel' package there are two
different pairs of functions which do that. Only one variable is
transfered at a time.

fsave/fload have less overhead and are meant for communication within
the local machine. You should probably use these.

__bw_psend__/__bw_prcv__ are not meant for public use. They were
originally intended to be more flexible, and they are usable for
communication between different machines. Currently their interface
might still change.

Olaf

--
public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net

_______________________________________________
Help-octave mailing list
[hidden email]
https://mailman.cae.wisc.edu/listinfo/help-octave

signature.asc (853 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: save/load with pipes

Daniel Kraft
Hi!

On 2014-03-03 11:36, Olaf Till wrote:

> On Mon, Mar 03, 2014 at 10:00:15AM +0100, Daniel Kraft wrote:
>> In one of my codes, I'm using fork to split processing of my data set
>> onto multiple processes.  The subprocesses send their results back via
>> temporary files and the save/load mechanism for simplicity.  It would be
>> much cleaner to use pipes instead.  But since the pipe() function
>> returns a pair of file descriptors, I wonder whether one can use
>> save/load (or a similar high-level function which serialises the data
>> for me) also with file descriptors and not just with file names.  Is
>> there a way to do that?
>
> Yes, in the current version of the 'parallel' package there are two
> different pairs of functions which do that. Only one variable is
> transfered at a time.
>
> fsave/fload have less overhead and are meant for communication within
> the local machine. You should probably use these.

Ah great, that's exactly what I was looking for!  Thanks!

Yours,
Daniel

--
http://www.domob.eu/
OpenPGP: 901C 5216 0537 1D2A F071  5A0E 4D94 6EED 04F7 CF52
Namecoin: id/domob -> https://nameid.org/?name=domob
--
Done:  Arc-Bar-Cav-Hea-Kni-Ran-Rog-Sam-Tou-Val-Wiz
To go: Mon-Pri
_______________________________________________
Help-octave mailing list
[hidden email]
https://mailman.cae.wisc.edu/listinfo/help-octave