Passing variables between octave processes on the same machine

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Passing variables between octave processes on the same machine

phofman
Hi,

Please does anyone have any experience with passing complete variables
between two independent octave processes (not parent/child via popen)?

The transport channel is easy - sockets, zeromq etc. I assume for
serialization I could use the 'save' function:

bitstream = save('-', 'variable')


But I was not able to find a simple way to deserialize contents of the
'bitstream' variable back to 'variable'. The 'load' function does not
accept input from a variable (as far as I understand the documentation),
only from a real file.

Actually, I would love to use a sequence of plain files but several
messages must be sent every second and SSD drives do not like this
constant create/delete file load. Creating a RAM drive would work but I
need a multiplatform solution (linux, windows) and creating the
platform-specific RAM drive would overcomplicate the project deployment.

Thank you in advance for any suggestions, experience, etc.

Best regards,

Pavel.


Reply | Threaded
Open this post in threaded view
|

Re: Passing variables between octave processes on the same machine

Sergei Steshenko


On Sunday, January 20, 2019, 7:36:25 PM GMT+2, Pavel Hofman <[hidden email]> wrote:


Hi,

Please does anyone have any experience with passing complete variables
between two independent octave processes (not parent/child via popen)?

The transport channel is easy - sockets, zeromq etc. I assume for
serialization I could use the 'save' function:

bitstream = save('-', 'variable')


But I was not able to find a simple way to deserialize contents of the
'bitstream' variable back to 'variable'. The 'load' function does not
accept input from a variable (as far as I understand the documentation),
only from a real file.

Actually, I would love to use a sequence of plain files but several
messages must be sent every second and SSD drives do not like this
constant create/delete file load. Creating a RAM drive would work but I
need a multiplatform solution (linux, windows) and creating the
platform-specific RAM drive would overcomplicate the project deployment.

Thank you in advance for any suggestions, experience, etc.

Best regards,

Pavel.

================================================

Start from named pipes "Introduction to Named Pipes" - Introduction to Named Pipes | Linux Journal


.

Using named pipes you can save Octave variables using 'save' in one process and load Octave variables in another process using 'load'. And/or maybe 'fwrite' and 'fread' preceded by 'fopen'.

Named pipes also exist in Windows.

--Sergei.


Reply | Threaded
Open this post in threaded view
|

Re: Passing variables between octave processes on the same machine

phofman
Dne 20. 01. 19 v 18:42 Sergei Steshenko napsal(a):
>
>
> Using named pipes you can save Octave variables using 'save' in one
> process and load Octave variables in another process using 'load'.
> And/or maybe 'fwrite' and 'fread' preceded by 'fopen'.
>
> Named pipes also exist in Windows.
>

Hi Sergei,

Thanks a lot for your suggestion.

Do you have a practical experience with using named pipes for this? I
would love to avoid the blocking that pipes bring. If the reader fails,
the writer will get stuck once the pipe fills up. If the writer writes
two variables into the pipe, can the reader recover both of them
sequentially? That would be quite easy to do with octave-zeromq which
delivers each message payload individually, but there is no file to
'load' from...

Or maybe I am missing something important.

Anyway I very much appreciate your help.

Best regards,

Pavel.




Reply | Threaded
Open this post in threaded view
|

Re: Passing variables between octave processes on the same machine

Andreas Weber-6
In reply to this post by phofman
Am 20.01.19 um 18:35 schrieb Pavel Hofman:
> Please does anyone have any experience with passing complete variables
> between two independent octave processes (not parent/child via popen)?

I had the same problem and wrote:
https://github.com/octave-de/macgyver_utils/blob/master/serialize.m

But if you have the same octave and architecture on both sides you can
just use "typecast" on simple datatypes.

For human readable exchange of complex structures you may use JSON as
interexchange format:
https://github.com/Andy1978/octave-rapidjson

-- Andy


Reply | Threaded
Open this post in threaded view
|

Re: Passing variables between octave processes on the same machine

Olaf Till-2
In reply to this post by Sergei Steshenko
On Sun, Jan 20, 2019 at 05:42:40PM +0000, Sergei Steshenko wrote:

>  
>
>     On Sunday, January 20, 2019, 7:36:25 PM GMT+2, Pavel Hofman <[hidden email]> wrote:  
>  
>  Hi,
>
> Please does anyone have any experience with passing complete variables
> between two independent octave processes (not parent/child via popen)?
>
> The transport channel is easy - sockets, zeromq etc. I assume for
> serialization I could use the 'save' function:
>
> bitstream = save('-', 'variable')
>
>
> But I was not able to find a simple way to deserialize contents of the
> 'bitstream' variable back to 'variable'. The 'load' function does not
> accept input from a variable (as far as I understand the documentation),
> only from a real file.
>
> Actually, I would love to use a sequence of plain files but several
> messages must be sent every second and SSD drives do not like this
> constant create/delete file load. Creating a RAM drive would work but I
> need a multiplatform solution (linux, windows) and creating the
> platform-specific RAM drive would overcomplicate the project deployment.
>
> Thank you in advance for any suggestions, experience, etc.
>
> Best regards,
>
> Pavel.
>
> ================================================
> Start from named pipes "Introduction to Named Pipes" - Introduction to Named Pipes | Linux Journal
>
>
> |
> |
> |
> |  |  |
>
>  |
>
>  |
> |
> |  |
> Introduction to Named Pipes | Linux Journal
>
>
>  |
>
>  |
>
>  |
>
>
> .
> Using named pipes you can save Octave variables using 'save' in one process and load Octave variables in another process using 'load'. And/or maybe 'fwrite' and 'fread' preceded by 'fopen'.
> Named pipes also exist in Windows.
> --Sergei.
For processes within the same non-windows-machine, once you have a
channel with an Octave file number, an efficient way is with 'fsave'
and 'fload' from the 'parallel' package.

For data exchange between different machines, you could look at
'var2bytea' and 'bytea2var' from the 'database' package. 'var2bytea'
returns a binary string for each input variable, similarly to save
('-', ...). This string has to be sent over and given as input to
'bytea2var' at the other side.

Olaf

--
public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net



signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Passing variables between octave processes on the same machine

Sergei Steshenko
In reply to this post by phofman


On Sunday, January 20, 2019, 7:51:25 PM GMT+2, Pavel Hofman <[hidden email]> wrote:


Dne 20. 01. 19 v 18:42 Sergei Steshenko napsal(a):
>
>
> Using named pipes you can save Octave variables using 'save' in one
> process and load Octave variables in another process using 'load'.
> And/or maybe 'fwrite' and 'fread' preceded by 'fopen'.
>
> Named pipes also exist in Windows.
>

Hi Sergei,

Thanks a lot for your suggestion.

Do you have a practical experience with using named pipes for this? I
would love to avoid the blocking that pipes bring. If the reader fails,
the writer will get stuck once the pipe fills up. If the writer writes
two variables into the pipe, can the reader recover both of them
sequentially? That would be quite easy to do with octave-zeromq which
delivers each message payload individually, but there is no file to
'load' from...

Or maybe I am missing something important.

Anyway I very much appreciate your help.


Best regards,

Pavel.

===========================================

I do have practical experience with named pipe - they work well for me. I didn't use them with Octave, I used them e.g.with 'ecasound'. But the mechanism is generic.

You can have named pipe per variable. You can also send a number of variables simultaneously through named pipe using Octave struct.

The fundamental rule is: first start reading the pipe in the receiving process, then send data to it in the sending process.

I also used (zero size) files as semaphores. I.e. the sending process creates such a file when data is available, and the receiving process deletes the file upon receiving the data thus notifying the sending process that new data can be prepared. But this is not necessarily related to named pipes - it's just another mechanism.

--Sergei.


Reply | Threaded
Open this post in threaded view
|

Re: Passing variables between octave processes on the same machine

phofman
In reply to this post by Andreas Weber-6
Hi Andreas,

Dne 20. 01. 19 v 19:25 Andreas Weber napsal(a):
> Am 20.01.19 um 18:35 schrieb Pavel Hofman:
>> Please does anyone have any experience with passing complete variables
>> between two independent octave processes (not parent/child via popen)?
>
> I had the same problem and wrote:
> https://github.com/octave-de/macgyver_utils/blob/master/serialize.m

Great work, thank you. I will try it out. What is the license of your code?
>
> But if you have the same octave and architecture on both sides you can
> just use "typecast" on simple datatypes.
>
> For human readable exchange of complex structures you may use JSON as
> interexchange format:
> https://github.com/Andy1978/octave-rapidjson

I would prefer an efficient binary way but very good to know, thanks.

With regards,

Pavel.




Reply | Threaded
Open this post in threaded view
|

Re: Passing variables between octave processes on the same machine

phofman
In reply to this post by Olaf Till-2
Hi Olaf,

Thanks again.

Dne 20. 01. 19 v 19:30 Olaf Till napsal(a):
> For processes within the same non-windows-machine, once you have a
> channel with an Octave file number, an efficient way is with 'fsave'
> and 'fload' from the 'parallel' package.

This looks very interesting. I assume I would have to start all
processes in some special way to get the file number. Is there any way
that would work for two unrelated octave processes running on the same
machine under the same user? You mention a non-windows machine - working
only for linux? Unfortunately that would be a problem for me. While I
develop everything in linux, my project will have to support windows too.

> For data exchange between different machines, you could look at
> 'var2bytea' and 'bytea2var' from the 'database' package. 'var2bytea'
> returns a binary string for each input variable, similarly to save
> ('-', ...). This string has to be sent over and given as input to
> 'bytea2var' at the other side.

Great option, the database package is easy to install on any platform. I
will investigate if the fsave/fload option is not available. Please is
there any safe byte/byte combination to use as a separator between each
serialized block?

Thanks a lot again,

Pavel.


Reply | Threaded
Open this post in threaded view
|

Re: Passing variables between octave processes on the same machine

phofman
In reply to this post by Sergei Steshenko
Hi Sergei,

>
> I do have practical experience with named pipe - they work well for me.
> I didn't use them with Octave, I used them e.g.with 'ecasound'. But the
> mechanism is generic.

I have used named pipes many times in the past. But in this case the
blocking behaviour really is an obstacle. The writer must keep running
when the reader is down, the communication is not crucial for the
writer. The reader is a low-priority monitoring process with GUI.

> I also used (zero size) files as semaphores. I.e. the sending process
> creates such a file when data is available, and the receiving process
> deletes the file upon receiving the data thus notifying the sending
> process that new data can be prepared. But this is not necessarily
> related to named pipes - it's just another mechanism.

Creating small new files for each message is unfortunately not an option
on ssd drives.

Anyway, thansk a lot for your help!

Pavel.


Reply | Threaded
Open this post in threaded view
|

Re: Passing variables between octave processes on the same machine

Sergei Steshenko


On Sunday, January 20, 2019, 11:55:56 PM GMT+2, Pavel Hofman <[hidden email]> wrote:


Hi Sergei,

>
> I do have practical experience with named pipe - they work well for me.
> I didn't use them with Octave, I used them e.g.with 'ecasound'. But the
> mechanism is generic.

I have used named pipes many times in the past. But in this case the
blocking behaviour really is an obstacle. The writer must keep running
when the reader is down, the communication is not crucial for the
writer. The reader is a low-priority monitoring process with GUI.

> I also used (zero size) files as semaphores. I.e. the sending process
> creates such a file when data is available, and the receiving process
> deletes the file upon receiving the data thus notifying the sending
> process that new data can be prepared. But this is not necessarily
> related to named pipes - it's just another mechanism.

Creating small new files for each message is unfortunately not an option
on ssd drives.

Anyway, thansk a lot for your help!


Pavel.

================================

Implement write to named pipe through 'eval', trying to reopen named pipes for writing after writing to the fails.

I.e. not an issue.

--Sergei.


Reply | Threaded
Open this post in threaded view
|

Re: Passing variables between octave processes on the same machine - SOLVED

phofman
In reply to this post by phofman
Hi,
Dne 20. 01. 19 v 21:10 Pavel Hofman napsal(a):
> Hi Olaf,
>
>
>> For data exchange between different machines, you could look at
>> 'var2bytea' and 'bytea2var' from the 'database' package. 'var2bytea'
>> returns a binary string for each input variable, similarly to save
>> ('-', ...). This string has to be sent over and given as input to
>> 'bytea2var' at the other side.

The combination of var2bytea/bytea2var and zeromq in ZMQ_PAIR mode seems
to work great. No need to use any separator, every zmq_send results in
reception of one message with the serialized variable. Passing a struct
with details is very comfortable this way. The ZMQ_DONTWAIT flag
controls the blocking behavior properly - very convenient.

Both packages are pre-compiled in the Win64 build too, no extra
deployment steps required for that OS.

Guys, thanks a lot for your help!

Best regards,

Pavel.





Reply | Threaded
Open this post in threaded view
|

Re: Passing variables between octave processes on the same machine

Sergei Steshenko
In reply to this post by phofman


On Sunday, January 20, 2019, 11:55:28 PM GMT+2, Pavel Hofman <[hidden email]> wrote:


Hi Sergei,

>
> I do have practical experience with named pipe - they work well for me.
> I didn't use them with Octave, I used them e.g.with 'ecasound'. But the
> mechanism is generic.

I have used named pipes many times in the past. But in this case the
blocking behaviour really is an obstacle. The writer must keep running
when the reader is down, the communication is not crucial for the
writer. The reader is a low-priority monitoring process with GUI.

> I also used (zero size) files as semaphores. I.e. the sending process
> creates such a file when data is available, and the receiving process
> deletes the file upon receiving the data thus notifying the sending
> process that new data can be prepared. But this is not necessarily
> related to named pipes - it's just another mechanism.

Creating small new files for each message is unfortunately not an option
on ssd drives.

Anyway, thansk a lot for your help!


Pavel.

=========================================

I know that you've made a decision, but regarding "Creating small new files for each message is unfortunately not an option
on ssd drives" - it's not an issue. First of all, modern SSD drives have pretty long lifespan, and even swap area is located on them.

But, anyway, thanks to OS buffering if a file exists for several seconds, it's not even physically written to hard drive, it's operated on in OS RAM file buffer.

IIRC UNIX default time after files are flushed to disk are 25 seconds.

--Sergei.


Reply | Threaded
Open this post in threaded view
|

Re: Passing variables between octave processes on the same machine

phofman
Hi,

Dne 22. 01. 19 v 3:56 Sergei Steshenko napsal(a):
>
> I know that you've made a decision, but regarding"Creating small new
> files for each message is unfortunately not an option
> on ssd drives" - it's not an issue. First of all, modern SSD drives have
> pretty long lifespan, and even swap area is located on them.

Continuous creating/deleting 10 files a second for the purpose of
communication is simply bad enginnering. Modern SSD drives may be more
resilient, but people use old ones, some of my systems run on CF cards.
Definitely not designed for this type of (useless) load.


>
> IIRC UNIX default time after files are flushed to disk are 25 seconds.

For ext4 it is 5 secs. You would not want to loose 25 secs of data after
a power loss. I could not find the figure for NTFS.

But anyway, good to discuss.

Best regards,

Pavel.


Reply | Threaded
Open this post in threaded view
|

Re: Passing variables between octave processes on the same machine

Francesco Potortì
>> I know that you've made a decision, but regarding"Creating small new
>> files for each message is unfortunately not an option
>> on ssd drives" - it's not an issue. First of all, modern SSD drives have
>> pretty long lifespan, and even swap area is located on them.
>
>Continuous creating/deleting 10 files a second for the purpose of
>communication is simply bad enginnering. Modern SSD drives may be more
>resilient, but people use old ones, some of my systems run on CF cards.
>Definitely not designed for this type of (useless) load.
>>
>> IIRC UNIX default time after files are flushed to disk are 25 seconds.
>
>For ext4 it is 5 secs. You would not want to loose 25 secs of data after
>a power loss. I could not find the figure for NTFS.

As far as I know, on Unix short-lived files (and pipe data) are never
physically written to disk, they just live in the fs buffers in memory.
I imagine this is still true in all current operating systems that are
capable enough to run Octave.

Sure, you get all the fs overhead, so I agree that in principle it is
bad engineering, I do not think that the mass memory would suffer from
that.

--
Francesco Potortì (ricercatore)        Voice:  +39.050.621.3058
ISTI - Area della ricerca CNR          Mobile: +39.348.8283.107
via G. Moruzzi 1, I-56124 Pisa         Skype:  wnlabisti
(entrance 20, 1st floor, room C71)     Web:    http://fly.isti.cnr.it



Reply | Threaded
Open this post in threaded view
|

Re: Passing variables between octave processes on the same machine

Sergei Steshenko


On Tuesday, January 22, 2019, 11:09:19 PM GMT+2, Francesco Potortì <[hidden email]> wrote:


>> I know that you've made a decision, but regarding"Creating small new

>> files for each message is unfortunately not an option
>> on ssd drives" - it's not an issue. First of all, modern SSD drives have
>> pretty long lifespan, and even swap area is located on them.
>
>Continuous creating/deleting 10 files a second for the purpose of
>communication is simply bad enginnering. Modern SSD drives may be more
>resilient, but people use old ones, some of my systems run on CF cards.
>Definitely not designed for this type of (useless) load.
>>
>> IIRC UNIX default time after files are flushed to disk are 25 seconds.
>
>For ext4 it is 5 secs. You would not want to loose 25 secs of data after
>a power loss. I could not find the figure for NTFS.

As far as I know, on Unix short-lived files (and pipe data) are never
physically written to disk, they just live in the fs buffers in memory.
I imagine this is still true in all current operating systems that are
capable enough to run Octave.

Sure, you get all the fs overhead, so I agree that in principle it is
bad engineering, I do not think that the mass memory would suffer from

that.


--
Francesco Potortì (ricercatore)        Voice:  +39.050.621.3058
ISTI - Area della ricerca CNR          Mobile: +39.348.8283.107
via G. Moruzzi 1, I-56124 Pisa        Skype:  wnlabisti
(entrance 20, 1st floor, room C71)    Web:    http://fly.isti.cnr.it


==============================================


Regarding bad engineering - it all depends. I.e. the files as semaphores are trivially easy to debug using just shell commands.

We may discuss speed. I.e. creating a file is context switch. I.e. current task is suspended and the OS is called to perform the file operation. So this won't be fast for many short variables to be transferred from process to process.

I was using the file approach for, say, 10 transfers per second. And I needed to transfer blocks of 8Kbytes or so.

--Sergei.