Re: Help-octave Digest, Vol 157, Issue 40

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Help-octave Digest, Vol 157, Issue 40

John Donoghue-3
On 4/22/19 9:41 AM, [hidden email] wrote:

> Message: 2
> Date: Sun, 21 Apr 2019 23:54:09 +0000
> From: "Ardid, Salva"<[hidden email]>
> To:"[hidden email]"  <[hidden email]>
> Subject: Workaround missing functions in the zeromq package
> Message-ID: <1655660.5vOfAfxstU@eksa>
> Content-Type: text/plain; charset="utf-8"
>
> Hi,
>
> I started working to support Octave in Transplant using the zeromq package in Octave Forge. It?s the first time I use anything related to ZMQ and I reached a point where I?m stacked because Transplant uses zmq_msg_init, zmq_msg_recv, zmq_msg_data and zmq_msg_close, which are not yet supported in Octave?s package.
>
> So, I wonder, would there be some way to implement the same functionality just using zmq_recv?
>
> For the case it is useful, this is the piece of code I would need to adapt:
>
> msg = libstruct('zmq_msg_t', struct('hidden', zeros(1, 64, 'uint8')));
> calllib('libzmq', 'zmq_msg_init', msg); % always returns 0
> msglen = calllib('libzmq', 'zmq_msg_recv', msg, obj.socket, 0);
> assert(msglen >= 0, obj.errortext('zmq_msg_recv'));
> msgptr = calllib('libzmq', 'zmq_msg_data', msg);
> if not(msgptr.isNull)
>      setdatatype(msgptr, 'uint8Ptr', 1, msglen);
>      str = uint8(msgptr.Value);
> else
>      str = uint8([]);
> end
> err = calllib('libzmq', 'zmq_msg_close', msg);
> assert(err == 0, obj.errortext('zmq_msg_close'));
>
>
> [Transplant also uses zmq contexts, which are not supported in the Octave package either. Although I?m not totally sure, I think this shouldn?t be strictly needed to make it work]
>
> Any help on this is very much appreciated!
>
> Thanks in advance and best, Salva

the octave zmq package uses a zmq context within the package, but
doesn't expose it to the user. The same goes for the zmq_msg_xxxx
functions - they are used internally, but the user doesnt have direct
access to them.

So the user (after creating the zmq socket) can just call,   msg =
zmq_recv(sock, 100) to read up to 100 bytes to msg, rather than having
to prep a msg_t.

The examples and documentation that come with the package give a
overview of using it, and what calls are not required. [1]


[1] https://octave.sourceforge.io/zeromq/package_doc/





Reply | Threaded
Open this post in threaded view
|

Re: Help-octave Digest, Vol 157, Issue 40

Ardid, Salva
El dilluns, 22 d’abril de 2019, a les 11:47:48 EDT, John Donoghue va escriure:

  On 4/22/19 9:41 AM, [hidden email] wrote:
  > Message: 2
  > Date: Sun, 21 Apr 2019 23:54:09 +0000
  > From: "Ardid, Salva"<[hidden email]>
  > To:"[hidden email]"  <[hidden email]>
  > Subject: Workaround missing functions in the zeromq package
  > Message-ID: <1655660.5vOfAfxstU@eksa>
  > Content-Type: text/plain; charset="utf-8"
  >
  > Hi,
  >
  > I started working to support Octave in Transplant using the zeromq package in Octave Forge. It?s the first time I use anything related to ZMQ and I reached a point where I?m stacked because Transplant uses zmq_msg_init, zmq_msg_recv, zmq_msg_data and zmq_msg_close, which are not yet supported in Octave?s package.
  >
  > So, I wonder, would there be some way to implement the same functionality just using zmq_recv?
  >
  > For the case it is useful, this is the piece of code I would need to adapt:
  >
  > msg = libstruct('zmq_msg_t', struct('hidden', zeros(1, 64, 'uint8')));
  > calllib('libzmq', 'zmq_msg_init', msg); % always returns 0
  > msglen = calllib('libzmq', 'zmq_msg_recv', msg, obj.socket, 0);
  > assert(msglen >= 0, obj.errortext('zmq_msg_recv'));
  > msgptr = calllib('libzmq', 'zmq_msg_data', msg);
  > if not(msgptr.isNull)
  >      setdatatype(msgptr, 'uint8Ptr', 1, msglen);
  >      str = uint8(msgptr.Value);
  > else
  >      str = uint8([]);
  > end
  > err = calllib('libzmq', 'zmq_msg_close', msg);
  > assert(err == 0, obj.errortext('zmq_msg_close'));
  >
  >
  > [Transplant also uses zmq contexts, which are not supported in the Octave package either. Although I?m not totally sure, I think this shouldn?t be strictly needed to make it work]
  >
  > Any help on this is very much appreciated!
  >
  > Thanks in advance and best, Salva
 
  the octave zmq package uses a zmq context within the package, but
  doesn't expose it to the user. The same goes for the zmq_msg_xxxx
  functions - they are used internally, but the user doesnt have direct
  access to them.
 
  So the user (after creating the zmq socket) can just call,   msg =
  zmq_recv(sock, 100) to read up to 100 bytes to msg, rather than having
  to prep a msg_t.
 
  The examples and documentation that come with the package give a
  overview of using it, and what calls are not required. [1]
 
 
  [1] https://octave.sourceforge.io/zeromq/package_doc/
 
I see. I read the docs but I guess what I missed was UP TO in "msg =
  zmq_recv(sock, 100) to read up to 100 bytes to msg"

That's awesome!

Thank you very much,
Salva
 
 
 




Reply | Threaded
Open this post in threaded view
|

Zero_mq stacked while receiving msg

Ardid, Salva
In reply to this post by John Donoghue-3

Unfortunately the receive function below gets stacked in Octave (i.e., when obj.isMatlab == 0 ) at the zmq_recv call and I don’t have a clue what might be wrong or how to fix it. Please note that other calls to the function seem to work fine, and then for whatever reason one of them gets trapped. I tried changing the number of bits to very large numbers with no difference. Any help is more than appreciated.

function str = receive(obj)
            if obj.isMatlab
                msg = libstruct('zmq_msg_t', struct('hidden', zeros(1, 64, 'uint8')));
                calllib('libzmq', 'zmq_msg_init', msg); % always returns 0
                msglen = calllib('libzmq', 'zmq_msg_recv', msg, obj.socket, 0);
                assert(msglen >= 0, obj.errortext('zmq_msg_recv'));
                msgptr = calllib('libzmq', 'zmq_msg_data', msg);
                if not(msgptr.isNull)
                    setdatatype(msgptr, 'uint8Ptr', 1, msglen);
                    str = uint8(msgptr.Value);
                else
                    str = uint8([]);
                end
                err = calllib('libzmq', 'zmq_msg_close', msg);
                assert(err == 0, obj.errortext('zmq_msg_close'));
            else
                % this is all transparent in Octave (internally managed by zmq_recv)
                str = zmq_recv (obj.socket, 64, 0);
            end
        end


Reply | Threaded
Open this post in threaded view
|

RE: Zero_mq stacked while receiving msg

John Donoghue-3


From: Ardid, Salva [mailto:[hidden email]]
Sent: Thursday, April 25, 2019 12:37 PM
To: John Donoghue
Cc: [hidden email]
Subject: Zero_mq stacked while receiving msg

Unfortunately the receive function below gets stacked in Octave (i.e., when obj.isMatlab == 0 ) at the zmq_recv call and I don’t have a clue what might be wrong or how to fix it. Please note that other calls to the function seem to work fine, and then for whatever reason one of them gets trapped. I tried changing the number of bits to very large numbers with no difference. Any help is more than appreciated.
function str = receive(obj)
            if obj.isMatlab
                msg = libstruct('zmq_msg_t', struct('hidden', zeros(1, 64, 'uint8')));
                calllib('libzmq', 'zmq_msg_init', msg); % always returns 0
                msglen = calllib('libzmq', 'zmq_msg_recv', msg, obj.socket, 0);
                assert(msglen >= 0, obj.errortext('zmq_msg_recv'));
                msgptr = calllib('libzmq', 'zmq_msg_data', msg);
                if not(msgptr.isNull)
                    setdatatype(msgptr, 'uint8Ptr', 1, msglen);
                    str = uint8(msgptr.Value);
                else
                    str = uint8([]);
                end
                err = calllib('libzmq', 'zmq_msg_close', msg);
                assert(err == 0, obj.errortext('zmq_msg_close'));
            else
                % this is all transparent in Octave (internally managed by zmq_recv)
                str = zmq_recv (obj.socket, 64, 0);
            end
        end


--------------------------------------

Stacked ? Not sure what that means.

Note that in your malab code,  the 'zmq_msg_t' structure that is 64 bytes in size
That’s not the same as saying the received data will be 64 bytes in length

So just out of interest sake, try a larger zmq_recv size ... 1024 ?

Which version of libzmq are you using ?



Reply | Threaded
Open this post in threaded view
|

Re: Zero_mq stacked while receiving msg

Ardid, Salva

El dijous, 25 d’abril de 2019, a les 15:45:05 EDT, JohnD va escriure:

 
  From: Ardid, Salva [mailto:[hidden email]]
  Sent: Thursday, April 25, 2019 12:37 PM
  To: John Donoghue
  Cc: [hidden email]
  Subject: Zero_mq stacked while receiving msg
 
  Unfortunately the receive function below gets stacked in Octave (i.e., when obj.isMatlab == 0 ) at the zmq_recv call and I don’t have a clue what might be wrong or how to fix it. Please note that other calls to the function seem to work fine, and then for whatever reason one of them gets trapped. I tried changing the number of bits to very large numbers with no difference. Any help is more than appreciated.
  function str = receive(obj)
              if obj.isMatlab
                  msg = libstruct('zmq_msg_t', struct('hidden', zeros(1, 64, 'uint8')));
                  calllib('libzmq', 'zmq_msg_init', msg); % always returns 0
                  msglen = calllib('libzmq', 'zmq_msg_recv', msg, obj.socket, 0);
                  assert(msglen >= 0, obj.errortext('zmq_msg_recv'));
                  msgptr = calllib('libzmq', 'zmq_msg_data', msg);
                  if not(msgptr.isNull)
                      setdatatype(msgptr, 'uint8Ptr', 1, msglen);
                      str = uint8(msgptr.Value);
                  else
                      str = uint8([]);
                  end
                  err = calllib('libzmq', 'zmq_msg_close', msg);
                  assert(err == 0, obj.errortext('zmq_msg_close'));
              else
                  % this is all transparent in Octave (internally managed by zmq_recv)
                  str = zmq_recv (obj.socket, 64, 0);
              end
          end
 
 
  --------------------------------------
 
  Stacked ? Not sure what that means.
 
  Note that in your malab code,  the 'zmq_msg_t' structure that is 64 bytes in size
  That’s not the same as saying the received data will be 64 bytes in length
 
  So just out of interest sake, try a larger zmq_recv size ... 1024 ?
 
  Which version of libzmq are you using ?
 
 
Hi John,

Thanks for your email.

"Stacked" means that nothing seems to happen, if I run the code with some verbose info I get:

> python3 pyTransplantOctaveTest.py
zeromq loaded
socket done
connection done
receiving msg
msg received
sending msg
msg sent
receiving msg
msg received
sending msg
msg sent
receiving msg
msg received
sending msg
msg sent
receiving msg

And then it stays in last `receiving message` for ever without actually receiving it (it would be indicated by `msg received`)

I know, 64 does not need to be the maximum number, however changing that number didn't change the behavior. I set it up to 2^31-1, because that is what the maximum seems to be in the python side according to an exception (response = msgpack.unpackb(self.socket.recv(), raw=False, max_bin_len=2**31-1)

I use libzmq 4.2.5

I don't see what might be wrong. Any help is more than welcome.

Thanks,
Salva



Reply | Threaded
Open this post in threaded view
|

RE: Zero_mq stacked while receiving msg

John Donoghue-3


> -----Original Message-----
> From: Ardid, Salva [mailto:[hidden email]]
> Sent: Thursday, April 25, 2019 4:20 PM
> To: JohnD
> Cc: [hidden email]
> Subject: Re: Zero_mq stacked while receiving msg
>
>
> El dijous, 25 d’abril de 2019, a les 15:45:05 EDT, JohnD va escriure:
>
>
>   From: Ardid, Salva [mailto:[hidden email]]
>   Sent: Thursday, April 25, 2019 12:37 PM
>   To: John Donoghue
>   Cc: [hidden email]
>   Subject: Zero_mq stacked while receiving msg
>
>   Unfortunately the receive function below gets stacked in Octave (i.e., when
> obj.isMatlab == 0 ) at the zmq_recv call and I don’t have a clue what might be
> wrong or how to fix it. Please note that other calls to the function seem to work
> fine, and then for whatever reason one of them gets trapped. I tried changing
> the number of bits to very large numbers with no difference. Any help is more
> than appreciated.
>   function str = receive(obj)
>               if obj.isMatlab
>                   msg = libstruct('zmq_msg_t', struct('hidden', zeros(1, 64, 'uint8')));
>                   calllib('libzmq', 'zmq_msg_init', msg); % always returns 0
>                   msglen = calllib('libzmq', 'zmq_msg_recv', msg, obj.socket, 0);
>                   assert(msglen >= 0, obj.errortext('zmq_msg_recv'));
>                   msgptr = calllib('libzmq', 'zmq_msg_data', msg);
>                   if not(msgptr.isNull)
>                       setdatatype(msgptr, 'uint8Ptr', 1, msglen);
>                       str = uint8(msgptr.Value);
>                   else
>                       str = uint8([]);
>                   end
>                   err = calllib('libzmq', 'zmq_msg_close', msg);
>                   assert(err == 0, obj.errortext('zmq_msg_close'));
>               else
>                   % this is all transparent in Octave (internally managed by zmq_recv)
>                   str = zmq_recv (obj.socket, 64, 0);
>               end
>           end
>
>
>   --------------------------------------
>
>   Stacked ? Not sure what that means.
>
>   Note that in your malab code,  the 'zmq_msg_t' structure that is 64 bytes in
> size
>   That’s not the same as saying the received data will be 64 bytes in length
>
>   So just out of interest sake, try a larger zmq_recv size ... 1024 ?
>
>   Which version of libzmq are you using ?
>
>
> Hi John,
>
> Thanks for your email.
>
> "Stacked" means that nothing seems to happen, if I run the code with some
> verbose info I get:
>
> > python3 pyTransplantOctaveTest.py
> zeromq loaded
> socket done
> connection done
> receiving msg
> msg received
> sending msg
> msg sent
> receiving msg
> msg received
> sending msg
> msg sent
> receiving msg
> msg received
> sending msg
> msg sent
> receiving msg
>
> And then it stays in last `receiving message` for ever without actually receiving
> it (it would be indicated by `msg received`)
>
> I know, 64 does not need to be the maximum number, however changing that
> number didn't change the behavior. I set it up to 2^31-1, because that is what
> the maximum seems to be in the python side according to an exception
> (response = msgpack.unpackb(self.socket.recv(), raw=False,
> max_bin_len=2**31-1)
>
> I use libzmq 4.2.5
>
> I don't see what might be wrong. Any help is more than welcome.
>
> Thanks,
> Salva
>


Is it possible that whatever is sending the message is failing or considering that a message sent is invalid and not responding back with a reply ?



Reply | Threaded
Open this post in threaded view
|

Re: Zero_mq stuck while receiving msg

Ardid, Salva

El dijous, 25 d’abril de 2019, a les 17:20:50 EDT, JohnD va escriure:

>  Is it possible that whatever is sending the message is failing or considering that a message sent is invalid and not responding back with a reply ?


I guess it's what happens, but the code works in Matlab (with loadlibrary, calllib, and so on). Is there any way to make zeromq more verbose? Perhaps that could help to understand the issue better...

[ Sorry I realized now that I wrote stacked several times instead of stuck ]




Reply | Threaded
Open this post in threaded view
|

Re: Zero_mq stuck while receiving msg

Ardid, Salva


El dijous, 25 d’abril de 2019, a les 17:32:59 EDT, Ardid, Salva va escriure:

 
  El dijous, 25 d’abril de 2019, a les 17:20:50 EDT, JohnD va escriure:
 
  >  Is it possible that whatever is sending the message is failing or considering that a message sent is invalid and not responding back with a reply ?
 
 
  I guess it's what happens, but the code works in Matlab (with loadlibrary, calllib, and so on). Is there any way to make zeromq more verbose? Perhaps that could help to understand the issue better...
 
  [ Sorry I realized now that I wrote stacked several times instead of stuck ]
 
 
 
 
 
I put extra info in matlab and octave:

Matlab:

libzmq loaded
socket done
connection done
receiving msg:
{"name":"eval","type":"get_global"}
msg received.
sending msg:
{"type":"value","value":["__function__","eval"]}
msg sent.
receiving msg:
{"name":"eval","args":["0;"],"nargout":-1,"type":"call"}
msg received.
sending msg:
{"type":"value","value":0}

Octave:

zeromq loaded
socket done
connection done
receiving msg:
{"name":"eval","type":"get_global"}
msg received.
sending msg:
{"identifier":"Octave:undefined-function","message":"'idx' undefined near line 236 column 13","stack":[{"file":"\/home\/miniconda\/envs\/cenv1\/lib\/python3.7\/site-packages\/transplant\/parsejson.m","name":"parsejson>false","line":236,"column":11,"scope":null,"context":0},{"file":"\/home\/miniconda\/envs\/cenv1\/lib\/python3.7\/site-packages\/transplant\/parsejson.m","name":"parsejson>object","line":143,"column":9,"scope":null,"context":0},{"file":"\/home\/miniconda\/envs\/cenv1\/lib\/python3.7\/site-packages\/transplant\/parsejson.m","name":"parsejson>value","line":50,"column":20,"scope":null,"context":0},{"file":"\/home\/miniconda\/envs\/cenv1\/lib\/python3.7\/site-packages\/transplant\/parsejson.m","name":"parsejson","line":27,"column":16,"scope":null,"context":0},{"file":"\/home\/miniconda\/envs\/cenv1\/lib\/python3.7\/site-packages\/transplant\/transplant_remote.m","name":"transplant_remote>receive_msg","line":200,"column":21,"scope":null,"context":0},{"file":"\/home\/miniconda\/envs\/cenv1\/lib\/python3.7\/site-packages\/transplant\/transplant_remote.m","name":"transplant_remote","line":71,"column":17,"scope":null,"context":0}],"type":"error"}
msg sent.
receiving msg:
{"name":"what","type":"get_global"}
msg received.
sending msg:
{"type":"value","value":["__function__","what"]}
msg sent.
receiving msg:
{"name":"what","args":["eval"],"nargout":-1,"type":"call"}
msg received.
sending msg:
{"identifier":null,"message":"what: could not find the directory eval","stack":[{"file":"\/app\/share\/octave\/5.1.0\/m\/miscellaneous\/what.m","name":"what","line":85,"column":9,"scope":null,"context":0},{"file":"\/home\/miniconda\/envs\/cenv1\/lib\/python3.7\/site-packages\/transplant\/transplant_remote.m","name":"transplant_remote","line":129,"column":38,"scope":null,"context":0}],"type":"error"}
msg sent.
receiving msg:

So I think now that the actual problem may start with those weird sent messages, which do not occur in Matlab

Reply | Threaded
Open this post in threaded view
|

Re: Zero_mq stuck while receiving msg

Ardid, Salva


El dijous, 25 d’abril de 2019, a les 18:12:43 EDT, Salva Ardid va escriure:

 
  El dijous, 25 d’abril de 2019, a les 17:32:59 EDT, Ardid, Salva va escriure:
 
   
    El dijous, 25 d’abril de 2019, a les 17:20:50 EDT, JohnD va escriure:
   
    >  Is it possible that whatever is sending the message is failing or considering that a message sent is invalid and not responding back with a reply ?
   
   
    I guess it's what happens, but the code works in Matlab (with loadlibrary, calllib, and so on). Is there any way to make zeromq more verbose? Perhaps that could help to understand the issue better...
   
    [ Sorry I realized now that I wrote stacked several times instead of stuck ]
   
   
   
   
   
  I put extra info in matlab and octave:
 
  Matlab:
 
  libzmq loaded
  socket done
  connection done
  receiving msg:
  {"name":"eval","type":"get_global"}
  msg received.
  sending msg:
  {"type":"value","value":["__function__","eval"]}
  msg sent.
  receiving msg:
  {"name":"eval","args":["0;"],"nargout":-1,"type":"call"}
  msg received.
  sending msg:
  {"type":"value","value":0}
 
  Octave:
 
  zeromq loaded
  socket done
  connection done
  receiving msg:
  {"name":"eval","type":"get_global"}
  msg received.
  sending msg:
  {"identifier":"Octave:undefined-function","message":"'idx' undefined near line 236 column 13","stack":[{"file":"\/home\/miniconda\/envs\/cenv1\/lib\/python3.7\/site-packages\/transplant\/parsejson.m","name":"parsejson>false","line":236,"column":11,"scope":null,"context":0},{"file":"\/home\/miniconda\/envs\/cenv1\/lib\/python3.7\/site-packages\/transplant\/parsejson.m","name":"parsejson>object","line":143,"column":9,"scope":null,"context":0},{"file":"\/home\/miniconda\/envs\/cenv1\/lib\/python3.7\/site-packages\/transplant\/parsejson.m","name":"parsejson>value","line":50,"column":20,"scope":null,"context":0},{"file":"\/home\/miniconda\/envs\/cenv1\/lib\/python3.7\/site-packages\/transplant\/parsejson.m","name":"parsejson","line":27,"column":16,"scope":null,"context":0},{"file":"\/home\/miniconda\/envs\/cenv1\/lib\/python3.7\/site-packages\/transplant\/transplant_remote.m","name":"transplant_remote>receive_msg","line":200,"column":21,"scope":null,"context":0},{"file":"\/home\/miniconda\/envs\/cenv1\/lib\/python3.7\/site-packages\/transplant\/transplant_remote.m","name":"transplant_remote","line":71,"column":17,"scope":null,"context":0}],"type":"error"}
  msg sent.
  receiving msg:
  {"name":"what","type":"get_global"}
  msg received.
  sending msg:
  {"type":"value","value":["__function__","what"]}
  msg sent.
  receiving msg:
  {"name":"what","args":["eval"],"nargout":-1,"type":"call"}
  msg received.
  sending msg:
  {"identifier":null,"message":"what: could not find the directory eval","stack":[{"file":"\/app\/share\/octave\/5.1.0\/m\/miscellaneous\/what.m","name":"what","line":85,"column":9,"scope":null,"context":0},{"file":"\/home\/miniconda\/envs\/cenv1\/lib\/python3.7\/site-packages\/transplant\/transplant_remote.m","name":"transplant_remote","line":129,"column":38,"scope":null,"context":0}],"type":"error"}
  msg sent.
  receiving msg:
 
  So I think now that the actual problem may start with those weird sent messages, which do not occur in Matlab
 

These are errors in transplant_remote.m when calling the "what" and "message" functions. The first exists in Octave but there should be some sort of compatibility issue, whereas the function message is not yet implemented in Octave. At least I know better what's going on. I'll need to go to that file and adapt it for Octave...

Thanks