Saving symbolic variables in a database

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Saving symbolic variables in a database

diego9j
Hi,

First of all, I might be a little bit colloquial in my explanations (I recently started programming in Octave), but I hope that there's no problem in that.

As a part of a bigger project, I need to store a symbolic variable (from symbolic package) into a postgres database (with database package). I got one idea of how to do it but I'm having some trouble that I can't solve, so both new ideas and solutions for my problem are welcome.

Since there's no "direct" way of storing symbolic variables in postgres, I thought about storing the binary data of the variable, something like:

binary_data=save("-binary","-","symbolic_variable_name");
8bit_data=uint8(binary_data);

Then I store it with pq_exec_params as bytea postgres type. The problem is that
1 - I found no function to parse back from 8-bit to binary
2 - I don't know how to make that binary data a variable again

When doing simple file i/o, you save the binary to a file and then with the "load" function, the variable "pops out" again, but I don't know how to do that with an octave string.


As I said, anything that helps me will be greatly appreciated, I'm really stuck with this...

Best regards
Diego
Reply | Threaded
Open this post in threaded view
|

Re: Saving symbolic variables in a database

Doug Stewart-4


On Wed, Mar 9, 2016 at 1:05 PM, diego9j <[hidden email]> wrote:
Hi,

First of all, I might be a little bit colloquial in my explanations (I
recently started programming in Octave), but I hope that there's no problem
in that.

As a part of a bigger project, I need to store a symbolic variable (from
symbolic package) into a postgres database (with database package). I got
one idea of how to do it but I'm having some trouble that I can't solve, so
both new ideas and solutions for my problem are welcome.

Since there's no "direct" way of storing symbolic variables in postgres, I
thought about storing the binary data of the variable, something like:

binary_data=save("-binary","-","symbolic_variable_name");
8bit_data=uint8(binary_data);

Then I store it with pq_exec_params as bytea postgres type. The problem is
that
1 - I found no function to parse back from 8-bit to binary
2 - I don't know how to make that binary data a variable again

When doing simple file i/o, you save the binary to a file and then with the
"load" function, the variable "pops out" again, but I don't know how to do
that with an octave string.


As I said, anything that helps me will be greatly appreciated, I'm really
stuck with this...

Best regards
Diego



--
View this message in context: http://octave.1599824.n4.nabble.com/Saving-symbolic-variables-in-a-database-tp4675312.html
Sent from the Octave - General mailing list archive at Nabble.com.

_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave



Would anything onĀ http://wiki.octave.org/Symbolic_package
help you?
 ff=function_handle(rhs(f))

--
DAS


_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave
Reply | Threaded
Open this post in threaded view
|

Re: Saving symbolic variables in a database

Colin Macdonald-2
On 09/03/16 20:20, Doug Stewart wrote:
> On Wed, Mar 9, 2016 at 1:05 PM, diego9j <[hidden email]
>     As a part of a bigger project, I need to store a symbolic variable (from
>     symbolic package) into a postgres database (with database package).
>
>
> Would anything on http://wiki.octave.org/Symbolic_package
> help you?
>
>  ff=function_handle(rhs(f))

Following is undocumented: I might break it anytime...  Either
`sym(char(x))` or `sym(x.pickle)` should both restore x.  So you can
save the string `x.pickle` to your database, then pass it through sym it
on load...

I filed this to improve the saving/loading of sym:
https://github.com/cbm755/octsympy/issues/376

Colin

_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave
Reply | Threaded
Open this post in threaded view
|

Re: Saving symbolic variables in a database

Olaf Till-2
In reply to this post by diego9j
On Wed, Mar 09, 2016 at 10:05:53AM -0800, diego9j wrote:

> Hi,
>
> First of all, I might be a little bit colloquial in my explanations (I
> recently started programming in Octave), but I hope that there's no problem
> in that.
>
> As a part of a bigger project, I need to store a symbolic variable (from
> symbolic package) into a postgres database (with database package). I got
> one idea of how to do it but I'm having some trouble that I can't solve, so
> both new ideas and solutions for my problem are welcome.
>
> Since there's no "direct" way of storing symbolic variables in postgres, I
> thought about storing the binary data of the variable, something like:
>
> binary_data=save("-binary","-","symbolic_variable_name");
> 8bit_data=uint8(binary_data);
>
> Then I store it with pq_exec_params as bytea postgres type. The problem is
> that
> 1 - I found no function to parse back from 8-bit to binary
> 2 - I don't know how to make that binary data a variable again
>
> When doing simple file i/o, you save the binary to a file and then with the
> "load" function, the variable "pops out" again, but I don't know how to do
> that with an octave string.
I wasn't aware of the "-" feature of 'save', it seems to be a new
addition, specific to Octave. But as I see it, there might be some
issues:

- Saving "-binary" into a string seems not the right thing to me,
  e.g. it will break if the binary data contains a zero byte. Maybe
  better if Octave would save into an uint8-array if saving binary?

- A 'symmetric' operation -- loading from a variable -- should be
  provided, but seemingly isn't currently available.

I don't quite understand your item '1 - I found no function to parse
back from 8-bit to binary' -- char() would convert from uint8 back to
a string, but I don't see that this would help you.

It's nice to see someone using the database package. But it wasn't
designed to store _arbitrary_ Octave variables into a database. But if
there really is such a need, maybe the database package itself should
provide a way, as long as there is no satisfying possibility with
Octaves save/load. Some functions like 'var2bytea' and
'bytea2var'. I'd guess I could add these within a few days, if really
necessary ...

Olaf

--
public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net

_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Saving symbolic variables in a database

Olaf Till-2
Another thought, would it be an alternative to you to save/load
into/from a temporary file and use the packages functions for
import/export of large objects into/from the Postgresql database?

Or have you masses of such variables with no corresponding SQL type to
import into the database? Then it would probably be better to provide
a function to register a new converter in the 'database' package; this
would require you to create a corresponding composite type (with one
bytea element) in your Postgresql database ... so you would only have
to specify the typename, without manual conversion. I currently don't
know how feasible this would be for me to implement.

Olaf

--
public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net

_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Saving symbolic variables in a database

Olaf Till-2
In reply to this post by Olaf Till-2
On Thu, Mar 10, 2016 at 09:25:19AM +0100, Olaf Till wrote:

> On Wed, Mar 09, 2016 at 10:05:53AM -0800, diego9j wrote:
> > ...
> > As a part of a bigger project, I need to store a symbolic variable (from
> > symbolic package) into a postgres database (with database package). I got
> > one idea of how to do it but I'm having some trouble that I can't solve, so
> > both new ideas and solutions for my problem are welcome.
> >
> > Since there's no "direct" way of storing symbolic variables in postgres, I
> > thought about storing the binary data of the variable, something like:
> >
> > binary_data=save("-binary","-","symbolic_variable_name");
> > 8bit_data=uint8(binary_data);
> >
> > Then I store it with pq_exec_params as bytea postgres type. The problem is
> > that
> > 1 - I found no function to parse back from 8-bit to binary
> > 2 - I don't know how to make that binary data a variable again
> >
> > When doing simple file i/o, you save the binary to a file and then with the
> > "load" function, the variable "pops out" again, but I don't know how to do
> > that with an octave string.
> ...
> It's nice to see someone using the database package. But it wasn't
> designed to store _arbitrary_ Octave variables into a database. But if
> there really is such a need, maybe the database package itself should
> provide a way, as long as there is no satisfying possibility with
> Octaves save/load. Some functions like 'var2bytea' and
> 'bytea2var'. I'd guess I could add these within a few days, if really
> necessary ...
I didn't hear anything again from you, but the functions suggested
above are now added to 'database', see function documentation. A new
package release still must wait a bit, tell me if you should urgently
need this functionality but need help in accessing the code
repository.

Olaf

--
public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net

_______________________________________________
Help-octave mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-octave

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Saving symbolic variables in a database

diego9j
In reply to this post by Colin Macdonald-2
Colin Macdonald-2 wrote
Following is undocumented: I might break it anytime...  Either
`sym(char(x))` or `sym(x.pickle)` should both restore x.  So you can
save the string `x.pickle` to your database, then pass it through sym it
on load...
This is exactly what I needed, lots of thanks Colin!  This avoids using octave's save/load commands and all the parsing problems. Also, I think that saving a string is way more elegant than saving <binary data>.


Olaf Till-2 wrote
I wasn't aware of the "-" feature of 'save', it seems to be a new
addition, specific to Octave. But as I see it, there might be some
issues:

- Saving "-binary" into a string seems not the right thing to me,
  e.g. it will break if the binary data contains a zero byte. Maybe
  better if Octave would save into an uint8-array if saving binary?

- A 'symmetric' operation -- loading from a variable -- should be
  provided, but seemingly isn't currently available.

I don't quite understand your item '1 - I found no function to parse
back from 8-bit to binary' -- char() would convert from uint8 back to
a string, but I don't see that this would help you.
I don't know if it's because of my inexperience, but Octave's file and variable I/O is terrible. I skimmed through octave-forge packages and I found none extending the I/O functionalities. I guess that it's something that could be done, if someone wanted to take that project.

Furthermore, most of the "binary data parsing" that I proposed in the main message makes no sense. I was just testing things to avoid the lack of bytea type in octave (which makes no sense by the way, why can octave return binary data but not in binary variable type?)

Olaf Till-2 wrote
It's nice to see someone using the database package. But it wasn't
designed to store _arbitrary_ Octave variables into a database. But if
there really is such a need, maybe the database package itself should
provide a way, as long as there is no satisfying possibility with
Octaves save/load. Some functions like 'var2bytea' and
'bytea2var'. I'd guess I could add these within a few days, if really
necessary ...
Those two functions seem appropiate additions to the package since it would add lots of compatibility between octave forge packages. My problem is solved but other people could use them...

Olaf Till-2 wrote
I didn't hear anything again from you, but the functions suggested
above are now added to 'database', see function documentation. A new
package release still must wait a bit, tell me if you should urgently
need this functionality but need help in accessing the code
repository.
Are those functions 'var2bytea' and 'bytea2var'? If so, that's really nice. I don't need them urgently, but will check the function reference from time to time to see when the new package release is public :)