Raw file i/o

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Raw file i/o

Guido Dietz
You wrote:

> Superuser AIA <[hidden email]> wrote:
>
> : Hello OCTAVE users,
> :
> : For further use of our old MATLAB programs and datas we needed a
> : procedure for a raw file i/o like in the new MATLAB fread and
> : fwrite.
>
> Thanks for the changes.
>
> We already have an implementation of these functions installed for the
> next release, but they don't seem to have all the functionality of
> your changes.  I would like to try to merge in any improvements, but I
> think that will have to wait until after 1.1 is released.
>
> Thanks,
>
> jwe

I wrote you a few days before, now the work is beeing tested:

We just encoded the feature to read and write by fread and
fwrite with specified formats. Therefore we implemented the new
builtin-variable "raw_io_format" wich can have the values "default",
"ieee-little", "ieee-big", "vax-d" or "vax-g".

Method:
* Changes in "src/variables.cc", "src/user-prefs.cc",
  "src/user-prefs.h" for implementing the new builtin-variable.
* Extracting and modifying the int and float conversion routines
  from "src/load-save.cc" to "liboctave/FConc.cc",
  "liboctave/FConc.h" and "liboctave/FSwap.h". Changes in
  "liboctave/Makefile.in".
* implementing the conversion routines in "liboctave/dMatrix.cc".

ToDo:
* All the various conversion routines vax>cray, ieee>cray, .......;
  but I don't know whether I can do that ...

After some testings (two or three days) I will send you a patch and
hope that you will add it to the next release.

Bye G.
 


Reply | Threaded
Open this post in threaded view
|

Re: Raw file i/o

Guido Dietz
You wrote:

>One thing I would like to add is to add an optional third argument to
>fopen() that would set the default conversion type.  The argument
>should be a string, and should be compatible with the way Matlab
>works (this is from the Matlab help and was posted to
>comp.soft-sys.matlab this week):
>
>[FID, MESSAGE] = FOPEN('filename',permission, machineformat) opens the
>        specified file with the specified permission and treats data read
>        using FREAD or data written using FWRITE as having a format given
>        by machineformat. machineformat is one of the following strings:
>
>        'native'      or 'n' - local machine format - the default
>        'ieee-le'     or 'l' - IEEE floating point with little-endian
>                               byte ordering
>        'ieee-be'     or 'b' - IEEE floating point with big-endian
>                               byte ordering
>        'vaxd'        or 'd' - VAX D floating point and VAX ordering
>        'vaxg'        or 'g' - VAX G floating point and VAX ordering
>        'cray'        or 'c' - Cray floating point with big-endian
>                               byte ordering
>        'ieee-le.l64' or 'a' - IEEE floating point with little-endian
>                               byte ordering and 64 bit long data type
>                               (currently supported on: alpha, sgi, ibm_rs,
>                                hp300, vax, and cray platforms only)
>
>Perhaps the strings you have chosen for the new built-in variables
>should be changed to match these (or at least accept them too).
>
>: After some testings (two or three days) I will send you a patch and
>: hope that you will add it to the next release.
>
>Thanks, I'm looking forward to seeing them.
>
>jwe

I reckon this is a good idea: I am just working on it. The builtin variable
'raw_io_format' I put in the waste paper basket.

Therefore the new method:
* Extracting and modifying the int and float conversion routines
  from "src/load-save.cc" to "liboctave/FConc.cc",
  "liboctave/FConc.h" and "liboctave/FSwap.h". Changes in
  "liboctave/Makefile.in".
* implementing the conversion routines in "liboctave/dMatrix.cc".
* adding 'file_arch' and others to the file_info structure and modifying
  the 'fopen*' calls as  well as 'fread_internal' and 'fwrite_internal' in
  "src/load-save.cc".

Only for the 'ieee-le.l64' I am to lazy, but I hope you'll respond:

>I wouldn't worry too much about this.  Perhaps someone who has a need
>for these conversions will eventually implement them.

as in the last mail!

Bye bye
G.



--

Reply | Threaded
Open this post in threaded view
|

Re: Raw file i/o

Przemek Klosowski-4
There has been some discussion of binary data file formats in octave, with
several people suggesting extensions to handle various binary data types.

I would suggest that instead of inventing yet another scheme for
portable binaries, octave uses some standard method. I would recommend
using the HDF library, which provides architecture-independent binary
data format, plus it allows for self-describing data (it is possible
to store attributes with the data, such as the data topology
(rank/dimensions), units, equations used to generate the data.)

        przemek



Reply | Threaded
Open this post in threaded view
|

Re: Raw file i/o

John Eaton-4
In reply to this post by Guido Dietz
[hidden email] (Przemek Klosowski) wrote:

: There has been some discussion of binary data file formats in octave, with
: several people suggesting extensions to handle various binary data types.
:
: I would suggest that instead of inventing yet another scheme for
: portable binaries, octave uses some standard method. I would recommend
: using the HDF library,

I agree that using something like this would be desirable.

But I think Octave needs the other features as well, to provide
compatibility with Matlab's binary I/O.  I believe that this is also
important, since people seem to want Octave to be compatible.

If there is a (mostly) standard library for binary data that is freely
available, I would like to use it.  I would even drop Octave's own
binary format in favor of the new method if it is easy to write a
conversion program (something that I expect should not be too hard).

In addition to HDF, I have heard about netCDF, which also looks like
it might be a reasonable alternative.

Can anyone comment on which data file formats (netCDF, HDF, or others)
are likely to be become standards?

(Please e-mail me directly unless you really think it is something
that would be of interest to most people who read help-octave.)

Thanks,

jwe