loading big gzip compressed and uncompressed numerical matrices

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

loading big gzip compressed and uncompressed numerical matrices

Andreas Weber-6
Dear all,

I've implemented a CSV parser as OCT file [1] to load gzip compressed
numerical matrices.

The reason for this was that I'm running GNU Octave on embedded systems
and I had to deal with increasing, gzip compressed files (see my
question 3 weeks ago [2]).

The result is quite surprising (at least for me): It's approx. 12 times
faster than the builtin "load" even for non-incremental reads.

I hope it can be useful for others who deal with large numerical CSVs.

-- Andy

[1] https://github.com/Andy1978/load_gz/tree/master
[2]
https://octave.1599824.n4.nabble.com/incremental-read-of-gzipped-matrix-td4694821.html