Optimal way to handle big data table ?

[CdeMills]

Hello,

I was doing recently lamp spectrum analysis to extract photometric properties. This implies to compute the integral of the spectrum by the CIE1931 sensitivity functions; they are tabulated at 400 wavelength, each time 4 values. What's the best way to use those data inside a function ?
1) encode them inside the function body ? It will be compiled once.
CIE31Table = [360 0.000130 0.000004 0.000606
              361 0.000146 0.000004 0.000681
              362 0.000164 0.000005 0.000765 ... ];
2) read them from a text file ?
3) read them from a binary file ?
4) other ?

The point is to minimise the computational load of each time refilling this matrix with its 1600 entries.

Regards

Pascal

[Sergei Steshenko-2]

----- Original Message -----

> From: CdeMills <[hidden email]>
> To: [hidden email]
> Cc:
> Sent: Thursday, March 7, 2013 12:07 PM
> Subject: Optimal way to handle big data table ?
>
> Hello,
>
> I was doing recently lamp spectrum analysis to extract photometric
> properties. This implies to compute the integral of the spectrum by the
> CIE1931 sensitivity functions; they are tabulated at 400 wavelength, each
> time 4 values. What's the best way to use those data inside a function ?
> 1) encode them inside the function body ? It will be compiled once.
> CIE31Table = [360 0.000130 0.000004 0.000606
>               361 0.000146 0.000004 0.000681
>               362 0.000164 0.000005 0.000765 ... ];
> 2) read them from a text file ?
> 3) read them from a binary file ?
> 4) other ?
>
> The point is to minimise the computational load of each time refilling this
> matrix with its 1600 entries.
>
> Regards
>
> Pascal
>
>
>
> --
> View this message in context:
> http://octave.1599824.n4.nabble.com/Optimal-way-to-handle-big-data-table-tp4650575.html
> Sent from the Octave - General mailing list archive at Nabble.com.
> _______________________________________________
> Help-octave mailing list
> [hidden email]
> https://mailman.cae.wisc.edu/listinfo/help-octave
>

This is not that big - I, for example, deal with loading about 1e6 audio samples in ASCII, and it takes, say, half a minute or so.

Anyway, the obvious way is to first create your table in ASCII, load it, then save it as binary, and then load it as needed from the binary.

Regards,
  Sergei.

_______________________________________________
Help-octave mailing list
[hidden email]
https://mailman.cae.wisc.edu/listinfo/help-octave

[Alexander Barth-3]

Hi Pascal,

You get the best input/output performance when using a binary file format (such as octave's mat format).

Cheers,

Alex

On Thu, Mar 7, 2013 at 11:07 AM, CdeMills <[hidden email]> wrote:

Hello,

I was doing recently lamp spectrum analysis to extract photometric
properties. This implies to compute the integral of the spectrum by the
CIE1931 sensitivity functions; they are tabulated at 400 wavelength, each
time 4 values. What's the best way to use those data inside a function ?
1) encode them inside the function body ? It will be compiled once.
CIE31Table = [360 0.000130 0.000004 0.000606
              361 0.000146 0.000004 0.000681
              362 0.000164 0.000005 0.000765 ... ];
2) read them from a text file ?
3) read them from a binary file ?
4) other ?

The point is to minimise the computational load of each time refilling this
matrix with its 1600 entries.

Regards

Pascal



--
View this message in context: http://octave.1599824.n4.nabble.com/Optimal-way-to-handle-big-data-table-tp4650575.html
Sent from the Octave - General mailing list archive at Nabble.com.
_______________________________________________
Help-octave mailing list
[hidden email]
https://mailman.cae.wisc.edu/listinfo/help-octave

_______________________________________________
Help-octave mailing list
[hidden email]
https://mailman.cae.wisc.edu/listinfo/help-octave

[Alexander Barth-3]

and of course you should write your code such that you read the data only once (e.g. passing the data as parameter or using global or persistent variables)

see "help global" and "help persistent".

On Wed, Mar 20, 2013 at 1:39 PM, Alexander Barth <[hidden email]> wrote:

Hi Pascal,

You get the best input/output performance when using a binary file format (such as octave's mat format).

Cheers,

Alex

On Thu, Mar 7, 2013 at 11:07 AM, CdeMills <[hidden email]> wrote:

Hello,

I was doing recently lamp spectrum analysis to extract photometric
properties. This implies to compute the integral of the spectrum by the
CIE1931 sensitivity functions; they are tabulated at 400 wavelength, each
time 4 values. What's the best way to use those data inside a function ?
1) encode them inside the function body ? It will be compiled once.
CIE31Table = [360 0.000130 0.000004 0.000606
              361 0.000146 0.000004 0.000681
              362 0.000164 0.000005 0.000765 ... ];
2) read them from a text file ?
3) read them from a binary file ?
4) other ?

The point is to minimise the computational load of each time refilling this
matrix with its 1600 entries.

Regards

Pascal



--
View this message in context: http://octave.1599824.n4.nabble.com/Optimal-way-to-handle-big-data-table-tp4650575.html
Sent from the Octave - General mailing list archive at Nabble.com.
_______________________________________________
Help-octave mailing list
[hidden email]
https://mailman.cae.wisc.edu/listinfo/help-octave

_______________________________________________
Help-octave mailing list
[hidden email]
https://mailman.cae.wisc.edu/listinfo/help-octave