# Optimal way to handle big data table ?

4 messages
Open this post in threaded view
|
Report Content as Inappropriate

## Optimal way to handle big data table ?

 Hello, I was doing recently lamp spectrum analysis to extract photometric properties. This implies to compute the integral of the spectrum by the CIE1931 sensitivity functions; they are tabulated at 400 wavelength, each time 4 values. What's the best way to use those data inside a function ? 1) encode them inside the function body ? It will be compiled once. CIE31Table = [360 0.000130 0.000004 0.000606               361 0.000146 0.000004 0.000681               362 0.000164 0.000005 0.000765 ... ]; 2) read them from a text file ? 3) read them from a binary file ? 4) other ? The point is to minimise the computational load of each time refilling this matrix with its 1600 entries. Regards Pascal
Open this post in threaded view
|
Report Content as Inappropriate

## Re: Optimal way to handle big data table ?

 ----- Original Message ----- > From: CdeMills <[hidden email]> > To: [hidden email] > Cc: > Sent: Thursday, March 7, 2013 12:07 PM > Subject: Optimal way to handle big data table ? > > Hello, > > I was doing recently lamp spectrum analysis to extract photometric > properties. This implies to compute the integral of the spectrum by the > CIE1931 sensitivity functions; they are tabulated at 400 wavelength, each > time 4 values. What's the best way to use those data inside a function ? > 1) encode them inside the function body ? It will be compiled once. > CIE31Table = [360 0.000130 0.000004 0.000606 >               361 0.000146 0.000004 0.000681 >               362 0.000164 0.000005 0.000765 ... ]; > 2) read them from a text file ? > 3) read them from a binary file ? > 4) other ? > > The point is to minimise the computational load of each time refilling this > matrix with its 1600 entries. > > Regards > > Pascal > > > > -- > View this message in context: > http://octave.1599824.n4.nabble.com/Optimal-way-to-handle-big-data-table-tp4650575.html> Sent from the Octave - General mailing list archive at Nabble.com. > _______________________________________________ > Help-octave mailing list > [hidden email] > https://mailman.cae.wisc.edu/listinfo/help-octave> This is not that big - I, for example, deal with loading about 1e6 audio samples in ASCII, and it takes, say, half a minute or so. Anyway, the obvious way is to first create your table in ASCII, load it, then save it as binary, and then load it as needed from the binary. Regards,   Sergei. _______________________________________________ Help-octave mailing list [hidden email] https://mailman.cae.wisc.edu/listinfo/help-octave
Open this post in threaded view
|
Report Content as Inappropriate

## Re: Optimal way to handle big data table ?

 In reply to this post by CdeMills Hi Pascal,You get the best input/output performance when using a binary file format (such as octave's mat format).Cheers,Alex On Thu, Mar 7, 2013 at 11:07 AM, CdeMills wrote: Hello, I was doing recently lamp spectrum analysis to extract photometric properties. This implies to compute the integral of the spectrum by the CIE1931 sensitivity functions; they are tabulated at 400 wavelength, each time 4 values. What's the best way to use those data inside a function ? 1) encode them inside the function body ? It will be compiled once. CIE31Table = [360 0.000130 0.000004 0.000606               361 0.000146 0.000004 0.000681               362 0.000164 0.000005 0.000765 ... ]; 2) read them from a text file ? 3) read them from a binary file ? 4) other ? The point is to minimise the computational load of each time refilling this matrix with its 1600 entries. Regards Pascal -- View this message in context: http://octave.1599824.n4.nabble.com/Optimal-way-to-handle-big-data-table-tp4650575.html Sent from the Octave - General mailing list archive at Nabble.com. _______________________________________________ Help-octave mailing list [hidden email] https://mailman.cae.wisc.edu/listinfo/help-octave _______________________________________________ Help-octave mailing list [hidden email] https://mailman.cae.wisc.edu/listinfo/help-octave