slow code, optimisation advice needed

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

slow code, optimisation advice needed

tomkut
Dear Octavers

Would anyone be so kind and offer advice on speeding up the below (the loop
takes a long time)?
a = fileread ("C:/Backup/Thermo Backups/2018-05-17.txt");
d = strsplit (a, "\n");
for i = 1 : length(d)
 e (i , :) = strsplit ( d{i} );
endfor




--
Sent from: http://octave.1599824.n4.nabble.com/Octave-General-f1599825.html


Reply | Threaded
Open this post in threaded view
|

Re: slow code, optimisation advice needed

Ian McCallion
Can you send a fragment of the input file and an idea of its size? Ian

On 25 May 2018 at 15:13, tomkut <[hidden email]> wrote:

> Dear Octavers
>
> Would anyone be so kind and offer advice on speeding up the below (the loop
> takes a long time)?
> a = fileread ("C:/Backup/Thermo Backups/2018-05-17.txt");
> d = strsplit (a, "\n");
> for i = 1 : length(d)
>  e (i , :) = strsplit ( d{i} );
> endfor
>
>
>
>
> --
> Sent from: http://octave.1599824.n4.nabble.com/Octave-General-f1599825.html
>
>


Reply | Threaded
Open this post in threaded view
|

Re: slow code, optimisation advice needed

tomkut
This post was updated on .
Yes Ian, here is a sample  2018-05-17_-_Copy.txt
<http://octave.1599824.n4.nabble.com/file/t372862/2018-05-17_-_Copy.txt
size: about 6MB . There are some heading rows that are filtered prior to the code here.
I added:
n=length(strsplit(d{1}))
e = cell(length(d),n);
and that improved speed greatly already.



--
Sent from: http://octave.1599824.n4.nabble.com/Octave-General-f1599825.html


Reply | Threaded
Open this post in threaded view
|

Re: slow code, optimisation advice needed

Ian McCallion
From the sample the data is numeric and has a fixed number of columns.
So do you really want it in cells containing character strings? If so
there is not much else you can do IMHO, but if what you really want is
the numbers try

e = dlmread ("C:/Backup/Thermo Backups/2018-05-17.txt");

which runs about 15 times faster.

HTH
Ian

On 29 May 2018 at 13:49, tomkut <[hidden email]> wrote:

> Yes Ian, here is a sample  2018-05-17_-_Copy.txt
> <http://octave.1599824.n4.nabble.com/file/t372862/2018-05-17_-_Copy.txt>
> size: about 6MB
> I added:
> n=length(strsplit(d{1}))
> e = cell(length(d),n);
> and that improved speed greatly already.
>
>
>
> --
> Sent from: http://octave.1599824.n4.nabble.com/Octave-General-f1599825.html
>
>


Reply | Threaded
Open this post in threaded view
|

Re: slow code, optimisation advice needed

tomkut
Unfortunately, Ian, there are some heading rows that are filtered prior to
the code here.



--
Sent from: http://octave.1599824.n4.nabble.com/Octave-General-f1599825.html


Reply | Threaded
Open this post in threaded view
|

Re: slow code, optimisation advice needed

PhilipNienhuis
In reply to this post by tomkut
tomkut wrote

> Dear Octavers
>
> Would anyone be so kind and offer advice on speeding up the below (the
> loop
> takes a long time)?
> a = fileread ("C:/Backup/Thermo Backups/2018-05-17.txt");
> d = strsplit (a, "\n");
> for i = 1 : length(d)
>  e (i , :) = strsplit ( d{i} );
> endfor

Others have focused on the file I/O, but the rest can be speeded up as well.
It is useful to do as follows:

profile clear;
profile on;
:
<your code>
:
profile off;
profshow

to find out where most of the time is spent.

OK, as regards your code:
for loops are usually killing for performance.
I haven't tried myself but instead of the for loop you can try vectorizing
with cellfun along the lines of;

e = cellfun (@strsplit, d)

(maybe "uni, 0" are required as 3rd and 4th arg to cellfun)

(It seems that cellfun() works the fastest when given handles to functions.)

Philip



--
Sent from: http://octave.1599824.n4.nabble.com/Octave-General-f1599825.html


Reply | Threaded
Open this post in threaded view
|

Fwd: slow code, optimisation advice needed

Ian McCallion
In reply to this post by tomkut
On 30 May 2018 at 15:40, tomkut <[hidden email]> wrote:
> Unfortunately, Ian, there are some heading rows that are filtered prior to
> the code here.

The developers of dlmread thought of that. see the documentation:

https://octave.sourceforge.io/octave/function/dlmread.html

Cheers... Ian


Reply | Threaded
Open this post in threaded view
|

Re: slow code, optimisation advice needed

tomkut
In reply to this post by PhilipNienhuis
thx PhilipNienhuis, I tried the timing function. The e = cellfun ( @strsplit,
(d(1:3)) ,'UniformOutput', false );  works but the structure of the
resulting cell array is different e.g.:
e =
{
  [1,1] =
  {
    [1,1] = 36.5333
    [1,2] = 41.8667
   }
  [2,1] =
  {
    [1,1] = 36.5333
    [1,2] = 41.8667
  }
}
as oppose to:
f =
{
  [1,1] = 36.5333
  [2,1] = 36.5333
  [1,2] = 41.8667
  [2,2] = 41.8667
  }
How can I correct this?



--
Sent from: http://octave.1599824.n4.nabble.com/Octave-General-f1599825.html


Reply | Threaded
Open this post in threaded view
|

Re: Fwd: slow code, optimisation advice needed

tomkut
This post was updated on .
In reply to this post by Ian McCallion
Initially had problems with the heading (imaginary numbers read), but found the right syntax, thanks Ian.
--
Sent from: http://octave.1599824.n4.nabble.com/Octave-General-f1599825.html


Reply | Threaded
Open this post in threaded view
|

Re: slow code, optimisation advice needed

PhilipNienhuis
In reply to this post by tomkut
tomkut wrote

> thx PhilipNienhuis, I tried the timing function. The e = cellfun (
> @strsplit,
> (d(1:3)) ,'UniformOutput', false );  works but the structure of the
> resulting cell array is different e.g.:
> e =
> {
>   [1,1] =
>   {
>     [1,1] = 36.5333
>     [1,2] = 41.8667
>    }
>   [2,1] =
>   {
>     [1,1] = 36.5333
>     [1,2] = 41.8667
>   }
> }
> as oppose to:
> f =
> {
>   [1,1] = 36.5333
>   [2,1] = 36.5333
>   [1,2] = 41.8667
>   [2,2] = 41.8667
>   }
> How can I correct this?

Try one of the cat, horzcat or vertcat method. You'd have to experiment a
bit to get it right.

Philip



--
Sent from: http://octave.1599824.n4.nabble.com/Octave-General-f1599825.html