Speed of Ocave vs Matlab?

classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Speed of Ocave vs Matlab?

Michael Barnes
I just got asked this question at a computer vision conference. Does anyone have any benchmark examples I can quote?

Thanks

Michael Barnes
Reply | Threaded
Open this post in threaded view
|

Re: Speed of Ocave vs Matlab?

Markus Bergholz


On Mon, Sep 7, 2015 at 11:14 PM, Michael Barnes <[hidden email]> wrote:
I just got asked this question at a computer vision conference. Does anyone have any benchmark examples I can quote?

Thanks

Michael Barnes


function ret = e3(n)
    % calculating e, because matlab doesn't have it
    if (nargin == 0)
        n = 1000;
    end
    ret = cumsum([ 1 1./factorial(1:n)]);
    ret = ret(end);
end

Matlab: 0.00068 sec
Octave: 0.00045 sec


>> size(data)
ans =
        3401          15

>> tic, a = str2double(data); toc % this is matlab
Elapsed time is 1.999473 seconds.

octave:2> tic, a = str2double(data); toc % this is octave
Elapsed time is 0.122138 seconds.



It always depends on your code and data structure. When you're benchmarking loops, octave will lose.

PS: don't trust any benchmarks


--
icq: 167498924
XMPP|Jabber: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Speed of Ocave vs Matlab?

rocketsound
In reply to this post by Michael Barnes
In general: Any loop-heavy code is slow in Octave because it has no JIT compiler as Matlab does. I think any further statement to this topic requires more details about your desired application of Octave. My experience is: Some parts of the language are equally fast to Matlab (like vector calculations, inverse, stuff like this) and so are a lot of functions. But for example the function qp (comparable to quadprog) is terribly slow. My MPC controller calculations take like 10 seconds with Matlab but at least one whole day with Octave.
Reply | Threaded
Open this post in threaded view
|

Re: Speed of Ocave vs Matlab?

John Swensen-3
In reply to this post by Markus Bergholz

On Sep 7, 2015, at 3:03 PM, Markus Bergholz <[hidden email]> wrote:


It always depends on your code and data structure. When you're benchmarking loops, octave will lose.

PS: don't trust any benchmarks


--
icq: 167498924
XMPP|Jabber: [hidden email]

How much does the not-enabled-by-default JIT help in the loop department? Does it need to be able to JIT compile everything in the loop also? Or, can loop performance be improved just by turning JIT on regardless of the code contents of the loop?
Reply | Threaded
Open this post in threaded view
|

Re: Speed of Ocave vs Matlab?

Sebastian Schöps
In reply to this post by Michael Barnes
Michael Barnes wrote
I just got asked this question at a computer vision conference. Does anyone have any benchmark examples I can quote?
Yet another comment: an obvious reason why Octave is often as fast as Matlab is that both rely on the same numerical libraries (e.g. ARPACK, SuiteSparse etc). Consequently, benchmarking x=A\b would not make much sense.
Reply | Threaded
Open this post in threaded view
|

Re: Speed of Ocave vs Matlab?

Olaf Till-2
In reply to this post by rocketsound
On Mon, Sep 07, 2015 at 03:11:43PM -0700, rocketsound wrote:
> ... But for example the function qp (comparable to quadprog) is
> terribly slow. My MPC controller calculations take like 10 seconds with
> Matlab but at least one whole day with Octave.

This sounds as if there were several different algorithms involved,
where it is natural that not the same algorithms are automatically
used by Matlab and Octave, so that this is not just a general
comparison of Matlab and Octave, but of some different special
algorithms. But I don't understand what you are doing.

MPC? Do you optimize dynamic models?

Do you really mean qp? It doesn't evaluate a user function at
all... What are the sizes of your input arguments of qp? Is it really
the call to qp which lasts so long? Is qp called several times?

Olaf

--
public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Speed of Ocave vs Matlab?

Alexander Barth-3
In reply to this post by Michael Barnes

On Mon, Sep 7, 2015 at 11:14 PM, Michael Barnes <[hidden email]> wrote:
I just got asked this question at a computer vision conference. Does anyone have any benchmark examples I can quote?


It strongly depends on the type of benchmark. In my tests, octave turned out the be faster than Matlab:

http://www.geosci-model-dev.net/7/225/2014/gmd-7-225-2014.pdf
(table 4 on page 9, open access journal under creative commons).

The code needs to solve a sparse system defined by a partial differential equation. Large-parts of the code are vectorized and the only loops are essentially the number of dimensions (which is typically 2 or 3).

I was actually surprised by this results and I did not perform any optimization specific to octave. I only vectorized the code which is beneficial for octave and Matlab.

In the above mentioned paper, you will however also find some references where Matlab was faster than Octave.

The conclusion for me is: if performance is important, you need to vectorize the code as much as possible. This is not only necessary for octave but also for Matlab. If this is not possible, then I use a compiled language like Fortran.

I hope this helps,
Alex

Reply | Threaded
Open this post in threaded view
|

Re: Speed of Ocave vs Matlab?

Carlo de Falco-2
On 8 Sep 2015, at 10:01, Alexander Barth <[hidden email]> wrote:

>
> On Mon, Sep 7, 2015 at 11:14 PM, Michael Barnes <[hidden email]> wrote:
> I just got asked this question at a computer vision conference. Does anyone have any benchmark examples I can quote?
>
>
> It strongly depends on the type of benchmark. In my tests, octave turned out the be faster than Matlab:
>
> http://www.geosci-model-dev.net/7/225/2014/gmd-7-225-2014.pdf
> (table 4 on page 9, open access journal under creative commons).
>
> The code needs to solve a sparse system defined by a partial differential equation. Large-parts of the code are vectorized and the only loops are essentially the number of dimensions (which is typically 2 or 3).
>
> I was actually surprised by this results and I did not perform any optimization specific to octave. I only vectorized the code which is beneficial for octave and Matlab.
>
> In the above mentioned paper, you will however also find some references where Matlab was faster than Octave.
>
> The conclusion for me is: if performance is important, you need to vectorize the code as much as possible. This is not only necessary for octave but also for Matlab. If this is not possible, then I use a compiled language like Fortran.
>
> I hope this helps,
> Alex
>


Alexander,

I guess you still have all the scripts to run these tests again?
I would be curiuos to see what is the comparison between Octave 3.6.4 and 4.x,
as a few users reported significant slowdown in some aplications.

c.





Reply | Threaded
Open this post in threaded view
|

Re: Speed of Ocave vs Matlab?

rocketsound
In reply to this post by Olaf Till-2
For linear systems like a PT3 system and (positive definite) quadratic cost functions, standard MPC controllers (like quasi infinite horizon MPC) can always be written as a (quadratically constrained) quadratic programs thus I'm using qp/quadprog in order to find the optimizer that I want to apply in every MPC interation. I noticed that qp's speed is acceptable if the problem's dimension is not "too large" which of course is pretty sketchy to claim so let me give you more details: While writing this answer I run an script that calculates a zero-terminal constraint MPC which is numerically demanding because of the punctiform terminal constraint of the state vector. I profiled two runs with a different prediciton horizon: The first run had 40 optimization variables and the second run had 100 optimization variables. This is the result of the profiler:

40 optimization variables wrote
   #               Function Attr     Time (s)   Time (%)        Calls
---------------------------------------------------------------------
  83                blkdiag             7.751      21.19        14200
  85               cell2mat             5.892      16.11        14200
 153               __line__             3.648       9.98          251
 112                 __qp__             3.424       9.36           50
  84                cellfun             3.355       9.17       129063
 155            __go_line__             2.236       6.11          251
 123                    set             1.987       5.43         1272
  13               ishandle             1.313       3.59          521
   1 MPC_Exercise1_Solution             1.220       3.34            1
   9                    get             0.674       1.84         7521
   2                  clear             0.404       1.11          151
  88               num2cell             0.304       0.83        14254
  46                    all             0.221       0.60        57672
 140                __plt__             0.214       0.58          200
  37                  zeros             0.192       0.53        15004
  89                    cat             0.185       0.50        14200
  29               prefix !             0.149       0.41       120895
 125   __plt_get_axis_arg__             0.140       0.38          906
  87                   sort             0.138       0.38        14200
   6              binary ==             0.129       0.35        86151
100 optimization variables wrote
   #               Function Attr     Time (s)   Time (%)        Calls
---------------------------------------------------------------------
 121                 __qp__            35.160      34.83           50
  94                blkdiag            21.028      20.83        35200
  96               cell2mat            14.811      14.67        35200
  95                cellfun             5.888       5.83       318063
 158               __line__             3.730       3.70          251
   1 MPC_Exercise1_Solution             3.217       3.19            1
 160            __go_line__             2.772       2.75          251
 130                    set             2.071       2.05         1272
  52                  zeros             1.961       1.94        36004
  13               ishandle             1.305       1.29          530
  28               num2cell             0.734       0.73        35257
   9                    get             0.676       0.67         7531
  32                    all             0.543       0.54       141675
  99                    cat             0.482       0.48        35200
   2                  clear             0.424       0.42          151
  21               prefix !             0.377       0.37       288904
  98                   sort             0.366       0.36        35200
   6              binary ==             0.299       0.30       191160
  63                  false             0.293       0.29        74164
  58                   size             0.293       0.29       106169
As you can see, qp is the bottelneck here. Doubling the problem dimension creatly increases the simulation time. I added Matlab's results as an attachment to this reply (it is 1,206 s and 5,158 s).

40 optimization variables: profiler_40.png
100 optimization variables: profiler_100.png


For even more optimization variables and more complex constraints the overall simulation time gets worse.
Reply | Threaded
Open this post in threaded view
|

Re: Speed of Octave vs Matlab?

Olaf Till-2
In reply to this post by Olaf Till-2
Ok, thanks for the information. But, although this maybe an issue to
tackle, it is not the difference between 10 seconds and one whole day
you mentioned, which would rather be a blocker. Or have I understood
something wrong?

Olaf

(Hope this reply makes it to the correct position in the thread, I
lost the original mail and had to regenerate it and its Message-Id
from the archive.)

--
public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Speed of Ocave vs Matlab?

Alexander Barth-3
In reply to this post by Carlo de Falco-2
Dear Carlo,

Unfortunately I do not have octave 4 on this machine. For octave 3.8.1 the numbers were very similar to octave 3.6.4 (and even better for 3.8.1)

size  octave 3.6.4  octave 3.8.1 
100   0.415             0.2507   
200   1.808             1.2061   
300   5.155             3.7093   
400   8.104             6.5474   
500   13.905           11.7405   
600   24.906           23.7764   

(run-time in seconds, median of 10 realization)


On Tue, Sep 8, 2015 at 11:43 AM, Carlo De Falco <[hidden email]> wrote:
On 8 Sep 2015, at 10:01, Alexander Barth <[hidden email]> wrote:

>
> On Mon, Sep 7, 2015 at 11:14 PM, Michael Barnes <[hidden email]> wrote:
> I just got asked this question at a computer vision conference. Does anyone have any benchmark examples I can quote?
>
>
> It strongly depends on the type of benchmark. In my tests, octave turned out the be faster than Matlab:
>
> http://www.geosci-model-dev.net/7/225/2014/gmd-7-225-2014.pdf
> (table 4 on page 9, open access journal under creative commons).
>
> The code needs to solve a sparse system defined by a partial differential equation. Large-parts of the code are vectorized and the only loops are essentially the number of dimensions (which is typically 2 or 3).
>
> I was actually surprised by this results and I did not perform any optimization specific to octave. I only vectorized the code which is beneficial for octave and Matlab.
>
> In the above mentioned paper, you will however also find some references where Matlab was faster than Octave.
>
> The conclusion for me is: if performance is important, you need to vectorize the code as much as possible. This is not only necessary for octave but also for Matlab. If this is not possible, then I use a compiled language like Fortran.
>
> I hope this helps,
> Alex
>


Alexander,

I guess you still have all the scripts to run these tests again?
I would be curiuos to see what is the comparison between Octave 3.6.4 and 4.x,
as a few users reported significant slowdown in some aplications.

c.





Reply | Threaded
Open this post in threaded view
|

Re: Speed of Ocave vs Matlab?

chechu

Dear all,
If those scripts are mailed, we could run them on Octave 4.0 with jit and openblas enabled.

Yours sincerely Chechu

Reply | Threaded
Open this post in threaded view
|

Re: Speed of Octave vs Matlab?

rocketsound
In reply to this post by Olaf Till-2
Olaf Till-2 wrote
Ok, thanks for the information. But, although this maybe an issue to
tackle, it is not the difference between 10 seconds and one whole day
you mentioned, which would rather be a blocker. Or have I understood
something wrong?
I don't have much time until the end of the week, I will replay you in more detail.

The profiler results above are taken from another script because I had it at my fingertips. I'll try to find out the source of the long execution time I was talking above (might be because of the huge amount of constraints I was using) but the result is the same: it's qp that slows down the code. Looking at __qp__ it seems like it is implementing a quite simple solving algorithm so my prediction is it simply scales bad with problem dimension. Sadly, no more information is given about what kind of algorithm lies behind the code.

Anyway, and beyond this thread, thanks for the interessent and your work you put into the GSCO this year, the code is pretty valuable to me.
Reply | Threaded
Open this post in threaded view
|

Re: Speed of Ocave vs Matlab?

Alexander Barth-3
In reply to this post by chechu
Dear Chechu,
Thank you for your proposition! I attach here the benchmark script.
But you will also need to install the octave forge package divand (which does not have any extra dependencies).

Do no hesitate to ask if something is unclear.

Note that the Xeon E5420 is a quite old CPU (launched in 2007). Even my
laptop i3 (3120M) CPU is 2 times faster.

Regards,
Alex


test_2dvar_benchmark.m (3K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Speed of Octave vs Matlab?

Olaf Till-2
In reply to this post by rocketsound
On Tue, Sep 08, 2015 at 11:10:41AM -0700, rocketsound wrote:
> The profiler results above are taken from another script because I had it at
> my fingertips. I'll try to find out the source of the long execution time I
> was talking above (might be because of the huge amount of constraints I was
> using) but the result is the same: it's qp that slows down the code. Looking
> at __qp__ it seems like it is implementing a quite simple solving algorithm
> so my prediction is it simply scales bad with problem dimension. Sadly, no
> more information is given about what kind of algorithm lies behind the code.

An example for the very long execution time of __qp__ would indeed be
good. I'll probably check if an alternative to __qp__ can be applied
to quadprog in optim.

> Anyway, and beyond this thread, thanks for the interessent and your work you
> put into the GSCO this year, the code is pretty valuable to me.

Thanks a lot, but I mostly pass this on to Asma Afzal, who did most of
the work.

Olaf

--
public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Speed of Ocave vs Matlab?

Jordi Gutiérrez Hermoso-2
In reply to this post by John Swensen-3
On Mon, 2015-09-07 at 21:21 -0700, John Swensen wrote:
> How much does the not-enabled-by-default JIT help in the loop
> department?

It's a white elephant. It doesn't help at all except in very trivial
cases.

- Jordi G. H.



Reply | Threaded
Open this post in threaded view
|

Re: Speed of Ocave vs Matlab?

Michael Godfrey
This seems to be true. But, it should be added that the implementation is
very incomplete, and does not work on recent versions of llvm. It is
likely that it could be made to work better. Or, does anyone have a better
idea for a JIT?

On 09/08/2015 10:15 PM, Jordi Gutiérrez Hermoso wrote:
On Mon, 2015-09-07 at 21:21 -0700, John Swensen wrote:
> How much does the not-enabled-by-default JIT help in the loop
> department?
It's a white elephant. It doesn't help at all except in very trivial
cases.

- Jordi G. H.




Reply | Threaded
Open this post in threaded view
|

Re: Speed of Ocave vs Matlab?

Richard Crozier
On 08/09/15 22:26, Michael Godfrey wrote:
> This seems to be true. But, it should be added that the implementation is
> very incomplete, and does not work on recent versions of llvm. It is
> likely that it could be made to work better. Or, does anyone have a better
> idea for a JIT?
>

There's been a talk on the bug tracker of using the experimental gcc-jit:

https://savannah.gnu.org/patch/index.php?8395

Since Octave mostly uses gcc to compile this would seem like an
interesting direction to go. The author of gcc-jit also seems keen to
disseminate his work so might help Octave develop this.

Richard

--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.


Reply | Threaded
Open this post in threaded view
|

Re: Speed of Ocave vs Matlab?

Michael Godfrey
This looks promising. libgccjit and libgccjit-devel are available in the current release of Fedora.
After a very quick look, the level of support looks useful (as compared to llvm). Since the llvm jit
is now really broken it might make sense to integrate the libgccjit code as the optional jit enable
code, if there is code that actually works for some cases.

The libjit folks do mention that the API is nor yet stable, but it is not likely to be worse than
llvm, where they regularly change the API and the locations of all the files that need to be
included...

Obviously, someone with more detailed knowledge of the choices should help decide.
Michael

On 09/09/2015 08:09 AM, Richard Crozier wrote:
On 08/09/15 22:26, Michael Godfrey wrote:
This seems to be true. But, it should be added that the implementation is
very incomplete, and does not work on recent versions of llvm. It is
likely that it could be made to work better. Or, does anyone have a better
idea for a JIT?


There's been a talk on the bug tracker of using the experimental gcc-jit:

https://savannah.gnu.org/patch/index.php?8395

Since Octave mostly uses gcc to compile this would seem like an interesting direction to go. The author of gcc-jit also seems keen to disseminate his work so might help Octave develop this.

Richard

-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

Reply | Threaded
Open this post in threaded view
|

Re: Speed of Ocave vs Matlab?

chechu
In reply to this post by Alexander Barth-3
Good morning, it has taken a while but we have tested the given script on the very same computer with the last stable releases of octave. This way it seems a fair comparative.

Host computer details:

GNU/Debian 8.0 Jessie amd64
Intel Core i5-4200U - Portable - 2 cores - 4 threads with hyperthreading
8Gbytes RAM
Openblas 0.2.14
LLVM 3.3

ALL tests with octave/libraries single-threaded

---- No big differences ----


Octave 3.6.4


Median results
size   100 time     0.1595
size   200 time     0.6576
size   300 time     1.9180
size   400 time     3.1008
size   500 time     5.3241
size   600 time     9.7821


Octave 3.8.2 - JIT Enabled


Median results
size   100 time     0.1710
size   200 time     0.6614
size   300 time     1.8991
size   400 time     3.0877
size   500 time     5.3217
size   600 time     9.8336


Octave 3.8.2 - JIT Disabled


Median results
size   100 time     0.1641
size   200 time     0.6634
size   300 time     1.9230
size   400 time     3.1107
size   500 time     5.2869
size   600 time     9.8104


Octave 4.0.0 - JIT Enabled


Median results
size   100 time     0.1849
size   200 time     0.6627
size   300 time     1.9341
size   400 time     3.0775
size   500 time     5.3132
size   600 time     9.8861


Octave 4.0.0 - JIT Disabled


Median results
size   100 time     0.1773
size   200 time     0.6737
size   300 time     1.9490
size   400 time     3.0896
size   500 time     5.3405
size   600 time     9.7848



Thanks, Chechu
12