# octave benchmark test

16 messages
Open this post in threaded view
|
Report Content as Inappropriate

## octave benchmark test

 I am posting the results of this Sciviews benchmark test on the help list to hit the widest possible audience.  John Eaton and the other contributors to octave and octave-forge have my applause for the effort that they have put in over the last couple of months to extend and speed octave up. I also wish to encourage you to go the whole hog and update to ATLAS, FFW, octave-2.1.56(when it becomes the recommended version) and octave-forge (in that order).  On an Athlon 1700 RH9 machine with only 256Mbyte memory (of which more in a moment), the Sciviews benchmark tests  (see http://www.sciviews.org/other/benchmark.htm  ) gave:   Octave Benchmark 2   ================== Number of times each test is run__________________________: 3                                                                                 I. Matrix calculation                                  octave-2.1.56 (Matlab6.5)   --------------------- Creation, transp., deformation of a 1500x1500 matrix (sec): 1.117       (0.6733) 800x800 normal distributed random matrix ^1000______ (sec): 0.1454     (0.2846) Sorting of 2,000,000 random values__________________ (sec): 1.696       (0.9027) 700x700 cross-product matrix (b = a' * a)___________ (sec): 0.4062     (0.5264) Linear regression over a 600x600 matrix (c = a \ b') (sec): 0.247       (0.2885)                  ------------------------------------------------------                Trimmed geom. mean (2 extremes eliminated): 0.4822                                                                                 II. Matrix functions   -------------------- FFT over 800,000 random values______________________ (sec): 0.3724     (0.5914) Eigenvalues of a 320x320 random matrix______________ (sec): 0.9604     (1.0192) Determinant of a 650x650 random matrix______________ (sec): 0.2894     (0.2450) Cholesky decomposition of a 900x900 matrix__________ (sec): 0.3212     (0.3306) Inverse of a 400x400 random matrix__________________ (sec): 0.1626     (0.1957)                  ------------------------------------------------------                Trimmed geom. mean (2 extremes eliminated): 0.3259                                                                                 III. Programmation   ------------------ 750,000 Fibonacci numbers calculation (vector calc)_ (sec): 0.6075     (0.9792) Creation of a 2250x2250 Hilbert matrix (matrix calc) (sec): 1.137****   (1.1501) Grand common divisors of 70,000 pairs (recursion)___ (sec): 0.6918     (0.4376) Creation of a 220x220 Toeplitz matrix (loops)_______ (sec): 1.449       (0.0021) Escoufier's method on a 37x37 matrix (mixed)________ (sec): 2.2         (1.1663)                  ------------------------------------------------------                Trimmed geom. mean (2 extremes eliminated): 1.045                                                                                                                                                             Total time for all 15 tests_________________________ (sec): 11.8       (8.7927) Overall mean (sum of I, II and III trimmed means/3)_ (sec): 0.5475                      --- End of test --- **** The Hilbert matrix test uses a lot of memory.  With one memory bank AWOL, 256Mbyte is not enough.  It took 7-8 repeats to allow the system to clean up the memory enough that no swapping occurred during the test. It will be noted that most of the difference now lies in the last two tests, where Matlab must either be employing automatic vectorisation or an on the fly compiler, when it can.  This seems to be where Matworks have been concentrating their effort since the Sciviews tests were posted). Octave, with all the trimmings, now has roughly the performance of Matlab6.0. Vectorising the Toeplitz test, reduces its execution time in octave to 0.018s.  TAKE NOTE! ------------------------------------------------------------- Octave is freely available under the terms of the GNU GPL. Octave's home on the web:  http://www.octave.orgHow to fund new projects:  http://www.octave.org/funding.htmlSubscription information:  http://www.octave.org/archive.html-------------------------------------------------------------
Open this post in threaded view
|
Report Content as Inappropriate

## Re: octave benchmark test

 Daprès Paul Thomas <[hidden email]> (le 07/03/2004): > Sorting of 2,000,000 random values__________________ (sec): 1.696       > (0.9027) This slowup with octave is also a choice. The sort code in octave-forge uses the mergesort code from python, while matlab uses a quicksort algorithm. The benchmark above is reversed if partially ordered lists are used, as for instance often happen in interpolation codes. So octaves loss in the above benchmark shows more about the choices of the person writing the benchmark than the octaves relative performance. Benchmarks are all figments of someones deranged imagination... D. -- David Bateman                                [hidden email] Motorola CRM                                 +33 1 69 35 48 04 (Ph) Parc Les Algorithmes, Commune de St Aubin    +33 1 69 35 77 01 (Fax) 91193 Gif-Sur-Yvette FRANCE The information contained in this communication has been classified as: [x] General Business Information [ ] Motorola Internal Use Only [ ] Motorola Confidential Proprietary ------------------------------------------------------------- Octave is freely available under the terms of the GNU GPL. Octave's home on the web:  http://www.octave.orgHow to fund new projects:  http://www.octave.org/funding.htmlSubscription information:  http://www.octave.org/archive.html-------------------------------------------------------------
Open this post in threaded view
|
Report Content as Inappropriate

## Re: octave benchmark test

 I agree entirely, where details  (ie. less factor of 1.5-2 either way) are concerned.  However, you and others gained factors of 5 or more that have made the difference that I find impressive.  Whether deranged or not....... Paul T David Bateman wrote: >Daprès Paul Thomas <[hidden email]> (le 07/03/2004): >   > >>Sorting of 2,000,000 random values__________________ (sec): 1.696       >>(0.9027) >>     >> > >This slowup with octave is also a choice. The sort code in octave-forge >uses the mergesort code from python, while matlab uses a quicksort >algorithm. The benchmark above is reversed if partially ordered lists >are used, as for instance often happen in interpolation codes. So octaves >loss in the above benchmark shows more about the choices of the person >writing the benchmark than the octaves relative performance. Benchmarks >are all figments of someones deranged imagination... > >D. > >   > ------------------------------------------------------------- Octave is freely available under the terms of the GNU GPL. Octave's home on the web:  http://www.octave.orgHow to fund new projects:  http://www.octave.org/funding.htmlSubscription information:  http://www.octave.org/archive.html-------------------------------------------------------------
Open this post in threaded view
|
Report Content as Inappropriate

## Re: octave benchmark test

Open this post in threaded view
|
Report Content as Inappropriate

## Re: octave benchmark test

 Since we are speaking about benchmarks, there is an old benchmark of mine at , and the companion . Both are outdated, that is, they list results for machines that are now obsolete (1997).  At the time, I stopped collecting results because of some inconcistencies caused by compilation options which were not clear. Anyway, it would be easy to update that bechmark and use it for a quick assessment of how Octave performs on a given box.  Probably, because of the new Octave features, the set of tests should be updated, which is fairly easy. So, if anyone feels like upgrading this, I will put it under the GPL. -- Francesco Potortì (ricercatore)        Voice: +39 050 315 3058 (op.2111) ISTI - Area della ricerca CNR          Fax:   +39 050 313 8091 via G. Moruzzi 1, I-56124 Pisa         Email: [hidden email] Web: http://fly.isti.cnr.it/           Key:   fly.isti.cnr.it/public.key ------------------------------------------------------------- Octave is freely available under the terms of the GNU GPL. Octave's home on the web:  http://www.octave.orgHow to fund new projects:  http://www.octave.org/funding.htmlSubscription information:  http://www.octave.org/archive.html-------------------------------------------------------------
Open this post in threaded view
|
Report Content as Inappropriate

## Re: octave benchmark test

 In reply to this post by Michael Martin-8 On Mar 8, 2004, at 12:58 PM, Michael Martin wrote: > Now some of the differences are quite likely due to algorithmic > differences. The difference in sorting is certainly likely > algorithmic, though given that my other numbers are slightly better > than the times in Paul Thomas' post, I have to wonder why my sort is > quite so much worse than his. He is using David Bateman's sort from octave-forge (http://octave.sf.net). It is reported to have better performance on partially ordered lists, but worse on random data compared to matlab. Hmmm... I wonder if it is worthwhile to run through the data once to see how ordered it is, then do quicksort or merge sort as needed? Also, somebody with access to matlab might want to see if "[y,idx]=sort(x)" is any slower than "y=sort(x)".  Maybe they have a fast stable sort algorithm that we don't know about. Paul Kienzle [hidden email] ------------------------------------------------------------- Octave is freely available under the terms of the GNU GPL. Octave's home on the web:  http://www.octave.orgHow to fund new projects:  http://www.octave.org/funding.htmlSubscription information:  http://www.octave.org/archive.html-------------------------------------------------------------
Open this post in threaded view
|
Report Content as Inappropriate

## Re: octave benchmark test

 Paul Kienzle wrote: > It is reported to have better performance on partially ordered lists, > but worse on random data compared to matlab. Well, may be I misunderstood what does "partially ordered means", but here is what I got. (Matlab 6.5):  >> x1=rand(3000); % random  >> x2=[1:3000]; % ordered  >> x3=repmat(x2,3000,1); %  >> x=x3+2*x1; % partially ordered ?  >> tic ; sort(x1) ; toc elapsed_time =      1.4351  >> tic ; sort(x3) ; toc elapsed_time =      0.8567  >> tic ; sort(x) ; toc elapsed_time =      1.4285  >> x=x3+1.1*x1;  >>  tic ; sort(x) ; toc elapsed_time =      1.4355  >>  tic ; [y,idx]=sort(x) ; toc elapsed_time =      2.2073  >>  tic ; [y,idx]=sort(x) ; toc   % So we will not need to allocate y and idx elapsed_time =      2.0607 (Octave 2.1.56): octave:1> x1=rand(3000); octave:2> x2=[1:3000]; octave:3> x3=repmat(x2,3000,1); octave:4> x=x3+2*x1; octave:5> tic ; sort(x) ; toc ans = 2.1968 octave:6> tic ; sort(x1) ; toc ans = 1.9328 octave:7> tic ; sort(x3) ; toc ans = 0.58155 octave:8> tic ; [y,idx]= sort(x) ; toc ans = 3.5003 octave:9> tic ; [y,idx]= sort(x) ; toc ans = 3.4638 > Paul Kienzle > [hidden email] > Dmitri. ------------------------------------------------------------- Octave is freely available under the terms of the GNU GPL. Octave's home on the web:  http://www.octave.orgHow to fund new projects:  http://www.octave.org/funding.htmlSubscription information:  http://www.octave.org/archive.html-------------------------------------------------------------
Open this post in threaded view
|
Report Content as Inappropriate

## Re: octave benchmark test

 On Mar 8, 2004, at 7:17 PM, Dmitri A. Sergatskov wrote: > Paul Kienzle wrote: > >> It is reported to have better performance on partially ordered lists, >> but worse on random data compared to matlab. > > Well, may be I misunderstood what does "partially ordered means", but > here is > what I got. Thanks.  I don't know the details of the algorithm, but I believe it is a merge sort variant.    These tend to do better if you start with a small number of long 'runs' rather than a large number of short 'runs', as in the following: octave:42> w=[1:1000000]; octave:43> x=repmat([1:500000],1,2); octave:44> y=repmat([1:100000],1,10); octave:45> z=rand(1000000,1); octave:46> tic; sort(w); toc ans = 0.60063 octave:47> tic; sort(x); toc ans = 0.80005 octave:48> tic; sort(y); toc ans = 1.0342 octave:49> tic; sort(z); toc ans = 1.4839 x is an imporant case for table lookups. It would be nice if the random case were 1x rather 1.5x matlab.  Still,  1.5x is loads better than the 7x it used to be. Paul Kienzle [hidden email] ------------------------------------------------------------- Octave is freely available under the terms of the GNU GPL. Octave's home on the web:  http://www.octave.orgHow to fund new projects:  http://www.octave.org/funding.htmlSubscription information:  http://www.octave.org/archive.html-------------------------------------------------------------
Open this post in threaded view
|
Report Content as Inappropriate

## Re: octave benchmark test

Open this post in threaded view
|
Report Content as Inappropriate

## Re: octave benchmark test

 In reply to this post by Dmitri A. Sergatskov-2 Dmitri, Please find attached my test code for the speed of the sort code. This is based on a simialr benchmark within Python itself. Code is called like octave:1> sort_time = testsort('sort'); Test of the sorting function (sort)     \  N  |1.00e+02  1.00e+03  1.00e+04  1.00e+05  1.00e+06   Test \    | *sort     |3.82e-05  5.26e-04  6.41e-03  8.65e-02  1.18e+00   \sort     |1.15e-05  1.12e-04  1.13e-03  1.75e-02  2.08e-01   /sort     |1.10e-05  1.07e-04  1.09e-03  1.49e-02  1.69e-01   3sort     |2.06e-05  1.26e-04  1.19e-03  1.79e-02  2.22e-01   +sort     |1.69e-05  1.19e-04  1.14e-03  1.66e-02  1.99e-01   =sort     |1.09e-05  1.06e-04  1.11e-03  1.52e-02  1.67e-01    Normalized times against Matlab R12 on an IBM T23     \  N  |1.00e+02  1.00e+03  1.00e+04  1.00e+05  1.00e+06   Test \    | *sort     |2.25e+00  2.30e+00  2.35e+00  2.40e+00  2.31e+00   \sort     |9.94e-01  8.17e-01  6.47e-01  7.78e-01  6.33e-01   /sort     |1.03e+00  8.33e-01  6.59e-01  6.75e-01  5.49e-01   3sort     |1.88e+00  9.63e-01  7.28e-01  8.20e-01  7.15e-01   +sort     |1.12e+00  6.38e-01  5.17e-01  5.43e-01  4.27e-01   =sort     |8.97e-01  6.76e-01  5.22e-01  5.64e-01  4.43e-01   octave:2> sort_time_index = testsort('sort, [], 1); Test of the sorting function (sort) with indexing     \  N  |1.00e+02  1.00e+03  1.00e+04  1.00e+05  1.00e+06   Test \    | *sort     |6.39e-05  8.74e-04  1.15e-02  1.80e-01  2.55e+00   \sort     |2.50e-05  2.54e-04  2.66e-03  3.95e-02  4.70e-01   /sort     |2.46e-05  2.43e-04  2.51e-03  3.33e-02  4.03e-01   3sort     |4.03e-05  2.73e-04  2.63e-03  3.66e-02  4.30e-01   +sort     |3.34e-05  2.61e-04  2.56e-03  3.52e-02  4.20e-01   =sort     |2.49e-05  2.49e-04  2.54e-03  3.40e-02  4.07e-01    Normalized times against Matlab R12 on an IBM T23     \  N  |1.00e+02  1.00e+03  1.00e+04  1.00e+05  1.00e+06   Test \    | *sort     |2.69e+00  2.89e+00  3.09e+00  3.12e+00  3.04e+00   \sort     |1.54e+00  1.37e+00  1.19e+00  1.24e+00  1.14e+00   /sort     |1.75e+00  1.50e+00  1.25e+00  1.18e+00  1.11e+00   3sort     |2.67e+00  1.68e+00  1.31e+00  1.30e+00  1.19e+00   +sort     |1.59e+00  1.08e+00  8.78e-01  8.42e-01  7.50e-01   =sort     |1.28e+00  9.49e-01  6.93e-01  6.62e-01  6.07e-01   Note the times calibrated against Matlab R12 are only valid on my machine. D. According to Dmitri A. Sergatskov <[hidden email]> (on 03/09/04): > Paul Kienzle wrote: > > >It is reported to have better performance on partially ordered lists, > >but worse on random data compared to matlab. > > Well, may be I misunderstood what does "partially ordered means", but here > is > what I got. > (Matlab 6.5): > > >> x1=rand(3000); % random > >> x2=[1:3000]; % ordered > >> x3=repmat(x2,3000,1); % > >> x=x3+2*x1; % partially ordered ? > > > > >> tic ; sort(x1) ; toc > > elapsed_time = > >     1.4351 > > >> tic ; sort(x3) ; toc > > elapsed_time = > >     0.8567 > > >> tic ; sort(x) ; toc > > elapsed_time = > >     1.4285 > > >> x=x3+1.1*x1; > >>  tic ; sort(x) ; toc > > elapsed_time = > >     1.4355 > > >>  tic ; [y,idx]=sort(x) ; toc > > elapsed_time = > >     2.2073 > > >>  tic ; [y,idx]=sort(x) ; toc   % So we will not need to allocate y > and idx > > elapsed_time = > >     2.0607 > > > (Octave 2.1.56): > > octave:1> x1=rand(3000); > octave:2> x2=[1:3000]; > octave:3> x3=repmat(x2,3000,1); > octave:4> x=x3+2*x1; > octave:5> tic ; sort(x) ; toc > ans = 2.1968 > octave:6> tic ; sort(x1) ; toc > ans = 1.9328 > octave:7> tic ; sort(x3) ; toc > ans = 0.58155 > octave:8> tic ; [y,idx]= sort(x) ; toc > ans = 3.5003 > octave:9> tic ; [y,idx]= sort(x) ; toc > ans = 3.4638 > > > >Paul Kienzle > >[hidden email] > > > > Dmitri. > > > > ------------------------------------------------------------- > Octave is freely available under the terms of the GNU GPL. > > Octave's home on the web:  http://www.octave.org> How to fund new projects:  http://www.octave.org/funding.html> Subscription information:  http://www.octave.org/archive.html> ------------------------------------------------------------- -- David Bateman                                [hidden email] Motorola CRM                                 +33 1 69 35 48 04 (Ph) Parc Les Algorithmes, Commune de St Aubin    +33 1 69 35 77 01 (Fax) 91193 Gif-Sur-Yvette FRANCE The information contained in this communication has been classified as: [x] General Business Information [ ] Motorola Internal Use Only [ ] Motorola Confidential Proprietary ------------------------------------------------------------- Octave is freely available under the terms of the GNU GPL. Octave's home on the web:  http://www.octave.orgHow to fund new projects:  http://www.octave.org/funding.htmlSubscription information:  http://www.octave.org/archive.html-------------------------------------------------------------
Open this post in threaded view
|
Report Content as Inappropriate

## Re: octave benchmark test

Open this post in threaded view
|
Report Content as Inappropriate

## Re: octave benchmark test

Open this post in threaded view
|
Report Content as Inappropriate

## Re: octave benchmark test

 In reply to this post by David Bateman-3 > > Vectorize code wherever possible!!!!! Octave doesn't have JIT which is > where Matlab wins on this case. Can someone please tell me what JIT is, and explain why Octave does not have/could not do JIT, if Octave has/does almost everything else that Matlab has/does? Henry ------------------------------------------------------------- Octave is freely available under the terms of the GNU GPL. Octave's home on the web:  http://www.octave.orgHow to fund new projects:  http://www.octave.org/funding.htmlSubscription information:  http://www.octave.org/archive.html-------------------------------------------------------------
Open this post in threaded view
|
Report Content as Inappropriate

## Re: octave benchmark test

 On Mar 9, 2004, at 5:50 PM, Henry F. Mollet wrote: > >> >> Vectorize code wherever possible!!!!! Octave doesn't have JIT which is >> where Matlab wins on this case. > > Can someone please tell me what JIT is, and explain why Octave does not > have/could not do JIT, if Octave has/does almost everything else that > Matlab > has/does? Just In Time compiler. For best effect, you want to turn:         function total=sum(x)                 total=0; for i=1:n, total+=x(i); end into something like:         double total = 0;         int i; for (i=0; i < n; i++) total += x[i];         return total; If that looks easy, consider that x may be real or complex or a sparse matrix or a galois field or a indefinite precision number, ...  And consider that the size of x may be less than n.  And consider that x may be a function (matlab syntax does not distinguish between functions and array indexing), which returns a different value type for each i.  You don't know until you call the function what the type of x is. The function case is particularly tricky since there are no guarantees.  For example, I can override the builtin sin function so that it returns strings.  Or the function that I'm calling could replace a builtin function during the loop. Yes you can build in checks to make sure the results are as expected.  If you are not careful, though, you will end up rebuilding the octave interpreter, and it won't be any faster. We could probably hack in some simple cases which would make the benchmarks look good.  Some of them would even be practical.  I'm sure if you poke at the matlab interpreter enough you will find places where their optimization doesn't result in any speed up.  Try for example putting an eval statement in the loop (after eval, all bets are off). Paul Kienzle [hidden email] ------------------------------------------------------------- Octave is freely available under the terms of the GNU GPL. Octave's home on the web:  http://www.octave.orgHow to fund new projects:  http://www.octave.org/funding.htmlSubscription information:  http://www.octave.org/archive.html-------------------------------------------------------------
Open this post in threaded view
|
Report Content as Inappropriate

## Re: octave benchmark test

 And for those of you that live in North America, here is an opportunity that I found, whilst researching JIT a couple of weeks back: Work on the Just In Time (JIT) Compiler Team to improve MATLAB performance. JIT Compiler Team Member 10/14/2003 Natick, MA The JIT (Just In Time compiler) team is a small group within The MathWorks charged with improving the performance of MATLAB to near FORTRAN / C levels. Because MATLAB is a typeless, interactive language, this effort involves technical challenges rarely seen in other languages. We are looking for another team member who is experienced, creative, can work effectively independently as well as in a group, and is willing to expand the state of the art while delivering value to tens of thousands of customers. For the reasons that Paul describes below, "technical challenge" is an understatement.   I prefer the Japanese phrase - chotto muri desu-ka > "it's slightly impossible" Paul T Paul Kienzle wrote: > > If that looks easy, consider that x may be real or complex or > a sparse matrix or a galois field or a indefinite precision > number, ...  And consider that the size of x may be less than > n.  And consider that x may be a function (matlab syntax does > not distinguish between functions and array indexing), which > returns a different value type for each i.  You don't know until > you call the function what the type of x is. > ------------------------------------------------------------- Octave is freely available under the terms of the GNU GPL. Octave's home on the web:  http://www.octave.orgHow to fund new projects:  http://www.octave.org/funding.htmlSubscription information:  http://www.octave.org/archive.html-------------------------------------------------------------