Improving pkg.m

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Improving pkg.m

JuanPi
Hi,

It has been a while since we talked about improving pkg.m.

Shall we set a schedule for a sprint?

Who is joining?

--
JuanPi Carbajal
-----
"It is one thing not to be able to perform a certain feat, but quite
another to prove that it cannot be done." - Henry Ernest Dudeney
-----
http://ailab.ifi.uzh.ch/carbajal/

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://ad.doubleclick.net/clk;258768047;13503038;j?
http://info.appdynamics.com/FreeJavaPerformanceDownload.html
_______________________________________________
Octave-dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/octave-dev
Reply | Threaded
Open this post in threaded view
|

Re: Improving pkg.m

c.-2

On 23 Sep 2012, at 21:06, JuanPi wrote:

> Hi,
>
> It has been a while since we talked about improving pkg.m.
>
> Shall we set a schedule for a sprint?
>
> Who is joining?

I'm in.

We should have the description of the required improvements at

http://wiki.octave.org/Projects#Packaging

a bit more specific before the sprint.

c.

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://ad.doubleclick.net/clk;258768047;13503038;j?
http://info.appdynamics.com/FreeJavaPerformanceDownload.html
_______________________________________________
Octave-dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/octave-dev
Reply | Threaded
Open this post in threaded view
|

Re: Improving pkg.m

Claudio Freire
On Sun, Sep 23, 2012 at 5:06 PM, c. <[hidden email]> wrote:
> I'm in.
>
> We should have the description of the required improvements at
>
> http://wiki.octave.org/Projects#Packaging
>
> a bit more specific before the sprint.

I just added:
 make pkg able to supply extra configure and make flags, useful for
distributions, including -j for make

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://ad.doubleclick.net/clk;258768047;13503038;j?
http://info.appdynamics.com/FreeJavaPerformanceDownload.html
_______________________________________________
Octave-dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/octave-dev
Reply | Threaded
Open this post in threaded view
|

Re: Improving pkg.m

Paul Dreik-2


2012-09-23 22:33, Claudio Freire skrev:

> On Sun, Sep 23, 2012 at 5:06 PM, c. <[hidden email]> wrote:
>> I'm in.
>>
>> We should have the description of the required improvements at
>>
>> http://wiki.octave.org/Projects#Packaging
>>
>> a bit more specific before the sprint.
>
> I just added:
>  make pkg able to supply extra configure and make flags, useful for
> distributions, including -j for make
Just curios, are there any octave packages that benefit from parallel
compilation?
paul

>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://ad.doubleclick.net/clk;258768047;13503038;j?
> http://info.appdynamics.com/FreeJavaPerformanceDownload.html
> _______________________________________________
> Octave-dev mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/octave-dev
>

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://ad.doubleclick.net/clk;258768047;13503038;j?
http://info.appdynamics.com/FreeJavaPerformanceDownload.html
_______________________________________________
Octave-dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/octave-dev
Reply | Threaded
Open this post in threaded view
|

Re: Improving pkg.m

Juan Pablo Carbajal
On Sun, Sep 23, 2012 at 11:29 PM, Paul Dreik <[hidden email]> wrote:

>
>
> 2012-09-23 22:33, Claudio Freire skrev:
>> On Sun, Sep 23, 2012 at 5:06 PM, c. <[hidden email]> wrote:
>>> I'm in.
>>>
>>> We should have the description of the required improvements at
>>>
>>> http://wiki.octave.org/Projects#Packaging
>>>
>>> a bit more specific before the sprint.
>>
>> I just added:
>>  make pkg able to supply extra configure and make flags, useful for
>> distributions, including -j for make
> Just curios, are there any octave packages that benefit from parallel
> compilation?
> paul
>>
>> ------------------------------------------------------------------------------
>> Everyone hates slow websites. So do we.
>> Make your web apps faster with AppDynamics
>> Download AppDynamics Lite for free today:
>> http://ad.doubleclick.net/clk;258768047;13503038;j?
>> http://info.appdynamics.com/FreeJavaPerformanceDownload.html
>> _______________________________________________
>> Octave-dev mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/octave-dev
>>
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://ad.doubleclick.net/clk;258768047;13503038;j?
> http://info.appdynamics.com/FreeJavaPerformanceDownload.html
> _______________________________________________
> Octave-dev mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/octave-dev

I do not understand the question and it seem off-topic. Can you open
another thread to discuss that?

Thanks

--
M. Sc. Juan Pablo Carbajal
-----
PhD Student
University of Zürich
http://ailab.ifi.uzh.ch/carbajal/

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://ad.doubleclick.net/clk;258768047;13503038;j?
http://info.appdynamics.com/FreeJavaPerformanceDownload.html
_______________________________________________
Octave-dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/octave-dev
Reply | Threaded
Open this post in threaded view
|

Re: Improving pkg.m

Montgomery-Smith, Stephen
On 09/23/2012 05:20 PM, Juan Pablo Carbajal wrote:

> On Sun, Sep 23, 2012 at 11:29 PM, Paul Dreik <[hidden email]> wrote:
>>
>>
>> 2012-09-23 22:33, Claudio Freire skrev:
>>> On Sun, Sep 23, 2012 at 5:06 PM, c. <[hidden email]> wrote:
>>>> I'm in.
>>>>
>>>> We should have the description of the required improvements at
>>>>
>>>> http://wiki.octave.org/Projects#Packaging
>>>>
>>>> a bit more specific before the sprint.
>>>
>>> I just added:
>>>   make pkg able to supply extra configure and make flags, useful for
>>> distributions, including -j for make
>> Just curios, are there any octave packages that benefit from parallel
>> compilation?
>> paul
>>>
>>> ------------------------------------------------------------------------------
>>> Everyone hates slow websites. So do we.
>>> Make your web apps faster with AppDynamics
>>> Download AppDynamics Lite for free today:
>>> http://ad.doubleclick.net/clk;258768047;13503038;j?
>>> http://info.appdynamics.com/FreeJavaPerformanceDownload.html
>>> _______________________________________________
>>> Octave-dev mailing list
>>> [hidden email]
>>> https://lists.sourceforge.net/lists/listinfo/octave-dev
>>>
>>
>> ------------------------------------------------------------------------------
>> Everyone hates slow websites. So do we.
>> Make your web apps faster with AppDynamics
>> Download AppDynamics Lite for free today:
>> http://ad.doubleclick.net/clk;258768047;13503038;j?
>> http://info.appdynamics.com/FreeJavaPerformanceDownload.html
>> _______________________________________________
>> Octave-dev mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/octave-dev
>
> I do not understand the question and it seem off-topic. Can you open
> another thread to discuss that?

The question made sense to me, and wasn't off topic.  One person said it
would be nice if "-j" could be passed to make via pkg.m.  Another person
asked if there would be any advantage to doing this (since "-j" causes
make to do parallel builds).  Filling in the blanks, I assume he meant
that most packages build so fast that passing "-j" would be of very
little benefit.


------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://ad.doubleclick.net/clk;258768047;13503038;j?
http://info.appdynamics.com/FreeJavaPerformanceDownload.html
_______________________________________________
Octave-dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/octave-dev
Reply | Threaded
Open this post in threaded view
|

Re: Improving pkg.m

Juan Pablo Carbajal
On Mon, Sep 24, 2012 at 12:33 AM, Stephen Montgomery-Smith
<[hidden email]> wrote:

> On 09/23/2012 05:20 PM, Juan Pablo Carbajal wrote:
>> On Sun, Sep 23, 2012 at 11:29 PM, Paul Dreik <[hidden email]> wrote:
>>>
>>>
>>> 2012-09-23 22:33, Claudio Freire skrev:
>>>> On Sun, Sep 23, 2012 at 5:06 PM, c. <[hidden email]> wrote:
>>>>> I'm in.
>>>>>
>>>>> We should have the description of the required improvements at
>>>>>
>>>>> http://wiki.octave.org/Projects#Packaging
>>>>>
>>>>> a bit more specific before the sprint.
>>>>
>>>> I just added:
>>>>   make pkg able to supply extra configure and make flags, useful for
>>>> distributions, including -j for make
>>> Just curios, are there any octave packages that benefit from parallel
>>> compilation?
>>> paul
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Everyone hates slow websites. So do we.
>>>> Make your web apps faster with AppDynamics
>>>> Download AppDynamics Lite for free today:
>>>> http://ad.doubleclick.net/clk;258768047;13503038;j?
>>>> http://info.appdynamics.com/FreeJavaPerformanceDownload.html
>>>> _______________________________________________
>>>> Octave-dev mailing list
>>>> [hidden email]
>>>> https://lists.sourceforge.net/lists/listinfo/octave-dev
>>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Everyone hates slow websites. So do we.
>>> Make your web apps faster with AppDynamics
>>> Download AppDynamics Lite for free today:
>>> http://ad.doubleclick.net/clk;258768047;13503038;j?
>>> http://info.appdynamics.com/FreeJavaPerformanceDownload.html
>>> _______________________________________________
>>> Octave-dev mailing list
>>> [hidden email]
>>> https://lists.sourceforge.net/lists/listinfo/octave-dev
>>
>> I do not understand the question and it seem off-topic. Can you open
>> another thread to discuss that?
>
> The question made sense to me, and wasn't off topic.  One person said it
> would be nice if "-j" could be passed to make via pkg.m.  Another person
> asked if there would be any advantage to doing this (since "-j" causes
> make to do parallel builds).  Filling in the blanks, I assume he meant
> that most packages build so fast that passing "-j" would be of very
> little benefit.
>
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://ad.doubleclick.net/clk;258768047;13503038;j?
> http://info.appdynamics.com/FreeJavaPerformanceDownload.html
> _______________________________________________
> Octave-dev mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/octave-dev

oh! sorry then for my miss interpretation. Being able to pass "-j" to
pkg would be perfect!

@Paul: Sorry! If that was your question. Definitely, control for
example would compile much faster if we can use multiple
processors/cores.

--
M. Sc. Juan Pablo Carbajal
-----
PhD Student
University of Zürich
http://ailab.ifi.uzh.ch/carbajal/

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://ad.doubleclick.net/clk;258768047;13503038;j?
http://info.appdynamics.com/FreeJavaPerformanceDownload.html
_______________________________________________
Octave-dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/octave-dev
Reply | Threaded
Open this post in threaded view
|

Re: Improving pkg.m

Claudio Freire
In reply to this post by Paul Dreik-2
On Sun, Sep 23, 2012 at 6:29 PM, Paul Dreik <[hidden email]> wrote:
>> I just added:
>>  make pkg able to supply extra configure and make flags, useful for
>> distributions, including -j for make
> Just curios, are there any octave packages that benefit from parallel
> compilation?

Well, for openSUSE, we build all packages at once, and it takes a very
long time, like an hour and a half, for about 80 packages. That's an
average build time of a minute per package, big enough to make a
difference with parallel compilation.

I actually haven't tried though, convincing pkg.m to add extra flags
wasn't documented and seemed left out. Packaging would benefit from
being able to send config and make-time flags, besides the parallel
compilation aspect. Things like CFLAGS, for instance.

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://ad.doubleclick.net/clk;258768047;13503038;j?
http://info.appdynamics.com/FreeJavaPerformanceDownload.html
_______________________________________________
Octave-dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/octave-dev
Reply | Threaded
Open this post in threaded view
|

Re: Improving pkg.m

Paul Dreik-2
2012-09-24 01:59, Claudio Freire skrev:

> On Sun, Sep 23, 2012 at 6:29 PM, Paul Dreik <[hidden email]> wrote:
>>> I just added:
>>>  make pkg able to supply extra configure and make flags, useful for
>>> distributions, including -j for make
>> Just curios, are there any octave packages that benefit from parallel
>> compilation?
>
> Well, for openSUSE, we build all packages at once, and it takes a very
> long time, like an hour and a half, for about 80 packages. That's an
> average build time of a minute per package, big enough to make a
> difference with parallel compilation.
>
> I actually haven't tried though, convincing pkg.m to add extra flags
> wasn't documented and seemed left out. Packaging would benefit from
> being able to send config and make-time flags, besides the parallel
> compilation aspect. Things like CFLAGS, for instance.
>
I think this is good and I do not oppose it at all! My thought was that
make -j makes child processes of make run in parallel, which is
different from having several instances of make run in parallel.
That is why I asked if there is a package that improves from it - I
guess a package has to use multiple source files, where at least some
are nondependent so they can be compiled in parallel.

One could of course imagine a situation where a Makefile spanning
several packages is possible, made from the dependency information in
the packages. Invoking make at the root of the package tree would
certainly improve package generation if -j is used for that make
invokation. But that is maybe what you do for opensuse?

@Stephen:
no, I did not mean the benefit was negligible, I meant if -j would make
any difference at all because it requires the build to be possible to
parallelize.

@Juan Pablo:
You answered my question! I tried with -j4 and the compilation time
dropped certainly dropped, but only when the slicot library has been
built. Below, the "real" time output from the time command is used:
cd control/src
time make
72 seconds
make clean (to get the caches warm. note, make clean does not remove
slicotlibrary.a. Maybe it should?)
time make
18 seconds
make clean
time make -j4
12 seconds
make clean
rm slicotlibrary.a
time make -j4
63 seconds


Paul

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Octave-dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/octave-dev
Reply | Threaded
Open this post in threaded view
|

Re: Improving pkg.m

Claudio Freire
On Mon, Sep 24, 2012 at 3:55 AM, Paul Dreik <[hidden email]> wrote:

> time make
> 72 seconds
> make clean (to get the caches warm. note, make clean does not remove
> slicotlibrary.a. Maybe it should?)
> time make
> 18 seconds
> make clean
> time make -j4
> 12 seconds
> make clean
> rm slicotlibrary.a
> time make -j4
> 63 seconds

secs2d:

time make:
real    0m11.763s
user    0m10.929s
sys     0m0.679s

time make -j4:
real    0m5.897s
user    0m19.935s
sys     0m0.945s

(system is dual core with HT, so this is an example of fully
parallelizable build)

Anyway, like I said, the speed benefit is only secondary to being able
to specify proper CFLAGS.

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Octave-dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/octave-dev
Reply | Threaded
Open this post in threaded view
|

Re: Improving pkg.m

Juan Pablo Carbajal
On Mon, Sep 24, 2012 at 9:09 AM, Claudio Freire <[hidden email]> wrote:

> On Mon, Sep 24, 2012 at 3:55 AM, Paul Dreik <[hidden email]> wrote:
>> time make
>> 72 seconds
>> make clean (to get the caches warm. note, make clean does not remove
>> slicotlibrary.a. Maybe it should?)
>> time make
>> 18 seconds
>> make clean
>> time make -j4
>> 12 seconds
>> make clean
>> rm slicotlibrary.a
>> time make -j4
>> 63 seconds
>
> secs2d:
>
> time make:
> real    0m11.763s
> user    0m10.929s
> sys     0m0.679s
>
> time make -j4:
> real    0m5.897s
> user    0m19.935s
> sys     0m0.945s
>
> (system is dual core with HT, so this is an example of fully
> parallelizable build)
>
> Anyway, like I said, the speed benefit is only secondary to being able
> to specify proper CFLAGS.
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Octave-dev mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/octave-dev

Good, we should improve the options parsing in pkg.m. This is already
in the wiki.

Can we go back to the scheduling of a "pkg.m improvement code sprint".

Does anyone know a free/libre web-based event scheduling service?
(otherwise we use doodle.com)


--
M. Sc. Juan Pablo Carbajal
-----
PhD Student
University of Zürich
http://ailab.ifi.uzh.ch/carbajal/

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Octave-dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/octave-dev
Reply | Threaded
Open this post in threaded view
|

Re: Improving pkg.m

c.-2
In reply to this post by Paul Dreik-2

On 24 Sep 2012, at 08:55, Paul Dreik wrote:

> That is why I asked if there is a package that improves from it - I
> guess a package has to use multiple source files, where at least some
> are nondependent so they can be compiled in parallel.

openmpi_ext is a good example for this, it has many .cc files that can all be compiled independently.
c.
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Octave-dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/octave-dev
Reply | Threaded
Open this post in threaded view
|

Re: Improving pkg.m

Juan Pablo Carbajal
On Mon, Sep 24, 2012 at 9:50 AM, c. <[hidden email]> wrote:

>
> On 24 Sep 2012, at 08:55, Paul Dreik wrote:
>
>> That is why I asked if there is a package that improves from it - I
>> guess a package has to use multiple source files, where at least some
>> are nondependent so they can be compiled in parallel.
>
> openmpi_ext is a good example for this, it has many .cc files that can all be compiled independently.
> c.
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Octave-dev mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/octave-dev

I created a dedicated page for the effort.
http://wiki.octave.org/Code_sprint:_pkg.m

Some items need a better description. I did not copy the "more like
apt-get" item cause is too unspecific. If you know what was meant in
any of the item, maybe you can add some user-stories to its
description.

FLOSS Scheduling tool?

--
M. Sc. Juan Pablo Carbajal
-----
PhD Student
University of Zürich
http://ailab.ifi.uzh.ch/carbajal/

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Octave-dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/octave-dev