octave-mpi questions

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

octave-mpi questions

John W. Eaton-6
On 13-Feb-2003, John W. Eaton <[hidden email]> wrote:

| On  8-Feb-2003, Andy Jacobson <[hidden email]> wrote:
|
| |         I recently received a request from Martin Siegert to help get
| |         the MPI patches working with octave-2.1.44.  (There appear to
| |         be some issues with changes in the standard C++ library, but I
| |         haven't looked into the problem in much detail yet).  I
| |         vaguely recall that you and I recently had an email exchange
| |         in which you mentioned that you had looked into incorporating
| |         some MPI capability into the standard distribution.  Not
| |         wanting to duplicate efforts, I wonder whether you have
| |         undertaken such a thing, or if Martin and I should do so.
|
| Sorry for the delay.  I stalled on this because after I looked at the
| code, I started thinking that it would be better to have thin wrapper
| for the MPI library, then implement higher level functions in
| M-files.  But that's going to be a lot of work, and I don't understand
| the MPI library as I don't use it (yet).  So I suppose it would be
| best do go ahead and include the functions that you have written with
| a note that what we probably really want is a more complete interface
| to MPI.  So, let me take another look at it.  Probably it will not be
| hard to get it in to the CVS archive quickly.

OK, I have adapted the patches for the current Octave sources.  I
haven't checked the changes in yet because I have a few questions.

It seems to me that it would be best for this code to be loaded
dynamically, and have the mpi library linked only to the dynamically
loaded .oct file so that users who don't want MPI don't have to pay
for it (at all) and so we don't have to have two different binaries.

I can see a couple of ways to do this.

One is to add a --mpi option for Octave which will cause Octave do do
the inits (but by using feval() to call .oct files so that the MPI
library doesn't have to be linked to the main Octave binary).  You
could run Octave with a command like

  mpirun OPTIONS octave --mpi

to start multiple Octave processes on one or more systems and give
yourself a prompt for the master process (or just run a script with
the master).  This seems fairly straightforward, but still requires
Octave to be modified to handle MPI.

Another way to add the MPI functions to Octave would be to not modify
Octave's startup procedure at all, but require the user to call some
kind of mpi_start function that does the initialization and also
includes the functionality of mpirun (if that is possible -- I suppose
we will need it to somehow tell the other processes that they should
perform the MPI startup steps too).  This startup function can also
call atexit ("mpi_quit") (or similar) to ensure that MPI is stopped
properly when Octave exits.  The user can also call mpi_quit without
exiting the main Octave process, then when mpi_quit is called by
atexit it will do nothing.  The problems I see with this approach are
the warnings (which seem strange to me) about not doing much before
calling MPI_Init and passing pointers to the actual argv/argc from the
C main.  (Surely this is not a requirement, because there are a number
of add-on packages for Matlab that use MPI and are implemented entirly
in .m and mex files.)

I like the idea of having the flexibility of starting and stopping MPI
processing on the fly, but I don't know whether that is practical.

Comments?

Thanks

jwe


Reply | Threaded
Open this post in threaded view
|

Re: octave-mpi questions

JD Cole-2
>
>
>OK, I have adapted the patches for the current Octave sources.  I
>haven't checked the changes in yet because I have a few questions.
>  
>
Before you commit those changes, you may want to see if the .oct-file,
MPI interface I posted a while back will fit Mr. Siegart's needs. I have
attached them to this e-mail. I had discussed these with Paul as a
possible addition to Octave-Forge. [See attached tar file] These are
mearely wrappers to the basic MPI API with the added convenience of
sending any variable type rather than sending a void* data block.
(Thanks to Andy's original work on octave-mpi.) Of course, these
routines could be extended to other functions in the MPI API. (They
function with existing octave binaries.)

>It seems to me that it would be best for this code to be loaded
>dynamically, and have the mpi library linked only to the dynamically
>loaded .oct file so that users who don't want MPI don't have to pay
>for it (at all) and so we don't have to have two different binaries.
>
>Another way to add the MPI functions to Octave would be to not modify
>Octave's startup procedure at all, but require the user to call some
>kind of mpi_start function that does the initialization and also
>includes the functionality of mpirun (if that is possible -- I suppose
>we will need it to somehow tell the other processes that they should
>perform the MPI startup steps too).  This startup function can also
>call atexit ("mpi_quit") (or similar) to ensure that MPI is stopped
>properly when Octave exits.  The user can also call mpi_quit without
>exiting the main Octave process, then when mpi_quit is called by
>atexit it will do nothing.  The problems I see with this approach are
>the warnings (which seem strange to me) about not doing much before
>calling MPI_Init and passing pointers to the actual argv/argc from the
>C main.  (Surely this is not a requirement, because there are a number
>of add-on packages for Matlab that use MPI and are implemented entirly
>in .m and mex files
>
The attached code follows your second idea of not modifying the octave
binary at, and letting the user call MPI functions such as MPI_Init and
MPI_Finalize. One problem I found with this approach is by allowing the
user to call MPI_Finalize (), analogous with mpi_quit, is that some MPI
implementations will not allow the user to call MPI_Init, or
mpi_startup, again and will cause Octave to prematurely exit, i.e.
core-dump, when MPI_Init is called a second time. (This specifically
happens when using the LAM-MPI implementation.) I have added code in the
mpi_* functions to try to prevent such errors from happening.

>I like the idea of having the flexibility of starting and stopping MPI
>processing on the fly, but I don't know whether that is practical.
>
>  
>
Are you speaking of, say, allowing a control-C to halt processing on
other nodes? I'm not sure this can be done. Simply speaking, we want to
send a UNIX signal to other MPI nodes. Perhaps a "caretaker" daemon
which sending an interrupt signal upon our request? If the functionality
were to be built into octave it may require a polling mechenism, which
would inevitably slow processing down.

-JD



mpi_octave-ss20021218.tar.gz (7K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: octave-mpi questions

Andy Jacobson-2-2
Howdy,

I think JD's on the right track with the oct file idea, but I can't
confirm that his implementation works.  I don't have access to a
working MPI platform at this time.  

No matter how this works out, I think it's rather important that the
load/save functions for reading and writing octave values to and from
streams (and thus to other MPI-aware octave instances) be included.
This could be considered an octave-specific analog of the MPI_Send()
and MPI_Recv() functions.

        Regards,

                Andy
--
Andy Jacobson

[hidden email]

Program in Atmospheric and Oceanic Sciences
Sayre Hall, Forrestal Campus
Princeton University
PO Box CN710 Princeton, NJ 08544-0710 USA

Tel: 609/258-5260  Fax: 609/258-2850


Reply | Threaded
Open this post in threaded view
|

Re: octave-mpi questions

John W. Eaton-6
In reply to this post by JD Cole-2
On 17-Feb-2003, JD Cole <[hidden email]> wrote:

| The attached code follows your second idea of not modifying the octave
| binary at, and letting the user call MPI functions such as MPI_Init and
| MPI_Finalize. One problem I found with this approach is by allowing the
| user to call MPI_Finalize (), analogous with mpi_quit, is that some MPI
| implementations will not allow the user to call MPI_Init, or
| mpi_startup, again and will cause Octave to prematurely exit, i.e.
| core-dump, when MPI_Init is called a second time.

OK, so it seems that what I was hoping to be able to do may not be
possible (at least with current MPI implementations).

| > I like the idea of having the flexibility of starting and stopping MPI
| > processing on the fly, but I don't know whether that is practical.

| Are you speaking of, say, allowing a control-C to halt processing on
| other nodes?

No, I was thinking that it might be possible to do the following:

 0. Start Octave without mpirun or any other special options.  At this
    point, Octave doesn't know anything about parallel processing,
    same as usual.

 1. Call some sequence of functions to start the MPI system
    (equivalent of lamboot and mpirun to start the other processes)
    and initialize MPI (MPI_init).  At this point, Octave knows about
    MPI and can send jobs to other processes and collect results.

 2. Do parallel tasks.

 3. Eventually, call some sequence of functions to stop the MPI
    system, terminate the other processes, and return to single
    processor mode.  The MPI code would remain loaded unless all the
    MPI functions are cleared.

 4. Go on with other tasks, possibly restarting the MPI processing
    (steps 1--3) as many times as desired without having to exit
    Octave.

If this is not possible given the current MPI implementations, is it a
fundamental limitation of MPI?

jwe


Reply | Threaded
Open this post in threaded view
|

Re: octave-mpi questions

Martin Siegert
In reply to this post by Andy Jacobson-2-2
On Tue, Feb 18, 2003 at 01:36:18PM -0500, Andy Jacobson wrote:

> I think JD's on the right track with the oct file idea, but I can't
> confirm that his implementation works.  I don't have access to a
> working MPI platform at this time.  
>
> No matter how this works out, I think it's rather important that the
> load/save functions for reading and writing octave values to and from
> streams (and thus to other MPI-aware octave instances) be included.
> This could be considered an octave-specific analog of the MPI_Send()
> and MPI_Recv() functions.

I am not sure whether you can get around the requirements of the
MPI standard, namely that you must not call any MPI functions
before having called MPI_Init and you must not call any MPI functions
after having called MPI_Finalized, in particular, the standard says
that MPI_Init cannot be called after a call to MPI_Finalize - one cannot
restart MPI once the MPI environment has been cleaned up. Thus,
the observation that Octave exits when MPI_Init is called a second
time seems to be what is supposed to happen (and not only a speciality
of LAM).
Thus, I do not believe that there is a way to let users restart MPI
and therefore I think a solution that ensures that MPI_Init and
MPI_Finalize are called just once is the only way to go.

If you want me to test solutions, I am happy to do this. I have
current versions of MPICH and LAM installed on my cluster.

Cheers,
Martin

========================================================================
Martin Siegert
Academic Computing Services                        phone: (604) 291-4691
Simon Fraser University                            fax:   (604) 291-4242
Burnaby, British Columbia                          email: [hidden email]
Canada  V5A 1S6
========================================================================


Reply | Threaded
Open this post in threaded view
|

Re: octave-mpi questions

JD Cole-2
>
>
>Thus, I do not believe that there is a way to let users restart MPI
>and therefore I think a solution that ensures that MPI_Init and
>MPI_Finalize are called just once is the only way to go.
>  
>
I agree with Martin on this. Short of creating a process separate from
Octave to handle the MPI communications, users would have to initialize
once.

>If you want me to test solutions, I am happy to do this. I have
>current versions of MPICH and LAM installed on my cluster.
>
>  
>
That would be great if you could test the code I posted earlier, I
currently don't have access to a cluster. I have been developing with
LAM by running multiple instances on one node.

Best,
JD


Reply | Threaded
Open this post in threaded view
|

Re: octave-mpi questions

Martin Siegert
On Tue, Feb 18, 2003 at 03:09:07PM -0800, JD Cole wrote:

> That would be great if you could test the code I posted earlier, I
> currently don't have access to a cluster. I have been developing with
> LAM by running multiple instances on one node.

* The code works with LAM (version 6.5.9)
* the code segfaults in mpi_init immediately with MPICH (version 1.2.5)

It took me a week to figure out why:

- passing NULL to MPI_Init as in MPI_Init(NULL,NULL); is a feature of
  the MPI-2 standard.
- there is (to my knowledge) no MPI implementation that fully implements
  the MPI-2 standard.
- different MPI implementations implement different parts of the MPI-2
  standard
- MPI_Init(NULL,NULL); works in LAM but not in MPICH

I asked the MPICH maintainers about the last point. Here is the answer:

On Fri, Feb 21, 2003 at 09:45:22AM -0600, William Gropp wrote:
> For the ch_p4 device, there isn't any easy patch (the command line
> arguments are used to communicate essential information).  One possibility
> is to use the MPICH2 beta test; MPICH2 is our new implementation of MPI and
> will support all of MPI-2; so far all of MPI-1 and parts of MPI-2 are
> supported (including the use of null args to MPI_Init).  Another choice is
> to use the ch_shmem device of MPICH; that doesn't use command line
> arguments.

My impression is that MPICH2 is really beta (e.g., the F90 interface does
not exist yet).

The only MPI-1 compliant way of calling MPI_Init is
MPI_Init(&argc,&argv);
The question now is: is there a way to provide octave's command line
arguments to the mpi_init module?

Martin

========================================================================
Martin Siegert
Academic Computing Services                        phone: (604) 291-4691
Simon Fraser University                            fax:   (604) 291-4242
Burnaby, British Columbia                          email: [hidden email]
Canada  V5A 1S6
========================================================================


Reply | Threaded
Open this post in threaded view
|

Re: octave-mpi questions

John W. Eaton-6
On 21-Feb-2003, Martin Siegert <[hidden email]> wrote:

| My impression is that MPICH2 is really beta (e.g., the F90 interface does
| not exist yet).

We only need a C interface, so I don't think this matters much.  Other
than missing interfaces, are there serious problems with MPICH2?

| The only MPI-1 compliant way of calling MPI_Init is
| MPI_Init(&argc,&argv);
| The question now is: is there a way to provide octave's command line
| arguments to the mpi_init module?

Sure, they are already available in the argv variable that is defined
when Octave starts.  But this is just a copy in an octave_value
object, so you would need to convert it back to an array of C
character strings.

In the thread starting here:

  http://www.octave.org/octave-lists/archive/help-octave.2001/msg00138.html

a claim was made that the argv/argc passed to MPI_Init must be
pointers to the actual variables passed to main, and that MPI_Init
needs to see them before they are processed by the program that calls
MPI_Init.  The MPI standard may claim that those are requirements, but
so far, no one has been able to explain *why* the standard includes
these requirements.  Also, there was a claim that bad things happen if
the function that calls MPI_Init returns, and I'm not sure whether
that is another limitation of the standard or was simply due to a
buggy implementation.

So, can anyone tell me *why* these things should cause problems?  It
seems to me that we should be able to define  MPI_Init in Octave that
does something like

  mpi_init (argv)

where argv is a cell array of strings that are packaged as appropriate
and passed on to the C MPI_Init.  Then users could pass whatever
arguments they like to the MPI system and we wouldn't have to modify
Octave's main program or argument processing at all (except perhaps we
would need a way start Octave and have it run a function and then
continue processing, so the client processes would know to go into
listener mode, waiting for commands from the server process).

It would also be nice if Octave's mpi_init could start the client
processes, so you wouldn't have to use mpirun (or similar) to do that
when initially starting Octave.

There was another thread about these issues starting here:

  http://www.octave.org/octave-lists/archive/octave-maintainers.2002/msg00207.html


jwe


Reply | Threaded
Open this post in threaded view
|

Re: octave-mpi questions

JD Cole-2
John W. Eaton wrote:
On 21-Feb-2003, Martin Siegert [hidden email] wrote:

| My impression is that MPICH2 is really beta (e.g., the F90 interface does
| not exist yet).

We only need a C interface, so I don't think this matters much.  Other
than missing interfaces, are there serious problems with MPICH2?

| The only MPI-1 compliant way of calling MPI_Init is
| MPI_Init(&argc,&argv);
| The question now is: is there a way to provide octave's command line
| arguments to the mpi_init module?

Sure, they are already available in the argv variable that is defined
when Octave starts.  But this is just a copy in an octave_value
object, so you would need to convert it back to an array of C
character strings.
  
IMHO I think it might be a mistake to require Octave to rely on the MPI-2 standard until support for it is more widespread.  Taken that into considering it may be value to implement support for dynamically linked startup files. Kind of an octaverc directory, which is executed prior to any octave startup. (It would also be equally appropriate to provide an octavecr directory which allows for shutdown routines to be called, such as MPI_Finalize.)  This is not only useful for MPI_Init function, but, may also be valuable to a user who wishes to use VSIPL standard of libraries, or other libraries which require "close to main" initialization calls. In addition to allowing Octave to "startup" external libraries, these files could also install appropriate variables, etc. (It is also would be worth considering specification of such files in .octaverc, perhaps a better solution which would cause less clutter as for as the distribution is concerned.)

JWE mentioned this concept here:
http://www.octave.org/octave-lists/archive/octave-maintainers.2003/msg00074.html
in the context of using feval (). The caveat to this implementation is that these initialization calls must ignore command line arguments not intended for the library startup AND the user must beware of arguments which may be relevant to multiple libraries, not excluding Octave.
In the thread starting here:

  http://www.octave.org/octave-lists/archive/help-octave.2001/msg00138.html

a claim was made that the argv/argc passed to MPI_Init must be
pointers to the actual variables passed to main, and that MPI_Init
needs to see them before they are processed by the program that calls
MPI_Init.  The MPI standard may claim that those are requirements, but
so far, no one has been able to explain *why* the standard includes
these requirements.  Also, there was a claim that bad things happen if
the function that calls MPI_Init returns, and I'm not sure whether
that is another limitation of the standard or was simply due to a
buggy implementation.
For starters, the standard dictates that the argv/argc dependency stems from implementation dependent needs. Specifically in Section 7.5 "Startup" of the MPI-1.1 document (available here MPI-1.1 document ) it states that MPI does not specify "how an MPI program is started or launched from the command line". I think this is due to the manner in which an MPI processes may be implemented, in addition to operating system interfacing. As stated in previous threads, the MPI-2 standard does not impose argv/argc passing to MPI specifically for the reason we are troubled by it, libraries may wish to use MPI functionality. (See MPI-2.0 document, section 4.2 "Passing NULL to MPI_Init".)

So, can anyone tell me *why* these things should cause problems?  It
seems to me that we should be able to define  MPI_Init in Octave that
does something like

  mpi_init (argv)

where argv is a cell array of strings that are packaged as appropriate
and passed on to the C MPI_Init.  Then users could pass whatever
arguments they like to the MPI system and we wouldn't have to modify
Octave's main program or argument processing at all (except perhaps we
would need a way start Octave and have it run a function and then
continue processing, so the client processes would know to go into
listener mode, waiting for commands from the server process).
  
I refer back to the octaverc/octavecr directory idea, above.
It would also be nice if Octave's mpi_init could start the client
processes, so you wouldn't have to use mpirun (or similar) to do that
when initially starting Octave.
  
    This may be a problem. The MPI-1.1 standard specifcally states that the process model is static, however, the MPI-2 standard provides new capability to dynamically create processes. (See MPI-2.0 document, section 5 "Process Creation and Management"). The new capability only gets half there, though. While some MPI implementations, such as LAM-MPI, allow applications to be started without the commands mpirun or mpiexec, is still no standardize way to start an MPI application (section 4.1 "Standardized Process Startup"). I tend to think that anyone using MPI in the first place could either write a script "mpioctave" to do the command line startup. For dynamic creation of processes, the user is still going to have to describe their distributed environment to spawn new processes.
    A second, less desirable, solution could be implemented to give this kind of functionality, however it involves creating separate (unix) processes to do the MPI communication, and would undoubtedly slow down the evaluation process.

There are obvious reasons why we may want to choose MPI-2 over MPI-1.1, however, it is my opinion that with dynamic startup capabilities, we don't need to go there yet. I am also of the opinion that many users running Octave on parallel architectures may not yet have the choice to use the newer standard. (Or it may come at quite a steep price.)

-JD
Reply | Threaded
Open this post in threaded view
|

Re: octave-mpi questions

Martin Siegert
In reply to this post by John W. Eaton-6
On Fri, Feb 21, 2003 at 07:01:25PM -0600, John W. Eaton wrote:
> On 21-Feb-2003, Martin Siegert <[hidden email]> wrote:
>
> | My impression is that MPICH2 is really beta (e.g., the F90 interface does
> | not exist yet).
>
> We only need a C interface, so I don't think this matters much.  Other
> than missing interfaces, are there serious problems with MPICH2?

It's not that octave may be able to live with an incomplete version -
replacing the MPI library means that all other MPI programs have to
be able to use that version. It is impossible (at least from a support
point of view) to have a different MPI implementation for every and
each program.

> | The only MPI-1 compliant way of calling MPI_Init is
> | MPI_Init(&argc,&argv);
> | The question now is: is there a way to provide octave's command line
> | arguments to the mpi_init module?
>
> Sure, they are already available in the argv variable that is defined
> when Octave starts.  But this is just a copy in an octave_value
> object, so you would need to convert it back to an array of C
> character strings.
>
> In the thread starting here:
>
>   http://www.octave.org/octave-lists/archive/help-octave.2001/msg00138.html
>
> a claim was made that the argv/argc passed to MPI_Init must be
> pointers to the actual variables passed to main, and that MPI_Init
> needs to see them before they are processed by the program that calls
> MPI_Init.  The MPI standard may claim that those are requirements, but
> so far, no one has been able to explain *why* the standard includes
> these requirements.  Also, there was a claim that bad things happen if
> the function that calls MPI_Init returns, and I'm not sure whether
> that is another limitation of the standard or was simply due to a
> buggy implementation.
>
> So, can anyone tell me *why* these things should cause problems?  It
> seems to me that we should be able to define  MPI_Init in Octave that
> does something like
>
>   mpi_init (argv)
>
> where argv is a cell array of strings that are packaged as appropriate
> and passed on to the C MPI_Init.

>From the MPI-1.1 Standard:
<quote>
An MPI implementation is free to require that the arguments in the C binding
must be the arguments to main.

Rationale.

The command line arguements are provided to MPI_Init to allow an MPI
implementation to use them in initializing the MPI environment. They are
passed by reference to allow an MPI implementation to provide them in
environments where the command-line arguments are not provided to main.
</quote>

E.g., under mpich running "mpirun -np 5 ./octave-2.1.44 -q mpi_test.m" will
cause an exec of

./octave-2.1.44 -q mpi_test.m -p4pg /usr/local/src/mpi_octave-ss20021218/PI9261 -p4wd /usr/local/src/mpi_octave-ss20021218

> Then users could pass whatever
> arguments they like to the MPI system and we wouldn't have to modify
> Octave's main program or argument processing at all (except perhaps we
> would need a way start Octave and have it run a function and then
> continue processing, so the client processes would know to go into
> listener mode, waiting for commands from the server process).

I users cannot pass whatever they like to the MPI system. They must pass
those command line arguments that are important for the MPI environment:
In the example above that would be "-p4pg ... -p4wd ...".
That part must be included when calling MPI_Init().

> It would also be nice if Octave's mpi_init could start the client
> processes, so you wouldn't have to use mpirun (or similar) to do that
> when initially starting Octave.
>
> There was another thread about these issues starting here:
>
>   http://www.octave.org/octave-lists/archive/octave-maintainers.2002/msg00207.html

I cannot see how you can avoid "mpirun". Furthermore, I'd find I that highly
undesirable: MPI users are used to start their programs with mpirun.

Cheers,
Martin


Reply | Threaded
Open this post in threaded view
|

Re: octave-mpi questions

Taza
In reply to this post by Martin Siegert
Hi
am have been working on buiding a High Performance Computing Cluster, i decided to use  Michaels PelicanHPC v2.2 and have been practicing generic mpi programming and then tried to understand the octave- mpi examples on the live cd, like kernel_example. reffered to also the old parallel knoppix mpi examples so that i could understand the concept of parallel programming, i seem to well appreciate the concept of parallel programming.
However  now  i want to have hands on based approach with the examples and concepts of octave and mpi i have bn understanding. On opening the octave script, it looks like i cant write my code and save it so that i can compile and run it, like how i was doing when i was doing the generic mpi, I go to pico/nano editor and write my code, save it then run it at the command prompt. so my problem is i have problem to do something similar in octave script. help me on this.
Reply | Threaded
Open this post in threaded view
|

Re: octave-mpi questions

Taza
In reply to this post by John W. Eaton-6
an image of the coding i do on my virtual machine

Hi this is the image of what i try to code on my virtual machine frontend, i first generate some dependent variables then try to use mpi to pass the variables around the cluster. but when i use mpi, i get stuck on how to begin running my code, like u can see in my image, i fail to exit the code so that i can save my work and compile and run it, am really lost at this area. how to go about it.
Reply | Threaded
Open this post in threaded view
|

Re: octave-mpi questions

Michael Creel


On Thu, Dec 1, 2011 at 6:19 AM, Taza <[hidden email]> wrote:
http://octave.1599824.n4.nabble.com/file/n4127009/octave-mpi_image.png

Hi this is the image of what i try to code on my virtual machine frontend, i
first generate some dependent variables then try to use mpi to pass the
variables around the cluster. but when i use mpi, i get stuck on how to
begin running my code, like u can see in my image, i fail to exit the code
so that i can save my work and compile and run it, am really lost at this
area. how to go about it.

--
View this message in context: http://octave.1599824.n4.nabble.com/octave-mpi-questions-tp1643254p4127009.html
Sent from the Octave - Maintainers mailing list archive at Nabble.com.

Hi Taza,
This mail list is not the proper place for these questions. Please make a post at the PelicanHPC forum (http://pelicanhpc.788819.n4.nabble.com/), and I'll answer there. 

Michaeñ