 |
Jobs: MPICH-G2 for Globus 4.0
Home > User Info > Jobs > MPICH-G2 for Globus 4.0
On this page
Related Links
Need Help?
MPICH-G2 will not be deployed on TeraGrid resources which do not support distributed MPI applications. MPICH-G2 is supported on the following TeraGrid platforms:
| Site |
Resource |
| IU |
Big Red |
| NCSA |
Cobalt |
| NCSA |
Mercury |
| ORNL |
NSTG |
| PSC |
Rachel |
| Purdue |
Lear |
| SDSC |
DataStar |
| SDSC |
IA-64 |
| TACC |
LoneStar |
| TACC |
Maverick |
| UC/ANL |
IA-64 |
| UC/ANL |
IA-32 Visualization Cluster |
MPICH-G2 is a distributed MPI implementation that allows parallelized, message-passing applications (which are usually simulations) to use more than one TeraGrid resource in a single application run. MPICH-G2 uses the Globus GRAM service to submit parts of the run to multiple TeraGrid resources and it uses both the local MPI implementation and TCP/IP to achieve message passing between subjobs. Grid authentication is used both for job submission and to secure the communication channels between resources. This deployment of MPICH-G2 will include support for each of the locally supported MPI/compiler combinations on suitable resources. Some systems support more than one local MPI implementation, and some support more than one compiler.
The following page details the steps necessary for running MPICH-G2 jobs on TeraGrid. They assume that the user already has acquired their Globus security credentials and has their DN entered in the gridmap files across the TeraGrid.
1. Setting up your environment with your ~/.soft
NOTE: These instructions are for use with CTSS 3 and Globus 4.
See the MPICH-G2 page if you are using CTSS 2 or Globus 2.
See Using SoftEnv for details on what a .soft file is and how to make changes to it.
At the end of your .soft file you will need to add a corresponding MPICH-G2 macro listed below in the section Access to MPICH-G2. Add the macro that correpsonds to the resource and compiler that you will be using. For example, if you are using NCSA's Mercury system with Intel compilers in non-debug mode, you're .soft file might look like this:
@teragrid-basic
@globus-4.0
@mpich-g2-1.2.6e-mpich-gm-intel81
For other systems, simply replace the last line of the .soft file with the corresponding macro from the table below.
2. Compiling your MPICH-G2 application
Compile your application at each TeraGrid site. There are four MPICH-G2 compilers; for C programs use mpicc, for C++ use mpicxx, for Fortran77 use mpif77, and for Fortran90 use mpif90. These compilers should already be in your path based on the ~/.soft specfications in step 1.
3. Write your Globus RSL file
Your MPICH-G2 application can be executed on one or more TeraGrid sites in a single run. From your application's point of view it will appear as a single MPI_COMM_WORLD that spans the TeraGrid sites you're running on.
You describe the job you want to run using the Globus Resource Specification Language (RSL). Click for a full description of the RSL syntax.
Here is an example RSL file to run the "allall_g2" program as a single 6-proc job across NCSA, SDSC, and ANL (2 processes at each site). Please note that for "count", ranks are packed onto clusters such that if a compute host has 2 processors, both will be used. Also note the use of queue= and project= RSL parameters where appropriate. (The project parameter is currently required at IU, Purdue, and UC/ANL.)
+
( &(resourceManagerContact="grid-hg.ncsa.teragrid.org/jobmanager-pbs")
(count=2)
(queue=debug)
(jobtype=mpi)
(project="TG-STA040001N")
(maxwalltime=5)
(label="subjob 0")
(environment=(GLOBUS_DUROC_SUBJOB_INDEX 0)
(GLOBUS_TCP_PORT_RANGE ""))
(directory="/home/ncsa/arnoldg/mpi")
(executable="/home/ncsa/arnoldg/mpi/allall_g2")
(arguments="10" "700" "2000")
)
( &(resourceManagerContact="tg-login1.sdsc.teragrid.org/jobmanager-pbs_intel")
(count=2)
(jobtype=mpi)
(project="TG-STA040001N")
(maxwalltime=5)
(label="subjob 1")
(environment=(GLOBUS_DUROC_SUBJOB_INDEX 1)
(GLOBUS_TCP_PORT_RANGE ""))
(directory="/users/ux453915/mpi")
(executable="/users/ux453915/mpi/allall_g2")
(arguments="10" "700" "2000")
)
( &(resourceManagerContact="tg-grid1.uc.teragrid.org/jobmanager-pbs_intel")
(count=2)
(host_count=1:ia64-compute)
(jobtype=mpi)
(project="TG-STA040001N")
(maxwalltime=5)
(label="subjob 2")
(environment=(GLOBUS_DUROC_SUBJOB_INDEX 2)
(GLOBUS_TCP_PORT_RANGE "")
)
(directory="/home/arnoldg/mpi")
(executable="/home/arnoldg/mpi/allall_g2")
(arguments="10" "700" "2000")
)
 Access to MPICH-G2
To build and run MPICH-G2 applications users should replace the @teragrid-dev macro with a macro from the table below for the resource being used. All of the libraries enabled by these keys are non-threaded. Notes indicate the local MPI implementation used, the compilers used, and the presence or absence of debugging information.
| Resource |
MPICH-G2 macro |
Notes (MPI, compilers, debug) |
| Indiana Big Red |
|
OpenMPI, Intel compilers, non-debug |
| Indiana Big Red |
|
OpenMPI, Intel compilers, debug |
| Indiana Big Red |
|
OpenMPI, GNU compilers, non-debug |
| Indiana Big Red |
|
OpenMPI, GNU compilers, debug |
| NCSA Cobalt |
@mpich-g2-1.2.6e-mpi12-intel81 |
Intel MPI, Intel compilers, non-debug |
| NCSA Cobalt |
@mpich-g2-1.2.6e-mpi12-intel81-dbg |
Intel MPI, Intel compilers, debug |
| NCSA Cobalt |
@mpich-g2-1.2.6e-mpi12-gcc33 |
MPICH-GM, GNU compilers, non-debug |
| NCSA Cobalt |
@mpich-g2-1.2.6e-mpi12-gcc33-dbg |
MPICH-GM, GNU compilers, debug |
| NCSA Mercury |
@mpich-g2-1.2.6e-mpich-gm-intel81 |
MPICH-GM, Intel compilers, non-debug |
| NCSA Mercury |
@mpich-g2-1.2.6e-mpich-gm-intel81-dbg |
MPICH-GM, Intel compilers, debug |
| NCSA Mercury |
@mpich-g2-1.2.6e-mpich-gm-gcc32 |
MPICH-GM, GNU compilers, non-debug |
| NCSA Mercury |
@mpich-g2-1.2.6e-mpich-gm-gcc32-dbg |
MPICH-GM, GNU compilers, debug |
| ORNL NSTG |
|
MPICH-P4, Intel compilers, non-debug |
| ORNL NSTG |
|
MPICH-P4, Intel compilers, debug |
| ORNL NSTG |
|
MPICH-P4, GNU compilers, non-debug |
| ORNL NSTG |
|
MPICH-P4, GNU compilers, debug |
| PSC Rachel |
|
HP MPI, HP compilers, non-debug |
| PSC Rachel |
|
HP MPI, HP compilers, debug |
| Purdue Lear |
|
MPICH-P4, Intel compilers, non-debug |
| Purdue Lear |
|
MPICH-P4, Intel compilers, debug |
| Purdue Lear |
|
MPICH-P4, GNU compilers, non-debug |
| Purdue Lear |
|
MPICH-P4, GNU compilers, debug |
| SDSC DataStar |
|
PPE.POE, Intel compilers, non-debug |
| SDSC DataStar |
|
PPE.POE, Intel compilers, debug |
| SDSC DataStar |
|
MPICH-P4, GNU compilers, non-debug |
| SDSC DataStar |
|
MPICH-P4, GNU compilers, debug |
| SDSC DTF |
@mpich-g2-1.2.6e-mpich-gm-intel81 |
MPICH-GM, Intel compilers, non-debug |
| SDSC DTF |
@mpich-g2-1.2.6e-mpich-gm-intel81-dbg |
MPICH-GM, Intel compilers, debug |
| SDSC DTF |
@mpich-g2-1.2.6e-mpich-gm-gcc32 |
MPICH-GM, GNU compilers, non-debug |
| SDSC DTF |
@mpich-g2-1.2.6e-mpich-gm-gcc32-dbg |
MPICH-GM, GNU compilers, debug |
| TACC Lonestar |
@mpich-g2-1.2.6e-mvapich-intel91 |
MVAPICH, Intel compilers, non-debug |
| TACC Lonestar |
@mpich-g2-1.2.6e-mvapich-intel91-dbg |
MVAPICH, Intel compilers, debug |
| TACC Maverick |
|
|
| TACC Maverick |
|
|
| UC/ANL DTF |
@mpich-g2-1.2.6e-mpich-gm-intel81 |
MPICH-GM, Intel compilers, non-debug |
| UC/ANL DTF |
@mpich-g2-1.2.6e-mpich-gm-intel81-dbg |
MPICH-GM, Intel compilers, debug |
| UC/ANL DTF |
@mpich-g2-1.2.6e-mpich-gm-gcc32 |
MPICH-GM, GNU compilers, non-debug |
| UC/ANL DTF |
@mpich-g2-1.2.6e-mpich-gm-gcc32-dbg |
MPICH-GM, GNU compilers, debug |
Access to Globus MPI Libraries for MPICH-G2
Developers and advanced users who need a Globus MPI flavor in order to build their own versions of MPICH-G2 should replace @teragrid-dev and @globus-4.0 with a macros from the following table. All of the libraries enabled by these keys are non-threaded. Notes indicate the local MPI implementation used, the compilers used, and the presence or absence of debugging information.
| Resource |
Globus MPI Libraries macro |
Notes (MPI, compilers, debug) |
| Indiana Big Red |
|
OpenMPI, Intel compilers, non-debug |
| Indiana Big Red |
|
OpenMPI, Intel compilers, debug |
| Indiana Big Red |
|
OpenMPI, GNU compilers, non-debug |
| Indiana Big Red |
|
OpenMPI, GNU compilers, debug |
| NCSA Cobalt |
+globus-mpicc64-4.0.1-mpi12-intel81-r1 |
Intel MPI, Intel compilers, non-debug |
| NCSA Cobalt |
+globus-mpicc64dbg-4.0.1-mpi12-intel81-r1 |
Intel MPI, Intel compilers, debug |
| NCSA Cobalt |
+globus-mpicc64-4.0.1-mpi12-gcc33-r1 |
Intel MPI, GNU compilers, non-debug |
| NCSA Cobalt |
+globus-mpicc64dbg-4.0.1-mpi12-gcc33-r1 |
Intel MPI, GNU compilers, debug |
| NCSA Mercury |
@globus-mpicc64pthr-4.0.1-mpich-gm-intel81 |
MPICH-GM, Intel compilers, non-debug |
| NCSA Mercury |
@globus-mpicc64dbg-4.0.1-mpich-gm-intel81 |
MPICH-GM, Intel compilers, debug |
| NCSA Mercury |
@globus-mpicc64pthr-4.0.1-mpich-gm-gcc32 |
MPICH-GM, GNU compilers, non-debug |
| NCSA Mercury |
@globus-mpicc64dbgpthr-4.0.1-mpich-gm-gcc32 |
MPICH-GM, GNU compilers, debug |
| ORNL NSTG |
|
MPICH-P4, Intel compilers, non-debug |
| ORNL NSTG |
|
MPICH-P4, Intel compilers, debug |
| ORNL NSTG |
|
MPICH-P4, GNU compilers, non-debug |
| ORNL NSTG |
@mpich-g2-1.2.6e-gcc32dbg |
MPICH-P4, GNU compilers, debug |
| PSC Rachel |
|
HP MPI, HP compilers, non-debug |
| PSC Rachel |
|
HP MPI, HP compilers, debug |
| Purdue Lear |
|
MPICH-P4, Intel compilers, non-debug |
| Purdue Lear |
|
MPICH-P4, Intel compilers, debug |
| Purdue Lear |
|
MPICH-P4, GNU compilers, non-debug |
| Purdue Lear |
|
MPICH-P4, GNU compilers, debug |
| SDSC DataStar |
|
PPE.POE, Intel compilers, non-debug |
| SDSC DataStar |
|
PPE.POE, Intel compilers, debug |
| SDSC DataStar |
|
MPICH-P4, GNU compilers, non-debug |
| SDSC DataStar |
|
MPICH-P4, GNU compilers, debug |
| SDSC DTF |
@globus-mpicc64-gm-4.0.1-mpich-gm-intel81 |
MPICH-GM, Intel compilers, non-debug |
| SDSC DTF |
@globus-mpicc64dbg-gm-4.0.1-mpich-gm-intel81 |
MPICH-GM, Intel compilers, debug |
| SDSC DTF |
@globus-mpicc64-gm-4.0.1-mpich-gm-gcc32 |
MPICH-GM, GNU compilers, non-debug |
| SDSC DTF |
@globus-mpicc64dbg-gm-4.0.1-mpich-gm-gcc32 |
MPICH-GM, GNU compilers, debug |
| TACC Lonestar |
@globus-mpicc64-4.0.1-mvapich-intel91 |
MVAPICH, Intel compilers, non-debug |
| TACC Lonestar |
@globus-mpicc64dbg-4.0.1-mvapich-intel91 |
MVAPICH, Intel compilers, debug |
| TACC Maverick |
|
|
| TACC Maverick |
|
|
| UC/ANL DTF |
@globus-mpicc64-4.0.1-mpich-gm-intel81 |
MPICH-GM, Intel compilers, non-debug |
| UC/ANL DTF |
@globus-mpicc64dbg-4.0.1-mpich-gm-intel81 |
MPICH-GM, Intel compilers, debug |
| UC/ANL DTF |
@globus-mpicc64-4.0.1-mpich-gm-gcc32 |
MPICH-GM, GNU compilers, non-debug |
| UC/ANL DTF |
@globus-mpicc64dbg-4.0.1-mpich-gm-gcc32 |
MPICH-GM, GNU compilers, debug |
Access to Globus MPI Libraries for mpiG
A new version of MPICH-G2 is under development. The name of this version is mpiG, and it uses Globus Web service interfaces rather than pre-Web service interfaces. Developers and very advanced users who need a Globus MPI flavor in order to build their own versions of mpiG should replaced @teragrid-dev and @globus-4.0 with a macros from the following table. All of the libraries enabled by these keys are threaded. Notes indicate the local MPI implementation used, the compilers used, and the presence or absence of debugging information.
| Resource |
Globus MPI Libraries macro |
Notes (MPI, compilers, debug) |
| Indiana Big Red |
|
OpenMPI, Intel compilers, non-debug |
| Indiana Big Red |
|
OpenMPI, Intel compilers, debug |
| Indiana Big Red |
|
OpenMPI, GNU compilers, non-debug |
| Indiana Big Red |
|
OpenMPI, GNU compilers, debug |
| NCSA Cobalt |
|
Intel MPI, GNU compilers, debug |
| NCSA Cobalt |
|
Intel MPI, GNU compilers, non-debug |
| NCSA Cobalt |
|
Intel MPI, Intel compilers, non-debug |
| NCSA Cobalt |
|
Intel MPI, Intel compilers, debug |
| NCSA Mercury |
@globus-mpicc64pthr-4.0.1-mpich-gm-intel81 |
MPICH-GM, Intel compilers, non-debug |
| NCSA Mercury |
@globus-mpicc64dbgpthr-4.0.1-mpich-gm-intel81 |
MPICH-GM, Intel compilers, debug |
| NCSA Mercury |
@globus-mpicc64pthr-4.0.1-mpich-gm-gcc32 |
MPICH-GM, GNU compilers, non-debug |
| NCSA Mercury |
@globus-mpicc64dbgpthr-4.0.1-mpich-gm-gcc32 |
MPICH-GM, GNU compilers, debug |
| ORNL NSTG |
|
MPICH-P4, Intel compilers, non-debug |
| ORNL NSTG |
|
MPICH-P4, Intel compilers, debug |
| ORNL NSTG |
|
MPICH-P4, GNU compilers, non-debug |
| ORNL NSTG |
|
MPICH-P4, GNU compilers, debug |
| PSC Rachel |
|
HP MPI, HP compilers, non-debug |
| PSC Rachel |
|
HP MPI, HP compilers, debug |
| Purdue Lear |
|
MPICH-P4, Intel compilers, non-debug |
| Purdue Lear |
|
MPICH-P4, Intel compilers, debug |
| Purdue Lear |
|
MPICH-P4, GNU compilers, non-debug |
| Purdue Lear |
|
MPICH-P4, GNU compilers, debug |
| SDSC DataStar |
|
PPE.POE, Intel compilers, non-debug |
| SDSC DataStar |
|
PPE.POE, Intel compilers, debug |
| SDSC DataStar |
|
MPICH-P4, GNU compilers, non-debug |
| SDSC DataStar |
|
MPICH-P4, GNU compilers, debug |
| SDSC DTF |
@globus-mpicc64pthr-4.0.1-mpich-gm-intel81 |
MPICH-GM, Intel compilers, non-debug |
| SDSC DTF |
@globus-mpicc64dbgpthr-4.0.1-mpich-gm-intel81 |
MPICH-GM, Intel compilers, debug |
| SDSC DTF |
@globus-mpicc64pthr-4.0.1-mpich-gm-gcc32 |
MPICH-GM, GNU compilers, non-debug |
| SDSC DTF |
@globus-mpicc64dbgpthr-4.0.1-mpich-gm-gcc32 |
MPICH-GM, GNU compilers, debug |
| TACC Lonestar |
|
MVAPICH, Intel compilers, non-debug |
| TACC Lonestar |
|
MVAPICH, Intel compilers, debug |
| TACC Maverick |
|
|
| TACC Maverick |
|
|
| UC/ANL DTF |
@globus-mpicc64pthr-4.0.1-mpich-gm-intel81 |
MPICH-GM, Intel compilers, non-debug |
| UC/ANL DTF |
@globus-mpicc64dbgpthr-4.0.1-mpich-gm-intel81 |
MPICH-GM, Intel compilers, debug |
| UC/ANL DTF |
@globus-mpicc64pthr-4.0.1-mpich-gm-gcc32 |
MPICH-GM, GCC compilers, non-debug |
| UC/ANL DTF |
@globus-mpicc64dbgpthr-4.0.1-mpich-gm-gcc32 |
MPICH-GM, GCC compilers, debug |
Example MPICH-G2 Application
If you are interested in running a test application there is an example available from the MPICH-G2 home page. This link provides a complete code sample and instructions on running the ring test.
The environment variable $MPICH_G2_GLOBUS_LOCATION is defined in the MPICH-G2 macro. To use this test in the TeraGrid environment edit the location of mpicc in Makefile as follows:
MPICH_INSTALL_PATH = $(MPICH_G2_GLOBUS_LOCATION)/mpich-g2
To run the test, type the following commands (the PATH is not required, since it is set by the MPICH-G2 macro):
% make ring
% mpirun -np 4 ring
|
|
