Last modified 4 years ago
MPI on Grid
This wiki page is a summary of the more detailed documentation. Though in this wiki page some files have been adapted and corrected.
MPI-start wrapper
Let's call it mpi-start-wrapper.sh
#!/bin/bash
# Pull in the arguments.
MY_EXECUTABLE=`pwd`/$1
MPI_FLAVOR=$2
# Convert flavor to lowercase for passing to mpi-start.
MPI_FLAVOR_LOWER=`echo $MPI_FLAVOR | tr '[:upper:]' '[:lower:]'`
# Pull out the correct paths for the requested flavor.
eval MPI_PATH=`printenv MPI_${MPI_FLAVOR}_PATH`
# Ensure the prefix is correctly set. Don't rely on the defaults.
eval I2G_${MPI_FLAVOR}_PREFIX=$MPI_PATH
export I2G_${MPI_FLAVOR}_PREFIX
# Touch the executable. It exist must for the shared file system check.
# If it does not, then mpi-start may try to distribute the executable
# when it shouldn't.
touch $MY_EXECUTABLE
# Setup for mpi-start.
export I2G_MPI_APPLICATION=$MY_EXECUTABLE
export I2G_MPI_APPLICATION_ARGS=
export I2G_MPI_TYPE=$MPI_FLAVOR_LOWER
export I2G_MPI_PRE_RUN_HOOK=mpi-hooks.sh
export I2G_MPI_POST_RUN_HOOK=mpi-hooks.sh
# If these are set then you will get more debugging information.
export I2G_MPI_START_VERBOSE=1
#export I2G_MPI_START_DEBUG=1
# Invoke mpi-start.
$I2G_MPI_START
Hooks
Let's call the file mpi-hooks.sh
#!/bin/sh
#
# This function will be called before the MPI executable is started.
# You can, for example, compile the executable itself.
#
pre_run_hook () {
# Compile the program.
echo "Compiling ${I2G_MPI_APPLICATION}"
# Actually compile the program.
cmd="mpicc ${MPI_MPICC_OPTS} -o ${I2G_MPI_APPLICATION} ${I2G_MPI_APPLICATION}.c"
echo $cmd
$cmd
if [ ! $? -eq 0 ]; then
echo "Error compiling program. Exiting..."
exit 1
fi
# Everything's OK.
echo "Successfully compiled ${I2G_MPI_APPLICATION}"
return 0
}
#
# This function will be called before the MPI executable is finished.
# A typical case for this is to upload the results to a storage element.
#
post_run_hook () {
echo "Executing post hook."
echo "Finished the post hook."
return 0
}
The JDL file
Let's call it mpi-job.jdl
JobType = "MPICH";
NodeNumber = 4;
Executable = "mpi-start-wrapper.sh";
Arguments = "mpi-test OPENMPI";
StdOutput = "mpi-test.out";
StdError = "mpi-test.err";
InputSandbox = {"mpi-start-wrapper.sh","mpi-hooks.sh","mpi-test.c"};
OutputSandbox = {"mpi-test.err","mpi-test.out"};
Requirements =
Member("MPI-START", other.GlueHostApplicationSoftwareRunTimeEnvironment)
&& Member("OPENMPI", other.GlueHostApplicationSoftwareRunTimeEnvironment)
# && (other.GlueCEUniqueID == "gridce.iihe.ac.be:2119/jobmanager-pbs-betest")
;
The other supported implementations can be selected by changing "OPENMPI" (in two places) to the name of the desired implementation. The other names are "LAM", "MPICH", and "MPICH2". Note again that the "JobType?" attribute must be "MPICH" in all cases
Executable Source Code
Let's call it mpi-test.c
/* hello.c
*
* Simple "Hello World" program in MPI.
*
*/
#include "mpi.h"
#include <stdio.h>
int main(int argc, char *argv[]) {
int numprocs; /* Number of processors */
int procnum; /* Processor number */
/* Initialize MPI */
MPI_Init(&argc, &argv);
/* Find this processor number */
MPI_Comm_rank(MPI_COMM_WORLD, &procnum);
/* Find the number of processors */
MPI_Comm_size(MPI_COMM_WORLD, &numprocs);
printf ("Hello world! from processor %d out of %d\n", procnum, numprocs);
/* Shut down MPI */
MPI_Finalize();
return 0;
}
Output
As a result, in mpi-test.out you should have something similar to:
Modified mpirun: Executing command: ./mpi-start-wrapper.sh mpi-test OPENMPI ************************************************************************ UID = betest046 HOST = node15-9.wn.iihe.ac.be DATE = Thu Mar 6 14:22:10 CET 2008 VERSION = 0.0.52 ************************************************************************ mpi-start [INFO ]: search for scheduler mpi-start [INFO ]: activate support for pbs mpi-start [INFO ]: activate support for openmpi mpi-start [INFO ]: call backend MPI implementation mpi-start [INFO ]: start program with mpirun -<START PRE-RUN HOOK>--------------------------------------------------- Compiling /user/betest046/gram_scratch_fkK23UZSoS/.mpi/https_3a_2f_2flaranja.iihe.ac.be_3a9000_2f4TtvVjtG6CQgaftNJz2flQ/mpi-test mpicc -m32 -o /user/betest046/gram_scratch_fkK23UZSoS/.mpi/https_3a_2f_2flaranja.iihe.ac.be_3a9000_2f4TtvVjtG6CQgaftNJz2flQ/mpi-test /user/betest046/gram_scratch_fkK23UZSoS/.mpi/https_3a_2f_2flaranja.iihe.ac.be_3a9000_2f4TtvVjtG6CQgaftNJz2flQ/mpi-test.c Successfully compiled /user/betest046/gram_scratch_fkK23UZSoS/.mpi/https_3a_2f_2flaranja.iihe.ac.be_3a9000_2f4TtvVjtG6CQgaftNJz2flQ/mpi-test -<STOP PRE-RUN HOOK>---------------------------------------------------- =[START]================================================================ Hello world! from processor 1 out of 4 Hello world! from processor 3 out of 4 Hello world! from processor 2 out of 4 Hello world! from processor 0 out of 4 =[FINISHED]============================================================= -<START POST-RUN HOOK>--------------------------------------------------- Executing post hook. Finished the post hook. -<STOP POST-RUN HOOK>----------------------------------------------------
Other MPI exercises
https://computing.llnl.gov/tutorials/mpi/exercise.html
Simple job
/* hello.c
*
* Simple "Hello World" program in MPI.
*
*/
#include "mpi.h"
#include <stdio.h>
int main(int argc, char *argv[])
{
int numprocs; /* Number of processors */
int procnum; /* Processor number */
/* Initialize MPI */
MPI_Init(&argc, &argv);
/* Find this processor number */
MPI_Comm_rank(MPI_COMM_WORLD, &procnum);
/* Find the number of processors */
MPI_Comm_size(MPI_COMM_WORLD, &numprocs);
printf ("Hello world! from processor %d out of %d\n", procnum, numprocs);
/* Shut down MPI */
MPI_Finalize();
return 0;
}
check.jdl
Type = "Job";
JobType = "MPICH";
NodeNumber = 6;
Executable = "check.sh";
StdOutput = "check.out";
StdError = "check.err";
InputSandbox = {"check.sh","mpihello.c","Makefile"};
OutputSandbox = {"check.err","check.out"};
#Requirements = (other.GlueCEInfoLRMSType == "PBS");
RetryCount = 0;
Lrms_Type = "PBS";
check.sh
#!/bin/sh -x date echo "SITE: $SITE_NAME" echo "GIIS: $SITE_GIIS_URL" echo "LOG: $GLITE_WMS_LOG_DESTINATION" echo "==== Info ====" env | sort echo "==== Task ====" make mpiexec mpihello echo "==== End ====="
