FNAL - KICP Joint Cluster

Cosmos Cluster Documentation

New Users

Fair Use Policy

Strong Authentication

User Authentication

Hardware Details

Software Details

Filesytem Details

Tool Documentation

Data Transfer

TORQUE Batch System

Cluster Usage

Software Details

Software will be acquired, depending on the user needs. Here is the list of available software:

  • Portland group compilers are in /opt/pgi/. You can setup the Portland group compiler by executing the following command (substitute setups.csh for csh/tcsh):
    source /opt/pgi/setup.sh
    
  • Intel compilers are in /opt/intel/. You can setup the Intel compilers by executing the following command (substitute setups.csh for csh/tcsh):
    source /opt/intel/intelc.sh intel64
    source /opt/intel/intelf.sh intel64
    
  • Note: The code generated with the Intel compiler option -fast is not a valid executable for AMD processors. Please use the following options instead to compile your code on fulla:

    -O3 -ipo -no-prec-div -static -xW

  • The MPICH2 versions for gcc, PGI, and Intel are installed in the directories /usr/local/mpich2-gcc, /usr/local/mpich2-pgi, and /usr/local/mpich2-intel, respectively. To use mpich2-pgi or mpich2-intel, you will have to first setup the respective compiler as mentioned above. To launch the MPI jobs use /usr/local/mpich2-xxx/bin/mpiexec. Execute the above mpiexec commands with the option --help to obtain a list of the available options with the mpiexec command.

  • OpenMPI for PGI is in /usr/local/openmpi-pgi, for Intel in /usr/local/openmpi-intel, and for gcc in /usr/local/openmpi-gcc. For PGI and Intel you will need to setup the respective compiler environment variables as discussed above.

  • Globus Toolkit (GT) is located in /usr/local/gt5.0.3.

Launching MPI Jobs

For mpich2, please use mpiexec command found in /usr/local/mpich2-xxx/bin to launch your MPI jobs.

mpiexec will automatically determine the hosts and the number of processes from your batch job submission. You may optionally specify the number of processes with command line option "-n #'.

For openmpi, use the mpirun command in /usr/local/openmpi-xxx/bin. Set the following environment variable: and for PGI or Intel setup the compiler environment as discussed above. Use the mpirun command in the bin directory. You should not specify a hostfile or a machinefile, as the hosts will be automatically detected from the batch scheduler.

Last Modified 2/06/2008   webmaster@fulla.fnal.gov
Security, Privacy, Legal
Fermilab Policy on Computing
Fermi National Accelerator Laboratory