Kruskal Cluster

Martin David Kruskal

Introduction

The kruskal cluster consists of 32 HP DL165G7 servers each with dual 16 core AMD Opteron processors (total 1024 cores). They are interconnected by Infiniband Quad Data Rate (40Gb) switches. These systems each boot from the same Red Hat Enterprise Linux Operating System image.

The kruskal queue schedules batch jobs on the kruskal cluster. The cluster is optimized for mid scale (>=32 core) parallel jobs, and is not intended to run small jobs which can be accommodated by other clusters.

Tips To Run Kruskal Jobs

1. Users are encouraged to build their executables with "pathscale/3.2" and "openmpi". Applications built with MPICH will run on kruskal through IPoIB, but this won't be able to fully utilize underlying Infiniband capacity through RDMA.

2. OpenMPI is integrated with the batch system. It uses a considerable amount of shared memory with RDMA technology. Users running openmpi need to increase this limit:

for a C shell user, put the following line in ./cshrc file:

limit memorylocked 1048576

for a Bash shell user, put the following line in ./bashrc file:

ulimit -l 1048576

3. OpenMPI attemps to determine which BTL driver to use at run time. The following example shows how to use the mca option (--mca) to instruct it to use InfiniBand driver: ("np" is number of cores)

mpirun --mca btl openib -np 128 myprog

4. Here is a sample job script:

      ## --- run on the kruskal queue
      #PBS -q kruskal
      ## --- the name of the job
      #PBS -N my_job
      ## --- get 8 systems, 16 cores on each, and 32GB of total memory
      #PBS -l nodes=8:ppn=16,mem=32000mb
      ## --- don't rerun if the job fails
      #PBS -r n
      ## --- the time needed
      #PBS -l walltime=24:00:00
      #
      cd $PBS_O_WORKDIR
      date
      mpirun --mca btl openib -np 128 ./prog
      date