Software Development Cluster

colony2a.emsl.pnl.gov

Purpose

The cluster is used to support parallel software development efforts, including the DoE OASCR funded projects. By providing multiple interconnects (Myrinet, Infiniband, and Ethernet) it allows to compare attributes of these networks with the same compute nodes.

Hardware Configuration

24 dual 1GHz Itanium-2 compute nodes (48 CPUs) from Hewlett Packard

36GB disk and 6GB memory per node

Front-end login/compilation node: 900 MHz Itanium-2 single CPU node

1GB memory

NFS server node: dual 800MHz Itanium-1

2GB memory
RAID-5 720GB storage

partitioned into:

200GB for /home

500GB for /nfstmp

Note that there is a protection against h/w failure but no backup at this time

Scratch area 2 x 140GB available as /nfsscratch1 and /nfsscratch2

data is volatile, no backup

Networks

Myrinet-2000 on all nodes
Infiniband on all nodes
Ethernet on all nodes
GigE on all nodes

Software Configuration

Example settings are given below for the C shell users.

Operating System

Red Hat Linux with kernel 2.4-20 with patches provided by HP

Compilers

GCC 2.96 and 3.2.7
Intel C and Fortran compilers

set ICOMPILER = /afs/msrc/ia64_linux24/lapps/intel/compiler70/ia64
setenv LD_LIBRARY_PATH /opt/mlib/lib/linux:{$ICOMPILER}/lib 
set path = ( $path {$ICOMPILER}/bin )

MPI

To use MPI with ARMCI/GA you need to specify a few network dependent and network independent settings:

Network dependent setting specified in the table below points to the location of MPI through the environment variable MPI_LOC

MPI version	MPI_LOC Set it as "setenv MPI_LOC xxx" where xxx is specified below
MPI over Myrinet: MPICH-GM from Myricom
----intel fortran 7 compilers (but still with 4 byte fortran integers)-use this if you don't know what to use	/home/software/mpich-gm
----GNU compilers (and hence 4 byte fortran integers)	/home/software/mpich-gm_gnu_i4
----intel fortran 7 compiler and 8 byte fortran integers(see note below)	/home/software/mpich-gm_efc7_i8
----intel fortran 8 compiler and 8 byte fortran integers(see note below)	/home/software/mpich-gm_ifort8_i8

MPI over sockets: MPICH/p4 from Argonne	/home/software/mpich-p4

MPI over Infiniband: MVAPICH from OSU compiled with:
----intel fortran 7 compilers (but still with 4 byte fortran integers)-use this if you don't know what to use	/home/software/mvapich
----GNU compilers (and hence 4 byte fortran integers)	/home/software/mvapich_gnu_i4
----intel fortran 7 compiler and 8 byte fortran integers(see note below)	/home/software/mvapich_efc7_i8
----intel fortran 8 compiler and 8 byte fortran integers(see note below)	/home/software/mvapich_ifort8_i8

NOTE FOR MPI COMPILED FOR 8 BYTE INTEGERS: MVAPICH (MPI over InfiniBand from OSU) has been compiled for 8 byte integers using the efc 7 compiler(/home/software/mvapich_efc7_i8) and ifort 8.0 compiler (/home/software/mvapich_ifort8_i8). Similarly, MPICH-GM (from Myricom) has been compiled for 8 byte integers using the efc 7 compiler (/home/software/mpich-gm_efc7_i8) and ifort 8.0 compiler (/home/software/mpich-gm_ifort8_i8). To use ifort 8.0 you have to source /home/edo/intel_fc_80/bin/ifortvars.csh or /home/edo/intel_fc_80/bin/ifortvars.sh based on your shell.

Network independent settings for using MPI with ARMCI/GA are:

setenv MPI_LIB $MPI_LOC/lib
setenv MPI_INCLUDE $MPI_LOC/include
setenv LIBMPI -lmpich
set path = ($MPI_LOC/bin  $path)

ARMCI

Settings for using Myrinet GM as network in ARMCI

setenv TARGET LINUX64
setenv ARMCI_NETWORK GM

setenv GM_LOC /home/software/gm
setenv GM_INCLUDE $GM_LOC/include
setenv GM_LIB $GM_LOC/lib
set path = ($path $GM_LOC/bin $GM_LOC/sbin)

Note: that ARMCI can only use GM when running with MPICH-GM

settings for using Mellanox Infiniband as network in ARMCI

setenv TARGET LINUX64
setenv ARMCI_NETWORK MELLANOX

setenv IB_INCLUDE /usr/mellanox/include
setenv IB_LIB /usr/mellanox/lib

Math Libraries

HP Math library installed at /opt/mlib/lib/linux

Running Jobs

Rules

Any jobs runing on the compute nodes must be submitted through PBS.
The front-end/login node should not be used to run computational jobs.

Procedures

For PBS interactive work two scripts are available in /home/software/bin

getprocs - allocates the specified number of processors from PBS and starts interactive shell
qmpirun - runs the specified MPI (over GM) program on the allocated nodes (2tasks/node) using Myrinet
qmpichrun - runs the specified MPI (over p4) program on the allocated nodes (2tasks/node) using sockets over Ethernet

The batch jobs are submitted through the regular PBS procedures. There is only one queue available.
The Maui scheduler commands are also available, e.g. "showq"

Disk Space Quota

The cluster has a quota system on the user home directories. The default limit is 1GB. The command /usr/bin/quota can be used to verify the limit on the user home directory.