Skip to Main Content U.S. Department of Energy
PNNL: Global Arrays Toolkit

Global Arrays and ARMCI Platform-Specific Notes

The purpose of this page is to provide updated information on the operating system configurations/settings that are relevant for the Global Arrays and ARMCI users on different platforms.

If you have questions or need support/help/report-bug, contact the GA/ARMCI developers at or visit our Google Group to see the collection of prior postings to the list.

Platforms

IBM BlueGene/P

  • Supported in GA 4.1 or higher.
  • For GA 5.0 and later
    configure --with-dcmf # for 32-bit integer mode
    configure --with-dcmf --enable-i8 # for 64-bit integer mode
  • For GA 4.3 and earlier
    make FC=mpixlf77 CC=mpixlc
    =============================================
    # GA/ARMCI Env Settings
    export TARGET=BGP
    export ARMCI_NETWORK=DCMFMPI
    export MSG_COMMS=DCMFMPI
    export USE_MPI=yes
    export BGCOMPILERS=/bgsys/drivers/ppcfloor/gnu-linux/bin
    export BGP_INSTALLDIR=/bgsys/drivers/ppcfloor
    export BGP_RUNTIMEPATH=/bgsys/drivers/ppcfloor/runtime
    export ARMCIDRV=${BGP_INSTALLDIR}
    export BGDRIVER=${ARMCIDRV}
    export MPI_LIB=${BGDRIVER}/comm/lib
    export MPI_INCLUDE=${BGDRIVER}/comm/include
    export LIBMPI="-L${MPI_LIB} -lfmpich_.cnk -lmpich.cnk -ldcmf.cnk -ldcmfcoll.cnk -lpthread -lrt -L${BGP_RUNTIMEPATH}/SPI -lSPI.cna"
    export BGMLMPI_INCLUDE=${MPI_INCLUDE}
    =============================================
    # GA/ARMCI OPTIONAL Env Settings
    export USE_INTEGER8=YES # to enable 64-bit integer mode

IBM BlueGene/L

  • For GA 5.0 and later
    configure --with-bgml # for 32-bit integer mode
    configure --with-bgml --enable-i8 # for 64-bit integer mode
  • For GA 4.3 and earlier
    gmake USE_INTEGER8=YES FC=blrts_xlf # (enables 64-bit integer mode)
    or
    gmake # (for plain 32 bit integer)
    =============================================
    # GA/ARMCI Env Settings
    export BGLSYS_DRIVER=/bgl/BlueLight/ppcfloor
    export BGLSYS_ROOT=${BGLSYS_DRIVER}/bglsys
    export BLRTS_GNU_ROOT=${BGLSYS_DRIVER}/blrts-gnu
    export BGDRIVER=${BGLSYS_DRIVER}
    export BGCOMPILERS=${BLRTS_GNU_ROOT}/bin
    export TARGET=BGL
    export ARMCI_NETWORK=BGMLMPI
    export USE_MPI=yes
    export MPI_LIB=${BGLSYS_ROOT}/lib
    export MPI_INCLUDE=${BGLSYS_ROOT}/include
    export LIBMPI='-lfmpich_.rts -lmpich.rts -lmsglayer.rts -lrts.rts -ldevices.rts'

Cray XT4/XT5

  • For GA 5.0 and later
    • configure --with-portals
    • If OS is Linux on Compute Nodes
      • (Required) export MA_USE_ARMCI_MEM=1
    • (Required) Set Symmetric Heap Size. Some examples..
      • export XT_SYMMETRIC_HEAP_SIZE=1G (i.e.1 GB of symmetric heap)
      • export XT_SYMMETRIC_HEAP_SIZE=128M (i.e. 128 MB of symmetric heap)
  • Prior to GA 5.0
    • If OS is CATAMOUNT on Compute Nodes
      • export TARGET=CATAMOUNT
    • If OS is Linux on Compute Nodes
      • export TARGET=LINUX64
      • (Required) export MA_USE_ARMCI_MEM=1
    • Common Settings for CNL and CATAMOUNT
      • (Required) Set Symmetric Heap Size. Some examples..
        • export XT_SYMMETRIC_HEAP_SIZE=1G (i.e.1 GB of symmetric heap)
        • export XT_SYMMETRIC_HEAP_SIZE=128M (i.e. 128 MB of symmetric heap)
      • export USE_MPI=y
      • export MPI_INCLUDE=$MPICH_DIR/include
    • You must select one of the network protocol
      • ARMCI_NETWORK=PORTALS (recommended)
      • ARMCI_NETWORK=CRAY-SHMEM. Do not use -VN mode due to a shmem bug.
    • Issues:
      • If GA accumulate calls are hanging or slow, please set SHMEM_SWAP_BACKOFF variable.
      • Example: export SHMEM_SWAP_BACKOFF=100

Infiniband/OpenIB

  • GA/ARMCI is not supported on QLogic hardware as QLogic's OFED interfaces don't work well with GA/ARMCI.
  • GA versions >= 4.1 supports OpenIB. It has been tested with OFED 1.2.5.1. Please send a message to with your system configuration details if you face any problems running on OFED 1.2.5.1 or later versions.
  • Here is a sample set of ENV settings on a 64-bit Linux cluster to build GA.

    export TARGET=LINUX64
    export ARMCI_NETWORK=OPENIB

    # MPI Settings - Using MVAPICH here.
    export MPI_HOME=/usr/mpi/gcc/mvapich-0.9.9
    export MPI_LIB=$MPI_HOME/lib
    export MPI_INCLUDE=$MPI_HOME/include
    export LIBMPI="-lmpich"

    # IB Settings (path to "ofed")
    export IB_HOME=/opt/ofed
    export IB_INCLUDE=$IB_HOME/include
    export IB_LIB=$IB_HOME/lib64
    export IB_LIB_NAME="-libverbs -libumad -lpthread -lrt"

Compiling GA/ARMCI with ARMCI_NETWORK=MPI-MT

  • When ARMCI_NETWORK is set to MPI-MT (configure --with-mpi-mt), GA/ARMCI uses additional threads for communication (communication coprocesses) which make MPI calls.
  • Due to the above reason, please compile GA using an MPI implementation that supports MPI_Init_thread() and a required threading level of MPI_THREAD_MULTIPLE.

Compiling GA/ARMCI with ARMCI_NETWORK=MPI-SPAWN

  • When ARMCI_NETWORK is set to MPI-SPAWN, GA/ARMCI uses MPI-2's dynamic process management to spawn additional processes for communication (communication coprocesses).
    • Due to the above reason, please compile GA using an MPI implementation that supports dynamic process management (i.e. MPI_Comm_spawn_multiple).
  • On Cray XT systems, setting the following environment variable is required for MPI-SPAWN option
    • export ARMCI_SPAWN_CRAY_XT=y

Myrinet

  • MX: MX does not support one sided communication. So please use ARMCI_NETWORK=MPI-SPAWN option. See previous section for MPI-SPAWN support.
  • GM2: GA has been tested with GM 2.0.6. Please send a message to with your system configuration details if you face any problems running on GM 2.0.3 or later versions.

Systems with QSnet Interconnects

  1. Due to a bug in the Elan library (versions between and including 1.3 and 1.4.1), programs using this library can crash when transmiting 4-byte C Floats. Please contact Quadrics support for further information about this problem. Since ARMCI uses the Elan library on the QSnet interconnect, this bug can cause a problem if your system has version on. This problem does not occur with the 1.4.2 version.
  2. True64 (Compaq Alphaserver)
    • it might be required to set the Quadrics envrionment variable SHMEM_SMP_ENABLE=0 as a workaround for shmem_fadd problem, see the follwing note.
    • The following environment settings are recommended: ARMCI_DEFAULT_SHMMAX=440 and LIBELAN_ALLOC_SIZE=461373440

Linux

  • The release 3.2 of GA supports Intel(x86), PowerPC, Alpha, Itanium-1/2 and Sparc Ultra processors.
    • The Linux kernel has traditionally fairly small limit for the shared memory segment size (SHMMAX). In kernels 2.2.x it is 32MB on Intel, 16MB on Sun Ultra, and 4MB on Alpha processors. There are two ways to increase this limit:
      • rebuild the kernel after changing SHMMAX in /usr/src/linux/include/asm-i386/shmparam.h, for example, setting SHMMAX as 0x8000000 (128MB)
      • a system admin can increase the limit without rebuilding the kernel, for example: echo "134217728" >/proc/sys/kernel/shmmax
      More info on this subject: http://ps-ax.com/shared-mem.html and on the Linux SMP discussion list.
      Also see Note.
  • Issues related to Myrinet
    • On Linux/x86 clusters with Myrinet, the release of GM 1.4 leads to hangs in ARMCI and GA. This problem has been solved in GM 1.4.1pre6 which is available on the Myricom ftp site. Versions 1.2, 1.3, 1.4pre48 of GM do not have that problem.
    • With MPICH/GM versions >1.2.3, the GM version 1.4.1pre14 or higher must be used.

IBM SP

  • POE environment variable settings for the parallel environment PSSP 3.1 (Troutbeck) and higher:

    Global Arrays applications like any other LAPI-based codes must define MP_MSG_API=lapiorMP_MSG_API=mpi,lapi (when using GA and MPI)
  • The LAPI-based implementation of GA cannot be used on the very old (made 5-7 years ago) SP-2 systems because LAPI does not support the TB2 switch used in these models. If in doubt which switch you got use odmget command: odmget -q name=css0 CuDv
  • under AIX 4.3.1 and later must set environment variable AIXTHREAD_SCOPE=S to assure correct operation of LAPI (IBM should do it in PSSP by default).
  • under AIX 4.3.3 and later an additional environment variable is required RT_GRQ=ON to restore the original thread scheduling LAPI relies upon.
  • Enabling the recent (Fall, 2002) shared memory implementation of LAPI by setting the LAPI_USE_SHM variable to yes is not recommended due to possible stability issues.
  • In AIX >= 5.2, set the environment varibale USE_OLDAIO=y to use legacy AIO, if new posix AIO is not always on by default.

SGI

In older versions of GA (<3.1) there is a possibility of conflict between the SGI implementation of MPI (but not others, MPICH for example) and ARMCI in their use of the SGI specific interprocessor communication facility called arena.The release 3.1 does not use arenas to avoid the problem.

If processors are oversubscribed (you are using more processes than processors), the SGI spin locks used for synchronization in GA are a very bad idea. In such a case Sys V semaphores work much better

SGI Altix

The default value of SMA_SYMMETRIC_SIZE is the total machine memory divided by the number of processors on the system. If there is "shmalloc failed" error message when creating global array, increase the memory by setting the environment variable SMA_SYMMETRIC_SIZE (in bytes) to the required amount not exceeding system limit. To know more about this variable.
     man shmalloc
     man intro_shmem

Sun

Solaris by default provides only 1MB limit for the largest shared memory segment. You need to increase this value to do any useful work with GA. For example to make SHMMAX= 2GB, add either of the lines to /etc/system:
set shmsys:shminfo_shmmax=0x80000000 /* hexidecimal */
set shmsys:shminfo_shmmax=2147483648 /* decimal */
After rebooting, you should be able to take advantage of the increased shared memory limits. For more info, please refer to the article in the SunWorld. Also see Note.

Compaq True64

Tru64 is another example of an OS with a pityfully small size of the shared memory limit. Here are instruction on how to modify shared memory max segment size to 256MB on the Tru64 UNIX Version 4.0F:

  1. create a file called /etc/sysconfig.shmmax
    cat > /etc/sysconfig.shmmax << EOF
    ipc:
    shm-max = 268435456
    EOF
    You can check if the file created is OK by typing: /sbin/sysconfigdb -l -t /etc/sysconfig.shmmax
  2. Modify kernel values: sysconfigdb -a -f /etc/sysconfig.shmmax ipc
  3. Reboot
  4. To check new values: /sbin/sysconfig -q ipc|egrep shm-max

More info is here. Also see Note.

HP-UX

In most HP-UX/11 installations, the defult limit for the largest shared memory segment is 64MB. A system administrator should be able to easily increment this value to better suit the applications needs:

  1. Start sam (HP System Administration Manager).
  2. Go to Kernel Configuration, select Configurable Parameters. Click on the shmmax entry on the list.
  3. From Actions menu select Modify Configurable Parameter. Click on the radio button Specify New Formula/Value and fill in a desired value for SHMMAX in the hexadecimal format e.g., 0X8000000 (128MB).
  4. Finally, select Process New Kernel from the Actions menu. It will apply the new value of SHHMAX into the kernel after rebooting the system.

The C compiler (commercial version) under HP-UX 10.2 performs incorrect address calculations for shared memory references. GNU gcc works fine. This problem manifests itself in global/testing/test.x failing. If it happens, GA has to recompiled with gcc.

Fujitsu VX/VPP

The code will not complile because of the missing header file. This file references MPlib library, a trade secret of Fujitsu. The GA/ARMCI binaries can be obtained from .

Cray systems

J90 and SV1 - MPI related issues:

  • Only MPI message-passing library is supported (TCGMSG-MPI also).
  • Must use "mpirun -nt" rather than "mpirun -np" command to run GA based codes.
  • The Cray MPI_Abort implementation is broken(hangs) - GA aborts using _exit().
  • MPI_Initialized() is also broken - we cannot detect if some other library already initialized MPI.

FreeBSD

To increase the shared memory segments on FreeBSD the following two sysctl's should be added to the startup scripts (e.g. /etc/rc.local):
sysctl -w kern.ipc.shmmax=67108864
sysctl -w kern.ipc.shmall=16384
the first sysctl allocates 64Mbytes of memory, the second does the same thing in 4k pages (4k * 16384 = 64M), you must set both sysctl.

Note on SHMMAX

Setting very large values of SHMMAX is not recommended. In particular, it should not be bigger than the swap space or even the amount of physical memory in the system. For most applications, 128-256MB values should work fine.

Global Arrays

News and Events

Feedback