The Global Arrays and ARMCI Support Page

• Seperate TARGET available for BlueGene/L systems (setenv TARGET BGL).
• You can download and build with the following settings. It may vary in your system. Please contact your GA sys admin or send an email to GA support.
• Example GA Build instructions on IBM BG/L:

• Here is a sample set of ENV Settings to build GA on Cray XT3/XT4. Please use appropriate settings for your environment. Contact your sys admin or GA Support if you need more information.
  
• If OS is CATAMOUNT on Compute Nodes
• export TARGET=CATAMOUNT
  
• If OS is Linux on Compute Nodes
• export TARGET=LINUX64
• (Required) export MA_USE_ARMCI_MEM=1
• Common Settings for CNL and CATAMOUNT
• (Required) Set Symmetric Heap Size. Some examples..
  
• export XT_SYMMETRIC_HEAP_SIZE=1G (i.e.1 GB of symmetric heap)
• export XT_SYMMETRIC_HEAP_SIZE=128M (i.e. 128 MB of symmetric heap)
• export USE_MPI=y
• export MPI_INCLUDE=$MPICH_DIR/include
  

• You must select one of the network protocol
• ARMCI_NETWORK=PORTALS (to use PORTALS)
  
• ARMCI_NETWORK=CRAY-SHMEM. Do not use -VN mode due to a shmem bug.
  

• GA versions >= 4.1 supports OpenIB. It has been tested with OFED 1.2.5.1. Please send a message to hpctools@pnl.gov with your system configuration details if you face any problems running on OFED 1.2.5.1 or later versions.
• Here is a sample set of ENV settings on a 64-bit Linux cluster to build GA.
  

        export TARGET=LINUX64
        export ARMCI_NETWORK=OPENIB

        # MPI Settings - Using MVAPICH here.
        export MPI_HOME=/usr/mpi/gcc/mvapich-0.9.9
        export MPI_LIB=$MPI_HOME/lib
        export MPI_INCLUDE=$MPI_HOME/include
        export LIBMPI="-lmpich"

        # IB Settings (path to "ofed")
        export IB_HOME=/opt/ofed
        export IB_INCLUDE=$IB_HOME/include
        export IB_LIB=$IB_HOME/lib64
        export IB_LIB_NAME="-libverbs -libumad -lpthread -lrt"

• When ARMCI_NETWORK is set to MPI-SPAWN, GA/ARMCI uses MPI-2's dynamic process management to spawn additional processes for communication (communication coprocesses).
  
• Due to the above reason, please compile GA using an MPI implementation that supports dynamic process management (i.e. MPI_Comm_spawn_multiple).
• On Cray XT systems, setting the following environment variable is required for MPI-SPAWN option
  
• export ARMCI_SPAWN_CRAY_XT=y
  

• GA has been tested with GM 2.0.6. Please send a message to hpctools@pnl.gov with your system configuration details if you face any problems running on GM 2.0.3 or later versions.

1. Due to a bug in the Elan library (versions between and including 1.3 and 1.4.1), programs using this library can crash when transmiting 4-byte C Floats. Please contact Quadrics support for further information about this problem. Since ARMCI uses the Elan library on the QSnet interconnect, this bug can cause a problem if your system has version on. This problem does not occur with the 1.4.2 version.
2. True64 (Compaq Alphaserver)
•  it might be required to set the Quadrics envrionment variable SHMEM_SMP_ENABLE=0 as a workaround for shmem_fadd problem, see the follwing note.
• The following environment settings are recommended: ARMCI_DEFAULT_SHMMAX=440  and  LIBELAN_ALLOC_SIZE=461373440

Linux

• The release 3.2 of GA supports  Intel (x86), PowerPC, Alpha, Itanium-1/2 and Sparc Ultra processors.

The Linux kernel has traditionally fairly small limit for the shared memory segment size (SHMMAX). In kernels 2.2.x it is 32MB on Intel, 16MB on Sun Ultra, and 4MB on Alpha processors. There are two ways to increase this limit:

• rebuild the kernel after changing SHMMAX in /usr/src/linux/include/asm-i386/shmparam.h, for example, setting SHMMAX as 0x8000000 (128MB)
• a system admin can increase the limit without rebuilding the kernel, for example: echo "134217728" >/proc/sys/kernel/shmmax
More info on this subject: http://ps-ax.com/shared-mem.html and on the Linux SMP discussion list.
Also see Note.
 
• Issues related to Myrinet
• On Linux/x86 clusters with Myrinet, the release of GM 1.4 leads to hangs in ARMCI and GA. This problem has been solved  in GM 1.4.1pre6 which is available on the Myricom ftp site. Versions 1.2, 1.3, 1.4pre48 of GM do not have that problem.
• With MPICH/GM versions >1.2.3, the GM version 1.4.1pre14 or higher must be used.

• POE environment variable settings for the parallel environment PSSP 3.1 (Troutbeck) and higher:

Global Arrays applications like any other LAPI-based codes must define MP_MSG_API=lapiorMP_MSG_API=mpi,lapi(when using GA and MPI)
• The LAPI-based implementation of GA cannot be used on the very old (made 5-7 years ago) SP-2 systems because LAPI does not support the TB2 switch used in these models.  If in doubt which switch you got use odmget command: odmget -q name=css0 CuDv
• under AIX 4.3.1 and later must set environment variable AIXTHREAD_SCOPE=S to assure correct operation of LAPI (IBM should do it in PSSP by default).
• under AIX 4.3.3 and later  an additional environment variable is required RT_GRQ=ON to restore the original thread scheduling LAPI relies upon.
• Enabling the recent (Fall, 2002) shared memory implementation of LAPI by setting the LAPI_USE_SHM variable to yes is not recommended due to possible stability issues.
• In AIX >= 5.2, set the environment varibale USE_OLDAIO=y to use legacy AIO, if new posix AIO is not always on by default.

If processors are oversubscribed (you are using more processes than processors), the SGI spin locks used for synchronization in GA are a very bad idea. In such a case Sys V semaphores work much better

SGI Altix
The default value of SMA_SYMMETRIC_SIZE is the total  machine  memory divided  by the number of processors on the system. If there is "shmalloc failed" error message when creating global array, increase the memory by setting the environment  variable SMA_SYMMETRIC_SIZE (in bytes) to the required amount not exceeding system limit. To know more about this variable.
    man shmalloc
    man intro_shmem

Sun
Solaris by default provides only 1MB limit for the largest shared memory segment. You need to increase  this value to do any useful work with GA. For example to make SHMMAX= 2GB, add either of the lines to /etc/system:
set shmsys:shminfo_shmmax=0x80000000 /* hexidecimal */
set shmsys:shminfo_shmmax=2147483648 /* decimal     */
After rebooting, you should be able to take advantage of the increased shared memory limits. For more info, please refer to the article in the SunWorld. Also see Note.

Compaq True64
Tru64 is another example of an OS with a pityfully small size of the shared memory limit. Here are instruction on how to modify shared memory max segment size to 256MB on  the Tru64 UNIX Version 4.0F:
1) create a file called /etc/sysconfig.shmmax
cat > /etc/sysconfig.shmmax << EOF
ipc:
shm-max = 268435456
EOF
You can check if the file created is OK by typing: /sbin/sysconfigdb -l -t /etc/sysconfig.shmmax
2) Modify kernel values: sysconfigdb -a -f /etc/sysconfig.shmmax ipc
3) Reboot
4) To check new values: /sbin/sysconfig -q ipc|egrep shm-max

More info is here.Also see Note.

HP-UX
In most HP-UX/11 installations, the defult limit  for the largest shared memory segment is 64MB. A system administrator should be able to easily increment this value to better suit the applications needs:
1) Start sam (HP System Administration Manager).
2) Go to Kernel Configuration, select Configurable Parameters. Click on the shmmax entry on the list.
3) From Actions menu select Modify Configurable Parameter. Click on the radio button Specify New Formula/Value and fill in a desired value for SHMMAX in the hexadecimal format e.g.,  0X8000000 (128MB).
4) Finally, select Process New Kernel from the Actions menu. It will apply the new value of SHHMAX into the kernel after rebooting the system.

The C compiler (commercial version) under HP-UX 10.2 performs incorrect address calculations for shared memory references. GNU gcc works fine. This problem manifests itself in global/testing/test.x failing. If it happens, GA has to recompiled with gcc.

Fujitsu VX/VPP
The code will not complile because of the missing header file. This file references MPlib library, a trade secret of Fujitsu. The GA/ARMCI binaries can be obtained from nobes@fecit.com.

Cray systems
  J90 and SV1 - MPI related issues:

•    Only MPI message-passing library is supported (TCGMSG-MPI also).
•    Must use "mpirun -nt" rather than "mpirun -np" command to run GA based codes.
•    The Cray MPI_Abort implementation is broken(hangs) - GA aborts using _exit().
•    MPI_Initialized() is also broken - we cannot detect if some other library already initialized MPI.

FreeBSD
To increase the shared memory segments on FreeBSD the following two sysctl's should be added to the startup scripts (e.g. /etc/rc.local):
sysctl -w kern.ipc.shmmax=67108864
sysctl -w kern.ipc.shmall=16384
the first sysctl allocates 64Mbytes of memory, the second does the same thing in 4k pages (4k * 16384 = 64M), you must set both sysctl.