2. Writing, Building and Running GA Programs

The web pagewww.emsl.pnl.gov/docs/global/support.html  contains updated information about using GA on different platforms. Please refer to this page frequently for most recent updates and platform information.

2.1 Platform and Library Dependencies

2.1.1 Supported Platforms

Older versions of GA supported some additional (now obsolete) platforms such as: IPSC, KSR, PARAGON, DELTA, CONVEX. They are not supported in the newer (>3.1) versions because we do not have access to these systems. We recommend using GA 2.4 on these platforms.

For most of the platforms, there are two versions available: 32-bit and 64-bit. This table specifies valid TARGET names for various supported platforms.
 
Platform
 32-bit TARGET name
64-bit TARGET name
 Remarks
Sun ultra  SOLARIS  SOLARIS64  64-bit version added in GA 3.1
IBM BlueGene/PBGPsupported in GA 4.1 and later (Contact your BlueGene sys admin for GA instalation). More info in support page...
IBM BlueGene/L
BGL
 added in GA 4.0.2 (Contact your BlueGene sys admin for GA instalation). More info in support page...
Cray XT3/XT4
LINUX64 (or)
CATAMOUNT
 TARGET based on the OS in compute Nodes (Catamount/Linux). More info and sample settings in support page...
IBM RS/6000   IBM  IBM64  64-bit version added in GA 3.1
IBM SP  LAPI  LAPI64
 no support yet for user-space communication in the 64-bit mode by IBM
Compaq/DEC alpha  not available  DECOSF
HP pa-risc  HPUX  HPUX64  64-bit version added in GA 3.1
Linux (32-bit):
x86, ultra, powerpc		  LINUX  not available
Linux (64-bit):
ia64 (Itanium),
x86_64 (Opteron), ppc64, etc
  not available  LINUX64
Linux alpha  not available  LINUX64 64-bit version added in GA 3.1; 
Compaq compilers rather than GNU required
Cray T3E  not available  CRAY-T3E
Cray J90  not available  CRAY-YMP
Cray SV1  not available  CRAY-SV1
Cray X1
  not available
  cray-sv2
 In X1,  by default, TARGET is defined by the operating system as cray-sv2
SGI IRIX mips  SGI_N32, SGI  SGITFP
Hitachi SR8000  HITACHI  not available
Fujitsu VPP systems  FUJITSU-VPP  FUJITSU-VPP64  64-bit version added in GA 3.
NEC SX series
  
  NEC
Apple
 MACX
 MACX64
  Running MAC X or higher

To aid development of fully portable applications, in 64-bit mode Fortran integer datatype is 64-bits. It is motivated by 1) the need of applications to use very large data structures and 2) Fortran INTEGER*8 not being fully portable. The 64-bit representation of integer datatype is accomplished by using appropriate Fortran compiler flag.

Because of limited interest in heterogenous computing among known us GA users, the Global Array library still does not support heterogeonous platforms. This capability can be added if required by new applications.
 

2.1.2 Selection of the communication network for ARMCI

Some cluster installations can be equipped with a high performance network which offer instead, or in addition to TCP/IP some special communication protocol, for example GM on Myrinet network. To achieve high performance in Global Arrays, ARMCI must be built to use these protocols in its implementation of one-sided communication. Starting with GA 3.1, this is accomplished by setting an environment variable ARMCI_NETWORK to specify the protocol to be used. In addition, the it might be necessary to provide location for the header files and library path corresponding to location of s/w supporting the appropriate protocol API, see g/armci/config/makecoms.h for details.
 
 Network  Protocol name  ARMCI_NETWORK setting  Supported platforms
 Ethernet  TCP/IP  SOCKETS   (optional/default)  workstation clusters (32 and 64-bit)
 Quadrics/QsNet  Elan3/Shmem  QUADRICS or ELAN3
  Linux (alpha,x86,IA64,..), Compaq
 Quadrics/QsNet II
  Elan4
  ELAN4
  Linux (32 and 64-bit)
 Infiniband
  OpenIB
  OPENIB  Linux (32 and 64-bit).
NOTE: This network is supported in GA versions>=4.1. For more info see support page...
 Infiniband  VAPI
  MELLANOX
  Linux  (32 and 64-bit)
 Myrinet  GM  GM  Linux  (x86,ultra,IA64)
 Giganet cLAN  VIA  VIA  Linux (32 and 64-bit)
 MPI MPI-SPAWNSupported in GA 4.1 or higher. This network setting can be used on any platform that has MPI-2 dynamic process management support. Using this setting is recommended only if your network is not listed above.

Other Platforms: (More settings info for these platforms in support page)

 Platforms
  Protocol Name
  ARMCI_NETWORK Setting
IBM BG/L
  BGML
  BGMLMPI
Cray XT3/XT4
  Shmem
 Portals
  CRAY-SHMEM
 PORTALS

2.1.3 Selection of the message-passing library

As explained in Section 3, GA works with either MPI or TCGMSG message-passing libraries. That means that GA applications can use either of these interfaces. Selection of the message-passing library takes place when GA is built. Since the TCGMSG library is small and compiles fast, it is included with the GA distribution package and built on Unix workstations by default so that the package can be built as fast and as convenientlly to the user as possible. There are three possible configurations for running GA with the message-passing libraries:
  1. GA with MPI (recommended): directly with MPI. In this mode, GA program should contain MPI initialization calls.
  2. GA with TCGMSG-MPI (MPI and TCGMSG emulation library): TCGMSG-MPI implements functionality of TCGMSG using MPI. In this mode, the message passing library is initialized using a TCGMSG PBEGIN(F) call which internally references MPI_Initialize. To enable this mode, define the environmental variable USE_MPI.
  3. GA with TCGMSG: directly with TCGMSG. In this mode, GA program should contain TCGMSG initialization calls.
For the MPI versions, the optional environmental variables MPI_LIB and MPI_INCLUDE are used to point to the location of the MPI library and include directories if they are not in the standard system location(s). GA programs are started with the mechanism that any other MPI programs use on the given platform.

The recent versions of MPICH (an MPI implementation from ANL/Mississippi State) keep the MPI header files in more than one directory and provide compiler wrappers that implicitly point to the appropriate header files. One can :

One disadvantage of the second approach it that GA makefile in some circumstances might be not able to determine which compiler (e.g., GNU or PGI) is called underneath by the MPICH compiler wrappers. Since different compilers provide different Fortran/C interface, the package might fail to build. This problem is most likely to occur on non-Linux Unix systems with non-native compilers (e.g., gcc).

On Windows NT, the current version of GA was tested with WMPI, an NT implementation derived from MPICH in Portugal.

2.1.3 Dependencies on other software

In addition to the message-passing library, GA requires:


GA may also depend on other software depending on the functions being used.

2.2 Writing GA Programs

C programs that use Global Arrays should include files `global.h',  'ga.h',  `macdecls.h'. Fortran programs should include the files `mafdecls.fh', `global.fh'. Fortran source must be preprocessed as a part of compilation.

The GA program should look like:

      call mpi_init(..)   MPI_Init(..)      ! start MPI
      call ga_initialize() GA_Initialize()   ! start global arrays
      status = ma_init(..) MA_Init(..)       ! start memory allocator

      .... do work           .... do work

      call ga_terminate() GA_Terminate()    ! tidy up global arrays
      call mpi_finalize() MPI_Finalize()    ! tidy up MPI
      stop                                      ! exit program

      call pbeginf()     PBEGIN_(..)       ! start TCGMSG
      call ga_initialize() GA_Initialize()   ! start global arrays
      status = ma_init(..) MA_Init(..)       ! start memory allocator

      .... do work       .... do work

      call ga_terminate() GA_Terminate()    ! tidy up global arrays
      call pend()        PEND_()           ! tidy up tcgmsg
      stop                                     ! exit program
 

The ma_init call looks like :

      status = ma_init(type, stack_size, heap_size)

and it basically just goes to the OS and gets stack_size+heap_size elements of size type. The amount of memory MA allocates need to be sufficient for storing global arrays on some platforms. Please refer to section 3.3.1 for the  details and information on more advanced usage of MA in GA programs.

2.3 Building GA

Use GNU make to build the GA library and application programs on Unix and Microsoft nmake on Windows. The structure of the available makefiles are The user must specify TARGET as an environment variable (setenv TARGET TARGET_name) or in the GNUmakefile or on the command line when calling make. For example:
           setenv TARGET LAPI  (for IBM/SP platform)
          (or) from command line, gmake TARGET=LAPI

Valid TARGET_name for various supported platforms can be found in the above table. Valid TARGETs can also be listed by calling make in the top level distribution directory on UNIX family of systems when TARGET is not defined. On Windows, WIN32, CYGNUS and INTERIX (previously known as OpenNT) are supported. 

Compiler Settings (optional): For various supported platforms, the default compilers and compiler options are specified in config/makefile.h. One could change the predefined default compilers and compiler flags in GA package either by specifying them on the command line or in the file config/makefile.h. Note: editing config/makefile.h for any platform requires extra care and intended for intermediate/advanced users.

For example,
gmake FC=f90 CC=cc FOPT=-O4 COPT=-g
Note that GA provides only Fortran-77 interfaces. To use and compile with a Fortran 90 compiler, it has to support a subset of Fortran-77.

2.3.1 Unix Environment

As mentioned in earlier section, there are three possible configurations for building GA.

1. GA with MPI (recommended): To build GA directly with MPI, the user needs to define environmental variables MPI_LIB and MPI_INCLUDE which should point to the location of the MPI library and include directories. Additionally, the make/environmental variable MSG_COMMS must  be defined as MSG_COMMS = MPI. (In csh/tcsh,  setenv MSG_COMMS MPI)

2. GA with TCGMSG-MPI: To build GA with the TCGMSG-MPI, user needs to define environmental variables USE_MPI, MPI_LIB and MPI_INCLUDE which should point to the location of the MPI library and include directories.

   Example: using csh/tcsh (assume using MPICH installed in /usr/local on IBM workstation)

      setenv USE_MPI y
      setenv MPI_LOC /usr/local/mpich
      setenv MPI_LIB $MPI_LOC/lib/rs6000/ch_shmem
      setenv MPI_INCLUDE $MPI_LOC/include

3. GA with TCGMSG: To build GA directly with TCGMSG, the user must define the environmental variable MSG_COMMS=TCGMSG.
    Note:  When MSG_COMMS=TCGMSG, make sure to unset the environment variable USE_MPI (e.g. unsetenv USE_MPI).


After chosing the configuration, to build the GA library, type
        make or gmake

If the build is successfull, a test program test.x will be created in global/testing directory. Refer Section "Running GA programs" on how to run this test.

To build an application based on GA located in g/global/testing, for example, the application's name is app.c (or app.F, app.f), type
        make app.x or gmake app.x

Please refer to compiler flags in file g/config/makefile.h to make sure that Fortran and C compiler flags are consistent with flags use to compile your application. This may be critical when Fortran compiler flags are used to change the default length of the integer datatype.

Interface to ScaLAPACK: GA interface routines to ScaLAPACK are only available, when GA is build with MPI and ScaLAPACK. Before building GA, the user is required to define the environment variables USE_SCALAPACK or USE_SCALAPACK_I8 (for scalapack libraries compiled with 8-byte integers), and the location of ScaLAPACK & Co. libraries in the env variable SCALAPACK.

   Example: using csh/tcsh

      setenv USE_SCALAPACK y  (or)  setenv USE_SCALAPACK_I8 y
      setenv SCALAPACK '-L/msrc/proj/scalapack/LIB/rs6000
                -lscalapack -lpblas -ltools -lblacsF77cinit -lblacs'
      setenv USE_MPI y

Since there are certain interdependencies between blacs and blacsF77cinit, some system might require specification of -lblacs twice to fix the unresolved external symbols from these libs.

Installing GA C++ Bindings: By default, GA C++ bindings are not built. GA++ is built only if GA_C_CORE is defined as follows:
    setenv GA_C_CORE y
    cd GA_HOME
    make clean; make (This will build GA with C core and C++ bindling).

Using GA_C_CORE: GA's internal core is implemented using Fortran and C. When GA_C_CORE is set, core Fortran functionalites are replaced by their C counterparts to eliminate the hassle involved in mixing Fortran and C with C++ bindings on certain platforms or for some compilers (like, missing Fortran symols/libraries during the linking phase). NOTE: C and C++ compilers should be from the same family. GA_C_CORE doesnot support mixing C and C++ compilers (e.g.using Intel compiler for C and GNU compiler for C++).
    make FC=ifort CC=icc CXX=g++ (not supported if GA_C_CORE is set)
    make FC=ifort CC=icc CXX=icpc (Intel compiler family - supported)

2.3.2 Windows NT

To buid GA on Windows NT, MS Power Fortran 4 or DEC Visual Fortran 5 or later, and MS Visual C 4 or later are needed. Other compilers might need the default compilation flags modified. When commercial Windows compilers are not available, one can choose to use CYGNUS or INTERIX and build it as any other Unix box using GNU compilers.

First of all, one needs to set environment variables (same as in Unix enviroment). GA needs to know where find the MPI include files and libraries. To do this, select the Environment tab under the Control Panel, then set the variables to point to the location of MPI, for example for WMPI on disk D:

set MPI_INCLUDE as d:\Wmpi\Include
set MPI_LIB as d:\Wmpi\Console
Make sure that the dynamic link libraries required by the particular implementation of MPI are copied to the appropriate location for the system DLLs. For WMPI, copy VWMPI.dll to \winnt.

In the top directory do,

nmake
The GA test.exe program can be built in the g\global\testing directory:
nmake test.exe
In addition,  the HPVM package from UCSD offers the GA interface  in the NT/Myrinet cluster environment.

GA could be built on Windows 95/98. However, due to the DOS shell limitations, the top level NTmakefile will not work. Therefore, each library has to be made separately in its own directory. The environment variables referring to MPI can be hardcoded in the NT makefiles.

2.3.3 Writing and building new GA programs

For small programs contained in a single file, the most convenient approach is to put your program file into the g/global/testing directory. The existing GNU make suffix rules would build an executable with the ".x" suffix from any C or Fortran source file. You do not have to modify makefiles in g/global/testing at all.  For example, if your program is contained in myfile.c or myfile.F and you place it in that directory, all you need to do to create an executable called myfile.x  is to type: make myfile.x .

Windows nmake is not as powerful as GNU make - you would need to modify the NT makefile.

This approach obviously is not feasible for large packages that contain multiple source files and directories. In that case you need to provide apropriate definitions  in your makefile:

Starting with GA 3.1, one could simplify linking of applications by including g/armci/config/makecoms.h and g/armci/config/makemp.h that define all the necessary platform specific libraries that are required by GA.

2.4 Running GA Programs

Assume the GA program app.x had already been built. To run it,

Running on shared memory systems and clusters: (i.e. network of workstations/linux clusters)
    If the app.x is built based on MPI, run the program the same way as any other MPI programs.
Example: to run on four processes on clusters, use
   
    mpirun -np 4 app.x

    Example: If you are using MPICH (or MPICH-like Implementations), and mpirun requires a machinefile or hostfile, then run the GA program same as any other MPI programs. The only change required is to make sure the hostnames are specified in a consecutive manner in the machinefile. Not doing this, will prevent SMP optimizations and would lead to poor resource utilization.

        mpirun -np 4 -machinefile machines.txt app.x

    Contents of machines.txt: (Let us say we have two 2-way SMP nodes (host1 and host2, and correct formats for a 4-processor machinefile is shown in the below table)

correct
 correct
 incorrect
host1
host1
host2
host2
 host2
host2
host1
host1		 host1
host2
host1 (This is wrong. Same hosts should be specified together
host2


    If app.x is built based on TCGMSG (not including, Fujitsu, Cray J90, and Windows, because there are no native ports of TCGMSG), to execute the program on Unix workstations/servers, one should use the 'parallel' program (built in tcgmsg/ipcv4.0). After building the application, a file called 'app.x.p' would also be generated (If there is not such a file, make it: make app.x.p). This file can be edited to specify how many processors and tasks to use, and how to load the executables. Make sure that 'parallel' is accessible (you might copy it into your 'bin' directory).  To execute, type:

           parallel app.x  

  1. On MPPs, such as Cray XT3/XT4, or IMB SPs, use appropriate system command to specify the number of processors, load and run the programs.

    Example:

  2. On Microsoft NT, there is no support for TCGMSG, which means you can only build your application based on MPI. Run the application program the same way as any other MPI programs. For, WMPI you need to create the .pg file.

    Example:  

        R:\nt\g\global\testing> start /b test.exe