next up previous contents
Next: 7. Utilities Up: NWCHEM Programmer's Guide, Release Previous: 5. Integral Application Programmer's   Contents

Subsections

6. Software Development Toolkit

The Software Development Toolkit is the foundation of the functional architecture in NWChem. It consists of various useful elements for memory management and data manipulation that are needed to facilitate the development of parallel computational chemistry algorithms. The memory management elements implement the NUMA memory management module for efficient execution in parallel enviroments and provides the means for interfacing between the calculation modules of the code and the system hardware. Efficient data manipulation is accomplished using the runtime data base, which stores the information needed to run particular calculations and allows different modules to have access to the same information. This chapter describes the various elements of the Software Development Toolkit in detail.


6.1 Non-Uniform Memory Allocation (NUMA)

All computers have several levels of memory, with parallel computers generally having more than computers with only a single processor. Typical memory levels in a parallel computer include the processor registers, local cache memory, local main memory, and remote memory. If the computer also supports virtual memory, local and remote disk memory are added to this heirarchy. These levels vary in size, speed, and method of access, and in NWChem the differences among them are lumped under the general concept Non-Uniform Memory Access (NUMA). This approach allows the developer to think of all memory anywhere in the system as accessible to any processor as needed. It is then possible to focus independently on the questions of memory access methods and memory access costs. Memory access methods are determined by the programming model and available tools and the desired coding style for an application. Memory access costs are determined by the program structure and the performance characteristics of the computer system. The design of a code's major algorithms, therefore, is critical to the creation of an efficient parallel program.

In order to scale to massively parallel computer architectures in all aspects of the hardware (i.e., CPU, disk, and memory), NWChem uses Non-Uniform Memory Access to distribute the data across all nodes. Memory access is achieved through explicit message passing using the TCGMSG interface. The Memory Allocator (MA) tool is used to allocate memory that is local to the calling process. The Global Arrays (GA) tool is used to share arrays between processors as if the memory were physically shared. The complex I/O patterns required to accomplish efficient memory management are handled with the abstract programming interface ChemIO.

The following subsections discuss the TCGMSG message passing tool, the Memory Allocator library, the Global Arrays library, and ChemIO, and describe how they are used in NWChem.

6.1.1 Message Passing

TCGMSG6.1is a toolkit for writing portable parallel programs using a message passing model. It is relatively simple, having limited functionality that includes point-to-point communication, global operations, and a simple load-balancing facility, and was designed with chemical applications in mind. This simplicity contributes to the robustness of TCGMSG and its expemlary portability, and also to its high performance for a wide range of problem sizes.

The model used by TCGMSG operates as if it is sending a block until the message is explicitly received, and the messages from a particular process can be received only in the order sent. Processes should be thought of as being connected with ordered synchronous channels, even though messages are actually sent without any synchronization between sender and receiver, so far as buffering permits. The amount of buffering is greatly dependent on the mechanism used by the particular platform, so it is best not to count on this feature. Detailed information that includes documentation of the programming interface is available on-line as part of the EMSL webpage, at

       /docs/parsoft/tcgmsg/

A more general tool for message passing is MPI, which includes concepts such as process groups, communication contexts, and virtual topologies. Process groups can be used to specify that only certain processes are involved in a particular task, or to allow separate groups of processes to work on different tasks. Communication context provides an additional criterion for message selection, enhancing internal communication flexibility without incurring conflicts with other modules. MPI has been implemented in NWChem as an alternative to TCGMSG, and the code can be compiled with this option specified. However, it is not an undertaking for the faint of heart and it is highly advisable to contact nwchem-support@emsl.pnl.gov before trying this option.

The TCGMSG-MPI library is distributed with the Global Arrays package. This library is an implementation of the TCGMSG message passing inteface on top of MPI and system-specific resources. Using this library, it is possible to use both MPI and TCGMSG interfaces in the same application. TCGMSG offers a much smaller set of operations than MPI, but these include some unique capabilties, such as

The nxtval operation is implemented in TCGMSG-MPI in different ways, depending on the platform.

Detailed information that includes documentation of the programming interface is available on-line as part of the EMSL webpage, at

       /docs/parsoft/tcgmsg-mpi/

6.1.2 Memory Allocator (MA)

The Memoray Allocator (MA) is used to allocate data that will generally not be directly shared with other processes, such as workspace for a particular local calculation or for replication of very small sets of data. The MA tool is a library of routines that comprises a dynamic memory allocator for use by C, FORTRAN, or mixed-language applications. It provides both heap and stack memory management disciplines, debugging and verification support (for detecting memory leaks, for example), usage statistics, and quantitative memory availability information.

Applications written in FORTRAN require this sort of library because the language does not support dynamic memory allocation. Applications written in C can benefit from using MA instead of the ordinary malloc() and free() routines because of the extra features MA provides, which include both heap and stack memory management disciplines, debugging and verification support, usage statistics, and quantitative memory availability information. MA is designed to be portable across a large variety of platforms.

Detailed information on specific routines is available in the MA man pages. This can be accessed by means of the command, man ma. (Note: this will work only if the local environmental variable MANPATH includes the path $(NWCHEM_TOP)/src/man/ma/man. See Section 8.3 for information on system and environmental requirements for running NWChem.) The following subsections present a summary list of the MA routines, and a brief discussion of the implementation of this feature.

6.1.2.1 MA Data Types

All MA memory must be explicitly assigned a specific type by defining each data item in units of integer, logical, double precision, or character words. The type of data is specified in arguments using predefined Fortran parameters (or macros in C). These parameters are available in the include files mafdecls.fh in Fortran and in macdecls.h in C. The parameters are typed as follows:

MT_INT -- integer
MT_DBL -- double precision
MT_LOG -- logical
MT_CHAR -- character*1

6.1.2.2 Implementation

To access required MA definitions, C applications should include macdecls.h and FORTRAN applications should include mafdecls.fh. These are public header files for a dynamic memory allocator, and are included in the .../src/ma subdirectory of the NWChem directory tree. The files contain the type declarations and parameter declarations for the datatype constants, and define needed functions and variable types.

The memory allocator uses the following memory layout definitions:

A segment of memory is obtained from the OS upon initialization. The low end of the segment is managed as a heap. The heap region grows from low addresses to high addresses. The high end of the segment is managed as a stack. The stack region grows from high addresses to low addresses.

Each region consists of a series of contiguous blocks, one per allocation request, and possibly some unused space. Blocks in the heap region are either in use by the client (allocated and not yet deallocated) or not in use by the client (allocated and already deallocated). A block on the rightmost end of the heap region becomes part of the unused space upon deallocation. Blocks in the stack region are always in use by the client, because when a stack block is deallocated, it becomes part of the unused space.

A block consists of the client space, i.e., the range of memory available for use by the application. Guard words adjacent to each end of the client space to help detect improper memory access by the client. Bookkeeping information is stored(?) in an "allocation descriptor" AD Two gaps, each zero or more bytes long, are defined to satisfy alignment constraints (specifically, to ensure that AD and client_space are aligned properly).

6.1.2.3 List of MA routines

All MA routines are shown below, grouped by category and listed alphabetically within each category. The FORTRAN interface is given here. Information on the the C interface are available in the man pages. (The man pages also contain more detailed information on the arguments for these routines.)

Initialization:

Allocation:

Deallocation:

Debugging:

Iteration Over Allocated Blocks:

Statistics:

6.1.2.4 MA Errors

Errors considered fatal by MA result in program termination. Errors considered nonfatal by MA cause the MA routine to return an error value to the caller. For most boolean functions, false is returned upon failure and true is returned upon success. (The boolean functions for which the return value means something other than success or failure are MA_set_auto_verify(), MA_set_error_print(), and MA_set_hard_fail().) Integer functions return zero upon failure; depending on the function, zero may or may not be distinguishable as an exceptional value.

An application can force MA to treat all errors as fatal via MA_set_hard_fail().

If a fatal error occurs, an error message is printed on the standard error (stderr). By default, error messages are also printed for nonfatal errors. An application can force MA to print or not print error messages for nonfatal errors via MA_set_error_print().

6.1.3 Global Arrays (GA)

Globally addressable arrays have been developed to simplify writing portable scientific software for both shared and distributed memory computers. Programming convenience, code extensibility and maintainability are gained by adopting the shared memory programming model. The Global Array (GA) toolkit provides an efficient and portable "shared memory" programming interface for distributed memory computers. Each process in a MIMD parallel program can asynchronously access logical blocks of physically distributed matrices without need for explicit cooperation by other processes. The trade-off with this approach is that access to shared data will be slower than access to local data, and the programmer must be aware of this in designing modules.

From the user perspective, a global array can be used as if it was stored in the shared memory. Details of the data distribution, addressing and communication are encapsulated in the global array objects. However, the information on the actual data distribution can be obtained and taken advantage of whenever data locality is important.

The Global Arrays tool has been designed to complement the message-passing programming model. The developer can use both shared memory and message passing paradigms in the same program, to take advantage of existing message-passing software libraries such as TCGMSG. This tool is also compatible with the Message Passing Interface (MPI). The Global Arrays toolkit has been in the public domain since 1994 and is actively supported. Additional documentation and information on performance and applications is available on the web site /docs/global/.

Currently support is limited to 2-D double precision or integer arrays with block distribution, at most one block per array per processor.

6.1.3.1 Interaction Between GA and MA

Available global (GA) and local (MA) memory can interact within NWChem in only two ways,

  1. GA is allocated within MA, and GA is limited only by the available space in MA.

  2. GA is not allocated within MA, and GA is limited at initialization (within NWChem input this is controlled by the MEMORY directive)

If GA is allocated within MA, then the available GA space is limited to the currently available MA space. This also means that the total allocatable memory for GA and MA must be no more than the available MA space. If GA is not allocated within MA, then local and global arrays occupy essentially independent space. The allocatable memory for GA is limited only by the available space for GA, and similarly, the allocatable memory for MA is limited only by the available local memory.

When allocating space for GA, some care must be exercised in the treatment of the information returned by the routine ga_memory_avail(), whether or not the allocation is done in MA. The routine ga_memory_avail() returns the amount of memory (in bytes) available for use by GA in the calling process. This returned value must be converted to double precision words when using double precision. If a uniformly distributed GA is desired, it is also necessary to find the minimum of this value across all nodes. This value will in general be a rather large number. When running on a platform with many nodes and having a large memory, the agreggate GA memory, even in double precision words, could be a large enough value to overflow a 32-bit integer. Therefore, for calculations that require knowing the size of total memory, it is advisable to first store the size of memory on each node in a double precision number and then sum these values across all the nodes.

The following pseudo-code illustrates this process for an application.

#include "global.fh"
#include "mafdecls.fh"

    integer avail_ma, avail_ga

    avail_ma = ma_inquire_avail(mt_dbl)
    avail_ga = ga_memory_avail()/ma_sizeof(mt_dbl,1,mt_byte)

    if (ga_uses_ma()) then
c
c  available GA space is limited to currently available MA space,
c  and GA and MA share the same space
c
      allocatable_ga + allocable_ma <= avail_ma = avail_ga

    else
c
c  GA and MA are independent
c
         allocatable_ga <= avail_ga
         allocatable_ma <= avail_ma

    endif
c
c find the minimum value of available GA space over all nodes
c
    call ga_igop(msgtype,avail_ga,1,'min')
c
c determine the total available GA space
c
    double precision davail_ga
    davail_ga = ga_memory_avail()/ma_sizeof(mt_dbl,1,mt_byte)
    call ga_dgop(msgtype,davail_ga,1,'+')

6.1.3.2 List of GA Routines

The following routines are invoked for operations that are globally collective. That is, they must be simultaneously invoked by all processes as if in SIMD mode.

Operations that may be invoked by any process in true MIMD style:

Operations that may be invoked by any process in true MIMD style and are intended to support writing of new functions:

Operations to support portability between implementations:

Other utility operations:

Note that consistency is only guaranteed for

  1. Multiple read operations (as the data does not change)
  2. Multiple accumulate operations (as addition is commutative)
  3. Multiple disjoint put operations (as there is only one writer for each element)
The application has to worry about everything else (usually by appropriate insertion of ga_sync calls).

6.1.3.3 New(?) Stuff

Subroutines that appear in the files of directory .../src/global/src, but are not in the (ga.tex) document;

6.1.3.4 Use of TCGMSG global operation routines

In some cases (notably workstation clusters) the global array tools use a ``data-server'' process on each node in addition to the compute processes. Data-server processes don't follow the same flow of execution of compute processes, so TCGMSG global operations (brdcst, igop, and dgop) will hang when invoked. The global array toolkit provides ``wrapper'' functions (ga_brdcst, ga_igop, and ga_dgop) which properly exclude data server processes from the global communication and must be used instead of the corresponding TCGMSG functions.

6.1.3.5 Interaction between GA and message-passing

The limited buffering available on the IBM SP-1/2 means that GA and message-passing operations cannot interleave as readily as they do on other machines. Basically, in transitioning from GA to message passing or vice versa the application must call ga_sync().

6.1.4 ChemI/O

ChemIO is a high-performanc parallel I/O abstract programming interface for computational chemistry applications6.2. The development of out-of-core methods for computational chemistry requires efficient and portable implementation of often complex I/O patterns. The ChemIO interface addresses this problem by providing high performance implementations on multiple platforms that hides some of the complexity of the underlying I/O patterns from the programmer through the use of high-level libraries. The interface is tailored to the requirements of large-scale computational chemistry problems and supports three distinct I/O models. These are

  1. Disk Resident Arrays (DRA) -- for explicit transfer between global memory and secondary storage, allowing the programmer to manage the movement of array data structures between local memory, remote memory, and disk storage. This component supports collective I/O operations, in which multiple processors cooperate in a read or write operation and thereby enable certain useful optimizations.

  2. Exclusive Access Files (EAF) -- for independent I/O to and from scratch files maintained on a per-processor basis. It is used for out-of-core computations in calculational modules that cannot easily be organized to perform collective I/O operations.

  3. Shared Files (SF) -- for creation of a scratch file that can be shared by all processors. Each processor can perform noncollective read or write operations to an arbitrary location in the file.

These models are implemented in three user-level libraries in ChemIO; Disk Resident Arrays, Exclusive Access Files, and Shared Files. These libraries are layered on a device library, the Elementary I/O library (ELIO), which provides a portable interface to different file systems. The DRA, EAF, and SF modules are fully independent. Each one can be modified or even removed without affecting the others. ELIO itself is not exposed to applications.

6.1.4.1 Elementary I/O Library (ELIO)

The ELIO library implements a set of elementary I/O primitives including blocking and non-blocking versions of read and write operations, as well as wait and probe operations to control status of non-blocking read/writes. It also implements file operations such as open, close, delete, truncate, end-of-file detection, and an inquiry function for the file/filesystem that returns the amount of available space and the filesystem type. Most of these operations are commonly seen in various flavors of the UNIX filesystem. ELIO provides an abstract portable interface to such functionality.

(Insert gory details here.)

6.1.4.2 Disk Resident Arrays

The computational chemistry parallel algorithms in NWChem have been implemented in terms of the Global Arrays shared memory programming model. The GA library (see Section 6.1.3) uses a shared memory programming model in which data locality is managed explicitly by the programmer. This management is achieved by explicit calls to functions that transfer data between a global address space (a distributed array) and local storage. The GA library allows each process in a MIMD parallel program to access asynchronously logical blocks of physically distributed matrices without the need for explicit cooperation from other processes.

The GA model exposes to the programmer the non-uniform memory access (NUMA) characteristics of modern high-performance computer systems. The disk resident array (DRA) model extends the GA model to another level in the storage hierarchy, namely, secondary storage. It introduces the concept of a disk resident array -- a disk-based representation of an array -- and provides functions for transferring blocks of data between global arrays and disk arrays. It allows the programmer to access data located on disk via a simple interface expressed in terms of arrays rather than files.

At the present time, (NOTE: The source of this statement is a document created 5/10/95) all operations are declared to be collective. This simplifies implementation on machines where only some processors are connected to I/O devices.

Except where stated otherwise, all operations are synchronous (blocking) which means that control is returned to the calling process only after the requested operation completes.

All operations return an error code with value 0 if successful, greater than zero if not successful.

A program that uses Disk Resident Arrays should look like the following example:

      program foo
#include "mafdecls.h"
#include "global.fh"
#include "dra.fh"
c
      call pbeginf()                      ! initialize TCGMSG
      if(.not. ma_init(...)) ERROR        ! initialize MA
      call ga_initialize()                ! initialize Global Arrays
      if(dra_init(....).ne.0) ERROR       ! initialize Disk Arrays 

c     do work

      if(dra_terminate().ne.0)ERROR       ! destroy DRA internal data structures
      call ga_terminate                   ! terminate Global Arrays
      call pend()                         ! terminate TCGMSG
      end

List of DRA operations:

6.1.4.3 Exclusive Access Files (EAF)

The EAF module supports a particularly simple I/O abstraction in which each processor in a program is able to create files that it alone has access to. The EAF interface is similar to the standard C UNIX I/O interface and is implemented as a thin wrapper on the ELIO module. It provides Fortran and C applications with capabilities that include

The syntax of EAF is similar to the standard Unix C file operations, although there are some differences, as a result of introducing new semantics or extended features available through EAF.

The primary functionality of EAF is illustrated here by tracing execution of example program segments.

Example 1: basic open-write-read-close sequence.

#include "chemio.h"
#include "eaf.fh"

	integer fh 		! File Handle
	integer sz 		! Return value of size written
	integer stat		! Return status
	integer buf(100) 	! Data to write

	fh = EAF_OpenPersist('/tmp/test.out', ELIO_RW) <- We probably want
						          CHEMIO_RW here

	sz = EAF_Write(fh, 0, buf, 100*EAF_SZ_INT)     <- What's the NWChem 
                                                          macro for int size?
	if(sz .ne. 100*EAF_SZ_INT) 
      $       write(0,*) 'Error writing, wrote ', sz, ' bytes'

	sz = EAF_Read(fh, 0, buf, 100*EAF_SZ_INT)
	if(sz .ne. 100*EAF_SZ_INT) 
      $       write(0,*) 'Error reading, read ', sz, ' bytes'

	stat = EAF_Close(fh)
	end

The include file 'chemio.h' defines the permission macros ELIO_R, ELIO_W, and ELIO_RW for read, write, and read-write permissions, respectively. The header file 'eaf.fh' is a Fortran program segment externally defining the EAF routines and must appear before any executable code using EAF.

EAF_OpenPersist opens a persistent file, as opposed to a scratch file (EAF_OpenScratch) which is deleted when it is closed. This file is named '/tmp/test.out' and has read-write permissions. The returned value is the file handle for this file and should not be directly manipulated by the user.

EAF_Write writes to the file opened with file handle, fh, at absolute offset 0. It is legal to write a scalar or array, for instance in the above example both 'buf' and 'buf(1)' have the same meaning. The last argument is the number of bytes to be written. It is important to multiply the number of array elements by the element size. The following macros are provided in 'eaf.fh':

The return value is the number of bytes written. If this number does not match the requested number of bytes to be written, an error has occured.

Example 2: read/write operations

EAF_Read is syntactically and semantialy identical to EAF_Write, except the buffer is read, not written.

#include "chemio.h"
#include "eaf.fh"

	integer fh 		! File Handle
	integer id1, id2 	! asynchronous ID handles
	integer stat		! Return status
	integer pend		! Pending status
	integer iter		! Iterations counter
	integer buf(100), x	! Data

	iter = 0

	fh = EAF_OpenScratch('/piofs/mogill/test.out', ELIO_RW)

	stat = EAF_AWrite(fh, 0,  buf, 100*EAF_SZ_INT, id1)
	if(stat .ne. 0) write(0,*) 'Error doing 1st asynch write.  stat=', stat

	stat = EAF_AWrite(fh, 100*EAF_SZ_INT,  x, 1*EAF_SZ_INT, id2)
	if(stat .ne. 0) write(0,*) 'Error doing 2nd asynch write.  stat=', stat

100	stat = EAF_Probe(id1, pend)
	iter = iter + 1
	write(0,*) 'Waiting', iter
	if(iter .lt. 100  .and.  pend .eq. ELIO_PENDING) goto 100
	EAF_Wait(id1)

	stat = EAF_ARead(fh, 0, buf, 100*EAF_SZ_INT, id1)
	if(stat .ne. 0) write(0,*) 'Error doing 1st asynch read.  stat=', stat

	EAF_Wait(id2)
	stat = EAF_AWrite(fh, 100*EAF_SZ_INT,  x, 1*EAF_SZ_INT, id2)
	if(stat .ne. 0) write(0,*) 'Error doing 2nd asynch write.  stat=', stat
	EAF_Wait(id2)
	EAF_Wait(id1)

	stat = EAF_Close(fh)
	end

This example demonstrates use of asynchronous reading and writing. The entire buffer 'buf' is written to offset 0, the beginning of the. The file is simultaniously written to from the scalar x in the position following the buffer. The positions in the file are determined by abosulte offset argument as with the synchronous write.

The first write, id1, is repeatedly probed for completion for 100 tries or until completion, whichever comes first. The two possible pending statuses are ELIO_DONE and ELIO_PENDING.

When a completed asynchronous operation is detected with EAF_Wait or EAF_Probe, the id is invalidated with ELIO_DONE. The following EAF_Wait(id1) blocks until id1 completes. Using EAF_Probe or EAF_Wait with an invalidated ID has no effect.

Once id1 is freed, it is reused in the first asynchronous read statement. The following EAF_Wait blocks for completion and invalidation of id2, which is then used to asynchronously read the scalar X.

The EAF_Close deletes the file because it was opened as a scratch file.

List of EAF Functions

6.1.4.4 Shared Files (SF)

The Shared File module supports the abstraction of a single contiguous secondary storage address space (a "file") that every processor has access to. Processes create and destroy SF objects in a collective fashion, but all other file I/O operations are non- collective. A shared file can be thought of as a one-dimensional array of bytes located in shared memory, except that the library interface is required to actually access the data.

The library is capable of determining the striping factor and all other internal optimizations for the "file". The programmer has the option, however, of giving the library a few helpful hints, to reduce the number of decisions the interface must take care of. These hints are supplied when the shared file is created, and can be any or all of the following:

  1. Specify a hard limit (not to be exceeded) for the file size.
  2. Specify a soft limit for the file size; that is, an estimate of the expected shared file size, which can be exceeded at run time, if necessary.
  3. Specify the size of a "typical" request.

Non-collective I/O operations in SF include read, write, and wait operations. Read and write operations transfer the specifeid number of bytes between local memory and disk at a specified offset. The library does not perform any explicit control of consistency in concurrent accesses to overlapping sections of the shared files. For example, SF semantics allow a write operation to return before the data transfer is complete. This requires special care in programs that perform write operations in critical sections, since unlocking access to a critical section before write completes is unsafe. To allow mutual exclusion control in access to shared files, the sf_wait function is provide. It can be used to enforce completion of the data transfer so that the data can be safely accessed by another process after access to the critical section is released by the writing process. The function sf_waitall can be used to force the program to wait for completion of multiple SF operations specified through an arugment arry of request identifiers.

The actual size of a shared file might grow as processes perform write operations beyond the current end-of-file boundary. Data in shared files are implicitly initialized to zero, which means that read operations at locations that have not been written to return zero values. However, reading behond the current end-of-file boundary is an error.

Shared files can be used to build other I/O abstractions. In many cases, this process requires adding an additional consistency control layer. A single file pointer view, for example, can be implemented by adding an automatically modifiable pointer variable located in shared memory by using the GA toolkit, or some other means.

The shared files model consists of the following elements:

List of SF Functions:

integer sf_create(fname, size_hard_limit, size_soft_limit, req_size, handle)
        fname            -- meta-file name
        size_hard_limit  -- max file size in bytes not to be exceeded (a hint)
        size_soft_limit  -- estimated file size (a hint)
        req_size         -- size of  a typical request (a hint)
        handle           -- returned handle to the created file

Creates shared file using name and path specified in fname as a template. Function req_size specifies size of a typical request (-1 = "don't know").

integer sf_write(handle, offset, bytes, buffer, request_id)
        handle           -- file handle returned from sf_create   [in]
        offset           -- location in file (from the beginning)
                            where data should be written to       [in]
        buffer           -- local array to put the data           [in]
        bytes            -- number of bytes to read               [in]
        request_id       -- id identifying asynchronous operation [out]

asynchronous write operation

integer sf_read(handle, offset, bytes, buffer, request_it)
        handle           -- file handle returned from sf_create   [in]
        offset           -- location in file (from the beginning)
                            where data should be read from        [in]
        buffer           -- local array to put the data           [in]
        bytes            -- number of bytes to read               [in]
        request_id       -- id identifying asynchronous operation [out]

asynchronous read operation

integer sf_wait(request_id)
        request_id       -- id identifying asynchronous operation [in/out]

blocks calling process until I/O operation associated with id completed, invalidates request_id

integer sf_waitall(list, num)
        list(num)        -- array of ids for asynchronous operations [in/o]
        num              -- number of entries in list                [in]

blocks calling process until all "num" I/O operations associated with ids specified in list completed, invalidates ids on the list

integer sf_destroy(handle)
        handle           -- file handle returned from sf_create      [in]

6.2 The Run Time Data Base (RTDB)

The run time data base is the parameter and information repository for the independent modules (e.g., SCF, RIMP2) comprising NWChem. This approach is similar in spirit to the GAMESS dumpfile or the Gaussian checkpoint file. The only way modules can share data is via the database or via files, the names of which are stored in the database (and may have default values). Information is stored directly in the database as typed arrays, each of which is described by

  1. a name, which is a simple string of ASCII characters (e.g., "reference energies"),
  2. the type of the data (real, integer, logical, or character),
  3. the number of data items, and
  4. the actual data (an array of items of the specified type).

A database is simply a file and is opened by name. Usually there is just one database per calculation, though multiple databases may be open at any instant.

By default, access to all open databases occur in parallel, meaning that

Alternatively, database operations can occur sequentially. This means that only process zero can read/write the database, and this happens with no communication or synchronization with other processes. Any read/write operations by any process other than process zero is an error.

Usually, all processes will want the same data at the same time from the database, and all processes will want to know of the success or failure of operations. This is readily done in the default parallel mode. An exception to this is during the reading of input. Usually, only process zero will read the input and needs to store the data directly into the database without involving the other processes. This is done using sequential mode.

The following subsections contain a detailed listing of the C and Fortran API. Programs using RTDB routines must include the appropriate header file; rtdb.fh for Fortran, or rtdb.h for C. These files define the return types for all rtdb functions. In addition, rtdb.fh specifies the following parameters

The Fortran routines return logical values; .true.on success, .false. on failure. The C routines return integers; 1 on success, or 0 on failure. All rtdb_* functions are also mirrored by routines rtdb_par_* in which process 0 performs the operation and all other processes are broadcast the result of a read and discard writes.

6.2.1 Functions to Control Access to the Runtime Database

The functions that control opening, closing, writing to and reading information from the runtime database are described in this section.

6.2.1.1 rtdb_parallel

C routine:

  int rtdb_parallel(const int mode)

Fortran routine:

  logical function rtdb_parallel(mode)
  logical mode              [input]
This function sets the parallel access mode of all databases to mode and returns the previous setting. If mode is true then accesses are in parallel, otherwise they are sequential.

6.2.1.2 rtdb_open

C routine:

  int rtdb_open(const char *filename, const char *mode, int *handle)

Fortran routine:

  logical function rtdb_open(filename, mode, handle)
  character *(*) filename   [input]
  character *(*) mode       [input]
  integer handle            [output]
This function opens a database. It requires the following arguments:

6.2.1.3 rtdb_close

C routine:

  int rtdb_close(const int handle, const char *mode)

Fortran routine:

  logical function rtdb_close(handle, mode)
  integer handle            [input]
  character*(*) mode        [input]
This function closes a database. It requires the following arguments: When closing a database file that has been opened with the rtdb_open argument mode specified as scratch, the value for mode for the function rtdb_close is automatically set to delete. Database files needed for restart must not be opened as scratch files.

6.2.1.4 rtdb_put

C routine:

  int rtdb_put(const int handle, const char *name, const int ma_type,
               const int nelem, const void *array)

Fortran routine:

  logical function rtdb_put(handle, name, ma_type, nelem, array)
  integer handle            [input]
  character *(*) name       [input]
  integer ma_type           [input]
  integer nelem             [input]
  <ma_type>                 [input]
  nelem                     [input]
  array                     [input]
This function inserts an entry into the database, replacing the previous entry. It requires the following arguments:

6.2.1.5 rtdb_get

C routine:

  int rtdb_get(const int handle, const char *name, const int ma_type,
               const int nelem, void *array)

Fortran routine:

  logical function rtdb_get(handle, name, ma_type, nelem, array)
  integer handle            [input]
  character *(*) name       [input]
  integer ma_type           [input]
  integer nelem             [input]
  <ma_type>                 [output]
  nelem                     [output]
  array                     [output]
This function gets an entry from the data base. It requires the following arguments:

6.2.1.6 rtdb_cput and rtdb_cget

  logical function rtdb_cput(handle, name, nelem, buf)
  integer handle            [input]
  character *(*) name       [input]
  character *(*) buf        [input]

  logical function rtdb_cget(handle, name, nelem, buf)
  integer handle            [input]
  character *(*) name       [input]
  character *(*) buf        [output]
These functions are Fortran routines to provide put/get functionality for character variables. The functions have identical argument lists, the only difference between them is that for rtdb_cput, the specified character data is put into the database, and for rtdb_cget the data is copied from the database. The arguments are as follows;

6.2.1.7 rtdb_ma_get

C routine:

  int rtdb_ma_get(const int handle, const char *name, int *ma_type,
                  int *nelem, int *ma_handle)

Fortran routine:

  logical function rtdb_ma_get(handle, name, ma_type, nelem, ma_handle)
  integer handle            [input]
  character *(*) name       [input]
  integer ma_type           [output]
  integer nelem             [output]
  integer ma_handle         [output]
This function returns the MA type, number of elements of that type, and the MA handle of the entry specified. (The MA handle is to memory automatically allocated to hold the data read from the database.) the function requires the following arguments:

6.2.1.8 rtdb_get_info

C routine:

  int rtdb_get_info(const int handle, const char *name, int *ma_type, 
                    int *nelem, char date[26])

Fortran routine:

  logical function rtdb_get_info(handle, name, ma_type, nelem, date)
  integer handle            [input]
  character *(*) name       [input]
  integer ma_type           [output]
  integer nelem             [output]
  character*26 date         [output]

This function queries the database to obtain the number of elements in the specified entry, it's MA type, and the date of its insertion into the rtdb. It requires the following arguments:

6.2.1.9 rtdb_first and rtdb_next

C routines:

  int rtdb_first(const int handle, const int namelen, char *name)

  int rtdb_next(const int handle, const int namelen, char *name)

Fortran routines:

  logical function rtdb_first(handle, name)
  integer handle            [input]
  character *(*) name       [output]

  logical function rtdb_next(handle, name)
  integer handle            [input]
  character *(*) name       [output]
These routines enable iteration through the items in the database in an effectively random order. The function rtdb_first returns the name of the first user-inserted entry in the datbase. The function rtdb_next returns the name of the user-inserted entry put into the data base after the entry identified on the previous call to rtdb_next (or the call to rtdb_first, on the first call to rtdb_next).

The arguments required for the C routines are as follows:

The Fortran routines require the same arguments for handle and name, but it is not necessary to define the length of the buffer required.

An example of the use of these functions in C is to count and print the name of all entries in the database. The coding for this can be implemented as follows;

  char name[256];
  int n, status, rtdb;

  for (status=rtdb_first(rtdb, sizeof(name), name), n=0;
       status;
       status=rtdb_next(rtdb, sizeof(name), name), n++) 
    printf("entry %d has name '%s'\n", n, name);

6.2.1.10 rtdb_delete

C routine:

  int rtdb_delete(const int handle, const char *name)

Fortran routine:

  logical function rtdb_delete(handle, name)
  integer handle            [input]
  character *(*) name       [input]
This function deletes an entry from the database. This function does not return any arguments. The value the function itself returns as indicates success or failure of the delete operation. The function returns as

6.2.1.11 rtdb_print

C routine:

  int rtdb_print(const int handle, const int print_values)

Fortran routine:

  logical function rtdb_print(handle, print_values)
  integer handle            [input]
  logical print_values      [input]
This function prints the contents of the data base to STDOUT. It requires the following arguments:


next up previous contents
Next: 7. Utilities Up: NWCHEM Programmer's Guide, Release Previous: 5. Integral Application Programmer's   Contents
Dunyou Wang 2009-03-13