PETSc (the Portable, Extensible Toolkit for Scientific Computation) is developed by the Argonne National Laboratory. It is part of the ACTS Toolkit in the DOE 2000 Initiative. PETSc is a suite of data structures and routines for the scalable (parallel) solution of scientific applications modeled by partial differential equations. It employs the MPI standard for all message-passing communication, and is written in a data-structure-neutral manner to enable easy reuse and flexibility.
The following summarizes the inter-operability status of Global Arrays and PETSc:
Global Arrays is inter-operable with PETSc. In an application using Global Arrays, the PETSc solvers can be called to solve PDEs that require solving large-scale, sparse nonlinear systems of equations. The only issue related to inter-operablity is how to convert the data structures of Global Arrays to those of PETSc before calling the PETSc solvers, and how to convert the data structures of PETSc back to Global Arrays after calling the PETSc solvers. Fortunately, PETSc provides enough mechanisms for this purpose. For vector operations, there are VecCreateMPI()
, VecSetValues()
, VecGetArray()
, VecRestoreArray()
, etc. The same functions exist for matrix operation.
The packages used in the testing are:
PETSc online documentation is a well-maintained site for PETSc resources. Examples can be accessed both online or from the package itself.
A typical scenario to use PETSc in a Global Arrays application is that there is a global array x that represents the approximate solution initialized with some initial values. It needs to call one of the PETSc solvers to solve the problem, and restore the results back to x.
Here are the instructions for implementing an example Ax = b, where A is the matrix defining the linear system, b is the right-hand side, and x is the approximate solution and an global array.
There are detailed instructions for setting up environment variables on different platforms with the PETSc package. Users on Cray T3E at NERSC only need to the load the petsc module: insert
module load petsc
into the .login
file.
Data conversion between the Global Arrays and PETSc is the key issue for inter-operability. PETSc provides several ways to create Vectors and Matrices and to set values to them. We found that the most efficient way to connect the Global Arrays and PETSc is to use the GetArray and RestoreArray mechanism. GetArray and RestoreArray are not intended to set values though, they open a window to access and update the local Vector/Matrix of PETSc. Global Arrays provide the one-sided operations, get and put, which are a perfect match for PETSc's GetArray and RestoreArray mechanism. The array segment of Global Arrays can be sent to or received from PETSc in block fashion, instead of updating element by element.
Here is how it works: