The FMS HIM User Guide

This program (HIM) simulates the ocean by numerically solving the Boussinesq primitive equations in isopycnal vertical coordinates and general orthogonal horizontal coordinates. These equations are horizontally discretized on an Arakawa C-grid. There are a range of options for the physical parameterizations, from those most appropriate to highly idealized models for studies of geophysical fluid dynamics to a rich suite of processes appropriate for realistic ocean simulations. The thermodynamic options range from an adiabatic model with fixed density layers to a model with temperature and salinity as state variables and using a full nonlinear equation of state. The uppermost few layers may be used to describe a bulk mixed layer, including the effects of penetrating shortwave radiation. Either a split- explicit time stepping scheme or a non-split scheme may be used for the dynamics, while the time stepping may be split (and use different numbers of steps to cover the same interval) for the forcing, the thermodynamics, and for the dynamics. Most of the numerics are second order accurate in space. HIM can run with an absurdly thin minimum layer thickness (often an Angstrom is used).

Details of the numerics and physical parameterizations are provided in the appropriate source files. Most of the available options are selected by the settings in HIM_init.h and may be overridden at run time through a parsed input file with the same format, although some (such as the equation of state) are selected by specifying which file to use in the path_names file.

There are a range of closure options available in HIM. Horizontal velocities are subject to a combination of horizontal biharmonic and Laplacian friction (based on a stress tensor formalism) and a vertical Fickian viscosity (perhaps using the kinematic viscosity of water). The horizontal viscosities may be constant, spatially varying or may be dynamically calculated using Smagorinsky's approach. A diapycnal diffusion of density and thermodynamic quantities is also allowed, but not required, as is horizontal diffusion of interface heights (akin to the Gent-McWilliams closure of geopotential coordinate models). The diapycnal mixing may use a fixed diffusivity or it may use the shear Richardson number dependent closure described in Hallberg (MWR, 2000). When there is diapycnal diffusion, it applies to momentum as well. As this is in addition to the vertical viscosity, the vertical Prandtl always exceeds 1.

HIM has a number of noteworthy debugging capabilities. Excessively large velocities are truncated and HIM will stop itself after a number of such instances to keep the model from crashing altogether, and the model state is output with a reported time of 9.9e9. This is useful in diagnosing failures, or (by accepting some truncations) it may be useful for getting the model past the adjustment from an ill-balanced initial condition. In addition, all of the accelerations in the columns with excessively large velocities may be directed to a text file.

While HIM is qualitatively similar to MICOM, most of the details of the numerics differ.

A deliberately long (but somewhat out of date) technical description of the dynamical core is also available for the brave. This description of the dynamical core is largely based on the appendix of R. Hallberg's thesis, and an appropriate reference for HIM is: Hallberg, R., 1995: Some Aspects of the Circulation in Ocean Basins with Isopycnals Intersecting the Sloping Boundaries. Ph. D. Thesis, University of Washington. 244 pp.

Many of the more recent algorithmic developments with HIM are documented in the peer reviewed literature.

The purpose of this web page is to provide general information about HIM and particular information for how to download and run the code.

HIM users can acquire the source code and associated data sets from GForge, and are required to register at the GFDL GForge location. Therefore, users need to register only once to get both the source code and datasets of HIM. More details can be found in the quickstart_guide.html.

Email concerning HIM are to be directed to the HIM-email list located at oar.gfdl.himnoaa.gov. All questions, comments, and suggestions are to be referred to this list. An archive of all emails is maintained at email archive. Note that by registering at GForge to access the code, you are automatically subscribed to the email list.

This is the first public release of the F90 HIM code. However, the careful conversion of the C HIM code should ensure that there are relatively few bugs for a wholly new code. The Fortran HIM code has been in scientific use at GFDL for about a year now. In addition, the F90 coding style benefits from our experience with both the C HIM code and MOM4. We believe that this will be a valuable and proficient tool for simulating the ocean.

In addition to this user guide, documentation for HIM is provided by two postscript documents:

HIM possesses a number of computational, numerical, and physical characteristics that are noteworthy. The following provides an overview of the main characteristics of HIM (please refer to A Technical Guide to HIM for references).

HIM has been coded within GFDL's Flexible Modeling System (FMS). Doing so allows for HIM developers to use numerous FMS infrastructure and superstructure modules that are shared amongst various atmospheric, ocean, sea ice, land, vegetative, etc. models. Common standards and shared software tools facilitate the development of high-end earth system models, which necessarily involves a wide variety of researchers working on different computational platforms. Such standards also foster efficient input from computational scientists and engineers as they can more readily focus on common computational issues.

The following list represents a sample of the FMS shared modules used by HIM.

The FMS infrastructure (the "Lima version") has been released to the public on GForge, with further releases every three-four months.

The Flexible Modeling System ( FMS) is free software; you can redistribute it and/or modify it and are expected to follow the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

FMS is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with HIM; if not, write to:

HIM is distributed with a set of test cases located in src/HIM_examples/. These tests are taken from models used at GFDL for testing the numerical and computational integrity of the code.

	Warning
	These experiments are NOT sanctioned for their physical relevance. They are instead provided for the user to learn how to run HIM, and to verify the numerical and/or computational integrity of the code. PLEASE do not assume that the experiments will run for more than the short time selected in the sample runscripts.

2gyre: A two-layer, adiabatic, wind-driven double gyre in a basin with sloping sides. There are no thermodynamics in this case. This model is very small and can be easily run on a single processor workstation. It should provide the user with a basic experience of running HIM.

DOME: An entraining gravity current, driven by an inflow at the top of a slope. This is a coarse resolution version of one of the cases described in Legg et al., Ocean Modelling, 2005. This case has no surface forcing or surface mixed layer submodel, and only buoyancy as a thermodynamic variable, but it is otherwise a full ocean model.

benchmark: An idealized wind- and buoyancy-driven high-resolution model. This has all the physics of a full ocean model, but in a configuration that requires no input files. Versions of this have been used by NOAA for benchmarking computers. To change the resolution, only the number of gridpoints and timestep need to be changed.

global: A 1-degree, 48-layer ocean-only prototype for a global ocean-climate model on a tripolar grid. This is driven by the CORE forcing and uses restoring of surface properties to climatological values. THIS HAS NOT BEEN FULLY TUNED, AND IS NOT DIRECTLY USEFUL FOR CLIMATE STUDIES!.

him_sis: The same model as global, but coupled to GFDL's sea ice model (SIS). Running this model is somewhat different from HIM-only runs in that flow control is ceded to the standard GFDL coupler, and the HIM code is not used to specify the forcing. From a user's perspective, running this model is virtually identical to running a MOM4 model coupled to SIS.

The FMS development team uses a local implementation of GForge to serve FMS software, located at http://fms.gfdl.noaa.gov. In order to obtain the source code and data sets, you must register as an FMS user on our software server. After submitting the registration form on the software server, you should receive an automatically generated confirmation email within a few minutes. Clicking on the link in the email confirms the creation of your account.

After your account has been created, you should log in and request access to the Flexible Modeling System project. Once the FMS project administrator grants you access, you will receive a second email notification. This email requires action on the part of the project administrator and thus may take longer to arrive. The email will contain a software access password along with instructions for obtaining the release package, which are described below.

To check out the release package containing source code, scripts, and documentation via CVS, type the following commands into a shell window. You might wish to first create a directory called fms in which to run these commands. You should enter the software access password when prompted by the cvs login command. At cvs login, the file ~/.cvspass is read. If this file does not already exist, an error message may display and the cvs login may fail. In this event, you should first create this file via touch ~/.cvspass.

      cvs -z3 -d:pserver:cvs@fms.gfdl.noaa.gov:/cvsroot/him login
      cvs -z3 -d:pserver:cvs@fms.gfdl.noaa.gov:/cvsroot/him co -r lima him
    

This will create a directory called HIM in your current working directory containing the release package.

If you prefer not to use CVS, you may download the tar file from https://fms.gfdl.noaa.gov/projects/HIM/. Sample output is also available there for download. See Section 5.1, “Sample model output” for more information on the sample output.

All data sets that are needed to run HIM test cases are available for download from the same place in GForge where users get the source code. Therefore, users need to register only once to get both the source code and datasets of HIM. More details can be found in the quickstart_guide.html.

The topography data set for HIM_global and HIM_SIS is a coarsened version of that kindly provided by Andrew Coward and David Webb at the Southampton Oceanography Centre. Their topography is a montage of that developed by Smith and Sandwell (1997) by satellite data in the region of 72°S to 72°N, the NOAA (1988) 5-minute global topography ETOPO5, and the International Bathymetric Chart of the Arctic Ocean (IBCAO). All of the forcing is taken from the "Common Ocean Reference Experiments" (CORE) (Large and Yeager, 2005). Temperature and salinity initial and boundary conditions are provided by the NOAA National Oceanographic Data Center (NODC) World Ocean Atlas (WOA).

A runscript is provided in each test case directory (scripts/him_test_case ) for each test case. Details can be found in quickstart_guide.html.

Incorporated in the FMS infrastructure is MPP (Massively Parallel Processing), which provides a uniform message-passing API interface to the different message-passing libraries. If MPICH is installed, the user can compile the HIM source code with MPI. If the user does not have MPICH or the communications library, the HIM source code can be compiled without MPI by omitting the CPPFLAGS value -Duse_libMPI in the example runscript.

The diagnostics table allows users to specify the sampling rates and choose the output fields prior to executing the HIM source code. It is included in the experiment directory for each test case (HIM/exp/$test_case/HIM_diag_table). A portion of a sample HIM diagnostic table is displayed below. Reference diag_table_tk.html for detailed information on the diagnostics table.

"HIM Experiment"
1 1 1 0 0 0
"prog_%4yr_%3dy",     5,"days",1,"days","Time",365,"days"
"ave_prog_%4yr_%3dy", 5,"days",1,"days","Time",365,"days"
"cont_%4yr_%3dy",     5,"days",1,"days","Time",365,"days"
#This is the field section of the diag_table.
# Prognostic Ocean fields:
#=========================
"ocean_model","u","u","prog_%4yr_%3dy","all",.false.,"none",2
"ocean_model","v","v","prog_%4yr_%3dy","all",.false.,"none",2
"ocean_model","h","h","prog_%4yr_%3dy","all",.false.,"none",1
"ocean_model","e","e","prog_%4yr_%3dy","all",.false.,"none",2
"ocean_model","temp","temp","prog_%4yr_%3dy","all",.false.,"none",2
"ocean_model","salt","salt","prog_%4yr_%3dy","all",.false.,"none",2
  

The diagnostics manager module, diag_manager_mod, is a set of simple calls for parallel diagnostics on distributed systems. It provides a convenient set of interfaces for writing data to disk, namely in NetCDF format. The diagnostics manager is packaged with the HIM source code. The FMS diagnostic manager can handle scalar fields as well as arrays. For more information on the diagnostics manager, reference diag_manager.html.

Running the HIM source code in a parallel processing environment will produce one output NetCDF diagnostic file per processor. mppnccombine joins together an arbitrary number of data files containing chunks of a decomposed domain into a unified NetCDF file. If the user is running the source code on one processor, the domain is not decomposed and there is only one data file. mppnccombine will still copy the full contents of the data file, but this is inefficient and mppnccombine should not be used in this case. Executing mppnccombine is automated through the runscripts. The data files are NetCDF format for now, but IEEE binary may be supported in the future.

mppnccombine requires decomposed dimensions in each file to have a domain_decomposition attribute. This attribute contains four integer values: starting value of the entire non-decomposed dimension range (usually 1), ending value of the entire non-decomposed dimension range, starting value of the current chunk's dimension range and ending value of the current chunk's dimension range. mppnccombine also requires that each file have a NumFilesInSet global attribute which contains a single integer value representing the total number of chunks (i.e., files) to combine.

The syntax of mppnccombine is:

        mppnccombine [-v] [-a] [-r] output.nc [input ...] 
    

An output file must be specified and it is assumed to be the first filename argument. If the output file already exists, then it will not be modified unless the option is chosen to append to it. If no input files are specified, their names will be based on the name of the output file plus the extensions '.0000', '.0001', etc. If input files are specified, they are assumed to be absolute filenames. A value of 0 is returned if execution is completed successfully and a value of 1 indicates otherwise.

The source of mppnccombine is packaged with the HIM module in the postprocessing directory. mppnccombine.c should be compiled on the platform where the user intends to run the FMS HIM source code so the runscript can call it. A C compiler and NetCDF library are required for compiling mppnccombine.c:

       cc -O -o mppnccombine -I/usr/local/include -L/usr/local/lib mppnccombine.c -lnetcdf
    

Sample HIM model output data files are available to registered HIM users on GFDL's NOMADS server. The output data are organized into directories that bear the same names as the test cases . For example, output for test case 2gyre can be found in directory sample_output/2gyre. Output files are classified into two categories:

Note that these output files are compressed using tar. All .tar files should be decompressed for viewing. The decompress command is:

       tar -xvf filename.tar
    

There are several graphical packages available to display the model output. These packages vary widely depending on factors, such as the number of dimensions, the amount and complexity of options available and the output data format. The data will first have to be put into a common format that all the packages can read. FMS requires the data to be stored in NetCDF format since it is so widely supported for scientific visualization. The graphical package is also dependent upon the computing environment. For ocean modeling, ncview, Ferret and Matlab are most commonly used.