Hydrosys

Hydrosys is a set MATLAB routines originally written in 1997 by Alexandre Ganachaud, in parallel to the hydroinv programs. The purpose is to organize the data treatment and program in a memory efficent and easy acess way, as the data sets can be large. The programs have been written mostly using the matlab language for modifications to be easy. For compatibility purposes, interface routines allow one to go from the hydroinv to the hydrosys data systems and sometimes reciprocally. The problem with hydroinv was the difficulty to accessing the data.

DISCLAIMER:

This software is provided "as is" without warranty of any kind.

BUGS:

If you use this software, please send an email to Alexandre Ganachaud so that we can keep you informed of any modification/correction. Send an email if you have any questions or comments, bug report.

DOWNLOAD:

  1. hydroinv.tar (compressed tar file)
  2. routines.tar (useful tools)
  3. Get also sea water routines from CSIRO
  4. And R. Pawlowicz' convenient m_map toolbox
  5. Paul Robbins made available useful routines that read the WOCE format files: more info on these routines:http://crusty.er.usgs.gov/sea-mat/.
  6. my own interface of CSIRO Neutral surface calculations (D. Jackett)

 HYDROSYS FLOWCHART

flowchart


To edit HTML, one can use the ASHE by typing ashe from any machine. You can also get the documentation. Also, see the beginner's guide to HTML.
 


Contents  

About the scripts

Most scripts related to this system are in the subdirectory Hydrosys. The same contains .f files for some routines that use Fortran code with mex4/mexsol files and associated g.f files.

Hydrosys/Neutral contains the neutral surfaces stuff (see this section).

Up: General Menu


Data file system

The hydrosys data file system was created to meetthe following criteria: easy access, reasonable memory use. Each section is associated to a header file, <section_name>_hdr.mat. This file contains some basic information on the data files (name...) plus any exotic information on the section, data treatment, pressure, bottom depths, positions... The data file names are in the Pairfiles variable. The storage is in fortran binary format. The storage precision, given in Precision is often float32. The advantage to the .mat files is that the storage size is divided by two, which is useful for large ctd files for example. The drawback of this storage is that for each property there is one file. Lumping everything into one single file would make the access uneasy and rigid (one could want, for example, to store CTD and velocity data on a 2-db scale, and nutrient on another scale).

The structure of the data is relatively staightforward and described in the routines getsdat.m (for the station data) and getpdat.m (for the pair data).

Access to the data

Station data
An example is in getsdat.m. This script, given the data directory (datadir) and the section name (secid), loads all the data at the stations. The data variables are described in the first lines of this routine (type help getsdat). If the data are not present in the data directory, they may have to be either created by the mergech.m, of converted from the old data system (see section "conversion from hydroinv to hydrosys").

Example: loads the data for section A36N, in the Atlantic.
path(path,'/data4/ganacho/web/Hydrosys')
datadir='/data4/ganacho/web/Hydrosys/Hdata/';
secid='a36n';
getsdat
g_pltopo(Nstat, Slon,Pres, Maxd(:,Itemp), Botd)
hold on;title(['Potential Temperature, Station data, ' secid ])
path(path,'/data4/ganacho/web/Matlab')
cint=[0 1 2 3 4 5 10 15 20];
[c,h]=contour(Slon,-Presctd,temp,cint)'
clabel(c,h)
xlabel('longitude');set(gca,'ylim',[-6000,0]);
 

Pair data
As for the station data, one can use getpdat.m to load all the pair data. The data variables are described in the first lines of this routine (type help getpdat).

Example: to plot the potential temperature, section A36N
path(path,'/data4/ganacho/web/Hydrosys')
datadir='/data4/ganacho/web/Hydrosys/Hdata/';
secid='a36n';
getpdat
g_pltopo(Nstat, Slon,Pres, Maxd(:,Itemp), Botd)
hold on;title(['Potential Temperature, Pair data, ' secid ])
path(path,'/data4/ganacho/web/Matlab')
cint=[0 1 2 3 4 5 10 15 20];
[c,h]=contour(Plon,-Presctd,temp,cint);
clabel(c,h) xlabel('longitude');set(gca,'ylim',[-6000,0]);


Accessing the individual data files and saving data
Given the information from the header file, the data files can be accessed automatically using the routines rhydro.m for reading and whydro.m for writing.

To read data, one wants to read the header file first: datadir='/data4/ganacho/web/Hydrosys/Hdata/';
secid='a36n';
hdrname = [secid '_pair.hdr.mat']; eval(['load ' datadir hdrname ])

For example, to read the temperature only, type : temp=rhydro([datadir Pairfiles(Itemp,:)], Precision(Itemp,:),...
MPres(Itemp), Npair, Maxdp(:,Itemp));

To read all the properties (indice iprop), one by one: for iprop = 1:Nprop pname = Propnm(iprop,:); %property name punit = Propunits(iprop,:); %property unit pfile=[datadir Pairfiles(iprop,:)];%property file name prop=rhydro(pfile,Precision(iprop,:),MPres(iprop),Npair,Maxdp(:,iprop)); eval([Propnm(iprop,:) '=prop;']); end So that each property is loaded in each variable (temp, sali, ...).

To read the velocity: gvel = rhydro([datadir Velfile], Velprec, ... MPres(iprop), Npair, Maxdp(:,iprop));

To save the properties: pprop=temp; %save the temperature ovw=0; %do not overwrite existing file whydro(pprop,[OPdir Pairfiles(Itemp,:)],Precision(Itemp,:),Maxdp,ovw)

The examples are taken from getpdat.m and g_pairset.m

Up: General Menu


Conversion from hydroinv to hydrosys

This section shows how to convert the data from the old system used by Alison, that we named hydroinv, to the new data file system, hydrosys.

Station data conversion

The station data are created from the program mergech (it merges the data from ctd with the bottle data) after interpolation of the observations to standard depths (program obs2std). The parameters to read the output of mergech are given in the following table for some of the sections. For other sections, the information can be found in the corresponding parameter files used to run geovel, in /data39/alison/phd/progs/geovel/<cruise name>/*.par, and complete the table.
 

Mergech output files

Section root name  nstat first last nvar ndep CTD
A36NA24Nflst at109 213 1 102192 101191202 5 37 yes
A48N hudson 78 * * 5 37 yes
I32S i5 106 * * 5 37 yes
I18S at93 68 * * 5 37 no
MZ_NMZ_S mozam 348 * * 5 37 no

The data from a particular cruise or set of cruises are in the "root name" file, in the directory /data39/alison/phd/data/<rootname>.<ext>, where ext='.hdr' for the header, and '.std' for the data. The data file size must be nstat*nvar(+1)*ndep*4. In the case of ctd data, then it includes the dynamic height, so there is one more variable (+1). The record length is nvar(+1)*ndep*4, as there is one record for each station. To convert a file to the new format, copy the example file conv_at109.m, that does the conversion for the North Atlantic, and modify it to the new set of parameters. For converting the station data only, set the parameter p_conv_stat to 1, and p_conv_pair to zero. Verify afterwards how the fields look like. The latter routine calls automatically conv_sdata.m, that actually does the conversion. 


Pair data conversion

The routine conv_pdata.m does the conversion from the old (hydroinv) files. The inputs file (created by geovel.f, hydroinv system) can contain several sections. conv_pdata.m is called by a parameter file that defines the variables for each section. For example, conv_at109.m does the conversion for some sections in the North Atlantic. One cn copy it and adapt it to the new conversion using the information in the table below.

Most of the geovel output files sit in the /data39/alison/phd/data/ directory. The names of the pair data files are organized as follows.
 
 

Geovel output files

Section root name  nstat npair first last nvar Rec length ndep
A11N hall 85 79 1 79 5
37+1
A36NA24Nflst at109 213 209 1101190 100189199 5 912 37+1
A48N hudson 78 76

5

I32S i5
103 1stpair last nvar 912 37+1

The root name takes the extension .hdr, .dat for station data and _pair.hdr, _pair.dat for pair data, .hdr standing for "header information" and .dat for the data fields. Information about the missing parameters can be found in the mkequats parameter file (see hydroinv ) as the mkequats program reads the pair data files. nstat stands for the total number of station in the station header file; npair stands for the total number of pairs in header and data files; first and last pairs are given for each section, when one file contains several sections. nvar is the number of properties in each file (temperature, ...). The velocity is always read after the nvar properties.

ndep contains the total number of standard depths plus the depth at the bottom. The "37" standard depths are in std.37 the "38" standard depths are in std.38
In the routine read_pairdata, the values at the bottom are removed from the data because they contain no actual iinformation and they intriduce complications in the plotting routines.

The record lenght depends on the cruise. It is usually (nvar+1)*ndep*4. See the comments on the first lines of geovel.f for a complete description, although there are unclear things in the comments. To check, verify the size of the pair data file: it should be npairtotal*(nvar+1)*ndep*4*npair.

Note: it is also possible to access directly the data from the old system (without translation into the hydrosys format) with the routines read_pairdata.m and read_stathdr.m. They read all the information in the pair header and data files, and in the station header file.

Up: General Menu


General diagram

*

Procedure

1) run woce2obs -> create observed files
2) run in a separate window ctd2std as it takes 10 min to read the ctd info for a trans-Atlantic section. Meanwhile, proceed by running obs2std. Then, finish the ctd2std run. Run mergech, then geovel. Up: General Menu

woce2obs

Purpose:

Convert the ASCII Woce format into binary direct access files. The programs is Hydrosys/woce2obs.m (read the header).
As I run it, I use a small table on a sheet of paper or in a woce2obs.log file. It looks like
Station | sili |phos | nita
--------------------------------
| 400-600 |

....
i.e., when I spot an outlier, (in the last step of the program) I zoom to get the station number, and then I write down the depth range.
After w2o_spotoutliers is run, I set gis=[ ] and run w2o_checkstat to see if those are actual anomalies. If necessary, they are manually removed:
sili( idepth_outlier , istat_outlier ) = NaN
for each outlier. These lines are copied in the woce2obs.log filein the run directory for documentation.

Finally, the data are saved: w2o_saveobsfile (and plotted)

Up: General Menu


ctd2std

Purpose:

First interpolates the ctd data at standard depths Hydrosys/ctd2std.m (read the header). ctd_step3 then fills blank stations and missing data through an interactive process. using several choices including copy top/bottom and vertical and horizontal interp/extrapolation. In addition, the p_ooxyg parameter allows the user to see oxygen bottle data along with the ctd data for each oxygen plot. The user can then use a mouse to alter the ctd data to fit the bottle data if desired. Finally, the data is saved in a memory-efficient format and contour plots of the section are drawn and printed.

Parameter file example: ctd2std_input_p21w.m Copy and adapt this file.

Parameters:
 

  • p_plot: Plots raw data along with standardized data for each station during ctd_get.
  • hdrfile: Header file from bottle data.
  • ctddir: CTD directory.
  • fctdp: Prefix of CTD file.
  • fctds: Suffix of CTD file.
  • stddpfile: Standard Depth file (DEFAULT is 0:50:8000).
  • gstatctd: Station subselection (optional).
  • opdir: Output directory.
  • cnamesec: Section name for output files.
  • p_ooxyg: shows every oxygen plot during ctd_step3 and plots the bottle data along with the ctd data.
  • IPdir: must = 'Obsdata/', which contains the bottle data, to carry on operations for p_ooxyg.

  • To run ctd2std, first type the name of the adapted input file at the matlab prompt; for example: ctd2std_input_p21w. The program will then load the header file of the specified section, in this case p21w, load the standard depths, and read the ctd data from that section. Then ctd_get.m averages the data around each standard depth point, interp/extrapolating in the intervals if necessary. Next, ctd_step3.m fills in missing data and stations, first by automatically copying near the surface and near deep bottoms if necessary. Any remaining missing data or stations are plotted for the user, who can then choose from a number of different filling methods including copy top/bottom and vertical and horizontal interp/extrapolation for each continuous hole. The user can also remove data and refill it. If p_ooxyg is activated, the oxygen data for each station will be plotted along with that station's oxygen bottle data. These stations can either be accepted as they are or altered graphically with a mouse. Any missing or altered data is recorded in the cell variable ctd2std_rec, which is then saved in the header file. This variable contains the indices of the previously missing or altered data for each station and property (for example, ctd2std_rec{57,3} would contain indices of filled or altered oxygen data for station 57). These can then be used as input for cpres to find the corresponding pressure ranges or prop(station number,property index) to find the filled/altered data values. After ctd_step3, the program saves the treated data, plots contours of the section for each property, and prints them if desired.

    Up: General Menu


    obs2std

    Purpose:

    First interpolates the observed data at standard depths Hydrosys/obs2std.m (read the header). obs2_step3 then fills holes and missing stations interactively. obs2_step4 looks for data anomalies and can treat them with user input. Finally, the standardized data is saved in a memory-efficient format and contours of the section for each property are plotted and printed.

    Parameter file example: obs2std_input_p21w.m Copy and adapt this file.

    Parameters:
     

  • gip2treat: Property subselection. Corresponds to indices of bpropnm (eg., 'oxyg'==opropnm(3))
  • gis2get: Station Subselection (optional).
  • stddpfile: Standard depth file (every 50 m by default).
  • IPdir: Source directory for the observed bottle data.
  • hdrobs: Header file from bottle data.
  • OPdir: Output directory for final saved files.
  • hdrctd: Header file from standardized CTD data (ctdstd), used to find density values.
  • p_isopyc: Parameter for the obs2_step3 option of horizontal inter/extrapolation along isopycnals.

  • To run obs2std, first type the name of the adapted input file at the matlab prompt; for example: obs2std_input_p21w. The program will then load the header file of the specified section, in this case p21w, load the standard depths, and read the observed data from that section. obs2_step1.m vertically interpolates automatically within certain adjustable limits and removes closely spaced data. obs2_step2.m vertically extrapolates automatically at deep enough bottoms and at the surface. obs2_step3.m fills holes and missing stations interactively, using user input choices such as vertical inter/extapolation, horizontal inter/extrapolation along isopycnals, and copy to the top/bottom. Any user-altered data are stored by their corresponding indices in stdd (the standard pressure vector) in the cell variable obs2std_rec, which is later saved in the header file. This variable contains the indices of the previously missing or altered data for each station and property (for example, obs2std_rec{57,1} would contain indices of filled or altered oxygen data (or whichever was the first observation property treated for that section) for station 57). These can then be used as input for stdd to find the corresponding pressure ranges or bprop(station number,property index) to find the filled/altered data values. obs2_step4.m looks for data anomalies and plots graphs with red diamonds around suspicious points. The user can either OK these plots by double clicking or remove data around these areas, which will be filled later on with obs2_step3. Finally, the standardized data is saved in a memory-efficient format and contours of the section for each property can be plotted and printed if desired.

    Up: General Menu


    merge_ctd_hydro

    Purpose:

    Creates a common header referring to ctd and bottle data at standard depths Hydrosys/merge_ctd_hydro.m (read the header).

    Up: General Menu


    geovel

    geovel.m is based on the geovel.f from the hydroinv programs. It is matlab, uses either variables that are already in memory or loads them from files (see the Data file system format). It is generally called by a parameter file for each section. The first lines of geovel.m describe the inputs/op... The bottom wedge has several options for its treatment: plane fit; constant slope (as done in geovel.f); and constant shear, or horizontal extrapolation.

    Parameter file example: gv_input_a36n.m. Copy and adapt this file. The parameter secid must correspond to the file names.
    Parameters:

  • Gis2do: station id to select from the station files. If not precized, geovel asks for them interactively.
  • deffaulttreat: default treatment for the bottom wedge. geovel treats all station with the default method first.
  • slopmx: maximum iso T or S slopes for the constant slope treatement.
  • Ptreat(optional): Treatment to do for each pair. If there Ptreat is not precized, geovel uses the default treatment. Then, each suspicious pair is analyzed interactively. The suspicious pairs are 1) too close station (less than 20 Km); 2) shallow water stations (<310m); 3) large geostrophic velocities (>1m/s) and 4) Steep topography (see g_hotpair.m for details). For each suspicious pairs, the fields are plotted and the user has choice among different methods to change the geostrophic velocity if necessary. However if all problems have been checked, geovel can be run without warning/interaction on suspicious pairs by setting the treatment id (see gv_input_a36n.m) to use for the bottom wedge in Ptreat.
  • P2plot(optional): If Ptreat is specified BUT one wants to plot some suspicious pairs, those can be specified in P2plot.
  • dynh_option(0/1). This option was used to recover exactly the results of geovel.f with deffaulttreat=4 ONLY (constant slope). The difference: with dynh_option, T and S are extrapolated and the dynamic height is computed in the bottom triangle while normally the dynamic height is directly extrapolated. Up: General Menu 


  • section plots

    To plot different properties along a section (contours with good intervals, nice topography and labels, use plt_prop.m.

    Up: General Menu


    isopycnals

    Several routines concerning isopycnals (all apply to fields along a hydrographic line). Most routines are written in Fortran and the interface (<routine>g.mex4 for SUN or <routine>g.mexsol for SOLARIS) is needed too.
  • getsigpres.m: given a field (density or other), find the pressure of a prescribed isoline. As many others, the it is a fortran routine. For details, see getsigpres.f. An example of use: find_sig_interface.m
  • find_sig_interface.m: (trivial loop using getsigpres.m. Given a number of isopycnals defined on potential density relative to a given reference, get the isopycnal positions. find_sig_interface.m does NOT alow switching reference level as was done in laybound.f. It is not needed if one uses neutral surfaces.
  • neutral surfaces: the software from Jackett and McDougall (JPO, February 1997) can be also used. It is used in getns.m. Neutral surface routines:
  • The code won't work unless the gamma.nc file is in the /data4/ganacho/ML/Neutral. To have it setup, one needs to recompile the library (gamma.a) with the makefiles Mfile-SOLARIS or Mfile-lib-SPARC. Then, the MEX files should be compiled using fmex or rather the command lines in MEXMAKE.README. More info ? here.
  • getsigprop.m/getsigprop.f: Interpolates linearily a field (T, S or other) along one or more isopycnals (or any line given by its pressure positions). Used in getsigprop_test.m.

  • Up: General Menu


    layer integrations

    Given a set of layers and a property, integlay.m integrates the property within the layers. See also the code: integlay.f. Example: layint.m
  • laybound.m computes mean, average of a property along layer interfaces and over the vertical area of a layer. Up: General Menu 


  • transport computation

    An example of transports computation for a single section and using the preceeding routines (transport due to the thermal wind relative to a given isopycnal): geotran.m

    Up: General Menu


    build equations

    mkequats, given the pair data + geost. velocity and a bunch of parameters, build a matrix for one enclosed box containing conservation equations plus flux equations for all properties in the data. mkequats is called by the input parameter file. See the example file: mk_input_natl2.m which also contains a description of all input parameters / outputs. When precizing the layer interfaces, always start with 0 or a very small number compared with the seawater sigma density range (for the surface) and finish with 100 or a large number for the bottom.

    The mkequats.m routine first call the subroutine mk_set_layprops.m that computes layer interface position, integrated properties/transports through each section and layer, and averages over the whole box.

    To compute horizontal areas of layer interfaces:
    Section 1
    *<------- length 1 --------->
    *oooooooooooooooooooooooooooo*
    *
    *
    ******ooooooooooooooooooooooooo*
    <-zc-><-------length 2-------->
    Section 2
    ydist=distance between the two sections
    zc=zonal coast
    The area is computed as in a trapeze:
    A=ydist*(sum(lay. interface. lenght)+zc)/2
    So that the decrease of the area with depth (because of topography) is taken into account by lay. interface. lenght. Of course we do not have a trapeze everywhere. This is why the parameter ydist is not provided but computed from the harea parameter and ydist*(sum(lay. interface. lenght)+zc)/2 so that it gives the right area at the surface. Zonal coast is used to increase the area from that of a trapezoidal calculation if needed. The area does not need to be very accurate in the inverse calculation.

    To compute a horizontal area enclosed by a section/continent polygon:
    areas.f, binary SUN areas (from /data39/alison/phd/progs/bin/), or the Matlab equivalent areas.m which, given a set of (latitude, longitude) locations returns the area of the polygon they define (as long as all interior angles measure less than 180 degrees). Input file example: natl.inp

    An alternative to areas.f and its equivalents is neutarea.m, which finds the area of various neutral (gamma) surfaces within a longitude/latitude box. The program reads gridded Levitus data with a program called readlev.m. The inputs are the northern and southern Latitude boundaries, the eastern and western Longitude boundaries, and Input directory for the levitus data file, the Output directory for the ascii file containing the computed areas, and the values of gamma at which to find the areas (an input gamma of 0 will produce the surface area of all ocean within the box that is both recognized by the Levitus data set and defined in the program gamman.m, which is also called by neutarea.m).

    To make inversions involving more than one box, these matrix equations will have to be combined to form a global matrix (script weldquats.m).

    optional parameters

    In addition to the main parameters (see the example linked above), there are several options. Up: General Menu

    boxinvert

    boxinvert.m does a Gauss-Markov inversion to get reference level velocities and vertical velocities. Procefure: (set parameters/type boxinvert).

    Example for Gauss-Markov:

    Options for SVD: Up: General Menu

    hydrotrans

    hydrotrans.m computes transports and uncertainties. Plot them (by calling hydrotrans_plot.m). Procedure: set parameters/type hydrotrans.
    Main parameters: Up: General Menu

    Global transports and residuals

    To compute global transports and residuals:

    1) run_global for each box. In the loop, include save_transec, get_sumresiduals and hydrotrans (p_plt_streamf=0;p_pltstream=0). 

    2) save global transports and residuals :

    eval(['save ' ModelidG '_Transec.mat Transec Amatnets_glb'])

    eval(['save ' ModelidG '_lay2sum.mat lay2sum'])

    3) make global calculation based on bhat_global, P, Transec, Amatnets_glb and lay2sum. See glbdiagnostic.m

    Summation over consecutive layers

    cumtranslay.m sum cumulatively transport from west in consecutive layers.

    Main parameters:

    Glbfigs/glbcumtransmap.m: cumulative transports on map

    Up: General Menu