ERDDAP Easier access to oceanographic data
|
Brought to you by
NOAA
NMFS
SFSC
ERD
|
ERDDAP > Information
ERDDAP (the Environmental Research Division's Data Access Program) helps humans
and computer programs download
scientific data (for example, satellite data, climate model data, buoy data)
in common file formats and make graphs and maps.
The Problems that ERDDAP Tries To Solve
Without ERDDAP, when a person (or a computer program) looks on the internet
for a specific type of scientific data (for example, satellite sea surface temperature data),
there are problems ...
- The datasets of interest are hard to find because they are at many different web sites.
- Each site requires a different protocol to request the data
(for example,
HTTP GET,
XML,
SOAP+XML,
DAP,
WCS,
WFS,
SOS,
or an HTML form).
- Each site returns the data in a different format
(for example,
XML, SOAP+XML, DAP binary data stream, ASCII text, HDF 4, HDF 5, NetCDF, ...)
and it isn't the common file format that you want.
- Data from different sites is hard to compare
because the dates+times are expressed in different formats
(for example, "Jan 2, 1985", "02-JAN-1985", "1/2/85", "2/1/85",
"1985-01-02", or days since has Jan 1, 1980, or ...).
ERDDAP's Solutions
- ERDDAP is a web-based service that aggregates data from diverse sources
and offers a simple, consistent way to access the data.
- ERDDAP can serve local or remote data.
Datasets in ERDDAP can be local (on the same computer) or remote
(accessed via the web).
- ERDDAP offers different ways to search for datasets of interest.
For example,
full text search and
search by category.
- ERDDAP lets you make requests in a standardized way, regardless of the data source's request protocol.
ERDDAP also provides Data Access Forms (web pages) which help humans create the DAP requests.
OPeNDAP's
DAP
is the recommended
IOOS
DMAC
data transport mechanism a
NASA EOSDIS standard.
(DAP is great!)
ERDDAP translates the your request from the DAP format to the data source's request format
and converts the response to one of the simple data structures.
Then ERDDAP reformats the data in the common file format of your choice
(for example, .html table, ESRI .asc, Google Earth .kml, .mat, .nc, .csv, .tsv, .json, .xhtml, .png)
and sends the file to you.
Other protocols for requesting the data (for example,
WCS,
WFS, and
SOS)
may be added/supported in the future.
ERDDAP is structured for these additions and there don't seem to be any impediments.
- ERDDAP uses just two basic data structures.
- Since it is difficult for human clients and computer clients to deal with a large
number of dataset structures, ERDDAP uses just two basic data structures:
- gridded data (for example, for satellite data and model data) and
- tabular data (for example, for in-situ buoy, station, and trajectory data).
- Certainly, not all data can be expressed in these structures, but much of it can.
Tables, in particular, are very flexible data structures
(look at the success of relational database programs).
- This makes data queries easier to construct.
- This makes data responses have a simple structure,
which makes it easier to serve the data in a wider variety of standard file types
(which often just support simple data structures).
- This makes it easier to compare data from different sources.
- And even if this sounds odd to you, most ERDDAP clients will never notice --
they will simply see that all of the datasets have a nice simple structure
and they will be thankful that they can get data from a wide variety of sources
returned in a wide variety of file formats.
- Other data structures could be supported in the future if called for.
- ERDDAP isn't just for oceanographic data.
ERDDAP is a Data Access Program that was written at
NOAA
NMFS
SFSC
ERD.
The ERDDAP server at ERD serves oceanographic data,
but ERDDAP (the program) can serve any gridded or tabular data.
- ERDDAP requests can be made in user units.
Although requests for gridded data in ERDDAP can be made with array indices (as with DAP),
requests can also be in user units (for example, degrees east).
- ERDDAP sends results in common data file formats.
The results can be returned in any of several common data file formats
(for example,
.html table,
ESRI .asc,
Google Earth .kml,
.mat,
.nc,
.csv,
.tsv,
.json,
.xhtml),
instead of just the original format
or just the DAP transfer format (which has no standard file manifestation).
These files are created on-the-fly.
Since there are few internal
data structures, it is easy to add additional file-type drivers.
- ERDDAP standardizes the variable names and units for longitude, latitude, altitude, and time in the results.
To facilitate comparisons of data from different datasets,
the requests and results in ERDDAP use standardized space/time axis units:
- longitude is always in degrees_east.
- latitude is always in degrees_north.
- altitude is always in meters with positive=up.
- time when formatted as a number is always in "seconds since 1970-01-01T00:00:00Z"
(which is UDUNITS-compatible)
and,
when formatted as a string, is formatted according to the
ISO 8601:2004 "extended" format standard
(YYYY-MM-DDThh:mm:ssZ, for example, "1985-01-02T00:00:00Z").
Also, to avoid time zone and daylight savings time confusion,
time values are always converted to the UTC time zone.
This makes it easy to specify constraints in requests without having to worry about
the altitude data format (are positive values up or down? in meters or fathoms?)
or the time data format (a nightmarish realm of possible formats and time zones).
This makes the results from different data sources easy to compare.
- ERDDAP lets you request .png and .pdf image files with graphs and maps of the data
in addition to the actual data.
Three special uses of these images are:
- Web page authors can
embed a graph with the latest data in a web page
using an HTML <img> tag.
- Anyone can use ERDDAP's Slide Sorter
to build a personal web page that displays graphs with the latest data
(or other images or HTML content), each in its own, draggable slide.
- Anyone can use or make
Google Gadgets
to display images with the latest data on their iGoogle home page.
- ERDDAP adds metadata.
Many data sources have little or no
metadata
(for example, CF metadata)
describing the data.
ERDDAP lets (and encourages) the administrator to describe metadata which
will be added to datasets and their variables on-the-fly.
- ERDDAP has
REST- and
ROA-style
links to make its services available to computer programs.
These features can be used to build another web service on top of ERDDAP (making ERDDAP do all the work!).
ERDDAP is not intended to be a high-level data exploration/graphing service.
Instead, ERDDAP is intended to provide services for such web sites and programs.
So if you don't like ERDDAP's interface, we encourage you to build your own, and use ERDDAP as the foundation.
Read more about the Computer Program interface.
- ERDDAP processes data in chunks.
To save memory (a big issue) and make responses start sooner,
ERDDAP processes data requests in chunks --
repeatedly getting a chunk of data from the source, cleaning it up (for example, adding
metadata),
and sending that to the client.
For many data sources (for example,
SOS
sources), this means that the first chunk of data (for example, from the first sensor)
gets to the client in seconds
instead of minutes (for example, after data from the last sensor has been retrieved),
reassuring the client that the data is coming (albeit slowly).
From a memory standpoint, this allows numerous large requests
(each larger than available memory) to be handled simultaneously.
- ERDDAP has a modular structure.
ERDDAP is structured so that it is easy to add different components
(for example, a class to request data from SOS server and store it as a table).
The new component then gains all the features
and capabilities of the parent
(for example, support for DAP requests
and the ability to save the data in several common file formats).
Is ERDDAP a solution to everyone's data distribution problems?
No. But hopefully ERDDAP fills some common needs.
You can
Set Up Your Own ERDDAP Server
and serve your own data.
Contact - If you have questions, suggestions, or comments about ERDDAP in general
(not this specific ERDDAP installation),
please email bob dot simons at noaa dot gov .
ERDDAP Version 1.10
DISCLAIMER OF ENDORSEMENT
Any reference obtained from this server to a specific commercial product,
process, or service does not constitute or imply an endorsement by CoastWatch,
NOAA, or the United States Government of the product, process, or service, or
its producer or provider. The views and opinions expressed in any referenced
document do not necessarily state or reflect those of CoastWatch, ERD,
NOAA, or the United States Government.
DISCLAIMER FOR EXTERNAL LINKS
The appearance of external links on this World Wide Web site does not
constitute endorsement by the
Department of Commerce/National
Oceanic and Atmospheric Administration
of external Web sites or the information, products or services contained
therein. For other than authorized activities, the Department of Commerce/NOAA does not
exercise any editorial control over the information you may find at these locations. These
links are provided consistent with the stated purpose of this Department of Commerce/NOAA
Web site.
DISCLAIMER OF LIABILITY
Neither the data providers, ERD, CoastWatch, NOAA, nor the United States Government,
nor any of their employees or contractors, makes any warranty, express or implied,
including warranties of merchantability and fitness for a particular purpose,
or assumes any legal liability for the accuracy, completeness, or usefulness,
of any information at this site.
USAGE LIMITATIONS
The SeaWiFS images and data from this site may be used for free, but not
redistributed; all other images and data from this site may be used and
redistributed for free.
CONTACT
Please email questions, suggestions, or comments
regarding this web page to bob dot simons at noaa dot gov .
We would really appreciate knowing if you use ERDDAP.
If a something doesn't work and you think it should,
please include the URL?query and the error message in the email.