About the GDS

User's Guide

Administrator's Guide

Configuration file reference

About the GDS

The GrADS-DODS server is a stable, secure, easy to install, and easy to configure data server that provides subsetting and analysis services across a network.

The subsetting capability allows a user to request a specified range of time and space from a large dataset, eliminating the need to download everything simply to access a small relevant portion of a dataset.

The analysis capability allows users to retrieve the results of a GrADS operation applied to one or more datasets on the server. Examples of analysis operations include basic math functions, averages, smoothing, differencing, correlation, and regression. This eliminates the need for a client to download large datasets to perform an operation that only generates a small amount of data---instead, the operation is performed on the server, and the small amount of generated data is transferred over the network. Analysis can include previous analysis results and datasets uploaded by the user, as well as datasets hosted on the server.

These services can be provided for any GrADS-readable dataset. Making datasets available on the server is simply a matter of typing in the location of the dataset in the configuration file. Datasets can be added to and removed from the server listings on-the-fly, with no need to restart the server, providing flexibility for data providers whose holdings change rapidly.

In addition, there are a number of parameters to control the performance and resource usage of the server. The central component is a Java servlet, which connects two pieces of pre-exising software: GrADS (see below), which handles the actual data operations; and the DODS core, which implements the protocol by which information is exchanged with the client across the internet. The Jakarta Project's Tomcat package is used to provide an execution environment for the servlet. All required software is free and open-source, except the JVM implementation in some instances.

For more on GrADS, see the GrADS home page.
For more on Tomcat, see the Tomcat home page.
For more on servlets, see Sun's Java Servlet home page.

What is DODS?

DODS (Distributed Oceanographic Data System) is a software framework used for data networking. DODS provides tools for making local data accessible to remote locations regardless of local storage format. See the DODS home page for more information.

For more on DODS, see the DODS home page.

back to document top

GrADS-DODS Server - User's Guide

Table of Contents

• Accessing data from a web browser
  • Browsing server contents
  • Retrieving data subsets as ASCII text
    
    
• Accessing data from a DODS-enabled analysis tool
  • Opening a dataset and retrieving data subsets
  • Performing remote analysis
  • Uploading data
    
    
• Using remote data in scripts

Accessing data from a web browser

Browsing server contents

To browse a directory of the datasets being served on a GDS, point your web browser to the base URL of the GDS. This will usually be a URL of the form http://machine.domain:9090/dods/.

This directory listing will provide links to "info", "dds" and "das" for each dataset. The first link provides a Web page with a brief summary, followed by a complete metadata listing, for the dataset. The other two provide links to the DODS Data Descriptor Structure, which specifies the logical structure of the dataset, and Data Attribute Structure, which provides descriptive information about the dataset.

You can also retrieve a complete dataset listing for a GDS by adding /xml to its base URL.

If you are given a GDS dataset URL, you can enter that URL in your web browser, and get the "info" listing. This listing will contain links back to the dataset directory for the GDS.

Note: Many DODS data objects with distinct URLs will often be considered a single "dataset" from a scientific point of view. However, the word "dataset" is used here in a technical sense, to mean a single DODS data object.

Retrieving data subsets as ASCII text

The GDS can provide subsets of any dataset it is serving, in ASCII comma-delimited format. To retrieve a subset, enter a URL of the form http://gds-base-url/dataset.ascii?constraint.

The constraint portion of the URL should be a DODS constraint expression. Some basic constraints:

A constraint of the form var will request the complete contents of the variable.

A constraint of the form var[a:b] will return the subset of the variable defined by a and b.

A constraint of the form var[a:n:b] will return every n th element of the subset defined by a and b.

For subsets of variables with multiple dimensions, each dimension must have a constraint. So a constraint for a subset of a three-dimensional variable would appear as var[a1:b1][a2:b2][a3:b3], or var[a1:n1:b1][a2:n2:b2][a3:n3:b3]

back to table of contents on User's Guide

Accessing data from a DODS-enabled analysis tool

Opening a dataset and retrieving data subsets

You can retrieve data from a GrADS-DODS server using any DODS-enabled desktop analysis tool (aka "client") such as GrADS, Ferret, Matlab, or IDL. To do this, provide a URL instead of a path name to your client's open command. You can then use the data as if it were a local data set; the client will automatically retrieve data as needed. You may want to also read the notes below on optimizing your scripts to use remote data.

Performing remote analysis

Remote analysis is very useful for doing calculations on remote data that use a large quantity of input data but generate a small output, such as averaging and correlation functions. This type of calculation will run much faster on the server, and you will only need to download the small result instead of the entire set of inputs.

In order to do data analysis on the server, you construct a URL containing a GrADS expression, and then open that URL with your client. The server will perform the analysis task, and return the results to the client as a DODS dataset with a single variable called result, containing the results. This result dataset can then be used exactly as it if were an original data set (see above). It can even be used as input to further analysis expressions on the server, allowing calculations that use multiple stages of intermediate results to be performed remotely.

The URL for an analysis operation is created by appending _expr_ to the server's base URL, as follows:
http://machine_name:9090/dods/_expr_; followed immediately by three sets of curly braces containing arguments, as follows:

{dataset1,dataset2,...}{expression}{x1:x2,y1:y2,z1:z2,t1:t2}

The first set of curly braces contains a list of all the datasets on the server that are used in the GrADS expression. If the datasets are in a subdirectory, the name of the subdirectory should be included in the dataset name.

Source datasets can include the results of previous analysis expressions, allowing you to perform multi-stage calculations. To use a previous analysis result as a source, put its shorthand name in the list of datasets. The shorthand name for a result dataset is contained in the dataset's title attribute (in GrADS you can view this by typing q file), and has the form _expr_nnnn where nnnn will be some number.

The second set of curly braces contains the GrADS expression to be evaluated. This describes the actual calculation to be performed, using GrADS syntax (see the GrADS home page).

The third set of curly braces contains the boundaries for the expression evaluation in world coordinates (latitude, longitude, elevation, time). These boundaries may not vary in more than two of the four dimensions. The first three coordinate pairs should be given as real numbers. The last pair are time coordinates, and should be in the format recognized by the set time command in GrADS: [hh[mm]z][dd][mon][yyyy]. For example, 0z1jan2000.

Specifically, the analysis is performed as follows:

1. GrADS is invoked.
2. The source datasets are opened in the order they are listed in the first set of curly braces
3. The dimension environment is set according to the parameters in the third set of curly braces.
4. The expression in the second set of curly braces is evaluated and saved as a new dataset.

Thus, a variable in the nth listed dataset should be referred to as var_name.n in the analysis expression. For instance, if dataset2 contains a variable called foo, this variable should be referred to in the expression as foo.2. The expression will be evaluated against the grid of the last dataset opened.

Here are some examples of remote analysis, using GrADS as a client. Please note that for clarity and avoidance of strange browser behavior, the URLs in the following examples have been split into more than one line because they are so long, but they should be entered as one line.

1. Global Averaging:
   The following expression will return a timeseries of globally-averaged monthly mean surface air temperatures based on NCEP reanalysis data being served on the GDS at the Climate Diagnostic Center (http://web2.cdc.noaa.gov:9090/):
   
   ga-> sdfopen http://web2.cdc.noaa.gov:9090/dods/_expr_{sfc_air_mon_mean}
   {tloop(aave(air,global))}{0:0,0:0,1:1,jan1948:jan2001}
   
   
2. Variable Comparison:
   A GDS running at NCAR (http://motherlode.ucar.edu:9090/) is distributing a set of ensemble members from the "Climate of the 20th Century" runs of the COLA atmospheric general circulation model. We will compare the relative humidity "rh" from the first two datasets, namely "C20C_A" and "C20C_B". Suppose we want to find a global time-average of their difference at the 1000 mb level in 1960. Using GrADS as our client, we would open the following URL:
   
   ga-> sdfopen http://motherlode.ucar.edu:9090/dods/_expr_{C20C_A,C20C_B}
   {ave((rh.1-rh.2),time=1jan1960,time=1dec1960)}
   {0:360,-90:90,1000:1000,1nov1976:1nov1976}
   ga-> display result
   
   The analysis results are returned in the variable "result" in the opened dataset. Note that the world coordinate boundaries specified in the third set of curly braces fix the time to 1nov1976 -- this can be set to any arbitrary time because the time dimension specification is overridden by the GrADS expression which tells the server to average over the period from January 1960 to December 1960.
   
   
3. A More Complex Analysis Operation:
   Suppose you wanted to calculate the mean 500mb height anomaly associated with warm tropical SST anomalies. Use the Reynolds SST Analyses to create a time series of the area-averaged SST anomaly between 180 and 90W and 10N and 10S. An "ENSO" mask is then defined for SST anomalies greater than 1 degree. Using this mask, calculate a mean 500mb height from the the NCEP/NCAR Reanalysis Data associated with the warm SST anomalies. All these operations are packaged into a single URL:
   
   ga-> sdfopen http://cola8.iges.org:9090/dods/_expr_{ssta,z5a}
   {tmave(const(maskout(aave(ssta.1,lon=-180,lon=-90,lat=-10,lat=10),
   aave(ssta.1,lon=-180,lon=-90,lat=-10,lat=10)-1.0),1),
   z5a.2(lev=500),t=1, t=600)}
   {0:360,0:90,500:500,jan1950:jan1950}
   
   The GrADS script sstmask.gs illustrates the use of this example and contains some additional graphics commands to display the analysis result.

Uploading data

The GDS also allows the client to upload data that can then be used as a source in analysis expressions. This capability is still experimental, and will be fully documented and made officially available with the release of GrADS 1.9.

back to table of contents on User's Guide

Using remote data in scripts

You do not have to do anything special to adapt a script to work with remote data. All you need to do is replace local filenames with URLs. This is because from your client program's point of view, a remote dataset behaves exactly like a local dataset except that access is slower.

However, because remote data retrieval is not instantaneous, existing scripts that do not take this into account may run very slowly. Thus it is often desirable to modify the script to improve its efficiency.

The key to writing efficient scripts is fine-tuning your use of I/O requests. DODS-enabled clients such as GrADS only provides the illusion of a continuous connection with a remote dataset. In fact, a new connection is made to the server every time you request data from the I/O layer (for instance by using the "display" command). The speed of these connections is dependent on network latency and server response time, but is generally much slower than an equivalent request from a local disk. Thus, reducing the number of network connections, and the quantity of data sent over the network, will often significantly speed up your script.

Here are some guidelines for writing efficient scripts. The examples given use the GrADS scripting language, but the principles apply to most DODS-enabled clients:

• Avoid multiple opens. Opening a GDS data file generates as many as eight separate network requests, so try to avoid opening the same file more than once.
  
  
• Store remote data locally if you plan to reuse it. DODS has a limited ability to cache remote data locally, the way a web browser does with web pages. However, this only works when you request the exact same subset. Thus, if you use different parts of the same remote data subset in multiple places in your script, you are actually requesting it multiple times over the network.
  
  To avoid this, request data once from the server, and then store it in local memory or on disk. In GrADS, you can do this using 'define' or 'set gxout fwrite'. For example:
  
    'sdfopen http://cola8.iges.org:9090/dods/mrf0930'
    'set lat 22 52'
    'set lon 233 295'
    'set t 1 15'
    'define slp = slp/100'
    'd slp'
  
  Now you can use the variable 'slp' as many times as you wish in your script without any additional network requests.
  
  Note to GrADS users: The 'define' command automatically loops through each time step in the dimension environment, so using 'define' may not always improve your performance if you are accessing time series at a single point. For example:
  
     'set lon -90'
    'set lat 40'
    'set lev 500'
    'set t 1 15'
    'define ztser = z'
  
  The above example will result in 15 separate requests for data from the server, one for each time. Each request will only obtain a single data value! If time is the only varying dimension, it is far better to display the data using the 'display' command (which doesn't automatically loop through time) or, if you're going to display the data more than once, use 'set gxout fwrite' to preserve a local copy. We will be fixing this behavior in version 1.9 of GrADS.
  
  For a more complex example, take a look at the GrADS script meteogram.gs, which uses 'define' and the 'set gxout fwrite' commands to draw a graphically detailed meteogram for any location in the forecast model domain.
  
  
• Evaluate expressions on the server side when appropriate. It may save time and server resources to package your request into an analysis expression. A good rule of thumb is to use analysis expressions when the size of the result data set is smaller than the total size of the input data, e.g. when doing spatial or time averaging.
  
  For example, the following script example opens two separate MRF forecast data files and then uses the 'const' function to merge one variable from each of them to form one continuous time series:
  
     'sdfopen http://cola8.iges.org:9090/dods/mrf010800'
    'sdfopen http://cola8.iges.org:9090/dods/mrf010800b'
    'set lat 0'
    'set lon 0'
    'set t 1 31'
    'define tt = const(t.1,0,-u) + const(t.2,0,-u)'
    'd tt'
  
  This second version of the script example creates the same continuous time series using an analysis expression. This script runs three times faster than the first version, and hits the server half as many times.
  
    baseurl    = 'http://cola8.iges.org:9090/dods/_expr_'
    datasets   = '{mrf010800,mrf010800b}'
    expression = '{const(t.1,0,-u)+const(t.2,0,-u)}'
    dimensions = '{0:0,0:0,1000:1000,00Z08JAN2002:00Z23JAN2002}'
    'sdfopen '%baseurl%datasets%expression%dimensions
    'set t 1 31'
    'define tt = result.1'
    'd tt'
  
  Note however, that there is an overhead on the server associated with each analysis expression. Thus, if the size of the expression output is the same as, or larger than, its inputs, it will be more efficient to retrieve the inputs first, and do the analysis locally.
  
  
• Try to move data requests outside loops. If you are looping over a grid of data points, when possible you should retrieve the whole area you intend to use with a single request, and store it locally for use in the loop. Otherwise you will be making a new network request for each data point inside the loop, which can cause extremely slow performance.
  

GrADS-DODS Server - Administrator's Guide

Table of Contents

• Installation
  • Requirements
  • Downloading and setup
  • Putting your datasets online
    
    
• Controlling the server
  
  • Starting and stopping
  • Web-based administration
  • Checking the server status
    
    
• Configuration
  • Startup parameters
  • Configuration file settings
  • Tomcat settings
  • Security
    
    
• Advanced topics
  • Static web pages
  • Integrating with Apache
  • Deploying to a different servlet container
  • Building from source

Installation

Requirements

The GDS can run on any UNIX platform for which both Java and GrADS are available.

You will need a Java Virtual Machine (JVM) that supports Java v1.2 or higher. Enter java -version at the Unix command prompt to find out what JVM you have currently installed on your system. The Java Virtual Machine is a free download either from Sun Microsystems, or your operating system manufacturer's website.

You will also need GrADS. Because the server uses some new features in GrADS, you will need version 1.8 or higher. Handling station data and client uploads requires version 1.9 or higher. The latest version of GrADS is available at the GrADS home page.

Download and setup

The latest version of the GDS is available at the GDS home page as a compressed tar archive.

You do not need root user access to run the GDS. There is no build or system install process, because it is a cross-platform Java application. And any number of GDSes can be run on the same system, as long as they are configured to use different ports (see Tomcat settings)

After unpacking the archive, all you need to do is edit the configuration file, and tell the GDS where to find GrADS, by editing the <invoker> tag. If you are not using a full GrADS distribution, make sure the GrADS executable you specify is capable of opening the types of dataset you wish to use.

At ths point you should be able to start the server and view the example dataset.

Putting your datasets online

Next you will want to put your data online.

First, make sure that all of your datasets are ready to be opened by GrADS. (for instance, if you have GRIB data, you will need to generate CTL and map files).

Once this is done, all you need to do to put your datasets online is tell the GDS where they are, using the configuration file. This is done by adding <dataset>, <datadir>, and <datalist> tags inside the <data> tag. If you want, you can organize the way the data appear online using <mapdir> tags.

Note that the GDS does not attempt to access datasets until the first time they are requested by a client, so it may not immediately complain about unusable datasets. Before you invite others to use your server, therefore, it is a good idea to make sure that all of the datasets you are serving work properly, by opening them in your own DODS-enabled client.

Also note that DODS does not provide a way to notify clients that a given dataset has changed. Thus, if you are planning to serve a changing collection of data, such as real-time observations, plan to always post new data under new handles, rather than overwriting the contents of existing datasets. Otherwise, a user making the wrong assumption could unknowingly generate incorrect results, or become very confused.

Once you have your data loaded and working, it is highly recommended that you familiarize yourself with the configuration options and administrative tools available by reading the remainder of this documentation.

return to table of contents on Administrator's Guide

Controlling the server

Starting and stopping

There are four scripts in the server home directory that are used to control the GDS:

startserver - Starts the GDS as a background task.

stopserver - Shuts down the GDS.

rebootserver - Restarts the GDS

cleanup - Restarts the GDS, deleting all temporary files

Notes:

• In general, the stopserver script should be preferred to killing the server process, as it allows the GDS to shut down gracefully.
  
  
• Once the GDS is running, it must be stopped before it can be started again. Otherwise, startserver will print a Java "ConnectException" message and fail.
  
  
• If the GDS fails because of an invalid configuration, Tomcat will start anyway. Thus, when the GDS prints an invalid configuration message, the stopserver script must be run before starting it again.
  

Web-based administration

The GDS has a web-based administration interface which is accessed by URLs of the form:

/admin?auth=authorization_string&cmd=command_string

The authorization_string given must match the auth setting given in the <service-admin> configuration tag. The command_string can be one of the following:

reload: Checks for changes to the server configuration. This updates the data catalog, privilege sets, and all other settings contained in the GDS configuration file, without the need to take the server offline.

clear: Removes all temporary entries from the catalog.

Checking the server status

To use the script, edit it to use the base URL and home directory of the server it is to check, and add it as a cron job with the desired frequency. Once in the crontab, the script can be temporarily disabled by placing a file called block_check_gds in the home directory of the GDS being checked.

return to table of contents on Administrator's Guide

Configuration

Startup parameters

The startup script, startserver, contains the following settings.

HEAP_SIZE: the amount of memory that the Java Virtual Machine will reserve for Tomcat and the GDS.

By default, the GDS is set to use 150MB. However, under heavy usage, with large numbers of datasets, it may need more than this. Conversely, with only a few datasets online and occasional use, it will need far less. It is worth experimenting with different heap sizes if you want to reduce the GDS footprint, or if the GDS is reporting out of memory errors.

CONFIG_FILE: the name of the configuration file to use. By default this is set to gds.xml. If the pathname is relative, it is resolved with respect to the home directory of the server.

Configuration file settings

The GDS derives settings for all of its modules from an XML configuration file. The name of this file is specified in the startup script; the default is gds.xml. This file is read every time the server starts, and every time a reload command is given using the administrative web interface.

The configuration file reference describes the XML tags that can be used in this file.

Tomcat settings

The configuration file for Tomcat is tomcat/conf/server.xml. This file can be edited to change the network port that the GDS runs on, and to increase the size of the connection pool, among other things.

An explanation of Tomcat and the settings in its configuration file can be found at the Tomcat home page

Security

Both Tomcat and the GrADS-DODS Server support IP-address-based security. Each can be given its own security settings. Tomcat will allow or deny access to the server based on the settings in its configuration files. For finer grained control, use the <ip_range> and <privilege> tags in the GDS configuration file. These settings allow or deny access to specific datasets and server features.

return to table of contents on Administrator's Guide

Advanced Topics

Static web pages

The GDS comes with some static web pages which can be accessed using Tomcat (including this manual). These pages are located in the subdirectory tomcat/webapps/ROOT. In general, putting your static content on a separate web server is recommended, since will usually be more efficient than using Tomcat for this. However, Tomcat's static content directory provides an easy way to put up some information about the datasets you are serving, without setting up a separate web server.

Integrating with Apache

The GDS can fairly easily be integrated into an existing Apache web site. All that is required is to set up the link between Tomcat and Apache. Consult the Tomcat home page for more on this process.

Deploying to a different servlet container

It is not necessary to use the copy of Tomcat that is distributed with the GDS, to run the GDS. Any servlet container that supports the Java Servlet API 2.2 or higher can be used.

The GDS web application is in the directory tomcat/webapps/dods. You can generate a Web Application archive (WAR) file by running the makewar script under the path src/. This will generate an archive called dods.war. To add the WAR to your servlet container, follow the servlet container's instructions.

Once you have added it, you will need to set the property anagram.home to the directory containing the rest of the GDS distribution. This can be set either as a Java system property (using the -d switch when invoking Java) or as a servlet context property (follow the servlet container's instructions).

Building from source

Source code is included with the GDS, under the path src/.

• To generate Java documentation from the source code, change to src/ and run the makedoc script.The documentation will be placed in src/doc/.
  
  
• To recompile the source code, change to src/ and run the makejar script.
  

Please note that, while you are welcome to browse and modify the source for your own purposes, and submit patches and/or new features to COLA, you are not granted permission to distribute modified versions of this software.

GrADS-DODS Server - Configuration File Reference

Table of Contents

• Basic format description
  
  
• Example of a configuration file
  
  
• Tag hierarchy
  
  
• Tag definitions
  • <allow>
  • <analyzer>
  • <catalog>
  • <data>
  • <datadir>
  • <datalist>
  • <dataset>
  • <deny>
  • <dods>
  • <filter-*>
  • <gds>
  • <grads>
  • <invoker>
  • <ip_range>
  • <log>
  • <log_override>
  • <mapdir>
  • <mapper>
  • <privilege>
  • <privilege_mgr>
  • <service-*>
  • <servlet>
  • <uploader>

Basic format description

The GDS configuration file is in XML format. This means it must start with the following line:

<?xml version="1.0" encoding="ISO-8859-1"?>

and then contain a set of nested "tags" .Each tag specifies the configuration options for a particular module of the server. In cases where configuration for a module is more complex, the tag for that module may contain further tags. Tags are written in one of two forms. For a tag with no contents, the syntax is:

<tagname attribute="value" ... />

and for a tag with contents, it is:

<tagname attribute="value" ...>
  (other tags)
</tagname>

The GDS configuration file does not currently use plain text or CDATA segments.

back to table of contents on Configuration File Reference

Example of a configuration file

<?xml version="1.0" encoding="ISO-8859-1"?>
<gds name="my_server home="http://www.some.edu/~jdoe/data_server_info.html">
  <catalog temp_entries="500">
    <data>
       <dataset name="my_data" file="/data/my_data.ctl" format="ctl"/>
       <datadir name="my_model_runs" file="/data/mruns/" suffix=".nc" format="nc"
                doc="http://www.some.edu/my_online_data/mrun_info.html" />
       <mapdir name="private_data">
         <datalist file="secret_datasets.lst"/>
       </mapdir>
    </data>
  </catalog>
  <log mode="rotate">
    <log_override module="tool/invoker" level="verbose"/>
  </log>
  <grads>
    <invoker grads_dir="/home/jdoe/grads/" time="600"/>
    <analyzer storage="250" time="60"/>
    <dods subset_size="2000" />
    <uploader/>
  </grads>
  <mapper>
    <service-admin auth="sDFe294f3nv034u8"/>
  </mapper>
  <privilege_mgr default="public">
    <ip_range mask="127.0.0.1" privilege="full" />
    <ip_range mask="192.168" privilege="full" />
    <privilege name="full" /> <!-- no restrictions -->
    <privilege name="public"
               analyze_allowed="false"
               abuse_hits="1000"
               abuse_timeout="24/>
      <deny path="/private_data">
    </privilege>
  </privilege_mgr>
  <servlet>
    <filter-overload limit="20" />
  </servlet>
</gds>  

back to table of contents on Configuration File Reference

Tag hierarchy

The following table shows the tag structure of the configuration file. This corresponds to the runtime structure of the server. The top level tag must always be <gds>. Each tag must be contained by the tag immediately to its left, and can contain any of the tags to its right. Tags followed by * can appear multiple times.

back to table of contents on Configuration File Reference

Tag definitions