Date: Tue, 12 Sep 2006 16:21:52 -0400
From: Wesley Ebisuzaki <Wesley.Ebisuzaki@noaa.gov>
Subject: Re: RT NOMADS servers
In-reply-to: <450701D5.4020808@ucar.edu>
To: Robert Rozumalski <rozumal@ucar.edu>
Cc: Jun.Wang@noaa.gov, Brandt Maxwell <Brandt.Maxwell@noaa.gov>,
 Tony Mostek <mostek@comet.ucar.edu>, Jordan Alpert <Jordan.Alpert@noaa.gov>

Robert,

   Think it is great that someone is supporting the forecast offices.  
Jun Wang is supporting
the ftp2u program and the policy is to make the program available to 
others.  

   The ftp2u program can extract specific records from a GRIB file and
do it quite quickly.  However, the partial-http transfer is a better for 
full records.

        
http://www.cpc.ncep.noaa.gov/products/wesley/fast_downloading_grib.html

It's advanges are (1) no server scripts to maintain. (2) lower CPU 
requirements on server,
(3) easier for the client to "automate" and (4) is grib2 ready.

The ftp2u program has one advantage; it can extract geographical 
subregions.  This reduces the bandwidth
but puts a heavy burden on the server.  With grib2, the cpu demands will 
increase with the jpeg2000
compression scheme. 

   One "better" method may be to combining tiling with the partial-http 
transfers.  The client would
need software to figure out the needed tiles, and to recombine them into 
a larger file at the end.

   Another possibility is that the jpeg2000 scheme allows 
"random-access".  May be possible to
download just the needed parts of a grib2 file.

      Wesley Ebisuzaki
      CPC/NCEP




Robert Rozumalski wrote:
>
>
> Greetings Wesley and Jun,
>
>   I have contacted you previously regarding the RT NOMADS server and 
> the ability of NWS Forecast
>   offices to request subsets of the 0.5 degree GFS GRIB files for 
> initializing local model runs.
>   Several SOOs have asked about the availability of the NOMADS server 
> and have commented that
>   the server is typically saturated with users.  As I understand from 
> our previous exchange is
>   that the process that spatially subsets a grib file is CPU intensive 
> and thus you limit the
>   number of users on the system, which leads me to my request.
>
>   I would like to set up a server here as a ftp2u system similar to 
> that you have running on
>   NOMADS that would provide data to NWS offices. The hardware is not 
> an issue and I have good
>   access to global GFS grib files, so what I am asking from you are 
> the necessary ftp2u scripts
>   (ftp2u_gfs_dir.sh) and grib file processing routines. Can you make 
> these files available to
>   me for this purpose? One benefit for you is that it might take some 
> of the load off the NOMADS
>   server but most importantly it is helping out the SOOs and field 
> offices.
>
>
> Regards,
>
> Bob
>
>