The File Exchange Interface (FEI)

John Rector
Jet Propulsion Laboratory
MS 186-414
4800 Oak Grove Drive
Pasadena, CA 91109
818.354.4313
jar@next1.jpl.nasa.gov

The File Exchange Interface (FEI) is a distributed file access
system supporting a single logical view of the file sets. The file
set can be divided among several servers running on different
machines at several sites. FEI supports the following basic file
transfer functions: get, add, replace, delete and rename. While
transferring a file, the user can elect to include compression,
encryption and cyclic redundancy checks (CRC). Compression during
transfer speeds delivery to remote sites. A simple compression
scheme is used so the compression time is not long. Experience has
shown image files compress from 50% to 80%. Encryption and CRCs
allow the save transfer of critical files like up-link commands
where we must be sure that no changes have occurred.

All files are grouped into file types. Each file type has a name,
like gllSsiEdr, Galileo's camera images. The file type is used to
locate the proper server and directory and to check user access on
the file type. File type names are short and easy for users to
remember. They also allow administrators to change the placement of
files, moving them to different machines and directories, even at
different sites, without effecting the user community.

Authentication is handled by Kerberos. A connection can be made and
then used days later, so Kerberos ticket expiration is less of a
factor. Access control is implemented by assigning users to roles. A
role gives the user transfer capabilities on specific file types.
Since a user can have multiple roles, for different instruments on
different missions, spanning a set of machines and servers,
administration is complex requirements is possible. Role information
is stored in a central database, so users to not have to be given
accounts on the server machines, thus increasing security.

All FEI data is stored in a central database, currently Sybase, but
we have plans to move to an object oriented database in within a
year. FEI data stored in the database, includes, server
configuration parameters, file type descriptions and mappings to
machines and directories, access roles information, significant
event information reported by servers and specification of demon
processes that should be called using RPCs whenever a file of a
defined type enters the system. The database is also used to stored
meta-data about files.

Since configuration information is central, an FEI server can be
brought-up on any machine having access to the database. Servers are
multi-threaded so they can use SMP machine architectures making
them scalable. A mission can increase performance by adding more
CPUs. This need not be done in the early planning stages when
performance requirements are hazy, but can be done later, even after
a mission is in operation, when exact requirements are realized.
Currently the servers run on SUN Solaris, but ports could be made to
any operating system that conforms to the POSIX thread standard.

FEI supports demons, processes that will fire when a file transfer
operation is performed on a particular file type. Currently the FEI
servers communicate with demons using RPCs. In the future we plan to
add distributed object technology. Here are a few examples of the
use of demons. When a file comes in, the demon can parse the file's
header, automatically placing keyword/value pairs in the database.
This information can then be used to locate files with particular
attributes. The file type and file name is included with the
meta-data. Also, a demon, can check a file's header to see it a copy
should be move to another machine. For example, some images should
be made available to the public immediately. A demon can check a
file, an if appropriate, make a copy and send it to a W3 server on
another machine across a fire-wall where it can be accesses over the
Internet.

FEI also supports subscription. A process can subscribe to a file
type. As soon as a file of that file type arrives in the system, a
copy is sent to the client process. The client can then process the
file or store the local copy at a remote site. Subscription permits
automation of image processes in a world-wide distributed science
community. Once a file is processed, it can be sent back to an FEI
server as another file type where it can then be read by other
subscribers waiting for the processed file.

FEI is current in operation in JPL's Multimission Image Processing
Section supporting the Galileo Mission.