FEI Concepts

What is the FEI?

The File Exchange Interface (FEI) is a distributed file access system supporting a single logical view of the file sets. The file set can be divided among several servers running on different machines at several sites. FEI supports the following basic file transfer functions: get, add, replace, delete and rename. While transferring a file, the user can elect to include compression, encryption and cyclic redundancy checks (CRC). Compression during transfer speeds delivery to remote sites. A simple compression scheme is used so the compression time is not long. Experience has shown image files compress from 50% to 80%. Encryption and CRCs allow the save transfer of critical files like up-link commands where we must be sure that no changes have occurred.

All files are grouped by file type. Each file type has a name, like gllSsiEdr, Galileo's camera images. The file type is used to connect the proper server and directory and to check user access on the file type. File type names are short and easy for users to remember. They also allow administrators to change the placement of files, moving them to different machines and directories, even at different sites, without effecting the user community.

FEI supports a transmission mode called subscription. A client process can subscribe to a file type. As soon as a file of that file type arrives in the system, a copy is sent to the client process. The client can then process the file or store the local copy at a remote site. Subscription permits automation of image processes in a world-wide distributed science community. Once a file is processed, it can be sent back to an FEI server as another file type where it can then be read by other subscribers waiting for the processed file.

In the next release of the FEI server, notification and conference transmission modes will also be supported. Notification is like subscription, but only information about a file, not the file's contents, are sent to the subscribing process. The process can then query a database to determine if it should down-load the file. Conference, also like subscription, transmits a file among a group of subscribers but doesn't store it in FEI. Conference is used to securely transfer images among a group of users like a telephone conference call.

Authentication is handled by Kerberos. Once a user connection is authenticated it can be used for as long as necessary without further authentication checking, so Kerberos ticket expiration is less of a factor. Access control is implemented by assigning a user access roles. An access role gives the user transfer capabilities on specific file types. Since a user can have multiple access roles, possibly for different instruments on different missions, administration of complex requirements is possible. Access role information is stored in a central database, so users don't require login accounts on the server machines, thus increasing security.

All FEI data is stored in a central database, currently Sybase; but other DBMS's would work as well. Data stored in the FEI database includes server configuration information, file type and file descriptions, access roles records, system significant event records and descriptions of demon processes FEI calls for specified types of file transmissions. The FEI database can be coordinate with a project database that contains meta-data about file types managed by FEI.

Since configuration information is central, an FEI server can be brought-up on any machine having access to the database. Servers are multi-threaded so they can use SMP machine architectures making FEI scalable. A mission can increase performance by adding more CPUs and other computer resources. In the early stages of mission planning, requirements can be hazy, so the target computer requirements can be upgraded incrementally as more detailed requirements are realized. Currently FEI servers run on SUN Solaris machines. Users can access FEI servers from a number of workstation environments, including SUN OS, SUN Solaris, HP-UX and SGI-Irix. (Microsoft Windows NT support is planned for the Spring of 1997 and Apple Macintosh in the Fall of 1997.)

FEI supports auxiliary demons - processes activated when a file transfer operation is performed on a particular file type. Currently FEI servers communicate with demons using remote procedure calls. In the future we plan to add distributed object technology. Here are a few examples of how demons are used. When a file comes in, a demon parses the file's header, automatically placing keyword/value pairs in a database table. This information can be used later to locate files with particular attributes. A demon can also transfer a copy of a file through a fire-wall to a Web server directory where the file can be accessed by the general public.

FEI: A Client/Server Architecture

FEI is a client/server architecture. In order for a file to be transmitted, two pieces of software must be running and they must be able to connect with each other to send data back and forth. In this sense, FEI is like a telephone conversation; and it has some of the same implications for you the user.

First, like a telephone conversation, both parties must be present to successfully make a call. If the server is not running or if you don't have the proper information to connect to it (its number), then nothing is going to happen.

Second, you need an account to use the system, just as you must subscribe with some carrier to make telephone calls. To get an account, you must make arrangements with the FEI administrator. Explaining what this entails is one of the objectives of this document, but we'll defer the details until you know a little more about FEI.

Third, you'll need some client software to communicate with an FEI server, just as you need a communications device to use the telephone system. Notice that we didn't say you needed a telephone, you might use a modem or some other communication device for use with the telephone system. FEI offers a basic connectivity server represented by its Application Programming Interface (API) Any program (telephony device) you use depends on the API, but the look-and-feel and the use of different programs varies. A command line utility, named fei is supplied with FEI to provide basic services. But if that program doesn't fit your requirements, a different program can be written to do so. It just has to use the API to connect to the FEI system.

For you, the client/server architecture has the following implications:

• You need software to communicate with FEI and FEI servers must be present to transmit data back and forth with your programs.

• You need to set up an environment to use FEI, and you need to get authorization to use the system.

• While basic services are provided by fei, the supplied command line utility, you can also have other programs written to meet particular needs.

We should also mention that FEI server's are multi-threaded. That means that you connection runs in the same server with many other connections. This has little impact on you day to day use of the system, but it does allow FEI to provide services that other types of system find hard to perform. One implication the a multi-threaded server that can effect you directly is that of sizing. By adding more resources to a computer, in particular more CPU's, an FEI server can be made to run faster. Mission's don't always know what resources they'll need in early planning stages. This is all right, because if you need more power later on to handle more users or to transfer more data, this can be accommodated with hardware additions that don't impact the organization of the system as a whole.

What's In A Name: File Types

FEI is organized around the concept of a file type. A file type is name you use when connecting to an FEI server and you can thing of it as the name of the person you call when using a telephone.

A file type describes a set of files that have something in common like the raw or processed image files for a particular instrument on a mission. For example, the Galileo mission's raw image files might have a file type of gllSsiRaw. All files within FEI are categories by file type and each file type within the system is assigned to and is managed by one FEI server. Although several servers can make up an FEI system, each manages its own set of file types, and the file types don't overlap servers - the assignments are unique.

A file type server several functions:

• It connects you to the proper FEI server managing the file belonging to the file type.

• It locates the directory on the server machine where files are stored.

• It is used to ascertain whether or not you have the right to access files associated with this file type.

• In some cases, it is used by the FEI server to communicate with auxiliary processes to carry out specialized processing steps on a file. For example, when a file is added to the system, an auxiliary process may be called to send a copy of the file to another server that provides Web server support so the file is made available to a wider, and more general, audience.

Using file types is straight forward. As an example, we show how a file is added to the system using the fei command line utility. The command is:

	fei a gllSsiRaw/file123.image

In the example, fei takes two parameters. The first is 'a', which means "add the file". The second is the file type, gllSsiRaw, followed by a separator, the forward slash '/', followed by the name of a file, file123.image. The file type is used to connect to the server, find the directory where the file should be stored, verify that you have the right to add a file of this type, and to notify an auxiliary process to carry out any addition actions on this file. Considering what's being done, the syntax is fairly simple.

There is one additional function of a file type that is not part of the FEI system strictly speaking. And that is, it can be used to get meta-data about a file. For instance, each file of file type gllSsiRaw has a lot of additional information about it along with the image itself. This additional data is called meta-data, and it's often stored in a mission database table. Now if a mission uses file type names for database table names, getting to information is much simpler. In fact, a program can use the same name to access both FEI and the database.

So, everything you need to know about files and the information about files can depend on the file type name. They next question becomes, "How do I find out what file types are present and which ones are important to me?" There are several answers to that question. First, since missions design their own file types, there may be one or more documents available that describe file type - maybe even a Web page, if you're lucking.

But it's a fact of life that mission requirements change and documents aren't always available or up to date. So, there's a second way to get the information you require. Each FEI system has a central database where all of the information about the system is stored. In that database is a table named fileTypes If you have access to the database, you can get the information you want by executing the simple SQL query

	select name, notes
	from fileType
	order by name.
	

The name field contains the names of file types, and if you FEI administrator has been kind to you, the notes field will tell you something about the file type and its use. This method of getting file type information is particularly attractive for two reasons. First, the FEI servers use the data in this table, so its accurate - no spelling errors, nothing missing. Second, the information is always up to date. This is the information the system uses; it is the primary source.

A note about database access. Your FEI administrator must give you access to the database, and you must have some utility for accessing the database and displaying query results. Currently, FEI uses a Sybase DBMS as its database, so any tool that accesses Sybase can be used. MDMS provides such a tool called dbView. (For more information see dbView.)

There's a third way to get a list of file types, but it only provides the names and no descriptive information. On your system, the client side, there will be one or more files called FEI interfaces files. FEI recognizes a default name of feiDomain for one of these files, so it's probably present on you machine. The location of this file is referenced by the environmental variable FEI, so it's easy to find. On Unix systems, you can view file types by typing this command:

	cat $FEI/feiDomain | more
	

Microsoft system users try:

	type %FEI%\feiDomain | more
	

The fei command line utility has a handy command that also lists the file type in the default file. The command is:

	fei l
	

(That's and "el", for list.)

As we mentioned, your system may have more than one interfaces file. Ask whoever is responsible for FEI at your site or contact your FEI administrator for more information about these files.

The FEI Domain File

At the end of the last section, we introduced the FEI interfaces file, in particular, the default file, feiDomain. In that section we used it as one source of a list of file types. That's not its primary reason for being on your system.

The FEI interfaces file contains the names of file types you can use and the connection information need to connect to the FEI server that manages that file type. FEI uses the TCP/IP network protocol, the same one used by the Internet. For each file type listed in the interfaces file there is a node name (or IP address) and a TCP port number. You don't need to know that information, just the file type name. The interfaces file is used like a telephone directory by FEI. It looks up a name and dials a number, or in FEI's case, it makes a connection over the network using the TCP/IP protocol and the parameter found in the interfaces file.

The interfaces file, like a telephone directory, can change from time to time. Names get added and deleted, numbers change. And so it is with FEI. So, the question arises, "How do I get the latest copy of an interfaces file?" Your FEI administrator may deliver new ones to you as changes are made, by email for example or through some other arrangement. But, if you're the self-sufficient type, you can get it on your own from the FEI database. Remember, the FEI database is used by the servers, so it's always the most up to date and accurate source. To get the file from the database follow these steps:

1. If it exists save a copy of your current interfaces file. Add a file extension like ".old" to it. (Hardcore Unix hackers will choose a tilde "~". Ignore such people. We live in our own world.)

2. Use a database utility to access the FEI database and execute the Sybase stored procedure command:

   		genFeiDomainFile
   		

   The interfaces file, properly formatted, is returned. You'll have to save the results to a file somehow. dbView does this nicely, so you might think of using it. (We'll give you an example once we've finished with these general instructions.)

3. Finally, once you've got the new file, give it the name of your old domain file, probably feiDomain. Be sure to place it in the same directory as the old file, and, if this is the default file, make sure that the environmental variable FEI references the directory. That's it.

Here's the dbView session we mentioned. The commands you type are in bold. The interfaces file with the displayed contents was written to to the data file newInterfacesFile Just rename it and you're ready to go. (The lines in the interfaces file beginning with a '#' character are comments. The line (return status = 0) isn't part of the file; it's dbView's response line for the query.)

	% dbView

	dbView, version 1.9, (dblib), 20 Jan 1997
	Copyright 1993, The Jet Propulsion Laboratory.  All rights reserved.

	User name [jake]: 
	password: 
	server [MY_MISSION]: 
	database [fei]:
	DBMS Type [Sybase]: 

	1> open dataFile newInterfacesFile

	1> genFeiDomainFile
	2> go

	# feiDomain file.
	# Generated on Feb  8 1997 11:19AM by dbo
 
	conference1
		milano  fei1  12005  JPL-MDMS.JPL.NASA.GOV
	image
		milano  fei1  12005  JPL-MDMS.JPL.NASA.GOV
	text
		milano  fei1  12005  JPL-MDMS.JPL.NASA.GOV

	(return status = 0)

	1> close dataFile

	1> exit
	

File Type Naming Conventions

A file type name can be up to 30 characters in length. The first character in a name must be a lower or upper case letter. Following the first character, names can contain letters, numbers and the underscore character '_".

FEI imposes no other naming conventions. However, here are some rules you might consider adopting:

• A name should start with the mission's acronym, like 'gll...' for the Galileo Mission.

• An instrument acronym should follow the name, like 'gllSsi..." for the SSI instrument on the Galileo spacecraft.

• An instrument acronym should be following with the type of file, like 'gllSsiRaw' or 'gllSsiEdr'.

• Further distinguishing characteristics should follow the name as described so far, like 'gllSsiRaw' and 'gllSsiRawTest', where test is a set of files used just for testing.

• Each section of a name but the first should begin with a capital letter, like 'gllSsiRaw', 'gllSsiEdr', etc.

FEI file types don't impose a hierarchy of names, but you can see the naming conventions just described imply a hierarchy for easy reference. Here's a short example, for three missions and four instruments:

   Galileo
      NIMS
         gllNimsEdr
         gllNimsRaw
      SSI
         gllSsiEdr
         gllSsiRaw
   Mars Pathfinder
      IMP
         mpfImpEdr
         mpfImpRaw
   Mars 98
      RAC
         m98RacEdr
         m98RacRaw
	

Security: Authentication And Access Control

FEI protects the files it manages in two ways. First it requires authentication of a user when the user connects to a server. Second, transmission modes, like add, get, replace, delete, etc. are protected by access control lists that only allow authorized users to perform a specified type of transmission on a particular file type.

Authentication

Access Roles

Basic Commands

Now with all the preliminaries out of the way, we discuss transmitting files. In this section we cover the basic commands. In following sections we'll describe subscription and batch processing.

Here are FEI's basic transmission modes. Remember, each transmission mode takes place in a particular file type context. That is, we always transmit or effect a file of a particular type.

• Add a file

• Get a file

• Replace a file. If the file doesn't exits, it's added.

• Delete a file.

• Rename a file. The file can't move from one file type to another. It must remain in its file type context. To move a file to a new file type, you have to get it from the old type, add it to the new file type, and delete the file from the old file type.

The FEI application program interface supports calls for each of these modes. How a particular program represents these to you is up to the programming team and to you. But, to show some concrete examples, we'll show the the command line utility fei carries out simple transmissions of this sort. (More complex examples are covered in FEI Command Line Utilities.)

These simple examples use a file file type of image and file names of image1, newImage1 and myImage. Each type of transmission is performed by a separate command. The type of command is the first parameter to fei and is represented by a single letter.

	kinit jake

	Add image1
	fei a image/image1
	
	Get image1 and store it locally as newImage1
	fei g image/image1 newImage1
	
	Replace image1 with the local file newImage1
	fei r image/image1 newImage1
	
	Rename newImage1 as myImage
	fei n image/newImage1 myImage
	 
	Delete myImage
	fei d image/myImage
	

We placed the Kerberos kinit command in bold to remind you that you've got to get a Kerberos ticket for FEI authentication before you can do anything. Also, you'd need the capability for each of the operations on the file type image before you could execute the commands successfully.

At this point you may have a lot of questions. "Hey! Can I do this? What about that?" We're just introducing concepts here. How you solve particular tasks is covered in other documents. (See "Where To Go Next" at the end of this document for suggestions.) For now we'll concentrate on the basic ideas.

Auxiliary Functions: Compression, Encryption, CRC's

When transmitting a file, you can have FEI apply several auxiliary functions. These functions only apply during transmission. The file at the end of transmission is deposited in the same format as the original. The functions are:

• Compress the file using a lossless LZW compression scheme. Sometimes, especially for sites far removed from an FEI server site or when using congested networks, compression can speed file transmission. However, compression and decompression take time and it's being done dynamically at the time of transmission, so its hard to tell when compression can help. How do you tell? You experiment. Normally, files of the same type compress about the same. So try one or more. First compressed and then not. Did compression help? If so, use it, but check once in a while. Over time conditions may change and therefore the need for compression may change. (Image files with a lot of dark sky may compress well. So if you find you have a lot of them, you might try it.)

• Encryption encrypts the file while its transmitted. If you don't want people to see the contents of a file, encrypt it. (As a rule, we don't give out information on our encryption scheme. But the particulars can be discussed with MDMS developers if necessary.)

• CRC's are cyclic redundancy checks applied to transmitted files. If you add or replace a file with function, the CFC value is stored in the FEI database along with the file's record. It's all located in the files table. Use this when you want to make sure that nothing was lost during transmission and that no agent tampered with you file. Once transmission is complete, you are given a copy of the CRC value if the computations at both ends were the same and the transmission completed successfully. You can keep this value and compare it with the one in the database later to verify that the file is still as you left it.

File Profiles

Each time a file is transmitted, a file profile record is generated along with the file. A file profile record is represented by a structure in the FEI API. Different programs provide the information in the structure in different manners. Here's what the fei command line utility returns:

1. type The file type.
2. name The file name.
3. date The date and time the file was received into the FEI system. The date-time format is taken from the SQL database language. An example is: "1 Jan 1997 12:00 PM"
4. size The file size in bytes
5. cfc The cyclic redundancy check value. Only valid if the CFC option was used when transmitting the file.

For more detailed information on the file profile record, see the FEI API Concepts Guide.

Subscription

Subscription is a way to get a copy of a file automatically automatically when it entered in the FEI system. There are two subscriptions modes:

Note: The term subscriptionis used to described all of the modes and it's also the name of one of the modes. The use of the term should be clear from the context, but when it's not, we shall use the term subscription mode to make it clear that we are discussing the mode itself and not using the term in its general sense.

1. Subscription Mode

   Returns a description of the file and the file itself.

   This is the most commonly used mode of subscription. Following our example, each time a file of gllSsiEdr entered the system, a copy of it is sent to all subscribers along with a description of the file called a file profile

   The file is written to whatever directory you designate and the file's name is the same as that used within the FEI system. The file's profile is delivered as a separate data item and is represented differently in different programs. The fei command line utility returns the profile to the STDOUT file stream, so each time a file is delivered you'll see its profile appear. (We'll describe the contents of a file profile below.)

   Here's an example of subscribing to the file type "gllSsiEdr" with the fei command line utility:

   	fei s gllSsiEdr
   	

2. Notify Mode

   Notify returns just the description of the file.

   It's like subscription mode, but no file is transferred, just the file's profile. This mode is used when you only want some of the files associated with a file type. Usually, you use the file type and the file's name - returned in the file's profile - to query a database table to determine if this file is of interest to you. For example, let's say a satellite is in an Earth polar orbit and is recording images of ice flows. If you're only interested in ice flows within a particular latitude and longitude, you can subscribe with notification. Each time a file arrives, you're notified of that fact. Using the file type and file name, you can query a database. (Assume that things are well structured, and the name of the database table is the same name as the file type.) The query would look something like this:

   	select minLat, minLon, maxLat, maxLon
   	from <file type name>
   	where name = "<file name>"
   	

   Once this information is returned, your program can determine if this file is important to you, and if it is, the program can get a copy using FEI basic transmission mode. After that, your program can do whatever is appropriate.

   Notify can save the transmission of a lot of unwanted data.

   Here's an example of subscribing to the file type "gllSsiEdr" with the fei command line utility:

   	fei n gllSsiEdr
   	

   Note: Notify mode is available starting with FEI server version 2.

Conference Calls (Subscription With A Twist)

A Conference Call is a special uses of subscription mode. It's best understood with an example. Suppose you are looking at an image, and you've made some enhancements to it the revealed something interesting. You'd like to show this image to a number of fellow scientist and discuss it. Do transmit the image you use a special file type called a conference file type that the FEI administrator has made available to you and a group of other investigators. Now, you contact these others - possibly with a telephone conference call - and tell them to subscribe to your conference file type; we'll call the type "conference1". You then add the image showing your newly found discovery using mpfConference and it's immediately distributed to all of your subscribers do they can see it and all of you can discuss it. During the discussion, others may make relevant observations, perhaps by changing your image by by transmitting a different image of their own. All of the subscribing investigators see the new image.

Now lets describe some of the characteristics of a conference call.

• A conference call used a special file type called a conference file type The record for the file type in the FEI database table fileTypes has the conferenceFlag value set to 1. For example:

  	1> select name, conferenceFlag
  	2> from fileTypes
  	3> go
  
  	name                    conferenceFlag  
  	----------------------  --------------
  	text                                 0  
  	image                                0  
  	conference1                          1  
  	

• The FEI administrator sets up conference call file types and grants you and other relevant investigators access to it, so it has restricted access. That means that not just anyone can join a conference call. Note: You could encrypt any files sent using this file type to further ensure privacy.

• Conference file types don't save a copy of files sent in the FEI system. The files just pass through the FEI server managing the file type. If you later wanted to save a copy in FEI, you'd use a different file type.

The FEI Database

We've discussed the FEI database several times in this document. In this section we'll summarize what's already been said and add a few more details.

An FEI system keeps all persistent information in a database. Currently a Sybase database management system is used, but other's could be supported without undue effort. The database is what really defines an FEI system. The system may have several servers and servers may run on different machines; but, if they are part of the same FEI system, they all use the same database.

The types of information stored in the database include:

• Server configuration
• File types
• Files
• Access rolls
• Auxiliary processes associated with file types and transmission modes
• FEI significant events

From the user perspective, only two tables are relevant, fileTypes and files. (For a more complete description of the FEI database, see the section The FEI Database in the in The FEI Administration Guide.)

We already pointed out that the fileTypes table can be used to find out about the file types defined in the system. We also mentioned that the information in that table can be used to generate an FEI interfaces file with the stored procedure genFeiDomainFile.

The other relevant table is files, but if you want to view information in that table, you'd be better served to use the database view fileView. (The files table doesn't have a direct reference to the file type of a file while fileView does.)

fileView has the following attributes:

• fileType The file's file type found in the fileTypes table.
• file The name of the file.
• contributor The name of the FEI user who added, or contributed, the file.
• receivedAt The date and time the file entered the FEI system.
• status The status of the file. 0 means the file is fine; any other value means there's something wrong with the file, and the FEI administrator should investigate.
• crcValue The cyclic redundancy check value if the file was added with the cfc option.
• errMsg An error message if there's something wrong with the file.

Getting a list of newly arrived files is one use of fileView. For example, you could run the following query on January 2nd to get all the files that arrived on January 1st.:

	select name, receivedAt, size
	from files
	where receivedAt > "1 Jan 1997 12:00AM"
	and fileType = "image"
	order by receivedAt
	

Then, save the last receivedAt value returned; and on January 3rd, run a similar query using the value you save. Each time the query is run, you retrieve a new list of files for the specified file type.

Use the same approach to get other information about a set of files. Change the WHERE clause to qualify different areas of interest.

Actually, there's a faster and better way to execute the last query. Use the stored procedure show files; then you only need type the following:

	showfiles image, "1 January 1997 12:00AM"
	

The parameter "image" is the file type and information is returned for any file received after "1 January 1997 12:00AM". Even if you don't feel comfortable with SQL, this is pretty easy.

Running Batch Jobs