TWiki . Lqcd . FermilabILDGUsersManual

International Lattice Data Grid at Fermilab Users Manual

Author: Eric H. Neilsen, Jr.

Contact: neilsen@fnal.gov

$Date: 2004/04/13 17:36:14 $

$Name: $

$Revision: 1.6 $

$State: Exp $

Organization: Fermi National Accelerator Laboratory

Introduction

About this document

This document will provide instructions and reference material for users of International Lattice Data Grid (ildg) software at Fermilab. It will include a general introduction to the ildg, recipees for typical uses, and reference material describing user level operations. It does not include information intended for developers or administrators.

The template presented in IEEE Std 1063-2001, the IEEE Standard for Software User Documentation, inspired the structure and content of this document, but is incomplete and under active development; some elements in the recommendation are omitted or appear in an abbreviated form.

The International Lattice Data Grid

Physics computations executed by the lattice QCD community produce large volumes of data. Because the data produced by completed computations form the basis for future work, the maintenance of archives of existing data and metadata describing the provenance and generation parameters for that data is essential not only as a reference, but also as a basis for future work. The Fermilab ildg software provides a database for metadata storage, a database that records what data is available in each archive and how it may be obtained, and an interface mass storage systems at Fermilab and elsewhere. The user interface provides a convenient mechanism for coordinating changes to these databases and archives.

Several collaborations generate data which should be archived, and members of each collaboration have access to a different set of computational and archival facilities. A standardized interface to the archives and metadata databases will simplify the sharing of data between these collaborations. Standards for these interfaces are still under development; the interfaces implemented in the current version of this software represents speculation based on current discussions, and the Fermilab implementation should not yet be expected to be interoperate with other implementations.

Currently, ildg software is installed only on the head node of the Fermilab lattice QCD cluster, lqcd.fnal.gov. It requires the Globus toolkit to be installed, a GSI ftp client, and Python version 2.3 or later. While transfer of data requires users to have a grid certificate accepted both on the host node and the SRM mass storage system being used, such a certificate is not required for metadata operations provided the metadata database server is running on the same node as the ildg client.

The tools described here are in an early, experimental stage; they should not yet be considered reliable enough for production use.

Use of this document

New users should begin by reading the introduction and description of typical use subsectons in the concept of operations section and the first few procedure descriptions that follow to get a general idea of how things work. Before use, the entire concept of operations section should be read, and any procedures of interest.

Concept of operations

Introduction

The ildg provides a set of services supporting the description, transportation, and archiving of lattice QCD data. The primary services are:

Client software provides the user (either directly or scripting language API's) with convenient ways of using these databases. To simplify use, operations often performed in tandem (adding data to the mass storage system and corresponding metadata to the metadata database, for example) are combined into a single operation from the users point of view.

Description of a typical use

A user begins with a lattice QCD data set, say a set of files containing configurations that are members of a common ensemble. Metadata files describe the ensemble and each configuration. When the data is ready to be made public, the user issues a single ildg command that copies the data files to a convenient mass storage system, adds metadata describing these files to a metadata database, and updates a database associating the entries in the metadata database with the locations of these files in the mass storage system.

At some later point in time, a second user uses ildg tools to search the metadata database, and discovers these data. Not being ready to use the data immediately, the user issues an ildg command that copies the data from the mass storage system in which it was originally stored into a more convenient, local one. This command automatically updates the replica database with the second location.

When the second user is ready to use these files, an single ildg command is then used to copy the files from mass storage to the desired location on disk. This tool uses the replica database to determine in which mass storage systems the data resides, and copies the data from the most convenient one.

Global file names

A global file name (GFN) is a string used as a persistant, location independant identifier of a set of data. Once assigned, a GFN always identifies the same set of data, even if the data changes location or is deleted everywhere. A single GFN may identify data stored in files on many systems, in fields of a database, or even in memory, provided every instance is an exact copy of the data.

The ildg uses GFN's to identify sets of data in mass storage. For example, the each stored lattice configuration has its own GFN. While descriptive strings may be (and are, by default) used as GFNs, users are discouraged from using data contained in these strings; metadata in the metadata catalog should be used instead. Metadata encoded in the GFN may be incorrect, and not all GFN's are likely to have metadata encoded in the same way. If an error is made in the initial assignment of a GFN (because of a typo, for example), it cannot be corrected; parameters derived from this name will be permanantly incorrect. A better approach is to store all metadata in the metadata database, or in separate files that refer to the described data.

Metadata parameters and qcdml

In addition to the software described here, the ildg community is developing an XML format for the description of lattice QCD metadata. The ildg software uses this format as the primary means of specifying and transferring metadata; when metadata is added to or retreived from the metadata database, the data is transfered in qcdml.

The Fermilab implementation of the ildg metadata database stores the qcdml document in a relational database. Unfortunately, searching data in xml format with such tools as XPath and XQuery can be very slow, and the generation of queries can be cumbersome. Therefore, in addition to storing the qcdml itself, the metadata catalog server extracts a subset of the most useful information from the qcdml and stores it in tables in the database. These parameters can then be used to express conditions as SQL logical expressions. Seaches using these expressions will be considerably faster than their XQuery counterparts.

The following are examples of such queries:

The qcdml describing the ensemble and configuration are included in the available fields, and in future a function that extracts data from these fields using XPath expressions will be availble. (Even now, ambitious users may use string functions to extract data from the stored xml. This is not, however, particularly practical.) Data searched in this way will, of course, suffer in performance.

Data fields that may be referred to in these expressions, together with the QCDML path from which they are determined, can be found in the "Available metadata fields" appendix of this document.

Dynamic generation of file names

Several ildg uses require that the user specify a format for the dynamic generation of file names. These commands require two parameters: a format string, and a list of metadata parameters. The metadata parameters an be any of those used that can be used in queries (see the "Available metadata fields" appendix for a complete list). The format string specifies the template into which the values for these parameters are to be substituted, in a format resembling the C printf rules. For example, if the following format string is specified:

with the paramter list

The following files names may be generated:

and so forth for all file names that need generation.

Procedures

General information

determining the uri of a file in mass storage from its path in pnfs

Several operations require the user to supply the URI of a file in mass storage. The URI for files in the Fermilab mass storage system may be determined from the path and name of the file in the pnfs mount on lqcd.fnal.gov by replacing the /pnfs in the full path name on lqcd.fnal.gov with srm://fndca.fnal.gov:8443//pnfs/fnal.gov/usr. In other words, one can convert a file named /pnfs/lqcd/test/myFile.dat as seen on lqcd to an appropriate URI using:

bash$ echo /pnfs/lqcd/test/myFile.dat | sed 's%/pnfs%srm://fndca.fnal.gov:8443//pnfs/fnal.gov/usr%'

preparing the shell environment on lqcd.fnal.gov

The ildg software is currently installed on lqcd.fnal.gov as a UPS product; the UPS setup utility sets environmental variables (including the path) required to run the ildg client.

First, environmental variables for UPS itself must be set. This is accomplished by sourcing /usr/local/etc/setups.sh (or /usr/local/etc/setups.csh, if you use csh or tcsh):

bash$ . /usr/local/etc/setups.sh

No output is expected from this script.

Once this is done, setup ildg itself:

bash$ setup ildg

This command will not produce any output if successful.

security

Remote access to the database and archives used by ildg is controlled using globus Grid Security Infrastructure (GSI). At present, a GSI certificate is not required for access to the databases from lqcd, but such a certificate is required for access to the mass storage system.

To use GSI certificate authenticated services at Fermilab, you must obtain a GSI personal certificate from a certificate authority trusted by Fermilab, and that certificate must be added to the list of authorized users for each service. Fermilab users require an appropriate certificate should request a DOEGrids certificate.

Once a DOEGrids certificate is obtained and installed as described on the DOEGrids web page (see the link above), your identity must be added to the list of authorized users on the service where GSI authentication is required. If you are running on lqcd.fnal.gov, the only such service is the Fermilab mass storage system.

You must determine your GSI identity so you can provided it when requesting authorization. On lqcd.fnal.gov, this can be done by setting up globus (see the UPS instructions in the "preparing the shell environment on lqcd.fnal.gov" section above) and initializing your certificate:

bash$ . /usr/local/etc/setups.sh ;# if this was done previously in
bash$                             # this shell, you need not repeat it
bash$ setup globus               ;# this is unnecessary if you have
bash$                             # already setup idlg in this shell
bash$ grid-proxy-init

Provide your grid passphrase when prompted for it. The output of grid-proxy-init will look something like:

bash$ grid-proxy-init
Your identity: /DC=org/DC=doegrids/OU=People/CN=Eric H. Neilsen, Jr. 886417
Enter GRID pass phrase for this identity:
Creating proxy ............................................... Done
Your proxy is valid until: Thu Apr  8 22:53:23 2004

Your grid identity is listed in the first line of output from grid-proxy-init.

The grid-proxy-init command generates a proxy which allows you to use the grid services, and must be repeated whenever the proxy expires. The expiration date of the proxy is listed in the output of the grid-proxy-init command, and may also be obtained using grid-proxy-info.

routine preparation for ildg use

Under typical use (after certificates have been obtained and you have been authorized to use the necessary services), the initial preparation for use will look something like this:

bash$ . /usr/local/etc/setups.sh ;# this can be put in your .bashrc
bash$ setup ildg ;# this can also be put in your .bashrc
bash$ grid-proxy-init
Your identity: /DC=org/DC=doegrids/OU=People/CN=Eric H. Neilsen, Jr. 886417
Enter GRID pass phrase for this identity:
Creating proxy ............................................... Done
Your proxy is valid until: Thu Apr  8 22:53:23 2004

Creating a qcdml metadata file

The Fermilab ildg software currently supports a slightly modified version of qcdml v0.1 only. The first task in adding data to the database is to generate such a metadata file for each data file you will be exporting.

The Fermilab ildg software includes an interactive tool for generating this qcdml file using the .info file present for much of the currently archived data and interactive user input. Because the qcdml contains significatnly more information than the .info files typically do, and because the format of these file varies significantly, only interactive creation of qcdml is supported. qcdml will be generated only for the milc a squared tadpole action.

To generate qcdml metadata for a set of files in a single ensemble, change the current working directory to a directory containing the .info files for the ensemble. This may require using dccp, encp, or srmcp to copy these files out the the mass storage system, because changing the current working directory to a pnfs directory with the files is not sufficient. For example:

bash$ cd ~/myData/myEnsemble
bash$ ls *.info
configuration1.info  configuration2.info  configuration3.info

Then begin the tool, providing the .info files as arguments:

bash$ info2qcdml *.info

All files included as arguments must belong to the same ensemble. info2qcdml then displays a form showing the various parameters stored by qcdml. These fields are often filled in with defaults, or are generated from the provided .info files.

Entries which take lists of values (dimension names, dimension lengths, quark flavors, quark masses, quark mass determination, quark boundary condition, and gluon boundary condition) accept space separated lists.

Enter times and dates in the ISO-8601 format (YYYY-MM-DDTHH:MM:SSZ) in the UT (formerly GMT).

The final two entries, "step reg ex" and "series reg ex" determine how the step and series for each configuration is derived from the file name. Enter regular expressions matching the file names, where the first subexpression of the regular expression is the designated quantity (step or series).

When all values are filled in to your satisfaction, hit the "write" button at the bottom of the form. This will cause qcdml files to be generates, one for each .info file provided on the command line. The first qcdml file written will contain data both for that configuration and the ensemble as a whole, while the others containly only data pertaining to that specific configuration and a reference to the ensemble defined in the first one. The names of the generated files will correspond to the names of the supplied .info files, with the .info replaced by .xml.

Registering archived data with the ildg

Although users are eventually expected to use ildg tools to simultaneouly export data, metadata, and replica information with a single command, at present there already exists a substantial volume of data already in mass storage. The "register" command allows a user to update the metadata and replica databases with information about data already present in a mass storage system without needlessly creating an additional copy of the data itself.

To register an existing copy of a data file in a mass storage system, the user must have a qcdml metadata file describing that file, and know the URI of the data file. The section entitled "determining the uri of a file in mass storage from its path in pnfs" provides further information about these URI's.

One can add the metadata and replica information for this file using the following command:

bash$ idlg register srm://fndca.fnal.gov:8443//pnfs/fnal.gov/usr/lqcd/test/myFile.dat myFile.xml

where myFile.xml holds the qcdml metadata.

It is also possible to register many file at once, provided that the URI can be determined using a substitution from the base file name of the qcdml file. The rules for substitution resemble the formatting rules of a C printf statement, where the base file name of the qcdml file is the only substitution variable. For example, if you have two qcdml files,

and their URI's are

one can add them both using

bash$ ildg register srm://fndca.fnal.gov:8443//pnfs/fnal.gov/usr/lqcd/test/%s.dat myFirstFile.xml mySecondFile.xml

In combination with wildcards, this can be particularly useful for adding directories of files containing entire ensembles.

Exporting data to the ildg

When adding new data to the ildg, a user exports data to the mass storage system, adds metadata to the metadata database, and records the location of the data in the replica catalog in a single operation. In addition to the file containing the data itself and the qcdml formatted metadata file describing it, the user must have the destination URI of the location in mass storage into which the data is to be copied. The section entitled "determining the uri of a file in mass storage from its path in pnfs" provides further information about these URI's.

Use the ildg add command to add data to the ildg. This command takes three arguments:

Note that the full absolute paths for the files must be given. (A file in the current working directory may always be added using eg `pwd`/thisFile.)

For example, to add the file /tmp/myData.dat with metadata in /tmp/myData.xml to mass storage at srm://fndca.fnal.gov:8443//pnfs/fnal.gov/usr/lqcd/test/myFile.dat, one might use:

bash$ ildg add /tmp/myData.dat /tmp/myData.xml srm://fndca.fnal.gov:8443//pnfs/fnal.gov/usr/lqcd/test/myData.dat

the output from this command will look similar to this:

lqcd.fnal.gov:configuration:ILDG:test:myData:a:1003 added to metadata database
lqcd.fnal.gov:configuration:ILDG:test:myData:a:1003 inserted into replica catalog database at srm://fndca.fnal.gov:8443//pnfs/fnal.gov/usr/lqcd/test/myData.dat
Connecting to https://fndca.fnal.gov:8443/srm/managerv1
calling connect h - fndca.fnal.gov  p - 8443
inside client_auth_callback. Remote user is /DC=org/DC=doegrids/OU=Services/CN=host/stkendca3a.fnal.gov
SRM copy return state is Pending
calling connect h - fndca.fnal.gov  p - 8443
inside client_auth_callback. Remote user is /DC=org/DC=doegrids/OU=Services/CN=host/stkendca3a.fnal.gov
calling connect h - fndca.fnal.gov  p - 8443
inside client_auth_callback. Remote user is /DC=org/DC=doegrids/OU=Services/CN=host/stkendca3a.fnal.gov
calling connect h - fndca.fnal.gov  p - 8443
inside client_auth_callback. Remote user is /DC=org/DC=doegrids/OU=Services/CN=host/stkendca3a.fnal.gov
calling connect h - fndca.fnal.gov  p - 8443
inside client_auth_callback. Remote user is /DC=org/DC=doegrids/OU=Services/CN=host/stkendca3a.fnal.gov
calling connect h - fndca.fnal.gov  p - 8443
inside client_auth_callback. Remote user is /DC=org/DC=doegrids/OU=Services/CN=host/stkendca3a.fnal.gov
calling connect h - fndca.fnal.gov  p - 8443
inside client_auth_callback. Remote user is /DC=org/DC=doegrids/OU=Services/CN=host/stkendca3a.fnal.gov
calling connect h - fndca.fnal.gov  p - 8443
inside client_auth_callback. Remote user is /DC=org/DC=doegrids/OU=Services/CN=host/stkendca3a.fnal.gov
calling connect h - fndca.fnal.gov  p - 8443
inside client_auth_callback. Remote user is /DC=org/DC=doegrids/OU=Services/CN=host/stkendca3a.fnal.gov
gsiftp://neilsen@lqcd.fnal.gov:2812///home/neilsen/testData/milc/myData.1003 copied to srm://fndca.fnal.gov:8443//pnfs/fnal.gov/usr/lqcd/test/myData.dat

The lines beginning "calling connect h" and "inside client_auth_callback" may be repeated many times, with increasing time delays between successive appearances; these indicate that the client is polling the mass storage system for the status of the copy.

If the destination of the data was the Fermilab mass storage system (as in the above example), its presence may be verified using the pnfs mount. For example, one could find the file added in the example above using:

bash$ ls /pnfs/lqcd/test

on lqcd.fnal.gov.

Obtaining data from a mass storage system

Copies of the data itself may also be obtained from a mass storages system using ildg tools. The ildg get command is used for this purpose. It has two required arguments and one optional one:

See the "Metadata parameters and qcdml" section of this document for a description of how to specify the conditions in the first argument, and the "Dynamic generation of file name" section for more information on the format of the second and third parameters.

For example, to obtain copies of all data in ensembe "myEnsemble" in series A or B with updates less than 10, one could use:

bash$ ildg get "(ensemble = 'myEnsemble' and ((series = 'A') or (series = 'B')) and (update < 10)" \
> '/data/%s-series%s-step%s.dat' 'ensemble series update'

which would produce the following files (provided these are the data that exist):

Obtaining metadata

Copies of the qcdml describing stored data sets may be obtained from the metadata database using ildg tools. The ildg query command is used for this purpose. It has two required arguments and one optional one:

See the "Metadata parameters and qcdml" section of this document for a description of how to specify the conditions in the first argument, and the "Dynamic generation of file name" section for more information on the format of the second and third parameters.

For example, to obtain copies of all qcdml metadata in ensembe "myEnsemble" in series A or B with updates less than 10, one could use:

bash$ ildg query "(ensemble = 'myEnsemble' and ((series = 'A') or (series = 'B')) and (update < 10)" \
> '/data/%s-series%s-step%s.xml' 'ensemble series update'

The output would resemble the following:

Connecting to https://fndca.fnal.gov:8443/srm/managerv1
calling connect h - fndca.fnal.gov  p - 8443
inside client_auth_callback. Remote user is /DC=org/DC=doegrids/OU=Services/CN=host/stkendca3a.fnal.gov
SRM copy return state is Pending
calling connect h - fndca.fnal.gov  p - 8443
inside client_auth_callback. Remote user is /DC=org/DC=doegrids/OU=Services/CN=host/stkendca3a.fnal.gov
calling connect h - fndca.fnal.gov  p - 8443
inside client_auth_callback. Remote user is /DC=org/DC=doegrids/OU=Services/CN=host/stkendca3a.fnal.gov
calling connect h - fndca.fnal.gov  p - 8443
inside client_auth_callback. Remote user is /DC=org/DC=doegrids/OU=Services/CN=host/stkendca3a.fnal.gov
srm://fndca.fnal.gov:8443//pnfs/fnal.gov/usr/theory/milc/someFile1 copied to gsiftp://neilsen@lqcd.fnal.gov:2812///data/myEnsemble-seriesA-step1.xml=
Connecting to https://fndca.fnal.gov:8443/srm/managerv1
calling connect h - fndca.fnal.gov  p - 8443
inside client_auth_callback. Remote user is /DC=org/DC=doegrids/OU=Services/CN=host/stkendca3a.fnal.gov
SRM copy return state is Pending
calling connect h - fndca.fnal.gov  p - 8443
inside client_auth_callback. Remote user is /DC=org/DC=doegrids/OU=Services/CN=host/stkendca3a.fnal.gov
calling connect h - fndca.fnal.gov  p - 8443
inside client_auth_callback. Remote user is /DC=org/DC=doegrids/OU=Services/CN=host/stkendca3a.fnal.gov
calling connect h - fndca.fnal.gov  p - 8443
and so forth.

The lines beginning "calling connect h" and "inside client_auth_callback" may be repeated many times, with increasing time delays between successive appearances; these indicate that the client is polling the mass storage system for the status of the copy. If the file is large or the mass storage system is busy, there may be a great many of these before the command completes.

This command would produce the following files (provided these are the data that exist):

Exploring available data using a web interface

The metadata stored in the metadata database on lqcd.fnal.gov may be explored using the web interface at http://lqcd.fnal.gov/cgi-bin/explore

Instructions for using the page may be found on the page itself.

Finding data that matches a specific set of criteria

The ildg discover command allows the user to list the GFN's of data that matches a set of conditions. Its only argument is the string specifying the conditions.

For example, to list the GFN's of data in ensemble myEnsemble would look like this:

=bash$ ildg discover "ensemble = 'myEnsemble'"

The output will be a list of the GFNs of all matching data.

Knowledge of the GFN's will not generally be of interest to users; most commands which require designation of data or sets of data take restrictions on metadata as their parameters.

Changing metadata stored in the metadata database

Stored metadata may be modified using the ildg alter command. This command takes two arguments:

Note that, if the qcdml file contains ensemble data, the ensemble data for all configurations in that ensemble will be updated.

Currently, the alter command uses the GFN to designate the name of the data whose metadata is to be modified; in future versions, metadata restrictions will be used instead.

To obtain GFNs using metadata restrictions, use the ildg discover command. A full typical use in a use wishes to modify the metadata for update 10 of ensemble myEnsemble might look like this:

bash$ ildg discover "ensemble = 'myEnsemble' and updates = 10"
myEnsemble:a:010
bash$ ildg alter "myEnsemble:a:010" myData-10.qcdml 

You can even get fancy with shell tools, and avoid using GFNs altogether:

bash$ ildg discover "ensemble = 'myEnsemble' and updates = 10" | \
> xargs --replace ildg alter {} myData-10.qcdml 

Upon success, ildg alter will emit a message that looks like this:

myEnsemble:a:010 added to metadata database

Note that the ensemble, collaboration, and project metadata parameters cannot be changed.

Undoing changes made to the metadata database

Changes made to the metadata may be undone using the ildg revert command. This command takes two arguments:

Withdrawing data

While data may not be removed from the ildg, it may be declared withdrawn. Withdrawn data will not be returned by queries to the database. (In future versions, options may be added to allow the user to obtain withdrawn data.) The ildg withdraw command takes one argument, an expression that specifies which data is to be withdrawn.

Moving data files from one mass storage system to another

The ildg replicate command may be used to copy data from one mass storage system to another, or to a second location within the same mass storage system.

Command reference

Error message and problem resolution

Appendixes

Available metadata fields

metadata field name qcdml path
ensemble markovStep/markovChainLFN
lfn markovStep/data
config_xml  
ensemble_xml  
series markovStep/series
updates markovStep/update
crc management/crcCheckSum
revision management/revision
author management/generateData/participant/name
generation_date management/generateData/date
machine machine/name
machine_type machine/type
code code/name
code_version code/version
code_date code/date
timestamp management/submitData/date
project management/project
reference algorithm/reference
collaboration management/collaboration
institution machine/institution
algorithm markovStep/algorithm
algorithm_reference markovStep/algorithm
algorithm_exact markovStep/algorithm
x markovStep/markovChain/physics/size
y markovStep/markovChain/physics/size
z markovStep/markovChain/physics/size
time markovStep/markovChain/physics/size
gauge_fixing markovStep/gaugeFixing
ensemble_precision markovStep/precision
asqtad markovStep/markovChain/action/milcASquardTadpoleAction
gluon_x_boundary markovStep/markovChain/action/*/field/gluonField/boundaryCondition
gluon_y_boundary markovStep/markovChain/action/*/field/gluonField/boundaryCondition
gluon_z_boundary markovStep/markovChain/action/*/field/gluonField/boundaryCondition
gluon_t_boundary markovStep/markovChain/action/*/field/gluonField/boundaryCondition
quark_x_boundary markovStep/markovChain/action/*/field/quarkField/boundaryCondition
quark_y_boundary markovStep/markovChain/action/*/field/quarkField/boundaryCondition
quark_z_boundary markovStep/markovChain/action/*/field/quarkField/boundaryCondition
quark_t_boundary markovStep/markovChain/action/*/field/quarkField/boundaryCondition
degenerate_mass markovStep/markovChain/action/milcASquardTadpoleAction/seaQuarks
nondegenerate_mass markovStep/markovChain/action/milcASquardTadpoleAction/seaQuarks
mass_determination markovStep/markovChain/action/milcASquardTadpoleAction/seaQuarks
asqtad_gauge_description markovStep/markovChain/action/milcASquardTadpoleAction/gaugeDescription
asqtad_n_loops markovStep/markovChain/action/milcASquardTadpoleAction/nLoops
asqtad_n_reps markovStep/markovChain/action/milcASquardTadpoleAction/nReps
asqtad_beta markovStep/markovChain/action/milcASquardTadpoleAction/beta
asqtad_u0 markovStep/markovChain/action/milcASquardTadpoleAction/u0
asqtad_u0_determination markovStep/markovChain/action/milcASquardTadpoleAction/u0Determination
asqtad_quark_description markovStep/markovChain/action/milcASquardTadpoleAction/quarkDescription

EricNeilsen - 13 Apr 2004

----- Revision r1.6 - 13 Apr 2004 - 17:36 GMT - TWikiGuest
Copyright © 1999-2003 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback.