This document describes how to create an ACFILE with TRANSP and how to write code to read an ACFILE, i.e. an Ascii Compressed File which has been output by TRANSP. ACFILEs are a new mechanism for output of arbitrary information from TRANSP COMMON. It is expected that ACFILEs will be used to output fast ion distribution function data from TRANSP, which has long been requested, but it is a general mechanism that may be used for other purposes as well. For example: ALL of TRANSP COMMON may be written to an ACFILE, which could serve as input to a TRANSP post-processor which uses TRANSP facilities (e.g. straight line tracker) in its computations. This document is a first draft and may not be "complete". I expect to work with pioneer users of TRANSP ACFILEs. Please feel free to contact me ... D. McCune, PPPL, USA (609) 243-2731. Written by D. McCune 4 Dec 1991. -------------------------------- Partial Review and Edit: DMC 17 Feb 1994 Update DMC 2007 -- see end of document I. Introduction -- what ACFILES are and how they are created. ACFILEs are compressed sequential ascii files containing arbitrary data from TRANSP COMMON at a single selected output time. One TRANSP run may produce up to 5 ACFILEs. ACFILEs can contain all of TRANSP COMMON, and so can be very large (several Megabytes), or, smaller ACFILEs with a reduced data set may be generated. The content of ACFILEs generated by TRANSP are controlled by the setting of CHARACTER arrays SELOUT and SELAVG in the TRANSP namelist. SELOUT specifies the content of any output ACFILEs. If defaulted, the ACFILE will contain *everything* in TRANSP COMMON. SELAVG specifies which quantities are to be "averaged" before being written into the ACFILE. If defaulted, *no* quantities are averaged before output. The list of quantities to average must be a subset of the list of quantities to output. Example (1): SELOUT(1)='RMOM0 RMOM YMOM FBM EFBM NZNBMA NZNBME' SELOUT(2)='NMOM NZNBMR TBM1 TBM2 BMVOL RHB NZONES TI TE RHOEL' SELAVG='FBM' specifies the named TRANSP COMMON symbols for ACFILE output; of these only FBM will be "averaged" prior to output. Example (2): (SELOUT defaulted) SELAVG='FBM RHB' specifies that ALL TRANSP COMMON symbols will be output to the ACFILE; of these only FBM and RHB will be averaged prior to output. The time of output of the ACFILEs is controlled by TRANSP namelist array OUTTIM; OUTTIM(1)= the first ACFILE output time; OUTTIM(2)= the second ACFILE output time, etc. OUTTIM must be specified in ascending order; if OUTTIM(j+1).le.OUTTIM(j) then no j+1'th ACFILE is written. No more than 5 ACFILEs per run are allowed. If TRANSP run TFTR.85 12345A01 generates say 3 ACFILEs, these files will be archived in the disk area RUNDATA:[TRANSP.TFTR.85] using the following filenames: 12345A01.DATA1 (at time OUTTIM(1)) 12345A01.DATA2 (at time OUTTIM(2)) 12345A01.DATA3 (at time OUTTIM(3)). (The ACFILE "header" also gives the file data output time). For the jth output ACFILE, AVERAGING of data is defined using TRANSP namelist controls: AVGTIM -- average the data over time range OUTTIM(j)-AVGTIM to OUTTIM(j) AVGSAMP -- averaging sampling time (if MTHDAVG=1) MTHDAVG -- if =1, average SELAVG selected data at the end of a TRANSP energy balance timestep, if AVGSAMP or more simulated time has passed since the last time the data was averaged. If =2, average the data after each Monte Carlo fast ion model timestep. The ACFILE header will contain NUMAVG = the number of time points summed in computing averaged data. TRANSP uses temporary files to store the averaging sums while these are being accumulated. II. Contents of an ACFILE. The ACFILE consists of three parts: o file header o averaging header o data items The file header contains the following information: TRANSP run id TRANSP ACFILE sequence number (1,2, ... ,5) TRANSP simulation time -- time of output of ACFILE. ACFILE format code Size of ACFILE contents selector CHARACTER data buffers. Contents selector data Averaging selector data. The averaging header contains: The number of time points averaged (NUMAVG). The averaging method (MTHDAVG, described above). The averaging time interval (AVGTIM, described above). The averaging sampling time (AVGSAMP, described above). Data items are written and read sequentially in *alphabetic order*. Each data item consists of an item header and item data. The item header specifies the TYPE of data (REAL, INTEGER, or LOGICAL), the DIMENSIONALITY of the data (0 for scalar, 1 for 1d array, 2 for 2d array, etc.), the NAME of the data (i.e. the TRANSP COMMON symbol which contained the data at output time), and the size of individual array dimensions if applicable. The item data is the data itself in encoded form. The encoding is such that if the data is read and decoded on the same machine as where it was written, the binary data as written is reproduced precisely. If the data is read on a different machine from where it was written, the data will be reproduced as precisely as possible given different resolutions of representations of floating point numbers on different machines. Different encoding methods are used for real, integer, and logical data. If the data is a scalar, both the header and data are on the same line. For example: *R0 XPI M4AfT0 encodes the TRANSP scalar symbol XPI as having the value 3.141593. In this entry, the "*" denotes the start of a header, "R" denotes REAL, 0 denotes scalar, the (8 character) name is "XPI ", and the value field "M4AfT0" is a base 64 ascii encoding of the sign, exponent, and mantissa of the 32 bit floating point number XPI. If the data is a logical array its header and data records might look like *L2 LLOGIC 7 2 TFTFTFTFTFTFTF This would encode an array which in FORTRAN was declared LOGICAL LLOGIC(7,2) with values LLOGIC(1,1)=.TRUE., LLOGIC(2,1)=.FALSE., ... etc. (The sample data yields .TRUE. if the sum of element indices is even, .FALSE. otherwise). Note that integers are also encoded base 64. The ascii digits for this encoding are: 0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ<> so that in this code, decimal 10 = a, decimal 63 = >, decimal 66 = 12, etc. Positive integers are preceded by a blank; negative integers by a "-" character; a variable number of characters are used to encode the integer, depending on its size. 32 bit floating point numbers are encoded always in 6 characters, one 6 character block per floating point word. Each character is from the base 64 character set and represents 6 bits of accuracy. Thus 36 bits are avaible in the 6 character encoding; more than enough for the desired 32 bit accuracy. The floating point representation is portable across machine boundaries: if an ACFILE written on one machine is read on another, the resulting decoded floating point numbers will be the same in so far as differences in machine representations allow. This is accomplished by deriving for each floating point number its IEEE standard representation (not a machine specific representation) before converting to ascii code. To save file space, consecutive blocks of floating zeroes are encoded via the special six character code _0:nnn, where nnn is a base 64 integer giving the number of consecutive zeroes. This method of data compression has proven highly effective in TRANSP tests-- which means of course that there often are large blocks of consecutive zeroes in TRANSP COMMON. III. Cautionary Note -- ACFILE contents definition evolves in time. The contents of an ACFILE generated by TRANSP are essentially an image of all or a subset of TRANSP COMMON. TRANSP COMMON is the internal communications infrastructure for TRANSP, and therefore it is under continual development. If TRANSP COMMON symbols are redefined or redimensioned, the image of such symbols in ACFILEs created prior to the modification will no longer match the current definition or dimensioning. This may cause these parts of the old ACFILE to be inaccessible to a program which has an ACFILE input facility but uses the new TRANSP COMMON. Similarly, a program which reads ACFILE data but does not use the new TRANSP COMMON would not be able to read the modified symbols from new ACFILEs because its internal idea of the symbol definition would not have kept pace with the definition in the new TRANSP COMMON. ***Therefore***, ACFILEs should not be considered permanent, stable storage mechanisms for TRANSP output data. Rather, they have a limited shelf life, in terms of software compatibility. It should be noted, however, that the shelf life will be far greater than what was available previously. Previously, TRANSP COMMON images were binary files, which became completely inaccessible in case of any modification whatever to TRANSP COMMON. In ACFILEs, access on a symbol by symbol basis is possible, and will remain possible as long as the symbol is not redefined in TRANSP COMMON. Typically, symbols are not redefined for years at a time. However, it is possible that one day a major upgrade will be done (e.g. increase the number of radial zones, or, increase the number of fast ion species) which will make large categories of symbols in old ACFILEs inaccessible. The point is not to rely on the permanent utility of data stored in ACFILEs. This will not be supported, as it would unduly constrain the ability to make modifications and improvements in TRANSP itself. IV. Possible uses for ACFILEs. ACFILEs have a number of potential uses: (1) Quasi-stable storage facility for TRANSP output data which is not available through RPLOT. Example: higher dimensional structures such as the fast ion distribution function FBM. (2) Quasi-stable TRANSP post-processors. TRANSP COMMON could be reloaded from an ACFILE into a specialized post-processor which could do detailed calculations not included in ordinary TRANSP runs. Such calculations could involve use of TRANSP facilities (e.g. the straight line tracker) provided the ACFILE defines a sufficient subset of TRANSP COMMON. (3) Quasi-stable storage facility for TRANSP restart records. A binary restart record may be converted to ACFILE format using the program TREXE:CONVERT.EXE. Then, a TRANSP BUILD may be executed. This will cause CONVERT to be relinked with a new TRANSP COMMON image. Then, CONVERT may read back in the restart ACFILE and create a binary image restart file compatible with the new COMMON block. The new restart file will contain everything defined except for new symbols or symbols whose dimensioning was changed by the TRANSP BUILD. In most cases, this will be good enough to allow a debug project based on a single run to continue through one or two TRANSP BUILDs. Previously, all debug runs had to be scrapped when a TRANSP BUILD was run and TRANSP COMMON changed. (4) Quasi-stable restart and crash records that are portable across machine boundaries. ACFILEs are encoded in ASCII. This creates new possibilities for debugging. A TRANSP crash occurring on one machine may be studied on a different machine. Thus in principle an ACFILE image of TRANSP COMMON could be created at a remote machine, e.g. an IBM system at JET, and then shipped to PPPL for analysis. V. Programs that read or write ACFILEs. TRANSP will write ACFILEs under namelist control, as described in section I. above. In addition, the following programs are available in TREXE:. (The sources are in CMS libraries; readonly copies are in the CMS reference directories indicated by the logical name SOURCE:, defined to TRANSP users). ACTEST -- standalone test driver for ACFILE i/o. This program writes, reads, and rewrites a small test ACFILE. The source gives a good example of calls to ACFILE i/o routines. ADISPLAY -- read and display contents of arbitrary ACFILE. The user must know the names of data (real, integer, or logical scalars or arrays) stored in the ACFILE to be read. However, the user does not need to know in advance the data structure (i.e. array dimensioning information). Assuming the ACFILE was produced by TRANSP, then the names of available symbols are given in the TRANSP run namelist setting for "SELOUT", or, if this is defaulted, then, any name in TRANSP COMMON can be used. The contents of TRANSP COMMON are listed in SOURCE_FREESHARE:PORT.FOR. CONVERT -- converts TRANSP binary COMMON image (i.e. a restart file) to ACFILE format, or vice versa. CONVERT includes an up-to-date version of TRANSP COMMON. Input binary restart files must match this image precisely. In other words, if there is a special need, use CONVERT to store binary image files in quasi-stable ACFILE format **BEFORE** changing TRANSP COMMON by running the TRANSP BUILD procedure. VI. Subroutines used to read ACFILEs. There are two modes for accessing ACFILEs: (1) Standalone mode. In this mode, an ACFILE is opened and selected symbols are read out into arrays that are local to the reading program. The program must have arrays that will hold the precise array structures that are defined in the ACFILE. The code for such a program will use the following TRANSP subroutine libraries (in TRANSP$:[OBJ.LIB]): ACLIB HPSORT I3ENCOD VAXONLY which must be linked to allow use of the ACFILE i/o subroutines needed to read the file. Existing sample programs: ACTEST and ADISPLAY. The ADISPLAY program (source in SOURCE_ADISPLAY:ADISPLAY.FOR) can be used to read and display the contents of any symbol in any ACFILE. (2) TRANSP mode. In this mode, the ACFILE is assumed to contain an up-to-date image of (all or part of) TRANSP COMMON. The TRANSP generated subroutine ASCXALL is called to load TRANSP COMMON from the contents of the ACFILE. The code for such a program will use TRANSP subroutine libraries TRCORE TR_VAX TRULIB TRNGEN ASCXLIB OLYMPS BALLIB ACLIB HPSORT I3ENCOD TRNDUM VAXONLY which must be linked to allow use of the ASCXALL routine to control the ACFILE i/o operation. There are so many libraries, because a substantial portion of the TRANSP i/o system needs to be linked. Existing sample program: CONVERT (see the main program source file SOURCE_CONVERT:CONVERT.FOR). Of course, a program which read an ACFILE to load TRANSP COMMON and then wanted to do a calculation using TRANSP facilities, would need additional TRANSP libraries. All of TRANSP is built out of approximately 40 libraries. User code will call subroutines in the library ACLIB, plus the ASCXLIB subroutine ASCXALL if access mode (2) is used. Code for all main program drivers and subroutines are in the directories SOURCE:, in files .FOR where gives the name of the main program or subroutine. In this document, I will not give a description of calling arguments of all subroutines, since this is already documented in the source code. Instead I show valid call sequences for ACFILE read operations, with commentary. Arguments which are input to subroutines are shown in upper case. Call Sequence for standalone program which reads an ACFILE: =========================================================== Examples: TRANSP programs ACTEST and ADISPLAY. Program outline: start of program: CALL ACINIT ... initialize AC system file loop: CALL ACOPRD(...) CALL ACAVHRD(...) CALL ACLCLR(...) CALL ACLSET(...) data read loop: CALL ACDATRH(...) ! optional CALL ACDATRD(...) if not done, goto data read loop CALL ACCLOS(...) if not done, goto file loop end of program. This program is reading ACFILE data into local arrays (not TRANSP COMMON). 1. Once at the beginning of the program, CALL ACINIT to initialize the AC file system internal COMMON block. 2. To open an ACFILE and read file header, CALL ACOPRD(LUN,RUNID,NUMID,DIRECTORY, > ztime,iformat,MAXSEL,nsel,selout,selavg, ier) ACOPRD receives enough information to name and open the desired ACFILE; it returns the indicated arguments based on values read from the ACFILE header. If all is successful, IER=0 is returned; if there is an error, error messages will be written and IER.NE.0 will be returned. The character arrays CHARACTER*80 SELOUT(MAXSEL),SELAVG(MAXSEL), elements 1 through NSEL, give file contents information. These items are defined from originating TRANSP run namelist (see section I above). Presently, the parameter MAXSEL is set to 5 in TRANSP COMMON. A call to ACOPRD is required. IER should be checked. 3. Next, read averaging header, CALL ACAVHRD(numavg,mthdavg,avgtim,avgsamp, ier) which reads averaging information (as per TRANSP run namelist, section I above). A call to ACAVHRD is required. IER should be checked. 4. Next, select the list of items to read. The values of SELOUT and SELAVG may be as defined by the call to ACOPRD, or, SELOUT may be modified to specify only a subset of the file contents. I.e. the code DO I=1,NSEL SELOUT(I)=' ' ENDDO SELOUT(1)='TE TI RHOEL RHOB FBM' reduces the read list to the 5 named TRANSP COMMON arrays TE, etc... It is NOT required to set SELOUT explicitly, unless a subset of the ACFILE data is to be read and the ACNEXT routine (below) is to be used to sequence read operations. Then: CALL ACLCLR(IFORMAT) ! IFORMAT read from ACOPRD CALL ACLSET(NSEL,SELOUT,SELAVG, ier) ACLCLR initializes the AC file internal i/o symbol list, and ACLSET defines and alphabetizes the internal i/o symbol list. These calls are required. IER should be checked. 5. Fetch next name from ACFILE internal i/o list: CHARACTER*8 ZNAME, and CALL ACNEXT(zname,iend) where ZNAME is the next name, and IEND=1 specifies end-of-list (no more names). CAUTION -- ACNEXT does not advance the list pointer, so, subsequent calls to ACNEXT will return the same name unless some action is taken. This can lead to infinite loops if programming is not done carefully. It is not required to use ACNEXT; the program may read the ACFILE directly, but symbols names must be given in ALPHABETIC ORDER! 6. Skip a symbol (advance the AC internal list) with CALL ACSKIP ...not required, but may be useful e.g. in error recovery. 7. Read a symbol data header (optional): CALL ACDATRH(ZNAME,ztypi,idims,isize,ier) Reads the header data for the named symbol. The header data indicates what type of data is in the file (CHARACTER*1 ZTYPI is set to 'R', 'I', or 'L' for real, integer, or logical); the symbol array dimensionality and dimension sizes (INTEGER IDIMS(10); IDIMS(1)=dimensionality, IDIMES(J+1) gives the size of the jth array dimension; scalars have IDIMS(1)=0). ACDATRH is optional but gives the programmer more control by allowing a peek ahead at the dimensioning of the next symbol to be read. See for example its use in the program ADISPLAY. If used, IER MUST BE CHECKED. IER=0 is normal. If there is an error, the caller should exit any read loop and go back to the file loop start (e.g. step 2. above). 8. Read a symbol data header and data (required): ** modified DMC May 2007 ** portability issues require that separate routines be used for each data type; this is a change not backwards compatible with old software versions (but I believe ACLIB routines are de facto private to TRANSP now; TRANSP has been updated). To read REAL data (that was stored as REAL or REAL*8: CALL ACDATRD_R(IDIMS,ZNAME, zdata, iavg,NERRLEV, ier) To read REAL*8 data (that was stored as REAL or REAL*8: CALL ACDATRD_R(IDIMS,ZNAME, z8data, iavg,NERRLEV, ier) To read INTEGER data: CALL ACDATRD_I(IDIMS,ZNAME, iidata, iavg,NERRLEV, ier) To read LOGICAL data: CALL ACDATRD_L(IDIMS,ZNAME, lldata, iavg,NERRLEV, ier) This reads both the header and the data itself. Note that the data type and dimensioning information as well as name are INPUT to this routine -- the data will not be read unless there is a perfect match of name, type, and dimensioning. If the match is not satisfied, the data item is cleared (all elements set to 0.0 if real, 0 if integer, or .FALSE. if logical). ZDATA can be a real, integer, or logical array or scalar, depending on the symbol characteristics. It is up to the programmer to assure that the storage ZDATA actually has the dimensioning characteristics specified. NERRLEV=0 is set to "ignore" the following types of errors: (a) symbol ZNAME not found in the ACFILE (b) symbol ZNAME type or dimensioning mismatch in which case IER=0 is returned, and ZDATA is cleared (no file data is read). ACDATRH may be used to peek ahead at the ZNAME symbol's type and dimensioning characteristics; after which an appropriate ACDATRD call may be generated-- see for example the TRANSP program ADISPLAY. No matter what the setting for NERRLEV, IER MUST BE CHECKED. IER=0 is normal. If there is an error, the caller should exit any read loop and go back to the file loop start (e.g. step 2. above). 9. After all data has been read (or after errors), close the file. CALL ACCLOS(IER) Call Sequence for program which uses TRANSP COMMON and reads an ACFILE: ======================================================================= Example: TRANSP program CONVERT. Program outline: start of program: CALL ACINIT ... initialize AC system file loop: set up run id ... identify a TRANSP run CALL BASIC ... initialize i/o section of TRANSP COMMON CALL ACOPRD(...) CALL ACAVHRD(...) CALL ACLCLR(...) CALL ACLSET(...) data read: NACTION=4 CALL ASCXALL(KRET) if(KRET.NE.0) then ... error ... else ... process data .... CALL ACCLOS(...) if not done, goto file loop end of program. This program is reading ACFILE data into TRANSP COMMON. This means that the program should have the declaration line INCLUDE 'TRCOM/NOLIST' where TRCOM is a logical name pointing to the TRANSP COMMON block declaration file SOURCE_INCLSHARE:TRANS.COM. Be aware that this is a HUGE declaration containing almost 3000 names! It is recommended that programs and subroutines which include TRCOM use the OLYMPUS variable/array naming convention: if the variable or array name starts with A-H or O-Y, it is a TRANSP COMMON REAL VARIABLE or ARRAY name. if the variable or array name starts with L-N it is a TRANSP COMMON INTEGER or LOGICAL VARIABLE or ARRAY name. if the variable or array name starts with I-K it is a local integer or logical variable or array name. if the variable or array name starts with Z it is a local real variable or array name. for CHARACTER data, all first letters but I,J,K, and Z are reserved for TRANSP COMMON. Also, TRCOM already includes some symbols, e.g. the CHARACTER*80 buffers SELOUT(MAXSEL) and SELAVG(MAXSEL), that are useful for doing ACFILE i/o. Sequence of calls: 1. Once at the beginning of the program, CALL ACINIT to initialize the AC file system internal COMMON block. 2. A call is required to set up TRANSP i/o facilities. The program must first obtain a TRANSP run id and load this in a special COMMON block "ZFILCON". ZFILCON contains two CHARACTER items, one of which is set to the run id, and the other to the string 'PRESET'. This mechanism gives the calling program (instead of a low level TRANSP library routine) control over the TRANSP run id. The thing to do is to copy what is done at the start of the program CONVERT. When it is done, the following call sets up TRANSP i/o facilities: CALL BASIC 3. To open an ACFILE and read file header, CALL ACOPRD(LUN,RUNID,NUMID,DIRECTORY, > ztime,iformat,MAXSEL,nsel,selout,selavg, ier) ACOPRD receives enough information to name and open the desired ACFILE; it returns the indicated arguments based on values read from the ACFILE header. If all is successful, IER=0 is returned; if there is an error, error messages will be written and IER.NE.0 will be returned. The character arrays CHARACTER*80 SELOUT(MAXSEL),SELAVG(MAXSEL), elements 1 through NSEL, give file contents information. These items are defined from originating TRANSP run namelist (see section I above). A call to ACOPRD is required. IER should be checked. 4. Next, read averaging header, CALL ACAVHRD(numavg,mthdavg,avgtim,avgsamp, ier) which reads averaging information (as per TRANSP run namelist, section I above). A call to ACAVHRD is required. IER should be checked. 5. Set up the AC file internal i/o list with these calls: CALL ACLCLR(IFORMAT) ! IFORMAT read from ACOPRD CALL ACLSET(NSEL,SELOUT,SELAVG, ier) ACLCLR initializes the AC file internal i/o symbol list, and ACLSET defines and alphabetizes the internal i/o symbol list. These calls are required. IER should be checked. 6. READ ALL DATA: set TRANSP COMMON switch NACTION=4 and call the TRANSP generated code subroutine CALL ASCXALL(kret) and check that KRET=0 (no errors) on return. 7. After all data has been read (or after errors), close the file. CALL ACCLOS(IER) ---------------------------------------------------------------------- Questions and comments about this documented should be forwarded to D. McCune at Princeton PPL, USA (609) 243-2731. PPPL VAX user ids: TRANSP or DMCCUNE. Internet address: transp@pppl.gov or dmccune@pppl.gov Best wishes to you. ---------------------------------------------------------------------- 2007 update (D. McCune) Character string scalars (character*nnn str) or 1d vectors (character*nnn strvec(idim)) are now supported by aclib, BUT the length of any character string or string vector element must be no more than 250 characters. Preservation of TRANSP RESTART files through version changes has been revived. See the cvs commit log, copied here: June 20, 2007: Revived the "convert" program, which maps between binary and encoded ASCII versions of the trcom state. Modifications were done so that "convert" will work with the new, dynamically allocated trcom. The i3encod library was also improved for speed and accuracy of encode/decode of real*8 numbers-- bit for bit preservation of all trcom state except the "NACTION" control variable used by convert itself... has been achieved. Also part of this: AClib now supports character strings and vectors of character strings, and, the gensis code generator incorporates I/O of trcom character strings in the ascxlib library. Use of "convert" can often allow a TRANSP run that was interrupted to be restarted even though the version of "trcom" has changed. The old RS.DAT binary cannot be read directly after the "trcom" change, but, TRANSP now produces RS.DAT_ASCII and RS.DAT_DIMS as well. Feeding these to "convert" allows a readable binary RS.DAT to be produced. All variables and arrays that exist in both the old and new version of trcom are preserved bit for bit. However, any new variables and arrays are initialized to 0, 0.0, .FALSE., and blank. This method works if the changes to trcom consist in the addition and/or removal of variables and arrays. If however new dimensions are added, or, if the dimensioning of arrays (rank, or which symbolic dimensions are used) has changed, the method is not expected to work.