BBSGI.MAN - BRANCH-BUS/SILICON GRAPHICS DATA ACQUISITION SYSTEM (BBSGI) For IRIX 5.x systems - AT&T UNIX System V Release 4 r.imossi/oldf/bnl 25-Oct-1993 BRANCH-BUS/SILICON GRAPHICS DATA ACQUISITION SYSTEM (BBSGI) TABLE OF CONTENTS ----------------- 1.0 INTRODUCTION 2.0 HARDWARE-SOFTWARE CONFIGURATION 3.0 USER INTERFACE SUBROUTINES 4.0 ANSI/IEEE FAST BUS INTERFACE 5.0 DEVICE LIST INTERFACE 6.0 APPLICATION PROGRAM COMPILATION AND LINK 7.0 LOCAL VMEBUS-TO-BRANCH-BUS INTERFACE 8.0 REMOTE BRANCH-BUS INTERFACE 9.0 INSTALLATION OF HARDWARE AND SOFTWARE 10.0 BRANCH BUS DIAGNOSTIC PROGRAM (bbdiag.c) 11.0 MISCELLANEOUS INTERFACE LIBRARY ROUTINES 12.0 DESIGN DECISIONS 13.0 REFERENCES APPENDIX A HARDWARE REGISTER DEFINITIONS APPENDIX B BRANCH BUS OBJECT LIST (LAYER 2) INTERFACE APPENDIX C ERROR CODES 1.0 INTRODUCTION The BRANCH-BUS/SILICON GRAPHICS DATA ACQUISITION SYSTEM (BBSGI) is a product of the Online Data Facility (OLDF) at Brookhaven National Laboratory, Upton, New York and Experiment 787. The product uses a Branch Bus driver created by Fermi National Laboratory, Batavia, Illinois. The product's purpose is to provide data acquisition programs with fast and easy access to Fast Bus and VMEBUS modules. A choice of DMA (VBDMA) or Programmed IO (VBBC) interfaces is permitted. Users have several choices of software interfaces for their data acquisition programs. One is based on the ANSI/IEEE (std 1177-1989) Fast Bus Standard Routines. Another is a device list driven set of routines. There are lower level interfaces if the user desires his own interface creation or greater control of the IO. A test program (bbdiag.c), used by OLDF for system debugging, is available as a template data acquisition program. The program has an interactive/prompting interface that uses the software interface library. +----> FAST BUS MODULES SILICON Branch Bus | GRAPHICS <-------------------------------+ COMPUTER | +----> VMEBUS MODULES 2.0 HARDWARE-SOFTWARE CONFIGURATION Branch Bus +----------------------------+ V | | B | | S D | B G M | V user I A | I modules... |___|_____________|________| | |_____|_________|_|_|_|_|__| SGI local VMEBUS (crate 0) | Remote VMEBUS (crate 1) | | +-----------+ | B | B | F user | C modules... | |____|_________|_|_|_|_|__| | Remote Fast Bus (crate 2) | | +-----------More FastBus/VMEBUS crates (crates 3 to 31) The BBSGI hardware-software consists of o Any Silicon Graphics computer - this project used a Silicon Graphics 4D/320S computer; MIPS 33 MHZ R3000 CPU; MIPS R3010 floating point unit; 256 MB memory; o Local VMEBUS - Silicon Graphics computer's internal VMEBUS; assigned crate number 0; user VMEBUS modules can be placed here if slots available; o Local VMEBUS-to-Branch-Bus interface - resides in CPU's local VMEBUS; a Branch Bus master and VMEBUS slave; there is a choice of two devices: a VBBC for programmed IO or a VBDMA for DMA transfers; VBDMA is a combination of modified VBBC and BVI boards; See a later section for details; o Branch Bus - max transfer rate of 20 Megabytes/sec; Max block size of 256K bytes; max length of 50 feet; o Remote BVI - Branch Bus/VMEBUS Interface; a Branch Bus slave and VMEBUS master; Max transfer count of 256K bytes; Physical switches on unit identify a unique crate number for the connected VMEBUS; o Remote BBFC - BNL's Branch-Bus/Fast-Bus Controller interface; a Branch Bus slave and a Fast Bus master; Max transfer count of 256K bytes; Physical switches on unit identify a unique crate number for the connected Fast Bus; o Remote VMEBUS crate - max transfer rate of 40 Megabytes/sec; holds user VMEBUS modules; o Remote Fast Bus crate - max transfer rate of 40 Megabytes/sec; holds user Fast Bus modules; o Software interface library (bblib.c); o Modified Fermi Branch Bus driver for Silicon Graphics computers (bb_.c, vbbcreg.c); o Diagnostic program (bbdiag.c) - program the staff used to test hardware and software interface library; user can use it to test equipment and as a template for a data acquisition program; 3.0 USER INTERFACE SUBROUTINES +----------------------------------------------------------+ Layer 4 |User Application program/Test program (eg,bbdiag.c,bnlssp)| +----------------------------------------------------------+ | | | | | | | | | | | | | | | V V | | | +------------------------+ | | | Layer 3 |Device List |ANSI/IEEE | | | | |Interface |Fast Bus | | | | | (dl...) |Interface | | | | | | (f...) | V | | +------------------------+----------+ | | Layer 2 |Branch Bus Object List Interface | | | | (ol...) | V | +-----------------------------------+-----------+ | Layer 1 |Branch Bus driver Interface | | | (bb...) | V +-----------------------------------------------+-----------+ Layer 0 |Unix System Call Interface/Branch Bus driver | | (bb_...) | +-----------------------------------------------------------+ | | V +-----------------------------------------------------------+ | VBBC/Branch Bus Hardware | +-----------------------------------------------------------+ The user interface library (bblib.c) is designed into 3 layers: 1, 2 and 3. Each layer hides details from the next higher layer. The user data acquisition program can be considered as layer 4, the application layer. The lowest layer, which is the Unix operating system where the Branch Bus driver exists, can be considered as layer 0. The user will normally access the system at layer 3. There are two interfaces at layer 3 to choose. The first is the device list interface that provides routines that are driven by a device list file of VMEBUS and Fast Bus modules read and write operations. The second choice is the ANSI/IEEE Fast Bus interface that provides routines that read/write one Fast Bus module at a time. The user can access the system at more primitive layers: 0, 1 or 2, but this adds little functionality at the risk of crashing the computer. If the user is unhappy with the two interfaces at layer 3, he can create his own interface by using layer 2 routines; see the appendix. All software is written in the C language. The user program can be written in any computer language. A general purpose, layer 4, data acquisition program is available for those who use the SLAC Scanner Processors (SSPs). It is called the BNL SSP Data Acquisition System. This document file (bbsgi.mem) and all software described in this document is found in /usr/var/sysgen/bb. 4.0 ANSI/IEEE FAST BUS INTERFACE This implementation supports only Immediate Execution Mode, meaning only one Fast Bus device is read/written per execution. These routines are written in C like everything else, except they were made to easily be called from a Fortran program. It was assumed most programs using this interface are written in Fortran. The calling string declarations in this section show first the Fortran call followed by the C call. 4.1 FAST BUS INTERFACE ROUTINES Here are all the implemented ANSI/IEEE Fast Bus Interface routines. Subroutine arguments are described in the succeeding section. OPEN Call open routine before calling any other routine in this standard. Device name "/dev/bb0" is DMA mode. Device name "/dev/bb1" is Programmed IO mode. call fbopen(iretcod,devicename) fbopen_(&iretcod,devicename); CLOSE Call close routine when use of Fast Bus no longer required. call fbclose(iretcod) fbclose_(&iretcod); SINGLE DATA WORD TRANSFERS Read One word from data space. call frd(iretcod,ieid,ipad,isad,ibufmode,iword) frd_(&iretcod,&ieid,&ipad,&isad,&ibufmode,iword); Write One word to data space. call fwd(iretcod,ieid,ipad,isad,ibufmode,iword) fwd_(&iretcod,&ieid,&ipad,&isad,&ibufmode,iword); Read One word from control space. call frc(iretcod,ieid,ipad,isad,ibufmode,iword) frc_(&iretcod,&ieid,&ipad,&isad,&ibufmode,iword); Write One word to control space. call fwc(iretcod,ieid,ipad,isad,ibufmode,iword) fwc_(&iretcod,&ieid,&ipad,&isad,&ibufmode,iword); BLOCK TRANSFERS Read buffer from data space. call frdb(iretcod,ieid,ipad,isad,ibufmode,ibuf,imaxbytes) frdb_(&iretcod,&ieid,&ipad,&isad,&ibufmode,ibuf,&imaxbytes); Write buffer to data space. call fwdb(iretcod,ieid,ipad,isad,ibufmode,ibuf,imaxbytes) fwdb_(&iretcod,&ieid,&ipad,&isad,&ibufmode,ibuf,&imaxbytes); Read buffer from control space. call frcb(iretcod,ieid,ipad,isad,ibufmode,ibuf,imaxbytes) frcb_(&iretcod,&ieid,&ipad,&isad,&ibufmode,ibuf,&imaxbytes); Write buffer to control space. call fwcb(iretcod,ieid,ipad,isad,ibufmode,ibuf,imaxbytes) fwcb_(&iretcod,&ieid,&ipad,&isad,&ibufmode,ibuf,&imaxbytes); STATUS INFORMATION Get Summary Status. This routine must be called after each transfer to get the error status. ( The error status is not the function return code.) Summary Status information is generated after each data transfer. It provides the total number of bytes read and written, and an error code indicating a good or bad transfer. See appendix for error code definitions. If ierror is nonzero, then an error occurred and supplementary status is needed to further describe the error or errors. call fsgsum(iretcod,ieid,ierror,ireadct,iwritect) fsgsum_(&iretcod,&ieid,&ierror,&ireadct,&iwritect); Get Supplementary Status. The Supplementary Status routine is called when the Summary Status routine returns an error (ierror is nonzero). Repeated calls to the supplementary status routine extracts all the available status information. The first call to this routine returns the same Error Code as that returned by FSGSUM (Get Summary status function). The Supplementary Status routine returns an Error Code FENMST (No more status) when no more information is available. An associated-parameter, iassoc, is returned with each error code, ierror. It is a 32-bit hardware register reading that adds detail to the error. For error code FEHERR (implementation dependent hardware error), the associated-parameter is the VBBC CSR Register; for error code FENMST (No more status), it is zero; for all other error codes, it is the BBFC Fast Bus Status Register; call fsgsup(iretcod,ieid,ierror,iassoc,iwhere) fsgsup_(&iretcod,&ieid,&ierror,&iassoc,&iwhere); Translate error code to message string. An error message can be generated from any returned error code and associated-parameter. The user must provide the error code, ierror, the associated-parameter, iassoc, and the unused arguments: ieid, iwhere. call fstran(iretcod,ieid,ierror,iassoc,iwhere,imessage) fstran_(&iretcod,&ieid,&ierror,&iassoc,&iwhere,imessage); 4.2 FAST BUS INTERFACE ARGUMENTS These defined the parameters used in the calls: Fortran integer iretcod,ieid,ipad,isad,ibufmode,iword,ibuf(xxx),imaxbytes integer ireadct,iwritect,ierror,iassoc,iwhere character*80 imessage character*20 devicename C Language int iretcod,ieid,imaxbytes,ireadct,iwritect,ierror,iwhere; unsigned int ipad,isad,ibufmode,iword,ibuf[xxx],iassoc,iwhere; char imessage[81],devicename[20]; Where o iretcod: Return code, returned by the called routine. There are two possible codes: FENORMAL for a normal (good) return and FEERR for an error return. See the appendix for error definitions. This code only indicates whether the function executed correctly, the ierror argument indicates whether the data is good. o ieid: Environment identifier, not used; all execution is immediate; delayed execution (device list) is a possible future enhancement. o ipad: Crate id and Primary address; Crate is in the leftmost byte, Primary Address in the rightmost byte. o isad: Secondary address. o ibufmode: Buffer mode, not used; all buffers are addressed as reference rather than value. o iword: One word variable used in single word transfers. o ibuf[xxx]: Array used in multiword transfers where xxx is the buffer size in words (4 bytes to a word). o imaxbytes: Maximum number of bytes to be read or written. o ireadct: Total number of bytes read into the buffer. Returned by the called routine. o iwritect: Total number of bytes written from the buffer. Returned by the called routine. o ierror: Fast Bus error code. This indicates whether the data is good. When the function return code (iretcod) is good, this item must be tested to see if the data is good. Good data is represented with error code FENORMAL. o iassoc: Associated-parameter is a value that further describes the error; it is normally the VBBC CSR or BBFC Status Register. o iwhere: Where-occurred parameter specifies the error-causing routine; not used. o imessage: Fixed 80 character string that is the error message translated from an error code and associated parameter. o devicename: Branch Bus device (/dev/bb0 for DMA or /dev/bb1 for Programmed IO) 4.3 FAST BUS INTERFACE EXAMPLE PROGRAM A sample Fortran program, bbfc.f, is part of the software distribution in /usr/var/sysgen/bb. See the beginning of the file for compilation instructions. An include file, fberr.inc, defining error codes is also present. This is a simple Fortran program calling the Fast Bus Interface routines. For a C language example see the diagnostic program (bbdiag.c). c cc -cckr -g -o fbtest fbtest.f bblib.c program fbtest character*20 bbdevice integer iretcod,ieid,ipad,isad,ibufmode,ibuf(10) bbdevice = '/dev/bb0' call fbopen(iretcod,bbdevice) ipad=16 isad=0 imaxbytes=16 200 call frdb(iretcod,ieid,ipad,isad,ibufmode,ibuf,imaxbytes) 400 stop end 5.0 DEVICE LIST INTERFACE This is a device-list driven set of routines. The device list consists of commands to read/write any number of Fast Bus and VMEBUS devices. The entire list is executed inside the driver with one call. The list is created with your favorite text editor or with the provided interactive dlcreate() routine. 5.1 DEVICE LIST DEFINITION The device list consists of lines of parameters. Each parameter is free form and separated by blank space. There are two types of lines: device read/write and DATA for a write. A device read/write line consists of parameters: bus-type (FASTBUS/VMEBUS), io-mode (READ/WRITE), byte-count, crate, Fast Bus primary-address or VMEBUS address, Fast Bus secondary-address, Fast Bus addressing space (DATA/CONTROL). FASTBUS io-mode byte-count crate primary-address secondary-addr space READ (decimal) (decimal) (decimal) (hexidecimal) DATA WRITE CONTROL VMEBUS io-mode byte-count crate VMEBUS-address READ (decimal) (decimal) (hexidecimal) WRITE A DATA line provides write data for the preceding write line. There are several formats. The following places user-defined hex data into sequential words of the write buffer starting at the first location. A maximum of 8 words per line, but any number of DATA lines can be strung together. DATA word1 word2 word3 word4 word5 word6 word7 word8 (hex) (hex) (hex) (hex) (hex) (hex) (hex) (hex) The following DATA lines fill the entire write buffer with data. These fillers start at the next location following the user-defined data. If there is no user-defined data, these fillers fill up the entire write buffer. If two or more fillers refer to the same write buffer, then the last filler definition takes presidency. This fills the write buffer with the same hexidecimal-word pattern: DATA PATTERN pattern-word (hexidecimal) This fills the write buffer with sequentially increasing (by one) numbers. If starting number is not provides, zero is used for the first location. DATA SEQUENTIAL starting-number (hexidecimal) This fills the write buffer with random numbers. Seed defaults to one if not provided. This random data filler requires the system to write data into the write buffer every time the device list is executed. This will slow down execution. The other filler types are static in that they are written only once at start up time. DATA RANDOM seed-number (decimal) 5.1.1 DEVICE LIST EXAMPLE Here is a sample device list that could be created with a text editor but in this case was made by dlcreate(). FASTBUS WRITE 1024 2 16 00000000 DATA DATA SEQUENTIAL 1 FASTBUS READ 1024 2 16 00000000 DATA VMEBUS WRITE 512 1 82000000 DATA 11111111 22222222 33333333 DATA PATTERN ffffffff VMEBUS READ 128 1 82000010 FASTBUS READ 520000 5 21 00001234 DATA The first line is a Fast Bus write of 1024 bytes, to crate 2, primary address 16, secondary address 0, data space. The second line provides the data for the preceding write command, which is sequentially increasing numbers starting with 1 in the first buffer location. The third line reads the Fast Bus module described in line 1. The fourth line is a VMEBUS write of 512 bytes, crate 1, VMEBUS address 82000000. The fifth line places words of data into the first three buffer words of the preceding VMEBUS write. The sixth line fills out the remainder of the buffer with pattern of ones. The seventh line is a VMEBUS read of 128 bytes, crate 1, VMEBUS address 82000010. The last line is a Fast Bus read of 520,000 bytes, from crate 5, primary address 21, secondary address 1234, data space. 5.2 BUFFER FORMAT The user provides a large buffer to this interface which is returned to the user with all the data read from each device in the list. preceding each device buffer is a header that provides information about the read. There are headers inserted for devices that are written as well, but write buffers are dynamically allocated from a system memory pool so these are not part of the user buffer; this saves space. The header can be changed to the user's needs. Here is the current header layout: RB_NEXT Number of words to next buffer header; last buffer has 0. RB_HDRSIZE Size of header in words RB_BCOUNT Actual byte count of device read/written RB_CSR VBBC CSR register immediately after device read/written RB_FBSTAT BBFC Fast Bus Status register after FAST Bus IO RB_ERROR Returned error code for device; Zero if OK. See appendix for error code definitions. RB_BUF Start of Read buffer 5.3 DEVICE LIST INTERFACE ROUTINES These routines were made to interface to C Language calling programs. OPEN Call open routine before calling any other routine. Device name "/dev/bb0" is DMA mode. Device name "/dev/bb1" is Programmed IO mode. dlopen(devicename); CLOSE Call close routine when use of Branch Bus no longer required. dlclose(); CREATE DEVICE LIST FILE This routine walks you through creating a device list, prompting you for all the necessary information. The list is written out to the specified disk file. dlcreate(filename); RETRIEVE DEVICE LIST FILE This routine inputs the device list file into memory. Any previous device list is replaced. The device list file was created either by dlcreate() or a text editor. dlretrieve(filename,buffer,size); EXECUTE DEVICE LIST The current device list is executed one time. All results are returned in the buffer your specified originally in the dlretrieve routine. dlexec(); TRANSLATE DEVICE LIST ERROR CODE Any error number returned in a header field can be translated to a message. dltran(errnum,errmess); 5.4 DEVICE LIST INTERFACE ARGUMENTS These defined the parameters used in the calls: C Language char filename[132],errmess[81]; int bsize,errnum; unsigned buffer[xxx]; Where o filename: path/filename of device list. o buffer[xxx]: Buffer to read device list IO where xxx o bsize: Size of buffer in bytes. is the buffer size in words (4 bytes to a word). o errnum: Error number from the error code field in the header. o errmess: Fixed 80 character string that is the error message translated from the error code field in the header. 5.5 DEVICE LIST INTERFACE EXAMPLE PROGRAM For a C language example see the diagnostic program (bbdiag.c). 6.0 APPLICATION PROGRAM COMPILATION AND LINK 6.1 C PROGRAMS Add the following to the beginning of your program: #include "/usr/var/sysgen/bb/vbbcreg.h" To compile, link and run your C application program, proga.c, with the interface software do: # cc -cckr -o proga proga.c /usr/var/sysgen/bb/bblib.o # proga Now, to compile, link and run your C application program with the debugger do: # cc -cckr -g -o proga proga.c /usr/var/sysgen/bb/bblib.o # dbx proga 6.2 FORTRAN PROGRAMS Add the following to the beginning of your program: #include "/usr/var/sysgen/bb/fberr.inc" To compile, link and run your Fortran application program, proga.f, with the interface software do: # f77 -o proga proga.f /usr/var/sysgen/bb/bblib.o # proga Now, to compile, link and run your Fortran application program with the debugger do: # f77 -g -o proga proga.f /usr/var/sysgen/bb/bblib.o # dbx proga 7.0 LOCAL VMEBUS-TO-BRANCH-BUS INTERFACE This interface resides in the computer's local VMEBUS and connects to Branch Bus. You can use either the BNL created VBDMA interface or the commercially available VBBC interface. The VBDMA runs in DMA mode at approximately 18 Megabytes/sec, whereas, the VBBC runs in programmed IO mode at approximately 3 Megabytes/sec. 7.1 VBDMA INTERFACE The VBDMA is actually a standard VBBC and BVI with BNL modifications to their daughter boards. The modifications allows the Silicon Graphics computer to initiate DMA transfers, to and from, VMEBUS and Fast Bus modules. The host Silicon Graphics computer must have three VMEBUS slots available: one for the VBBC, one for the BVI and one for a BBT, Branch Bus terminator card. A future enhancement will place the functions of all three boards ( VBBC, BVI and BBT) into one VME based module. A possible short term enhancement could place the BBT terminator on the BVI daughter board to reduce the needed slots to two. The VBDMA normally runs in DMA mode which is the reason for its existence, however, it can be made to run in programmed IO. There are eight "mode select" dip switches on the front panel of the VBBC's modified daughter board. These are the only two valid settings: DMA MODE PROGRAMMED IO MODE -------- ------------------ SWITCH OFF ON OFF ON 1 X X 2 X X 3 X X 4 X X 5 X X 6 X X 7 X X 8 X X 7.1.1 VBDMA INTERFACE LIMITATIONS Mode must be indicated in hardware and software. If the above dip switches are set for DMA, then device /dev/bb0 is opened in software. If the above dip switches are set for Programmed IO, then device /dev/bb1 is opened in software. CAUTION: the computer will crash on data transmission with a VMEBUS time out error when the hardware is set for DMA but the software is set for Programmed IO. There is no way of determining a nonexistent crate in DMA IO transmission. No error is returned. The byte count returned is the original requested byte count. 7.2 VBBC INTERFACE The VBBC is a standard unmodified VBBC that requires two VMEBUS slots in the Silicon Graphics local VMEBUS, one for the VBBC and one for the Branch Bus terminator card. Only programmed IO is possible. However, if the terminator is placed on the VBBC, workstations with only one free slot can run data acquisition. 8.0 REMOTE BRANCH-BUS INTERFACE These interfaces reside in remote crates, connecting remote crates to Branch Bus. Remote VMEBUS crates use a standard BVI interface. A unique crate number (not 0) must be set in the BVI's physical switches. Remote Fast Bus crates use the BNL-engineered BBFC interface. The BBFC emulates a BVI, making the remote FAST Bus Crate look like a remote VMEBUS crate. A unique crate number (not 0) must be set in the BBFC's physical switches. 9.0 INSTALLATION OF HARDWARE AND SOFTWARE Before starting, collect information on the Silicon Graphics computer. Determine the CPU type by executing: # hinv 9.1 HARDWARE INSTALLATION Each VME device has a set of VMEBUS addresses to which it responds. The driver dynamically maps the VME addresses into the host processor address space. The addressing mode chosen is A32NP. For each addressing class, Silicon Graphics has reserved a range of addresses for user-written drivers. These ranges are listed in /usr/var/sysgen/system/irix.sm. The chosen VMEBUS address is 0x1A000000. Set the local VBBC's VMEBUS address switches to the 32 bit address previously chosen: 0x1A000000. This number must match the address set in the VECTOR line of the system configuration file. If you are using the VBDMA interface, set the local BVI crate id to 0. Shutdown the computer, and insert the interface board(s) into the local VMEBUS. Don't forget the VBDMA interface requires a terminator board. Set unique crate numbers on all remote BBFC and BVI boards. Crate 0 is reserved for the local VMEBUS in the computer. Connect the Branch Bus cable to the BBFC on remote Fast Bus crates and the BVI on remote VMEBUS crates. 9.2 SOFTWARE INSTALLATION You must login as root to perform the software installation. Create special device files and set protection for public access: # mknod /dev/bb0 c 240 0 # mknod /dev/bb1 c 240 1 # chmod 666 /dev/bb0 # chmod 666 /dev/bb1 Copy the driver master file into its directory: # cp /usr/var/sysgen/bb/bb_ /usr/var/sysgen/master.d The master file should look like this for a static driver module: *FLAG PREFIX SOFT #DEV DEPENDENCIES c bb_ 240 2 Add a "d" flag to the master file for a dynamically loadable driver: *FLAG PREFIX SOFT #DEV DEPENDENCIES dc bb_ 240 2 Copy the driver system file into its directory: # cp /usr/var/sysgen/bb/bb.sm /usr/var/sysgen/system The system file should look like this, but all in one long line: VECTOR: bustype=VME module=bb_ adapter=0 ipl=1 iospace=(A32NP,0x1A000000,0x20000) At this point you compile and load the driver, but you have two choices: statically load the driver into the kernel or dynamically load/unload the driver into the running kernel. 9.2.1 DRIVER AS A STATIC MODULE IN KERNEL This is the traditional way of permanently installing a driver into the UNIX kernel. Compile the driver using the proper CPU identification determined from the hinv command. For example IP19 compiles as: # cd /usr/var/sysgen/bb # setenv CPUBOARD IP19 # make -f /usr/var/sysgen/Makefile.kernio bb_.o Build the new unix kernel containing the Branch Bus driver. First copy the driver object file into the boot subdirectory, then link a bootable Unix kernel # cp /usr/var/sysgen/bb/bb_.o /usr/var/sysgen/boot # cd /usr/var/sysgen # lboot -u /usr/var/sysgen/bb/unix.install Save the current Unix kernel and reboot the new kernel with the Branch Bus driver: # cp /unix /unix.no_bb # mv /usr/var/sysgen/bb/unix.install /unix.install # reboot If the new kernel is unbootable, boot the original one: >>boot dksc(0,1,0)unix.no_bb then login and restore the original kernel as the default: # cp /unix.no_bb /unix 9.2.2 DRIVER AS A DYNAMICALLY LOADABLE MODULE (DLM) IN RUNNING KERNEL A new IRIX feature supports dynamic loading and unloading of modules into a running kernel, A loadable module must be compiled and linked with cc options that differ from those used on a static module. Create a Makefile by copying : # cp /usr/var/sysgen/Makefile.kernio usr/var/sysgen/bb/Makefile and modifying the cc options pertaining to your processor in Makefile. For example, processor IP19's cc options would be changed from #if $(CPUBOARD) == "IP19" CFLAGS=-D_K32U64 -D_KERNEL -DSTATIC=static -DEVEREST \ -DBADVA_WAR -DIP19 -DR4000 -Wx,-G8 -non_shared -coff -Wc,-pic0 #endif to #if $(CPUBOARD) == "IP19" CFLAGS=-D_K32U64 -D_KERNEL -DSTATIC=static -DEVEREST \ -DBADVA_WAR -DIP19 -DR4000 -Wx,-G0 -non_shared -coff -Wc,-pic0 -r -d -c \ -Wc,-jalr #endif Now compile the driver using the proper CPU identification determined from the hinv command. For example IP19 compiles as: # cd /usr/var/sysgen/bb # setenv CPUBOARD IP19 # make bb_.o Use the "lboot" command to dynamically load the driver, but first unload a possible previous version: # cd /usr/var/sysgen/bb # cp bb_.o /usr/var/sysgen/boot # lboot -U 240 # lboot -L bb_ 9.2.2.1 LOAD DYNAMICALLY LOADABLE DRIVER MODULE AT REBOOT TIME To avoid manually loading the dynamically loadable driver every time the kernel is rebooted, have IRIX do it automatically. On start up, Irix looks in the directory /etc/rc2.d for all S* files and executes them in alphabetic order. Create a file /etc/rc2.d/S99bbdriver with the following contents: lboot -L bb_ If you wish to automatically unload the driver on shutdown create a file /etc/rc2.d/K99bbdriver with the following contents: lboot -U 240 9.3 UNIX KERNEL DEBUGGING For debugging purposes, you can compile the driver with symbol table information to symbolically analyze a kernel dump. Add the -g switch to the cc compilation statement. Note the -g switch prevents driver code optimization, so eliminate it when no longer debugging. To analyze a crash dump, say of vmcore.10: # cd /usr/var/sysgen/bb # dbx -k /unix /usr/var/adm/crash/vmcore.10 To analyze a running kernel: # cd /usr/var/sysgen/bb # dbx -k /unix /dev/kmem To list the system log: # cat /usr/var/adm/SYSLOG To shutdown the system, say to remove power, execute "shutdown" or "halt": # shutdown 9.4 SHAREABLE DEVICE The software interface devices, /dev/bb0 and /dev/bb1, are shareable devices. More than one program can use them at the same time. This is possible because each IO operation runs until completion. No other IO operation can start until the previous is complete. One use of this feature is to run your data acquisition program on one terminal/window and run the Diagnostic Program (bbdiag.c) in another. The Diagnostic Program can monitor your data acquisition program using the GET command or even dynamically modify the environment with the SET command. The SET command can turn on or off tracing, change physical transmission block size, etc. 10.0 BRANCH BUS DIAGNOSTIC PROGRAM (bbdiag.c) The Branch Bus Diagnostic program is not only useful for testing hardware but the user can check results from his program with it. Copy its execution file into your directory and execute: # cp /usr/var/sysgen/bb/bbdiag * # bbdiag There is one main menu and three submenus. You start off in the main menu. Whenever, you are prompted for a command, all the possible commands are displayed on the prompt line. You are given a default command which is the previous command and you select it by just typing carriage return. Commands can be truncated to one character in all menus except the Register IO menu. Here is a hierarchical diagram of the menus: --------------------MAIN MENU------------------ fbio, listio, regio, get, put, menu, exit | | | | | | | | +--REGISTER IO SUBMENU---------------- | | RC,W1,W2,WC,IN,OU,RE,GET,PUT,MENU,EXIT | | | +----------DEVICE LIST IO SUBMENU-------------------- | Retrieve,Create,Save,Display,Xlist,Get,Put,Menu,Exit | +-----------------FAST BUS IO SUBMENU------------------ Xecute,Fbstatus,Service,Reset,Get,Put,Menu,Exit All menus have commands: GET, PUT, MENU, EXIT. GET displays useful hardware and software status information. PUT allows you to change hardware and software parameters. MENU displays help on all the current menu commands. EXIT returns you to the main menu or, if you are in the main menu, exits the program. The main menu merely provides you with the ability to enter one of the submenus. 10.1 FAST BUS IO SUBMENU The FAST BUS IO submenu allows you to execute one Fast bus IO operation. The Fast bus IO operation is defined using the PUT submenu command. You can perform one Fast Bus read/write, single-word/block transfer. You can do the transfer once or looped, get a read buffer dump, etc., all declared within the PUT command. Execution occurs with the XECUTE command. Every time the program prompts you for action, the current Fast Bus IO operation is displayed to remind you what will get executed. The FBSTATUS command displays the current BBFC Fast Bus Status Register for a specific crate. The SERVICE command displays current BBFC Service Request for a specific crate. The RESET command executes a Fast Bus reset bus (FBRB). 10.2 DEVICE LIST IO SUBMENU The DEVICE LIST IO submenu is probably the most useful submenu. You can execute the same device list your acquisition program executes. The CREATE command creates a device list for you through prompts. SAVE writes it out to a file. The RETRIEVE command inputs a previously created device list file. The current device list can be listed on your terminal with the DISPLAY command. The PUT command provides options of looping and dumping read buffers on your terminal. PUT, in this submode only, has the option to compare all read and write buffers against each other. The XLIST command executes the device list. 10.3 REGISTER IO SUBMENU The REGISTER IO submenu read/writes the VBBC hardware registers and is best left to the computer staff for low level debugging. 10.4 MISCELLANEOUS PUT COMMAND OPTIONS The PUT command allows you to select DMA IO or Programmed IO for all subsequent Fast Bus or Device List interface executions. Only the VBDMA hardware allows this mode change. The mode must also be set physically in the eight "mode select" dip switches on the front panel of the VBBC's modified daughter board. There are two "sticky" PUT options that change computer operations even after the diagnostic program is terminated. The driver debug variable can be set to write specific driver errors to the console and the operator log file (/usr/var/adm/SYSLOG). Also the driver transfer block size can be varied from 4 to 262,140 bytes long. Large IO transfers are broken down into blocks. The larger the block size, the faster the transfer. 11.0 MISCELLANEOUS INTERFACE LIBRARY ROUTINES Here are useful routines not include in the layer 3 interface routines. All routines require the previous execution of the proper open routine. By the way, all these routines can be executed from within the Diagnostic Program, bbdiag.c, using the Fast Bus IO submenu's GET or PUT commands. 11.1 LAYER 1 INTERFACE LIBRARY ROUTINES Layer 1 routines are not described in this document because the user is not expected to use them directly. They can be found in a program listing of bblib.c. However, two layer 1 routines are described here for those interested in varying the physical block size of data transmissions. GET TRANSFER BLOCK SIZE IN BYTES This routine places the Branch Bus driver's physical block size, in bytes, into the unsigned integer 'xfer'. The returned function value is zero on a normal return and nonzero on an error. bbgetxfer(&xfer); SET TRANSFER BLOCK SIZE IN BYTES This routine sets a new value for the Branch Bus driver's physical block size, in bytes. The new value is contained in the unsigned integer variable 'xfer'. The returned function value is zero on a normal return and nonzero on an error. bbsetxfer(xfer); 11.2 NONCONFORMING INTERFACE LIBRARY ROUTINES These routines do not conform to the layered structure of the interface library but are presented here for their usefulness. GET BBFC SERVICE REQUEST This routine inputs the current BBFC Fast Bus Service Request setting. The user provides the BBFC's crate number in the unsigned integer 'crate' and the routine places the contents of the Fast Bus Service Request into the unsigned integer 'service'. The returned function value is zero on a normal return and nonzero on an error. call frsr(crate,service) Fortran call get_bbfc_service(crate,&service); C call GET BBFC FAST BUS STATUS REGISTER This routine inputs the current BBFC Fast Bus status register. The user provides the BBFC's crate number in the unsigned integer 'crate' and the routine places the contents of the BBFC Fast Bus status register into the unsigned integer 'status'. The returned function value is zero on a normal return and nonzero on an error. call frfb(crate,status) Fortran call get_bbfc_fbstat(crate,&status); C call RESET FAST BUS STATUS CRATE (FBRB) This routine resets Fast Bus at the specified crate. The user provides the BBFC's crate number in the unsigned integer 'crate'. The returned function value is zero on a normal return and nonzero on an error. call fwrb(crate) Fortran call set_bbfc_fbrb(crate); C call 12.0 DESIGN DECISIONS The project limited itself to commercially available hardware. If necessary, only slight modifications were allowed. Only the BBFC is totally new, but even that was limited to look like a BVI on the Branch Bus. Because the computer communicates with a Fast Bus Module over three buses, (VMEBUS-->Branch Bus-->Fast bus), control is difficult. Plus the VBBC provides only 8 control bits, 5 used for Primary Address, restricting the passage of control information. There is no implementation of Fast Bus Next Transfer Address (NTA), broadcast, and set arbitration level. Efforts failed in finding ways of scanning for active VMEBUS/FAST-Bus crates, scanning for active VMEBUS/FAST-Bus modules, determining setting of hardware mode switches (DMA/Programmed IO). Some features of the Branch Bus driver software such as scatter-gather are not carried up into the higher layers of the software interface. The original driver design uses "polling" rather than "interrupts" to determine VBBC IO completion. This design was kept. This locks up the computer during IO but for very short periods which is not too perceivable to others time sharing the computer. Some limitations of the Branch Bus driver software were kept to avoid breaking code that works. No interrupts are used. All hardware event completions are determined by testing status rather than waiting for an interrupt. A very small but useable subset of the ANSI/IEEE Fast Bus Standards Routines is implemented. Only 32-bit VMEBUS addressing and transferring is allowed. By default, driver functions open(), close(), Ioctl(), etc, are forced onto processor 0 on a multiprocessor (MP) system. Therefore, a device driver written for a single processor will work unmodified on an MP system. To avoid context switches to processor 0 for every IO call, you can modify a device driver to run on any processor by semaphoring a driver. This driver is semaphored, so you can make this an MP driver by adding flag D_MP to the bb_devflag driver global. Both devices, /dev/bb0 for DMA and /dev/bb1 for Programmed-IO, are sharable, so any number of processes can use them at the same time. Of course, only one mode ( DMA or PIO) is active at any one time because interface switches must be manually set to one of these modes. 13.0 REFERENCES o VME/Branch Bus Controller VBBC manual and schematics o Branch Bus/VMEBUS Interface BVI manual and schematics o VME/Branch Bus Terminator manual and schematics o Branch Bus Specification manual and schematics o Branch Bus/Fast Bus Controller BBFC schematics o VMEBUS/Branch Bus DMA VBDMA schematics o UNIX SYSTEM V RELEASE 4 Device Driver Interface/Driver-Kernel Interface Reference manual for Intel Processors (includes Multiprocessing). Also referred to as "the Red Book". o IRIX Device Driver Programming Guide o IRIX Device Driver Reference Pages manual o IEEE Standard FASTBUS Modular High-Speed Data Acquisition and Control System and FASTBUS IEEE Standard Routines manual o BNL SSP Data Acquisition System manuals o This document file (bbsgi.mem) and all software described in this document is found in /usr/var/sysgen/bb. APPENDIX A.0 - HARDWARE REGISTER DEFINITIONS ____________________________________________________________________________ ##VBBC CONTROL WORD 1## 31 28 24 20 16 12 08 04 00 | | | | | | | | | ----------------------------------------------------------------- | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | ----------------------------------------------------------------- |Crate--------->|Control Bits-->| Word Count------------------->| ^ ^ ^ ^ ^ ^ ^ ^ ##BBFC Fast Bus## | | | | | | | | | | | (20-16)Fast Bus Primary Address | | (21)Control Space* (23-22)Encoded: 1=BBFC Fast Bus Status Register Read* 2=Fast Bus Single Word Transfer 3=Fast Bus Service Request Read ^ ^ ^ ^ ^ ^ ^ ^ ##BVI VMEBUS## | | | | | | | | | | | | | | | (16)Reserved | | | | | | (17)BVI Performs VME Reset Before Transfer | | | | | (18)VME Broadcast Transfer | | | | (19)VME Standard 24-bit Addressing | | | (20)VME Single Word Transfer | | (21)VME 16-bit Transfer | (22)Reserved (23)Reserved * When Control Space and BBFC Fast Bus Status Register Read are set, the BBFC executes a Fast Bus reset bus (FBRB) and reads the Fast Bus register. There is not enough control bits for a reset so this kludge was necessary. ____________________________________________________________________________ ##VBBC CONTROL WORD 2## 31 28 24 20 16 12 08 04 00 | | | | | | | | | ----------------------------------------------------------------- | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | ----------------------------------------------------------------- |VMEBUS Address or Fast Bus Secondary Address------------------>| __________________________________________________________________________.0 ##VBBC CONTROL CSR (CONTROL AND STATUS REGISTER)## 31 28 24 20 16 12 08 04 00 | | | | | | | | | ----------------------------------------------------------------- | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | ----------------------------------------------------------------- ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^|Unused------------------------>| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (16)M16: 16 Bit Mode | | | | | | | | | | | | | | (17)OBT: Output Block Transfer | | | | | | | | | | | | | (18)IBT: Input Block Transfer | | | | | | | | | | | | (19)WS: Word Swap | | | | | | | | | | | (20)BS: Byte Swap | | | | | | | | | | (21)Unused | | | | | | | | | (22)RES: Reset VBBC and any connected BVI/BBFCs | | | | | | | | (23)BSY: Busy | | | | | | | (24)ILO: Illegal Operation | | | | | | (25)IDP: Input Data Present | | | | | (26)IPE Input Parity Error | | | | (27)IOF: Input Overflow | | | (28)SC: Slave Connected | | (29)BBE: Branch Bus Error | (30)BG: Bus Granted (31)AF: Arbitration Failure ____________________________________________________________________________ ##BBFC FAST BUS STATUS REGISTER## 31 28 24 20 16 12 08 04 00 | | | | | | | | | ----------------------------------------------------------------- | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | ----------------------------------------------------------------- |Words not transferred--------->|Unused------>|^ ^ ^ ^ ^ ^| ^ | | | | | | | | | | | | | | | (08)No Crate select | | | | | | (07)Parity Error | | | | | (06)Overflow | | | | (05)Data Time Out | | | (04)Address Time Out | | (03)Arbitration Time Out | (02-00)Slave Status APPENDIX B.0 - BRANCH BUS OBJECT LIST (LAYER 2) INTERFACE For those who wish to create their own Branch Bus interface library, here are the layer 2 routines you build it with. These routines deal with a data structure called an bb_operation object list defined in vbbcreg.h. An object is a Fast Bus or VMEBUS IO operation. More than one object is called an object list. This object list provides the Branch Bus device driver with all the information needed to perform IO. OPEN Call open routine before calling any other routine. Device name "/dev/bb0" is DMA mode. Device name "/dev/bb1" is Programmed IO mode. olopen(devicename) CLOSE Call close routine when use of Branch Bus no longer required. olclose() INITIALIZE OBJECT LIST Call initialize routine when creating first object list or whenever you to erase the current list in preparation for a replacement list. olinit() ADD FAST BUS OBJECT TO OBJECT LIST This routine adds a Fast Bus device IO operation to the Object list. oladdfb(mode,crate,pa,sa,space,bufaddr,count,sl) Where mode: "r" for read or "w" for write crate: unsigned integer crate number pa: unsigned integer Primary Address sa: unsigned integer Secondary Address space: "d" for data space, "c" for control space, "ds" for data space single-word "cs" for control space single-word bufaddr: read/write buffer count: byte count sl: bb_operation_status structure address for returned status info ADD FAST BUS OBJECT TO OBJECT LIST This routine adds a Fast Bus device IO operation to the Object list. oladdvme(mode,crate,vmeaddr,bufaddr,count,sl) Where mode: "r" for read or "w" for write crate: unsigned integer crate number vmeaddr: unsigned integer VMEBUS Address bufaddr: read/write buffer count: byte count sl: bb_operation_status structure address for returned status info EXECUTE OBJECT LIST This routine executes the object list build by the oladdfb() and oladdvme() routines. olexec() TRANSLATE ERROR CODE This routine translates the bb_operation_status error code, errnum, and returns it in string variable, errmessp. oltran(errnum,errmessp) APPENDIX C.0 - ERROR CODES Normally, error codes are translated into error messages by interface routines. This section describes the codes for those who wish to create their own messages. Error codes are decimal numbers grouped together according to where the error occurred. Errors occurring in the driver and layer 1 routines span 0 to 999. Layer 2 errors span 1001 to 1999. Layer 2 errors span 2001 to 2999. ERROR ERROR MEANING CODE SYMBOL ----- ----------------- ---------------------------------------------------- 0 FENORMAL No error BBERR_OK 1 BBERR_ARBITRATION Arbitration failure on VBBC 2 BBERR_CSR An error bit set in VBBC's CSR Register 3 BBERR_NOVBBC VBBC is missing from local VMEBUS 4 BBERR_ODDBYTE Odd byte transfer attempted 5 BBERR_TOOSHORT Supplied buffer was too short 6 BBERR_VBBCBSY VBBC's Busy Bit (BSY) is jammed on 7 BBERR_COPYFAULT Fault occurred while copying data in Programmed IO mode or VBBC's Slave Connect jammed-on in DMA mode 8 BBERR_NOSLAVE No slave for given crate address 9 BBERR_NOBG No bus grant was signalled 10 BBERR_LOCALACCESS Can't access local VME bus 11 BBERR_BLOCKOVF Too many segments locked 12 BBERR_LOCK Trouble locking page in memory 13 BBERR_LOSTBG Lost bus grant during transfer 14 BBERR_SPURIOUS1 Unexplainable problem during bus arbitration 15 BBERR_SPURIOUS2 Unexplainable problem during IO transfer 1001 OLERR_IOF VBBC Input Overflow error 1002 OLERR_IPE VBBC Input parity error 1003 OLERR_ILO VBBC Illegal Operation error 1004 OLERR_BBE VBBC Branch Bus error 1005 OLERR_AF VBBC Arbitration failure error 1006 OLERR_RES VBBC Reset set error 1007 OLERR_ARBTO BBFC Arbitration time out 1008 OLERR_ASTO BBFC Address time out 1009 OLERR_DSTO BBFC Data time out 1010 OLERR_OVFL BBFC Overflow 1011 OLERR_PE BBFC Parity error 1012 OLERR_NOCR No such Fast Bus crate 1013 OLERR_SS1 BBFC Slave Status = 1; Busy 1014 OLERR_SS2 BBFC Slave Status = 2; End of block 1015 OLERR_SS3 BBFC Slave Status = 3; Reserved 1016 OLERR_SS4 BBFC Slave Status = 4; Reserved 1017 OLERR_SS5 BBFC Slave Status = 5; User defined 1018 OLERR_SS6 BBFC Slave Status = 6; Error - invalid operation 1019 OLERR_SS7 BBFC Slave Status = 7; Error 2002 FEHERR Implementation dependent hardware error (IEEE FB) 2003 FEERR The function was not successful (IEEE Fast Bus) 2004 FENMST No more Supplementary Status (IEEE Fast Bus)