How-To: Configure an ATCA Short Rack Test Stand

Author: Claude Saunders
(saunders@aps.anl.gov)

This document is an ongoing guide on how to deploy a basic collection of software on an ATCA starter kit and companion rack-mount server. The guide is based on a particular collection of hardware and software, specified below, however much of this probably applies to a number of possible variations.

Please also go see Mike Haney and Mike Kasten's guide to the Arrow ATCA Starter Kit. This will provide much needed details on the basic capabilities, and how to get going with things like the various console ports.
http://docdb.fnal.gov/ILC-public/DocDB/ShowDocument?docid=313

The basic setup consists of a rack-mount server monitoring and managing the ATCA shelf. The rack-mount server runs the open-source OpenNMS network management system to monitor all the various hardware and software components. The rack-mount server also becomes the "System Controller" node for OpenClovis. The 2 processor blades of the ATCA shelf become OpenClovis "payload" nodes, and run both the EPICS soft IOC and EPICS PV Gateway.

The EPICS soft IOC and PV Gateway were selected as representative examples of software we would expect to be running on ATCA. As actual ATCA/uTCA analog I/O boards become available, the drivers and real-time databases available in EPICS could be used to control them. Aside from actually controlling hardware, another R&D goal is to "instrument" these EPICS software components with OpenClovis and explore the monitoring, manageability, and high-availability that SAF implementations can bring to the table.

Software Components

The following are the software components covered in this guide:

The choice of operating systems (CentOS and FC6) was largely driven by the need to have a supported environment for running the OpenClovis eval kit out-of-the-box. Other distributions and kernel versions are probably also acceptable. We did recently encounter a problem with FC6. An earlier release of FC6 had the 2.6.18 kernel, which worked fine with OpenClovis. A subsequent update brought the kernel level to 2.6.20, which did not allow OpenClovis to compile. Evidently there are some pretty finicky kernel dependencies in OpenClovis. If you have trouble with OpenClovis compiling on FC6, I do know that kernel 2.6.18 works, so you may need to upgrade/downgrade whatever level of FC6 you get at the time.

Hardware Components

The following is the assumed hardware configuration. Note that several variations are possible, so this is just one suggestion.

The hardware is inter-connected in the following fasion, with the assigned IP addresses in parentheses:

Windows-PC (10.90.91.10/24)
  |
  |
Centre-Com Switch
  |
  |
 eth1 (10.90.91.41/24)
HP Pro-Liant Rack-Server
 eth0 (10.90.90.40/24)
  |
  |
 zre2
Zynx ZX5000 ATCA Switch
 zre0   zre3   zre4
  |      |      |
  |      |      |
 eth0    |      | (10.90.90.50/24)
ShelfMgr |      |
         |      |
         |      |
       eth0   eth0 (10.90.90.122/24) (10.90.90.222/24)
      blade0  blade1

Installation Guides

Note that most of this installation was done off-the-network, due to restrictions onsite regarding putting equipment directly on the network. So some of the instructions could be simplified by using things like apt-get or yum.

Fedora Core 6 on HP Pro-Liant Rack-Server

  1. Simply download and burn the installation ISO's to CD and follow instructions.
  2. You will need root access for some of the subsequent installation work.
  3. Note - We've only verified that FC6 with the 2.6.18 kernel works properly with OpenClovis. Do NOT update your installation if you have 2.6.18, otherwise you will get upgraded to 2.6.20+, which apparently does not work.

OpenNMS 1.3.X on HP Pro-Liant Rack Server

  1. Go to www.opennms.org and follow the Install Guide for 1.3.x.
  2. Summarized here, you must install a number of prerequisites using your favorite RPM install technique.
  3. Install Java jdk 1.5 from java.sun.com
  4. Install Tomcat5 (note: has many dependencies on other RPM's so will pull in many things)
  5. Install Postgres 8.1.4
  6. Install rrdtool 1.2
  7. Install opennms-1.3.3-1_fc6.i386.rpm
  8. Install opennms-webapp-1.3.3-1_fc6.i386.rpm
  9. Install opennms-docs-1.3.3-1_fc6.i386.rpm
  10. Define OPENNMS_HOME=/opt/opennms (which is default install location)
  11. Edit /opt/opennms/etc/discovery-configuration.xml, adding IP address range of 10.90.90.1 to 10.90.90.254.
  12. Stop and start OpenNMS using "/etc/init.d/opennms stop" and "/etc/init.d/opennms start".
  13. Hit http://host:8080 to get initial OpenNMS page. Login with admin/admin. Note that it takes a while for initial discovery of all network nodes and capabilities.
  14. Follow the various guides for further configuration.

CentOS 4.5 on ATCA Intel CPU blades

  1. Find yourself a USB CD drive, as doing this with a USB Hard Drive was a pain in the rear.
  2. Get CentOS 4.5 ISO's from www.centos.org and burn to CD's.
  3. Plug in drive on blade, then reset it. Get to BIOS by repeatedly hitting F4 during boot.
  4. Via console port (9600 8-N-1), set BIOS to boot from CD.
  5. When CentOS linux installer boots, make sure to start install with boot: linux console=ttyS0,9600 text
  6. Pick the development install to get gcc, etc.. Otherwise pick Server and add those RPMs manually later.
  7. Name the blades blade0 and blade1 (in keeping with rest of this document).
  8. Do NOT install the firewall and disable SELinux

OpenClovis 2.2 OSS1 on HP Pro-Liant Rack Server

Note that I decided to deploy OpenClovis on the FC6 rack server, and cross-build for the CentOS4.5 ATCA blades. OpenClovis makefiles support this explicitly. Another option would have been to also deploy CentOS4.5 on the rack server, but I wasn't sure all the other software packages needed on there had RPM's for CentOS.

  1. Go to www.openclovis.org and download (you have to register now):
  2. Expand doc tarball, and follow InstallationGuide. Summarized and simplified here:
  3. Expand main tarball with "tar zxf openclovis-sdk-gpl-2.2.OSS1.tgz"
  4. Become root if you are not already, and run the preinstall-fedora-core-6.sh.
  5. Then run the install.sh.
  6. Add /sdk-2.2/bin to your PATH environment variable.
  7. You can now follow EvaluationGuide from doc directory. Summarized and simplified here:
  8. run cl-create-project-area to build an intial project-area. Select the eval model to install.
  9. cd to SISP directory of your project-area, and run ./configure, then ./make.
  10. This builds your local binaries. Now we need to cross-build target images.
  11. Type "./configure --with-cross-build=ia32-linux-rhel-2.6.9 --with-kernel-build=linux-2.6.9-5.EL-smp-i686"
  12. Edit /SISP/models/eval/target.conf. Below is what I used:
    • TRAP_IP=10.90.90.40 (the IP of rack server)
    • CMM_IP=10.90.90.50 (the IP the PigeonPoint Shelf Manager comes with, I think)
    • LOG_ADDR=10.90.90.40
    • IP_SCNodeI0=10.90.90.40 (rack server will be system controller)
    • IP_PayloadNodeI0=10.90.90.122 (blade0)
    • IP_PayloadNodeI1=10.90.90.222 (blade1)
    • SLOT_SCNodeI0=1
    • SLOT_PayloadNodeI0=3
    • SLOT_PayloadNodeI1=4
  13. Go back to /SISP and type "make images".
  14. You now have 3 installation tarballs:
  15. On rack server (as root), create /root/sisp directory and untar SCNodeI0.tgz in it.
  16. Copy PayloadNodeI0.tgz to blade0, and do the same as root on that host.
  17. Copy PayloadNodeI1.tgz to blade1, and do the same as root on that host.
  18. We are almost ready to proceed with OpenClovis evaluation examples. But first we must tweak a kernel parameter on all 3 hosts. On each of 3 hosts, add these two lines to /etc/sysctl.conf Then run "sysctl -p" to load the change (or just reboot).
  19. Now just follow OpenClovis_EvaluationGuide-2.2.pdf examples starting in Section 5.
  20. Note that you need to boot your CentOS4.5 blades in NON-smp mode. Getting OpenClovis to run on SMP targets requires a more tedious kernel-specific build of their kernel modules, which I have not got around to yet.
  21. Note also that you must run "iptables -F" on rack server to get rid of firewall rules that will block UDP packets from OpenClovis.

EPICS 3.14.9

  1. Go to www.aps.anl.gov/epics
  2. Get base 3.14.9 tarball from epics/base/R3-14/9.php page.
  3. Repeat all the following for each host (rack server, blade0, blade1). Note you can just build once on blade0 and copy executables to blade1. But rack server executables will not run properly on blades.
  4. Make an "epics" directory off your home directory.
  5. Untar all 4 tarballs above in epics directory.
  6. Set EPICS_HOST_ARCH=linux-x86
  7. Set EPICS_BASE=$HOME/epics/base3.14.9
  8. cd epics/base, then "make"
  9. mkdir $HOME/epicstop, cd $HOME/epicstop
  10. $EPICS_BASE/bin/linux-x86/makeBaseApp.pl -t example myexample
  11. $EPICS_BASE/bin/linux-x86/makeBaseApp.pl -i -t example myexample
  12. cd $HOME/epicstop, then "make"
  13. cd iocBoot/iocexample
  14. chmod +x st.cmd
  15. ./st.cmd - that's it, you are now running a soft IOC with a sample database.
  16. Type "dbl" in the IOC shell to see a list of process variables.
  17. In another window, go to $EPICS_BASE/bin/linux-x86
  18. Type "caget ". This does a get on the process variable value.
  19. Type "exit" from IOC shell to quit.

EPICS PV Gateway

  1. Go to www.aps.anl.gov/epics
  2. Get gateway and gnuregex tarballs from epics/extensions/gateway/index.php page.
  3. Get extensions build configure tarball from epics/extensions/configure/index.php page.
  4. Expand these in $HOME/epics directory.
  5. cd epics/extensions/configure
  6. Edit RELEASE file, giving full path for EPICS_BASE attribute.
  7. Type "make" in configure directory to build makefile framework.
  8. cd epics/extensions/src/gnuregex (we need to fix some out-of-date makefiles)
  9. cp Makefile.Host Makefile
  10. Edit Makefile, and change the following:
  11. make
  12. cd epics/extensions/src/gateway
  13. make
  14. cd $HOME/epics/extensions/bin/linux-x86
  15. ./gateway -help
  16. See gateway users guide for more info.