How-To: Configure an ATCA Short Rack Test Stand
Author: Claude Saunders
(saunders@aps.anl.gov)
This document is an ongoing guide on how to deploy a basic collection of software
on an ATCA starter kit and companion rack-mount server. The guide is based on a
particular collection of hardware and software, specified below, however much of
this probably applies to a number of possible variations.
Please also go see Mike Haney and Mike Kasten's guide to the Arrow ATCA Starter
Kit. This will provide much needed details on the basic capabilities, and how
to get going with things like the various console ports.
http://docdb.fnal.gov/ILC-public/DocDB/ShowDocument?docid=313
The basic setup consists of a rack-mount server monitoring and managing the
ATCA shelf. The rack-mount server
runs the open-source OpenNMS network management system to monitor all the
various hardware and software components. The rack-mount server also becomes
the "System Controller" node for OpenClovis. The 2 processor blades of the
ATCA shelf become OpenClovis "payload" nodes, and run both the EPICS soft IOC
and EPICS PV Gateway.
The EPICS soft IOC and PV Gateway were selected as representative examples of
software we would expect to be running on ATCA. As actual ATCA/uTCA analog I/O boards
become available, the drivers and real-time databases available in EPICS could be
used to control them. Aside from actually controlling hardware, another R&D goal
is to "instrument" these EPICS software components with OpenClovis and explore the
monitoring, manageability, and high-availability that SAF implementations can bring
to the table.
Software Components
The following are the software components covered in this guide:
- CentOS 4.5 (A free RHEL4 equivalent) Linux
- Fedora Core 6
- OpenNMS 1.3.X
- Java 1.5
- Tomcat5
- Postgres 8.1.4
- rrdtool 1.2
- OpenClovis 2.2 OSS1
- EPICS Base 3.14.9
- EPICS PV Gateway 2.0
The choice of operating systems (CentOS and FC6) was largely driven by the need
to have a supported environment for running the OpenClovis eval kit out-of-the-box.
Other distributions and kernel versions are probably also acceptable. We did recently
encounter a problem with FC6. An earlier release of FC6 had the 2.6.18 kernel, which
worked fine with OpenClovis. A subsequent update brought the kernel level to 2.6.20,
which did not allow OpenClovis to compile. Evidently there are some pretty finicky
kernel dependencies in OpenClovis. If you have trouble with OpenClovis compiling on
FC6, I do know that kernel 2.6.18 works, so you may need to upgrade/downgrade whatever
level of FC6 you get at the time.
Hardware Components
The following is the assumed hardware configuration. Note that several variations
are possible, so this is just one suggestion.
- Laptop
- HP ProLiant DL140 1U Server (running Fedora Core 6)
- CentreCom Ethernet Switch (any cheap switch)
- Arrow OCS ATCA Starter Kit
- Zynx ZX5000 Switch blade (Slot 1)
- Intel MPCBL0001 CPU blade (running CentOS 4.5) (Slot 3)
- Intel MPCBL0001 CPU blade (running CentOS 4.5) (Slot 4)
- Pigeon Point ShMM300 Shelf Manager (Slot 6)
The hardware is inter-connected in the following fasion, with the
assigned IP addresses in parentheses:
Windows-PC (10.90.91.10/24)
|
|
Centre-Com Switch
|
|
eth1 (10.90.91.41/24)
HP Pro-Liant Rack-Server
eth0 (10.90.90.40/24)
|
|
zre2
Zynx ZX5000 ATCA Switch
zre0 zre3 zre4
| | |
| | |
eth0 |   | (10.90.90.50/24)
ShelfMgr | |
| |
| |
eth0 eth0 (10.90.90.122/24) (10.90.90.222/24)
blade0 blade1
Installation Guides
Note that most of this installation was done off-the-network, due to restrictions onsite
regarding putting equipment directly on the network. So some of the instructions could be
simplified by using things like apt-get or yum.
Fedora Core 6 on HP Pro-Liant Rack-Server
- Simply download and burn the installation ISO's to CD and follow instructions.
- You will need root access for some of the subsequent installation work.
- Note - We've only verified that FC6 with the 2.6.18 kernel works properly with OpenClovis. Do NOT update
your installation if you have 2.6.18, otherwise you will get upgraded to 2.6.20+, which apparently does not work.
OpenNMS 1.3.X on HP Pro-Liant Rack Server
- Go to www.opennms.org and follow the Install Guide for 1.3.x.
- Summarized here, you must install a number of prerequisites using your favorite RPM install technique.
- Install Java jdk 1.5 from java.sun.com
- Install Tomcat5 (note: has many dependencies on other RPM's so will pull in many things)
- Install Postgres 8.1.4
- Install rrdtool 1.2
- Install opennms-1.3.3-1_fc6.i386.rpm
- Install opennms-webapp-1.3.3-1_fc6.i386.rpm
- Install opennms-docs-1.3.3-1_fc6.i386.rpm
- Define OPENNMS_HOME=/opt/opennms (which is default install location)
- Edit /opt/opennms/etc/discovery-configuration.xml, adding IP address range of 10.90.90.1 to 10.90.90.254.
- Stop and start OpenNMS using "/etc/init.d/opennms stop" and "/etc/init.d/opennms start".
- Hit http://host:8080 to get initial OpenNMS page. Login with admin/admin. Note that it takes
a while for initial discovery of all network nodes and capabilities.
- Follow the various guides for further configuration.
CentOS 4.5 on ATCA Intel CPU blades
- Find yourself a USB CD drive, as doing this with a USB Hard Drive was a pain in the rear.
- Get CentOS 4.5 ISO's from www.centos.org and burn to CD's.
- Plug in drive on blade, then reset it. Get to BIOS by repeatedly hitting F4 during boot.
- Via console port (9600 8-N-1), set BIOS to boot from CD.
- When CentOS linux installer boots, make sure to start install with boot: linux console=ttyS0,9600 text
- Pick the development install to get gcc, etc.. Otherwise pick Server and add those RPMs manually later.
- Name the blades blade0 and blade1 (in keeping with rest of this document).
- Do NOT install the firewall and disable SELinux
OpenClovis 2.2 OSS1 on HP Pro-Liant Rack Server
Note that I decided to deploy OpenClovis on the FC6 rack server, and cross-build for
the CentOS4.5 ATCA blades. OpenClovis makefiles support this explicitly. Another
option would have been to also deploy CentOS4.5 on the rack server, but I wasn't
sure all the other software packages needed on there had RPM's for CentOS.
- Go to www.openclovis.org and download (you have to register now):
- openclovis-sdk-gpl-2.2.OSS1.tgz
- openclovis-sdk-doc-2.2-oss1.tar.gz
- 3rdparty-base-oss-1.1.tar
- crossbuild-ia32-linux-rhel4-2.6.9-1.0-oss1.tar (note: gzip this after you get it)
- Expand doc tarball, and follow InstallationGuide. Summarized and simplified here:
- Expand main tarball with "tar zxf openclovis-sdk-gpl-2.2.OSS1.tgz"
- Become root if you are not already, and run the preinstall-fedora-core-6.sh.
- Then run the install.sh.
- Add /sdk-2.2/bin to your PATH environment variable.
- You can now follow EvaluationGuide from doc directory. Summarized and simplified here:
- run cl-create-project-area to build an intial project-area. Select the eval model to install.
- cd to SISP directory of your project-area, and run ./configure, then ./make.
- This builds your local binaries. Now we need to cross-build target images.
- Type "./configure --with-cross-build=ia32-linux-rhel-2.6.9 --with-kernel-build=linux-2.6.9-5.EL-smp-i686"
- Edit /SISP/models/eval/target.conf. Below is what I used:
- TRAP_IP=10.90.90.40 (the IP of rack server)
- CMM_IP=10.90.90.50 (the IP the PigeonPoint Shelf Manager comes with, I think)
- LOG_ADDR=10.90.90.40
- IP_SCNodeI0=10.90.90.40 (rack server will be system controller)
- IP_PayloadNodeI0=10.90.90.122 (blade0)
- IP_PayloadNodeI1=10.90.90.222 (blade1)
- SLOT_SCNodeI0=1
- SLOT_PayloadNodeI0=3
- SLOT_PayloadNodeI1=4
- Go back to /SISP and type "make images".
- You now have 3 installation tarballs:
- /target/eval/images/SCNodeI0.tgz (for rack server)
- /target/eval/images/PayloadNodeI0.tgz (for blade0)
- /target/eval/images/PayloadNodeI1.tgz (for blade1)
- On rack server (as root), create /root/sisp directory and untar SCNodeI0.tgz in it.
- Copy PayloadNodeI0.tgz to blade0, and do the same as root on that host.
- Copy PayloadNodeI1.tgz to blade1, and do the same as root on that host.
- We are almost ready to proceed with OpenClovis evaluation examples. But first
we must tweak a kernel parameter on all 3 hosts. On each of 3 hosts, add these
two lines to /etc/sysctl.conf
- net.ipv4.neigh.eth0.unres_qlen = 50
- net.ipv4.neigh.default.unres_qlen = 50
Then run "sysctl -p" to load the change (or just reboot).
- Now just follow OpenClovis_EvaluationGuide-2.2.pdf examples starting in Section 5.
- Note that you need to boot your CentOS4.5 blades in NON-smp mode. Getting OpenClovis
to run on SMP targets requires a more tedious kernel-specific build of their kernel
modules, which I have not got around to yet.
- Note also that you must run "iptables -F" on rack server to get rid of firewall rules
that will block UDP packets from OpenClovis.
EPICS 3.14.9
- Go to www.aps.anl.gov/epics
- Get base 3.14.9 tarball from epics/base/R3-14/9.php page.
- Repeat all the following for each host (rack server, blade0, blade1). Note you can just
build once on blade0 and copy executables to blade1. But rack server executables will
not run properly on blades.
- Make an "epics" directory off your home directory.
- Untar all 4 tarballs above in epics directory.
- Set EPICS_HOST_ARCH=linux-x86
- Set EPICS_BASE=$HOME/epics/base3.14.9
- cd epics/base, then "make"
- mkdir $HOME/epicstop, cd $HOME/epicstop
- $EPICS_BASE/bin/linux-x86/makeBaseApp.pl -t example myexample
- $EPICS_BASE/bin/linux-x86/makeBaseApp.pl -i -t example myexample
- cd $HOME/epicstop, then "make"
- cd iocBoot/iocexample
- chmod +x st.cmd
- ./st.cmd - that's it, you are now running a soft IOC with a sample database.
- Type "dbl" in the IOC shell to see a list of process variables.
- In another window, go to $EPICS_BASE/bin/linux-x86
- Type "caget ". This does a get on the process variable value.
- Type "exit" from IOC shell to quit.
EPICS PV Gateway
- Go to www.aps.anl.gov/epics
- Get gateway and gnuregex tarballs from epics/extensions/gateway/index.php page.
- Get extensions build configure tarball from epics/extensions/configure/index.php page.
- Expand these in $HOME/epics directory.
- cd epics/extensions/configure
- Edit RELEASE file, giving full path for EPICS_BASE attribute.
- Type "make" in configure directory to build makefile framework.
- cd epics/extensions/src/gnuregex (we need to fix some out-of-date makefiles)
- cp Makefile.Host Makefile
- Edit Makefile, and change the following:
- TOP=../../.. to TOP=../..
- include $(TOP)/config/CONFIG_EXTENSIONS to include $(TOP)/configure/CONFIG
- include $(TOP)/config/Rules.Host to include $(TOP)/configure/RULES
- make
- cd epics/extensions/src/gateway
- make
- cd $HOME/epics/extensions/bin/linux-x86
- ./gateway -help
- See gateway users guide for more info.