This section has a detailed description of all the components and procedures involved in Beaker Test Environment.
3.2.1.1. Submitting and Reviewing a Job Workflow
To submit a Job you must create Job Workflow. This is an XML file containing the tasks you want to run,as well as special environment variables and other options you want to setup for your Job.
3.2.1.2. Provisioning a system
If you would like to use one of these System you will need to provision it. Provisioning a System means to have the system loaded with an Operating System and reserved for the user. There are a couple of ways of doing this, which are outlined below.
3.2.1.2.1. Provision by System
Go to the System details page (see Section 5.1.3, “System Details Tabs”) of a System that is free (see Section 5.1.1, “System Searching”) and click on Take in the Current User field. After successfully taking the System, click the Provision tab of the System details page to provision the System.
Returning a System
After provisioning a System, you can manually return it by going to the above mentioned System details page, clicking on the Return link in the Current User field.
3.2.1.2.2. Provision by Distro
Go to the Distro search page(Section 5.2.1, “Distro Searching”) and search for a Distro you would like to provision onto a System. Once you have found the Distro you require, click Provision System, which is located in the far right column of your search results. If the Provision System link is not there, it's because there is no suitable System available to use with that Distro.
The resulting page lists the Systems you can use. Systems with Reserve Nowin the far right column mean that no on else is using them and you can take them, otherwise you will see Queue ReservationQueue Reservation; which means that someone is currently using the System but you can be appended to the queue of people wanting to use this System.
After choosing your System and clicking on the the aforementioned links, you will be presented with a form with the following fields:
System To Provision This is our System we will provision.
Distro To Provision The Distro we will be installing on the System.
Job Whiteboard This is a reference that will be displayed in Jobs list. You can enter anything in here, however it cannot be changed later.
KickStartMetaData Arguments passed to the KickStart script.
Kernel Options (install)
Kernel Options(Post)
Pressing the Queue Job button will submit this provisioning as a Job and redirect us to the details of the newly created Job.
3.2.1.2.3. Reserve Workflow
The Reserve Workflow page is accessed from the top menu by going to
Scheduler>Reserve. The Reserve Workflow process allows the ability to select which System and Distro is to be provisioned based on the following:
ArchArchitecture of the System we want to provision.
Distro FamilyThe family of Distro we want installed.
MethodHow we want the distro to be installed.
TagThe Distro's tag.
DistroBased on the above refinements we will be presented with a list of Distro's available to be installed.
Selecting values for the above items should be done in a top to bottom fashion, staring at Arch and ending with Distro.
Once the Distro to be installed is selected you have the option of showing a list of System's that you are able to provision (Show Systems button), or you can have Beaker automatically pick a system for you (Auto pick System). If you choose Show Systems you will be presented with a list of Systems you are able to provision. Ones that are available now show the link Reserve now beside them. This indicates the System is available to be provisioned immediately. If the System is currenty in use it will have the link Queue Reservation instead. This indicates that the System is currently in use, but can be provisioned for a later time.
Whether you choose to automatically pick a system or to pick one yourself, you will be presented with a page that asks you for the following options:
Once you are ready you can provision your System with your selected Distro by pressing Queue Job.
Beaker provides an inventory of Systems(These could be a physical machine,laptop,virtual guest, or resource) attached to lab controllers. Systems can be added, removed, have details changed, and be provisioned amongst other things.
3.2.1.3.1.1. System Searching
System searches are conducted by clicking on one of the items of the System menu at the top of the page.
The search panel has two modes; simple and advanced. The simple search is the default, and the default search is of the System Name, using the contains operator. To toggle between the two search modes, press the Toggle Search link.
The first column (Table) is the attribute on which the search is being performed; The second (Operation) is the type of search, and the third (Value) is the actual value to search on. To add another search criteria (row), click the Add(+) link just below the Table column. When using more than just one search criteria, the defaul operation between the criterias is an SQL AND operation. The operators change depending on what type of attribute is being searched.
Wildcards
No operator provides explicit wildcards other than the is operation, which allows the * wildcard when searching an attribute which is a string.
The kind of data returned in the System search can be changed by adding/removing the result columns. To do this the Toggle Result Columns link is pressed and the columns checked/unchecked.
Shortcut for finding Systems you are using
The top right hand corner has a menu which starts with Hello, followed by the name of the user currently logged in. Click on this, then down to My Systems
3.2.1.3.1.2. Adding a System
To add a System, go to any System search page, and click on the Add(+) link on the bottom left. You must be logged in to do this. After filling in the details, press theSave Changes button on the bottom left hand corner.
3.2.1.3.1.3. System Details Tabs
After finding a System in the search page, clicking on the System name will show the System details. To change these details, you must be logged in as either the owner of the System, or an admin.
System Details
Install Options
History
Shows the activity on this System for the duration of the systems life as an inventory item in Beaker. These activites can also be searched. By default, the simple search does a contains search on the Field attribute. Please see
Section 3.2.1.3.1.1, “System Searching” for details on searching.
3.2.1.3.1.4. System Activity
To search through the historical activity of all Systems, navigate to Activity>All at the top of the page. The default search is contains on the Property attribute.
Individual System history
Beaker can keep a record of Distros that are available to install on Systems in its Inventory.
3.2.1.3.1.5.1. Distro Searching
The find a particular Distro, click Distros>All. The default search is on the Distro's Name, with a contains clause
The purpose of a Job is to provide an encapsulation of Tasks. It is to provide a single point of submission of these Tasks, and a single point of reviewing the output and results of these Tasks. The Tasks within a Job may or may not be related to each other; although it would make sense to define Jobs based on the relationship of the Tasks within it. Once a Job has been submitted you can not alter its contents, or pause it. You can however cancel it (Section 5.3.3.4, “Job Results”), and alter its Recipe Set's priorities (you can only lower the priority level if you are not in the admin group). Adjusting this priority upwards will change which Recipe Set is run sooner, and vice a versa.
3.2.1.3.1.6.1. Job Workflow
To create a simple Job workflow, see the bkr workflow-simpl command in Chapter 2, Beaker client
3.2.1.3.1.6.2. Job Searching
To search for a Job, navigate to Scheduler>Jobs at the top of the page. To look up the Job ID, enter a number in the search box and press the Lookup ID button. Please see Section 5.1.1, “System Searching” for details on searching.
Quick Searches
By pressing the Running,Queued, or Completed buttons you can quickly display Recipes that have a status of running,queued, and completed respectively.
3.2.1.3.1.6.3. Job Submission
There are two ways of submitting a Job through the web app.They are outlined below.
3.2.1.3.1.6.3.1. Submitting a New Job
Once you have created an XML Job workflow, you able able to submit it as a new Job. To do this, go to theScheduler > New Job. Click Browse to select your XML file, and then hit the Submit Data button. The next page shown gives you an opportunity to check/edit your XML before queueing it as a Job by pressing the Queue button.
3.2.1.3.1.6.3.2. Cloning an existing Job
Cloning a Job means to take a Job that has already been run on the System, and re-submit it. To do this you first need to be on the Job search page. See Section 5.3.2, “Job Searching”.
Clicking on Clone under the Action column will take you to a page that shows the structure of the Job in the XML.
Submitting a slightly different Job
If you want to submit a Job that's very similar to a Job already in Beaker,you can use the Clone button to change details of a previous Job and resubmit it!
3.2.1.3.1.6.3.3. Job workflow details
A direct child is the whiteboard. The content is normally a mnemonic piece of text describing the Job:
<job>
<whiteboard>
Apache 2.2 test
</whiteboard>
</ob>
The next tag in the recipeSet tag (which describes a Recipe Set. See Section 5.4, “Recipes” for details). A Job workflow can have one or morerecipeSet. All Recipes within a Recipe Set are run simultaneously, whereas multiple Recipe Sets are run in no predetermined order. This should help you decide whether you wish to run tasks in one or many Recipe Set (i.e Multihost tests will require no more than one Recipe Set).
<job>
<whiteboard>
Apache 2.2 test
</whiteboard>
<recipeset>
</recipeset>
</job>
Of course a
recipeSet element needs one or more recipe children. As mentioned above,
Recipes run simultaneously. The
recipeSet element can have the following attributes
kernel_options
kernel_options_post
ks_meta
roleIn a Multihost environment, it could be either SERVERS, CLIENT or STANDALONE. If it is not important, it can be None.
whiteboardText that describes the Recipe
Here is an example:
<job>
<whiteboard>
Apache 2.2 test
</whiteboard>
<recipeset>
<ecipe kernel_options="" kernel_options_post="" ks_meta="" role="None" whiteboard="Lab Controller">
</recipe>
</recipeset>
</job>
Avoid having many Recipes in one Recipe Set
Because Recipes are run simultaneously, not one Recipe will commence until all other sibling Recipes are ready. This involves each Recipe reserving a machine, and waiting until every other Recipe has reserved a machine. This can tie up resources and keep them idle for long amounts of time. Try having many Recipe Sets containing few Recipes, rather than the opposite. Of course this only applies to Recipes that do not need to be run simultaneously (i.e not Multihost Jobs)
Within the recipe tag, you can specify what packages need to be installed on top of anything that comes installed by default.
<job>
<whiteboard>
Apache 2.2 test
</whiteboard>
<recipeSet>
<recipe kernel_options="" kernel_options_post="" ks_meta="" role="None" whiteboard="Lab Controller">
<packages>
<package name="emacs"/>
<package name="vim-enhanced"/>
<package name="unifdef"/>
<package name="mysql-server"/>
<package name="MySQL-python"/>
<package name="python-twill"/>
</packages>
</recipe>
</recipeSet>
</job>
If you would like you can also specify your own repository that provides extra packages that your Job requires. Use the repo tag for this. You can use any text you like for the name attribute.
<job>
<whiteboard>
Apache 2.2 test
</whiteboard>
<recipeSet>
<recipe kernel_options="" kernel_options_post="" ks_meta="" role="None" whiteboard="Lab Controller">
<packages>
<package name="emacs"/>
<package name="vim-enhanced"/>
<package name="unifdef"/>
<package name="mysql-server"/>
<package name="MySQL-python"/>
<package name="python-twill"/>
</packages>
<repos>
<repo name="myrepo_1" url="http://my-repo.com/tools/beaker/devel/"/>
</repos>
</recipe>
</recipeSet>
</job>
To actually determine what distro will be installed, the distroRequires element needs to be populated. Within, we can specify such elements as distro_arch, distro_name and distro_method. This relates to the Distro architecture, the name of the Distro, and it's install method (i.e nfs,ftp etc) respectively. The op determines if we do or do not want this value i.e = means we do want that value, != means we do not want that value. The distro_virt element will determine whether we install on a virtual machine or not.
<job>
<whiteboard>
Apache 2.2 test
</whiteboard>
<recipeSet>
<recipe kernel_options="" kernel_options_post="" ks_meta="" role="None" whiteboard="Lab Controller">
<packages>
<package name="emacs"/>
<package name="vim-enhanced"/>
<package name="unifdef"/>
<package name="mysql-server"/>
<package name="MySQL-python"/>
<package name="python-twill"/>
</packages>
<repos>
<repo name="myrepo_1" url="http://my-repo.com/tools/beaker/devel/"/>
</repos>
<distroRequires>
<and>
<distro_arch op="=" value="x86_64"/>
<distro_name op="=" value="RHEL5-Server-U4"/>
<distro_method op="=" value="nfs"/>
</and>
<distro_virt op="=" value=""/>
</distroRequires>
</recipe>
</recipeSet>
</job>
hostRequires has similar attributes to distroRequires
<job>
<whiteboard>
Apache 2.2 test
</whiteboard>
<recipeSet>
<recipe kernel_options="" kernel_options_post="" ks_meta="" role="None" whiteboard="Lab Controller">
<packages>
<package name="emacs"/>
<package name="vim-enhanced"/>
<package name="unifdef"/>
<package name="mysql-server"/>
<package name="MySQL-python"/>
<package name="python-twill"/>
</packages>
<repos>
<repo name="myrepo_1" url="http://my-repo.com/tools/beaker/devel/"/>
</repos>
<distroRequires>
<and>
<distro_arch op="=" value="x86_64"/>
<distro_name op="=" value="RHEL5-Server-U4"/>
<distro_method op="=" value="nfs"/>
</and>
<distro_virt op="=" value=""/>
</distroRequires>
<hostRequires>
<and>
<arch op="=" value="x86_64"/>
</and>
</hostRequires>
</recipe>
</recipeSet>
</job>
All that's left to populate our XML with, are the task elements. The two attributes we need to specify are the name and the role. Details of how to find which Task's are available, see Section 5.5.2, “Task Searching”. Also note that we've added in a param as a descendant of task. The value of this will be assigned to a new environment variable specified by name.
<job>
<whiteboard>
Apache 2.2 test
</whiteboard>
<recipeSet>
<recipe kernel_options="" kernel_options_post="" ks_meta="" role="None" whiteboard="Lab Controller">
<packages>
<package name="emacs"/>
<package name="vim-enhanced"/>
<package name="unifdef"/>
<package name="mysql-server"/>
<package name="MySQL-python"/>
<package name="python-twill"/>
</packages>
<repos>
<repo name="myrepo_1" url="http://my-repo.com/tools/beaker/devel/"/>
</repos>
<distroRequires>
<and>
<distro_arch op="=" value="x86_64"/>
<distro_name op="=" value="RHEL5-Server-U4"/>
<distro_method op="=" value="nfs"/>
</and>
<distro_virt op="=" value=""/>
</distroRequires>
<task name="/distribution/install" role="STANDALONE">
<params>
<param name="My_ENV_VAR" value="foo"/>
</params>
</task>
</recipe>
</recipeSet>
</job>
3.2.1.3.1.6.3.4. Job Results
The whole purpose of Jobs is to view the output of the Job, and more to the point, Tasks that ran within the Job. To do this, you must first go to the Job search screen (Section 5.3.2, “Job Searching”). After finding the Job you want to see the results of, click on the link in the ID column.You don't have to wait until the Job has completed to view the results. Of course only the results of those Tasks that have already finished running will be available.
The Job results page is divided by Recipe Set. To show the results of each Recipe within these Recipe Sets, click the Show All Results button. You can just show the tasks that have a status of Fail by clicking Show Failed Results.
While your Job is still Queued it's possible to change the priority. You can change the priority of individual Recipe Sets by changing the value of Priority, or you can change all the Job's Recipe Sets at once by clicking an option beside the text Set all RecipeSet priorities, which is at the top right of the page. If successful, a green success message will briefly display, otherwise a red error message will be shown.
Priority permissions
If you are not an Admin you will only be able to lower the priority. Admins can lower and raise the priority
Viewing Job results at a glance
If you would to be able to look at the Result of all Tasks within a particular Job, try the Matrix Report, See Section 5.6.1, “Matrix Report”.
Recipes are contained within a Job (although indirectly, as directly they are contained in a Recipe Set) and are themselves a container for Tasks. There can be more than one Recipe per Job. The purpose of a Recipe is to group a set of Tasks into a single logical unit.
3.2.1.3.2.1. Recipe Searching
The Recipe search is accessed through the Scheduler at the top of the page, and clicking on the Recipe link.
To look up the Recipe ID enter a number into the search box and press the Lookup ID button. See Section 5.1.1, “System Searching” for details on searching.
Quick Searches
By pressing the Running,Queued, or Completed buttons you can quickly display Recipes that have a status of running,queued, and completed respectively.
3.2.1.3.2.2. Recipe Actions
At any time you may wish to cancel the Recipe, you may press the Cancel link that is placed under the Action column.
Tasks are the lowest unit in the Job hierarchy, and running a Task is the primary purpose of running a Job. There purpose is to run one or more commands, and then collect the results of those commands in a way that other entities can access them. You can run as many or as few Tasks in a Job as you like.
3.2.1.3.3.1. Creating a Task
To create Tasks the beaker-devel package will need to be installed. From your terminal, type:
$ yum install beaker-devel
Now make a new directory from where you will create the test. Then cd into the newly created folder and run the following:
$ beaker-create-new-test
$ ls -l
Makefile PURPOSE runtest.sh
Below is a rundown of the files created and how to use them
3.2.1.3.3.1.1. runtest.sh
The core of each Beaker test is a runtest.sh shell script. It performs the testing (or delegates the work by invoking another script or executable) and reports the results. Either write the code that performs the test in the runtest.sh shell script or have runtest.sh execute another program that does the bulk of the work in perhaps another language. Choose a language appropriate to the job (and with which you are familiar): testing of a library could be written in C, parsing of text streams could be done in Perl, and GUI scripting in Python. Languages can be mixed and matched as appropriate within a single test - the runtest.sh script can call other code as necessary. Aim for correctness and readability: remember that others may have to debug this code if a test is flawed. Here is the example runtest.sh:
#!/bin/sh
# Copyright (c) 2006 Red Hat, Inc. All rights reserved. This copyrighted
# material is made available to anyone wishing to use, modify, copy, or
# redistribute it subject to the terms and conditions of the GNU General
# Public License v.2.
#
# This program is distributed in the hope that it will be useful, but WITHOUT
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
# FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301,USA.
# Author: Your Name <Your Email>
beaker-run-simple-test $TEST ./do-my-test.sh
The do-my-test.sh is a rather simple test, seen below, just enough to illustrate the required components of a task.
#!/bin/sh
# simple test example
ls -al /root/.ssh
The exit code from listing the contents of /root/.ssh will be saved to the $? system variable and based on this, beaker-run-simple-test will report the failure or success of the Task. So the most basic Beaker coding requirements are that a shell script must be sourced (beaker-environment.sh) and the report_result API must be called (this is called in beaker-run-simple-text).The output was sent to $OUTPUTFILE (in beaker-environment.sh). The Beaker API provides the $OUTPUTFILE variable so each test can create a log file to record its activities. When run from the command line, $OUTPUTFILE points to a temporary file in /tmp. When run in a test lab environment, the logs are stored in a central location. The report_result method is always called to inform the Beaker API of the success or failure of the test.
A standard Beaker Makefile coordinates many aspects of developing and running a Beaker test:
compiling test executables.
packaging test files into a single RPM.
downloading external source files for use in the test.
collect test files into a known location for execution.
running tests
A sample Makefile is copied into the local directory when beaker-create-new-test tool is invoked (or can be found at /usr/share/doc/beaker-devel-2.6/Makefile.template). The Makefile sets up certain targets and defines variables necessary for test execution and reporting. It is best to use this example when writing a new test, copying and modifying it as necessary. So for example, in order to have an executable compiled by the Makefile, the following two lines were changed.
BUILT_FILES=do-my-test
FILES=$(METADATA) runtest.sh Makefile PURPOSE do-my-test.c
When make run is typed, the Makefile will compile do-my-test.cintodo-my-test and run runtest.sh, which will then execute do-my-test. You do not have to enter anything in the BUILT_FILES directive if you are not compiling an executable. Also be sure to fill in the following section:
# The toplevel namespace within which the test lives.
# FIXME: You will need to change this:
TOPLEVEL_NAMESPACE=
# The name of the package under test:
# FIXME: you wil need to change this:
PACKAGE_NAME=
# The path of the test below the package:
# FIXME: you wil need to change this:
RELATIVE_PATH=
Those three place holders will be used to determine how your Task is named in Beaker, and also how it's mounted on a test System. It will be called /TOPLEVEL_NAMESPACE/PACKAGE_NAME/RELATIVE_PATH and it will be mounted the same in the /mnt/tests directory on a test System.
The test code directory contains a plain text file calledPURPOSE which explains what the test addresses along with any other information useful for troubleshooting or understanding it better. The PURPOSE file has no minimum or maximum length but should provide useful information. For example:
$ cat PURPOSE
This trivial test compiles a .c file, runs the resulting code, and checks that the output is as expected.
It is intended as a simple example of how to write a test that compiles source code to a binary and reports a single result using beaker-run-simple-test
It can also be used as a primitive smoketest for the compiler.
Before we can add a new Task to Beaker we need to package it. Firstly, you're able to run the task locally (if it makes sense) to ensure that the task operates as expected. If you would like to run the task locally:
$ make run
chmod a+x ./runtest.sh
./runtest.sh
/tmp/simple_test/do-my-test.sh
Running ./do-my-test.sh As root:
total 48
drwx------ 2 root root 4096 Dec 15 08:41 .
drwxr-x--- 24 root root 12288 Jun 2 14:57 ..
-rw------- 1 root root 415 Dec 13 19:12 authorized_keys
-rw-r--r-- 1 root root 8630 May 25 09:42 known_hosts
...finished running ./do-my-test.sh, exit code=0
/// result: PASS
Log: /tmp/tmp.c24401
Once you're happy it works as expected, you can run the following:
Tagging
If you wish to, and your Task is revisioned with git or CVS, you can increase the version of the package with the following command:
$ make tag
3.2.1.3.3.2. Task Searching
To search for a Task, go to Scheduler>Task Library at the top of the page. The default search is on the Name property, with the contains operator. See Section 5.1.1, “System Searching” for search details.
Once you've found a particular Task, you can see its details by clicking on the Link in the Name column.
It's also possible to search on the history of the running of Tasks. This is made possible by the Executed Tasks search, which is accessed by clicking on a task.
3.2.1.3.3.3. Adding a New Task
To add a Task which has already been packaged, click Scheduler>New Task. You will need to click on Browse to locate your Task, and then add it with the Submit Data button. See also Chapter 2, Beaker client for adding a task via the beaker client.
Beaker offers a few different reports. They can be accessed from the Reports menu at the top of the page. The Reserve report will give reservation details of Systems that are currently in use. The other report offered is the Matrix report.
3.2.1.3.4.1. Matrix Report
The Matrix report gives a user an overall picture of results for any given Job, or number of Jobs combined. It shows a matrix of Tasks run and the Arch that they were run on. The Reports->Matrix is accesable from the top menu.
There are two ways of defining what Job results to display. You can select the Job by its Whiteboard, or by its Job ID. To show a Job's Matrix report from its Whiteboard, click on the Whiteboard text in the Whiteboard select box. If you wish to select the Job by its ID, enter the Job ID into the Job ID text area. The Job Whiteboard and the Job ID are mutually exclusive when generating the Matrix report. To change between the two, click on their respective input areas. Click the Generate button to create the report.
Filtering Whiteboards
You can filter what is displayed in the Whiteboard select box by typing text into the Filter Whiteboard field, and then clicking anywhere outside the field.
Displaying reports of any combination of Jobs
Displaying the Matrix reports of any Jobs together, is possible when selecting by Job ID. Enter in all the relevant Job IDs seperate by whitespace or a newline.
The generated Matrix report shows the result of each Task with its corresponding Arch and Recipe Whiteboard. The points in the matrix describe the result of the Task, and how many occurences of that result there are. Clicking on these results will take you to the Executed Tasks page. See Section 5.5.2, “Task Searching”.
To have one or more Users grouped together, Beaker uses Groups. Systems can belong to one or more Groups.
3.2.1.3.5.1. Adding a Group
Groups can only be created/edited by a User in the Admin Group. To add a new Group go to Admin->Groups and click the Add( + ) link at the bottom left. You'll then be prompted to enter a Display Name and a Group Name. The former is the name that users of Beaker will see, and the latter is the name used internally. It's fine to have these names the same, or different.
3.2.1.4. Test Architecture Considerations
If you want your test to be smart, that intelligence must be in the test; the Beaker API can help. A test running in an automated environment does not have intelligence, hunches, or the ability to notice unusual activity. This intelligence must be programmed into the test. Naturally the return on investment for time required to add this intelligence should be considered, however the more intelligence a test has to handle false failures and false passes, the more valuable the automation is to the entity running it. Contrasted with manual testing where tests are run on a local workstation and suspicious results can be investigated easily, many organizations find that well written tests which can be trusted save time that can be used for any number of other activities.
Questions to consider
What is needed for a test run to return PASS?
What is needed for a test run to return FAIL?
How will PASS and FAIL conditions be determined pragmatically?
If it is not possible for a test to ever FAIL, does it make sense to automate it?
Writing Good Test Code
Check everything: all exit statuses, return values from function calls, etc. Unfortunately there are plenty of programs which return success codes even when a failure occurs.
Capture all debug output that might indicate an error; it may give clues as to what is going wrong when a test fails.
Comment your tests; good comments should describe the intent of what you are doing, along with caveats being followed, rather than simply parroting the code back as pseudo code.
In most (ideally all) situations a test should report true PASS and FAIL results, but test code is still code, and will invariably contain bugs.
Program defensively so that errors in test code report false FAIL results rather than false PASSes. For example, initialize a result variable to FAIL and only set it to PASS if no errors are detected.
Do not initialize a variable to PASS which fails only on a specific error & mdash;what if you missed another error? What if the shell function you called failed to execute?
It is easier to investigate and fix a failed test than a test that always passes (which it should not be).
3.2.1.5. Reporting Results
The philosophy of Beaker is that the engineers operating the system will want to quickly survey large numbers of tests, and thus the report should be as simple and clear as possible. "PASS" indicates that everything completed as expected. "FAIL" indicates that something unexpected occurred.
In general, a test will perform some setup (perhaps compiling code or configuring services), attempt to perform some actions, and then report on how well those actions were carried out. Some of these actions are your responsibility to capture or generate in your script:
a PASS or FAIL and optionally a value indicating a test-specific metric, such as a performance figure.
a debug log of information & mdash;invaluable when troubleshooting an unexpected test result. A test can have a single log file and report it into the root node of your results tree, or gather multiple logs, reporting each within the appropriate child node.
Other components of the result can be provided automatically by the framework when in a test lab environment:
the content of the kernel ring buffer (from dmesg). Each report clears the ring buffer, so that if your test reports multiple results, each will contain any messages logged by the kernel since the last report was made.
a list of all packages installed on the machine under test (at the time immediately before testing began), including name, version/release, and architecture.
a separate report of the packages listed in the RunFor of the metadata including name, version/release, and architecture (since these versions are most pertinent to the test run).
if a kernel panic occurs on the machine under test, this is detected for you from the console log output, and will cause an Abort Panic result in place of a PASS or FAIL for that test.
In addition, the Beaker framework provides a hierarchical namespace of results, and each test is responsible for a subtree of this namespace. Many simple tests will only return one result (the node they own), but a complex test can return an entire subtree of results as desired. The location in the namespace is determined by the value of variables defined in the Makefile. These variables will be discussed in the Packaging section.
A test may be testing a number of related things with a common setup (e.g. a setup phase of a server package onto localhost, followed by a collection of tests as a client). Some of these things may not work across every version/architecture combination. This will produce a list of "subresults", each of which could be classified as one of:
expected success: will lead to a PASS if nothing else fails
expected failure: should be as a PASS (as you were expecting it).
unexpected success: can be treated as a PASS (since it's a success), or a FAIL (since you were not expecting it).
unexpected failure: should always be a FAIL
Given that there may be more than one result, the question arises as to how to determine if the whole test passes or fails. One way to verify regression tests is to write a script that compares a set of outputs to an expected "gold" set of outputs which grants PASS or FAIL based on the comparison.
It is possible to write a script that silently handles unexpected successes, but it is equally valid for a script to report a FAIL on an unexpected success, since this warrants further investigation (and possible updating of the script).
To complicate matters further, expected success/failure may vary between versions of the package under test, and architecture of the test machine.
If the test is checking multiple bugs, some of which are known to work, and some of which are due to be fixed in various successive (Fedora) updates, ensure that the test checks everything that ought to work, reporting PASS and FAIL accordingly. If the whole test is reporting a single result, it will typically report this by ensuring that all expected things work; as bugs are fixed, more and more of the test is expected to work and can cause an overall FAIL.
If it is reporting the test using a hierarchy of results, the test can have similar logic for the root node, and can avoid reporting a result for a subtree node for a known failure until the bug is fixed in the underlying packages, and avoid affecting the overall result until the bug(s) is fixed.
As a general Beaker rule of thumb, a FAIL anywhere within the result subtree of the test will lead to the result for the overall test being a FAIL.
Indicate failure-causing conditions in the log clearly, with "FAIL" in upper case to make it easier to grep for.
Good log messages should contain three things: # what it is that you are attempting to do (e.g. checking to see what ls reports for the permission bits that are set on file foo) # what it is that you expect to happen (e.g. an expectation of seeing "-rw-r--r--" ) # what the actual result was an example of a test log showing success might look like:
Checking ls output: "foo" ought to have permissions "-rw-r--r--"
Success: "foo" has permissions: "-rw-r--r--"
An example of a failure might look like:
Checking ls output: "foo" ought to have permissions "-rw-r--r--"
FAIL: ls exit status 2
For multihost tests, time stamp all your logs, so you can interleave them.
Use of tee is also helpful to ensure that the output at the terminal as you debug a test is comparable to that logged from OUTPUTFILE in the lab environment.
Past experiences has shown problems where people confuse overwriting versus appending when writing out each line of a log file. Use tee -a $OUTPUT rather than tee > $OUTPUT or tee >> $OUTPUT.
Include a final message in the log, stating that this is the last line, and (for a single-result test) whether the result is a success or failure; for example:
echo "----- Test complete: result=$RESULT -----" | tee -a $OUTPUTFILE
Finish your runtest.sh: (after the report_result) to indicate that the final line was reached; for example:
echo "***** End of runtest.sh *****"
3.2.1.5.2. Passing Parameters to Tests
When you need a test to perform different steps in some specific situations there is an option available through Single package workflow command line interface called --test-params which allows you to pass the supplied parameter to runtest.sh where you can access it by TEST_PARAM_NAME=value.
For example you can launch the single workflow with a commandline like this:
single_package.py ... -t /some/test/name --test-params="PAR1=val1" --test-params="PAR2=val2"
And then make use of the passed parameter inside the runtest.sh script:
if [[ TEST_PARAM_PAR1 == 1 ]] ; then do something; fi
3.2.1.6. Unassimilated or Unfinished content
3.2.1.6.1. Using the startup_test function
The startup_test function can be used to provide a primitive smoketest of a program, by setting a shell variable named result. You will need to use report_result if you use it. The syntax is:
startup_test program [arg1] [arg2] [arg3]
The function takes the name of a program, along with up to three arguments. It fakes an X server for the test by ensuring that Xvfb is running (and setting DISPLAY accordingly), then enables core-dumping, and runs the program with the arguments provided, piping standard output and error into OUTPUTFILE (overwriting, not appending).
The function then checks various things:
any Gtk-CRITICAL warnings found in the resulting OUTPUTFILE cause result to be WARN.
that the program can be found in the PATH, using the which command; if it is not found it causes result to be FAIL, appending the problem to OUTPUTFILE
for binaries, it uses ldd to detect missing libraries; if any are missing it causes result to be FAIL, appending the problems to OUTPUTFILE
if any coredumps are detected it causes result to be FAIL
Finally, it kills the fake X server. You then need to report the result.
#!/bin/sh
# source the test script helpers
. /usr/bin/beaker-environment.sh
# ---- do the actual testing ----
result=PASS 1
startup_test /usr/bin/evolution
report_result $TEST $result 2
Normally it's a bad idea to start with a PASS and try to detect a FAIL, since an unexpected error that prevents further setting of the value will lead to a false PASS rather than a false FAIL. Unfortunately in this case the startup_test function requires it.
report_result $TEST $result
We report the result, using the special result shell variable set by startup_test
3.2.1.7. Writing and Running Multihosts Tests
All of the examples so far have run on a single host. Beaker has support for tests that run on multiple hosts, e.g. for testing the interactions of a client and a server.
When a multihost test is run in the lab, a machine will be allocated to each role in the test. Each machine has its own recipe. Multihost testing requires the concept of a recipe set: all of the per-machine recipes within a particular multihost test that go together to form the test as a whole.
Each machine under test will need to synchronize to ensure that they start the test together, and may need to synchronize at various stages within the test. Beaker has three notional roles: client, server and driver.
For many purposes all you will need are client and server roles. For a test involving one or more clients talking to one or more servers, a typical approach would be for the clients to block whilst the servers get ready. Once all servers are ready, the clients perform whatever testing they need, using the services provided by the server machines, and eventually report results back to Red Hat Test System Whilst this is happening the server tests block; the services running on these machines are carrying out work for the clients in parallel. Once all clients have finished testing, the server tests finish, and report their results.
Each participant in a test will be reporting results within the same job, and so must report to different places within the result hierarchy. For example, the server part of the test may PASS if it survives the load, but the client part might FAIL upon, say, getting erroneous data from the server; this would lead to an overall FAIL for the test.
If you have a more complex arrangement, it is possible to have a driver machine which controls all of the testing. Talk to Red Hat Quality Engineering if you need to do this.
All of the participants in a multihost test share a single
runtest.sh, which must perform every role within the test (e.g. the client role and server role). When a multihost test is run in the lab, the framework automatically sets environment variables to allow the various participants to know what their role should be, which other machines they should be talking to, and what roles those other machines are performing in the test. You will need to have logic in your
runtest.sh to examine these variables, and perform the necessary role accordingly. These variables are shared by all instances of the runtest.sh within a recipeset:
CLIENTS contains a space-separated list of hostnames of clients within this recipeset.
SERVERS contains a space-separated list of hostnames of servers within this recipeset.
DRIVER is the hostname of the driver of this recipeset, if any.
The variable HOSTNAME can be used by runtest.sh to determine its identity. It is set by beaker-environment.sh, and will be unique for each host within a recipeset.
Your test can thus decide whether it is a client, server or driver by investigating these variables: see the example below.
When you are developing your test outside the lab environment, only HOSTNAME is set for you (when sourcing the beaker-environment.sh script). Typically you will copy your test to multiple development machines, set up CLIENTS, SERVERS and DRIVER manually within a shell on each machine, and then manually run the runtest.sh on each one, debugging as necessary.
A multihost test needs to be marked as such in the Type: Multihost.
3.2.1.7.1. Synchronization Commands
Synchronization of machines within a multihost test is performed using per-host state strings managed on the Red Hat Test System server. Each machine's starting state is the empty string.
beaker-sync-set -s state
The beaker-sync-set command sets the state of this machine within the test to the given value.
beaker-sync-block -s state [hostnames...]
The beaker-sync-block command blocks further execution of this instance of the script until all of the listed hosts are in the given state.
Unfortunately, there is currently no good way to run these commands in the standalone helper environment.
3.2.1.7.1.1. Example of a runtest.sh for a multihost test
#!/bin/sh
# Source the common test script helpers
. /usr/bin/beaker_environment.sh
# Save STDOUT and STDERR, and redirect everything to a file.
exec 5>&1 6>&2
exec >> "${OUTPUTFILE}" 2>&1
client()
{
echo "-- wait the server to finish."
beaker_sync_block -s "DONE" ${SERVERS}
user="finger1"
for i in ${SERVERS}
do
echo "-- finger user \"$user\" from server \"${i}\"."
./finger_client "${i}" "${user}"
# It returns non-zero for failure.
if [ $? -ne 0 ]; then
beaker_sync_set -s "DONE"
report_result "${TEST}" "FAIL" 0
exit 1
fi
done
echo "-- client finishes."
beaker_sync_set -s "DONE"
result="PASS"
}
server()
{
# Start server and check it is up and running.
/sbin/chkconfig finger on && sleep 5
if ! netstat -a | grep "finger" ; then
beaker_sync_set -s "DONE"
report_result "${TEST}" "FAIL" 0
exit 1
fi
useradd finger1
echo "-- server finishes."
beaker_sync_set -s "DONE"
beaker_sync_block -s "DONE" ${CLIENTS}
result="PASS"
}
# ---------- Start Test -------------
result="FAIL"
if echo "${CLIENTS}" | grep "${HOSTNAME}" >/dev/null; then
echo "-- run finger test as client."
TEST=${TEST}/client
client
fi
if echo "${SERVERS}" | grep "${HOSTNAME}" >/dev/null; then
echo "-- run finger test as server."
TEST=${TEST}/server
server
fi
echo "--- end of runtest.sh."
report_result "${TEST}" "${result}" 0
exit 0
3.2.1.7.1.2. Tuning up multihost tests
Multihost tests can be easily tuned up outside Beaker using following code snippet based on $JOBID variable (which is set when running in Beaker environment). Just log in to two machines (let's say: client.redhat.com and server.redhat.com) and add following lines at the beginning of your runtest.sh script.
# decide if we're runnig on Beaker or in developer mode
if test -z $JOBID ; then
echo "Variable JOBID not set, assuming developer mode"
CLIENTS="client.redhat.com"
SERVERS="server.redhat.com"
else
echo "Variable JOBID set, we're running on Beaker"
fi
echo "Clients: $CLIENTS"
echo "Servers: $SERVERS"
Then you just run the script on both client and server. When scripts reach one of the synchronization commands (beaker-sync-set or beaker-sync-block) you will be asked for supplying actual state of the client/server by keyboard (usually just confirm readiness by hitting Enter). That's it! :-)
This guide provide in depth information for the required and optional Makefile variables in an Beaker test. A sample Makefile is copied into the local directory when beaker-create-new-test tool is invoked (or can be found at /usr/share/doc/beaker-devel-x.y/Makefile.template
).
# The name of the package under test:
PACKAGE_NAME=gcc
The package under test is the common command or executable being tested. This must be the name of an installable RPM in the distribution. If the focus of your test is a third party application, set PACKAGE_NAME equal to the primary package used by the third party application. For example, if you are writing a Beaker test to test a web application based on CGI, set PACKAGE_NAME to perl.
3.2.1.8.2. TOPLEVEL NAMESPACE
# The toplevel namespace within which the test lives.
TOPLEVEL_NAMESPACE=CoreOS
The Makefile contains three hierarchies resembling file systems, each with their own collections of paths. In order to ensure consistency between test creators and tests, the provided Makefile should be used to manage these hierarchies. Thus, the scheme in the Makefile template does all the work.
For clarity, it is worth noting the hierarchies at this time:
installation tests are built as packages for clean deployment on test machines. More than one test can be run on a given machine, so there is a hierarchy below /mnt/tests which keeps the files of the individual tests separate from each other. This is set in each test's Makefile.
result namespace a hierarchical namespace for results. For example, tests relating to the kernel report their results somewhere within the /kernel subtree, and tests relating to the NFS file system (as a part of the kernel) report their results inside /kernel/filesystems/nfs/. Each test "owns" a subtree of the namespace, specified by the Name: field of the metadata. Many tests report only a single result, but it is possible for a test to write out a complex hierarchy of results below its subtree.
The following top level reporting namespaces are predefined and should used to ensure consistent reporting of test results. These are the only valid accepted namespaces.
distribution contains tests that involve the distribution as a whole, or a large number of packages, for example /distribution/standards/posixtestsuite
kernel contains tests results relating to the kernel, for example
/kernel/xen/xm/dmesg
The
kernel namespace is unique in that it is also the name of a package. In this case it is usually best to define the
TOPLEVEL_NAMESPACE like this:
# The toplevel namespace definition for kernel tests
TOPLEVEL_NAMESPACE=$(PACKAGE_NAME)
Desktop contains tests results relating to desktop packages, for example /desktop/evolution/first-time-wizard-password-settings, which is a specific test relating to evolution
Tools contains tests results relating to the tool chain, for example /tools/gcc/testsuite/3.4
CoreOS all test results relating to user-space packages not covered by any of the above namespaces
Examples example tests that illustrate usage and functionality and are not activly maintained. This is a good place to experiment when you are getting hang of Beaker or to place simple examples to help others.
# The path of the test below the package:
RELATIVE_PATH=example-compilation
An implementation of Beaker should run the test from the directory containing the runtest.sh, as listed in the RELATIVE_PATH file of the Makefile. If the test needs to move around, store this somewhere with DIR=`pwd` or use pushd and popd.
# Version of the Test. Used with make tag.
export TESTVERSION=1.0
This is used when building a package of a test, and provides the "version" component of the RPM name-version-release triplet.
The value must be valid as an RPM version string.
It may consist of numbers, letters, and the dot symbol.
It may not include a dash symbol this is used by RPM to delimit the version string within the name-version-release triplet.
When writing a test from scratch, use 0.1 and increment gradually until the test has reached a level of robustness to merit a "1.0" release. When wrapping a test from an upstream location, use the upstream version string here, as closely as possibly given the restrictions on valid characters. The version should be incremented each time a change is made to the Makefile or test files and a RPM is created from these files to be publicly consumed in a test review or submission to a lab scheduler.
# The compiled namespace of the test.
export TEST=$(TOPLEVEL_NAMESPACE)/$(PACKAGE_NAME)/$(RELATIVE_PATH)
This variable defines the path to a test. This path should also be the same in source control.
BUILT_FILES=hello-world
List the files that need to be compiled to be used in the test.
Each test must supply a run target which allows an implementation of the framework to invoke make run. It is usually best to have this as the first target defined in the Makefile so that a simple invocation of make will use it as the default, and run the test. Note how the build target is set up as a dependency of run to ensure that this happens if necessary.
Additional targets and variables supplied by the include for /usr/share/beaker/lib/beaker-make.include. This file is supplied with beaker-devel as seen below.
[root@dhcp83-5 example-compilation]# cat /usr/share/beaker/lib/beaker-make.include
# Copyright (c) 2006 Red Hat, Inc. All rights reserved. This copyrighted material l
# is made available to anyone wishing to use, modify, copy, or
# redistribute it subject to the terms and conditions of the GNU General
# Public License v.2.
#
# This program is distributed in the hope that it will be useful, but WITHOUT AN Y
# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
# PARTICULAR PURPOSE. See the GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
#
# Author: Greg Nichols <gnichols@redhat.com>
#
# beaker-make.include
#
# default rules and settings for beaker makefiles
#
# Common Variables.
TEST_DIR=/mnt/tests$(TEST)
INSTALL_DIR=$(DEST)$(TEST_DIR)
METADATA=testinfo.desc
# tag: mark the test source as a release
tag:
beaker-mk-tag-release
release: tag
# prep: prepare the test(s) for packaging
install: $(FILES) runtest.sh testinfo.desc
mkdir -p $(INSTALL_DIR)
cp -a $(FILES) Makefile $(INSTALL_DIR)
install -m 0755 runtest.sh $(INSTALL_DIR)
# package: build the test package
package:
beaker-mk-build-package
# submit: submit the test package to Beaker
submit:
beaker-mk-build-package -s $(TESTSERVER)
##################################################
# example makefile
#
# include ~/devel/beaker/greg/beaker_nb/make.include
#
# FILES=prog1.c prog2.c
#
# ARENA=$(DEST)/mnt/tests/glibc/double-free-exploit
#
# install:
# mkdir -p $(ARENA)
# cp -a runtest.sh $(FILES) $(ARENA)
#
# run: tests
# runtest.sh
#
# tests: prog2 prog2
The tag target is used to tag a package in anticipation of submitting it to a test lab.
The submit target is used to submit a package to a test lab and requires the TESTSERVER variable to be defined. It builds an RPM of the test (if necessary) and uploads the test package to a test lab controller where it can be used to schedule tests.
Following is an example of the METADATA section needed to execute a basic test. Following subsections will comment briefly on the values that must be set manually (not set by variables) and optional values to enhance test reporting and execution.
$(METADATA): Makefile
@touch $(METADATA)
@echo "Owner: David Malcolm <dmalcolm@redhat.com>" > $(METADATA)
@echo "Name: $(TEST)" >> $(METADATA)
@echo "Path: $(TEST_DIR)" >> $(METADATA)
@echo "License: GPLv2" >> $(METADATA)
@echo "TestVersion: $(TESTVERSION)" >> $(METADATA)
@echo "Description: Ensure that compiling a simple .c file works as expected" >> $(METADATA)
@echo "TestTime: 1m" >> $(METADATA)
@echo "RunFor: $(PACKAGE_NAME)" >> $(METADATA) # add any other packages for which your test ought to run here
@echo "Requires: $(PACKAGE_NAME)" >> $(METADATA) # add any other requirements for the script to run here
Owner: (optional) is the person responsible for this test. Initially for Beaker, this will be whoever committed the test to Subversion. A naming policy may have to be introduced as the project develops. Acceptable values are a subset of the set of valid email addresses, requiring the form: Owner: human readable name <username@domain>.
Name: (required) It is assumed that any result-reporting framework will organize all available tests into a hierarchical namespace, using forward-slashes to separate names (analogous to a path). This field specifies the namespace where the test will appear in the framework, and serves as a unique ID for the test. Tests should be grouped logically by the package under test. This name should be consistent with the name used in source control too. Since some implementations will want to use the file system to store results, make sure to only use characters that are usable within a file system path.
Description (required) must contain exactly one string.
For example:
Description: This test tries to map five 1-gigabyte files with a single process.
Description: This test tries to exploit the recent security issue for large pix map files.
Description: This test tries to panic the kernel by creating thousands of processes.
Every Makefile must contain exactly one TestTime value. It represent the upper limit of time that the runtest.sh script should execute before being terminated. That is, the API should automatically fail the test after this time period has expired. This is to guard against cases where a test has entered an infinite loop or caused a system to hang. This field can be used to achieve better test lab utilization by preventing the test from running on a system indefinitely.
The value of the field should be a number followed by either the letter "m" or "h" to express the time in minutes or hours. It can also be specified it in seconds by giving just a number. It is recommended to provide a value in minutes, for readability.
The time should be the absolute longest a test is expected to take on the slowest platform supported, plus a 10% margin of error. It is usually meaningless to have a test time of less than a minute, since some implementations of the API may be attempting to communicate with a busy server such as writing back to an NFS share or performing an XML-RPC call.
For example:
TestTime: 90 # 90 seconds
TestTime: 1m # 1 minute
TestTime: 2h # 2 hours
Requires one or more. This field indicates the packages that are required to be installed on the test machine for the test to work. The package being tested is automatically included via the PACKAGE_NAME variable. Anything runtest.sh needs for execution must be included here.
This field can occur multiple times within the metadata. Each value should be a space-separated list of package names, or of Kickstart package group names preceded with an @ sign. Each package or group must occur within the distribution tree under test (specifically, it must appear in the comps.xml file).
For exmaple:
@echo "Requires: gdb" >> $(METADATA)
@echo "Requires: @legacy-software-development" >> $(METADATA)
@echo "Requires: @kde-software-development" >> $(METADATA)
@echo "Requires: -pdksh" >> $(METADATA)
The last example above shows that we don't want a particular package installed for this test. Normally you shouldn't have to do this unless the package is installed by default.
In a lab implementation, the dependencies of the packages listed can be automatically loaded using yum.
Note that unlike an RPM spec file, the names of packages are used rather than Provides: dependencies. If one of the dependencies changes name between releases, one of these approaches below may be helpful:
for major changes, split the test, so that each release is a separate test in a sub-directory, with the common files built from a shared directory in the Makefile.
if only a dependency has changed name, specify the union of the names of dependencies in the Requires: field; an implementation should silently ignore unsolvable dependencies.
it may be possible to work around the differences by logic in the section of the Makefile that generates the testinfo.desc file.
When writing a multihost test involving multiple roles client(s) and server(s), the union of the requirements for all of the roles must be listed here.
3.2.1.8.15. BeakerRequires
BeakerRequires one or more. This field indicates the other beaker tests that are required to be installed on the test machine for the test to work.
This field can occur multiple times within the metadata. Each value should be a space-separated list of Beaker Test RPM name without the version or .rpm. Each Test must exist on the Beaker Scheduler.
For example:
@echo "BeakerRequires: rh-tests-distribution-hts-common" >> $(METADATA)
RunFor allows for the specification of the packages which are relevant for the test. This field is the hook to be used for locating tests by package. For example, when running all tests relating to a particular package[1], an implementation should use this field. Similarly, when looking for results on a particular package, this is the field that should be used to locate the relevant test runs.
When testing a specific package, that package must be listed in this field. If the test might reasonably be affected by changes to another package, the other package should be listed here. If a package changes name in the various releases of the distribution, all its names should be listed here.
This field is optional; and can occur multiple times within the metadata. The value should be a space-separated list of package names.
Some tests are only applicable to certain distribution releases. For example, a kernel bug may only be applicable to RHEL3 which contains the 2.4 kernel. Limiting the release should only be used when a test will not execute on a particular release. Otherwise, the release should not be restricted so that your test can run on as many different releases as possible.
Releases can be used in two ways:
specifying releases you
want run your test for : For example, if you want to run your test on RHEL3 and RHEL4 only, add "Releases: RedHatEnterpriseLinux3 RedHatEnterpriseLinux4" to your Makefile METADATA variable, i.e.:
...
@echo "Requires: openldap-servers" >> $(METADATA)
@echo "Releases: RedHatEnterpriseLinux3 RedHatEnterpriseLinux4" >> $(METADATA)
@echo "Priority: Normal" >> $(METADATA)
...
specifiying releases you
don't want run your test for (using "-" sign before given releases): For example, if you don't want to run your test on RHEL3, but the other releases are valid for your test, add "Releases: -RedHatEnterpriseLinux3" to your Makefile METADATA variable, i.e.:
...
@echo "Requires: openldap-servers" >> $(METADATA)
@echo "Releases: -RedHatEnterpriseLinux3" >> $(METADATA)
@echo "Priority: Normal" >> $(METADATA)
...
3.2.1.9. Virtualization Workflow
Virtualization workflow is designed to take advantage all Beaker offers to be used for virtualization testing. The audience of this tutorial is expected to have basic familiarity with Beaker.
Virtualization testing framework in Beaker utilizes libvirt tools, particularly virt-install program to have a framework abstracted from the underlying virtualization technology of the OS. The crux of virtualization test framework is guestrecipe. Each virtual machine is defined in its own guestrecipe and guestrecipes are a part of the host's (dom0) recipe. To illustrate, let's say, we would like to create a job that will create a host and 2 guests, named guest1 and guest2 respectively. The skeleton of the recipe will look like this:
<recipe>
...
(dom0 test recipe)
...
<guestrecipe guestname="guest1">
...
(guest1 test recipe)
...
</guestrecipe>
<guestrecipe guestname="guest2">
...
(guest2 test recipe)
...
</guestrecipe>
</recipe>
Anything that can be described inside a recipe can also be described inside a guestrecipe. This allows the testers to run any existing Beaker test inside the guest just like it'd be run inside a baremetal machine.
When Beaker encounters a guestrecipe it does create an environmental variable to be passed on to virtinstall test. The tester-supplied elements of this variable all come from the guestrecipe element. Consequently, it's vital that the tester fully understand the properties of this element. guestrecipe element guestname and guestargs elements. guestname is the name of the guest you would like to give and is optional. If you omit this property then the Beaker will assign the hostname of the guest as the name of the guest. guestargs is where you define your guest. The values given here will be same as what one would pass to virt-install program with the following exceptions:
Name argument must not be passed on inside guestargs. As mentioned above, it should be passed with guestname property..
Other than name , -mac, -location, -cdrom (-c) , and -extra-args ks= must not be passed. Beaker does those based on distro information passed inside the guestrecipe.
In addition to what can be passed to virt-install, extra arguments -lvm or -part or -kvm can also be passed to guestargs, to indicate lvm-based or partition- based guests or kvm guests (instead of xen guests).
If neither one of -lvm or -part options are given, then a filebased guest will be installed. If -kvm option is not given then xen guests will be installed. See below for lvm-,partition-based guests section for more info on this topic.
The virtinstall test is very forgiving for the missed arguments, it'll use some default when it can. Currently these arguments can be omitted:
-ram or -r , a default of 512 is used
1.-nographics or -vnc, if the guest is a paravirtualized guest, then -nographics option will be used, if the guest is an hvm guest, then -vnc option will be used.
1.-file-size or -s, a default of 10 will be used.
-file or -f, if the guest is a filebased guest, then the default will be /var/lib/xen/images/${guestname}.img . For lvm-based and block-device based guest, this option MUST be provided.
3.2.1.9.1. KVM vs XEN GUESTS
Starting with RHEL 5.4, both xen and kvm hypervisors are shipped with the distro. To handle this situation, guest install tests take an extra argument (-kvm) to identify which type of guests will be installed. By default, kernel-xen kernel is installed hence the guests are xen guests. If -kvm is given in the guestargs, then the installation program decides that kvm guests are intended to be tested, so boots into the base kernel and then installs the guests. There can only be one hypervisor at work at one moment, and hence the installation test expects them all to be either kvm or xen guest, but not a mix of both.
3.2.1.9.2. Making More Sense of LVM and PARTITION based Guest Installations
Installing a file-based guest is the simplest of all. A specific file can be specified in guestarg with -file or -f arguments , or can just be omitted. However, for lvm and partition based guests, the virtinstall test will have to know where to install the guests exactly, because there is no way for it to know what the partition or volume name might be.
Obviously, if lvm or partition based guests are desired a custom partitioning will have to be done. Beaker allows the testers to submit a custom partition/lvm for their tests. The syntax for this below:
<partition> <type> type </type> <name> name </name> <size> size in GB </size> <fs> filesystem to format </fs> </partition>
type: Can be either lvm or part. This is required.
name: For partitions, this will be mount point, such as /mnt/guest1 , for lvm, it'll be the name of the volume, such as myguestlvm . In the case of lvm, it'll be named /TestVolumeGroup??/myguestlvm . This is required.
size: Size of the blockdevice in Gigabytes, this is required.
fs: filesystem you'd like to be formatted for this partition . This is optional, if omitted, ext3 will be used. * -file or -f guestargs, whatever the name is given inside the partition block should be passed on.
3.2.1.9.3. Other Uses of PARTITIONS and LVM Volumes
In virtualization testing, block devices can be used for other purposes than installing a guest on them. It's possible to have block devices attached to or detached from a guest. If you'd like to do this operation in your tests, you can either pass the information about the partitions you have created as environment variable to your test, or you can use the blockdevice utility, which is another test that lives in /distribution/utils/blockdevice .This testcase just creates a text file containing information about the block devices and manages them. Its commands are:
getdevice
Usage: getdevice <lvm|partition> <minimum size in GB>
Description: returns a free device that has enough space
Returns: string with device name and return code 0 on success, error string and return code 1 on failure
freedevice
Usage: freedevice <devicename>
Descriptions: Marks the device can free so it can be reused.
Returns: return code zero on success or error string and return code 1 on failure.
When running blockdevice test-util, you need to provide testing an environmental variable named BLOCKDB_ARGS, telling the test what partitions/lvms you have intended to be used in test script. The format of $BLOCKDB_ARGS
is name:type:size:fs;name:type:size:fs; for each block device. For example suppose you have these partitions in your job:
<partition>
<type>part</type>
<name>/mnt/block1</name>
<size>1</size> <!- 1 gig ->
</partition>
<partition>
<type>lvm</type>
<name>mylvm</name>
<size>5</size>
</partition>
<partition>
<type>part</type>
<name>/mnt/block4ext4</name>
<size>1</size> <!- 1 gig ->
<fs>ext4dev</fs>
</partition>
<partition>
<type>lvm</type>
<name>mylvm4ext4</name>
<size>5</size>
<fs>ext4dev</fs>
</partition>
then BLOCKDB_ARGS would be: /mnt/block1:part:1;mylvm:lvm:5;/mnt/block4ext4:part:1:ext4dev;mylvm4ext4:lvm:5:ext4dev
If you utilize blockdevices test, you should ensure that:
3.2.1.9.4. Dynamic Partitioning/IVM
On Beaker, each machine has its own kickstart for each OS family it supports. In it the partitioning area is marked so that it can be overwritten to allow having dynamic partitions/lvms in your tests.
The easiest way to specify dynamic partitions is to use the xml workflow and specify it in your xml file. Syntax of the partition tags is below:
<partition>
<type> type </type> <!- required ->
<name> name </name> <!- required ->
<size> size in GB </size> <!- required ->
<fs> filesystem to format </fs> <!- optional, defaulted to ext3 ->
</partition>
<partition> is the xml element for the partitioning. You can have multiple partition elements in a recipe. It has type, name, size and fs text contents all of which except for fs is required. Detailed information for each are:
type: Type of partition you'd like to use. This can be either part of lvm .
name: If the type is part, then this will be the mount point of the partition. For example, if you would like the partition to be mounted to /mnt/temppartition then just put it in here. For the lvm type, this will be the name of the volume and all custom volumes will go under its own group, prefixed with TestVolumeGroup? . For example, if you name your lvm type as "mytestvolume", it's go into /TestVolumeGroup??/mytestvolume.
size: The size of the partition or volume in GBs .
fs:This will be the filesystem the partition will be formatted in. If omitted, the partition will be formatted with ext3. By default, anaconda mounts all partitions. If you need the partition to be unmounted at the time of the test, you can use the blockdevice utility which is a test that lives on /distribution/utils/blockdevice . This test unmounts the specified partitions/volumes and lets users manage custom partitions thru its own scripts.
getdevice
Usage: getdevice <lvm|partition> <minimum size in GB>>
Description: returns a free device that has enough space
Returns: string with device name and return code 0 on success, error string
and return code 1 on failure
freedevice
Usage: freedevice <devicename>
Descriptions: Marks the device can free so it can be reused.
Returns: return code zero on success or error string and return code 1 on failure.
When running blockdevice test-util, you need to provide testing an environmental variable named BLOCKDB_ARGS, telling the test what partitions/lvms you have intended to be used in test script. The format of $BLOCKDB_ARGS is name,type,size;name,type,size; for each block device. For example suppose you have these partitions in your job:
<partition>
<type>part</type>
<name>/mnt/block1</name>
<size>1</size> <!- 1 gig ->
</partition>
<partition>
<type>lvm</type>
<name>mylvm</name>
<size>5</size>
</partition>
<partition>
<type>part</type>
<name>/mnt/block4ext4</name>
<size>1</size> <!- 1 gig ->
<fs>ext4dev</fs>
</partition>
<partition>
<type>lvm</type>
<name>mylvm4ext4</name>
<size>5</size>
<fs>ext4dev</fs>
</partition>
</screen>
<para>then BLOCKDB_ARGS would be:
</para>
<screen>/mnt/block1:part:1;mylvm:lvm:5;/mnt/block4ext4:part:1:ext4dev;mylvm4ext4:lvm:5:ext4dev
as in:
<test role='STANDALONE' name='/distribution/utils/blockdevice'>
<params>
<param name='BLOCKDB_ARGS' value='/mnt/block1:part:1;mylvm:lvm:5;/mnt/block4ext4:part:1:ext4dev;mylvm4ext4:lvm:5:ext4dev'/>
</params>
</test>
If you utilize blockdevices test, you should ensure that:
3.2.1.9.4.1. Dynamic Partitioning from Your Workflow
If you are using a different workflow and would like to add dynamic partitioning capability, you can do it by utilizing kickPart() call to the recipe object. The string you have to pass is exactly same format as the BLOCKDB_ARGS argument mentioned above. An example can be :
part_str = "/mnt/block1:part:1;mylvm:lvm:5;/mnt/block4ext4:part:1:ext4dev;mylvm4ext4:lvm:5:ext4dev"
rec = Recipe(scheduler=beaker_sched)
rec.kickPart(part_str)
3.2.1.9.4.2. Installing Package with Workflows
Workflow scripts are packaged in the beaker-redhat package.
3.2.1.9.5. Helper Programs Installed with Virtinstall
Virtinstall test also installs a few scripts that can later on be utilized in the tests. These are completely non-vital scripts, provided only for convenience to the testers.
guestcheck4up:
Usage: guestcheck4up <guestname>
Description: checks whether or not the guest is live or not.
Returns: 0 if guest is not shutoff, 1 if it is.
guestcheck4down:
Usage: guestcheck4down <guestname>
Description: checks whether or not the guest is live or not.
Returns: 0 if guest is shutoff, 1 if it is not.
startguest:
Usage: startguest <guestname> [timeout]
Description: Starts a guest and makes sure that it's console is reachable within optional $timeout seconds. If timeout value is omitted the default is 300 seconds.
Returns: 0 if the guest is started and a connection can be made to its console within $timeout seconds, 1 if it can't.
stopguest:
Usage: stopguest <guestname> [timeout]
Description: stops a guests and waits for shutdown by waiting for the "System Halted." string within the optional $timeout seconds. If timeout is omitted , then the default is 300 seconds.
Returns: 0 if the shutdown was successful, 1 if it wasn't.
getguesthostname:
Usage: getguesthostname <guestname>
Returns: A string that contains the hostname of the guest if successful, or an error string if it's an error.
wait4login:
Usage: wait4login <guestname> [timeout]
Description: It waits until it gets login: prompt in the guest's console within $timeout seconds. If timeout argument is not given, it'll wait indefinitely, unless there is an error!
Returns: 0 on success , or 1 if it encounters an error.
fwait4shutdown:
Usage: wait4shutdown <guestname> [timeout]
Description: It waits until it gets shutdown message in the guest's console within $timeout seconds. If timeout argument is not given, it'll wait indefinitely, unless there is an error!
Returns: 0 on success , or 1 if it encounters an error.
3.2.1.10. Multihost Testing
Running multihost tests on Beaker is done thru having different test roles in multihost tests amongst multiple recipes inside a recipeset. In it's simplest form, a job with multihost testing can look like:
<job>
<RecipeSet>
<recipe>
<test role='STANDALONE' name='/distribution/install'/>
<test role='SERVERS' name='/my/multihost/test'/>
</recipe>
<recipe>
<test role='STANDALONE' name='/distribution/install'/>
<test role='CLIENTS' name='/my/multihost/test'/>
</recipe>
</RecipeSet>
</job>
Note
For brewity some necessary parts are left out in the above job description
Submitting the job above will export environmental variables SERVERS and CLIENTS set to their respective hostnames. This allows a tester to write tests for each machines. So the runtest.sh in /my/multihost/test test might look like:
Server() {
# .. server code here
}
Client() {
# .. client code here
}
if test -z "$JOBID" ; then
echo "Variable jobid not set! Assume developer mode"
SERVERS="test1.beaker.bos.redhat.com"
CLIENTS="test2.beaker.bos.redhat.com"
DEVMODE=true
fi
if [ -z "$SERVERS" -o -z "$CLIENTS" ]; then
echo "Can not determine test type! Client/Server Failed:"
RESULT=FAILED
report_result $TEST $RESULT
fi
if $(echo $SERVERS | grep -q $HOSTNAME); then
TEST="$TEST/Server"
Server
fi
if $(echo $CLIENTS | grep -q $HOSTNAME); then
TEST="$TEST/Client"
Client
fi
Let's disect the code. First of, we have Server() and Client() functions which will be executed on SERVERS and CLIENTS machines respectively. Then we have an if block to determine if this is running as an beaker test, or if it's being run on the test developer's machine(s) to test it out. The last couple if blocks determine what code to run on this particular machine. As mentioned before, SERVERS and CLIENTS environmental variables will be set to their respective machines' names and exported on both machines.
Obviously, there will have to be some sort of coordination and synchronization between the machines and the execution of the test code on both sides. Beaker offers two utilities for this purpose, beaker-sync-set and beaker-sync-block . beaker-sync-set is used to setting a state on a machine. beaker-sync-block is used to block the execution of the code until a certain state on certain machine(s) are reached. Those familiar with parallel programming can think of this as a barrier operation . The detailed usage information about both of this utilities is below:
beaker-sync-set: It does set the state of the current machine. State can be anything. Syntax: beaker-sync-set -s STATE
beaker-sync-block: It blocks the code and doesn't return until a desired STATE is set on desired machine(s) . You can actually look for a certain state on multiple machines.. Syntax: beaker-sync-block -s STATE [-s STATE1 -s STATE2] machine1 machine2 ...
There are a couple of important points to pay attention. First of, the multihost testing must be on the same chronological order on all machines. For example, the below will fail:
<recipe>
<test role='STANDALONE' name='/distribution/install'/>
<test role='STANDALONE' name='/my/test/number1'/>
<test role='SERVERS' name='/my/multihost/test'/>
</recipe>
<recipe>
<test role='STANDALONE' name='/distribution/install'/>
<test role='CLIENTS' name='/my/multihost/test'/>
</recipe>
This will fail, because the multihost test is the 3rd test on the server side and it's the 2nd test on the client side.. To fix this, you can pad in dummy testcases on the side that has fewer testcases. There is a dummy test that lives in /distribution/utils/dummy for this purpose. So, the above can be fixed as:
<recipe>
<test role='STANDALONE' name='/distribution/install'/>
<test role='STANDALONE' name='/my/test/number1'/>
<test role='SERVERS' name='/my/multihost/test'/>
</recipe>
<recipe>
<test role='STANDALONE' name='/distribution/install'/>
<test role='STANDALONE' name='/distribution/utils/dummy'/>
<test role='CLIENTS' name='/my/multihost/test'/>
</recipe>
One shortcoming of the beaker-sync-block utility is that it blocks forever, so if there are multiple things being done in your test between the hosts, your test will timeout without possibly a lot of code being executed. There is a utility, blockwrapper.exp which can be used to put a limit on how many second it should block. The script lives in /CoreOS/common test, so be sure to add that test before your multihost tests in your recipes. The usage is exactly same as that of beaker-sync-block with the addition of a timeout value at the end, i.e.:
blockwrapper.exp -s STATE machine N
where N is the timeout value in seconds. If the desired state in the desired machine(s) haven't been set in N seconds, then the script will exit with a non-zero return code. In case of success it'll exit with code 0 .
3.2.1.10.1. A detailed example of multihost testing
For a detailed example of multihost testing, look at migration testing in $TOP/virt/xen/migrate . There, a machine is running as an origin machine with the role of "MIGRATE_FROM", which does export in /var/lib/xen directory as an NFS share, sets up passwordless-ssh between machines, copies over guest domain config files and then migrates the domains.
A workflow is a model to represent real work for further assessment. More abstractly, a workflow is a pattern of activity enabled by a systematic organizatio information flows, into a work process that can be documented and learned. Workflows are designed to achieve processing intents of some sort, such as information processing.
You can now specify Beaker jobs using an xml file that will be workflow agnostic. This allows users a more consistent, maintainable ways of storing and submitting jobs. You can now specify and save entire jobs, including many recipes and recipesets in them, in xml files and save them as regression testsuites and such. Here is a barebone XML file:
<job>
<workflow>Reserve ia64</workflow>
<submitter>tester@redhat.com</submitter>
<whiteboard>Reserving ia64 machine</whiteboard>
<recipeSet>
<recipe testrepo='development'>
<distroRequires>ARCH = ia64</distroRequires>
<distroRequires>FAMILY = RedHatEnterpriseLinuxServer5</distroRequires>
<distroRequires>NAME = RHEL5-Server-U2-RC-1</distroRequires>
<test role='STANDALONE' name='/distribution/install'/>
<test role='STANDALONE' name='/distribution/reservesys'>
<params>
<param name='RESERVEBY' value='gozen@redhat.com'/>
<param name='RESERVETIME' value='86400'/>
</params>
</test>
</recipe>
</recipeSet>
</job>
All xml tags above are pretty self explanatory. One of the biggest advantages of using xml for your jobs is not having to use different workflows for different types of tests. There is only one workflow, submit_job.py namely, that's being used. It does not have all the functionality/binding for all the workflows out there, but we are working on it to add more workflow functionalities into our xml binding and submit_job.py script. Currently as of Nov 2008, you should be able to use submit_job.py for any workflows that was used previously but not all cases have been tested yet and we are working on this .
The easiest way to get started on the xml definition would be seeing if your workflow has a way to provide xml definition for the job. For example, single_package workflow has -xml argument which prints out an xml definition of the job.
Below is a list of XML elements for Beaker:
<job> : It's the root element of any Beaker job definition. Has no attributes.
<workflow> : The name of the workflow. Has no attributes.
<submitter> : Name, email or other identifying string of the job submitter. Has no attributes.
<whiteboard> : A whiteboard string of anything descriptive about the job. Has no attributes.
<recipeSet> : Set of recipes, has multiple recipe child nodes. Has no attributes.
<recipe> : Test recipe itself. Has attributes of testrepo, whiteboard, bootargs .
<installPackage> : Package names to be installed. These will be appended to kickstart's %packages section.
<hostRequires> : Set host properties here.
<distroProperties> : Set properties of the distribution you'd like to install here.
<kickstart> : Pass in your custom kickstart with this element.
<accesskey> : Specify the accesskey for certain systems here, if you need to.
<partition> : Root element of partition definition
<name> : name of the partition. If you are intending the partition to be used as partition , then just give the mountpoint, such as /mnt/myblockdevice . If you'd rather have the spaces used for LVM , then give a logical volume name, for example, mylogvol .
<type> : This is either part, for partition , or lvm , for logical volume.
<size> : Size of the partition in Gigabytes.
<test> : Specify the tests to run. Has attributes testrole and name . Can have the params child elements.
<params> : Root element for parameters to be passed to the test.
<param> : It has name and value attributes with for setting name and value of the environmental variables respectively.
<guestrecipe> : This is a pseudo-recipe for virtual machine guests. Can have almost everything <recipe> can have. You can specify distroProperties, installPackage, etc. here. The attributes of this element are vitally important. First attribute is, guestname, which is the name you'd like to give to the guest. You can omit guestname, in which case Beaker will assign the hostname of the guest as the name of the guest. The second attribute is guestargs , which is identical to arguments that are given to virtinstall program, except for -lvm or -part argument which indicates if the guest is to be installed on blockdevice or lvm volume respectively.