Beaker provides an automated software testing system that should appeal to a variety of audiences for a variety of purposes. It provides a programming interface for developing automated unit tests, bug reproducers, hardware enablement, and regression tests. Beaker is composed of several components, but the most primary are the Beaker framework for writing the tests and the tests themselves. Everyone is encouraged to submit new tests to the Table Cloth repository and/or download stored tests for their use.
Most developers create new tests in a standalone environments (such as a workstation) first and later deploy them in the automated test lab where they are run on machines dedicated for testing. Many types of automated environments are possible:
running on a farm of real machines in some kind of lab situation
running as part of a continuous checkout/build/test "tinderbox"-style environment
running inside a fake "chroot" tree on your development machine
When tests are developed in accordance with the Beaker framework, they can be run either standalone or in a lab test environment without modification. The Beaker framework defines both an API and the format of required files that contain meta data. When run locally from a command line, the results are reported to stdout
. When deployed in the automated test lab, the results can be stored in a repository and viewed via a UI.
This section will first describe how to write tests within the Beaker framework. Later, once a new test is written and tested, it can be packaged, as a conventional RPM package from its source files, and submitted to the Table Cloth test repository, using Subversion source control system commands. This allows the test to be downloaded by others, but in addtion, it also allows the test to be run in a Fedora lab environment. The Fedora project has a dedicated test lab environment where tests stored in the test repository can be run and the results of the tests made public. The name of this lab is Beaker. The reporting of the test results will be done with the Testify UI.
A test lab has the advantage of being able to automatically schedule the test and collect test results on a variety of architectures and releases. The lab environment is also able to detect certain types of failure. For example, if a dedicated test machine goes into an infinite loop, the test is automatically killed and a failure is reported.