Regression Testing for CLARAty

Regression Testing

• Objectives:
  • Ensure changes do not break existing functionality
  • Ensure that software meets requirements
• Write tests as part of development process:
  1. Define a requirement
  2. Write a test that checks that requirement
  3. Modify your code until it passes the test
  4. Repeat

Regression Testing Conventions in CLARAty

• Tests will be run automatically as part of nightly build
• Appropriate tests should be run manually prior to committing changes
• Filenames:
  • Source files beginning with ``test_'' are regression tests
  • Ideally, there should be a test_ for each library source file
  • Note: programs that are for demos or interactive use will be named ``demo_''
• Program conventions
  • Program will perform any number of tests
  • Program will return an int indicating number of tests that failed
  • 0 indicates all tests were successful
• CppUnit to be used for structuring programs
  • C++ unit testing framework
  • helper macros for easier test suite declaration
  • text based and graphical test runners
  • runs on VxWorks

Example CppUnit Program in CLARAty (Thanks to Anne Wright)

// <sandbox>/src/points/utest_point
#include "claraty/common_defs.h"
#include "claraty/cppunit_defs.h"
#include <cppunit/ui/text/TestRunner.h>
#include <cppunit/TestResultCollector.h>
#include <cppunit/extensions/HelperMacros.h>
#include "claraty/point_t.h"

// Create a class that contains the suite of tests
class Utest_Point : public CppUnit::TestFixture {

  // Macros expand to code that defines a registerTests() method
  // that registers the specified tests to this suite
  CPPUNIT_TEST_SUITE (Utest_Point);
  CPPUNIT_TEST (test_equality);
  CPPUNIT_TEST (test_math);
  CPPUNIT_TEST (test_failure);
  CPPUNIT_TEST_SUITE_END ();
  
  // Members that can be used throughout tests
  Point<double> _p, _q;

public:
  Utest_Point() : _p(5.3, 6.3, 1.2), _q(1.0, -1.0, 3.0) {
  }

  // Can define a setUp() method to initialize things
  // Will be called automatically
  // Preferable to put code here rather than constructor as CppUnit
  // will check for exceptions
  void setUp() { 
    ... 
  }

  // Can define a tearDown() method to clean up
  // Will be called automatically
  // Preferable to put code here rather than destructor as CppUnit
  // will check for exceptions
  void tearDown() {
    ...
  }

  void test_equality() {
    CPPUNIT_ASSERT( _p == _p );
    CPPUNIT_ASSERT( _p != _q );

    Point<double> pc(_p);
    CPPUNIT_ASSERT( _p == pc );

    Point<double> pe(_p.get_x(), _p.get_y(), _p.get_z());
    CPPUNIT_ASSERT( _p == pe );
    CPPUNIT_ASSERT( pc == pe );
  }    

  void test_math() {
    CPPUNIT_ASSERT(_p+_q == Point<double>(6.3, 5.3, 4.2));
    CPPUNIT_ASSERT(_p-_q == Point<double>(4.3, 7.3, -1.8));
    CPPUNIT_ASSERT(_p+_q == _q+_p);
    CPPUNIT_ASSERT(_p-_q == (_q*-1)+_p);
  }

  void test_failure() {
    CPPUNIT_ASSERT(_p == _q);
  }

};

int 
main(int argc, char** argv)
{
  // Create the test runner
  CppUnit::TextUi::TestRunner runner;
  // Add test suite to runner
  runner.addTest (Utest_Point::suite());

  CPPUNIT_SET_OUTPUT (argc, argv, runner);

  // run the test
  runner.run();

  // check the results
  CppUnit::TestResultCollector& result_collector = runner.result();
  // uncaught exceptions
  int num_errors = result_collector.testErrors();
  // failed tests
  int num_failures = result_collector.testFailures();

  // return the sum of errors and failures
  // 0 means success
  return num_errors + num_failures;
}

Sample Output

# build example CppUnit program
> cd /claraty/src/points
> setenv USE_CPPUNIT 1
> ymk all

# run the test
> ../../bin/Drun ix86-linux-gcc3.2-glibc2.3/utest_point
...F
 
!!!FAILURES!!!
Test Results:
Run:  3   Failures: 1   Errors: 0
 
 
1) test: 11Utest_Point.test_failure (F) line: 41 /home/gaines/projects/nightly_r8_rover/claraty/include/claraty/utest_point.h
 "_p == _q"

# confirm return value 
> echo $?
1

Other Useful CppUnit Features

• Checking for exceptions (from CppUnit Cookbook)

  CPPUNIT_TEST_SUITE( ComplexNumberTest );
  // [...]
  CPPUNIT_TEST_EXCEPTION( testDivideByZeroThrows, MathException );
  CPPUNIT_TEST_SUITE_END();
  
  // [...]
  
    void testDivideByZeroThrows()
    {
      // The following line should throw a MathException.
      *m_10_1 / ComplexNumber(0);
    }
  

Getting More Information on CppUnit

• CppUnit Wiki
• CppUnit Cookbook (min-tutorial)
• CppUnit doxygen documentation

Notes on Regression Test Running

For each <arch>
  For each <module>
    cd <module>
    for each <program> in test_*
      number_of_errors = run <arch>/<program>
      log results

Notes

Will need to handle programs crashing. E.g., on Linux, if the program seg faults test run is aborted and an error code is returned.

May need to run a watchdog to check for programs that appear to be in an infinite loop (e.g. no test is allowed to run for more than x minutes).

Transition Plan

• Currently, not all programs that begin with ``test_'' are regression test programs
  • eventually these need to be renamed ``demo_''
• Each module will have a file called ``regression_tests.txt''
  • It will consist of a list of test program names (one per line)
  • The regression test script will use this file to determine which tests to run
• After all files have been named correctly, the regression_tests.txt files can be retired