CCL | Software | Download | Manuals | Forum | Papers
CCL Home

Research

Software Community Operations

Testing Framework

CCTools includes a small set of test scripts that allow users and developers to verify that the software is in proper working condition. The framework used to construct these tests is based on a simple set of shell scripting conventions:

Test Case

The primary component of the testing framework is the test case script. An example template can be found in dttools/src/test_case.sh.template, which looks like this:

#!/bin/sh

. ../../dttools/src/test_runner.common.sh

prepare()
{
    cd ../src/; make
    exit 0
}

run()
{
    exec ../src/hmac_test
}

clean()
{
    exit 0
}

dispatch $@

The first line of the script is the standard Unix she-bang to denote that is in fact a shell script. The next line imports a set of convenience functions from dttools/src/test_runner.common.sh to be used in the test script.

The remainder of the script consists of defining three shell functions: prepare, run, clean. These functions are normally executed in order and are used to specify the behavior of the test case as described below:

  1. Prepare:

    This function is responsible for setting up the test environment. For instance if the test requires input data to be pre-generated or for a server to be running, that should happen here.

    To signify successful execution, all shell functions should exit 0. Any other exit status will signify failure.

    In the template example above, we simply run make in the source directory to ensure that our test executable is compiled, though this is usually not necessary (especially if tests are ran as part of the build process) and exit 0 as required.

    A more interesting prepare example is shown in chirp/test/TR_001_chirp_benchmark.sh:

        prepare()
        {   
    	port=`find_free_port`
    	../src/chirp_server -p $port &
    	pid=$!
    
    	if ps ux | awk '{print $2}' | grep "^$pid$"; then
    	    echo $port> $PORT_FILE
    	    echo $pid > $PID_FILE
    	    exit 0
    	else
    	    exit 1
    	fi
        }
        

    In this case, we find a free port using the find_free_port which is provided by the dttools/src/test_runner.common.sh. We then use that port to start a chirp server. Next, we check that the server in fact started by verifying that the PID is active. If so, we store the port and pid for use in subsequent functions, and exit 0. Otherwise, we exit 1 to signify failure.

  2. Run:

    This function is responsible for executing the actual test program with the appropriate arguments. Usually it is sufficient to simply do exec program_test.

    A more interesting run example is shown in chirp/test/TR_001_chirp_benchmark.sh:

        run()
        {
            port=`cat $PORT_FILE`
    	exec ../src/chirp_benchmark localhost:$port $TEST_FILE 2 2 2
        }
        

    In this run function, we load the port stored in the previous prepare function and then execute the test program, chirp_benchmark with the appropriate arguments.

    Once again, to signify successful execution, all shell functions should exit 0. Any other exit status will signify failure.

  3. Clean:

    This function is responsible for cleaning up the environment after the test has been executed. For instance, if any files were generated during the preparation or execution phase, they should be removed. Likewise, if a server was started, then it should be killed in this function.

    Again, chirp/test/TR_001_chirp_benchmark.sh provides an illustrative example:

        clean()
        {   
    	kill -9 `cat $PID_FILE`
    	rm -f $TEST_FILE
    	rm -f $PID_FILE
    	rm -f $PORT_FILE
    	rm -f .__acl
    	exit 0
        }
        

    In this clean function, we kill the chirp server with the pid we stored in the prepare phase. Next we delete any files we generated in the prepare and run phases.

The final line of the test case script is calls the dispatch function provided by dttools/src/test_runner.common.sh. This function allows users to execute each phase independently by passing the phase name as an argument to the test case script:

$ ./test_case.sh prepare
$ ./test_case.sh run
$ ./test_case.sh clean 

This is convenient during debugging, as you can run each phase independently and fix them as necessary.

Test Runner

To simplify the execution of test cases, we also provide a wrapper script dttools/src/test_runner.sh. This script will execute the three test case phases in order and provide a pretty summary:
$ cd dttools/test
$ ../../dttools/src/test_runner.sh TR_001_hmac_test.sh
testing TR_001_hmac_test.sh ... ok
For more verbose information, you can pass the -v flag:
$ ../../dttools/src/test_runner.sh -v TR_001_hmac_test.sh
testing TR_001_hmac_test.sh ... 
  prepare ... ok
  run ... ok
  clean ... ok

Note, that test_runner.sh will always execute the clean phase even if the previous two fail.

The execution of the test script, including any unredirected standard output and error is logged to a file, normally ./cctools.test.log. Users may specify the location of the log file to test_runner.sh by using the -l command line option:

$ ../../dttools/src/test_runner.sh -l test.log TR_001_hmac_test.sh
testing TR_001_hmac_test.sh ... ok
$ cat test.log
--- testing TR_001_hmac_test.sh
MD5 Tests
===========
MD5 Test 1 ref:         0x9294727a3638bb1c13f48ef8158bfc9d
MD5 Test 1 digest:      0x9294727a3638bb1c13f48ef8158bfc9d

MD5 Test 2 ref:         0x750c783e6ab0b503eaa86e310a5db738
MD5 Test 2 digest:      0x750c783e6ab0b503eaa86e310a5db738

MD5 Test 3 ref:         0x56be34521d144c88dbb8c733f0e8b3f6
MD5 Test 3 digest:      0x56be34521d144c88dbb8c733f0e8b3f6


SHA1 Tests
===========
SHA1 Test 1 ref:        0xb617318655057264e28bc0b6fb378c8ef146be00
SHA1 Test 1 digest:     0xb617318655057264e28bc0b6fb378c8ef146be00

SHA1 Test 2 ref:        0xeffcdf6ae5eb2fa2d27416d5f184df9c259a7c79
SHA1 Test 2 digest:     0xeffcdf6ae5eb2fa2d27416d5f184df9c259a7c79

SHA1 Test 3 ref:        0x125d7342b9ac11cd91a39af48aa17b4f63f175d3
SHA1 Test 3 digest:     0x125d7342b9ac11cd91a39af48aa17b4f63f175d3
=== tested TR_001_hmac_test.sh: ok

Build Integration

To facilitate automated testing, the build system for CCTools includes integrated support for this test framework. Test for each sub-system go in project/test. For instance the dttools tests are in dttools/test. Inside the test directory will be a Makefile that includes the following rule:

test:
        @echo running dttools tests
	@${CCTOOLS_HOME}/dttools/src/test_runner.sh TR_*.sh

This means that if you execute make test, all the test cases that match the pattern TR_*.sh will be executed.

To execute the tests for any sub-system, simply go to the project folder and execute make test. To suppress make noise, run make -s test:

$ cd dttools
$ make -s test
running dttools tests
testing TR_001_hmac_test.sh ... ok
testing TR_002_chunk_test.sh ... ok

Autobuild Logs

Every time you make a commit to the CCTools SVN repository, the autobuild program will build your revision for various platforms and run all the test cases in the SVN branch on each platform. The results of the build and test cases are shown on the autobuild website. For each revision there is a Logs column. The summary file is a copy of the email sent for the build. For each platform there are four files:

  1. Build summary: This provides a summary of the build process, including the detected platform and packages.
  2. Build log: This provides a complete log of the build process and should contain any compilation errors of the build failures. On failure, it will also include a dump of the environment for debugging.
  3. Test summary: This corresponds to capturing the output of make -s tests from the root directory and thus provides a summary of the test successes and failures.
  4. Test log: This corresponds to capturing the output of test_runner.sh for all of the test cases and should be consulted to see why a test cases fails.

With all of this in mind, do your best to write tests any time an interesting bug comes up or as you develop new features. This way we can slowly build a set of tests that will help us capture any regressions in the future.