Contents: Overview of Steps Decide on a test name Create the directory for the test Get the source for a test tarball source git source script-based source Test script Sample test script Description of base test functions Test spec and plan Test results parser Pass criteria and reference info Jenkins job definition file Publishing the test Technical Details Directory structure Files

Overview of Steps [edit section]

To add a new test to Fuego, you need to perform the following steps:

• 1. Decide on a test name and type
• 2. Make the test directory
• 3. Get the source (or binary) for the test
• 4. Write a test script for the test
• 5. Add the test_specs (if any) for the test
• 6. Add log processing to the test
• 6-a. (if a benchmark) Add parser.py and criteria and reference files
• 7. Create the Jenkins test configuration for the test

Decide on a test name [edit section]

The first step to creating a test is deciding the test name. There are two types of tests supported by Fuego: functional tests and benchmark tests. A functional test either passes or fails, while a benchmark test produces one or more numbers representing some performance measurements for the system.

Usually, the name of the test will be a combination of the test type and a name to identify the test itself. Here are some examples: "bonnie" is a popular disk performance test. The name of this test in the fuego system is Benchmark.bonnie. A test which runs portions of the posix test suite is a functional test (it either passes or fails), and in Fuego is named Functional.posixtestsuite. The test name should be all one word (no spaces).

This name is used as the directory name where the test materials will live in the Fuego system.

Create the directory for the test [edit section]

The main test directory is located in /fuego-core/engine/tests/<test_name>

So if you just created a new Functional test called 'foo', you would create the directory:

• /fuego-core/engine/tests/Functional.foo

Get the source for a test [edit section]

The actual creation of the test program itself is outside the scope of Fuego. Fuego is intended to execute an existing test program, for which source code or a script already exists.

This page describes how to integrate such a test program into the Fuego test system.

A test program in Fuego is provided in source form so that it can be compiled for whatever processor architecture is used by the target under test. This source may be in the form of a tarfile, or a reference to a git repository, and one or more patches.

Create a tarfile for the test, by downloading the test source manually, and creating the tarfile. Or, note the reference for the git repository for the test source.

tarball source [edit section]

If you are using source in the form of a tarfile, you add the name of the tarfile (called 'tarball') to the test script.

The tarfile may be compressed. Supported compression schemes, and their associated extensions are:

• uncompressed (extension='.tar')
• compressed with gzip (extension='.tar.gz' or '.tgz')
• compressed with bzip2 (extension='.bz2')

For example, if the source for your test was in the tarfile 'foo-1.2.tgz' you would add the following line to your test script, to reference this source:

  tarball=foo-1.2.tgz

git source [edit section]

If you are using source from an online git repository, you reference this source by adding the variables 'gitrepo' and 'gitref' to the test script.

In this case, the 'gitrepo' is the URL used to access the source, and the 'gitref' refers to a commit id (hash, tag, version, etc.) that refers to a particular version of the code.

For example, if your test program is built from source in an online 'foo' repository, and you want to use version 1.2 of that (which is tagged in the repository as 'v1.2', on the master branch, you might have some lines like the following in the test's script.

  gitrepo=http://github.com/sampleuser/foo.git
  gitref=master/v1.2

script-based source [edit section]

Some tests are simple enough to be implemented as a single script (that runs on the board). For these tests, no additional source is necessary, and the script can just be placed directly in the test's home directory. In fuego_test.sh you must set the following variable:

local_source=1

During the deploy phase, the script is sent to the board directly from the test home directory instead of from the test build directory.

Test script [edit section]

The test script is a small shell script called fuego_test.sh. It specifies the source tarfile containing the test program, and provides implementations for the functions needed to build, deploy, execute, and evaluate the results from the test program.

The test script for a functional test should contain the following:

• source reference (either tarball or gitrepo and gitref)
• function test_pre_check (optional)
• function test_build
• function test_deploy
• function test_run
• function test_processing

The test_pre_check function is optional, and is used to check that the test environment and target configuration and setup are correct in order to run the test.

Sample test script [edit section]

Here is the fuego_test.sh script for the test Functional.hello_world. This script demonstrates a lot of the core elements of a test script.

#!/bin/bash

tarball=hello-test-1.0.tgz

function test_build {
    make
}

function test_deploy {
    put hello  $BOARD_TESTDIR/fuego.$TESTDIR/
}

function test_run {
    report "cd $BOARD_TESTDIR/fuego.$TESTDIR; ./hello $FUNCTIONAL_HELLO_WORLD_ARG"
}

function test_processing {
    log_compare "$TESTDIR" "1" "SUCCESS" "p"
}

Description of base test functions [edit section]

The base test functions (test_build, test_deploy, test_run, and test_processing) are fairly simple. Each one contains a few statements to accomplish that phase of the test execution.

You can find more information about each of these functions at the following links:

• test_pre_check
• test_build
• test_deploy
• test_run
• test_processing

Test spec and plan [edit section]

Another element of every test is the "test spec". A file is used to define a set of parameters that are used to customize the test for a particular use case.

You must define the test spec(s) for this test, and add an entry to the appropriate testplan for it.

Each test in the system must have a test spec file. This file is used to list customizable variables for the test.

If a test program has no customizable variables, or none are desired, then at a minimum a "default" test spec must be defined, with no test variables.

The test spec file is:

• named 'spec.json' in the test directory,
• in JSON format,
• provides a testName attribute, and a specs attribute, which is a list,
• may include any named spec you want, but must define at least the 'default' spec for the test
  • Note that the 'default' spec can be empty, if desired.

Here is an example one that defines no variables.

{
    "testName": "Benchmark.OpenSSL",
    "specs": {
        "default": {}
    }
}

And here is the spec.json of the Functional.hello_world example, which defines three specs:

{
    "testName": "Functional.hello_world",
    "specs": {
        "hello-fail": {
            "ARG":"-f"
        },
        "hello-random": {
            "ARG":"-r"
        },
        "default": {
            "ARG":""
        }
    }
}

Next, you may want to add an entry to one of the testplan files. These files are located in the directory /fuego-core/engine/overlays/testplans.

Choose a testplan you would like to include this test, and edit the corresponding file. For example, to add your test to the list of tests executed when the 'default' testplan is used, add an entry default to the 'testplan_default.json' file.

Note that you should add a comma after your entry, if it is not the last one in the list of "tests".

Please read Test Specs and Plans for more details.

Test results parser [edit section]

Each test should also provide some mechanism to parse the results from the test program, and determine the success of the test.

For a simple Functional test, you can use the log_compare function to specify a pattern to search for in the test log, and the number of times that pattern should be found in order to indicate success of the test. This is done from the test_processing function in the test script.

Here is an example of a call to log_compare:

function test_processing {
    log_compare "$TESTDIR" "11" "^TEST.*OK" "p"
}

This example looks for the pattern "^TEST.*OK", which finds lines in the test log that start with the word 'TEST' and are followed by the string 'OK' on the same line. It looks for this pattern 11 times.

log_compare can be used to parse the logs of simple tests with line-oriented output.

For tests with more complex output, and for Benchmark tests that produce numerical results, you must add a python program called 'parser.py', which scans the test log and produces a data structure used by other parts of the Fuego system.

See parser.py for information about this program.

Pass criteria and reference info [edit section]

You should also provide information to Fuego to indicate how to evaluate the ultimate resolution of the test.

For a Functional test, it is usually the case that the whole test passes only if all individual test cases in the test pass. That is, one error in a test case indicates overall test failure. However, for Benchmark tests, the evaluation of the results is more complicated. It is required to specify what numbers constitute success vs. failure for the test.

Also, for very complicated Functional tests, there may be complicated results, where, for example, some results should be ignored.

You can specify the criteria used to evaluate the test results, by creating a 'criteria.json' file for the test.

Finally, you may wish to add a file that indicates certain information about the test results. This information is placed in the 'reference.json' file for a test.

Please see the links for those files to learn more about what they are and how to write them, and customize them for your system.

Jenkins job definition file [edit section]

The last step in creating the test is to create the Jenkins job for it.

A Jenkins job describes to Jenkins what board to run the test on, what variables to pass to the test (including the test spec (or variant), and what script to run for the test.

Jenkins jobs are created using the command-line tool 'ftc'.

A Jenkin job has the name <node_name>.<spec>.<test_type>.<test_name>

You create a Jenkins job using a command like the following:

• $ ftc add-jobs -b myboard -t Functional.mytest [-s default]

The ftc 'add-jobs' sub-command uses '-b' to specify the board, '-t' to specify the test, and '-s' to specify the test spec that will be used for this Jenkins job.

In this case, the name of the Jenkins job that would be created would be:

• myboard.default.Functional.mytest

This results in the creation of a file called config.xml, in the /var/lib/jenkins/jobs/<job_name> directory.

Publishing the test [edit section]

Tests that are of general interest should be submitted for inclusion into fuego-core.

Right now, the method of doing this is to create a commit and send that commit to the fuego mailing list, for review, and hopefully acceptance and integration by the fuego maintainers.

In the future, a server will be provided where test developers can share tests that they have created in a kind of "test marketplace". Tests will be available for browsing and downloading, with results from other developers available to compare with your own results. There is already preliminary support for packaging a test using the 'ftc package-test' feature. More information about this service will be made available in the future.

Technical Details [edit section]

This section has technical details about a test.

Directory structure [edit section]

The directory structure used by Fuego is documented at Fuego directories

Files [edit section]

A test consists of the following files or items:

File or item  ^	format  ^	location  ^	description  ^	test type  ^
(deprecated) reference.log	Fuego-specific	/fuego-core/engine/tests/{test_name}	Has the threshold values and comparison operators for benchmark metrics measured by the test	benchmarks only
reference info	JSON	/fuego-core/engine/tests/{test_name}/reference.json	Has additional information about test results, such as the units for benchmark measurements	benchmark only
pass criteria	JSON	/fuego-core/engine/tests/{test_name}/criteria.json	Has the "pass" criteria for the test	all
test spec	JSON	/fuego-core/engine/tests/{test_name}/spec.json	Has groups of variables (and their values) that can be used with this test	all
test plan(s)	JSON	/fuego-core/engine/overlays/testplans	Has the testplans for the entire system	all
config.xml	Jenkins XML	/var/lib/jenkins/jobs/{test_name}	Has the Jenkins (front-end) configuration for the test	all
patches	patch format	/fuego-core/engine/tests/{test_name}	Zero or more patches to customize the test program (applied during the unpack phase	all
parser.py	python	/fuego-core/engine/tests/{test_name}	Python program to parse benchmark metrics out of the log, and provide a dictionary to the Fuego plotter	all, but can be missing for Functional tests
base script	shell script	/fuego-core/engine/tests/{test_name}/fuego_test.sh	Is the shell script that implements the different test phases in Fuego	all
tarfile	tar format	/fuego-core/engine/tests/{test_name}	Has the source code for the test program	all
(deprecated) p/n logs	text	/fuego-core/engine/tests/{test_name}	Are logs with the results (positive or negative) parsed out, for determination of test pass/fail	functional only

Query