Contents: Overview of Steps Decide on a test name Create the directory for the test Get the source for a test Test script Sample test script Description of base test functions Test spec and plan test results parsers and reference log files Jenkins test definition file Publishing the test technical details Directory structure Files

Overview of Steps [edit section]

To add a new test to Fuego, you need to perform the following steps:

• 1. Decide on a test name and type
• 2. Make the test directory
• 3. Get the source (or binary) for the test
• 4. Write a test script for the test
• 5. Add the test_specs (if any) for the test
• 6. (if a functional test) Add "expected results" (p/n) logs for the test
• 7. (if a benchmark) Add parser.py and reference.log files
• 8. Create the Jenkins test configuration for the test

Decide on a test name [edit section]

The first step to creating a test is deciding the test name. There are two types of tests supported by Fuego: functional tests and benchmark tests. A functional test either passes or fails, while a benchmark test produces one or more numbers representing some performance metric for the system.

Usually, the name of the test will be a combination of the test type and a name to identify the test itself. Here are some examples: "bonnie" is a popular disk performance test. The name of this test in the fuego system is Benchmark.bonnie. A test which runs portions of the posix test suite is a functional test (it either passes or fails), and in Fuego is named Functional.posixtestsuite. The test name should be all one word (no spaces).

This name is used as the directory name where the test materials will live in the Fuego system.

Create the directory for the test [edit section]

The main test directory is located in /home/jenkins/fuego/engine/tests/<test_name>

So if you just created a new Functional test called 'foo', you would create the directory:

• /home/jenkins/fuego/engine/tests/Functional.foo

Get the source for a test [edit section]

The actual creation of the test program itself is outside the scope of Fuego. Fuego is intended to execute an existing test program, for which source code or a script already exists.

This page describes how to integrate such a test program into the Fuego test system.

A test program in Jenkins is provided in source form so that it can be compiled for whatever processor architecture is used by the target under test. This source must be in the form of a tarfile, and one or more patches.

Create a tarfile for the test, by downloading the test source manually, and creating the tarfile.

The tarfile may be compressed. Supported compression schemes, and their associated extensions are:

• uncompressed (extension='.tar')
• compressed with gzip (extension='.tar.gz' or '.tgz')
• compressed with bzip2 (extension='.bz2')

Test script [edit section]

The test script is a small script, written the shell scripting language. It specifies the source tarfile containing the test program, and provides implementations for the functions needed to build, deploy, execute, and evaluate the results from the test program.

The test script for a functional test should contain the following:

• tarfile
• function test_pre_check (optional)
• function test_build
• function test_deploy
• function test_run
• function test_processing

The test_pre_check function is optional, and is used to check that the test environment and target configuration and setup are correct in order to run the test.

Sample test script [edit section]

Here is the test script for the test Functional.hello_world. This script demonstrates a lot of the core elements of a test script. The name of this script is hello_world.sh.

#!/bin/bash

tarball=hello-test-1.0.tgz

function test_build {
    make && touch test_suite_ready || build_error "error while building test"
}

function test_deploy {
    put hello  $FUEGO_HOME/fuego.$TESTDIR/
}

function test_run {
    report "cd $FUEGO_HOME/fuego.$TESTDIR; ./hello $FUNCTIONAL_HELLO_WORLD_ARG"
}

function test_processing {
    log_compare "$TESTDIR" "1" "SUCCESS" "p"
}

. $FUEGO_SCRIPTS_PATH/functional.sh

Description of base test functions [edit section]

The base test functions (test_build, test_deploy, test_run, and test_processing) are fairly simple. Each one contains a few statements to accomplish that phase of the test execution.

You can find more information about each of these functions at the following links:

• test_pre_check
• test_build
• test_deploy
• test_run
• test_processing

Test spec and plan [edit section]

Define the test spec(s) for this test, and add an entry to the testplan_default file for it.

Each test in the system can must have a test spec file. This file can be used to list customizable variables for the test.

If a test program has no customizable variables, or none are desired, then at a minimum a "default" test spec must be defined.

The test spec file is:

• in JSON format,
• have the same name as the test, with a '.json' extension (e.g. Functional.hello_world.json),
• should be placed in the directory /home/jenkins/engine/overlays/test_specs,
• provides a testName attribute, and a specs attribute, which is a list,
• may include any named spec you want, but must define at least the 'default' spec for the test
  • Note that the 'default' spec can be empty, if desired.

Here is an example one that defines no variables.

{
    "testName": "Benchmark.openssl",
    "specs":
    [
        {
            "name":"default"
        }
    ]
}

Next, add an entry in the file /home/jenkins/engine/overlays/testplans/testplan_default

Edit this file, and add an entry indicating to use the 'default' test spec when this testplan is used:

Here is what the added entry looks like for the Functional.hello_world test:

        {
            "testName": "Functional.hello_world",
            "spec": "default"
        }

Note that you should add a comma after your entry, if it is not the last one in the list of "tests".

Please read Test Specs and Plans for more details. You should do this if you have a filesystem test, or want to create your own test specs and test plans to add flexibility to your test execution, please read

test results parsers and reference log files [edit section]

FIXTHIS - add information about results parsing

• log_compare function
• <name>_p.log, <name>_n.log
• parser.py
• reference.log
• tests.info

Jenkins test definition file [edit section]

The last step in creating the test is to create the Jenkins configuration for it.

Go into the Jenkins interface, and select "Home" to get to the home page. Then, select "New Test" in the left navigation bar.

A dialog will appear.

FIXTHIS - add screenshot of "new test" dialog

Enter a test name, which should be something like: Functional.mytest

Select to copy from an existing Test, and, if it's a functional test, copy from Functional.hello_world. That is, select the radio button "Copy existing Test", and enter "Functional.hello_world" in the "Copy from" box.

The next dialog to appear will be the Jenkins configuration page for the test.

FIXTHIS - add screenshot of "configure test" dialog, showing Command box.

There are lots of things you can customize on this page, but you can skip most of the items, and just modify the following fields:

• Description - give a brief description of the test in English
• under Dynamic Choice Parameter (Scriptler), Parameters, Parameter value, change the name to Functional.mytest.
• under Test Run, Execute shell, inside the Command box, change the name of the shell script to run in the 'source' line. It should read something like this:

source $FUEGO_TESTS_PATH/$JOB_NAME/mytest.sh

This results in the creation of a file called config.xml, in the /var/lib/jenkins/jobs/<test_name> directory (which is symlinked to /home/jenkins/fuego/jobs/<test_name>)

Publishing the test [edit section]

Tests that are of general interest should be submitted for inclusion into fuego-core.

Right now, the method of doing this is to create a commit and send that commit to the fuego mailing list, for review, and hopefully acceptance and integration by the fuego maintainers.

technical details [edit section]

This section has technical details about a test.

Directory structure [edit section]

The directory structure used by Fuego is documented at Fuego directories

Files [edit section]

A test consists of the following files or items:

File or item  ^	format  ^	location  ^	description  ^	test type  ^
line added to tests.info	JSON	/home/jenkins/fuego/jobs/tests.info	provides the names of metrics to be plotted for this benchmark	benchmarks only
patches	patch format	/home/jenkins/tests/{test_name}	Zero or more patches to customize the test program (applied during the unpack phase	all
parser.py	python	/home/jenkins/tests/{test_name}	Python program to parse benchmark metrics out of the log, and provide a dictionary to the Fuego plotter	benchmarks only
reference.log	Fuego-specific	/home/jenkins/{test_name}	Has the threshold values and comparison operators for benchmark metrics measured by the test	benchmarks only
base script	shell script	/home/jenkins//tests/{test_name}	Is the shell script that implements the different test phases in Fuego	all
test spec	JSON	/home/jenkins/fuego/engine/overlays/test_specs	Has groups of variables (and their values) that can be used with this test	all
test plan(s)	JSON	/home/jenkins/fuego/engine/overlays/testplans/testplan_default.json (and others)	Has the testplan(s) for this test	all
p/n logs	text	/home/jenkins/tests/{test_name}	Are logs with the results (positive or negative) parsed out, for determination of test pass/fail	functional only
tarfile	tar format	/home/jenkins/tests/{test_name}	Has the source code for the test program	all
config.xml	Jenkins XML	/var/lib/jenkins/jobs/{test_name} (aka /home/jenkins/fuego/jobs/{test_name}	Has the Jenkins (front-end) configuration for the test	all

Query