{{TableOfContents}}

Overview of Steps [edit section]

= Overview of Steps =
To add a new test to Fuego, you need to perform the following steps:

• 1. Decide on a test name and type * 2. Make the test directory * 3. Get the source (or binary) for the test * 4. Write a test script for the test * 5. Add the test_specs (if any) for the test * 6. (if a functional test) Add "expected results" (p/n) logs for the test * 7. (if a benchmark) Add parser.py and reference.log files * 8. Create the Jenkins test configuration for the test

 * 1. Decide on a test name and type
 * 2. Make the test directory
 * 3. Get the source (or binary) for the test
 * 4. Write a test script for the test
 * 5. Add the test_specs (if any) for the test
 * 6. (if a functional test) Add "expected results" (p/n) logs for the test
 * 7. (if a benchmark) Add parser.py and reference.log files
 * 8. Create the Jenkins test configuration for the test

Decide on a test name [edit section]

= Decide on a test name =
The first step to creating a test is deciding the test name.  There are two
types of tests supported by Fuego: functional tests and benchmark tests.
A functional test either passes or fails, while a benchmark test produces one or more numbers representing some performance metric for the system.

Usually, the name of the test will be a combination of the test type
and a name to identify the test itself.  Here are some examples: "bonnie" is a popular disk performance test.  The name of this test in the fuego system is ''Benchmark.bonnie''.  A test which runs portions of the posix test suite is a functional test (it either passes or fails), and in Fuego is named ''Functional.posixtestsuite''.  The test name should be all one word (no spaces).

This name is used as the directory name where the test materials will live in the Fuego system.

Create the directory for the test [edit section]

= Create the directory for the test =
The main test directory is located in /home/jenkins/fuego/engine/tests/''<test_name>''

So if you just created a new Functional test called 'foo', you would create the directory:
 * /home/jenkins/fuego/engine/tests/Functional.foo

Get the source for a test [edit section]

= Get the source for a test =
The actual creation of the test program itself is outside the scope of Fuego.  Fuego is intended to execute an existing test program, for which source code or a
script already exists.

This page describes how to integrate such a test program into the Fuego test system.

A test program in Jenkins is provided in source form so that it can be compiled for whatever processor architecture is used by the target under test. This source must be in the form of a tarfile, and one or more patches.

Create a tarfile for the test, by downloading the test source manually, and
creating the tarfile.

The tarfile may be compressed.  Supported compression schemes, and their associated extensions are:
 * uncompressed (extension='.tar')
 * compressed with gzip (extension='.tar.gz' or '.tgz')
 * compressed with bzip2 (extension='.bz2')

Test script [edit section]

= Test script =
The test script is a small script, written the shell scripting language.  It
specifies the source tarfile containing the test program, and provides
implementations for the functions needed to build, deploy, execute, and evaluate the results from the test program.

The test script for a functional test should contain the following:
 * tarfile
 * function test_pre_check (optional)
 * function test_build
 * function test_deploy
 * function test_run
 * function test_processing

The test_pre_check function is optional, and is used to check that 
the test environment and target configuration and setup are correct
in order to run the test.

Sample test script [edit section]

== Sample test script ==
Here is the test script for the test Functional.hello_world.  This script demonstrates a lot of the core elements of a test script.  The name of this script is hello_world.sh.

{{{#!YellowBox
#!/bin/bash

tarball=hello-test-1.0.tgz

function test_build {
    make && touch test_suite_ready || build_error "error while building test"
}

function test_deploy {
    put hello  $FUEGO_HOME/fuego.$TESTDIR/
}

function test_run {
    report "cd $FUEGO_HOME/fuego.$TESTDIR; ./hello $FUNCTIONAL_HELLO_WORLD_ARG"
}

function test_processing {
    log_compare "$TESTDIR" "1" "SUCCESS" "p"          
}

. $FUEGO_SCRIPTS_PATH/functional.sh
}}}

Description of base test functions [edit section]

== Description of base test functions ==
The base test functions (test_build, test_deploy, test_run, and test_processing) are fairly simple.  Each one contains a few statements to accomplish that phase of the test execution.

You can find more information about each of these functions at the following links:
 * [[function_test_pre_check|test_pre_check]]
 * [[function_test_build|test_build]]
 * [[function_test_deploy|test_deploy]]
 * [[function_test_run|test_run]]
 * [[function_test_processing|test_processing]]

Test spec and plan [edit section]

= Test spec and plan =
Define the test spec(s) for this test, and add an entry to the testplan_default file for it.

Each test in the system can must have a test spec file.  This file
can be used to list customizable variables for the test.

If a test program has no customizable variables, or none are desired,
then at a minimum a "default" test spec must be defined.

The test spec file is:
 * in JSON format, 
 * have the same name as the test, with a '.json' extension (e.g. Functional.hello_world.json),
 * should be placed in the directory /home/jenkins/engine/overlays/test_specs,
 * provides a testName attribute, and a specs attribute, which is a list,
 * may include any named spec you want, but must define at least the 'default' spec for the test
   * Note that the 'default' spec can be empty, if desired.

Here is an example one that defines no variables.

{{{#!YellowBox
{
    "testName": "Benchmark.openssl",
    "specs": 
    [
        {
            "name":"default"
        }
    ]
}
}}}

Next, add an entry in the file /home/jenkins/engine/overlays/testplans/testplan_default

Edit this file, and add an entry indicating to use the 'default' test
spec when this testplan is used:

Here is what the added entry looks like for the Functional.hello_world test:
{{{#!YellowBox
        {
            "testName": "Functional.hello_world",
            "spec": "default"
        }
}}}

Note that you should add a comma after your entry, if it is not the last
one in the list of "tests".

Please read [[Test Specs and Plans]] for more details.  You should do this
if you have a filesystem test, or want to create your own test specs and 
test plans to add flexibility to your test execution, please read

test results parsers and reference log files [edit section]

= test results parsers and reference log files =
FIXTHIS - add information about results parsing
 * log_compare function
 * <name>_p.log, <name>_n.log
 * parser.py
 * reference.log
 * tests.info

Jenkins test definition file [edit section]

= Jenkins test definition file =
The last step in creating the test is to create the Jenkins configuration for it.

Go into the Jenkins interface, and select "Home" to get to the home page.
Then, select "New Test" in the left navigation bar.

A dialog will appear.

FIXTHIS - add screenshot of "new test" dialog

Enter a test name, which should be something like: Functional.mytest

Select to copy from an existing Test, and, if it's a functional test, copy from Functional.hello_world.  That is, select the radio button "Copy existing Test", and enter "Functional.hello_world" in the "Copy from" box.

The next dialog to appear will be the Jenkins configuration page for the test.

FIXTHIS - add screenshot of "configure test" dialog, showing Command box.

There are lots of things you can customize on this page, but you can skip
most of the items, and just modify the following fields:
 * Description - give a brief description of the test in English
 * under Dynamic Choice Parameter (Scriptler), Parameters, Parameter value, change the name to Functional.mytest.
 * under Test Run, Execute shell, inside the Command box, change the name of the shell script to run in the 'source' line.  It should read something like this:

{{{#!YellowBox
source $FUEGO_TESTS_PATH/$JOB_NAME/mytest.sh
}}} 

This results in the creation of a file called config.xml, in the /var/lib/jenkins/jobs/<test_name> directory (which is symlinked to /home/jenkins/fuego/jobs/<test_name>)

Publishing the test [edit section]

= Publishing the test =
Tests that are of general interest should be submitted for inclusion into fuego-core.

Right now, the method of doing this is to create a commit and send that commit
to the fuego mailing list, for review, and hopefully acceptance and
integration by the fuego maintainers.

technical details [edit section]

= technical details =
This section has technical details about a test.

Directory structure [edit section]

== Directory structure ==
The directory structure used by Fuego is documented at [[Fuego directories]]

Files [edit section]

== Files ==
A test consists of the following files or items:

{{{#!Table
||File or item||format||location||description||test type||
||config.xml  ||Jenkins XML||/var/lib/jenkins/jobs/{test_name} (aka /home/jenkins/fuego/jobs/{test_name}||Has the Jenkins (front-end) configuration for the test||all||
||tarfile     ||tar format ||/home/jenkins/tests/{test_name}||Has the source code for the test program||all||
||patches     ||patch format|| /home/jenkins/tests/{test_name}||Zero or more patches to customize the test program (applied during the unpack phase||all||
||base script ||shell script||/home/jenkins//tests/{test_name}||Is the shell script that implements the different test phases in Fuego||all||
||test spec   ||JSON        ||/home/jenkins/fuego/engine/overlays/test_specs||Has groups of variables (and their values) that can be used with this test||all||
||test plan(s)||JSON        ||/home/jenkins/fuego/engine/overlays/testplans/testplan_default.json (and others)||Has the testplan(s) for this test||all||
||p/n logs    ||text        ||/home/jenkins/tests/{test_name}||Are logs with the results (positive or negative) parsed out, for determination of test pass/fail||functional only||
||parser.py   ||python      || /home/jenkins/tests/{test_name}||Python program to parse benchmark metrics out of the log, and provide a dictionary to the Fuego plotter||benchmarks only||
||reference.log||Fuego-specific|| /home/jenkins/{test_name}||Has the threshold values and comparison operators for benchmark metrics measured by the test||benchmarks only||
||line added to tests.info||JSON|| /home/jenkins/fuego/jobs/tests.info||provides the names of metrics to be plotted for this benchmark||benchmarks only||

}}}