test.yaml

The test.yaml file is used to hold meta-information about a test. This is used by the Test package system for packaging a test and providing information for viewing and searching for tests in a proposed "test store". The test.yaml file can also can be used by human maintainers to preserve information (in a structured format) about a test, that is not included in the other test materials.

As an overview, the test.yaml file indicates where the source for the test comes from, it's license, the name of the test maintainer, a description of the test and tags for categorizing the test, and a formal list of parameters that are used by the test (what they mean and how to use them).

test.yaml fields [edit section]

Here are the fields supported in a test.yaml file:

fuego_package_version: indicates the version of package (in case of changes to the package schema). For now, this is always 1.
name: has the full Fuego name of the test. Ex: Benchmark.iperf
description: has an English description of the test
license: has an SPDX identifier for the test. This is the main license of the test project that the Fuego test uses, if the project has a tarfile or git repo. Otherwise it reflects the license of any non-Fuego-specific materials in the test directory. In such case, the test directory should include a LICENSE file. Fuego materials (fuego_test.sh, spec.json, chart_config.json, etc.) are considered to be under the default Fuego license (which is BSD-3-Clause) unless otherwise specifically indicated in these files. The license identifier for this field should be obtained from https://spdx.org/licenses/
author: the author or authors of the base test
maintainer: the maintainer of the Fuego materials for this test
version: the version of the base test
fuego_release: the version of Fuego materials for this test. This is a monotonically incrementing integer, starting at 1 for each new version of the base test.
type: either Benchmark or Functional
tags: a list of tags used to categorize this test. This is intended to be used in an eventual online test store.
tarball_src: a URL where the tarball was originally obtained from
gitrepo: a git URL where the source may be obtained from
host_dependencies: a list of Debian package names that must be installed in the docker container in order for this test to work properly. This field is optional, and indicates packages needed that are beyond those included in
params: a list of parameters that may be used with this test, including their descriptions, whether they are optional or required, and an example value for each one
data_files: a list of the files that are included in this test. This is used as the manifest for packaging the test (fuego_test.sh, and test.yaml are implicitly included in the packaging manifest).

More on params [edit section]

The 'params' field in the test.yaml file is a list of dictionaries with one item per test variable used by the test.

The name of the parameter is the short name of the parameter, without the test name prefix (e.g. FUNCTIONAL_LTP). The parameter name is the key for the dictionary with that parameters attributes.

Each parameter has a dictionary with attributes describing it. The dictionary has the following fields (keys):

'description' - text description of the parameter
'example' - an example of the parameter
'optional' - indicates whether the test requires this parameter (test variable) to be set or not. The value of the 'optional' field must be one of 'yes' or 'no'.

The test variables may be described by the test.yaml file can be defined in one of multiple locations in the Fuego test system. Most commonly the test variables are defined in a spec for the test, but they can also be defined in the board file, or as a dynamic board variable.

Example [edit section]

Here is an example test.yaml file, for the package Benchmark.iperf3:

fuego_package_version: 1
name: Benchmark.iperf3
description: |
    iPerf3 is a tool for active measurements of the maximum achievable
    bandwidth on IP networks.
license: BSD-3-Clause.
author: |
    Jon Dugan, Seth Elliott, Bruce A. Mah, Jeff Poskanzer, Kaustubh Prabhu,
    Mark Ashley, Aaron Brown, Aeneas Jaißle, Susant Sahani, Bruce Simpson,
    Brian Tierney.
maintainer: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
version: 3.1.3
fuego_release: 1
type: Benchmark
tags: ['network', 'performance']
tarball_src: https://iperf.fr/download/source/iperf-3.1.3-source.tar.gz
gitrepo: https://github.com/esnet/iperf.git
params:
    - server_ip:
        description: |
            IP address of the server machine. If not provided, then SRV_IP
            _must_ be provided on the board file. Otherwise the test will fail.
            if the server ip is assigned to the host, the test automatically
            starts the iperf3 server daemon. Otherwise, the tester _must_ make
            sure that iperf3 -V -s -D is already running on the server machine.
        example: 192.168.1.45
        optional: yes
    - client_params:
        description: extra parameters for the client
        example: -p 5223 -u -b 10G
        optional: yes
data_files:
    - chart_config.json
    - fuego_test.sh
    - parser.py
    - spec.json
    - criteria.json
    - iperf-3.1.3-source.tar.gz
    - reference.json
    - test.yaml