Release 1.2 (Combustion) Notes

Contents:
These are the Release Notes for the Fuego 1.2 release, which we are calling "Combustion".

Although this release is numbered as a "minor" release, it has some majorchanges in functionality. Here are some things to check out in this release:

Major changes [edit section]

The major changes to Fuego in this release are as follows:

These features are described in the sections that follow.

Unified output format [edit section]

Fuego now stores the results of all tests in a unified format.

Every test in the system now creates a run.json file, which includes that test's results, organized into testsets, testcases and measurements.

This format is intentionally modeled after kernelCI APIs, for easier integration with that test system's front-end.

See Unified Output Format for more information about this feature.

Test dependency system [edit section]

Fuego 1.2 introduces a test dependency system, that allows Fuego to determine whether a test can be run on a board or not, based on a set of variables declared in the tests base script (fuego_test.sh).

The dependency system allows for short-circuiting of test execution, when required features are not present. That is, these dependencies are checked during a pre_test phase, and if the dependencies are not met, Fuego aborts the test, before the test is built, deployed and executed on the target.

The dependencies are expressed as "NEED_" variables in fuego_test.sh

In the future, this features will allow for automatically detecting what tests are applicable to particular boards.

For more information, see Dependencies

Complex pass criteria handling [edit section]

Fuego allows for expressing complex pass criteria for a test. By default, most tests require that all individual testcases in the test succeed, in order for the test to pass. However, large complex tests often have individual testcases that fail. Fuego allows specifying complex pass criteria, including count-based pass criteria (number of pass vs. fail), as well as detailed list-based criteria (testcases whose failure is ignored, or which are required to pass). Additionally, the threshold values used to determine whether a benchmark test passes is configurable using the same criteria mechanism.

Fuego allows the default pass criteria to be overridden via per-board criteria definitions, or by a user-supplied criteria file.

This flexibility allows Fuego users to customize results determination to their boards and unique testing situations, and to share results analysis in a formalized way between users and sites.

By way of example, the Fuego criteria system allows a user to specify what the expected filesystem performance, or real-time latency should be on their board, in order to meet their product shipment requirements. It also allows them to indicate that certain tests, although they fail, can be safely ignored for the purposes of overall evaluation of the system.

See criteria.json for more information.

Dynamic board variables [edit section]

Fuego allows a user or a test to add additional variables to a board definition.

This allows persistent information to be saved about a board, which can be used to share information between tests, or between invocations of a test.

See the section "Dynamic variables" on the Test variables page for the rationale for this feature and examples of potential uses for it.

Charting core improvements [edit section]

Charting in Fuego was changed in the 1.2 release, to provide a foundation for future releases. A new chart configuration file was introduced, which controls what type of charting to show by default in the Jenkins interface for a test. Additionally, Fuego now supports 3 chart types:

Internally, the charting engine was modified to make it it more accessible to developers, and to allow (in the future) the same charting engine to be used for both realtime visualization and static report generation.

See Fuego charting

Test program source from git repositories [edit section]

Fuego has added, in this release, the ability to use git repositories for test program source. This is in addition to the default of providing a tarball of source for test programs.

A user can specify a git repository, and a git reference (commit id, tag, or version), and Fuego will use that source to build the test program. This provides additional flexibility in dealing with upstream source for test programs.

See function_unpack and Adding a new test.

Transport modifications [edit section]

In the 1.2 release, some slight modifications were made to support different transport mechanisms (particularly to support boards in LAVA-based board farms).

There are new functions: ov_transport_connect and ov_transport_disconnect for handling session setup and teardown (or board reservation and release).

Please see the documentation for these functions, for details.

Test improvements [edit section]

There were numerous adjustments to tests to deal with new features and fix errant behavior. Some of the major improvements to multiple tests were:

In addition, Functional.LTP was modified to support LTP binaries already installed on a target board. This work will likely serve as a template for handling additional tests with binaries already on the board, in future Fuego versions. See Functional.LTP for some information about board variables and specs that can be used in this situation.

Documentation updates [edit section]

Finally, a lot of these new features required extensive documentation updates, which have (hopefully) mostly been completed.