Release 1.2 (Combustion) Notes
Although this release is numbered as a "minor" release, it has some majorchanges in functionality. Here are some things to check out in this release:
- Unified output format
- Test dependency system
- Complex pass criteria handling
- Dynamic board variables
- Charting core improvements
- Support for test program source from git repositories
- Some transport modifications
- Test improvements
These features are described in the sections that follow.
Every test in the system now creates a run.json file, which includes that test's results, organized into testsets, testcases and measurements.
This format is intentionally modeled after kernelCI APIs, for easier integration with that test system's front-end.
See Unified Output Format for more information about this feature.
The dependency system allows for short-circuiting of test execution, when required features are not present. That is, these dependencies are checked during a pre_test phase, and if the dependencies are not met, Fuego aborts the test, before the test is built, deployed and executed on the target.
The dependencies are expressed as "NEED_" variables in fuego_test.sh
In the future, this features will allow for automatically detecting what tests are applicable to particular boards.
For more information, see Dependencies
Fuego allows the default pass criteria to be overridden via per-board criteria definitions, or by a user-supplied criteria file.
This flexibility allows Fuego users to customize results determination to their boards and unique testing situations, and to share results analysis in a formalized way between users and sites.
By way of example, the Fuego criteria system allows a user to specify what the expected filesystem performance, or real-time latency should be on their board, in order to meet their product shipment requirements. It also allows them to indicate that certain tests, although they fail, can be safely ignored for the purposes of overall evaluation of the system.
See criteria.json for more information.
This allows persistent information to be saved about a board, which can be used to share information between tests, or between invocations of a test.
See the section "Dynamic variables" on the Test variables page for the rationale for this feature and examples of potential uses for it.
- HTML table of testcases
- HTML table of test set aggregate results
Internally, the charting engine was modified to make it it more accessible to developers, and to allow (in the future) the same charting engine to be used for both realtime visualization and static report generation.
See Fuego charting
A user can specify a git repository, and a git reference (commit id, tag, or version), and Fuego will use that source to build the test program. This provides additional flexibility in dealing with upstream source for test programs.
Please see the documentation for these functions, for details.
- Added support for aarch64 toolchains
- Added dependency information
- Some older tests were fixed to address build issues with newer toolchains
- The source for some test programs was updated to newer versions
- Parsers were added for some Functional tests, to give results for individual testcases
- Parser improvements in general
- Lots of other bugfixes were made
In addition, Functional.LTP was modified to support LTP binaries already installed on a target board. This work will likely serve as a template for handling additional tests with binaries already on the board, in future Fuego versions. See Functional.LTP for some information about board variables and specs that can be used in this situation.