|
These are the Release Notes for the Fuego 1.2 release, which we are calling "Combustion".
|
{{TableOfContents}}
These are the Release Notes for the Fuego 1.2 release, which we are calling "Combustion".
|
|
Although this release is numbered as a "minor" release, it has
some majorchanges in functionality. Here are some things to
check out in this release:
|
Although this release is numbered as a "minor" release, it has
some majorchanges in functionality. Here are some things to
check out in this release:
|
|
|
= Major changes =
The major changes to Fuego in this release are as follows:
* Unified output format
* Test dependency system
* Complex pass criteria handling
* Dynamic board variables
* Charting core improvements
* Support for test program source from git repositories
* Some transport modifications
* Test improvements
|
|
These features are described in the sections that follow.
|
These features are described in the sections that follow.
|
|
|
== Unified output format ==
Fuego now stores the results of all tests in a unified format.
|
|
Every test in the system now creates a run.json file, which includes
that test's results, organized into testsets, testcases and measurements.
|
Every test in the system now creates a run.json file, which includes
that test's results, organized into testsets, testcases and measurements.
|
|
This format is intentionally modeled after kernelCI APIs, for easier
integration with that test system's front-end.
|
This format is intentionally modeled after kernelCI APIs, for easier
integration with that test system's front-end.
|
|
See Unified Output Format for more information about this feature.
|
See [[Unified Output Format]] for more information about this feature.
|
|
|
== Test dependency system ==
Fuego 1.2 introduces a test dependency system, that allows
Fuego to determine whether a test can be run on a board or not,
based on a set of variables declared in the tests base script (fuego_test.sh).
|
|
The dependency system allows for short-circuiting of test execution, when
required features are not present. That is, these dependencies are checked
during a pre_test phase, and if the dependencies are not met, Fuego aborts
the test, before the test is built, deployed and executed on the target.
|
The dependency system allows for short-circuiting of test execution, when
required features are not present. That is, these dependencies are checked
during a pre_test phase, and if the dependencies are not met, Fuego aborts
the test, before the test is built, deployed and executed on the target.
|
|
The dependencies are expressed as "NEED_" variables in fuego_test.sh
|
The dependencies are expressed as "NEED_" variables in fuego_test.sh
|
|
In the future, this features will allow for automatically detecting what
tests are applicable to particular boards.
|
In the future, this features will allow for automatically detecting what
tests are applicable to particular boards.
|
|
For more information, see Dependencies
|
For more information, see [[Dependencies]]
|
|
|
== Complex pass criteria handling ==
Fuego allows for expressing complex pass criteria for a test.
By default, most tests require that all individual testcases
in the test succeed, in order for the test to pass. However, large
complex tests often have individual testcases that fail.
Fuego allows specifying complex pass criteria, including count-based
pass criteria (number of pass vs. fail), as well as detailed
list-based criteria (testcases whose failure is ignored, or
which are required to pass). Additionally, the threshold
values used to determine whether a benchmark test passes is
configurable using the same criteria mechanism.
|
|
Fuego allows the default pass criteria to be overridden via
per-board criteria definitions, or by a user-supplied criteria file.
|
Fuego allows the default pass criteria to be overridden via
per-board criteria definitions, or by a user-supplied criteria file.
|
|
This flexibility allows Fuego users to customize results determination
to their boards and unique testing situations, and to share
results analysis in a formalized way between users and sites.
|
This flexibility allows Fuego users to customize results determination
to their boards and unique testing situations, and to share
results analysis in a formalized way between users and sites.
|
|
By way of example, the Fuego criteria system allows
a user to specify what the expected filesystem performance,
or real-time latency should be on their board, in order to meet
their product shipment requirements. It also allows them to
indicate that certain tests, although they fail, can be safely
ignored for the purposes of overall evaluation of the system.
|
By way of example, the Fuego criteria system allows
a user to specify what the expected filesystem performance,
or real-time latency should be on their board, in order to meet
their product shipment requirements. It also allows them to
indicate that certain tests, although they fail, can be safely
ignored for the purposes of overall evaluation of the system.
|
|
See criteria.json for more information.
|
See [[criteria.json]] for more information.
|
|
|
== Dynamic board variables ==
Fuego allows a user or a test to add additional variables to
a board definition.
|
|
This allows persistent information to be saved about a board,
which can be used to share information between tests, or
between invocations of a test.
|
This allows persistent information to be saved about a board,
which can be used to share information between tests, or
between invocations of a test.
|
|
See the section "Dynamic variables" on the Test variables page
for the rationale for this feature and examples of potential uses for it.
|
See the section "Dynamic variables" on the [[Test variables]] page
for the rationale for this feature and examples of potential uses for it.
|
|
|
== Charting core improvements ==
Charting in Fuego was changed in the 1.2 release, to provide
a foundation for future releases. A new chart configuration
file was introduced, which controls what type of charting to
show by default in the Jenkins interface for a test.
Additionally, Fuego now supports 3 chart types:
* HTML table of testcases
* HTML table of test set aggregate results
* Javascript plots of measurement data, for Benchmark tests
|
|
Internally, the charting engine was modified to make it
it more accessible to developers, and to allow (in the future)
the same charting engine to be used for both realtime visualization
and static report generation.
|
Internally, the charting engine was modified to make it
it more accessible to developers, and to allow (in the future)
the same charting engine to be used for both realtime visualization
and static report generation.
|
|
See Fuego charting
|
See [[Fuego charting]]
|
|
|
== Test program source from git repositories ==
Fuego has added, in this release, the ability to use
git repositories for test program source. This is in
addition to the default of providing a tarball of source
for test programs.
|
|
A user can specify a git repository, and a git reference
(commit id, tag, or version), and Fuego will use that source
to build the test program. This provides additional
flexibility in dealing with upstream source for test programs.
|
A user can specify a git repository, and a git reference
(commit id, tag, or version), and Fuego will use that source
to build the test program. This provides additional
flexibility in dealing with upstream source for test programs.
|
|
See function_unpack and Adding a new test.
|
See [[function_unpack]] and [[Adding a new test]].
|
|
|
== Transport modifications ==
In the 1.2 release, some slight modifications were made to
support different transport mechanisms (particularly to
support boards in LAVA-based board farms).
|
|
There are new functions: ov_transport_connect
and ov_transport_disconnect for
handling session setup and teardown (or board reservation and release).
|
There are new functions: [[function_ov_transport_connect|ov_transport_connect]]
and [[function_ov_transport_disconnect|ov_transport_disconnect]] for
handling session setup and teardown (or board reservation and release).
|
|
Please see the documentation for these functions, for details.
|
Please see the documentation for these functions, for details.
|
|
|
== Test improvements ==
There were numerous adjustments to tests to deal with new
features and fix errant behavior. Some of the major improvements
to multiple tests were:
* Added support for aarch64 toolchains
* Added dependency information
* Some older tests were fixed to address build issues with newer toolchains
* The source for some test programs was updated to newer versions
* Parsers were added for some Functional tests, to give results for individual testcases
* Parser improvements in general
* Lots of other bugfixes were made
|
|
In addition, Functional.LTP was modified to support LTP
binaries already installed on a target board. This work
will likely serve as a template for handling additional tests
with binaries already on the board, in future Fuego versions.
See Functional.LTP for some information about board
variables and specs that can be used in this situation.
|
In addition, Functional.LTP was modified to support LTP
binaries already installed on a target board. This work
will likely serve as a template for handling additional tests
with binaries already on the board, in future Fuego versions.
See [[Functional.LTP]] for some information about board
variables and specs that can be used in this situation.
|
|
|
== Documentation updates ==
Finally, a lot of these new features required extensive
documentation updates, which have (hopefully) mostly been
completed.
|