Release 1.3 Notes
These are the Release Notes for the Fuego 1.3 release, which we are calling "Draft".
Although this release is numbered as a "minor" release, it has some major changes in functionality. Here are some things to check out in this release:
Major changes [edit section]
The major changes to Fuego in this release are as follows:- Use of fuego.conf
- Ability to run ftc outside the docker container
- New report generation features
- more output formats
- customizable field lists
- more options for --where clauses (tguid, tguid:result)
- Support for board hardware reboot (ov_board_control_reboot)
- Log splitting by testcase
- Can invoke individual test phases with ftc
- New tests:
- vuls, LTP_one_test, many RT tests, year2038
- release test (self test)
- Addition of web and image comparison features
These features are described in the sections that follow.
Fuego configuration file [edit section]
The 1.3 release of Fuego introduces a global configuration file for Fuego. This file is used to hold global information about the Fuego installation, and is normally located at fuego-ro/conf/fuego.conf.The main purpose of this file in this release is to allow for ftc to be run outside the docker container. However, it will be used to hold additional global configuration variables in the future.
See Fuego configuration file for more details.
Running ftc outside docker container [edit section]
Up until this release, all ftc operations were performed inside the docker container. There was a convenience feature of ftc that allowed the tool to be executed outside the docker container, but in this case it would just re-invoke an instance of 'ftc' inside the docker container to perform the actual work.However, some ftc commands only need to access fuego directories, and don't need to actually execute inside the docker container. And most commands operate faster when they are performed directly on the host, rather than inside the container (for example, 'ftc gen-report').
This release adds the capability to control whether ftc runs outside the docker container or not. Two new command line options to ftc support this functionality. The '-x' option tells ftc to NOT re-invoke the operation inside the docker container. And the '-c' option allows the user to tell ftc where the fuego configuration file is. With these two options, a user can tell ftc to execute the indicated command outside the container.
For more details on this, please see the following page: Running ftc outside the container
New report generation features [edit section]
'ftc gen-report' supports several new features, which make the command more flexible and useful. The additions can be divided into three main groups:- new filtering (where clause) options
- new output formats
- ability to customize the fields reported
The --where clause is used to filter the data that appears in the report. The --where clause can now filter by test data by the tguid, tguid:result and duration_ms fields.
In addition to text output, you can now specify output in the following formats:
- html
- excel
- csv
- rst
Some of these formats are output to a file in the report directory, which by default is /fuego-rw/reports. The location of the report directory can be overridden using the -o option.
Using the options --header_fields and --fields, you can now control what fields from each run will be put in the report header and in the body (table) of the report.
See 'ftc gen-report help' to get more information about the report generation options.
Hardware board control [edit section]
Previous to 1.3, Fuego only did software rebooting of the board under test. The 1.3 release introduces a concept called "board control", which allows Fuego to perform hardware operations on the board. In the 1.3 release, the board may specify a BOARD_CONTROL variable in the board file, which indicates what board control system is being used.In the 1.3 release, only the 'ttc' board control system is supported, and the only hardware board control operation supported is reboot.
See ov_board_control_reboot for the function called by Fuego to do a hardware reboot.
Under some circumstances, Fuego will detect a hung board, and try to perform a hardware reboot of it during the post-test phase of test processing.
Log splitting by testcase [edit section]
In the 1.3 release, there is new functionality for splitting the testlog of a run into individual parts, and browsing them from the Jenkins user interface.The new functionality is in the parser code, and to use it the test must have a custom parser (its own parser.py). That parser must call the function split_output_per_testcase
See that function call for more information about adding this feature to a particular test.
Support for test phase execution [edit section]
ftc run-test now supports the '-p' option to specify that only a sub-set of test phases should be executed. This is useful for quick debugging of tests, or for special operations like pre-building all tests in a test plan.Use 'ftc run-test help' for details.
New tests in the 1.3 release [edit section]
Some new tests were added to the Fuego 1.3 release.There are 3 mains groups of tests:
- New real-time tests,
- Fuego self-tests
- Other new tests (including updated versions of tests)
New real-time tests [edit section]
This set of tests adds additional Fuego tests for existing programs in the rt-tests suite of tests.- Benchmark.backfire
- Benchmark.deadlinetest
- Benchmark.migratetest
- Benchmark.pmqtest
- Benchmark.ptsematest
- Benchmark.sigwaittest
- Benchmark.svsematest
New Fuego self-tests [edit section]
- Functional.fuego_lint
- Functional.fuego_releast_test
- Functional.fuego_tguid_check
Other new tests [edit section]
- Benchmark.dbench4
- Benchmark.dd
- Benchmark.iperf3
- Benchmark.vuls
- Functional.autopkgtest
- Functional.fuego_board_status
- Functional.LTP_one_test
- Functional.year2038
Test Descriptions [edit section]
- Benchmark.backfire
- Uses a kernel module (called backfire) to measure signal performance in the kernel.
- Benchmark.deadlinetest
- Test the deadline scheduler
- Benchmark.migratetest
- Test real-time multiprocessor scheduling of tasks to ensure the highest priority tasks are running on all available CPUs.
- Benchmark.pmqtest
- Starts pairs of threads and measures the latency of interprocess communication with POSIX message queues.
- Benchmark.ptsematest
- Starts two threads and measures the latency of inteprocess communication with POSIX mutex.
- Benchmark.sigwaittest
- Starts two threads or forks two processes, and measures the latency between sending and receiving a signal from one to the other.
- Benchmark.svsematest
- Starts two threads or forks two processes, and measures the latency of using SYSV semaphores between the two.
- Functional.fuego_lint
- Tests that Fuego internal programs follow preferred coding style.
- Functional.fuego_release_test
- This test builds the Fuego docker image, and tests basic core Fuego functionality. This is described more below.
- Functional.fuego_tguid_check
- This test verifies that Fuego core routines correctly parse and handle tguids with complicated formats.
- Benchmark.dbench4
- This is an updated version of the dbench file system performance test. Dbench was written by the authors of SAMBA, and it generates I/O workloads for filesystem testing. It uses "loadfiles" to simulate the I/O patterns, including timing, for real applications.
- Benchmark.dd
- This test measures the time for simple dd operation on the filesystem. It tests basic I/O performance.
- Benchmark.iperf3
- This is an updated version of the iperf network performance test.
- Benchmark.vuls
- Vuls is an agentless vulnerability scanner for Linux and FreeBSD. It uses multiple vulnerability databases. See https://vuls.io/
- Functional.autopkgtest
- Runs Debian package tests on the target. Note that this installs and leaves the package dependencies for the tests on the target board. This means that you should use a board control scheme which re-provisions the board as part of testing (e.g. LAVA).
- Functional.fuego_board_status
- Gathers a bunch of information about a board. This is used to establish a baseline of information about a board, which can be compared with a subsequent run later, to see if something has changed. Some of the items collected are: cpu information, memory information, loaded modules, usb devices, block devices, mount points, file system disk usage, network devices, running processes, kernel boot messages, and installed packages.
- Functional.LTP_one_test
- Allows running a single test program from the LTP test suite. This is used for isolating individual test cases of interest, with reduced overhead to see results for a single test.
- Functional.year2038
- A very simple test of correctly handling dates after the year 2038.
Release test (self test) [edit section]
Fuego now includes a selftest for itself, called Functional.fuego_release_test.Hey! We have a test system. We should be able to use it to test itself.
This test is fairly complicated, but it adds some interesting features to Fuego. As an overview, it clones the fuego repositories (fuego and fuego-core), builds the docker container, starts the container, and checks that various features are supported (both on the command line and in the Jenkins interface).
The reason this test is complicated is that you end up with two complete versions of Fuego running at the same time on your system. The Fuego driving the test, and the Fuego "under test". So, it takes a lot of memory and disk on your host system to be able to run this test.
Probably, this test is most useful to Fuego maintainers, to test releases of Fuego. However, the test also utilizes some new functionality that is now incorporated into the Fuego distribution in the docker container.
For more information about this test, and how to run it, see the file: fuego-core/engine/tests/Functional.fuego_release_test/README.md
Web and image comparison features [edit section]
Selenium and Chromium have been added to the base Fuego distribution, to support testing of results returned from a web-based interface.Using techniques demonstrated in Functional.fuego_release_test, you can use these host-side tools to test web functionality provided by your target board. This includes the capability for a test to scan a web page for specific text objects, as well as to check that client-side rendering of a web page (or part of the page) matches an expected image.
A test developer can take snapshots of a web page, or a region of the page, and save them for use with a test. Also, the developer can create image masks. These image masks can be used to omit sections of the page from comparison.
In this release, these capabilities are contained in a single test (the fuego_release_test). However, the intent is to migrate these capabilities into a generalized library that can be used by other Fuego tests, so that web page output can be tested easily with Fuego.
Experimental or unfinished features in this release [edit section]
Sphynx documentation [edit section]
In the 1.3 release, a Sphynx documentation directory and Makefile were created. Sphynx is now required on the host machine for building the documentation.See fuego/docs/Makefile and the fuego/docs/*.rst files for the initial (currently stubbed) files for the conversion of the documentation to Sphynx (reStructuredText) format.
Although the fuego repository was configured for supporting Sphynx documentation, the pages from the fuegotest wiki that comprise the current documentation for Fuego have not been converted to reStructuredText yet. There is a page with information about markup conversions at Markup Mapping
testcase documentation [edit section]
We have added a convention for storing test, testset and testcase documentation in reStructuredText.Tests can now have a 'docs' directory where this documentation is stored. The purpose of this documentation is to allow test users to share information about the test and testcases among themselves, in a way that will (eventually) be accessible via the Jenkins interface, and able to be placed in generated reports.
Note, however, that this system is still under development.
See Test documentation for more information about this system.