Release 1.3 Notes
These are the Release Notes for the Fuego 1.3 release, which we are calling "Draft".
Although this release is numbered as a "minor" release, it has some major changes in functionality. Here are some things to check out in this release:
- Use of fuego.conf
- Ability to run ftc outside the docker container
- New report generation features
- more output formats
- customizable field lists
- more options for --where clauses (tguid, tguid:result)
- Support for board hardware reboot (ov_board_control_reboot)
- Log splitting by testcase
- Can invoke individual test phases with ftc
- New tests:
- vuls, LTP_one_test, many RT tests, year2038
- release test (self test)
- Addition of web and image comparison features
These features are described in the sections that follow.
The main purpose of this file in this release is to allow for ftc to be run outside the docker container. However, it will be used to hold additional global configuration variables in the future.
See Fuego configuration file for more details.
However, some ftc commands only need to access fuego directories, and don't need to actually execute inside the docker container. And most commands operate faster when they are performed directly on the host, rather than inside the container (for example, 'ftc gen-report').
This release adds the capability to control whether ftc runs outside the docker container or not. Two new command line options to ftc support this functionality. The '-x' option tells ftc to NOT re-invoke the operation inside the docker container. And the '-c' option allows the user to tell ftc where the fuego configuration file is. With these two options, a user can tell ftc to execute the indicated command outside the container.
For more details on this, please see the following page: Running ftc outside the container
- new filtering (where clause) options
- new output formats
- ability to customize the fields reported
The --where clause is used to filter the data that appears in the report. The --where clause can now filter by test data by the tguid, tguid:result and duration_ms fields.
In addition to text output, you can now specify output in the following formats:
Some of these formats are output to a file in the report directory, which by default is /fuego-rw/reports. The location of the report directory can be overridden using the -o option.
Using the options --header_fields and --fields, you can now control what fields from each run will be put in the report header and in the body (table) of the report.
See 'ftc gen-report help' to get more information about the report generation options.
In the 1.3 release, only the 'ttc' board control system is supported, and the only hardware board control operation supported is reboot.
See ov_board_control_reboot for the function called by Fuego to do a hardware reboot.
Under some circumstances, Fuego will detect a hung board, and try to perform a hardware reboot of it during the post-test phase of test processing.
The new functionality is in the parser code, and to use it the test must have a custom parser (its own parser.py). That parser must call the function split_output_per_testcase
See that function call for more information about adding this feature to a particular test.
Use 'ftc run-test help' for details.
There are 3 mains groups of tests:
- New real-time tests,
- Fuego self-tests
- Other new tests (including updated versions of tests)
- Uses a kernel module (called backfire) to measure signal performance in the kernel.
- Test the deadline scheduler
- Test real-time multiprocessor scheduling of tasks to ensure the highest priority tasks are running on all available CPUs.
- Starts pairs of threads and measures the latency of interprocess communication with POSIX message queues.
- Starts two threads and measures the latency of inteprocess communication with POSIX mutex.
- Starts two threads or forks two processes, and measures the latency between sending and receiving a signal from one to the other.
- Starts two threads or forks two processes, and measures the latency of using SYSV semaphores between the two.
- Tests that Fuego internal programs follow preferred coding style.
- This test builds the Fuego docker image, and tests basic core Fuego functionality. This is described more below.
- This test verifies that Fuego core routines correctly parse and handle tguids with complicated formats.
- This is an updated version of the dbench file system performance test. Dbench was written by the authors of SAMBA, and it generates I/O workloads for filesystem testing. It uses "loadfiles" to simulate the I/O patterns, including timing, for real applications.
- This test measures the time for simple dd operation on the filesystem. It tests basic I/O performance.
- This is an updated version of the iperf network performance test.
- Vuls is an agentless vulnerability scanner for Linux and FreeBSD. It uses multiple vulnerability databases. See https://vuls.io/
- Runs Debian package tests on the target. Note that this installs and leaves the package dependencies for the tests on the target board. This means that you should use a board control scheme which re-provisions the board as part of testing (e.g. LAVA).
- Gathers a bunch of information about a board. This is used to establish a baseline of information about a board, which can be compared with a subsequent run later, to see if something has changed. Some of the items collected are: cpu information, memory information, loaded modules, usb devices, block devices, mount points, file system disk usage, network devices, running processes, kernel boot messages, and installed packages.
- Allows running a single test program from the LTP test suite. This is used for isolating individual test cases of interest, with reduced overhead to see results for a single test.
- A very simple test of correctly handling dates after the year 2038.
Hey! We have a test system. We should be able to use it to test itself.
This test is fairly complicated, but it adds some interesting features to Fuego. As an overview, it clones the fuego repositories (fuego and fuego-core), builds the docker container, starts the container, and checks that various features are supported (both on the command line and in the Jenkins interface).
The reason this test is complicated is that you end up with two complete versions of Fuego running at the same time on your system. The Fuego driving the test, and the Fuego "under test". So, it takes a lot of memory and disk on your host system to be able to run this test.
Probably, this test is most useful to Fuego maintainers, to test releases of Fuego. However, the test also utilizes some new functionality that is now incorporated into the Fuego distribution in the docker container.
For more information about this test, and how to run it, see the file: fuego-core/engine/tests/Functional.fuego_release_test/README.md
Selenium and Chromium have been added to the base Fuego distribution, to support testing of results returned from a web-based interface.
Using techniques demonstrated in Functional.fuego_release_test, you can use these host-side tools to test web functionality provided by your target board. This includes the capability for a test to scan a web page for specific text objects, as well as to check that client-side rendering of a web page (or part of the page) matches an expected image.
A test developer can take snapshots of a web page, or a region of the page, and save them for use with a test. Also, the developer can create image masks. These image masks can be used to omit sections of the page from comparison.
In this release, these capabilities are contained in a single test (the fuego_release_test). However, the intent is to migrate these capabilities into a generalized library that can be used by other Fuego tests, so that web page output can be tested easily with Fuego.
See fuego/docs/Makefile and the fuego/docs/*.rst files for the initial (currently stubbed) files for the conversion of the documentation to Sphynx (reStructuredText) format.
Although the fuego repository was configured for supporting Sphynx documentation, the pages from the fuegotest wiki that comprise the current documentation for Fuego have not been converted to reStructuredText yet. There is a page with information about markup conversions at Markup Mapping
Tests can now have a 'docs' directory where this documentation is stored. The purpose of this documentation is to allow test users to share information about the test and testcases among themselves, in a way that will (eventually) be accessible via the Jenkins interface, and able to be placed in generated reports.
Note, however, that this system is still under development.
See Test documentation for more information about this system.