Release 1.4 Notes
These are the Release Notes for the Fuego 1.4 release, which we are calling "Ember".
- Added of clitest to Fuego distribution in the docker container
- Added fuego_board_function_lib.sh
- Added some new functions to core, for use by test programs:
- Added (and used) FUEGO_LOGLEVELS to replace FUEGO_DEBUG
- Did major cleanup of console log verbosity
- Added new 'snapshot' phase of test execution
- Converted Jenkins jobs to call ftc run-test
- Added support for dynamic variables
- Simplified install and start of container
- PLATFORM was renamed to TOOLCHAIN
- Added many new tests
- at, bind, tar, etc.
- Did many fixes and enhancements for existing tests
It is installed in /usr/local/bin in the Fuego docker container, and is available for any tests that would like to use it. To use it on the board being tested, copy it to the board during the test's 'deploy' phase (in test_deploy in the test's fuego_test.sh file), with a command line so:
- put /usr/local/bin/clitest $BOARD_TESTDIR/fuego.$TESTDIR
Then use it in a command on the board (usually 'report'), like so:
- report "cd $BOARD_TESTDIR/fuego.$TESTDIR ; ./clitest --prefix TAB cli-command-file"
This script contains a few functions for performing some common operations in a distribution-independent way. Use of this library is completely optional for a test.
You might be interested in using this library if you have a board-side shell script that needs to:
- 1) detect the type of init manager (proc 1) running on the board
- 2) detect the type of logger service running on the board
- 3) start or stop services on the board, as part of your test
To use this library, your fuego_test.sh script should copy the library script to the board in the test_deploy function, and use it from a board-side shell script (during the 'run' phase of the test).
See fuego_board_function_lib.sh for more details.
The new log_this function captures the output of commands executed on the host, and puts it (ultimately) into the test log for a run. Any command executed with "log_this" is saved during test execution, and placed in the final testlog.txt, after any board-side log data (from report and report_append) commands.
There are a few tests (especially Fuego self-tests) that use this feature to avoid an awkward sequence of push-to-target, and report-cat, to get log data from the host into the testlog.
report_live function is available to capture data directly from the test program on the target onto the host. The normal report function saves the test program output to a log file, which is retrieved after the test completes. For some tests (e.g. tests that reboot the machine, or kexec into a new kernel), it is not possible to save the data for the whole test into a log on the target board.
The report_live function saves the data synchronously to the host, and puts it into the test log during post-processing.
assert_has_program is a Fuego core function that can be used to express a test dependency on a program on the board under test. It combines several previous functions into a more convenient single call.
is_abs_path is a small helper function added to the Fuego core library, for detecting if a path is absolute or not.
FUEGO_LOGLEVELS was added, to replace the previous FUEGO_DEBUG variable.
FUEGO_LOGLEVELS is more flexible than FUEGO_DEBUG, and easier to use, since it is string-based instead of number-based. FUEGO_LOGLEVELS supports multiple levels of verbosity, all the way from 'debug' to 'error'. Finally, the new FUEGO_LOGLEVELS feature provide functions for outputting messages at each verbosity level.
Please see the FUEGO_LOGLEVELS page for detailed information.
Many debug messages were moved to the 'debug' log level, and some informational messages were moved to the 'verbose' log level. Users can turn these on, if desired, by adjusting FUEGO_LOGLEVELS. Finally, some information that previously appeared in the log is now placed in the snapshot file for a run.
The end result of this is that Fuego test execution is much 'quieter'. Most of the important machine status information for a run is still available in the snapshot file (and is saved on every run, independent of log level). Also, increased verbosity is still available using a log level, if the user want's more details about the steps and progress of a run, during test execution.
Now, this status information is obtained in a separate phase of execution, and the operation can be overridden on a per-test or per-board basis. See Board Snapshot for more details.
Previously, Fuego created Jenkins jobs that included a call to
main.sh. However, in release 1.4, Fuego now creates Jenkins
jobs that call
ftc run-test instead. There should not be a noticeable
change in job execution behavior, but it is now possible to start a test
more easily from the command line.
One thing that was changed in 'ftc' was the addition of more flags for the 'ftc run-test' command line, so that options that used to be specified by environment variables can now be specified on the command line.
The purpose of these is to support being able to support "variant" testing, where a script can loop over a test multiple times, changing the variable to different values.
The dynamic variables are expanded to full variables names that are prefixed with the name of the test, during test execution.
Here is an example of using dynamic variables:
This would override the default value for BENCHMARK_DHRYSTONE_LOOPS, setting it to 100000000 (100 million) for this run.
This feature is intended to be useful for doing 'git bisect's of a bug, passing a different git commit id for each iteration of the test.
See Test_variables#Dynamic_variables for more information.
This has been simplified to just 2 steps:
The PLATFORM variable is still supported for backwards compatibility, if the TOOLCHAIN variable is not defined. However, it's use is now deprecated. Please adjust your board files by converting PLATFORM to TOOLCHAIN in the files.
See the individual tests for information about them.
Release 1.4 test commit log for descriptions of individual fixes and enhancements to tests in this release.
The prototype features enhanced in this release are:
- added "ftc reserve-resource" and "ftc release-resource"
- new target package feature
- improved support for (still prototype) central server (fserver.py)
- added support for server polling for remote job execution
- reworked support for fuego server operations in ftc
If a board in the test lab should be reserved, then the following should be used by the test:
- ftc reserve-resource -b other-board
Resources are held for a fixed period of time. If another test or user attempts to reserve a resource already reserved, then the reservation will fail.
See Target Packages for more information. A new phase was added 'm' for makepkg. if phase makepkg is specified, then deploy is altered to put the materials into /fuego-rw/stage/fuego.<testname>. Then, after deployment makepkg is called to create a test program package file, in /fuego-rw/cache/$TOOLCHAIN-$TESTNAME-board-package.tar.gz
The server is now available separately from Fuego, at: https://github.com/tbird20d/fserver.git
Additional features were added to the server, including the ability to update the status of a run-request that is in-progress of being executed.
The Fuego configuration is in fuego/fuego-ro/conf/fuego.conf. To use the object server, you must set the following variables in the configuration file:
The 'host_name' should be an alphanumeric string with a unique name in the Fuego ecosystem. The string can contain characters, digits and underscores, and must start with a character. (It explicitly may NOT include dashes or periods - as these are used for delimiters for request and run object names.)
The 'fuego_server' is the server name and path used to access a Fuego object server. The default global object server is at fuegotest.org, with path cgi-bin. In the default fuego.conf file, you will see the line:
The script is called
poll_requests.sh, and is located in the directory
If this script is executed inside the container, it will poll the server forever. When a request is found on the server that is intended for a board in this Fuego instances (this lab/host), then:
- the request is downloaded and executed
- the results are pushed back to the server
- the request is removed from the server
It is now possible to use ftc to remove a request from the server. To do this use the 'ftc rm-request' Also, the 'ftc run-request' command supports a new option '--put-run' to push the results of the run to the server, after the test is completed.
Finally, when a request from the server is being executed, its status is updated on the server in real-time, during the test execution.
This allows someone viewing the request list at the server to see the requests that are currently in-progress.