Release_1.4_Notes 

Fuego wiki

Login or create account

Release 1.4 Notes

These are the Release Notes for the Fuego 1.4 release, which we are calling "Ember".

Major changes [edit section]

The major changes to Fuego in this release are as follows:
  • Added of clitest to Fuego distribution in the docker container
  • Added fuego_board_function_lib.sh
  • Added some new functions to core, for use by test programs:
  • Added (and used) FUEGO_LOGLEVELS to replace FUEGO_DEBUG
  • Did major cleanup of console log verbosity
  • Added new 'snapshot' phase of test execution
  • Converted Jenkins jobs to call ftc run-test
  • Added support for dynamic variables
  • Simplified install and start of container
  • PLATFORM was renamed to TOOLCHAIN
  • Added many new tests
    • at, bind, tar, etc.
  • Did many fixes and enhancements for existing tests

Change details [edit section]

clitest [edit section]

clitest is a handy shell script for testing the results of command line operations. It is written purely in POSIX, so it can be used for both host-side and board-side testing.

It is installed in /usr/local/bin in the Fuego docker container, and is available for any tests that would like to use it. To use it on the board being tested, copy it to the board during the test's 'deploy' phase (in test_deploy in the test's fuego_test.sh file), with a command line so:

  • put /usr/local/bin/clitest $BOARD_TESTDIR/fuego.$TESTDIR

Then use it in a command on the board (usually 'report'), like so:

  • report "cd $BOARD_TESTDIR/fuego.$TESTDIR ; ./clitest --prefix TAB cli-command-file"

board utility functions [edit section]

This release includes a new shell script which contains a utility function library. The script can be used by board-side scripts, by first deploying it to the board, and then source-ing it into your test script, and then calling one or more of the functions.

This script contains a few functions for performing some common operations in a distribution-independent way. Use of this library is completely optional for a test.

You might be interested in using this library if you have a board-side shell script that needs to:

  • 1) detect the type of init manager (proc 1) running on the board
  • 2) detect the type of logger service running on the board
  • 3) start or stop services on the board, as part of your test

To use this library, your fuego_test.sh script should copy the library script to the board in the test_deploy function, and use it from a board-side shell script (during the 'run' phase of the test).

See fuego_board_function_lib.sh for more details.

new core functions [edit section]

log_this [edit section]

Some tests need to get information and data from host-side operations, that needs to be reported and analyzed by Fuego.

The new log_this function captures the output of commands executed on the host, and puts it (ultimately) into the test log for a run. Any command executed with "log_this" is saved during test execution, and placed in the final testlog.txt, after any board-side log data (from report and report_append) commands.

There are a few tests (especially Fuego self-tests) that use this feature to avoid an awkward sequence of push-to-target, and report-cat, to get log data from the host into the testlog.

report_live function [edit section]

A new report_live function is available to capture data directly from the test program on the target onto the host. The normal report function saves the test program output to a log file, which is retrieved after the test completes. For some tests (e.g. tests that reboot the machine, or kexec into a new kernel), it is not possible to save the data for the whole test into a log on the target board.

The report_live function saves the data synchronously to the host, and puts it into the test log during post-processing.

assert_has_program function [edit section]

assert_has_program is a Fuego core function that can be used to express a test dependency on a program on the board under test. It combines several previous functions into a more convenient single call.

is_abs_path function [edit section]

is_abs_path is a small helper function added to the Fuego core library, for detecting if a path is absolute or not.

FUEGO_LOGLEVELS feature [edit section]

A new variable is used to control FUEGO message output. FUEGO_LOGLEVELS was added, to replace the previous FUEGO_DEBUG variable.

FUEGO_LOGLEVELS is more flexible than FUEGO_DEBUG, and easier to use, since it is string-based instead of number-based. FUEGO_LOGLEVELS supports multiple levels of verbosity, all the way from 'debug' to 'error'. Finally, the new FUEGO_LOGLEVELS feature provide functions for outputting messages at each verbosity level.

Please see the FUEGO_LOGLEVELS page for detailed information.

log verbosity [edit section]

In this release, there was a major cleanup of console log output, to make it so that by default (at the default log level of 'info'), the log is fairly sparse. This is done so that the any messages (particularly warnings) will be more apparent and draw the attention of the Fuego user.

Many debug messages were moved to the 'debug' log level, and some informational messages were moved to the 'verbose' log level. Users can turn these on, if desired, by adjusting FUEGO_LOGLEVELS. Finally, some information that previously appeared in the log is now placed in the snapshot file for a run.

The end result of this is that Fuego test execution is much 'quieter'. Most of the important machine status information for a run is still available in the snapshot file (and is saved on every run, independent of log level). Also, increased verbosity is still available using a log level, if the user want's more details about the steps and progress of a run, during test execution.

snapshot feature [edit section]

In this release, a new mechanism was introduced to save the status information about the board for a test run. Previously, the machine status information was obtained during the pre_test phase, and appeared in the console log. Also, it was produced by a core Fuego function, always producing the same results.

Now, this status information is obtained in a separate phase of execution, and the operation can be overridden on a per-test or per-board basis. See Board Snapshot for more details.

ftc run-test [edit section]

In this release, the format of Fuego-related Jenkins jobs was changed, to execute tests from the Fuego command-line tool, 'ftc'.

Previously, Fuego created Jenkins jobs that included a call to

main.sh. However, in release 1.4, Fuego now creates Jenkins jobs that call ftc run-test instead. There should not be a noticeable change in job execution behavior, but it is now possible to start a test more easily from the command line.

One thing that was changed in 'ftc' was the addition of more flags for the 'ftc run-test' command line, so that options that used to be specified by environment variables can now be specified on the command line.

dynamic variables [edit section]

This release added support for dynamic test variables. These are test variables that can be specified on the 'ftc' command line, and override the values of variables specified in the spec.json file.

The purpose of these is to support being able to support "variant" testing, where a script can loop over a test multiple times, changing the variable to different values.

The dynamic variables are expanded to full variables names that are prefixed with the name of the test, during test execution.

Here is an example of using dynamic variables:

     $ ftc run-test -b beaglebone -t Benchmark.Dhrystone --dynamic-vars "LOOPS=100000000"

This would override the default value for BENCHMARK_DHRYSTONE_LOOPS, setting it to 100000000 (100 million) for this run.

This feature is intended to be useful for doing 'git bisect's of a bug, passing a different git commit id for each iteration of the test.

See Test_variables#Dynamic_variables for more information.

Install simplification [edit section]

There were modifications made to the startup scripts for Fuego. Previously, a sequence of 'install.sh', 'docker-create-container.sh' and 'docker-start-container.sh'.

This has been simplified to just 2 steps:

  • install.sh
  • start.sh

See Fuego Quickstart Guide and Installing Fuego for the new instructions for bootstrapping Fuego.

TOOLCHAIN board variable [edit section]

The board variable TOOLCHAIN is now used in place of 'PLATFORM', in a board file, to specify the toolchain that is used for building software for a target board.

The PLATFORM variable is still supported for backwards compatibility, if the TOOLCHAIN variable is not defined. However, it's use is now deprecated. Please adjust your board files by converting PLATFORM to TOOLCHAIN in the files.

New tests in the 1.4 release [edit section]

The following new tests have been added in this release:

Tests of Linux tools and functionality [edit section]

  • Functional.acpid
  • Functional.at
  • Functional.batch
  • Functional.bgpd
  • Functional.bind
  • Functional.busybox
  • Functional.cryptsetup
  • Functional.dovecot
  • Functional.ethtool
  • Functional.file
  • Functional.mcelog
  • Functional.nss
  • Functional.ospfd
  • Functional.ospf6d
  • Functional.pam
  • Functional.perl-xml-simple
  • Functional.rpm
  • Functional.tar
  • Functional.trousers

Fuego self-tests [edit section]

  • Benchmark.fuego_check_criteria_handling
  • Functional.fuego_compare_reports
  • Functional.fuego_ftc_test
  • Functional.fuego_report_live
  • Functional.fuego_sleep
  • Functional.fuego_slow_log
  • Functional.fuego_snapshot
  • Functional.fuego_test_variables

See the individual tests for information about them.

Test commit log [edit section]

See the Release 1.4 test commit log for descriptions of individual fixes and enhancements to tests in this release.

Prototype feature enhancements [edit section]

The following sections show enhancements to features that are in the prototype stage. The features are not complete-enough to be supported yet, but progress has been made on implementing the features in this release.

The prototype features enhanced in this release are:

  • added "ftc reserve-resource" and "ftc release-resource"
  • new target package feature
  • improved support for (still prototype) central server (fserver.py)
    • added support for server polling for remote job execution
    • reworked support for fuego server operations in ftc

Resource reservations [edit section]

This is the start of a feature for allowing the scheduling of lab resources for a test. For example, a test may need to use the 'netperf' server during a test of networking performance. It is desirable for this server to be used by only one test at a time, in order to avoid disturbing the results of the test on one board, with packets and networking activity from another board. A test can reserve the netperf server by doing ftc reserve-resource 'netperf-server'

If a board in the test lab should be reserved, then the following should be used by the test:

  • ftc reserve-resource -b other-board

Resources are held for a fixed period of time. If another test or user attempts to reserve a resource already reserved, then the reservation will fail.

Target Packages [edit section]

It is now possible to create a 'target package' for a test. A target packages contains the materials for a test that would normally be installed during a regular 'test deploy' phase. Usually, this is a test program binary, and any associated data and board-side scripts. A target package can be used to manually install the test program on the board, and for manually using the test.

See Target Packages for more information. A new phase was added 'm' for makepkg. if phase makepkg is specified, then deploy is altered to put the materials into /fuego-rw/stage/fuego.<testname>. Then, after deployment makepkg is called to create a test program package file, in /fuego-rw/cache/$TOOLCHAIN-$TESTNAME-board-package.tar.gz

Fuego Object Server [edit section]

In this release, some additional support was added for interaction between a local Fuego instance and a centralized testing server.

server re-write into standalone code [edit section]

The code for the server was derived from the prototype written for the 2017 server demo (at Embedded Linux Conference in March 2017). However, that code was cobbled together quickly based on the tbwiki wiki engine. The code has been re-written to be a standalone JSON test object server (independent of the tbwiki code).

The server is now available separately from Fuego, at: https://github.com/tbird20d/fserver.git

Additional features were added to the server, including the ability to update the status of a run-request that is in-progress of being executed.

Required Fuego configuration [edit section]

In order to use the object server, your lab must have its own unique hostname in the Fuego ecosystem, and you must configure your local Fuego instance with a reference to the server you want to use.

The Fuego configuration is in fuego/fuego-ro/conf/fuego.conf. To use the object server, you must set the following variables in the configuration file:

  • host_name
  • fuego_server

The 'host_name' should be an alphanumeric string with a unique name in the Fuego ecosystem. The string can contain characters, digits and underscores, and must start with a character. (It explicitly may NOT include dashes or periods - as these are used for delimiters for request and run object names.)

The 'fuego_server' is the server name and path used to access a Fuego object server. The default global object server is at fuegotest.org, with path cgi-bin. In the default fuego.conf file, you will see the line:

    fuego_server=fuegotest.org/cgi-bin

server polling script [edit section]

There is a script that supports polling the object server for tests and executing them.

The script is called poll_requests.sh, and is located in the directory

fuego-core/engine/scripts.

If this script is executed inside the container, it will poll the server forever. When a request is found on the server that is intended for a board in this Fuego instances (this lab/host), then:

  • the request is downloaded and executed
  • the results are pushed back to the server
  • the request is removed from the server

additional ftc support [edit section]

In this (the 1.4) release, the 'ftc' tool was enhanced with more support for Fuego server operations.

It is now possible to use ftc to remove a request from the server. To do this use the 'ftc rm-request' Also, the 'ftc run-request' command supports a new option '--put-run' to push the results of the run to the server, after the test is completed.

Finally, when a request from the server is being executed, its status is updated on the server in real-time, during the test execution.

This allows someone viewing the request list at the server to see the requests that are currently in-progress.

TBWiki engine 1.8.2 by Tim Bird