FrontPage 

Fuego wiki

Login or create account

Release 1.4 Notes in split format

{{TableOfContents}}
These are the Release Notes for the Fuego 1.4 release, which we are calling "Ember".
These are the Release Notes for the Fuego 1.4 release, which we are calling "Ember".

Major changes [edit section]

fuego_board_function_lib.sh * Added some new functions to core, for use by test programs: * log_this * report_live * assert_has_program * is_abs_path * Added (and used) FUEGO_LOGLEVELS to replace FUEGO_DEBUG * Did major cleanup of console log verbosity * Added new 'snapshot' phase of test execution * Converted Jenkins jobs to call ftc run-test * Added support for dynamic variables * Simplified install and start of container * PLATFORM was renamed to TOOLCHAIN * Added many new tests * at, bind, tar, etc. * Did many fixes and enhancements for existing tests
= Major changes =
The major changes to Fuego in this release are as follows:
 * Added of clitest to Fuego distribution in the docker container
 * Added [[fuego_board_function_lib.sh]]
 * Added some new functions to core, for use by test programs:
   * [[function_log_this|log_this]]
   * [[function_report_live|report_live]]
   * [[function_assert_has_program|assert_has_program]]
   * [[function_is_abs_path|is_abs_path]]
 * Added (and used) [[FUEGO_LOGLEVELS]] to replace FUEGO_DEBUG
 * Did major cleanup of console log verbosity
 * Added new 'snapshot' phase of test execution
 * Converted Jenkins jobs to call ftc run-test
 * Added support for dynamic variables
 * Simplified install and start of container
 * PLATFORM was renamed to TOOLCHAIN
 * Added many new tests
   * at, bind, tar, etc.
 * Did many fixes and enhancements for existing tests

Change details [edit section]

= Change details =
== clitest ==
clitest is a handy shell script for testing the results of command line
operations.  It is written purely in POSIX, so it can be used for both
host-side and board-side testing.
It is installed in /usr/local/bin in the Fuego docker container, and is available for any tests that would like to use it. To use it on the board being tested, copy it to the board during the test's 'deploy' phase (in test_deploy in the test's fuego_test.sh file), with a command line so: * put /usr/local/bin/clitest $BOARD_TESTDIR/fuego.$TESTDIR
It is installed in /usr/local/bin in the Fuego docker container, and is
available for any tests that would like to use it.  To use it on the board
being tested, copy it to the board during the test's 'deploy' phase
(in test_deploy in the test's fuego_test.sh file), with a command line so:
 * put /usr/local/bin/clitest $BOARD_TESTDIR/fuego.$TESTDIR
Then use it in a command on the board (usually 'report'), like so: * report "cd $BOARD_TESTDIR/fuego.$TESTDIR ; ./clitest --prefix TAB cli-command-file"
Then use it in a command on the board (usually 'report'), like so:
 * report "cd $BOARD_TESTDIR/fuego.$TESTDIR ; ./clitest --prefix TAB cli-command-file"

board utility functions [edit section]

== board utility functions ==
This release includes a new shell script which contains a
utility function library.  The script can be used by board-side
scripts, by first deploying it to the board, and then source-ing
it into your test script, and then calling one or more of the
functions.
This script contains a few functions for performing some common operations in a distribution-independent way. Use of this library is completely optional for a test.
This script contains a few functions for performing some
common operations in a distribution-independent way.
Use of this library is completely optional for a test.
You might be interested in using this library if you have a board-side shell script that needs to: * 1) detect the type of init manager (proc 1) running on the board * 2) detect the type of logger service running on the board * 3) start or stop services on the board, as part of your test
You might be interested in using this library if you
have a board-side shell script that needs to:
 * 1) detect the type of init manager (proc 1) running on the board
 * 2) detect the type of logger service running on the board
 * 3) start or stop services on the board, as part of your test
To use this library, your fuego_test.sh script should copy the library script to the board in the test_deploy function, and use it from a board-side shell script (during the 'run' phase of the test).
To use this library, your fuego_test.sh script should copy the library
script to the board in the test_deploy function, and use it from a board-side
shell script (during the 'run' phase of the test).
See fuego_board_function_lib.sh for more details.
See [[fuego_board_function_lib.sh]] for more details.

new core functions [edit section]

== new core functions ==
=== log_this ===
Some tests need to get information and data from host-side
operations, that needs to be reported and analyzed by Fuego.
The new log_this function captures the output of commands executed on the host, and puts it (ultimately) into the test log for a run. Any command executed with "log_this" is saved during test execution, and placed in the final testlog.txt, after any board-side log data (from report and report_append) commands.
The new [[function_log_this|log_this]] function captures the output of
commands executed on the host, and puts it (ultimately) into the test
log for a run. Any command executed with "log_this" is saved during test
execution, and placed in the final testlog.txt, after any board-side log
data (from report and report_append) commands.
There are a few tests (especially Fuego self-tests) that use this feature to avoid an awkward sequence of push-to-target, and report-cat, to get log data from the host into the testlog.
There are a few tests (especially Fuego self-tests) that 
use this feature to avoid an awkward sequence of push-to-target,
and report-cat, to get log data from the host into the testlog.

report_live function [edit section]

report_live function is available to capture data directly from the test program on the target onto the host. The normal report function saves the test program output to a log file, which is retrieved after the test completes. For some tests (e.g. tests that reboot the machine, or kexec into a new kernel), it is not possible to save the data for the whole test into a log on the target board.
=== report_live function ===
A new [[function_report_live|report_live]] function is available to
capture data directly from the test program on the target onto the
host.  The normal [[function_report|report]] function saves the test 
program output to a log file, which is retrieved after the test completes.
For some tests (e.g. tests that reboot the machine, or kexec into a new
kernel), it is not possible to save the data for the whole test
into a log on the target board.
The report_live function saves the data synchronously to the host, and puts it into the test log during post-processing.
The report_live function saves the data synchronously to the host,
and puts it into the test log during post-processing.

assert_has_program function [edit section]

assert_has_program is a Fuego core function that can be used to express a test dependency on a program on the board under test. It combines several previous functions into a more convenient single call.
=== assert_has_program function ===
[[function_assert_has_program|assert_has_program]] is a Fuego core
function that can be used to express a test dependency on a program
on the board under test.  It combines several previous functions
into a more convenient single call.

is_abs_path function [edit section]

is_abs_path is a small helper function added to the Fuego core library, for detecting if a path is absolute or not.
=== is_abs_path function ===
[[function_is_abs_path|is_abs_path]] is a small helper function added
to the Fuego core library, for detecting if a path is absolute or not.

FUEGO_LOGLEVELS feature [edit section]

FUEGO_LOGLEVELS was added, to replace the previous FUEGO_DEBUG variable.
== FUEGO_LOGLEVELS feature ==
A new variable is used to control FUEGO message output.
[[FUEGO_LOGLEVELS]] was added, to replace the previous [[FUEGO_DEBUG]]
variable.
FUEGO_LOGLEVELS is more flexible than FUEGO_DEBUG, and easier to use, since it is string-based instead of number-based. FUEGO_LOGLEVELS supports multiple levels of verbosity, all the way from 'debug' to 'error'. Finally, the new FUEGO_LOGLEVELS feature provide functions for outputting messages at each verbosity level.
FUEGO_LOGLEVELS is more flexible than FUEGO_DEBUG, and easier to use,
since it is string-based instead of number-based.  FUEGO_LOGLEVELS
supports multiple levels of verbosity, all the way from 'debug'
to 'error'.  Finally, the new FUEGO_LOGLEVELS feature provide
functions for outputting messages at each verbosity level.
Please see the FUEGO_LOGLEVELS page for detailed information.
Please see the [[FUEGO_LOGLEVELS]] page for detailed information.

log verbosity [edit section]

== log verbosity ==
In this release, there was a major cleanup of console log output, to
make it so that by default (at the default log level of 'info'), the
log is fairly sparse.  This is done so that the any messages
(particularly warnings) will be more apparent and draw the
attention of the Fuego user.
Many debug messages were moved to the 'debug' log level, and some informational messages were moved to the 'verbose' log level. Users can turn these on, if desired, by adjusting FUEGO_LOGLEVELS. Finally, some information that previously appeared in the log is now placed in the snapshot file for a run.
Many debug messages were moved to the 'debug' log level, and some
informational messages were moved to the 'verbose' log level.
Users can turn these on, if desired, by adjusting [[FUEGO_LOGLEVELS]].
Finally, some information that previously appeared in the log is
now placed in the snapshot file for a run.
The end result of this is that Fuego test execution is much 'quieter'. Most of the important machine status information for a run is still available in the snapshot file (and is saved on every run, independent of log level). Also, increased verbosity is still available using a log level, if the user want's more details about the steps and progress of a run, during test execution.
The end result of this is that Fuego test execution is much 'quieter'. 
Most of the important machine status information for a run is still
available in the snapshot file (and is saved on every run,
independent of log level). Also, increased verbosity is still
available using a log level, if the user want's more details about
the steps and progress of a run, during test execution.

snapshot feature [edit section]

== snapshot feature ==
In this release, a new mechanism was introduced to save the status
information about the board for a test run.  Previously, the machine status
information was obtained during the pre_test phase, and appeared in the
console log.  Also, it was produced by a core Fuego function, always
producing the same results.
Now, this status information is obtained in a separate phase of execution, and the operation can be overridden on a per-test or per-board basis. See Board Snapshot for more details.
Now, this status information is obtained in a separate phase of execution,
and the operation can be overridden on a per-test or per-board basis.
See [[Board Snapshot]] for more details.

ftc run-test [edit section]

== ftc run-test ==
In this release, the format of Fuego-related Jenkins jobs was changed,
to execute tests from the Fuego command-line tool, 'ftc'.
Previously, Fuego created Jenkins jobs that included a call to main.sh. However, in release 1.4, Fuego now creates Jenkins jobs that call ftc run-test instead. There should not be a noticeable change in job execution behavior, but it is now possible to start a test more easily from the command line.
Previously, Fuego created Jenkins jobs that included a call to 
``main.sh``.  However, in release 1.4, Fuego now creates Jenkins
jobs that call ``ftc run-test`` instead.  There should not be a noticeable
change in job execution behavior, but it is now possible to start a test
more easily from the command line.
One thing that was changed in 'ftc' was the addition of more flags for the 'ftc run-test' command line, so that options that used to be specified by environment variables can now be specified on the command line.
One thing that was changed in 'ftc' was the addition of more flags
for the 'ftc run-test' command line, so that options that used to be
specified by environment variables can now be specified on the command line.

dynamic variables [edit section]

== dynamic variables ==
This release added support for dynamic test variables.  These are test
variables that can be specified on the 'ftc' command line, and override
the values of variables specified in the spec.json file.
The purpose of these is to support being able to support "variant" testing, where a script can loop over a test multiple times, changing the variable to different values.
The purpose of these is to support being able to support "variant" testing,
where a script can loop over a test multiple times, changing the variable
to different values.
The dynamic variables are expanded to full variables names that are prefixed with the name of the test, during test execution.
The dynamic variables are expanded to full variables names that are prefixed
with the name of the test, during test execution.
Here is an example of using dynamic variables:
Here is an example of using dynamic variables:
{{{#!YellowBox
 $ ftc run-test -b beaglebone -t Benchmark.Dhrystone --dynamic-vars "LOOPS=100000000"
}}}
This would override the default value for BENCHMARK_DHRYSTONE_LOOPS, setting it to 100000000 (100 million) for this run.
This would override the default value for BENCHMARK_DHRYSTONE_LOOPS,
setting it to 100000000 (100 million) for this run.
This feature is intended to be useful for doing 'git bisect's of a bug, passing a different git commit id for each iteration of the test.
This feature is intended to be useful for doing 'git bisect's of
a bug, passing a different git commit id for each iteration of the test.
See Test_variables#Dynamic_variables for more information.
See [[Test_variables#Dynamic_variables]] for more information.

Install simplification [edit section]

== Install simplification ==
There were modifications made to the startup scripts for Fuego.
Previously, a sequence of 'install.sh', 'docker-create-container.sh' and
'docker-start-container.sh'.
This has been simplified to just 2 steps: * install.sh * start.sh
This has been simplified to just 2 steps:
 * install.sh
 * start.sh
See Fuego Quickstart Guide and Installing Fuego for the new instructions for bootstrapping Fuego.
See [[Fuego Quickstart Guide]] and [[Installing Fuego]] for the
new instructions for bootstrapping Fuego.

TOOLCHAIN board variable [edit section]

== TOOLCHAIN board variable ==
The board variable TOOLCHAIN is now used in place of 'PLATFORM', in a
board file, to specify the toolchain that is used for building software
for a target board.
The PLATFORM variable is still supported for backwards compatibility, if the TOOLCHAIN variable is not defined. However, it's use is now deprecated. Please adjust your board files by converting PLATFORM to TOOLCHAIN in the files.
The PLATFORM variable is still supported for backwards compatibility, if
the TOOLCHAIN variable is not defined.
However, it's use is now deprecated.  Please adjust your board files
by converting PLATFORM to TOOLCHAIN in the files.

New tests in the 1.4 release [edit section]

= New tests in the 1.4 release =
The following new tests have been added in this release:

Tests of Linux tools and functionality [edit section]

== Tests of Linux tools and functionality ==
 * Functional.acpid
 * Functional.at
 * Functional.batch
 * Functional.bgpd
 * Functional.bind
 * Functional.busybox
 * Functional.cryptsetup
 * Functional.dovecot
 * Functional.ethtool
 * Functional.file
 * Functional.mcelog
 * Functional.nss
 * Functional.ospfd
 * Functional.ospf6d
 * Functional.pam
 * Functional.perl-xml-simple
 * Functional.rpm
 * Functional.tar
 * Functional.trousers

Fuego self-tests [edit section]

== Fuego self-tests ==
 * Benchmark.fuego_check_criteria_handling
 * Functional.fuego_compare_reports
 * Functional.fuego_ftc_test
 * Functional.fuego_report_live
 * Functional.fuego_sleep
 * Functional.fuego_slow_log
 * Functional.fuego_snapshot
 * Functional.fuego_test_variables
See the individual tests for information about them.
See the individual tests for information about them.

Test commit log [edit section]

Release 1.4 test commit log for descriptions of individual fixes and enhancements to tests in this release.
= Test commit log =
See the [[Release 1.4 test commit log]] for descriptions of individual
fixes and enhancements to tests in this release.

Prototype feature enhancements [edit section]

= Prototype feature enhancements =
The following sections show enhancements to features that are in
the prototype stage.  The features are not complete-enough to be supported
yet, but progress has been made on implementing the features in this release.
The prototype features enhanced in this release are: * added "ftc reserve-resource" and "ftc release-resource" * new target package feature * improved support for (still prototype) central server (fserver.py) * added support for server polling for remote job execution * reworked support for fuego server operations in ftc
The prototype features enhanced in this release are:
 * added "ftc reserve-resource" and "ftc release-resource"
 * new target package feature
 * improved support for (still prototype) central server (fserver.py)
   * added support for server polling for remote job execution
   * reworked support for fuego server operations in ftc

Resource reservations [edit section]

== Resource reservations ==
This is the start of a feature for allowing the scheduling of
lab resources for a test.  For example, a test may need to use
the 'netperf' server during a test of networking performance.
It is desirable for this server to be used by only one test at
a time, in order to avoid disturbing the results of the test
on one board, with packets and networking activity from another
board.  A test can reserve the netperf server by doing
ftc reserve-resource 'netperf-server'
If a board in the test lab should be reserved, then the following should be used by the test: * ftc reserve-resource -b other-board
If a board in the test lab should be reserved, then the following
should be used by the test:
 * ftc reserve-resource -b other-board
Resources are held for a fixed period of time. If another test or user attempts to reserve a resource already reserved, then the reservation will fail.
Resources are held for a fixed period of time.  If another test
or user attempts to reserve a resource already reserved, then
the reservation will fail.

Target Packages [edit section]

== Target Packages ==
It is now possible to create a 'target package' for a test.
A target packages contains the materials for a test that would normally
be installed during a regular 'test deploy' phase.  Usually, this is
a test program binary, and any associated data and board-side scripts.
A target package can be used to manually install the test program on the
board, and for manually using the test.
See Target Packages for more information. A new phase was added 'm' for makepkg. if phase makepkg is specified, then deploy is altered to put the materials into /fuego-rw/stage/fuego.<testname>. Then, after deployment makepkg is called to create a test program package file, in /fuego-rw/cache/$TOOLCHAIN-$TESTNAME-board-package.tar.gz
See [[Target Packages]] for more information.
A new phase was added 'm' for makepkg.
if phase makepkg is specified, then deploy is altered to put the materials
into  /fuego-rw/stage/fuego.<testname>.
Then, after deployment makepkg is called to create a test program package
file, in /fuego-rw/cache/$TOOLCHAIN-$TESTNAME-board-package.tar.gz

Fuego Object Server [edit section]

== Fuego Object Server ==
In this release, some additional support was added for interaction between
a local Fuego instance and a centralized testing server.

server re-write into standalone code [edit section]

=== server re-write into standalone code ===
The code for the server was derived from the prototype written for the
2017 server demo (at Embedded Linux Conference in March 2017).  However,
that code was cobbled together quickly based on the tbwiki wiki engine.
The code has been re-written to be a standalone JSON test object
server (independent of the tbwiki code).
The server is now available separately from Fuego, at: https://github.com/tbird20d/fserver.git
The server is now available separately from Fuego, at: https://github.com/tbird20d/fserver.git
Additional features were added to the server, including the ability to update the status of a run-request that is in-progress of being executed.
Additional features were added to the server, including the ability
to update the status of a run-request that is in-progress of being executed.

Required Fuego configuration [edit section]

=== Required Fuego configuration ===
In order to use the object server, your lab must have its own unique hostname
in the Fuego ecosystem, and you must configure your local Fuego instance
with a reference to the server you want to use.
The Fuego configuration is in fuego/fuego-ro/conf/fuego.conf. To use the object server, you must set the following variables in the configuration file: * host_name * fuego_server
The Fuego configuration is in fuego/fuego-ro/conf/fuego.conf.
To use the object server, you must set the following variables in the configuration file:
 * host_name
 * fuego_server
The 'host_name' should be an alphanumeric string with a unique name in the Fuego ecosystem. The string can contain characters, digits and underscores, and must start with a character. (It explicitly may NOT include dashes or periods - as these are used for delimiters for request and run object names.)
The 'host_name' should be an alphanumeric string with a unique name in the
Fuego ecosystem.  The string can contain characters, digits and underscores,
and must start with a character.  (It explicitly may NOT include dashes or periods - as these are used for delimiters for request and run object
names.)
The 'fuego_server' is the server name and path used to access a Fuego object server. The default global object server is at fuegotest.org, with path cgi-bin. In the default fuego.conf file, you will see the line: {{{#!YellowBox fuego_server=fuegotest.org/cgi-bin }}}
The 'fuego_server' is the server name and path used to access a Fuego object
server.  The default global object server is at fuegotest.org, with path cgi-bin.  In the default fuego.conf file, you will see the line:
{{{#!YellowBox
fuego_server=fuegotest.org/cgi-bin
}}}

server polling script [edit section]

=== server polling script ===
There is a script that supports polling the object server for tests and
executing them.
The script is called poll_requests.sh, and is located in the directory fuego-core/engine/scripts.
The script is called ``poll_requests.sh``, and is located in the directory
``fuego-core/engine/scripts``.
If this script is executed inside the container, it will poll the server forever. When a request is found on the server that is intended for a board in this Fuego instances (this lab/host), then: * the request is downloaded and executed * the results are pushed back to the server * the request is removed from the server
If this script is executed inside the container, it will poll the
server forever.  When a request is found on the server that is intended
for a board in this Fuego instances (this lab/host), then:
 * the request is downloaded and executed
 * the results are pushed back to the server
 * the request is removed from the server

additional ftc support [edit section]

=== additional ftc support ===
In this (the 1.4) release, the 'ftc' tool was enhanced with
more support for Fuego server operations.
It is now possible to use ftc to remove a request from the server. To do this use the 'ftc rm-request' Also, the 'ftc run-request' command supports a new option '--put-run' to push the results of the run to the server, after the test is completed.
It is now possible to use ftc to remove a request from the server.  To do
this use the 'ftc rm-request'  Also, the 'ftc run-request' command
supports a new option '--put-run' to push the results of the run to
the server, after the test is completed.
Finally, when a request from the server is being executed, its status is updated on the server in real-time, during the test execution.
Finally, when a request from the server is being executed, its status
is updated on the server in real-time, during the test execution.
This allows someone viewing the request list at the server to see the requests that are currently in-progress.
This allows someone viewing the request list at the server to see the
requests that are currently in-progress.
TBWiki engine 1.8.3 by Tim Bird