{{TableOfContents}}

These are the Release Notes for the Fuego 1.4 release, which we are calling "Ember".

= Major changes =
The major changes to Fuego in this release are as follows:
 * Added of clitest to Fuego distribution in the docker container
 * Added [[fuego_board_function_lib.sh]]
 * Added some new functions to core, for use by test programs:
   * [[function_log_this|log_this]]
   * [[function_report_live|report_live]]
   * [[function_assert_has_program|assert_has_program]]
   * [[function_is_abs_path|is_abs_path]]
 * Added (and used) [[FUEGO_LOGLEVELS]] to replace FUEGO_DEBUG
 * Did major cleanup of console log verbosity
 * Added new 'snapshot' phase of test execution
 * Converted Jenkins jobs to call ftc run-test
 * Added support for dynamic variables
 * Simplified install and start of container
 * PLATFORM was renamed to TOOLCHAIN
 * Added many new tests
   * at, bind, tar, etc.
 * Did many fixes and enhancements for existing tests

= Change details =
== clitest ==
clitest is a handy shell script for testing the results of command line
operations.  It is written purely in POSIX, so it can be used for both
host-side and board-side testing.

It is installed in /usr/local/bin in the Fuego docker container, and is
available for any tests that would like to use it.  To use it on the board
being tested, copy it to the board during the test's 'deploy' phase
(in test_deploy in the test's fuego_test.sh file), with a command line so:
 * put /usr/local/bin/clitest $BOARD_TESTDIR/fuego.$TESTDIR

Then use it in a command on the board (usually 'report'), like so:
 * report "cd $BOARD_TESTDIR/fuego.$TESTDIR ; ./clitest --prefix TAB cli-command-file"

== board utility functions ==
This release includes a new shell script which contains a
utility function library.  The script can be used by board-side
scripts, by first deploying it to the board, and then source-ing
it into your test script, and then calling one or more of the
functions.

This script contains a few functions for performing some
common operations in a distribution-independent way.
Use of this library is completely optional for a test.

You might be interested in using this library if you
have a board-side shell script that needs to:
 * 1) detect the type of init manager (proc 1) running on the board
 * 2) detect the type of logger service running on the board
 * 3) start or stop services on the board, as part of your test

To use this library, your fuego_test.sh script should copy the library
script to the board in the test_deploy function, and use it from a board-side
shell script (during the 'run' phase of the test).

See [[fuego_board_function_lib.sh]] for more details.

== new core functions ==
=== log_this ===
Some tests need to get information and data from host-side
operations, that needs to be reported and analyzed by Fuego.
    
The new [[function_log_this|log_this]] function captures the output of
commands executed on the host, and puts it (ultimately) into the test
log for a run. Any command executed with "log_this" is saved during test
execution, and placed in the final testlog.txt, after any board-side log
data (from report and report_append) commands.
    
There are a few tests (especially Fuego self-tests) that 
use this feature to avoid an awkward sequence of push-to-target,
and report-cat, to get log data from the host into the testlog.

=== report_live function ===
A new [[function_report_live|report_live]] function is available to
capture data directly from the test program on the target onto the
host.  The normal [[function_report|report]] function saves the test 
program output to a log file, which is retrieved after the test completes.
For some tests (e.g. tests that reboot the machine, or kexec into a new
kernel), it is not possible to save the data for the whole test
into a log on the target board.

The report_live function saves the data synchronously to the host,
and puts it into the test log during post-processing.

=== assert_has_program function ===
[[function_assert_has_program|assert_has_program]] is a Fuego core
function that can be used to express a test dependency on a program
on the board under test.  It combines several previous functions
into a more convenient single call.

=== is_abs_path function ===
[[function_is_abs_path|is_abs_path]] is a small helper function added
to the Fuego core library, for detecting if a path is absolute or not.


== FUEGO_LOGLEVELS feature ==
A new variable is used to control FUEGO message output.
[[FUEGO_LOGLEVELS]] was added, to replace the previous [[FUEGO_DEBUG]]
variable.

FUEGO_LOGLEVELS is more flexible than FUEGO_DEBUG, and easier to use,
since it is string-based instead of number-based.  FUEGO_LOGLEVELS
supports multiple levels of verbosity, all the way from 'debug'
to 'error'.  Finally, the new FUEGO_LOGLEVELS feature provide
functions for outputting messages at each verbosity level.

Please see the [[FUEGO_LOGLEVELS]] page for detailed information.

== log verbosity ==
In this release, there was a major cleanup of console log output, to
make it so that by default (at the default log level of 'info'), the
log is fairly sparse.  This is done so that the any messages
(particularly warnings) will be more apparent and draw the
attention of the Fuego user.

Many debug messages were moved to the 'debug' log level, and some
informational messages were moved to the 'verbose' log level.
Users can turn these on, if desired, by adjusting [[FUEGO_LOGLEVELS]].
Finally, some information that previously appeared in the log is
now placed in the snapshot file for a run.

The end result of this is that Fuego test execution is much 'quieter'. 
Most of the important machine status information for a run is still
available in the snapshot file (and is saved on every run,
independent of log level). Also, increased verbosity is still
available using a log level, if the user want's more details about
the steps and progress of a run, during test execution.

== snapshot feature ==
In this release, a new mechanism was introduced to save the status
information about the board for a test run.  Previously, the machine status
information was obtained during the pre_test phase, and appeared in the
console log.  Also, it was produced by a core Fuego function, always
producing the same results.

Now, this status information is obtained in a separate phase of execution,
and the operation can be overridden on a per-test or per-board basis.
See [[Board Snapshot]] for more details.

== ftc run-test ==
In this release, the format of Fuego-related Jenkins jobs was changed,
to execute tests from the Fuego command-line tool, 'ftc'.

Previously, Fuego created Jenkins jobs that included a call to 
``main.sh``.  However, in release 1.4, Fuego now creates Jenkins
jobs that call ``ftc run-test`` instead.  There should not be a noticeable
change in job execution behavior, but it is now possible to start a test
more easily from the command line.

One thing that was changed in 'ftc' was the addition of more flags
for the 'ftc run-test' command line, so that options that used to be
specified by environment variables can now be specified on the command line.


== dynamic variables ==
This release added support for dynamic test variables.  These are test
variables that can be specified on the 'ftc' command line, and override
the values of variables specified in the spec.json file.

The purpose of these is to support being able to support "variant" testing,
where a script can loop over a test multiple times, changing the variable
to different values.

The dynamic variables are expanded to full variables names that are prefixed
with the name of the test, during test execution.

Here is an example of using dynamic variables:

{{{#!YellowBox
 $ ftc run-test -b beaglebone -t Benchmark.Dhrystone --dynamic-vars "LOOPS=100000000"
}}}

This would override the default value for BENCHMARK_DHRYSTONE_LOOPS,
setting it to 100000000 (100 million) for this run.

This feature is intended to be useful for doing 'git bisect's of
a bug, passing a different git commit id for each iteration of the test.

See [[Test_variables#Dynamic_variables]] for more information.

== Install simplification ==
There were modifications made to the startup scripts for Fuego.
Previously, a sequence of 'install.sh', 'docker-create-container.sh' and
'docker-start-container.sh'.

This has been simplified to just 2 steps:
 * install.sh
 * start.sh

See [[Fuego Quickstart Guide]] and [[Installing Fuego]] for the
new instructions for bootstrapping Fuego.

== TOOLCHAIN board variable ==
The board variable TOOLCHAIN is now used in place of 'PLATFORM', in a
board file, to specify the toolchain that is used for building software
for a target board.

The PLATFORM variable is still supported for backwards compatibility, if
the TOOLCHAIN variable is not defined.
However, it's use is now deprecated.  Please adjust your board files
by converting PLATFORM to TOOLCHAIN in the files.

= New tests in the 1.4 release =
The following new tests have been added in this release:

== Tests of Linux tools and functionality ==
 * Functional.acpid
 * Functional.at
 * Functional.batch
 * Functional.bgpd
 * Functional.bind
 * Functional.busybox
 * Functional.cryptsetup
 * Functional.dovecot
 * Functional.ethtool
 * Functional.file
 * Functional.mcelog
 * Functional.nss
 * Functional.ospfd
 * Functional.ospf6d
 * Functional.pam
 * Functional.perl-xml-simple
 * Functional.rpm
 * Functional.tar
 * Functional.trousers

== Fuego self-tests ==
 * Benchmark.fuego_check_criteria_handling
 * Functional.fuego_compare_reports
 * Functional.fuego_ftc_test
 * Functional.fuego_report_live
 * Functional.fuego_sleep
 * Functional.fuego_slow_log
 * Functional.fuego_snapshot
 * Functional.fuego_test_variables

See the individual tests for information about them.


= Test commit log =
See the [[Release 1.4 test commit log]] for descriptions of individual
fixes and enhancements to tests in this release.

= Prototype feature enhancements =
The following sections show enhancements to features that are in
the prototype stage.  The features are not complete-enough to be supported
yet, but progress has been made on implementing the features in this release.

The prototype features enhanced in this release are:
 * added "ftc reserve-resource" and "ftc release-resource"
 * new target package feature
 * improved support for (still prototype) central server (fserver.py)
   * added support for server polling for remote job execution
   * reworked support for fuego server operations in ftc

== Resource reservations ==
This is the start of a feature for allowing the scheduling of
lab resources for a test.  For example, a test may need to use
the 'netperf' server during a test of networking performance.
It is desirable for this server to be used by only one test at
a time, in order to avoid disturbing the results of the test
on one board, with packets and networking activity from another
board.  A test can reserve the netperf server by doing
ftc reserve-resource 'netperf-server'

If a board in the test lab should be reserved, then the following
should be used by the test:
 * ftc reserve-resource -b other-board

Resources are held for a fixed period of time.  If another test
or user attempts to reserve a resource already reserved, then
the reservation will fail.

== Target Packages ==
It is now possible to create a 'target package' for a test.
A target packages contains the materials for a test that would normally
be installed during a regular 'test deploy' phase.  Usually, this is
a test program binary, and any associated data and board-side scripts.
A target package can be used to manually install the test program on the
board, and for manually using the test.

See [[Target Packages]] for more information.
A new phase was added 'm' for makepkg.
if phase makepkg is specified, then deploy is altered to put the materials
into  /fuego-rw/stage/fuego.<testname>.
Then, after deployment makepkg is called to create a test program package
file, in /fuego-rw/cache/$TOOLCHAIN-$TESTNAME-board-package.tar.gz

== Fuego Object Server ==
In this release, some additional support was added for interaction between
a local Fuego instance and a centralized testing server.

=== server re-write into standalone code ===
The code for the server was derived from the prototype written for the
2017 server demo (at Embedded Linux Conference in March 2017).  However,
that code was cobbled together quickly based on the tbwiki wiki engine.
The code has been re-written to be a standalone JSON test object
server (independent of the tbwiki code).

The server is now available separately from Fuego, at: https://github.com/tbird20d/fserver.git

Additional features were added to the server, including the ability
to update the status of a run-request that is in-progress of being executed.

=== Required Fuego configuration ===
In order to use the object server, your lab must have its own unique hostname
in the Fuego ecosystem, and you must configure your local Fuego instance
with a reference to the server you want to use.

The Fuego configuration is in fuego/fuego-ro/conf/fuego.conf.
To use the object server, you must set the following variables in the configuration file:
 * host_name
 * fuego_server

The 'host_name' should be an alphanumeric string with a unique name in the
Fuego ecosystem.  The string can contain characters, digits and underscores,
and must start with a character.  (It explicitly may NOT include dashes or periods - as these are used for delimiters for request and run object
names.)

The 'fuego_server' is the server name and path used to access a Fuego object
server.  The default global object server is at fuegotest.org, with path cgi-bin.  In the default fuego.conf file, you will see the line:
{{{#!YellowBox
fuego_server=fuegotest.org/cgi-bin
}}}

=== server polling script ===
There is a script that supports polling the object server for tests and
executing them.

The script is called ``poll_requests.sh``, and is located in the directory
``fuego-core/engine/scripts``.

If this script is executed inside the container, it will poll the
server forever.  When a request is found on the server that is intended
for a board in this Fuego instances (this lab/host), then:
 * the request is downloaded and executed
 * the results are pushed back to the server
 * the request is removed from the server

=== additional ftc support ===
In this (the 1.4) release, the 'ftc' tool was enhanced with
more support for Fuego server operations.

It is now possible to use ftc to remove a request from the server.  To do
this use the 'ftc rm-request'  Also, the 'ftc run-request' command
supports a new option '--put-run' to push the results of the run to
the server, after the test is completed.

Finally, when a request from the server is being executed, its status
is updated on the server in real-time, during the test execution.

This allows someone viewing the request list at the server to see the
requests that are currently in-progress.