Release 1.5 Notes

Contents:

These are the Release Notes for the Fuego 1.5 release, which we are calling "Flame".

Major changes [edit section]

The major changes to Fuego in this release are as follows:

Change details [edit section]

New format for batch tests [edit section]

In previous versions of Fuego, testplans were used for running multiple Fuego tests at the same time. Fuego 1.5 introduces a new feature called "batch tests" that implements this functionality, and is intended to replace the testplan functionality.

See Using Batch Tests for a description of batch tests, and how to use them and write new ones.

Ability to manually group tests by batch-id, from the command line [edit section]

Even if tests are not executed as part of a batch test, it is sometimes useful to group multiple tests together. A user can manually set the batch-id for a test, by setting an environment variable FUEGO_BATCH_ID prior to calling 'ftc run-test'. If you use the same batch-id for multiple tests, then you can filter the results by that batch-id later, to see those test results grouped together in a report. See the command line help for 'ftc run-test' for information on specifying a batch_id in a report. (Or see Using Batch Tests, which also explains this.)

New test functions [edit section]

A few new functions were added to the Fuego test core, for use with batch tests. See their respective documentation pages for more information:

Simplified directory layout [edit section]

In the Fuego 1.5 release, the directory layout of the Fuego system was changed. This was done to simplify the directory structure, and to reduce some steps required to install Fuego.

Specifically:

The use of 'engine' in directory paths under fuego-core is now deprecated, and new tests should avoid referencing that directory.

Please note that all tests that come with the Fuego system (ie that are in the fuego-core repository) have already been converted to remove references to the 'engine' directory. However, a symlink was added to the fuego-core directory to support old tests that reference the 'engine' directory.

New default Jenkins server port [edit section]

This release changes the default port for the Jenkins server to 8090. That is, when running with a default installation of Fuego, you should use the following URL to access the Jenkins server interface:

This was done to make it less likely for Fuego to collide with other networked programs. Specifically, if a machine already has an installation of Jenkins using port 8080, a new, default installation of Fuego will not conflict with this.

More install options [edit section]

For a complete list of install.sh options, run install.sh --help

User configurable TCP port [edit section]

It is now possible to specify the TCP port for Jenkins. By default the Fuego uses TCP/IP port 8090, but this can be changed to another port so that multiple instances of Fuego can be run simultaneously, or to avoid a conflict with a service already using port 8090 on your host machine.

To use a different port than 8090 for Jenkins, specify it after the image name on the command line when you run install.sh. Note that this means that you must specify a Docker image name in order to specify a non-default port. For example:

New --nojenkins option [edit section]

It is now possible to install Fuego into a docker container that omits the Jenkins user interface. Some Fuego users have their own front-ends and back-ends, and don't need Jenkins. In Fuego version 1.5, install.sh now supports an option '--nojenkins' which produces a docker container to run Fuego without the Jenkins server. This reduces the overhead of the docker container by quite a bit, for those users.

New --no-cache option [edit section]

Sometimes, docker image cache data provides stale data for a Fuego docker container build. You can force Fuego to build a container from a clean slate by specifying the '--no-cache' option on the install.sh command line.

Ability to install Fuego without a container [edit section]

Usually, for security and test reproducibility reasons, Fuego is executed inside a docker container on your host machine. That is, by default installing Fuego will create a docker container using all the software that is needed for Fuego's tests.

However, in some configurations it is desirable to execute Fuego directly on a host machine (not inside a docker container). A user may have a dedicated machine, or they may want to avoid the overhead of running a docker container.

Several changes were made to support running Fuego (and specifically 'ftc') outside of a docker container. There is a separate install script, called 'install-debian.sh' that is used to install the Fuego system onto a Debian-based Linux distribution.

Use the script ./install-debian.sh in the top-level fuego directory, instead of ./install.sh, to install Fuego directly into your host machine. Please note that this is not advised unless you know exactly what you are doing. In this configuration, Fuego will likely not be able to manage host-side test dependencies for you correctly.

Note that Fuego tests execute bash instruction sequences on the host. So there is a danger when running tests from unknown third parties that they will execute something on your test host that breaches the security.

Upgraded Fuego base Linux distro to Debian stretch [edit section]

The Fuego base distribution of Linux (inside the Fuego container) is now Debian 'Stretch'. This is an update from Debian Jessie, used in previous versions of Fuego.

Also, the 'slim' version of the docker image for this distribution is used, in order to save filesystem space (the 'slim' docker image is much smaller than the 'standard' Debian Stretch image).

Upgraded Jenkins version to 2.164.1 [edit section]

Fuego now utilizes Jenkins version 2.164.1 for the interface server (front-end and back-end) of the system. This more recent version of Jenkins supports recently updated plugins and security fixes for the system.

Added new Tests [edit section]

Here is a list of new tests that were added in the v1.5 release:

Distribution functionality tests:

Fuego self-tests:

Tests using other system's tests:

Batch tests:

Support for results management backends (prototype) [edit section]

Normally, Fuego results can be examined using one of two methods:

However, this version of Fuego adds prototype support for sending Fuego test results to different results processing backends.

Two that are supported are the "Squad" backend, and the "fserver" backend. For both of these systems, you use the 'ftc put-run' command to transfer results for a run to the backend server.

See the following pages for more information:

NOTE: please note that support for external results management backends is in prototype form. This is not a fully supported Fuego feature yet, and aspects of this feature may change in future releases.

Added new specialized transport ssh2serial [edit section]

This release of Fuego adds a new specialized transport called 'ssh2serial'. This transport uses an intermediary computer between the host machine and the target board to perform operations on the board during a test. The intermediary machine is also referred to as the "proxy host".

Here is a diagram of the setup: 

  +--------+
  |        |  ssh  +---------------+         +--------------+
  | Fuego  |<----->| intermediary  | serial  |    board     |
  |  host  |       |   machine     |<------->|  under test  |
  |        |       | (proxy host)  |         +--------------+
  |        |       +---------------+
  +--------+

In order for this to work, you must install 'serio' on the intermediary machine. Also, the intermediary machine needs python (required to run serio tools). Specifically, the commands sercp and sersh should be placed in the directory /usr/local/bin on the intermediary machine.

Also, you need to provide the following variables to Fuego in the configuration file for the target board (ie fuego-ro/boards/<board>.conf):

You can either use a LOGIN/PASSWORD combination for access to the proxy host, or use an SSH_KEY file.

A staging area is created under /tmp on the proxy host where files are placed temporarily during transfer to and from the board. There must be sufficient filesystem space available to transfer whatever files any test may require (which may be big, for something like LTP (unless your configuration of LTP uses a pre-installed LTP binaries)).

Miscelaneous [edit section]

undocumented changes [edit section]