{{TableOfContents}}
This page describes a proposed "Test Package System" for Fuego.  This
describes "source" test packages, or the test definition and source code
necessary to build, deploy, execute and post-process a test.  The test
package is part of the overall [[Test server system]] proposed for Fuego.

Another system under development with Fuego is the "Test binary package system", which is a standalone package containing the materials needed to
execute a test on the target board (including pre-built binaries).
See [[Test binary package system]] for details of that system.

This page has obsolete information, as of the Fuego 1.1 release (April 2017)

= Introduction =
As of September, 2016, we add all tests to the system by adding them to
the fuego-core repository.  The files for a test are sprinkled
around in lots of different places
and some files (testplan_default.json, and tests.info) need to
be edited manually for each new test added to the system.  

This is a proposal to add a packaging system to Fuego, so that
individual tests can be shipped as standalone entities.

The Fuego-core repository can hold the source for tests that
are integrated into the system and vetted by fuego maintainers.
But having tests shippable in separate units will allow much
easier sharing of tests in the future.

As of May, 2018, several changes (refactorings) have been made to Fuego
to support the features proposed in the Test package system.  A demonstration
of how test packaging might work was given in March of 2017, at Embedded Linux
Conference.  ftc now has the capability to package a test, if it includes a
test.yaml file.

= test.yaml file =
Each test that participates in the test package system has a test.yaml
file, which defines meta-data used for packaging the test, as well as
data used for searching for it and presenting it in an online "test store".

See [[test.yaml]] for the specification for this file.

= Specification (ideas) =
I believe the following items are needed to create a test package system:
 * Specify the test package format
   * I propose to use debian package format (.deb files)
     * This allows us to use standard tools (dpkg, apt-get, etc.) to install and remove packages in a fuego system.
    * it also allows us to use standard Debian meta-data for the package
 * Specify test package contents
   * A test should include the following:
     * test tarfile (or tarfile reference)
     * patches
     * supporting files (data for test)
     * test script
     * testplan file(s)
     * test spec
     * test p/n files
     * parser.py (if a benchmark)
     * test info file (for plotting)
     * config.xml (Jenkins config file)
 * Write a program to install a test
   * this will be run in the post-install script for the debian package
 * Write a program to create a test package
   * When a user has created a test that works, they should be able to easily
   convert it into a test package to share
 * Modify Fuego to support adding a test by file-placement only
   * change the testplan system, to allow for new tests without having
   to hand-edit files.  Either change the overlay generator to aggregate
   testplan_default from multiple files, or make it so that testplan_default
   is not needed at all.
     * this work is in-progress (changes to ovgen.py are in Tim's local 'next' branch, to eliminate the needs for testplan_default)
   * change the plotting system, to allow for new tests without having to
   hand-edit files.
 
= Test Hub vision =
At some point in the future, I envision a "test hub" (kind of like
an app store), where people can place new test packages, and
individual sites can select the tests they want to use with their system.
There would be facilities for browsing the tests, downloading and
installing the tests, and possibly rating tests or reporting issues
about tests.

Creation of such a test hub is outside the scope of this test package proposal.

See [[Test server system]]

= work in progress =
Tim is working on the test package system, as of February 2017.
Here are some notes about that:

== to do ==
 * define the test package format (done)
 * document (formally) the test package format
 * write package tool (done)
   * ftc package <testname> (done)
 * write install tool
   * ftc install <package> (done)
   * Daniel's tool or jjb is used to populate Jenkins
     Config.xml is created from defaults
   * some other tool can be used to populate another system (tbwiki)
 * modify ftc to run tests without Jenkins (done)
   * this means ftc has to recognize other systems (or ignore the
       Jenkins stuff) in order to run tests
 * write ftc put-test tool (done)
 * write ftc list-tests (done)

Notes:
 * all items in ftc should be verb-object
   - package-test
   - install-test
   - lists-tests
   - set-var
   - get-var
   - put-test
   - put-run

== test package format ==
 * all materials originate in a single directory in Fuego
 * name: <testname>-<version>-<fuego-test-version>.ftp
 * format:
   * compressed tarfile format (.tgz)
 * contents:
    * test.yaml
       - fuego package version 1.0
       - sources - redundant with TARFILE (replace with SOURCE)
       - data
     * tarball (optional)
     * patches (optional)
     * base script (mandatory)
       - TARFILE
       - test_pretest (optional)
       - test_build (optional)
       - test_deploy
       - test_run
       - test_processing
     * tags (optional)
       - indicates test type, area
     * test name
     * test version
     * md5sum - list of md5sums for the files
     * specs
     * data
     * parser.py
     * 
 * package a test
 * publish a test
 * install a test
   * tar -xzvf p1.ftp

Here's a very simple sample test.yaml file:
{{{#!YellowBox
fuego_package_version: 1
name: hello_world
version: 1.1
fuego_release: 1
type: Functional
tags: ['example']
base_script: hello_world.sh
data_files:
 - hello-test-1.1.tgz
 - spec.json
}}}

Principles:
 * test execution should not require reading the manifest (only the base script)
 * test installation should be verifiable
 * should be easy to list tests by tag
 * would like to avoid a new language
 * should avoid too many languages if possible
   * adding yaml is not great, but probably inevitable
   * pyYaml doesn't have very good error reporting (It's fairly cryptic on errors)
     * maybe should fix that



==== questions ====
 * how are patches handled in the current system?
   * they are not copied to the build directory.  They just live in $TEST_HOME
 * 

== existing package format analysis ==
=== .deb ===
Features of .deb:
 - control file (with dependencies)
   - name
   - version
   - dependencies
   - maintainer
   - section (type of package)
   - priority
   - home page
   - description
   - architecture
 - md5sums (list for all files - maybe need)
 - conffiles (list files not to overwrite on upgrade - don't need)
 - preinst, postinst, prerm and postrm scripts (not needed)
 - config (not needed)
 - shlibs (not needed)

=== .pet ===
Features of .pet
 - pinstall.sh puninstall.sh - not needed
 - .desktop - used for the menu entry (not needed)
 - XPM or PNG file - for icon
 - folders (specifying directories to install at the top level e.g. opt, usr)
 - pet.spec - single line, fields delimited by |
   - name, type, size, description, target

=== .rpm ===
Features of .rpm
 - embedded lua language (not needed)
 - preamble:
   - summary
   - name
   - version
   - release
   - license
   - group
   - source
   - URL
   - distribution
   - vendor
   - packager
 - %description
 - %prep
 - %patch
 - %setup
 - %build
 - %install
 - %files
   - %doc
 - %clean
 - %post, %postun, %pre, %preun

=== ebuild ===
Features of ebuild
 - name: <name>-<version>.ebuild
 
phases:
 - pkg_setup
 - src_unpack
 - src_prepare
 - src_configure
 - src_compile
 - src_test
 - src_install
 - pkg_preinst
 - pkg_postinst
 - pkg_prerm
 - pkg_postrm