{{TableOfContents}}
This page describes a proposed "Test Package System" for Fuego.

This page has obsolete information, as of the Fuego 1.1 release (April 2017)

= Introduction =
As of September, 2016, we add all tests to the system by adding them to
the fuego-core repository.  The files for a test are sprinkled
around in lots of different places
and some files (testplan_default.json, and tests.info) need to
be edited manually for each new test added to the system.  

This is a proposal to add a packaging system to Fuego, so that
individual tests can be shipped as standalone entities.

The Fuego-core repository can hold the source for tests that
are integrated into the system and vetted by fuego maintainers.
But having tests shippable in separate units will allow much
easier sharing of tests in the future.



= Specification =
I believe the following items are needed to create a test package system:
 * Specify the test package format
   * I propose to use debian package format (.deb files)
     * This allows us to use standard tools (dpkg, apt-get, etc.) to install and remove packages in a fuego system.
    * it also allows us to use standard Debian meta-data for the package
 * Specify test package contents
   * A test should include the following:
     * test tarfile (or tarfile reference)
     * patches
     * supporting files (data for test)
     * test script
     * testplan file(s)
     * test spec
     * test p/n files
     * parser.py (if a benchmark)
     * test info file (for plotting)
     * config.xml (Jenkins config file)
 * Write a program to install a test
   * this will be run in the post-install script for the debian package
 * Write a program to create a test package
   * When a user has created a test that works, they should be able to easily
   convert it into a test package to share
 * Modify Fuego to support adding a test by file-placement only
   * change the testplan system, to allow for new tests without having
   to hand-edit files.  Either change the overlay generator to aggregate
   testplan_default from multiple files, or make it so that testplan_default
   is not needed at all.
     * this work is in-progress (changes to ovgen.py are in Tim's local 'next' branch, to eliminate the needs for testplan_default)
   * change the plotting system, to allow for new tests without having to
   hand-edit files.
 

= Test Hub vision =
At some point in the future, I envision a "test hub" (kind of like
an app store), where people can place new test packages, and
individual sites can select the tests they want to use with their system.
There would be facilities for browsing the tests, downloading and
installing the tests, and possibly rating tests or reporting issues
about tests.

Creation of such a test hub is outside the scope of this test package proposal.

See [[Test server system]]

= work in progress =
Tim is working on the test package system, as of February 2017.
Here are some notes about that:

== to do ==
 * define the test package format (done)
 * document (formally) the test package format
 * write package tool (done)
   * ftc package <testname> (done)
 * write install tool
   * ftc install <package> (done)
   * Daniel's tool or jjb is used to populate Jenkins
     Config.xml is created from defaults
   * some other tool can be used to populate another system (tbwiki)
 * modify ftc to run tests without Jenkins (done)
   * this means ftc has to recognize other systems (or ignore the
       Jenkins stuff) in order to run tests
 * write ftc put-test tool (done)
 * write ftc list-tests (done)

Notes:
 * all items in ftc should be verb-object
   - package-test
   - install-test
   - lists-tests
   - set-var
   - get-var
   - put-test
   - put-run

== test package format ==
 * all materials originate in a single directory in Fuego
 * name: <testname>-<version>-<fuego-test-version>.ftp
 * format:
   * compressed tarfile format (.tgz)
 * contents:
    * test.yaml
       - fuego package version 1.0
       - sources - redundant with TARFILE (replace with SOURCE)
       - data
     * tarball (optional)
     * patches (optional)
     * base script (mandatory)
       - TARFILE
       - test_pretest (optional)
       - test_build (optional)
       - test_deploy
       - test_run
       - test_processing
     * tags (optional)
       - indicates test type, area
     * test name
     * test version
     * md5sum - list of md5sums for the files
     * specs
     * data
     * parser.py
     * 
 * package a test
 * publish a test
 * install a test
   * tar -xzvf p1.ftp

Here's a very simple sample test.yaml file:
{{{#!YellowBox
fuego_package_version: 1
name: hello_world
version: 1.1
fuego_release: 1
type: Functional
tags: ['example']
base_script: hello_world.sh
data_files:
 - hello-test-1.1.tgz
 - Functional.hello_world.spec
}}}

Principles:
 * test execution should not require reading the manifest (only the base script)
 * test installation should be verifiable
 * should be easy to list tests by tag
 * would like to avoid a new language
 * should avoid too many languages if possible
   * adding yaml is not great, but probably inevitable
   * pyYaml doesn't have very good error reporting (It's fairly cryptic on errors)
     * maybe should fix that



==== questions ====
 * how are patches handled in the current system?
   * they are not copied to the build directory.  They just live in $TEST_HOME
 * 

== existing package format analysis ==
=== .deb ===
Features of .deb:
 - control file (with dependencies)
   - name
   - version
   - dependencies
   - maintainer
   - section (type of package)
   - priority
   - home page
   - description
   - architecture
 - md5sums (list for all files - maybe need)
 - conffiles (list files not to overwrite on upgrade - don't need)
 - preinst, postinst, prerm and postrm scripts (not needed)
 - config (not needed)
 - shlibs (not needed)

=== .pet ===
Features of .pet
 - pinstall.sh puninstall.sh - not needed
 - .desktop - used for the menu entry (not needed)
 - XPM or PNG file - for icon
 - folders (specifying directories to install at the top level e.g. opt, usr)
 - pet.spec - single line, fields delimited by |
   - name, type, size, description, target

=== .rpm ===
Features of .rpm
 - embedded lua language (not needed)
 - preamble:
   - summary
   - name
   - version
   - release
   - license
   - group
   - source
   - URL
   - distribution
   - vendor
   - packager
 - %description
 - %prep
 - %patch
 - %setup
 - %build
 - %install
 - %files
   - %doc
 - %clean
 - %post, %postun, %pre, %preun

=== ebuild ===
Features of ebuild
 - name: <name>-<version>.ebuild
 
phases:
 - pkg_setup
 - src_unpack
 - src_prepare
 - src_configure
 - src_compile
 - src_test
 - src_install
 - pkg_preinst
 - pkg_postinst
 - pkg_prerm
 - pkg_postrm