Contents: Introduction test.yaml file Specification (ideas) Test Hub vision work in progress to do test package format questions existing package format analysis .deb .pet .rpm ebuild

This page describes a proposed "Test Package System" for Fuego. This describes "source" test packages, or the test definition and source code necessary to build, deploy, execute and post-process a test. The test package is part of the overall Test server system proposed for Fuego.

Another system under development with Fuego is the "Test binary package system", which is a standalone package containing the materials needed to execute a test on the target board (including pre-built binaries). See Test binary package system for details of that system.

This page has obsolete information, as of the Fuego 1.1 release (April 2017)

Introduction [edit section]

As of September, 2016, we add all tests to the system by adding them to the fuego-core repository. The files for a test are sprinkled around in lots of different places and some files (testplan_default.json, and tests.info) need to be edited manually for each new test added to the system.

This is a proposal to add a packaging system to Fuego, so that individual tests can be shipped as standalone entities.

The Fuego-core repository can hold the source for tests that are integrated into the system and vetted by fuego maintainers. But having tests shippable in separate units will allow much easier sharing of tests in the future.

As of May, 2018, several changes (refactorings) have been made to Fuego to support the features proposed in the Test package system. A demonstration of how test packaging might work was given in March of 2017, at Embedded Linux Conference. ftc now has the capability to package a test, if it includes a test.yaml file.

test.yaml file [edit section]

Each test that participates in the test package system has a test.yaml file, which defines meta-data used for packaging the test, as well as data used for searching for it and presenting it in an online "test store".

See test.yaml for the specification for this file.

Specification (ideas) [edit section]

I believe the following items are needed to create a test package system:

• Specify the test package format
  • I propose to use debian package format (.deb files)
    • This allows us to use standard tools (dpkg, apt-get, etc.) to install and remove packages in a fuego system.
  • it also allows us to use standard Debian meta-data for the package
• Specify test package contents
  • A test should include the following:
    • test tarfile (or tarfile reference)
    • patches
    • supporting files (data for test)
    • test script
    • testplan file(s)
    • test spec
    • test p/n files
    • parser.py (if a benchmark)
    • test info file (for plotting)
    • config.xml (Jenkins config file)
• Write a program to install a test
  • this will be run in the post-install script for the debian package
• Write a program to create a test package
  • When a user has created a test that works, they should be able to easily convert it into a test package to share
• Modify Fuego to support adding a test by file-placement only
  • change the testplan system, to allow for new tests without having to hand-edit files. Either change the overlay generator to aggregate testplan_default from multiple files, or make it so that testplan_default is not needed at all.
    • this work is in-progress (changes to ovgen.py are in Tim's local 'next' branch, to eliminate the needs for testplan_default)
  • change the plotting system, to allow for new tests without having to hand-edit files.

Test Hub vision [edit section]

At some point in the future, I envision a "test hub" (kind of like an app store), where people can place new test packages, and individual sites can select the tests they want to use with their system. There would be facilities for browsing the tests, downloading and installing the tests, and possibly rating tests or reporting issues about tests.

Creation of such a test hub is outside the scope of this test package proposal.

See Test server system

work in progress [edit section]

Tim is working on the test package system, as of February 2017. Here are some notes about that:

to do [edit section]

• define the test package format (done)
• document (formally) the test package format
• write package tool (done)
  • ftc package <testname> (done)
• write install tool
  • ftc install <package> (done)
  • Daniel's tool or jjb is used to populate Jenkins Config.xml is created from defaults
  • some other tool can be used to populate another system (tbwiki)
• modify ftc to run tests without Jenkins (done)
  • this means ftc has to recognize other systems (or ignore the Jenkins stuff) in order to run tests
• write ftc put-test tool (done)
• write ftc list-tests (done)

Notes:

• all items in ftc should be verb-object
  • package-test
  • install-test
  • lists-tests
  • set-var
  • get-var
  • put-test
  • put-run

test package format [edit section]

• all materials originate in a single directory in Fuego
• name: <testname>-<version>-<fuego-test-version>.ftp
• format:
  • compressed tarfile format (.tgz)
• contents:
  • test.yaml
    • fuego package version 1.0
    • sources - redundant with TARFILE (replace with SOURCE)
    • data
  • tarball (optional)
    • patches (optional)
    • base script (mandatory)
      • TARFILE
      • test_pretest (optional)
      • test_build (optional)
      • test_deploy
      • test_run
      • test_processing
    • tags (optional)
      • indicates test type, area
    • test name
    • test version
    • md5sum - list of md5sums for the files
    • specs
    • data
    • parser.py *
• package a test
• publish a test
• install a test
  • tar -xzvf p1.ftp

Here's a very simple sample test.yaml file:

fuego_package_version: 1
name: hello_world
version: 1.1
fuego_release: 1
type: Functional
tags: ['example']
base_script: hello_world.sh
data_files:
 - hello-test-1.1.tgz
 - spec.json

Principles:

• test execution should not require reading the manifest (only the base script)
• test installation should be verifiable
• should be easy to list tests by tag
• would like to avoid a new language
• should avoid too many languages if possible
  • adding yaml is not great, but probably inevitable
  • pyYaml doesn't have very good error reporting (It's fairly cryptic on errors)
    • maybe should fix that

questions [edit section]

• how are patches handled in the current system?
  • they are not copied to the build directory. They just live in $TEST_HOME *

existing package format analysis [edit section]

.deb [edit section]

Features of .deb:

• control file (with dependencies)
  • name
  • version
  • dependencies
  • maintainer
  • section (type of package)
  • priority
  • home page
  • description
  • architecture
• md5sums (list for all files - maybe need)
• conffiles (list files not to overwrite on upgrade - don't need)
• preinst, postinst, prerm and postrm scripts (not needed)
• config (not needed)
• shlibs (not needed)

.pet [edit section]

Features of .pet

• pinstall.sh puninstall.sh - not needed
• .desktop - used for the menu entry (not needed)
• XPM or PNG file - for icon
• folders (specifying directories to install at the top level e.g. opt, usr)
• pet.spec - single line, fields delimited by |
  • name, type, size, description, target

.rpm [edit section]

Features of .rpm

• embedded lua language (not needed)
• preamble:
  • summary
  • name
  • version
  • release
  • license
  • group
  • source
  • URL
  • distribution
  • vendor
  • packager
• %description
• %prep
• %patch
• %setup
• %build
• %install
• %files
  • %doc
• %clean
• %post, %postun, %pre, %preun

ebuild [edit section]

Features of ebuild

• name: <name>-<version>.ebuild

phases:

• pkg_setup
• src_unpack
• src_prepare
• src_configure
• src_compile
• src_test
• src_install
• pkg_preinst
• pkg_postinst
• pkg_prerm
• pkg_postrm