{{TableOfContents}}

This page describes a proposed system (as of October 2019) for 
storing and distributing test binary packages.

A test package (described at [[Test package system]] is used for transferring
Fuego tests between systems, outside of Fuego's git repository.  This
is used with the [[Test server system]].

One important aspect of this system is that it is intended to allow
someone to run Fuego without a toolchain for their board.
They also don't need to have fuego-core/tests installed.

to do [edit section]

= to do =
Here is a list of things to do for this system:
 * (done) create test binary packages, on a Fuego host
   * (done) create a tarfile, and put it in /fuego-rw/cache
     * (done) using name $TOOLCHAIN-$TESTDIR-binary-package.tar.gz
 * (done) push test binary packages to a central server
   * (done) have fserver accept test binary packages
   * (done) allow someone to upload a test binary package
     * (done) add support for test binary packages to fserver.py
       * (done) create a json meta-data file to hold data for the package

• (done) have Fuego core use the local cache instead of the test's build and deploy phases * (done) check for USE_BINARY_PACKAGES during test execution * (done) check for existence of binary package during 'build' phase, and skip build * (done) check for existence of binary package during 'deploy' phase, and use that instead of materials from build dir

 * (done) have Fuego core use the local cache instead of the test's build and deploy phases
   * (done) check for USE_BINARY_PACKAGES during test execution
   * (done) check for existence of binary package during 'build' phase, and skip build
   * (done) check for existence of binary package during 'deploy' phase, and use that instead of materials from build dir

• (done) implement a use_binary_packages configuration option * (done) parse the config option in ftc * (done) use the config option in functions.sh to allow use of a binary package, if one is found in the cache

 * (done) implement a use_binary_packages configuration option
   * (done) parse the config option in ftc
   * (done) use the config option in functions.sh to allow use of a binary package, if one is found in the cache

• download test binary packages from the central server * have ftc check for binary package on server and put in cache (if use_binary_packages config is set), prior to test execution

 * download test binary packages from the central server
   * have ftc check for binary package on server and put in cache (if use_binary_packages config is set), prior to test execution

• improve binary package handling * (done) add '--binary' option to ftc package-test (instead of requiring a weird run-test invocation) * change 'make_cache.sh' to use 'ftc package-test .. --binary' * add '--binary' option to ftc install-test * add '--binary' option to ftc put-test * eliminate 'ftc put-binary-package' (re-use code)

 * improve binary package handling
   * (done) add '--binary' option to ftc package-test (instead of requiring a weird run-test invocation)
      * change 'make_cache.sh' to use 'ftc package-test .. --binary'
   * add '--binary' option to ftc install-test
   * add '--binary' option to ftc put-test
      * eliminate 'ftc put-binary-package' (re-use code)

• improve binary package format and content * (done) add &ftest directory at root * (done) add deploy directory at root * (done) when deploying a binary package, extract just the deploy directory * omit source files from the @ftest directory * parse source_files from the test.yaml file (using ydump?) * ydump test.yaml source_files | sed "s/['[]//g" | sed "s/]//g" | sed "s"/,/ /g" * when installing a binary test, put the ftpb in fuego-rw/cache * extract &ftest material into fuego-ro/test/<test-name> on the host * add spec to binary package filename * add version and release to binary package filename

 * improve binary package format and content
   * (done) add &ftest directory at root
   * (done) add deploy directory at root
     * (done) when deploying a binary package, extract just the deploy directory
   * omit source files from the @ftest directory
     * parse source_files from the test.yaml file (using ydump?)
       * ydump test.yaml source_files | sed "s/['[]//g" | sed "s/]//g" | sed "s"/,/ /g"
   * when installing a binary test, put the ftpb in fuego-rw/cache
     * extract &ftest material into fuego-ro/test/<test-name> on the host
   * add spec to binary package filename
   * add version and release to binary package filename

• support running a test with no prior test installation (e.g. in /fuego-core/tests) * if test is not present locally, but is available remotely as a binary package, then: * install enhanced binary package * binary package to /fuego-rw/cache * &ftest data to /fuego-rw/tests/<test-name> * (optionally), remove test after execution? * see some of the steps below for the no-build option

 * support running a test with no prior test installation (e.g. in /fuego-core/tests)
   * if test is not present locally, but is available remotely as a binary package, then:
     * install enhanced binary package
       * binary package to /fuego-rw/cache
       * &ftest data to /fuego-rw/tests/<test-name>
   * (optionally), remove test after execution?
   * see some of the steps below for the no-build option

• implement a no-build installation option * outline of behavior: * after test installation, remove test source materials * list test buildable source in test.yaml file * use that to avoid including it in test binary package * extend test binary package to include test materials (other than source) * see above for enhance binary package format (with $TEST_HOME inside the binary package) * put $TEST_HOME into <package>/&ftest directory * omit test source from &ftest directory * lists test on the server (ftc list-tests --remote --binary?) * add no-build option to install.sh * add no-build option to fuego.conf

 * implement a no-build installation option
   * outline of behavior:
     * after test installation, remove test source materials
       * list test buildable source in test.yaml file
       * use that to avoid including it in test binary package
     * extend test binary package to include test materials (other than source)
       * see above for enhance binary package format (with $TEST_HOME inside the binary package)
         * put $TEST_HOME into <package>/&ftest directory
           * omit test source from &ftest directory
     * lists test on the server (ftc list-tests --remote --binary?)
   * add no-build option to install.sh
   * add no-build option to fuego.conf

• implement a lightweight Fuego test-runner, that can execute on target without the rest of Fuego * outline of behavior: * list tests on the server (use wget or curl) * download a test binary package from the server * execute the test_pre_check function * need to support assert_has_program and NEED_ROOT * execute the test_run function * need to support report, report_live, cmd (only as local commands) * save the log somewhere, to be picked up by Fuego for post-processing * observation: this seems like we're abandoning the normal Fuego host/target model * define what needs to be in the binary test package: * test_pre_check, test_run, test variables (yes) * basically, test_prolog.sh (but there are board-specific items) * need fuego_test - what about parser, criteria, etc.?

 * implement a lightweight Fuego test-runner, that can execute on target without the rest of Fuego
   * outline of behavior:
     * list tests on the server (use wget or curl)
     * download a test binary package from the server
     * execute the test_pre_check function
       * need to support assert_has_program and NEED_ROOT
     * execute the test_run function
       * need to support report, report_live, cmd (only as local commands)
     * save the log somewhere, to be picked up by Fuego for post-processing
   * observation: this seems like we're abandoning the normal Fuego host/target model
   * define what needs to be in the binary test package:
     * test_pre_check, test_run, test variables (yes)
       * basically, test_prolog.sh (but there are board-specific items)
     * need fuego_test - what about parser, criteria, etc.?

How to create a binary package [edit section]

= How to create a binary package =
The following ftc command can be used to make a binary package using
the toolchain for a particular board:
{{{
  $ ftc package-test <test> --binary -b <board> [-s spec]
}}}

This will place the resulting binary package in the file:
/fuego-rw/cache/<toolchain>-<test>-binary-package.ftbp

So, as an example if you ran this command on the board 'local' with 
test "Functional.hello_world", you would end up creating the binary package:
/fuego-rw/cache/x86_64-Functional.hello_world-binary-package.ftbp

----

Note, the old (direct) method still works:
{{{
  $ ftc run-test -b <board> -t <test> -p "pbdm"
}}}

schema for test binary package meta-data [edit section]

= schema for test binary package meta-data =
Here are fields that are needed in the meta-data:
 * toolchain - what toolchain was used to build the package ($TOOLCHAIN)
 * (all toolchain variables):
   * arch ($ARCH)
   * compiler version
 * build date ($BUILD_TIMESTAMP)
 * build host ($FUEGO_HOST)
 * Fuego test name ($TESTDIR)

Here are the fields that eventually got used:
{{{
{
    "board": "$NODE_NAME",
    "build_host": "$FUEGO_HOST",
    "fuego_core_version": "$FUEGO_CORE_VERSION",
    "fuego_version": "$FUEGO_VERSION",
    "schema_version": "1.0"
    "test_name": "$TESTDIR",
    "testsuite_version": "$TESTSUITE_VERSION",
    "timestamp": "$BUILD_TIMESTAMP",
    "toolchain:": "$TOOLCHAIN",
}
}}}

Missing fields:
 * this also needs the test version number ('version' from test.yaml file' as well as 'fuego_release')
   * it looks like testsuite_version = test.yaml::version

contents of a test binary package [edit section]

= contents of a test binary package =
 * whatever is in the deploy directory after the 'deploy' function is called
 * binary-package.json with schema indicated above

Notes [edit section]

= Notes =
 * you have to have the source package (with fuego_test.sh, parser.py, test.haml, criteria.json, etc.) in order to run the test
 * the binary package only has the materials for the target board
 * the source package may have files that are not needed, like the source tar

to run a test from the server without any prior test installation [edit section]

== to run a test from the server without any prior test installation ==
 * you have to get the test from fserver
   * put the test in /fuego-rw/tests
 * you have to get the binary test package from fserver
   * put that in /fuego-rw/cache