Contents: to do How to create a binary package schema for test binary package meta-data contents of a test binary package Notes to run a test from the server without any prior test installation

This page describes a proposed system (as of October 2019) for storing and distributing test binary packages.

A test package (described at Test package system is used for transferring Fuego tests between systems, outside of Fuego's git repository. This is used with the Test server system.

One important aspect of this system is that it is intended to allow someone to run Fuego without a toolchain for their board. They also don't need to have fuego-core/tests installed.

to do [edit section]

Here is a list of things to do for this system:

• (done) create test binary packages, on a Fuego host
  • (done) create a tarfile, and put it in /fuego-rw/cache
    • (done) using name $TOOLCHAIN-$TESTDIR-binary-package.tar.gz
• (done) push test binary packages to a central server
  • (done) have fserver accept test binary packages
  • (done) allow someone to upload a test binary package
    • (done) add support for test binary packages to fserver.py
      • (done) create a json meta-data file to hold data for the package

• (done) have Fuego core use the local cache instead of the test's build and deploy phases
  • (done) check for USE_BINARY_PACKAGES during test execution
  • (done) check for existence of binary package during 'build' phase, and skip build
  • (done) check for existence of binary package during 'deploy' phase, and use that instead of materials from build dir

• (done) implement a use_binary_packages configuration option
  • (done) parse the config option in ftc
  • (done) use the config option in functions.sh to allow use of a binary package, if one is found in the cache

• download test binary packages from the central server
  • have ftc check for binary package on server and put in cache (if use_binary_packages config is set), prior to test execution

• improve binary package handling
  • (done) add '--binary' option to ftc package-test (instead of requiring a weird run-test invocation)
    • change 'make_cache.sh' to use 'ftc package-test .. --binary'
  • add '--binary' option to ftc install-test
  • add '--binary' option to ftc put-test
    • eliminate 'ftc put-binary-package' (re-use code)

• improve binary package format and content
  • (done) add &ftest directory at root
  • (done) add deploy directory at root
    • (done) when deploying a binary package, extract just the deploy directory
  • omit source files from the @ftest directory
    • parse source_files from the test.yaml file (using ydump?)
      • ydump test.yaml source_files | sed "s/['[]//g" | sed "s/]//g" | sed "s"/,/ /g"
  • when installing a binary test, put the ftpb in fuego-rw/cache
    • extract &ftest material into fuego-ro/test/<test-name> on the host
  • add spec to binary package filename
  • add version and release to binary package filename

• support running a test with no prior test installation (e.g. in /fuego-core/tests)
  • if test is not present locally, but is available remotely as a binary package, then:
    • install enhanced binary package
      • binary package to /fuego-rw/cache
      • &ftest data to /fuego-rw/tests/<test-name>
  • (optionally), remove test after execution?
  • see some of the steps below for the no-build option

• implement a no-build installation option
  • outline of behavior:
    • after test installation, remove test source materials
      • list test buildable source in test.yaml file
      • use that to avoid including it in test binary package
    • extend test binary package to include test materials (other than source)
      • see above for enhance binary package format (with $TEST_HOME inside the binary package)
        • put $TEST_HOME into <package>/&ftest directory
          • omit test source from &ftest directory
    • lists test on the server (ftc list-tests --remote --binary?)
  • add no-build option to install.sh
  • add no-build option to fuego.conf

• implement a lightweight Fuego test-runner, that can execute on target without the rest of Fuego
  • outline of behavior:
    • list tests on the server (use wget or curl)
    • download a test binary package from the server
    • execute the test_pre_check function
      • need to support assert_has_program and NEED_ROOT
    • execute the test_run function
      • need to support report, report_live, cmd (only as local commands)
    • save the log somewhere, to be picked up by Fuego for post-processing
  • observation: this seems like we're abandoning the normal Fuego host/target model
  • define what needs to be in the binary test package:
    • test_pre_check, test_run, test variables (yes)
      • basically, test_prolog.sh (but there are board-specific items)
    • need fuego_test - what about parser, criteria, etc.?

How to create a binary package [edit section]

The following ftc command can be used to make a binary package using the toolchain for a particular board:

  $ ftc package-test <test> --binary -b <board> [-s spec]

This will place the resulting binary package in the file: /fuego-rw/cache/<toolchain>-<test>-binary-package.ftbp

So, as an example if you ran this command on the board 'local' with test "Functional.hello_world", you would end up creating the binary package: /fuego-rw/cache/x86_64-Functional.hello_world-binary-package.ftbp

---

Note, the old (direct) method still works:

  $ ftc run-test -b <board> -t <test> -p "pbdm"

schema for test binary package meta-data [edit section]

Here are fields that are needed in the meta-data:

• toolchain - what toolchain was used to build the package ($TOOLCHAIN)
• (all toolchain variables):
  • arch ($ARCH)
  • compiler version
• build date ($BUILD_TIMESTAMP)
• build host ($FUEGO_HOST)
• Fuego test name ($TESTDIR)

Here are the fields that eventually got used:

{
    "board": "$NODE_NAME",
    "build_host": "$FUEGO_HOST",
    "fuego_core_version": "$FUEGO_CORE_VERSION",
    "fuego_version": "$FUEGO_VERSION",
    "schema_version": "1.0"
    "test_name": "$TESTDIR",
    "testsuite_version": "$TESTSUITE_VERSION",
    "timestamp": "$BUILD_TIMESTAMP",
    "toolchain:": "$TOOLCHAIN",
}

Missing fields:

• this also needs the test version number ('version' from test.yaml file' as well as 'fuego_release')
  • it looks like testsuite_version = test.yaml::version

contents of a test binary package [edit section]

• whatever is in the deploy directory after the 'deploy' function is called
• binary-package.json with schema indicated above

Notes [edit section]

• you have to have the source package (with fuego_test.sh, parser.py, test.haml, criteria.json, etc.) in order to run the test
• the binary package only has the materials for the target board
• the source package may have files that are not needed, like the source tar

to run a test from the server without any prior test installation [edit section]

• you have to get the test from fserver
  • put the test in /fuego-rw/tests
• you have to get the binary test package from fserver
  • put that in /fuego-rw/cache