Description [edit section]

Batch tests is a feature of Fuego that allows a user to easily define a test that runs other Fuego tests as sub-tests.

This feature is new in Fuego 1.5 and is intended to replace the test plan system.

To see Using Batch Tests for information on writing a batch test.

Rationale [edit section]

The test plan system is quite limited. It doesn't

Features [edit section]

• can easily compose one or more tests of interest, and run them as a group
• can run multiple tests with different specs
• can run different tests in sequence, and stop if something goes wrong
• can see the visualization of results of sub-tests by following a link in batch test results

• can be used to compose together a series of operations (like tbot)
  • this could be used to do a "full CI loop" test, where sub-tests consisted of building the code, deploying the system code, booting the system, then running a suite of Fuego tests
• the batch test can be programmed to continue or stop operation depending on the results of test
• the batch test can be programmed to run different tests depending on results

To do [edit section]

Here are features required for this:

• (done) sub-tests can be grouped by batch-id
  • (done) add batch-id generation function to fuego core - see function_allocate_next_batch_id
• (done) show result for each sub-test
• (done) add a link from each sub-test's results to that test's visualization data
  • (done) add link to jenkins job build page in parent test's testlog (hostlog.txt)
  • (done) have charting code read that link, and put into testcase chart
  • (done) should add visualization to the jenkins build page
• (done) generate jobs for sub-tests when creating jobs for batch test
  • (done) have ftc add-job automatically create sub-jobs? (yes)
    • (done) read json from batch test, and use as testplan for creating sub-jobs
    • (done) add to 'ftc add-jobs' - scan for BATCH_TESTPLAN variable in test.
      • scan for BATCH_TESTPLAN variable in test
      • use delimiters BATCH_TESTPLAN= and END_TESTPLAN in fuego_test.sh
• (done) create a sample batch job
  • batch_default = testplan_default
  • (done) batch_quicktest = internal testplan
  • (done) batch_bc
• replace all previous testplans:
  • smoketest
  • docker
  • fuego_nightly1
  • fuego_nightly2
  • hello_world
  • lava
  • ltsi
  • lts
  • reboot
  • filesystem
  • fuego_release
• (done) decide on naming conventions for batch tests
  • (done) short name of test starts with "batch_"
• (done) handle 'nested' CALLER situation
  • (done) prevent message about unknown FUEGO_CALLER
    • "Error: fuego_caller has an unexpected value: nested"
  • (defer) automatically handle the 'before log' removal/restoration problem
    • save the log differently if caller is 'nested'?
    • this is a problem for test recursion, not batch tests
• (done) put supporting routines in the Fuego core:
  • (done) allocate_next_batch_id
  • (done) run_test
  • (done) trim
• document supporting routines:
  • function_allocate_next_batch_id
  • function_run_test
  • function_trim