Description [edit section]

= Description =
Batch tests is a feature of Fuego that allows a user to easily define a
test that runs other Fuego tests as sub-tests.

This feature is new in Fuego 1.5 and is intended to replace the test
plan system.

To see [[Using Batch Tests]] for information on writing a batch test.

Rationale [edit section]

== Rationale ==
The test plan system is quite limited.  It doesn't 

Features [edit section]

= Features =
 * can easily compose one or more tests of interest, and run them as a group
 * can run multiple tests with different specs
 * can run different tests in sequence, and stop if something goes wrong
 * can see the visualization of results of sub-tests by following a link in batch test results

• can be used to compose together a series of operations (like tbot) * this could be used to do a "full CI loop" test, where sub-tests consisted of building the code, deploying the system code, booting the system, then running a suite of Fuego tests * the batch test can be programmed to continue or stop operation depending on the results of test * the batch test can be programmed to run different tests depending on results

 * can be used to compose together a series of operations (like tbot)
   * this could be used to do a "full CI loop" test, where sub-tests consisted of building the code, deploying the system code, booting the system, then running a suite of Fuego tests
 * the batch test can be programmed to continue or stop operation depending on the results of test
 * the batch test can be programmed to run different tests depending on results

To do [edit section]

= To do =
Here are features required for this:
 * (done) sub-tests can be grouped by batch-id
   * (done) add batch-id generation function to fuego core - see [[function_allocate_next_batch_id]]
 * (done) show result for each sub-test
 * (done) add a link from each sub-test's results to that test's visualization data
   * (done) add link to jenkins job build page in parent test's testlog (hostlog.txt)
   * (done) have charting code read that link, and put into testcase chart
   * (done) should add visualization to the jenkins build page
 * (done) generate jobs for sub-tests when creating jobs for batch test
   * (done) have ftc add-job automatically create sub-jobs? (yes)
     * (done) read json from batch test, and use as testplan for creating sub-jobs
     * (done) add to 'ftc add-jobs' - scan for BATCH_TESTPLAN variable in test.
       * scan for BATCH_TESTPLAN variable in test
       * use delimiters BATCH_TESTPLAN= and END_TESTPLAN in fuego_test.sh
 * (done) create a sample batch job
   * batch_default = testplan_default
   * (done) batch_quicktest = internal testplan
   * (done) batch_bc
 * replace all previous testplans:
   * smoketest
   * docker
   * fuego_nightly1
   * fuego_nightly2
   * hello_world
   * lava
   * ltsi
   * lts
   * reboot
   * filesystem
   * fuego_release
 * (done) decide on naming conventions for batch tests
   * (done) short name of test starts with "batch_"
 * (done) handle 'nested' CALLER situation
   * (done) prevent message about unknown FUEGO_CALLER
     * "Error: fuego_caller has an unexpected value: nested"
   * (defer) automatically handle the 'before log' removal/restoration problem
     * save the log differently if caller is 'nested'?
     * this is a problem for test recursion, not batch tests
 * (done) put supporting routines in the Fuego core:
   * (done) allocate_next_batch_id
   * (done) run_test
   * (done) trim
 * document supporting routines:
   * [[function_allocate_next_batch_id]]
   * [[function_run_test]]
   * [[function_trim]]