Fuego wiki

Login or create account

Batch Tests Feature

Description [edit section]

Batch tests is a feature of Fuego that allows a user to easily define a test that runs other Fuego tests as sub-tests.

This feature is new in Fuego 1.5 and is intended to replace the test plan system.

To see Using Batch Tests for information on writing a batch test.

Rationale [edit section]

The test plan system is quite limited. It doesn't

Features [edit section]

  • can easily compose one or more tests of interest, and run them as a group
  • can run multiple tests with different specs
  • can run different tests in sequence, and stop if something goes wrong
  • can see the visualization of results of sub-tests by following a link in batch test results

  • can be used to compose together a series of operations (like tbot)
    • this could be used to do a "full CI loop" test, where sub-tests consisted of building the code, deploying the system code, booting the system, then running a suite of Fuego tests
  • the batch test can be programmed to continue or stop operation depending on the results of test
  • the batch test can be programmed to run different tests depending on results

To do [edit section]

Here are features required for this:
  • (done) sub-tests can be grouped by batch-id
  • (done) show result for each sub-test
  • (done) add a link from each sub-test's results to that test's visualization data
    • (done) add link to jenkins job build page in parent test's testlog (hostlog.txt)
    • (done) have charting code read that link, and put into testcase chart
    • (done) should add visualization to the jenkins build page
  • (done) generate jobs for sub-tests when creating jobs for batch test
    • (done) have ftc add-job automatically create sub-jobs? (yes)
      • (done) read json from batch test, and use as testplan for creating sub-jobs
      • (done) add to 'ftc add-jobs' - scan for BATCH_TESTPLAN variable in test.
        • scan for BATCH_TESTPLAN variable in test
        • use delimiters BATCH_TESTPLAN= and END_TESTPLAN in
  • (done) create a sample batch job
    • batch_default = testplan_default
    • (done) batch_quicktest = internal testplan
    • (done) batch_bc
  • replace all previous testplans:
    • smoketest
    • docker
    • fuego_nightly1
    • fuego_nightly2
    • hello_world
    • lava
    • ltsi
    • lts
    • reboot
    • filesystem
    • fuego_release
  • (done) decide on naming conventions for batch tests
    • (done) short name of test starts with "batch_"
  • (done) handle 'nested' CALLER situation
    • (done) prevent message about unknown FUEGO_CALLER
      • "Error: fuego_caller has an unexpected value: nested"
    • (defer) automatically handle the 'before log' removal/restoration problem
      • save the log differently if caller is 'nested'?
      • this is a problem for test recursion, not batch tests
  • (done) put supporting routines in the Fuego core:
    • (done) allocate_next_batch_id
    • (done) run_test
    • (done) trim
  • document supporting routines:

TBWiki engine 1.8.3 by Tim Bird