In Fuego, a testplan is used to specify a set of tests to execute, and the settings to use for each one.
fuego-core/engine/overlay/testplans. Each testplan should start with the prefix "testplan_", and end with the filename extension ".json".
Each of these attributes, except for 'tests' has a value that is a string. Here are their meaninings and legal values:
The testPlanName is the name of this testplan. It must match the filename that holds this testplan (without the "testplan_" prefix or ".json" extension. This object is required.
'tests' is a list of tests that are part of this testplan. See below for a detailed description of the format of an element in the 'tests' list. This object is required.
These values are used if the related setting is not specified in the individual test definition.
For example, the testplan might define a default_timeout of "15m" (meaning 15 minutes). The plan could indicate timeouts different from this (say 5 minutes or 30 minutes) for individual tests, but if a test in the testplan doesn't indicate its own timout it would default to the one specified as the default at the top level of the testplan.
The ability to specify per-plan and per-test settings makes it easier to manage these settings to fit the needs of your Fuego board or lab.
Note that if neither the individual test nor the testplan provide a default value is not provided, then a Fuego global default value for that setting will be used.
Note that the default_spec specifies the name of the test spec to use for the test (if one is not is specified for the individual test definition). The name should match a spec that is defined for every test listed in the plan. Usually this will be something like "default", but it could be something that is common for a set of tests, like 'mmc' or 'usb' for filesystem tests.
See the individual test definitions for descriptions of these different test settings objects.
All object values are strings.
- 's' for seconds
- 'm' for minutes
- 'h' for hours
- 'd' for days
Most commonly, a number of minutes is specified, like so:
- "default_timeout" : "15m",
If no 'timeout' is specified, then the value of 'default_timeout' for this testplan is used.
If the value is 'false', then Fuego will do the following, when executing the test:
- if the test program is not built, then build it
- if the test program is already built, then use the existing test program
If the value is 'true', then Fuego will do the following:
- remove any existing program build directory and assets
- build the program (including fetching the source, unpacking it, and executing the instructions in the test's "test_build" function)
A user can always edit a Jenkins job (for a Fuego test), to override the test settings for that job.
The precedence of the settings encoded into the job definition at job creation time are:
- testplan individual test setting (highest priority)
- testplan default setting
- fuego default setting
The precedence of settings at job execution time are:
- 'ftc run-test' command line option setting (highest priority)
- fuego default setting