Fuego wiki

Test results formats

Contents:

kernelCI
avacado
xunit
junit
LTP
TAP
autotest
ctest

Here is a collection of test results formats from different programs:

kernelCI [edit section]

kernelCI
- see https://api.kernelci.org/schema-test.html
- and https://api.kernelci.org/collection-test.html

overview:
- test_suite:
  - version
  - _id:
  - name:
  - created_on: date:
  - lab_name:
  - time: duration
  - job:
  - kernel:
  - defconfig
  - defconfig_full:
  - arch:
  - board:
  - board_instance:
  - job_id:
  - build_id:
  - boot_id:
  - test_set: list of test sets
    - test_set: can be more than one
      - version:
      - _id:
      - name:
      - created_on: date:
      - time: duration
      - test_suite_id
      - test_case - list of test cases
      - definition_uri, vcs_commit
      - metadata
      - parameters: key-value pairs used during the test
      - test_job_url:
      - test_job_path:
      - test_job_id:
      - defects: the defects associated with the test set
  - test_case: list of test cases
    - test_case: can be more than one
      - version:
      - _id:: oid
      - name:
      - created_on: date:
      - test_set_id:
      - test_suite_id:
      - parameters: key-value pairs used during the test
      - status:
      - time: duration
      - definition_uri, vcs_commit
      - metadata
      - measurements: list of measurements
        name
        time
        unit
        measure
      - minimum:
      - maximum:
      - samples:
      - samples_sum:
      - samples_sqr_sum:
      - attachments:
  - metadata:
  - definition_uri:
  - vcs_commit:

avacado [edit section]

avacado
- see http://avocado-framework.readthedocs.io/en/53.0/ResultFormats.html
- by default, the machine-readable output format for avacado is xunit
- avacado can also output in it's own json format (but it doesn't claim that it will be a standard)

Kinds of results:

see http://avocado-framework.readthedocs.io/en/53.0/WritingTests.html
- PASS - test passed
- WARN - test passed, but something occurred which may need review
- SKIP - test was skipped
- CANCEL - test was canceled before execution
- FAIL - test failed
- ERROR - test encountered unexpected error (not a failure)
- INTERRUPTED - test was interrupted (timeout or ctrl-C) (not set by test but by framework)

Nomenclature:

whiteboard = place for a test to save information that might be used by other tests or the system
job id = sha1 hash
variant id = like a spec
test id = like our test name *

xunit [edit section]

xunit
- see http://help.catchsoftware.com/display/ET/JUnit+Format
- and a schema at: https://gist.github.com/erikd/4192748

junit [edit section]

junit
- see http://llg.cubic.org/docs/junit/
- overview:
  - testsuites: summary of status from all testsuites
    - disabled, errors, failures, tests: counts of tests with these conditions
    - name:
    - time: duration in seconds
    - testsuite: one testsuite instance (can appear multiple times)
      - disabled, errors, failures, tests, skipped: counts of these conditions
      - name: name of the test
      - hostname:
      - id:
      - package:
      - time: duration
      - timestamp: start time
      - properties: list of properties
        property: name, value: values used during test
      - testcase: can appear multiple times
        name
        assertions: count of assertions
        classname:
        status
        time
        skipped: message:
        error: message:
        failure: message:
        system-out: stdout text
        system-err: stderr text
      - system-out: stdout text
      - system-err:

Here's another presentation of the format:

<testsuites>        => the aggregated result of all junit testfiles
  <testsuite>       => the output from a single TestSuite
    <properties>    => the defined properties at test execution
      <property>    => name/value pair for a single property
      ...
    </properties>
    <error></error> => optional information, in place of a test case (e.g. if tests could not be ound etc.)
    <testcase>      => the results from executing a test method
      <system-out>  => data written to System.out during the test run
      <system-err>  => data written to System.err during the test run
      <skipped/>    => test was skipped
      <failure>     => test failed
      <error>       => test encountered an error
    </testcase>
    ...
  </testsuite>
  ...
</testsuites>

Kinds of results:

error = unanticipated problem with test (different from failure)
failiure = test failed (condition that was tested resulted in wrong answer)
skipped = test was skipped
<nothing> = if a testcase is listed, but does not have a qualifying attribute of error, failure or skipped, then it passed.

Nomenclature:

assertions = number of conditions checked during the test

LTP [edit section]

Each of the tests in the "main" body of LTP produces a report that looks like the following:

unlink07    1  TPASS  :  unlink(<non-existent file>) Failed, errno=2
unlink07    2  TPASS  :  unlink(<path is empty string>) Failed, errno=2
unlink07    3  TPASS  :  unlink(<path contains a non-existent file>) Failed, errno=2
unlink07    4  TPASS  :  unlink(<address beyond address space>) Failed, errno=14
unlink07    5  TPASS  :  unlink(<path contains a regular file>) Failed, errno=20
unlink07    6  TPASS  :  unlink(<address beyond address space>) Failed, errno=14
unlink07    7  TPASS  :  unlink(<pathname too long>) Failed, errno=36
unlink07    8  TPASS  :  unlink(<negative address>) Failed, errno=14

ltprun produces more data about each test that is run, like so:

<<<test_start>>>
tag=unlink07 stime=1502315351
cmdline="unlink07"
contacts=""
analysis=exit
<<<test_output>>>
unlink07    1  TPASS  :  unlink(<non-existent file>) Failed, errno=2
unlink07    2  TPASS  :  unlink(<path is empty string>) Failed, errno=2
unlink07    3  TPASS  :  unlink(<path contains a non-existent file>) Failed, errno=2
unlink07    4  TPASS  :  unlink(<address beyond address space>) Failed, errno=14
unlink07    5  TPASS  :  unlink(<path contains a regular file>) Failed, errno=20
unlink07    6  TPASS  :  unlink(<address beyond address space>) Failed, errno=14
unlink07    7  TPASS  :  unlink(<pathname too long>) Failed, errno=36
unlink07    8  TPASS  :  unlink(<negative address>) Failed, errno=14
<<<execution_status>>>
initiation_status="ok"
duration=0 termination_type=exited termination_id=0 corefile=no
cutime=0 cstime=0
<<<test_end>>>

The data from several different tests is consolidated into a single file by ltprun. There is one line per test program:

startup='Wed Aug  9 21:48:52 2017'
tag=access01 stime=1502315332 dur=0 exit=exited stat=32 core=no cu=0 cs=0
tag=access03 stime=1502315332 dur=0 exit=exited stat=32 core=no cu=0 cs=0
tag=alarm01 stime=1502315332 dur=0 exit=exited stat=0 core=no cu=0 cs=0
tag=alarm02 stime=1502315332 dur=0 exit=exited stat=0 core=no cu=0 cs=0
tag=alarm03 stime=1502315332 dur=0 exit=exited stat=0 core=no cu=0 cs=0
tag=asyncio02 stime=1502315332 dur=0 exit=exited stat=0 core=no cu=0 cs=0
tag=brk01 stime=1502315332 dur=0 exit=exited stat=0 core=no cu=0 cs=0
tag=chdir02 stime=1502315332 dur=0 exit=exited stat=0 core=no cu=0 cs=0
tag=chmod02 stime=1502315332 dur=0 exit=exited stat=0 core=no cu=0 cs=0
...

Key:

tag = program name or test name
stime = start time
dur = test duration in seconds
exit = exit status
stat = exit code
core = was a 'core' file produced (did the program fault)
cu = user time of program
cs = system time of program

LTP Kinds of results:

TPASS: test passed
TFAIL: test case had an unexpected result and failed.
TBROK: remaining test cases are broken and will not execute correctly, because some precondition not met, such as a resource not being available.
TCONF: test case was not written to run on the current hardware or software configuration such as machine type, or, kernel version.
TRETR: test cases is retired and should not used
TWARN: test case experienced an unexpected or undesirable event that should not affect the test itself such as being unable to cleanup resources after the test finished.
TINFO: Provides extra information about the test cast - does not indicate a problem.

TAP [edit section]

TAP stands for "test anything protocol", and it defines an output format for tests that is simple and can be parsed easily by lots of different tools.

The specification is here:

see https://testanything.org/tap-version-13-specification.html

Other resources:

Jenkins supports TAP format: https://wiki.jenkins-ci.org/display/JENKINS/TAP+Plugin
Python has a TAP parser: https://pypi.python.org/pypi/tap.py

Kinds of results:

ok - test passed (may have SKIP on line)
not ok - test failed

Nomenclature:

Bail out - test was aborted without finishing (could be due to missing pre-requisites)
SKIP - test was skipped due to missing pre-requisites

autotest [edit section]

ctest [edit section]

Someone commented that ctest output is good, but I can't find a specification for the output format.