There should be an effort to encourage people to adopt a single test program output format. This would make parsing results easier.

This page is about the output from the test programs, not the output from Fuego itself. For that, see the Unified Results Format Project.

Candidate formats are:

• json - using a schema similar to what kernelci use
• TAP - test anything protocol
• *unit - for the xunit, junit, pyunit family of test frameworks
  • are these are consistent in their output?

Here are projects that promote or use a single test output format:

• https://packages.debian.org/sid/autopkgtest
• https://wiki.yoctoproject.org/wiki/Ptest
• https://testanything.org/ (Test Anything Protocol)
• http://pyunit.sourceforge.net/

Notes on these:

ptest [edit section]

Yocto/OpenEmbedded's ptest framework decouples the compiling phase from the testing phase and produces "a consistent output format".

Ptest normally uses the test suite that comes with the original source code. For example, for the openssh recipe it uses some of the tests inside openssh's "regress" folder.

However, there are many recipes without their corresponding ptest definitions. I'm not sure if that is just because nobody added them yet, or because there was no test suite in the original source code.

TAP [edit section]

Test Anything Protocol

https://testanything.org/tap-version-13-specification.html

Example:

TAP version 13
1..6
#
# Create a new Board and Tile, then place
# the Tile onto the board.
#
ok 1 - The object isa Board
ok 2 - Board size is zero
ok 3 - The object isa Tile
ok 4 - Get possible places to put the Tile
not ok 5 - Placing the tile produces no error
  ---
  message: Could not place tile at row 1, column 1
  severity: fail
  ...
ok 6 - Board size is 1