Contents: vision input format, markers or tags output current dependencies on Fuego Miscelaneous notes Tim's notes Daniel's notes

This page has information about the project to de-couple Fuego's test log parser, to make it a standalone program.

Neil Williams had some good feedback (some of it negative) on the automated testing mailing list.

This page seeks to organize that content, and capture ideas about the process of making Fuego's parser standalone.

vision [edit section]

Create a standalone test program output parser, that can be used by multiple test systems, so that parsing is not baked into each test system.

Let it develop on its own, independent of a particular test system.

input [edit section]

Run artifacts from a test program

• stdout
• kernel output (console log during run)
• stderr
• return code
• trace during run
• system log during run (with items from the test program)

There can be multiple channels of information to extract data from during a test program run.

format, markers or tags [edit section]

• TAP
• LAVA

output [edit section]

Standardized output in multiple formats:

• junit
• xunit
• kernelci format
• squad format
• run.json (should be similar to kernelci)

current dependencies on Fuego [edit section]

• Fuego environment variables
  • FIXTHIS - list vars here
• Fuego paths?

Miscelaneous notes [edit section]

Tim's notes [edit section]

Up to Fuego 1.5 we invoke the parser with:

   run_python $PYTHON_ARGS $FUEGO_CORE/engine/tests/${TESTDIR}/parser.py

(with a whole lot of Fuego-specific environment variables).

I think it would be good to refactor this, so that the fuego core (functions.sh) calls a single program, indicating the test, the log file, and a parser name. Many fuego parsers could be combined by declaring a single regex pattern in the fuego_test.sh (similar to what is done with log_compare).

So, something like the following instead:

   run_python $PYTHON_ARGS --log=$FUEGO_RW/logs/.../testlog.txt --test=$TESTDIR --parser=TAP13

or

   run_python $PYTHON_ARGS --log=$FUEGO_RW/logs/.../testlog.txt --test=$TESTDIR \
     --parser=regex-2groups --parser-arg="regex_string= ^TEST-(\d+) (.*)$"

Daniel's notes [edit section]

I was thinking about a new python library called (e.g.: "fuegoparse") that you can use from any project (e.g.: install it with pip3) and also be called from the command line.

Example usage as a library:

import fuegoparse
import json

testparser = fuegoparse.TestlogParser(test="LTP", output_format="fuego_run_json")
data = testparser.parse("./testlog.txt")
with open('run.json', 'w') as outfile:
    json.dump(data, outfile)

Example as a command:

  $ fuegoparse -t LTP --log ./testlog.txt --format fuego_run_json -o run.json

Other output formats could be TAP, KernelCI, Squad, Junit, etc.

This could become a whole new project (the first spinoff of the Fuego project!).

We also need to think about isolating other parts (cross-build, dependency checks, transport abstractions), so they can be reusable. And finally be able to say that Fuego is just glue code that puts all those parts together.