Each benchmark should include an executable file called 'parser.py' in the test directory (/fuego-core/engine/tests/Benchmark.<testname>). Functional tests may also provide a parser.py, when they return more than a single testcase result from the test. However, this is optional. If a Functional test does not have a parser.py script, then a generic one is used (called generic_parser.py), that just sets the result for the test based on the return code from the test program and the single result from running executing log_compare in the test_processing portion of the test script.
The test log for the current run is parsed by parser.py, and one or more testcase results (measures or pass/fail results) are extracted, and then provided via a dictionary to the results processing engine. Normally this is done by scanning the log using simple regular expressions. However, since this is a python program, an arbitrarily complex parser can be written to extract result data from the test log.
- import the parser library
- specify a search pattern for finding one or more measurements (or testcases) from the test log
- call the parse_log function, to get a list of matches for the search pattern
- build a dictionary of result values
- call the process function, to save the information to the aggregate results files, and to re-generate the chart data for the test
- the process() function evaluates the results from the test, and determines the overall pass/fail status of a test, based on a criteria.json file
These test names must be consistent in the parser.py program, reference.json file and the criteria.json file.
Please see Fuego naming rules for rules and guidelines for test names in the Fuego system.
Note the two calls to parser library functions: parse_log() and process().
This is used to add /fuego-core/engine/scripts/parser to the python system path, for importing the 'common.py' module (usually as internal module name 'plib').
The parser library expects the following environment variables to be set:
parser.py is called with the following invocation, from processing: