Fuego wiki

Login or create account

Benchmark parser notes in 'raw' format

Here are notes about different elements of the log parser and
plot processing.

= files =
There are several files used:
 * system-wide (parser and results processing library):
   * [[]]
 * per-test:
   * inputs and control files:
     * [[]]
     * [[chart_config.json]]
       * replaces metrics.json
     * [[criteria.json]]
       * replaces reference.log
     * [[reference.json]]
       * replaces reference.log
   * outputs:
     * results.json
     * flat_plot_data.txt
     * flot_chart_data.json
 * per-run:
   * run.json
= overall flow =
The following occurs, during the processing phase of a test:
   * after running the test, fuego calls [[function_processing|processing]]
     * this calls
        * imports (usually as plib)
        * calls plib.parse_log with a search pattern
          * read the test log, and returns a list of matches
        * puts the results into a python dictionary
        * calls plib.process()
          * evaluates the results, according to '''criteria.json''' file
          * writes the results to '''results.json'''
          * writes the results to '''flat_plot_data.txt'''
          * generates chart data
            * uses '''chart_config.json''' as input
            * put output in '''flot_chart_data.json'''

In the Jenkins interface:
 * On the status page for the Benchmark test, there are plots automatically generated from the measure data from the test runs for the benchmark.
   * Jenkins calls the flot (jquery plotter) plugin to show charts for the data based on the flot_chart_data.json file
     * the code that reads the flot_chart_data.json is in /fuego-core/engine/scripts/mod.js

= parser elements =
FIXTHIS - These are described elsewhere.

== legacy files ==
These files are deprecated, but are still supported by Fuego:

=== reference.log ===
This file has been replaced by '''criteria.json'''.  However, if a test
does not have a criteria.json file, this file will be read and converted
into the necessary structure by the parser library.

Each section in the reference.log file has a threshold specifier, which consists
of 2 lines:
 * [metric operator]
 * threshold value

The metric operator line specifies the name of the metric, a vertical bar ('|'), and either 'ge' or 'le', all inside square brackets.

The threshold value is a numeric value to compare the benchmark result with,
to indicate success or failure.


See [[reference.log]]

TBWiki engine 1.8.2 by Tim Bird