Jenkins Visualization

Fuego test results are presented in the Jenkins interface via a number of mechanisms.

built-in Jenkins status [edit section]

Jenkins automatically presents the status of the last few tests (jobs) that have been executed, on the job page for those jobs.

A list of previous builds of the job are shown in the left-hand pane of the page for the job, showing colored balls indicating the test status.

FIXTHIS - add more details here

flot plugin [edit section]

The flot plugin for Jenkins provides visualization of Fuego test results.

In Fuego 1.0 and previous (JTA), this plugin only showed plot data for Benchmark results. In Fuego 1.2, all tests have charts presented, showing recent test results. For benchmarks, the results are shown as plots (graphs) of measure data, and for functional tests, tables are shown with either individual results for each testcase, or summary data for the testsets in the test.

See flot for more information.

Charting details [edit section]

Fuego results charts consists of either plots (a graph of results versus build number) or tables (a table of results versus build number).

There are 3 different chart output options in Fuego 1.2:

A user can control what type of visualization is used for a test using a file called chart_config.json. This file is in the test directory for each individual test. See the wiki page for that file for additional details.

Scope of data displayed [edit section]

The page for a particular job, in Jenkins, shows the data for all of the specs and boards related to the test. This can be confusing, but it allows users to compare results between boards, and between different test specs for the same test.

For example, a job that runs the test Benchmark.bonnie, using the 'default' test spec job (e.g. board1.default.Benchmark.bonnie) shows results for:

Planned for the future [edit section]

In future releases of Fuego, additional chart types are envisioned:

A fourth chart type is:

Detailed chart information [edit section]

Internally, output (by the flot module) is controlled by a file called: flot_chart_data.json

Inside that, there is a data structure indicating the configuration for one or more charts, called the chart_config. This is placed there during chart processing, by the results parser system. A section of that file, the chart_config element, is a direct copy of the data from chart_config.json that comes from the test directory for the test.

information flow [edit section]

The internal module fuego_parser_results.py is used to generate results.json. That module takes the results from multiple run.json files, and puts it into a single results file.

The internal module prepare_chart_data.py is used to generate flat_plot_data.txt. The data in this file is stored as a series of text lines, one per result for every testcase in every run of the test.

This file is then used to create a file called: flot_chart_data.json, which has the data pre-formated as either 'flot' data structures, or HTML tables.

A file called chart_config.json is used to determine what type of charts to include in the file, and what data to include.

Here's an ASCII diagram of this flow:

Note: Programs are in rectangles (with '+' corners), and data files are in "double-line rounded rectangles" with '/' and '\' corners.

+-------------+    //===========\\    +---------+    //========\\
|test program | -> ||testlog.txt|| -> |parser.py| -> ||run.json|| ----+
+-------------+    \\===========//    +---------+    \\========//     |
                                                                      |
  +-------------------------------------------------------------------+
  |
  |   +---------------------+     //==================\\
  +-> |prepare_chart_data.py| <-> ||flat_plot_data.txt||
      +---------------------+     \\==================//
          ^              |
          |              |
//=================\\    |   //===============================\\    +------+
||chart_config.json||    +-> ||  flot_chart_data.json         || -> |mod.js| -> (table or graph)
\\=================//        ||(HTML table or flot graph data)||    +------+
                             \\===============================//

The flot program mod.js is used to draw the actual plots and tables based on flot_chart_data.json. mod.js is included in the web page for the job view by Jenkins (along with the base flot libraries and jquery library, which flot uses).

measure_plot [edit section]

A measure_plot is a graph of measures for a benchmark, with the following attributes:
  title=<test>-<testset>
  X series=build number
  Y1 series=result
  Y1 label=<board>-<spec>-<test>-<kernel>-<tguid>
  Y2 series=ref
  Y2 label=<board>-<spec>-<test>-<kernel>-<tguid>-ref

It plots measures (y) versus build_numbers.

Here's example data for this:

FIXTHIS - add testset, and draw one plot per testset.

measure_table [edit section]

A measure_table is a table of test spec with the following attributes:

Here was the format of the first attempt:

  title=<board>-<test>-<spec> (kernel)
  headers:
     board:
     kernel(s):
     test spec:
  ---------------------------------------------------------------
                            |    build number
  measure items  | test set |   b1   |   b2   |   b3   |   bN   |
  X1             |  <ts1>   | value1 | value2 | value3 | valueN |
  X1(ref)        |  <ts1>   | ref(X1)| ref(X1)| ref(X1)| ref(X1)|
  <bn>           |  <ts2>   |                ...
    (row-span    |  <ts2>   |                ...
  as appropriate)|  <ts3>   |                ...
  <b2n>          |  <ts3>   |                ...

And, 'valueN' is displayed in a correct color, e.g. GREEN if value1 is in the expectation interval specified by 'ref', otherwise in RED, so that we can display more info in a chart.

testcase_table [edit section]

A testcase_table is a table of testcases (usually for a functional test), with the following attributes:
  title=<board>-<spec>-<test>-<kernel>-<tguid>
  headers:
     board:
     test:
     kernel:
     tguid:
  row=(one per line with matching tguid in flat_chart_data.txt)
  columns=build_number, start_time/timestamp, duration, result

It shows testcase results by build_id (runs).

Daniel's table has:

  overall title=<test>
    chart title=<board>-<spec>-<testset>-<testcase>
    headers:
       board:
       kernel_version:
       test_spec:
       test_case:
       test_plan:
       test_set:
       toolchain:
   build number | status | start_time | duration

Cai's table has:

   overall title=<test>
   summary:
      latest total:
      latest pass:
      latest fail
      latest untest:
   table:
   "no" | <test-name>  | test time |
                               | start-time |
                               | end-time |
                               | board version |
                               | test dir |
                               | test device |
                               | filesystem |
                               | command line |
   --------------------------------------------
   testcase number | testcase     | result |

This shows the result of only one run (the latest)

Tim's testcase table has: (one table per board-testname-testset)

   overall title=<test>
   header:
     board
     kernel version
     spec?
     filesystem
     test directory?
     command line?
   --------------------------------------------
   tguid | results
         | build_number |
         | b1 | b2 | bn |
   <tguid1>|result1|result2|resultn|
        totals
   pass: |    |    |    |
   fail: |    |    |    |
   skip: |    |    |    |
   error:|    |    |    |
   --------------------------------------------

testset_summary_table [edit section]

A testset_summary_table is a table of testsets (usually for a complex functional test) with the following attributes:

  title=<board>
  headers:
     board:
     kernel(s):
  -----------------------------------------------------
                            |    counts
  build number   | test set | pass | fail| skip | err |
  <bn>           |  <ts1>   |
    (row-span    |  <ts2>   |
  as appropriate)|  <ts3>   |
  <b2n>          |  <ts1>   |
                 |  <ts2>   |

Here was the format of the first attempt:

  title=<board>-<spec>-<test>-<kernel>
  headers:
     board:
     test:
     kernel:
  -------------------------------------------------------------------------
                                                |    counts
  testset | build_number | start time| duration | pass | fail| skip | err |
  <ts>    | ...|


It shows testset summary results by runs

Here's an alternate testset summary arrangement, that I'm not using at the moment:

   --------------------------------------------
   testset | results
           | b1                      | b2    | bn    |
           | pass | fail | skip | err |p|f|s|e|p|f|s|e|
   <ts>    | <cnt>| <cnt>| <cnt>| <cnt>...            |
        totals
   --------------------------------------------

testset_summary_plot [edit section]

A testset_summary_plot is a graph of testsets (usually for a complex functional test) with the following attributes:
  title=<board>-<spec>-<test>-<kernel>
  X series=build number
  Y1 series=pass_count
  Y1 label=<board>-<spec>-<test>-<kernel>-<testset>-pass
  Y2 series=fail_count
  Y2 label=<board>-<spec>-<test>-<kernel>-<testset>-fail

It graphs testset summary results versus build_ids

structure of chart_data.json [edit section]

Here's an example:

feature deferred to a future release [edit section]

Architecture for generic charting [edit section]

Assuming you have a flat list of entries with attributes for board, testname, spec, tguid, result, etc., then you can use treat this like a sql database, and do the following:

There's a similar set of data (keys, looping) for defining plot data. With keys selecting the axes.