Summary

Fuego does not support arbitrary arrays of results data (non-static testcases)

Owner

Tim

Reporter

Daniel

Status

open

Priority

medium

Description [edit section]

Fuego currently does not support testcase arrays of arbitrary size that can be populated dynamically. The reason is that reference.json is separate from parser.py, where the results data is separated.

Note: This issue, of dynamic lists of results, came up in Tim's test standards BOF session at Linaro Connect. It affects the ability to have static tguids for all tests, which is a core idea for comparing data between systems.

Basically, Fuego currently considers test set names, test case names and measure names to be statically defined. However, some tests run tests based on data determined at test run time.

This is a problem for how we currently declare reference values (measure thresholds) and units (currently found in criteria.json and reference.json respectively).

Notes [edit section]

Some ideas for how to solve this, without disturbing our current methods of encoding criteria and reference values, would be to:

 1) allow wildcarding of criteria and reference values
 2) allow criteria for non-present testcase results (that don't result in an error)

It might be possible to support a sparse set of tguids within a much larger space of possible tguids. For example, for cyclictest, there can be a latency result per thread, with an arbitrary number of threads supported. The number of threads depends on the spec and the parameters to the cyclictest program.

It is not practical to list all possible pass criteria statically, as it would require listing an arbitrary number of criteria (by tguid). We could support 1st thread latency, 2nd thread latency, etc. as static names, in the criteria file, without requiring that the thread2-latency be present in the testcase results.

One area where the testcase space is very large is with compiler error and warnings. Only the errors and warning cases are output on by the compiler, as the set of testcases is innumerable in practice (theoretically consisting of every possible compiler check, per token of code). The testcase names can be synthesized (line #, token, validity-check-name), but it would be impractical to list them all, and not all validity checks apply to all possible tokens. So the actual effective list of such testcase names would be a subset of all possible testcase names). (Also, it should be noted that Fuego has not, to date, had any pass criteria that reference compiler output.)

backlink

Fuego Issues List