; Summary: Fuego does not support arbitrary arrays of results data (non-static testcases)
; Owner: Tim
; Reporter: Daniel
; Status: open
; Priority: medium

= Description =
Fuego currently does not support testcase arrays of arbitrary size
that can be populated dynamically.  The reason is that reference.json
is separate from parser.py, where the results data is separated.

Note: This issue, of dynamic lists of results, came up in Tim's
test standards BOF session at Linaro Connect.  It affects the ability
to have static tguids for all tests, which is a core idea for comparing
data between systems.

Basically, Fuego currently considers test set names, test case names and
measure names to be statically defined.  However, some tests run tests
based on data determined at test run time.

This is a problem for how we currently declare reference values
(measure thresholds) and units (currently found in criteria.json
and reference.json respectively).

= Notes =
Some ideas for how to solve this, without disturbing our current methods
of encoding criteria and reference values, would be to:
{{{
 1) allow wildcarding of criteria and reference values
 2) allow criteria for non-present testcase results (that don't result in an error)
}}}

It might be possible to support a sparse set of tguids within a much larger
space of possible tguids.  For example, for cyclictest, there can be
a latency result per thread, with an arbitrary number of threads supported.
The number of threads depends on the spec and the parameters to the cyclictest program.
 
It is not practical to list all possible pass criteria statically, as it
would require listing an arbitrary number of criteria (by tguid).
We could support 1st thread latency, 2nd thread latency, etc. as static names, in the criteria file, without requiring that the thread2-latency be present in the testcase results.

One area where the testcase space is very large is with compiler error and warnings.  Only the errors and warning cases are output on by the compiler, as 
the set of testcases is innumerable in practice (theoretically consisting of
every possible compiler check, per token of code).  The testcase names can be
synthesized (line #, token, validity-check-name), but it would be impractical
to list them all, and not all validity checks apply to all possible tokens.  So the
actual effective list of such testcase names would be a subset of all possible 
testcase names).  (Also, it should be noted that Fuego has not, to date, had any
pass criteria that reference compiler output.)



; backlink: [[Fuego Issues List]]