Update Criteria

This page has information about a feature in progress, called "update criteria". This is the ability for Fuego users to update the criteria for a test using ftc.

The command to do this will be: ftc set-criteria.

It will read existing criteria from the currently applicable criteria file, and write a new criteria file to /fuego-rw/boards

to do for this feature [edit section]

operation [edit section]

This command should allow users to:

Should be able to set the criteria list based on currently observed behavior. That is, do something like:

The first one finds existing testcases that fail, and adds them to the 'fail_ok_list'.

The second one finds the current average max_latency, adds 10% to it, and saves it as the new reference value.

command syntax [edit section]

use "from run <src>", to use the value for this tguid from an existing data set (one run, or multiple runs)

src = run <run_id> (a specific run) src = run all (all previous runs)

What can be scripted?

manual operations [edit section]

Notes [edit section]

Expected Workflow [edit section]

A users sees that a test fails, and then determines they want to ignore it.

If they want to preserve the criteria file, as part of fuego-ro, then copy it from /fuego-rw/boards to /fuego-ro/boards.

Sometimes, if you have started ignoring failures, you want to check to see if they are still failing. You can:

More information for users [edit section]

It would be nice to put more information into the run.json file about the reason for a failure, or the reason that a failure was ignored. We could copy the comment from the criteria.json file, into the run.json file, and that might help users see what's going on.

We should save "reason" information in the run.json file.