Contents: to do information testplan file format spec file format ovgen.py outline, structure and notes ovgen.py classes ovgen.py functions call outline files and notes Desired features notes ovgen features questions prolog file structure the new model

This page has some miscellaneous notes about ovgen, how it is used, and how it might be modified in the future:

ovgen is used to construct a 'prolog' file, which is a shell script which has variables and definitions from the distro and board files, to control the execution of a test. The prolog script is generated when the base script is executed, and is sourced at some point during test execution.

Note that test execution is done in two separate phases. That is, Jenkins executes the test script itself to do the first phases of a test (pre_test, build, deploy, run), then does a "source functions.sh ; post_test" to execute the 'post_test' phase of the test.

to do [edit section]

• figure out what to do about missing specs: *

  missing spec in Benchmark.aim7
  missing spec in Benchmark.bfoo
  missing spec in Benchmark.fs_mark
  missing spec in Benchmark.nbench-byte
  missing spec in Benchmark.sysbench
  missing spec in Functional.arch_timer
  missing spec in Functional.cmt
  missing spec in Functional.fsfuzz
  missing spec in Functional.mesa-demos
  missing spec in Functional.scifab
  missing spec in Functional.sdhi_0
  

• move all plans to their respective test directories
  • bc
  • hello
  • sata, mmc, usb
    • tests are: Benchmark.fio, Benchmark.Bonnie, Benchmark.IOzone, Benchmakr.ffsb, Benchmark.Tiobench, Benchmark.aiostress, Functional.synctest, Benchmakr.Interbench, Benchmark.dbench

• specify per-test testplans in the Jenkins interface
  • have groovy script parse the testdir for testplan names
• ? if spec is 'default', and no test.spec file is found, then just synthesize one with the following:

{
    "testName": "Functional.zlib",
    "specs":
    [
        {
            "name":"default"
        }
    ]
}

• read testplans from overlays/testplans and testdir

information [edit section]

• Functional.bzip2.spec has no variables (no specs besides default)
• Functional.zlib.spec has no variables (no specs besides default)
• Functional.hello_world.spec has different specs

testplan file format [edit section]

testplan<name>.json file has:

• testPlanName
• tests:
  • testName
  • spec

spec file format [edit section]

Functional.<test_name>.json file has:

• testName:
• specs:
  • name
  • var1
  • var2 ...

ovgen.py outline, structure and notes [edit section]

ovgen.py classes [edit section]

• OFClass - overlay file class
  • name, description, funcs, vars, cap_list
• OFLayer - overlay file layer
  • name, description, vars
• TestSpecs - class to hold TestSpec information
  • name, variables, fail_case
• *Exception - Exceptions

ovgen.py functions [edit section]

• debug_print(string, lev=1) - utility routine
• parseOFVars(line, ofc) - check for and parse a OVERLAYFILE var
  • looks for a variable definition starting with OF
  • if OF.NAME and OF.DESCRIPTION are found, record those in ofc.name and description
• parseVars(line, ofc) - check for and parse a variable definition
• parseFunctionBody(name, f) - check for and parse the body of a function
• parseFunction(line, f) - check for and parse a function
• baseParseFunction(line, f, ofc) - parse a function and record it in ofc.funcs
• parseBaseFile(baseFilePath, ofc) - process itens in a class file
  • calls ParseOFVars, parseVars, baseParseFunction (?)
• parseBaseDir(baseDirPath, ofcls) - process all base files in a dir
  • looks for '*.fuegoclass' files
  • calls parseBaseFile
• parseInherit(line, ofcls) - check for and parse 'inherit' line
• parseInclude(line, ofcls) - check for and parse 'include' line
• parseLayerVarOverride(line, layer, inhclass) - check for and parse variable override line
  • looks for 'override VAR "definition"'
• parseLayerFuncOverride(line, layer, inhclass, f) - check for and parse function override
• parseLayerVarDefinition(line, layer, inhclass) - check for and parse variable definition
  • looks for VAR="definition"
• parseLayerCapList(line, layer, inhclass) - check for and parse CAP_ variables
  • looks for BOARD.CAP_LIST=.*
  • adds any items found to the inhclass.cap_list list.
• parseOverrideFile(overrideFile, layer, ofcls) - read an overaly file for vars and override functions
  • return a list of classes (actually, instances)
• generateProlog(outFilePath, ofcls, classes, testdir, testspec) - main routine to generate the output
• generateSpec(ts, fout) - write out the part of prolog from the spec information
• parseSpec(testdir, testspec) - read the spec file
• run(test_args=None) - equivalent to main()
  • parse arguments
  • parseBaseDir() to get classes
  • for each override file, parseOverrideFile()
  • generateProlog()
• testrun() - perform a test run

call outline [edit section]

• run
  • parseBaseDir
    • scan for *.fuegoclass files
    • parseBaseFile
      • parseOFVars
      • parseVars
      • baseParseFunction
        • parseFunction
          • parseFunctionBody
  • parseOverrideFile
    • parseInherit
    • parseInclude
    • parseLayerCapList
    • parseLayerVarOverride
    • parseLayerVarDefinition
    • parseLayerFuncOverride
      • parseFunctionBody
  • generateProlog
    • write vars from classes
    • parseSpec
    • generateSpec

files and notes [edit section]

• file ending in .fuegoclass is a class file - it should define OF.NAME and OF.DESCRIPTION
  • already referred to as 'base' files
• other files are "layer" files.
  • I can't find a use for these, other than debugging
• a layer file must inherit exactly one base file
• a layer file can include as many other files as it wants
• a layer file does not provide independent functions (only function overrides)

Desired features [edit section]

• support empty testplan
• parse testplan from test directory
• support empty test spec
• parse test specs from test directory

notes [edit section]

the overlay format is really just a shell script, with extensions for

• inherit
• include

Are these really needed for the type of object orientation that ovgen supports?

ovgen features [edit section]

• parsing of 'fuegoclass' files (base shell script class?)
• parsing of .dist files (overlay file = shell script class extension files)
• parsing of .board files (overlay file = shell script class extension files)

The model is that ovgen reads the dist, board, plan and spec files and produces the prolog file.

questions [edit section]

• is the prolog file intended to be persistent?
  • is it present after a test run? - yes
    • why? for reference or for reuse?
    • fact: it is overwritten when a new testplan is used
    • fact: it is written every time a test is run (see overlays.sh - rm $OF_OUTPUT_FILE)
  • it's main purpose seems to be for use by the post_test routine, which may be run in a separate invocation from the rest of the base script
  • is it ever re-used after the test completes?
    • does ovgen or any other thing read it or examine it?
• the default arguments for every test are included in the default testplan
  • However, only the arguments for a single spec are included in a custom testplan

• see if default specs are needed by unrelated tests:
  • I know that Functional.hello_world does not need BENCHMARK_DBENCH_NPROCS
  • How can I tell if any test needs variables from another spec?
    • it's referenced in their base script?
    • any required spec should be listed in the testplan!
  • an empty testplan only supports variables for that test?
    • an empty testplan doesn't support any spec variables
• do the following in /home/jenkins/fuego/engine/tests:
  • find . -type f -print0 | xargs -0 grep BENCHMARK
  • find . -type f -print0 | xargs -o grep FUNCTIONAL
• no test vars are used outside their own tests
  • Thus, having a global testplan_default is bogus!
• what happens if a test spec is missing?
  • ovgen.py aborts - if a test spec is mentioned in the testplan but it's not in the list of specs available, then ovgen.py errors out.
• what happens if a test plan is missing?
  • overlays.sh checks for presence of testplan file and aborts the job if not found
• does anyone call ovgen.py besides overlays.sh
  • answer: NO
• is there a use case for having all the default specs in the default testplan?
  • each test has to run it's own overlay generation, so I don't think so

prolog file structure [edit section]

The prolog filename is '<board_name>-prolog.sh"

It has the following sections:

• #class: base-distrib - stuff from the distribution
• #class: base-board - stuff from the board file
• #class: base-params - global/system-wide variables
• #class: base-funcs - global/system-wide functions
• #testplan: default - test variables for the indicated testplan

the new model [edit section]

• individual test shows list of available specs
  • modify testplan.groovy
  • show 'default' and any parsed spec names from the test dir
  • put specs in separate files
    • I think this is already supported!
  • testplan_mmc is no longer required for individual execution of e.g. Benchmark.bonnie

• testplan lists test.spec tuples
  • it already has a list of tuples
• have ftc 'run' a testplan
• job specifies a list of tests and a testplan