Fuego Developer Notes

This page has some detailed notes about Fuego, Jenkins and how they interact:

Resources [edit section]

Here are some pages in this wiki with developer information:

Notes [edit section]

specific questions to answer [edit section]

What happens when you click on the "run test" button:

Each Jenkins node is defined in Jenkins in:/var/lib/jenkins/nodes/config.xml

Jenkins calls:

Some Jenkins notes: Jenkins stores its configuration in plain files under JENKINS_HOME You can edit the data in these files using the web interface, or from the command line using manual editing (and have the changes take affect at runtime by selecting "Reload configuration from disk".

By default, Jenkins assumes you are doing a continuous integration action of "build the product, then test the product". It has default support for Java projects.

Fuego seems to use distributed builds (configured in a master/slave fashion).

Jenkins home has (from 2007 docs):

The docker container interfaces to the outside host filesystem via the following links:

What are all the fields in the "configure node" dialog: Specifically:

The fuego-core repository has: 

 engine
   overlays - has the base classes for fuego functions
     base - has core shell functions
     testplans - has json files for parameter specifications
     distribs - has shell functions related to the distro
   scripts - has fuego scripts and programs
    (things like overlays.sh, loggen.py, parser/common.py, ovgen.py, etc.
   slave.jar - java program that Jenkins calls to execute a test
   tests - has a directory for each test
     Benchmark.foo
       Benchmark.foo.spec
       foo.sh
       test.yaml
       reference.log
       parser.py
     Functional.bar
     LTP
     etc.

What is groovy:

What plugins are installed with Jenkins in the JTA configuration?

Which of these did Cogent write?

What scriptler scripts are included in JTA?

What language are scriptler scripts in?

What is the Maven plugin for Jenkins?

Jenkins refers to a "slave" - what does this mean?

How the tests work [edit section]

A simple test that requires no building is Functional.bc

This runs a shell script on target to test the 'bc' program.

Functional.bc has the files: 

    bc-script.sh
       declares "tarball=bc-script.tar.gz"
       defines shell functions:
         test_build - calls 'echo' (does nothing)
         test_deploy - calls 'put bc-device.sh'
         test_run - calls 'assert_define', 'report'
           report references bc-device.sh
         test_processing - calls 'log_compare'
           looking for "OK"
       sources $JTA_SCRIPTS_PATH/functional.sh
     bc-script.tar.gz
       bc-script/bc-device.sh

Variables used (in bc-script.sh): 

   BOARD_TESTDIR
   TESTDIR
   FUNCTIONAL_BC_EXPR
   FUNCTIONAL_BC_RESULT

A simple test that requires simple building: Functional.synctest

This test tries to call fsync to write data to a file, but is interupted with a kill command during the fsync(). If the child dies before the fsync() completes, it is considered success.

It requires shared memory (shmget, shmat) and semaphore IPC (semget and semctl) support in the kernel.

Functional synctest has the files: 

     synctest.sh
       declares "tarball=synctest.tar.gz"
       defines shell functions:
         test_build - calls 'make'
         test_deploy - calls 'put'
         test_run - calls 'assert_define', hd_test_mount_prepare, and 'report'
         test_processing - calls 'log_compare'
           looking for "PASS : sync interrupted"
       sources $JTA_SCRIPTS_PATH/functional.sh
     synctest.tar.gz
       synctest/synctest.c
       synctest/Makefile
     synctest_p.log
       has "PASS : sync interrupted"

Variables used (by synctest.sh) 

   CFLAGS
   LDFLAGS
   CC
   LD
   BOARD_TESTDIR
   TESTDIR
   FUNCTIONAL_SYNCTEST_MOUNT_BLOCKDEV
   FUNCTIONAL_SYNCTEST_MOUNT_POINT
   FUNCTIONAL_SYNCTEST_LEN
   FUNCTIONAL_SYNCTEST_LOOP

(NOTE: could be improved by checking for CONFIG_SYSVIPC in /proc/config.gz
to verify that the required kernel features are present)

MOUNT_BLOCKDEV and MOUNT_POINT are used by 'hd_test_mount_prepare'
but are prefaced with FUNCTIONAL_SYNCTEST or BENCHMARK_BONNIE


-------
from clicking "Run Test", to executing code on the target...
config.xml has the slave command: /home/jenkins/slave.jar
  -> which is a link to /home/jenkins/jta/engine/slave.jar

overlays.sh has "run_python $OF_OVGEN ..."
where OF_OVGEN is set to "$JTA_SCRIPTS_PATH/ovgen/ovgen.py"

How is overlays.sh called?
  it is sourced by /home/jenkins/scripts/benchmarks.sh and
    /home/jenkins/scripts/functional.sh

functional.sh is sourced by each Funcational.foo script.


For Functional.synctest:
{{{
Functional.synctest/config.xml
  for the attribute <hudson.tasks.Shell> (in <builders>)
    <command>....
      souce $JTA_TESTS_PATH/$JOB_NAME/synctest.sh</command>

synctest.sh
  '. $JTA_SCRIPTS_PATH/functional.sh'
     'source $JTA_SCRIPTS_PATH/overlays.sh'
     'set_overlay_vars'
         (in overlays.sh)
         run_python $OF_OVGEN ($JTA_SCRIPTS_PATH/ovgen/ovgen.py) ...
                $OF_OUTPUT_FILE ($JTA_SCRIPTS_PATH/work/${NODE_NAME}_prolog.sh)
           generate xxx_prolog.sh
         SOURCE xxx_prolog.sh

     functions.sh pre_test()

     functions.sh build()
        ... test_build()

     functions.sh deploy()

     test_run()
       assert_define()
       functions.sh report()

NOTES about ovgen.py [edit section]

What does this program do?

Here is a sample command line from a test console output: 

python /home/jenkins/scripts/ovgen/ovgen.py \
  --classdir /home/jenkins/overlays//base \
  --ovfiles /home/jenkins/overlays//distribs/nologger.dist /home/jenkins/overlays//boards/bbb.board \
  --testplan /home/jenkins/overlays//testplans/testplan_default.json \
  --specdir /home/jenkins/overlays//test_specs/ \
  --output /home/jenkins/work/bbb_prolog.sh

So, ovgen.py takes a classdir, a list of ovfiles a testplan and a specdir, and produces a xxx_prolog.sh file, which is then sourced by the main test script

Here is information about ovgen.py source: 

Classes:
 OFClass
 OFLayer
 TestSpecs


Functions:
 parseOFVars - parse Overlay Framework variables and definitions
 parseVars - parse variables definitions
 parseFunctionBodyName
 parseFunction
 baseParseFunction
 parseBaseFile
 parseBaseDir
 parseInherit
 parseInclude
 parseLayerVarOverride
 parseLayerFuncOverride
 parseLayerVarDefinition
 parseLayerCapList - look for BOARD.CAP_LIST
 parseOverrideFile
 generateProlog
 generateSpec
 parseGenTestPlan
 parseSpec
 parseSpecDir
 run

Sample generated test script [edit section]

bbb_prolog.sh is 195 lines, and has the following vars and functions: 

from class:base-distrib:
  ov_get_firmware()
  ov_rootfs_kill()
  ov_rootfs_drop_caches()
  ov_rootfs_oom()
  ov_rootfs_sync()
  ov_rootfs_reboot()
  ov_rootfs_state()
  ov_logger()
  ov_rootfs_logread()

from class:base-board:
 LTP_OPEN_POSIX_SUBTEST_COUNT_POS
 MMC_DEV
 SRV_IP
 SATA_DEV
 ...
 JTA_HOME
 IPADDR
 PLATFORM=""
 LOGIN
 PASSWORD
 TRANSPORT
 ov_transport_cmd()
 ov_transport_put()
 ov_transport_get()

from class:base-params:
 DEVICE
 PATH
 SSH
 SCP

from class:base-funcs:
 default_target_route_setup()

from testplan:default:
 BENCHMARK_DHRYSTONE_LOOPS
 BENCHMARK_<TESTNAME>_<VARNAME>
 ...
 FUNCTIONAL_<TESTNAME>_<VARNAME>

Logs [edit section]

When a test is executed, several different kinds of logs are generated: devlog, systemlogs, the testlogs, and the console log.

created by Jenkins [edit section]

created by ftc [edit section]

created by the test script [edit section]

Core scripts [edit section]

The main scripts that are sourced by a test script are:

These have the same pattern of operations in them:

functions available to test scripts: See Test Script APIs

notes about benchmark tests [edit section]

Benchmark tests must provide a reference.log file. This has a special syntax recognized by parser.py and the common.py library.

It consists of a series of lines, showing a name, test and value The syntax is: [<var-name>|<test>] <value>

An example is: [dhrystones|ge] 1

This says: the test is OK, if the value of 'dhrystones' is greater than or equal to 1.

(NOTE: this is an extremely weird syntax)

A reference.log file can have multiple tests.

Benchmark tests must provide a parser.py file, which extracts the (single value)? from the log data.

It does this by doing the following: import common as plib f = open(plib.TEST_LOG) lines = f.readlines() ((parse the data)) create a dictionary with a key and value, where the key matches the string in the reference.log file

The parser.py program builds a dictionary of values by parsing the log from the test (basically the test output). It then sends the dictionary, and the pattern for matching the reference log test criteria to the routine: common.py:process_data()

It defines ref_section_pat, and passes that to process_data() Here are the different patterns for ref_section_pat: 

      9  "\[[\w]+.[gle]{2}\]"
      1  "\[\w*.[gle]{2}\]"
      1  "^\[[\d\w_ .]+.[gle]{2}\]"
      1  "^\[[\d\w_.-]+.[gle]{2}\]"
      1  "^\[[\w\d&._/()]+.[gle]{2}\]"
      4  "^\[[\w\d._]+.[gle]{2}\]"
      2  "^\[[\w\d\s_.-]+.[gle]{2}\]"
      3  "^\[[\w\d_ ./]+.[gle]{2}\]"
      5  "^\[[\w\d_ .]+.[gle]{2}\]"
      1  "^\[[\w\d_\- .]+.[gle]{2}\]"
      1  "^\[[\w]+.[gle]{2}\]"
      1  "^\[[\w_ .]+.[gle]{2}\]"
Why are so many different ones needed?? Why couldn't the syntax be: <var-name> <test> <value> on one line?

It turns out this is processed by an 'awk' script. thus the weird syntax. We should get rid of the awk script and use python instead.

How is benchmarking graphing done? [edit section]

See Benchmark parser notes

docker tips [edit section]

See Docker Tips