Release 1.2 testing - 2017-08-24

Contents:

Tests conducted
Issues found

issues table
Functional.glib problem details
Functional.LTP problem details

run.json: no posix results
run.json: no realtime results in file
run.json: incorrect status for test_set and test_suite

Things to check for
Ideas for fixes
issues fixed

Here is some information about testing started on August 24, 2017 for the 1.2 release:

.(make room for TOC on right)

Tests conducted [edit section]

min1.testplan_ltsi

  Benchmark.Dhrystone         2017-08-24_18:09:14    timdesk:min1                   PASS
  Benchmark.IOzone            2017-08-24_19:20:19    timdesk:min1                   PASS
  Benchmark.Java              2017-08-24_18:10:37    timdesk:min1                   UNFINISHED
  Benchmark.Whetstone         2017-08-24_18:07:07    timdesk:min1                   PASS
  Benchmark.bonnie            2017-08-24_19:20:50    timdesk:min1                   UNFINISHED
  Benchmark.dbench            2017-08-24_19:21:09    timdesk:min1                   PASS
  Benchmark.ffsb              2017-08-24_18:07:59    timdesk:min1                   PASS
  Benchmark.fio               2017-08-24_18:10:49    timdesk:min1                   PASS
  Benchmark.hackbench         2017-08-24_18:07:39    timdesk:min1                   PASS
  Benchmark.linpack           2017-08-24_18:16:56    timdesk:min1                   PASS
  Benchmark.netperf           2017-08-24_18:31:37    timdesk:min1                   PASS
  Benchmark.tiobench          2017-08-24_18:05:06    timdesk:min1                   PASS
  Functional.LTP              2017-08-24_18:32:37    timdesk:min1                   PASS
  Functional.aiostress        2017-08-24_18:17:35    timdesk:min1                   UNFINISHED
  Functional.bc               2017-08-24_18:10:14    timdesk:min1                   PASS
  Functional.bc               2017-08-24_18:17:54    timdesk:min1                   PASS
  Functional.bzip2            2017-08-24_18:18:16    timdesk:min1                   PASS
  Functional.crashme          2017-08-24_18:05:52    timdesk:min1                   PASS
  Functional.expat            2017-08-24_18:18:35    timdesk:min1                   UNFINISHED
  Functional.ft2demos         2017-08-24_18:19:05    timdesk:min1                   PASS
  Functional.glib             2017-08-24_18:20:11    timdesk:min1                   UNFINISHED
  Functional.hello_world      2017-08-24_18:14:26    timdesk:bbb                    PASS
  Functional.hello_world      2017-08-24_18:04:41    timdesk:min1                   PASS
  Functional.hello_world      2017-08-24_18:06:49    timdesk:min1                   PASS
  Functional.ipv6connect      2017-08-24_18:08:53    timdesk:min1                   PASS
  Functional.jpeg             2017-08-24_18:06:22    timdesk:min1                   PASS
  Functional.kernel_build     2017-08-24_18:22:57    timdesk:min1                   PASS
  Functional.linus_stress     2017-08-24_18:27:33    timdesk:min1                   PASS
  Functional.netperf          2017-08-24_19:18:06    timdesk:min1                   PASS
  Functional.rmaptest         2017-08-24_18:28:33    timdesk:min1                   PASS
  Functional.stress           2017-08-24_18:29:05    timdesk:min1                   PASS
  Functional.synctest         2017-08-24_18:30:31    timdesk:min1                   PASS
  Functional.zlib             2017-08-24_18:31:15    timdesk:min1                   PASS

Issues found [edit section]

testplan_ltsi
- uses noroot spec for bonnie - is probably wrong
tests
- min1.default.Benchmark.Java - aborted
  - error message in console log is not human friendly
    - error is an abort from assert_define, from fuego_test.sh:test_pre_check()
    - this is not using need_check
  - error is not put into run.json file
- min1.default.Functional.aiostress - failed
  - test execution error
  - error loading shared libraries: libaio.so.1
- min1.default.Functional.expat - failed
  - build failure
  - directory (empty) was left on target
  - Warning: deprecated conversion from string constant to 'char*' [-Wwrite-strings]
  - youtube says this can be fixed with a line like this:
    - #pragma GCC diagnostic ignored "-Wwrite-strings"
- min1.default.Functional.glib - failed
  - runtime error with glib test
  - execution of bash script returns exit code 143
  - FUEGO has a bug handling the return code:
    - TRB: RETURN_VALUE=
    - generic_parser.py, line 19 invalid literal for int() with base 10:
      - value = "PASS" if int(RETURN_VAULE) == 0 else "FAIL"
    - result of "report" is not being handed to generic_parser.py
- min1.default.Functional.kernel_build - failed
  - fuego_test.sh uses 'report "cat build.log"', but the build.log is generated on the host, not the target. You can't 'report' that.
  - need to add build.log to target-side test log, if that's what's wanted
  - it's not using the right kernel config
- min1.docker.Functional.LTP - takes a very long time (45 minutes)
  - build duration was 584 seconds (~10 minutes)
  - subtest fork13 takes 337 seconds (~5 minutes)
  - subtest getrandom02 takes 296 seconds (~4 minutes)
- min1.noroot.Benchmark.bonnie failed
  - Fuego uses root account on min1
    - should make bonnie recognize situation and run correctly
- min1.default.Functional.LTP
- min1.selectionwithrt.Functional.LTP
  - run.json (4.4) does not have posix results
  - run.json (4.4) does not have realtime results
  - status of 'syscalls' testset is PASS, even though there are failures in individual testcases (e.g. sysctl01)
- min1.default.Benchmark.IOZone
  - there's no run.json link in the Jenkins interface
  - there are lots of HTML tables with SKIP in the measure area (not just the status area)
  - The test doesn't actually perform anything but:
    - 2048_Kb_Read_Write.ReWrite
    - 2048_Kb_Read_Write.Write
  - that's odd, are the parameters to iozone correct for the default spec?

Jenkins interface
- results on Jenkins job page:
  - HTML table
    - test_plan is always "none"
    - start_time is in seconds since the epoch (not human friendly)
    - duration is in milliseconds (not human-friendly)
    - measures are reported as PASS instead of valid value
      - see Dhrystone Score, or linpack.linpack
      - ftc gen-report gets this right
    - Functional.LTP has too much data!!
  - plots for benchmarks
    - are completely missing
ftc
- gen-report
  - columns are messy (columns don't auto-adjust to long field entries)
  - ftc gen-report --where test=.*LTP
    - tguid:status for syscalls are all None (grep fork13)
      - run.json has correct status (PASS)
general
- failures are very difficult to diagnose
  - some tests (like LTP) hid the test output
  - interpreting the console log is very difficult

issues table [edit section]

Test	problem	notes
Benchmark.Java	abort not recorded	should put abort message in run.json file
Functional.aiostress	missing libaio.so.1	is a toolchain/target mismatch
Functional.expat	build error - warning about const char conversion	Looks like maybe a warning that turned into an error?
Functional.glib	runtime error - gtester returns failure, and not the expected output	.
Functional.kernel_build	error in fuego_test.sh - build.log is not on the target	Need to change build.log
Functional.LTP	slow, result are too big for Jenkins page	.
Functional.LTP	run.json test_set status is wrong	Should be fail, with missing criteria.json file
Functional.LTP	run.json missing posix results	.
Functional.LTP	run.json missing realtime results	.
Functional.LTP	gen-report shows wrong values for tguids (None instead of PASS/FAIL)	.
Benchmark.bonnie	noroot spec didn't work	switched to default spec, should handle root or not in test
Benchmark.Dhrystone	Dhrystone.Score has PASS in Jenkins HTML table	.
Benchmark.IOzone	only has values for 2 measures	.

Functional.glib problem details [edit section]

on min1 and docker, gtester returns with exit code 142, which causes Fuego to freak out
- fuego_test.sh test_run calls "report"
  - it returns 143, which is an error
  - the signal handler is called
- fuego_test.sh test_processing is empty (normally it should have a log_compare)
  - the log_compares in this test don't match the test output
  - has this test every worked???
- post_test calls:
  - processing, which calls
    - test_processing(), which just returns 'true'
  - then it calls parser.py or generic_parser.py
    - if there were no errors, then RETURN_VALUE is undefined
on bbb, the build fails, trying to compile gatomic.c
- The assembler says:
  - Error: selected processor does not support Thumb mode 'swp r3,r5,[r4]'

Functional.LTP problem details [edit section]

run.json: no posix results [edit section]

fuego-core/engine/tests/Functional.LTP/parser.py does not even parse posix test results.

Should look at JTA or Fuego 1.0 and see what Functional.LTP-posix did. Maybe just logcompare?

run.json: no realtime results in file [edit section]

run.json: incorrect status for test_set and test_suite [edit section]

run.json has overall status of PASS and test_set syscalls has status of PASS, when some testcases under syscalls failed (and there's no criteria.json file - the default should be max_fail=0)

Output from min1.selectionwithrt.Functional.LTP consolelog.txt:

Need to not show parsed_results - it's too big with LTP.

Warning: missing or faulty criteria.json file, looking for reference.log
Applying criterion {'max_fail': 0, 'tguid': 'syscalls'}
    ^^^^^ this should result in FAIL ^^^^^
Applying criterion {'max_fail': 0, 'tguid': 'mm'}
Applying criterion {'max_fail': 0, 'tguid': 'timers'}
Applying criterion {'max_fail': 0, 'tguid': 'fs'}
Applying criterion {'max_fail': 0, 'tguid': 'pty'}
Applying criterion {'max_fail': 0, 'tguid': 'pipes'}
Applying criterion {'max_fail': 0, 'tguid': 'sched'}
Applying criterion {'max_fail': 0, 'tguid': 'dio'}
Applying criterion {'max_fail': 0, 'tguid': 'ipc'}
Applying criterion {'max_fail': 0, 'tguid': 'Functional.LTP'}
Writing run data to /fuego-rw/logs/Functional.LTP/min1.selectionwithrt.4.4/run.json
reference.json not available

Things to check for [edit section]

failures are easy to diagnose - no, it requires a ton of Fuego knowledge
I can skip LTP tests that cause problems
I can save results to establish new baseline for benchmark thresholds
- save as criteria file?

Ideas for fixes [edit section]

Benchmark.Java should put abort message into run.json file
for ftc gen-report column sloppiness:
- do two passes over data to get field lengths for column widths
- need to collect all data first, and run over it twice
Benchmark.bonnie should autodetect if running as root, and use correct arguments to bonnie++ program
should add "duration_hint" to spec.json file, so each spec can specify a hint
- idea is that this number could be multiplied by a board variable (timeout_factor?) to yield an approximate timeout for the test with this spec. The number in the testplan would override this guess, but this would be better than nothing.

issues fixed [edit section]

ftc list-runs does not process '!=' where operator
ftc list-runs help does not document '!=' where operator
ftc gen-report help does not document '!=' where operator
pass smaller number to fork13 command line for LTP time reduction