Release 1.5 testing - 2019-08-13
See an overview issues table at Release test results
to do [edit section]
- categorize all bugs
- likely location: core, test, board, toolchain, lab, unknown
- severity: high, medium, low
- time to fix: short, long
- priority: 5=high, 1=low, 0=defer
Resolution key:
- CANDIDATE - is a candidate for fixing this release
- RESEARCH - needs more research to categorize and prioritize
- TRANSIENT - problem went away with subsequent test and no change in Fuego
- LOCAL (lab,board) - problem was with setup, config, board status, board, or toolchain, and issue can be ignored
- IGNORE - ignore (not a problem)
- FIXED - problem fixed with change in Fuego
- DEFER - problem exists, but will not be fixed in this release
uncategorized issues list [edit section]
deferred from 1.4 release [edit section]
See http://fuegotest.org/wiki/Release_1.5_testing_-_2019-05-02#deferred_from_1.4_release
new errors [edit section]
- xx. (distro) docker.default.Functional.fuego_board_check - missing /sbin/reboot
- /sbin/reboot is present on host as a symlink to /bin/systemctl
- this is expected
- xx. Can't easily navigate to LTP individual test logs in Jenkins interface
old errors [edit section]
- Benchmark.GLMark - build failure on multiple platforms (missing glew.h, missing libSDL)
Errors by board [edit section]
bbb board [edit section]
- (old) bbb.default.Benchmark.GLMark - missing GL/glew.h
- (old) bbb.default.Benchmark.Interbench - run error: hangs in Audio test
- (old) bbb.default.Benchmark.gtkperf - build error: configure error: checking for gtk+-2.0 gdk-2.0
- (old) bbb.default.Benchmark.X11perf - missing X libs
- (RESEARCH) (old) bbb.default.Functional.glib - missing gettext lib
# ftc list-runs --where "start_time>4 days ago,status!=PASS,board=bbb"
Runs on this host: Test Status --------------------------- - ------ Benchmark.Dhrystone ERROR - fixed Benchmark.GLMark SKIP - build error: GL/glew.h: No such file or directory Benchmark.Interbench ERROR - run error: hangs in Audio test Benchmark.Java SKIP - no java: OK Benchmark.gtkperf SKIP - build error: configure error: checking for gtk+-2.0 gdk-2.0 ... Benchmark.netpipe FAIL - numbers below thresholds (size-262144.throughput = 37.54 which is not "gt 400") Benchmark.x11perf SKIP - missing X libs Functional.LTP FAIL - multiple failures (hangs) Functional.fontconfig SKIP - pre_check error: missing 'fc-list': OK Functional.ft2demos SKIP - build error: cannot find Xau, Xdmcp libs Functional.fuego_abort SKIP - run error: aborted = OK Functional.fuego_compare_reports FAIL - not configured (missing baseline results file): OK Functional.fuego_ftc_test FAIL - (not relevant to this board) Functional.fuego_lint FAIL - (not relevant to this board) Functional.fuego_release_test FAIL - (not relevant to this board) Functional.fuego_slow_log FAIL - (not relevant to this board) Functional.fuego_test_variables FAIL - (not relevant to this board) Functional.glib SKIP - build error: missing gettext, configure error. Functional.hello_world FAIL - expected (spec=fail)
min1 board [edit section]
Here are the tests that had issues:# ftc list-runs --where board=min1,status!=PASS Runs on this host: Test Status --------------------------- ------ Benchmark.GLMark SKIP - build errror, missing GL/glew.h (old) Benchmark.cyclictest FAIL - non-rt kernel Benchmark.dbench4 FAIL - (new) - failed to connect to board after reboot Benchmark.gtkperf SKIP - build error: configure error: checking for gtk+-2.0 gdk-2.0 (old) Benchmark.signaltest FAIL - non-rt kernel Benchmark.x11perf SKIP - runtime: unable to open display ':0 Functional.LTP FAIL - see below Functional.glib SKIP - build error: missing gettext support (RESEARCH)
Here are the LTP non-PASS results:
# ftc gen-report --where board=min1,test=LTP,tguid:result!=PASS
--- Here is the reason for each LTP failure (with some comments)
------------------------------------ tguid tguid:result ------------------------------------ syscalls.fanotify06 FAIL - segfault syscalls.msgctl11 FAIL - queue mismatch syscalls.open10 FAIL - setgid not set syscalls.creat08 FAIL - setgid not set sem_init.sem_init_7-1 SKIP - ??? (RESEARCH) ------------------------------------
ren1 board [edit section]
# ftc list-runs --where board=ren1,status!=PASS
Runs on this host: Test Status --------------------------- ------ Benchmark.Dhrystone FAIL - time too small (expected - but could fix test) Benchmark.GLMark SKIP - missing SDL/SDL.h (old) Benchmark.Java SKIP - pre_test: missing java (expected, ok) Benchmark.gtkperf SKIP - gdk-pixbuf-query-loaders: command not found Benchmark.reboot SKIP - runtime error: didn't capture reboot (RESEARCH) Benchmark.signaltest FAIL - non-RT kernel Benchmark.x11perf SKIP - build error: no package xmuu Functional.LTP FAIL - see below Functional.aiostress SKIP - pre_test: no /lib/*/libaio (ok?) Functional.crashme FAIL - 0 .. 507, but expected 3000 Functional.ft2demos SKIP - build error: missing /usr/lib/libxcb.la (RESEARCH) Functional.glib SKIP - missing gettext support (CANDIDATE) Functional.kernel_build FAIL - missing "Kernel: arch/.* is ready". bzImage, no such file or directory
Here is the LTP report for non-PASS tests:
# ftc gen-report --where board=ren1,test=LTP,tguid:result!=PASS
======================================================================
**** Fuego Test Report ****
test : LTP
board : ren1
kernel : 4.9.0-yocto-standard
timestamp : 2019-08-14_06:38:07
report_date : 2019-08-14_20:34:55
======================================================================
----------------------------------------------------------------------------------------
test_name spec board timestamp tguid tguid:result
----------------------------------------------------------------------------------------
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.cve-2017-6951 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.prot_hsymlinks FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.renameat202 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.add_key04 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.add_key02 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.keyctl08 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.keyctl03 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.keyctl02 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.keyctl05 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.keyctl04 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.keyctl07 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.keyctl06 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.msgctl09 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.msgctl08 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.fanotify06 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.utime02 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.mknod03 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.abort01 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.kill11 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.msgctl10 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.msgctl11 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.request_key02 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.open10 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.renameat201 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.creat08 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.add_key03 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.setsockopt02 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.cve-2017-5669 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.sendmsg01 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.request_key01 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.request_key03 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.setpriority01 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.utime01 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 sem_init.sem_init_7-1 SKIP
----------------------------------------------------------------------------------------
Here are LTP details:
To get sorted testcase names:
- $ ftc gen-report --where board=ren1,test=LTP,tguid:result!=PASS --fields tguid,tguid:result | sort
-------------------------------------- tguid tguid:result -------------------------------------- sem_init.sem_init_7-1 SKIP syscalls.abort01 FAIL - invalid argument syscalls.add_key02 FAIL - EINVAL syscalls.add_key03 FAIL - create keyring for another user syscalls.add_key04 FAIL - SIGSEGV syscalls.creat08 FAIL - setgid should be set syscalls.cve-2017-5669 FAIL - mapped VM addr in first 64Kb syscalls.cve-2017-6951 FAIL - timeout requesting dead key syscalls.fanotify06 FAIL - SIGSEGV syscalls.keyctl02 FAIL - timeout syscalls.keyctl03 FAIL - timeout syscalls.keyctl04 FAIL - timeout syscalls.keyctl05 FAIL - timeout syscalls.keyctl06 FAIL - timeout syscalls.keyctl07 FAIL - timeout syscalls.keyctl08 FAIL - timeout syscalls.kill11 FAIL - invalid argument to setrlimit syscalls.mknod03 FAIL - setgid bit not set syscalls.msgctl08 FAIL - ipcs not found syscalls.msgctl09 FAIL - ipcs not found syscalls.msgctl10 FAIL - ipcs not found syscalls.msgctl11 FAIL - ipcs not found syscalls.open10 FAIL - setgid bit not set syscalls.prot_hsymlinks FAIL - failed to run cmd: useradd hsym syscalls.renameat201 FAIL - Invalid argument syscalls.renameat202 FAIL - Invalid argument syscalls.request_key01 FAIL - timeout syscalls.request_key02 FAIL - timeout syscalls.request_key03 FAIL - timeout syscalls.sendmsg01 FAIL - ip/ifconfig not found syscalls.setpriority01 FAIL - useradd exited with non-zero code syscalls.setsockopt02 FAIL - allowed bad tp_sizeof_priv value syscalls.utime01 FAIL - tmp_file access and mod times not set syscalls.utime02 FAIL - tmp_file access and mod times not set --------------------------------------
fuego-test board [edit section]
- (CANDIDATE) fuego-test.default.Functional.fuego_abort - reports SKIP, but should be OK (behavior is correct)
- should turn this test into a nested test, that checks for correct messages in the console log
- (CANDIDATE) fuego-test.default.fuego_board_check - missing /sbin/reboot in container
- fuego-test.default.Functional.fuego_ftc_test - testcase 6 (ftc query-board differences)
- fuego-test.default.Functional.fuego_lint - expected failures
- fuego-test.default.Functional.fuego_release_test -
- fuego-test.default.Functional.fuego_slow_log -
# ftc list-runs --where board=fuego-test,status!=PASS
Runs on this host: Test Status --------------------------- ------ Functional.fuego_abort SKIP - OK (CANDIDATE) Functional.fuego_board_check FAIL - missing /sbin/reboot Functional.fuego_compare_reports FAIL - missing baseline Functional.fuego_ftc_test FAIL - (FIXED) testcase 3 - ftc help, testcase 6 - quoting of variables Functional.fuego_lint FAIL - 559 errors in ftc (ok for now) Functional.fuego_release_test FAIL - (CANDIDATE) password is required Functional.fuego_slow_log FAIL - (FIXED) syntax error near unexpected token done