Release 1.5 testing - 2019-05-02
See an overview issues table at Release test results
to do [edit section]
- review bugs labeled as FIXED in Release 1.4 testing - 2018-12-26
- categorize all bugs
- likely location: core, test, board, toolchain, lab, unknown
- severity: high, medium, low
- time to fix: short, long
- priority: 5=high, 1=low, 0=defer
Resolution key:
- CANDIDATE - is a candidate for fixing this release
- RESEARCH - needs more research to categorize and prioritize
- TRANSIENT - problem went away with subsequent test and no change in Fuego
- LOCAL (lab,board) - problem was with setup, config, board status, board, or toolchain, and issue can be ignored
- IGNORE - ignore (not a problem)
- FIXED - problem fixed with change in Fuego
- DEFER - problem exists, but will not be fixed in this release
uncategorized issues list [edit section]
deferred from 1.4 release [edit section]
- 1. (test) LTP runs for too long with nothing showing on the console. Jenkins should show the current testcase being run, or something.
- would require output from fuego_test.sh
- 4. (core) Jenkins doesn't show different color for ERROR (timeout or abort)
- 6. (core) bbb.default.Benchmark.cyclictest it's not easy to find the threshold value. It's not in run.json. It's on the chart, but with multiple boards it's hard to isolate and find
- could add checkboxes for series.
- could add reference value to run.json
- could copy reference.log or reference.json to log directory
- could have parser add reference check failure to console log
- 7. (core) ftc rm-jobs rpi3-1.*.*.* - can't do rm-jobs outside the container! (annoying, but whatever)
- required due to possible interactive confirmation
- should be able to rm-jobs without being inside the container
- should be able to do interactive commands even when forwarding to the container ftc
- 9. RESEARCH (test) min1.default.Functional.LTP - new failure for open10
- 10. RESEARCH (test) min1.default.Functional.LTP - new failure for creat08
- 11. (core?) need to have test indicate whether error code indicates error or failure
- example: x11perf - non-zero = error (not failure)
- 16. CANDIDATE, HIGH (core) fuego-test.default.Functional.fuego_release_test
- problem with Python 3 dependencies in docker project (requests requires urllib3, which is missing dependencies?)
... = ==== doing fuego phase: run ===== ------------------------------------------------- /usr/local/lib/python3.4/dist-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.9.1) or chardet (3.0.4) doesn't match a supported version! RequestsDependencyWarning) Traceback (most recent call last): File "/fuego-core/engine/tests/Functional.fuego_release_test/test_run.py", line 15, in <module> import docker File "/usr/local/lib/python3.4/dist-packages/docker/__init__.py", line 2, in <module> from .api import APIClient File "/usr/local/lib/python3.4/dist-packages/docker/api/__init__.py", line 2, in <module> from .client import APIClient File "/usr/local/lib/python3.4/dist-packages/docker/api/client.py", line 5, in <module> import requests File "/usr/local/lib/python3.4/dist-packages/requests/__init__.py", line 105, in <module> from urllib3.exceptions import DependencyWarning ImportError: cannot import name 'DependencyWarning'
- 17. (board) rpi3-2.default.Functional.fuego_dependencies - - can't find kconfig
- missing /proc/config.gz or /boot/config-4.9.41-v7+ (but has /lib/modules.../configs.ko)
- 18. (test) rpi3-2.default.Functional.fuego_sleep - timed out
- timeout from testplan_fuego_tests is too short
- 19. (core) test that fails pre-check is shown as 'not run' in Jenkins
- 20. (core) test that fails pre-check has no result code (listed as None) in 'ftc gen-report'
- 22. (test) ren1.default.Functional.LTP_one_test.syscalls.add_key_02 - timeout is not converted to Fuego ERROR
- the LTP parser should convert a timed-out test in LTP into a Fuego ERROR in the run.json file. It doesn't do this.
new errors [edit section]
- xx. batch testcase names end in underscore in underscore
- log_this should not put trailing space in hostlog.txt
- or, have parser trim trailing whitespace (regex?) or python code?
- xx. (core) docker.default.Functional.batch_nested - nested batch test has invalid job page link in hostlog.txt
- shown as "Info:", when it should be '2' (see run 1)
Running test Functional.batch_hello Log dir: logs/Functional.batch_hello/docker.Info:.Info:.Info: Job page: http://localhost:8090/fuego/job/docker.Info:.Functional.batch_hello/Info:/ [[nested-4]] ok 2 - Functional.batch_hello
- shown as "Info:", when it should be '2' (see run 1)
- xx. (distro) docker.default.Functional.fuego_board_check - missing /sbin/reboot
- /sbin/reboot is present on host as a symlink to /bin/systemctl
- xx. ren1.default.Functional.hello_world (run 3) - problem importing filelock
- python traceback in prepare_chart_data.py trying to import module 'filelock'
- problem is in poky-aarch64-tools.sh - re-declaration of ORIG_PATH
- xx. ftc query-board -b fuego_test - python error
root@timdesk:/fuego-rw# ftc query-board -b ftc-test
Information for board: ftc-test
ARCHITECTURE : "arm" EXPAT_SUBTEST_COUNT_NEG : "41" EXPAT_SUBTEST_COUNT_POS : "1769" FUEGO_HOME : "/home/a" Traceback (most recent call last): File "/usr/local/bin/ftc", line 5275, in <module> main() File "/usr/local/bin/ftc", line 5260, in main do_query_board(bvars, options) File "/usr/local/bin/ftc", line 4203, in do_query_board if value.find('\n') == -1: AttributeError: 'list' object has no attribute 'find'
[edit section]
log_compare: pattern 'SUCCESS' found 1 times (expected greater or equal than 1) Traceback (most recent call last): File "/fuego-core/scripts/generic_parser.py", line 30, in <module> sys.exit(plib.process(results)) File "/fuego-core/scripts/parser/common.py", line 701, in process from prepare_chart_data import store_flat_results, make_chart_data File "/fuego-core/scripts/parser/prepare_chart_data.py", line 36, in <module> from filelock import FileLock ImportError: No module named filelock- need to update all -tools.sh files in ~/work/fuego-reference
- need to make sure following are reviewed:
- ./aarch64-tools.sh:ORIG_PATH=$PATH
- ./debian-armhf-tools.sh:ORIG_PATH=$PATH
- ./poky-qemuarm-tools.sh:ORIG_PATH=$PATH
- ./lager-tools.sh:ORIG_PATH=$PATH
- ./x86_64-tools.sh:ORIG_PATH=$PATH
- ./debian-armel-tools.sh:ORIG_PATH=$PATH
old errors [edit section]
- Benchmark.GLMark - build failure on multiple platforms (missing glew.h, missing libSDL)
Errors by board [edit section]
# ftc list-runs --where "start_time>4 days ago,status!=PASS,board=bbb"
bbb board [edit section]
- (old) bbb.default.Benchmark.GLMark - missing GL/glew.h
- (old) bbb.default.Benchmark.Interbench - run error: hangs in Audio test
- bbb.default.Benchmark.gtkperf - build error: configure error: checking for gtk+-2.0 gdk-2.0
- bbb.default.Benchmark.X11perf - missing X libs
- (RESEARCH) bbb.default.Functional.glib - missing gettext lib
Runs on this host: Test Status --------------------------- - ------ Benchmark.Dhrystone ERROR - fixed Benchmark.GLMark SKIP - build error: GL/glew.h: No such file or directory Benchmark.Interbench ERROR - run error: hangs in Audio test Benchmark.Java SKIP - no java: OK Benchmark.gtkperf SKIP - build error: configure error: checking for gtk+-2.0 gdk-2.0 ... Benchmark.netpipe SKIP - missing NPtcp in host container, missing netpipe-tcp in container Benchmark.x11perf SKIP - missing X libs Functional.LTP FAIL - multiple failures Functional.bc ERROR Functional.fontconfig SKIP Functional.ft2demos SKIP Functional.fuego_abort SKIP Functional.fuego_compare_reports FAIL - not configured Functional.fuego_ftc_test FAIL Functional.fuego_lint FAIL Functional.fuego_release_test FAIL Functional.fuego_slow_log FAIL Functional.fuego_test_variables FAIL - expected Functional.glib SKIP Functional.hello_world FAIL - expected (spec=fail)
- CANDIDATE bbb.default.Benchmark.netpipe' - run error: missing NPtcp in docker container
- /usr/bin/NPtcp is provided by Debian package netpipe-tcp
- + add netpipe-tcp to Dockerfile
min1 board [edit section]
Here are the tests that had issues:# ftc list-runs --where board=min1,status!=PASS Runs on this host: Test Status --------------------------- ------ Benchmark.GLMark SKIP - build errror, missing GL/glew.h (old) Benchmark.cyclictest FAIL - non-rt kernel Benchmark.ffsb SKIP - current utilization = 1844660474359607984128.00% (RESEARCH) Benchmark.gtkperf SKIP - build error: configure error: checking for gtk+-2.0 gdk-2.0 (old) Benchmark.signaltest FAIL - non-rt kernel Benchmark.x11perf SKIP - runtime: unable to open display ':0 Functional.LTP FAIL - see below Functional.glib SKIP - build error: missing gettext support (RESEARCH)
Here are the LTP non-PASS results:
# ftc gen-report --where board=min1,test=LTP,tguid:result!=PASS ====================================================================== **** Fuego Test Report **** test : LTP board : min1 kernel : 4.15.0-33-generic timestamp : 2019-05-03_12:57:07 report_date : 2019-05-08_00:30:28 ====================================================================== -------------------------------------------------------------------------------------- test_name spec board timestamp tguid tguid:result -------------------------------------------------------------------------------------- Functional.LTP default min1 2019-05-03_12:57:07 syscalls FAIL Functional.LTP default min1 2019-05-03_12:57:07 syscalls.fanotify06 FAIL Functional.LTP default min1 2019-05-03_12:57:07 syscalls.msgctl11 FAIL Functional.LTP default min1 2019-05-03_12:57:07 syscalls.open10 FAIL Functional.LTP default min1 2019-05-03_12:57:07 syscalls.creat08 FAIL Functional.LTP default min1 2019-05-03_12:57:07 sem_init.sem_init_7-1 SKIP --------------------------------------------------------------------------------------
--- Here is the reason for each LTP failure (with some comments)
------------------------------------ tguid tguid:result ------------------------------------ syscalls.fanotify06 FAIL - segfault syscalls.msgctl11 FAIL - queue mismatch syscalls.open10 FAIL - setgid not set syscalls.creat08 FAIL - setgid not set sem_init.sem_init_7-1 SKIP - ??? (RESEARCH) ------------------------------------
ren1 board [edit section]
Runs on this host: Test Status --------------------------- ------ Benchmark.Dhrystone FAIL - time too small (expected - but could fix test) Benchmark.GLMark SKIP - missing SDL/SDL.h (old) Benchmark.Java SKIP - pre_test: missing java (expected, ok) Benchmark.gtkperf SKIP - gdk-pixbuf-query-loaders: command not found Benchmark.reboot SKIP - runtime error: didn't capture reboot (RESEARCH) Benchmark.signaltest FAIL - non-RT kernel Benchmark.x11perf SKIP - build error: no package xmuu Functional.LTP FAIL - see below Functional.aiostress SKIP - pre_test: no /lib/*/libaio (ok?) Functional.crashme FAIL - 0 .. 507, but expected 3000 Functional.ft2demos SKIP - build error: missing /usr/lib/libxcb.la (RESEARCH) Functional.glib SKIP - missing gettext support (CANDIDATE) Functional.kernel_build FAIL - missing "Kernel: arch/.* is ready". bzImage, no such file or directory
Here is the LTP report for non-PASS tests:
====================================================================== **** Fuego Test Report **** test : LTP board : ren1 kernel : 4.9.0-yocto-standard timestamp : 2019-05-03_13:41:21 report_date : 2019-05-08_00:39:12 ====================================================================== ---------------------------------------------------------------------------------------- test_name spec board timestamp tguid tguid:result ---------------------------------------------------------------------------------------- Functional.LTP default ren1 2019-05-03_13:41:21 syscalls FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.cve-2017-6951 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.prot_hsymlinks FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.renameat202 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.add_key04 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.add_key02 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.keyctl08 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.keyctl03 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.keyctl02 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.keyctl05 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.keyctl04 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.keyctl07 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.keyctl06 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.msgctl09 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.msgctl08 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.fanotify06 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.utime02 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.mknod03 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.abort01 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.kill11 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.msgctl10 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.msgctl11 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.request_key02 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.open10 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.renameat201 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.creat08 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.add_key03 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.setsockopt02 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.cve-2017-5669 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.sendmsg01 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.request_key01 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.request_key03 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.setpriority01 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.utime01 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 sem_init.sem_init_7-1 SKIP ----------------------------------------------------------------------------------------
Here are LTP details:
-------------------------------------- tguid tguid:result -------------------------------------- sem_init.sem_init_7-1 SKIP syscalls.abort01 FAIL - invalid argument syscalls.add_key02 FAIL - EINVAL syscalls.add_key03 FAIL - create keyring for another user syscalls.add_key04 FAIL - SIGSEGV syscalls.creat08 FAIL - setgid should be set syscalls.cve-2017-5669 FAIL - mapped VM addr in first 64Kb syscalls.cve-2017-6951 FAIL - timeout requesting dead key syscalls.fanotify06 FAIL - SIGSEGV syscalls.keyctl02 FAIL - timeout syscalls.keyctl03 FAIL - timeout syscalls.keyctl04 FAIL - timeout syscalls.keyctl05 FAIL - timeout syscalls.keyctl06 FAIL - timeout syscalls.keyctl07 FAIL - timeout syscalls.keyctl08 FAIL - timeout syscalls.kill11 FAIL - invalid argument to setrlimit syscalls.mknod03 FAIL - setgid bit not set syscalls.msgctl08 FAIL - ipcs not found syscalls.msgctl09 FAIL - ipcs not found syscalls.msgctl10 FAIL - ipcs not found syscalls.msgctl11 FAIL - ipcs not found syscalls.open10 FAIL - setgid bit not set syscalls.prot_hsymlinks FAIL - failed to run cmd: useradd hsym syscalls.renameat201 FAIL - Invalid argument syscalls.renameat202 FAIL - Invalid argument syscalls.request_key01 FAIL - timeout syscalls.request_key02 FAIL - timeout syscalls.request_key03 FAIL - timeout syscalls.sendmsg01 FAIL - ip/ifconfig not found syscalls.setpriority01 FAIL - useradd exited with non-zero code syscalls.setsockopt02 FAIL - allowed bad tp_sizeof_priv value syscalls.utime01 FAIL - tmp_file access and mod times not set syscalls.utime02 FAIL - tmp_file access and mod times not set
fuego-test board [edit section]
- (CANDIDATE) fuego-test.default.Functional.fuego_abort - reports SKIP, but should be OK (behavior is correct)
- should turn this test into a nested test, that checks for correct messages in the console log
- (CANDIDATE)
fuego-test.default.fuego_board_check - missing /sbin/reboot in container fuego-test.default.Functional.fuego_ftc_test - testcase 6 (ftc query-board differences) fuego-test.default.Functional.fuego_lint - expected failures fuego-test.default.Functional.fuego_release_test - fuego-test.default.Functional.fuego_slow_log -Runs on this host: Test Status --------------------------- ------ Functional.fuego_abort SKIP - OK (CANDIDATE) Functional.fuego_board_check FAIL - missing /sbin/reboot Functional.fuego_ftc_test FAIL - testcase 6, addition of ssh2serial, misc. Functional.fuego_lint FAIL Functional.fuego_release_test FAIL Functional.fuego_slow_log FAIL