Release 1.5 testing - 2019-05-02

Contents:

to do
uncategorized issues list

deferred from 1.4 release
new errors

old errors
Errors by board

bbb board
min1 board
ren1 board
fuego-test board

Here are some notes on testing the 1.4 release (using fuego-1211-container, with a rolling master branch through December 2018 and January 2019):

See an overview issues table at Release test results

to do [edit section]

review bugs labeled as FIXED in Release 1.4 testing - 2018-12-26
categorize all bugs
- likely location: core, test, board, toolchain, lab, unknown
- severity: high, medium, low
- time to fix: short, long
- priority: 5=high, 1=low, 0=defer

Resolution key:

CANDIDATE - is a candidate for fixing this release
RESEARCH - needs more research to categorize and prioritize
TRANSIENT - problem went away with subsequent test and no change in Fuego
LOCAL (lab,board) - problem was with setup, config, board status, board, or toolchain, and issue can be ignored
IGNORE - ignore (not a problem)
FIXED - problem fixed with change in Fuego
DEFER - problem exists, but will not be fixed in this release

uncategorized issues list [edit section]

deferred from 1.4 release [edit section]

1. (test) LTP runs for too long with nothing showing on the console. Jenkins should show the current testcase being run, or something.
- would require output from fuego_test.sh
4. (core) Jenkins doesn't show different color for ERROR (timeout or abort)
6. (core) bbb.default.Benchmark.cyclictest it's not easy to find the threshold value. It's not in run.json. It's on the chart, but with multiple boards it's hard to isolate and find
- could add checkboxes for series.
- could add reference value to run.json
- could copy reference.log or reference.json to log directory
- could have parser add reference check failure to console log
7. (core) ftc rm-jobs rpi3-1.*.*.* - can't do rm-jobs outside the container! (annoying, but whatever)
- required due to possible interactive confirmation
- should be able to rm-jobs without being inside the container
- should be able to do interactive commands even when forwarding to the container ftc
9. RESEARCH (test) min1.default.Functional.LTP - new failure for open10
10. RESEARCH (test) min1.default.Functional.LTP - new failure for creat08
11. (core?) need to have test indicate whether error code indicates error or failure
- example: x11perf - non-zero = error (not failure)
16. CANDIDATE, HIGH (core) fuego-test.default.Functional.fuego_release_test
- problem with Python 3 dependencies in docker project (requests requires urllib3, which is missing dependencies?)

...
 = ==== doing fuego phase: run =====
-------------------------------------------------
/usr/local/lib/python3.4/dist-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.9.1) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)
Traceback (most recent call last):
  File "/fuego-core/engine/tests/Functional.fuego_release_test/test_run.py", line 15, in <module>
    import docker
  File "/usr/local/lib/python3.4/dist-packages/docker/__init__.py", line 2, in <module>
    from .api import APIClient
  File "/usr/local/lib/python3.4/dist-packages/docker/api/__init__.py", line 2, in <module>
    from .client import APIClient
  File "/usr/local/lib/python3.4/dist-packages/docker/api/client.py", line 5, in <module>
    import requests
  File "/usr/local/lib/python3.4/dist-packages/requests/__init__.py", line 105, in <module>
    from urllib3.exceptions import DependencyWarning
ImportError: cannot import name 'DependencyWarning'

17. (board) rpi3-2.default.Functional.fuego_dependencies - - can't find kconfig
- missing /proc/config.gz or /boot/config-4.9.41-v7+ (but has /lib/modules.../configs.ko)
18. (test) rpi3-2.default.Functional.fuego_sleep - timed out
- timeout from testplan_fuego_tests is too short
19. (core) test that fails pre-check is shown as 'not run' in Jenkins
20. (core) test that fails pre-check has no result code (listed as None) in 'ftc gen-report'
22. (test) ren1.default.Functional.LTP_one_test.syscalls.add_key_02 - timeout is not converted to Fuego ERROR
- the LTP parser should convert a timed-out test in LTP into a Fuego ERROR in the run.json file. It doesn't do this.

new errors [edit section]

xx. batch testcase names end in underscore in underscore
- log_this should not put trailing space in hostlog.txt
- or, have parser trim trailing whitespace (regex?) or python code?

xx. (core) docker.default.Functional.batch_nested - nested batch test has invalid job page link in hostlog.txt

shown as "Info:", when it should be '2' (see run 1)

Running test Functional.batch_hello
Log dir: logs/Functional.batch_hello/docker.Info:.Info:.Info:
Job page: http://localhost:8090/fuego/job/docker.Info:.Functional.batch_hello/Info:/
[[nested-4]] ok 2 - Functional.batch_hello

xx. (distro) docker.default.Functional.fuego_board_check - missing /sbin/reboot
- /sbin/reboot is present on host as a symlink to /bin/systemctl
xx. ren1.default.Functional.hello_world (run 3) - problem importing filelock
- python traceback in prepare_chart_data.py trying to import module 'filelock'
- problem is in poky-aarch64-tools.sh - re-declaration of ORIG_PATH
xx. ftc query-board -b fuego_test - python error

root@timdesk:/fuego-rw# ftc query-board -b ftc-test Information for board: ftc-test

ARCHITECTURE : "arm" EXPAT_SUBTEST_COUNT_NEG : "41" EXPAT_SUBTEST_COUNT_POS : "1769" FUEGO_HOME : "/home/a" Traceback (most recent call last): File "/usr/local/bin/ftc", line 5275, in <module> main() File "/usr/local/bin/ftc", line 5260, in main do_query_board(bvars, options) File "/usr/local/bin/ftc", line 4203, in do_query_board if value.find('\n') == -1: AttributeError: 'list' object has no attribute 'find'

[edit section]

log_compare: pattern 'SUCCESS' found 1 times (expected greater or equal than 1) Traceback (most recent call last): File "/fuego-core/scripts/generic_parser.py", line 30, in <module> sys.exit(plib.process(results)) File "/fuego-core/scripts/parser/common.py", line 701, in process from prepare_chart_data import store_flat_results, make_chart_data File "/fuego-core/scripts/parser/prepare_chart_data.py", line 36, in <module> from filelock import FileLock ImportError: No module named filelock

need to update all -tools.sh files in ~/work/fuego-reference
need to make sure following are reviewed:
- ./aarch64-tools.sh:ORIG_PATH=$PATH
- ./debian-armhf-tools.sh:ORIG_PATH=$PATH
- ./poky-qemuarm-tools.sh:ORIG_PATH=$PATH
- ./lager-tools.sh:ORIG_PATH=$PATH
- ./x86_64-tools.sh:ORIG_PATH=$PATH
- ./debian-armel-tools.sh:ORIG_PATH=$PATH

xx. Can't easily navigate to LTP individual test logs in Jenkins interface

old errors [edit section]

Benchmark.GLMark - build failure on multiple platforms (missing glew.h, missing libSDL)

Errors by board [edit section]

# ftc list-runs --where "start_time>4 days ago,status!=PASS,board=bbb"

bbb board [edit section]

(old) bbb.default.Benchmark.GLMark - missing GL/glew.h
(old) bbb.default.Benchmark.Interbench - run error: hangs in Audio test
bbb.default.Benchmark.gtkperf - build error: configure error: checking for gtk+-2.0 gdk-2.0
bbb.default.Benchmark.X11perf - missing X libs
(RESEARCH) bbb.default.Functional.glib - missing gettext lib

Runs on this host:
  Test                           Status
  --------------------------- -  ------
  Benchmark.Dhrystone            ERROR - fixed
  Benchmark.GLMark               SKIP - build error: GL/glew.h: No such file or directory
  Benchmark.Interbench           ERROR - run error: hangs in Audio test
  Benchmark.Java                 SKIP - no java: OK
  Benchmark.gtkperf              SKIP - build error: configure error: checking for gtk+-2.0 gdk-2.0 ...
  Benchmark.netpipe              SKIP - missing NPtcp in host container, missing netpipe-tcp in container
  Benchmark.x11perf              SKIP - missing X libs
  Functional.LTP                 FAIL - multiple failures
  Functional.bc                  ERROR
  Functional.fontconfig          SKIP
  Functional.ft2demos            SKIP
  Functional.fuego_abort         SKIP
  Functional.fuego_compare_reports FAIL - not configured
  Functional.fuego_ftc_test        FAIL
  Functional.fuego_lint            FAIL
  Functional.fuego_release_test    FAIL
  Functional.fuego_slow_log        FAIL
  Functional.fuego_test_variables  FAIL - expected
  Functional.glib                  SKIP
  Functional.hello_world           FAIL - expected (spec=fail)

CANDIDATE bbb.default.Benchmark.netpipe' - run error: missing NPtcp in docker container
/usr/bin/NPtcp is provided by Debian package netpipe-tcp
+ add netpipe-tcp to Dockerfile

min1 board [edit section]
Here are the tests that had issues:
# ftc list-runs --where board=min1,status!=PASS Runs on this host: Test Status --------------------------- ------ Benchmark.GLMark SKIP - build errror, missing GL/glew.h (old) Benchmark.cyclictest FAIL - non-rt kernel Benchmark.ffsb SKIP - current utilization = 1844660474359607984128.00% (RESEARCH) Benchmark.gtkperf SKIP - build error: configure error: checking for gtk+-2.0 gdk-2.0 (old) Benchmark.signaltest FAIL - non-rt kernel Benchmark.x11perf SKIP - runtime: unable to open display ':0 Functional.LTP FAIL - see below Functional.glib SKIP - build error: missing gettext support (RESEARCH)

Here are the LTP non-PASS results:
# ftc gen-report --where board=min1,test=LTP,tguid:result!=PASS ====================================================================== **** Fuego Test Report **** test : LTP board : min1 kernel : 4.15.0-33-generic timestamp : 2019-05-03_12:57:07 report_date : 2019-05-08_00:30:28 ====================================================================== -------------------------------------------------------------------------------------- test_name spec board timestamp tguid tguid:result -------------------------------------------------------------------------------------- Functional.LTP default min1 2019-05-03_12:57:07 syscalls FAIL Functional.LTP default min1 2019-05-03_12:57:07 syscalls.fanotify06 FAIL Functional.LTP default min1 2019-05-03_12:57:07 syscalls.msgctl11 FAIL Functional.LTP default min1 2019-05-03_12:57:07 syscalls.open10 FAIL Functional.LTP default min1 2019-05-03_12:57:07 syscalls.creat08 FAIL Functional.LTP default min1 2019-05-03_12:57:07 sem_init.sem_init_7-1 SKIP --------------------------------------------------------------------------------------

--- Here is the reason for each LTP failure (with some comments)
------------------------------------ tguid tguid:result ------------------------------------ syscalls.fanotify06 FAIL - segfault syscalls.msgctl11 FAIL - queue mismatch syscalls.open10 FAIL - setgid not set syscalls.creat08 FAIL - setgid not set sem_init.sem_init_7-1 SKIP - ??? (RESEARCH) ------------------------------------

ren1 board [edit section]

Runs on this host: Test Status --------------------------- ------ Benchmark.Dhrystone FAIL - time too small (expected - but could fix test) Benchmark.GLMark SKIP - missing SDL/SDL.h (old) Benchmark.Java SKIP - pre_test: missing java (expected, ok) Benchmark.gtkperf SKIP - gdk-pixbuf-query-loaders: command not found Benchmark.reboot SKIP - runtime error: didn't capture reboot (RESEARCH) Benchmark.signaltest FAIL - non-RT kernel Benchmark.x11perf SKIP - build error: no package xmuu Functional.LTP FAIL - see below Functional.aiostress SKIP - pre_test: no /lib/*/libaio (ok?) Functional.crashme FAIL - 0 .. 507, but expected 3000 Functional.ft2demos SKIP - build error: missing /usr/lib/libxcb.la (RESEARCH) Functional.glib SKIP - missing gettext support (CANDIDATE) Functional.kernel_build FAIL - missing "Kernel: arch/.* is ready". bzImage, no such file or directory

Here is the LTP report for non-PASS tests:

====================================================================== **** Fuego Test Report **** test : LTP board : ren1 kernel : 4.9.0-yocto-standard timestamp : 2019-05-03_13:41:21 report_date : 2019-05-08_00:39:12 ====================================================================== ---------------------------------------------------------------------------------------- test_name spec board timestamp tguid tguid:result ---------------------------------------------------------------------------------------- Functional.LTP default ren1 2019-05-03_13:41:21 syscalls FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.cve-2017-6951 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.prot_hsymlinks FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.renameat202 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.add_key04 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.add_key02 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.keyctl08 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.keyctl03 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.keyctl02 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.keyctl05 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.keyctl04 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.keyctl07 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.keyctl06 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.msgctl09 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.msgctl08 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.fanotify06 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.utime02 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.mknod03 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.abort01 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.kill11 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.msgctl10 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.msgctl11 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.request_key02 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.open10 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.renameat201 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.creat08 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.add_key03 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.setsockopt02 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.cve-2017-5669 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.sendmsg01 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.request_key01 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.request_key03 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.setpriority01 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.utime01 FAIL Functional.LTP default ren1 2019-05-03_13:41:21 sem_init.sem_init_7-1 SKIP ----------------------------------------------------------------------------------------

Here are LTP details:
-------------------------------------- tguid tguid:result -------------------------------------- sem_init.sem_init_7-1 SKIP syscalls.abort01 FAIL - invalid argument syscalls.add_key02 FAIL - EINVAL syscalls.add_key03 FAIL - create keyring for another user syscalls.add_key04 FAIL - SIGSEGV syscalls.creat08 FAIL - setgid should be set syscalls.cve-2017-5669 FAIL - mapped VM addr in first 64Kb syscalls.cve-2017-6951 FAIL - timeout requesting dead key syscalls.fanotify06 FAIL - SIGSEGV syscalls.keyctl02 FAIL - timeout syscalls.keyctl03 FAIL - timeout syscalls.keyctl04 FAIL - timeout syscalls.keyctl05 FAIL - timeout syscalls.keyctl06 FAIL - timeout syscalls.keyctl07 FAIL - timeout syscalls.keyctl08 FAIL - timeout syscalls.kill11 FAIL - invalid argument to setrlimit syscalls.mknod03 FAIL - setgid bit not set syscalls.msgctl08 FAIL - ipcs not found syscalls.msgctl09 FAIL - ipcs not found syscalls.msgctl10 FAIL - ipcs not found syscalls.msgctl11 FAIL - ipcs not found syscalls.open10 FAIL - setgid bit not set syscalls.prot_hsymlinks FAIL - failed to run cmd: useradd hsym syscalls.renameat201 FAIL - Invalid argument syscalls.renameat202 FAIL - Invalid argument syscalls.request_key01 FAIL - timeout syscalls.request_key02 FAIL - timeout syscalls.request_key03 FAIL - timeout syscalls.sendmsg01 FAIL - ip/ifconfig not found syscalls.setpriority01 FAIL - useradd exited with non-zero code syscalls.setsockopt02 FAIL - allowed bad tp_sizeof_priv value syscalls.utime01 FAIL - tmp_file access and mod times not set syscalls.utime02 FAIL - tmp_file access and mod times not set

fuego-test board [edit section]

(CANDIDATE) fuego-test.default.Functional.fuego_abort - reports SKIP, but should be OK (behavior is correct)
should turn this test into a nested test, that checks for correct messages in the console log
(CANDIDATE) fuego-test.default.fuego_board_check - missing /sbin/reboot in container
fuego-test.default.Functional.fuego_ftc_test - testcase 6 (ftc query-board differences)
fuego-test.default.Functional.fuego_lint - expected failures
fuego-test.default.Functional.fuego_release_test -

fuego-test.default.Functional.fuego_slow_log -
Runs on this host: Test Status --------------------------- ------ Functional.fuego_abort SKIP - OK (CANDIDATE) Functional.fuego_board_check FAIL - missing /sbin/reboot Functional.fuego_ftc_test FAIL - testcase 6, addition of ssh2serial, misc. Functional.fuego_lint FAIL Functional.fuego_release_test FAIL Functional.fuego_slow_log FAIL