Release 1.5 testing - 2019-05-02 in 'raw' format
{{TableOfContents}}
Here are some notes on testing the 1.4 release (using fuego-1211-container,
with a rolling master branch through December 2018 and January 2019):
See an overview issues table at [[Release test results]]
= to do =
- review bugs labeled as FIXED in [[Release 1.4 testing - 2018-12-26]]
- categorize all bugs
- likely location: core, test, board, toolchain, lab, unknown
- severity: high, medium, low
- time to fix: short, long
- priority: 5=high, 1=low, 0=defer
Resolution key:
- CANDIDATE - is a candidate for fixing this release
- RESEARCH - needs more research to categorize and prioritize
- TRANSIENT - problem went away with subsequent test and no change in Fuego
- LOCAL (lab,board) - problem was with setup, config, board status, board, or toolchain, and issue can be ignored
- IGNORE - ignore (not a problem)
- FIXED - problem fixed with change in Fuego
- DEFER - problem exists, but will not be fixed in this release
= uncategorized issues list =
== deferred from 1.4 release ==
- 1. (test) LTP runs for too long with nothing showing on the console. Jenkins should show the current testcase being run, or something.
- would require output from fuego_test.sh
- 4. (core) Jenkins doesn't show different color for ERROR (timeout or abort)
- 6. (core) '''bbb.default.Benchmark.cyclictest''' it's not easy to find the threshold value. It's not in run.json. It's on the chart, but with multiple boards it's hard to isolate and find
- could add checkboxes for series.
- could add reference value to run.json
- could copy reference.log or reference.json to log directory
- could have parser add reference check failure to console log
- 7. (core) '''ftc rm-jobs rpi3-1.*.*.*''' - can't do rm-jobs outside the container! (annoying, but whatever)
- required due to possible interactive confirmation
- should be able to rm-jobs without being inside the container
- should be able to do interactive commands even when forwarding to the container ftc
- 9. RESEARCH (test) '''min1.default.Functional.LTP''' - new failure for open10
- 10. RESEARCH (test)''' min1.default.Functional.LTP''' - new failure for creat08
- 11. (core?) need to have test indicate whether error code indicates error or failure
- example: x11perf - non-zero = error (not failure)
- 16. CANDIDATE, HIGH (core) '''fuego-test.default.Functional.fuego_release_test'''
- problem with Python 3 dependencies in docker project (requests requires urllib3, which is missing dependencies?)
{{{#!YellowBox
...
= ==== doing fuego phase: run =====
-------------------------------------------------
/usr/local/lib/python3.4/dist-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.9.1) or chardet (3.0.4) doesn't match a supported version!
RequestsDependencyWarning)
Traceback (most recent call last):
File "/fuego-core/engine/tests/Functional.fuego_release_test/test_run.py", line 15, in <module>
import docker
File "/usr/local/lib/python3.4/dist-packages/docker/__init__.py", line 2, in <module>
from .api import APIClient
File "/usr/local/lib/python3.4/dist-packages/docker/api/__init__.py", line 2, in <module>
from .client import APIClient
File "/usr/local/lib/python3.4/dist-packages/docker/api/client.py", line 5, in <module>
import requests
File "/usr/local/lib/python3.4/dist-packages/requests/__init__.py", line 105, in <module>
from urllib3.exceptions import DependencyWarning
ImportError: cannot import name 'DependencyWarning'
}}}
- 17. (board) '''rpi3-2.default.Functional.fuego_dependencies''' - - can't find kconfig
- missing /proc/config.gz or /boot/config-4.9.41-v7+ (but has /lib/modules.../configs.ko)
- 18. (test) '''rpi3-2.default.Functional.fuego_sleep''' - timed out
- timeout from testplan_fuego_tests is too short
- 19. (core) test that fails pre-check is shown as 'not run' in Jenkins
- 20. (core) test that fails pre-check has no result code (listed as None) in 'ftc gen-report'
- 22. (test) '''ren1.default.Functional.LTP_one_test.syscalls.add_key_02''' - timeout is not converted to Fuego ERROR
- the LTP parser should convert a timed-out test in LTP into a Fuego ERROR in the run.json file. It doesn't do this.
== new errors ==
- xx. batch testcase names end in underscore in underscore
- log_this should not put trailing space in hostlog.txt
- or, have parser trim trailing whitespace (regex?) or python code?
- xx. (core) '''docker.default.Functional.batch_nested''' - nested batch test has invalid job page link in hostlog.txt
- shown as "Info:", when it should be '2' (see run 1)
{{{#!YellowBox
Running test Functional.batch_hello
Log dir: logs/Functional.batch_hello/docker.Info:.Info:.Info:
Job page: http://localhost:8090/fuego/job/docker.Info:.Functional.batch_hello/Info:/
[[nested-4]] ok 2 - Functional.batch_hello
}}}
- xx. (distro) '''docker.default.Functional.fuego_board_check''' - missing /sbin/reboot
- /sbin/reboot is present on host as a symlink to /bin/systemctl
- xx. '''ren1.default.Functional.hello_world''' (run 3) - problem importing filelock
- python traceback in prepare_chart_data.py trying to import module
'filelock'
- problem is in poky-aarch64-tools.sh - re-declaration of ORIG_PATH
- xx. '''ftc query-board -b fuego_test''' - python error
{{{#!YellowBlock
root@timdesk:/fuego-rw# ftc query-board -b ftc-test
Information for board: ftc-test
ARCHITECTURE : "arm"
EXPAT_SUBTEST_COUNT_NEG : "41"
EXPAT_SUBTEST_COUNT_POS : "1769"
FUEGO_HOME : "/home/a"
Traceback (most recent call last):
File "/usr/local/bin/ftc", line 5275, in <module>
main()
File "/usr/local/bin/ftc", line 5260, in main
do_query_board(bvars, options)
File "/usr/local/bin/ftc", line 4203, in do_query_board
if value.find('\n') == -1:
AttributeError: 'list' object has no attribute 'find'
= ==== doing fuego phase: processing =====
log_compare: pattern 'SUCCESS' found 1 times (expected greater or equal than 1)
Traceback (most recent call last):
File "/fuego-core/scripts/generic_parser.py", line 30, in <module>
sys.exit(plib.process(results))
File "/fuego-core/scripts/parser/common.py", line 701, in process
from prepare_chart_data import store_flat_results, make_chart_data
File "/fuego-core/scripts/parser/prepare_chart_data.py", line 36, in <module>
from filelock import FileLock
ImportError: No module named filelock
}}}
- need to update all -tools.sh files in ~/work/fuego-reference
- need to make sure following are reviewed:
- ./aarch64-tools.sh:ORIG_PATH=$PATH
- ./debian-armhf-tools.sh:ORIG_PATH=$PATH
- ./poky-qemuarm-tools.sh:ORIG_PATH=$PATH
- ./lager-tools.sh:ORIG_PATH=$PATH
- ./x86_64-tools.sh:ORIG_PATH=$PATH
- ./debian-armel-tools.sh:ORIG_PATH=$PATH
- xx. Can't easily navigate to LTP individual test logs in Jenkins interface
= old errors =
- [[Benchmark.GLMark]] - build failure on multiple platforms (missing glew.h, missing libSDL)
= Errors by board =
# ftc list-runs --where "start_time>4 days ago,status!=PASS,board=bbb"
== bbb board ==
- (old) '''bbb.default.Benchmark.GLMark''' - missing GL/glew.h
- (old) '''bbb.default.Benchmark.Interbench''' - run error: hangs in Audio test
- '''bbb.default.Benchmark.gtkperf''' - build error: configure error: checking for gtk+-2.0 gdk-2.0
- '''bbb.default.Benchmark.X11perf''' - missing X libs
- (RESEARCH) '''bbb.default.Functional.glib''' - missing gettext lib
{{{
Runs on this host:
Test Status
--------------------------- - ------
Benchmark.Dhrystone ERROR - fixed
Benchmark.GLMark SKIP - build error: GL/glew.h: No such file or directory
Benchmark.Interbench ERROR - run error: hangs in Audio test
Benchmark.Java SKIP - no java: OK
Benchmark.gtkperf SKIP - build error: configure error: checking for gtk+-2.0 gdk-2.0 ...
Benchmark.netpipe SKIP - missing NPtcp in host container, missing netpipe-tcp in container
Benchmark.x11perf SKIP - missing X libs
Functional.LTP FAIL - multiple failures
Functional.bc ERROR
Functional.fontconfig SKIP
Functional.ft2demos SKIP
Functional.fuego_abort SKIP
Functional.fuego_compare_reports FAIL - not configured
Functional.fuego_ftc_test FAIL
Functional.fuego_lint FAIL
Functional.fuego_release_test FAIL
Functional.fuego_slow_log FAIL
Functional.fuego_test_variables FAIL - expected
Functional.glib SKIP
Functional.hello_world FAIL - expected (spec=fail)
}}}
- CANDIDATE '''bbb.default.Benchmark.netpipe' - run error: missing NPtcp in docker container
- /usr/bin/NPtcp is provided by Debian package netpipe-tcp
- + add netpipe-tcp to Dockerfile
== min1 board ==
Here are the tests that had issues:
{{{
# ftc list-runs --where board=min1,status!=PASS
Runs on this host:
Test Status
--------------------------- ------
Benchmark.GLMark SKIP - build errror, missing GL/glew.h (old)
Benchmark.cyclictest FAIL - non-rt kernel
Benchmark.ffsb SKIP - current utilization = 1844660474359607984128.00% (RESEARCH)
Benchmark.gtkperf SKIP - build error: configure error: checking for gtk+-2.0 gdk-2.0 (old)
Benchmark.signaltest FAIL - non-rt kernel
Benchmark.x11perf SKIP - runtime: unable to open display ':0
Functional.LTP FAIL - see below
Functional.glib SKIP - build error: missing gettext support (RESEARCH)
}}}
----
Here are the LTP non-PASS results:
{{{
# ftc gen-report --where board=min1,test=LTP,tguid:result!=PASS
======================================================================
**** Fuego Test Report ****
test : LTP
board : min1
kernel : 4.15.0-33-generic
timestamp : 2019-05-03_12:57:07
report_date : 2019-05-08_00:30:28
======================================================================
--------------------------------------------------------------------------------------
test_name spec board timestamp tguid tguid:result
--------------------------------------------------------------------------------------
Functional.LTP default min1 2019-05-03_12:57:07 syscalls FAIL
Functional.LTP default min1 2019-05-03_12:57:07 syscalls.fanotify06 FAIL
Functional.LTP default min1 2019-05-03_12:57:07 syscalls.msgctl11 FAIL
Functional.LTP default min1 2019-05-03_12:57:07 syscalls.open10 FAIL
Functional.LTP default min1 2019-05-03_12:57:07 syscalls.creat08 FAIL
Functional.LTP default min1 2019-05-03_12:57:07 sem_init.sem_init_7-1 SKIP
--------------------------------------------------------------------------------------
}}}
---
Here is the reason for each LTP failure (with some comments)
{{{
------------------------------------
tguid tguid:result
------------------------------------
syscalls.fanotify06 FAIL - segfault
syscalls.msgctl11 FAIL - queue mismatch
syscalls.open10 FAIL - setgid not set
syscalls.creat08 FAIL - setgid not set
sem_init.sem_init_7-1 SKIP - ??? (RESEARCH)
------------------------------------
}}}
== ren1 board ==
{{{
Runs on this host:
Test Status
--------------------------- ------
Benchmark.Dhrystone FAIL - time too small (expected - but could fix test)
Benchmark.GLMark SKIP - missing SDL/SDL.h (old)
Benchmark.Java SKIP - pre_test: missing java (expected, ok)
Benchmark.gtkperf SKIP - gdk-pixbuf-query-loaders: command not found
Benchmark.reboot SKIP - runtime error: didn't capture reboot (RESEARCH)
Benchmark.signaltest FAIL - non-RT kernel
Benchmark.x11perf SKIP - build error: no package xmuu
Functional.LTP FAIL - see below
Functional.aiostress SKIP - pre_test: no /lib/*/libaio (ok?)
Functional.crashme FAIL - 0 .. 507, but expected 3000
Functional.ft2demos SKIP - build error: missing /usr/lib/libxcb.la (RESEARCH)
Functional.glib SKIP - missing gettext support (CANDIDATE)
Functional.kernel_build FAIL - missing "Kernel: arch/.* is ready". bzImage, no such file or directory
}}}
----
Here is the LTP report for non-PASS tests:
{{{
======================================================================
**** Fuego Test Report ****
test : LTP
board : ren1
kernel : 4.9.0-yocto-standard
timestamp : 2019-05-03_13:41:21
report_date : 2019-05-08_00:39:12
======================================================================
----------------------------------------------------------------------------------------
test_name spec board timestamp tguid tguid:result
----------------------------------------------------------------------------------------
Functional.LTP default ren1 2019-05-03_13:41:21 syscalls FAIL
Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.cve-2017-6951 FAIL
Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.prot_hsymlinks FAIL
Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.renameat202 FAIL
Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.add_key04 FAIL
Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.add_key02 FAIL
Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.keyctl08 FAIL
Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.keyctl03 FAIL
Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.keyctl02 FAIL
Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.keyctl05 FAIL
Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.keyctl04 FAIL
Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.keyctl07 FAIL
Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.keyctl06 FAIL
Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.msgctl09 FAIL
Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.msgctl08 FAIL
Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.fanotify06 FAIL
Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.utime02 FAIL
Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.mknod03 FAIL
Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.abort01 FAIL
Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.kill11 FAIL
Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.msgctl10 FAIL
Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.msgctl11 FAIL
Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.request_key02 FAIL
Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.open10 FAIL
Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.renameat201 FAIL
Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.creat08 FAIL
Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.add_key03 FAIL
Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.setsockopt02 FAIL
Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.cve-2017-5669 FAIL
Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.sendmsg01 FAIL
Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.request_key01 FAIL
Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.request_key03 FAIL
Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.setpriority01 FAIL
Functional.LTP default ren1 2019-05-03_13:41:21 syscalls.utime01 FAIL
Functional.LTP default ren1 2019-05-03_13:41:21 sem_init.sem_init_7-1 SKIP
----------------------------------------------------------------------------------------
}}}
----
Here are LTP details:
{{{
--------------------------------------
tguid tguid:result
--------------------------------------
sem_init.sem_init_7-1 SKIP
syscalls.abort01 FAIL - invalid argument
syscalls.add_key02 FAIL - EINVAL
syscalls.add_key03 FAIL - create keyring for another user
syscalls.add_key04 FAIL - SIGSEGV
syscalls.creat08 FAIL - setgid should be set
syscalls.cve-2017-5669 FAIL - mapped VM addr in first 64Kb
syscalls.cve-2017-6951 FAIL - timeout requesting dead key
syscalls.fanotify06 FAIL - SIGSEGV
syscalls.keyctl02 FAIL - timeout
syscalls.keyctl03 FAIL - timeout
syscalls.keyctl04 FAIL - timeout
syscalls.keyctl05 FAIL - timeout
syscalls.keyctl06 FAIL - timeout
syscalls.keyctl07 FAIL - timeout
syscalls.keyctl08 FAIL - timeout
syscalls.kill11 FAIL - invalid argument to setrlimit
syscalls.mknod03 FAIL - setgid bit not set
syscalls.msgctl08 FAIL - ipcs not found
syscalls.msgctl09 FAIL - ipcs not found
syscalls.msgctl10 FAIL - ipcs not found
syscalls.msgctl11 FAIL - ipcs not found
syscalls.open10 FAIL - setgid bit not set
syscalls.prot_hsymlinks FAIL - failed to run cmd: useradd hsym
syscalls.renameat201 FAIL - Invalid argument
syscalls.renameat202 FAIL - Invalid argument
syscalls.request_key01 FAIL - timeout
syscalls.request_key02 FAIL - timeout
syscalls.request_key03 FAIL - timeout
syscalls.sendmsg01 FAIL - ip/ifconfig not found
syscalls.setpriority01 FAIL - useradd exited with non-zero code
syscalls.setsockopt02 FAIL - allowed bad tp_sizeof_priv value
syscalls.utime01 FAIL - tmp_file access and mod times not set
syscalls.utime02 FAIL - tmp_file access and mod times not set
}}}
--------------------------------------
== fuego-test board ==
- (CANDIDATE) '''fuego-test.default.Functional.fuego_abort''' - reports SKIP, but should be OK (behavior is correct)
- should turn this test into a nested test, that checks for correct messages in the console log
- (CANDIDATE) '''fuego-test.default.fuego_board_check''' - missing /sbin/reboot in container
- '''fuego-test.default.Functional.fuego_ftc_test''' - testcase 6 (ftc query-board differences)
- '''fuego-test.default.Functional.fuego_lint''' - expected failures
- '''fuego-test.default.Functional.fuego_release_test''' -
- '''fuego-test.default.Functional.fuego_slow_log''' -
{{{
Runs on this host:
Test Status
--------------------------- ------
Functional.fuego_abort SKIP - OK (CANDIDATE)
Functional.fuego_board_check FAIL - missing /sbin/reboot
Functional.fuego_ftc_test FAIL - testcase 6, addition of ssh2serial, misc.
Functional.fuego_lint FAIL
Functional.fuego_release_test FAIL
Functional.fuego_slow_log FAIL
}}}