Release 1.4 testing - 2018-12-26
to do [edit section]
- categorize all bugs
- likely location: core, test, board, toolchain, lab, unknown
- severity: high, medium, low
- time to fix: short, long
- priority: 5=high, 1=low, 0=defer
Resolution key:
- CANDIDATE - is a candidate for fixing this release
- RESEARCH - needs more research to categorize and prioritize
- TRANSIENT - problem went away with subsequent test and no change in Fuego
- LOCAL (lab,board) - problem was with setup, config, board status, board, or toolchain, and issue can be ignored
- IGNORE - ignore (not a problem)
- FIXED - problem fixed with change in Fuego
- DEFER - problem exists, but will not be fixed in this release
uncategorized issues list [edit section]
- 1. FIXED (core) bbb.default.Benchmark.Interbench - timeout is reported as PASS in Jenkins
- see build_number 1 - test hung and it took 40 minutes
- 2. DEFER (test) LTP runs for too long with nothing showing on the console. Jenkins should show the current testcase being run, or something.
- would require output from fuego_test.sh
- 3. FIXED (core) min1.default.Benchmark.OpenSSL - took 15 minutes (which timed out) but is reported as PASS in Jenkins
- 4. DEFER (core) Jenkins doesn't show different color for ERROR (timeout or abort)
- 5. FIXED (core) main.sh may return success even when the test was interrupted
- e.g. when terminated with SIGTERM, to allow post-test to run and results are only partial
- 6. DEFER (core) bbb.default.Benchmark.cyclictest it's not easy to find the threshold value. It's not in run.json. It's on the chart, but with multiple boards it's hard to isolate and find
- could add checkboxes for series.
- could add reference value to run.json
- could copy reference.log or reference.json to log directory
- could have parser add reference check failure to console log
- 7. DEFER (core) ftc rm-jobs rpi3-1.*.*.* - can't do rm-jobs outside the container! (annoying, but whatever)
- required due to possible interactive confirmation
- should be able to rm-jobs without being inside the container
- should be able to do interactive commands even when forwarding to the container ftc
- 8. FIXED (core) min1.default.Benchmark.OpenSSL - has timeout of 15 minutes (should be default of 40, from what I can see)
- was from 15m default timeout in testplan_smoketest.json
- 9. RESEARCH (test) min1.default.Functional.LTP - new failure for open10
- 10. RESEARCH (test) min1.default.Functional.LTP - new failure for creat08
- 11. DEFER (core?) need to have test indicate whether error code indicates error or failure
- example: x11perf - non-zero = error (not failure)
- 12. FIXED (core) it was hard to add a default timeout to testplan_fuego_tests, because I couldn't find the schema for testplans very easily. I didn't know the variable name.
- added wiki page Testplan Reference
- 13. FIXED (core) testplans usb, mmc and sata could have used 'default_spec' to save lines
- 14. FIXED (core) fuego-test.default.Functional.fuego_dependencies
- testcase 5 NEED_KCONFIG=CONFIG_PRINTK=y
- problem finding /boot/config-$(uname -r) for need_KCONFIG
- 15. FIXED (core) ren1.default.Functional.fuego_test_minimal
- fails because of missing spec.json in ftc
- 16. CANDIDATE, HIGH (core) fuego-test.default.Functional.fuego_release_test
- problem with Python 3 dependencies in docker project (requests requires urllib3, which is missing dependencies?)
= ==== doing fuego phase: run =====
-------------------------------------------------
/usr/local/lib/python3.4/dist-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.9.1) or chardet (3.0.4) doesn't match a supported version!
RequestsDependencyWarning)
Traceback (most recent call last):
File "/fuego-core/engine/tests/Functional.fuego_release_test/test_run.py", line 15, in <module>
import docker
File "/usr/local/lib/python3.4/dist-packages/docker/__init__.py", line 2, in <module>
from .api import APIClient
File "/usr/local/lib/python3.4/dist-packages/docker/api/__init__.py", line 2, in <module>
from .client import APIClient
File "/usr/local/lib/python3.4/dist-packages/docker/api/client.py", line 5, in <module>
import requests
File "/usr/local/lib/python3.4/dist-packages/requests/__init__.py", line 105, in <module>
from urllib3.exceptions import DependencyWarning
ImportError: cannot import name 'DependencyWarning'
- 16. RESEARCH (unknown) rpi3-2.default.Functional.fuego_test_variables
- testcases 2,3, and 9
- 17. DEFER (board) rpi3-2.default.Functional.fuego_dependencies - - can't find kconfig
- missing /proc/config.gz or /boot/config-4.9.41-v7+ (but has /lib/modules.../configs.ko)
- 18. DEFER (test) rpi3-2.default.Functional.fuego_sleep - timed out
- timeout from testplan_fuego_tests is too short
- 19. DEFER (core) test that fails pre-check is shown as 'not run' in Jenkins
- 20. DEFER (core) test that fails pre-check has no result code (listed as None) in 'ftc gen-report'
- 21. FIXED (core) ren1.default.Functional.aiostress - (post-processing) Invalid literal for base 10 in generic_parser
- changed to a try here, and convert to ERROR in except clause
- or figure out why RETURN_VALUE is missing!!
- it's missing because:
- the build error caused early exit
- RETURN_VALUE was never set (by report, et al.)
- this test doesn't have a test_processing function
- it's missing because:
##### doing fuego phase: processing (from signal handler) #####
WARNING: program returned exit code
Log evaluation may be invalid
Traceback (most recent call last):
File "/fuego-core/engine/scripts/generic_parser.py", line 18, in <module>
value = "PASS" if int(RETURN_VALUE) == 0 else "FAIL"
ValueError: invalid literal for int() with base 10: ''
ERROR: results did not satisfy the threshold
- 22. DEFER (test) ren1.default.Functional.LTP_one_test.syscalls.add_key_02 - timeout is not converted to Fuego ERROR
- the LTP parser should convert a timed-out test in LTP into a Fuego ERROR in the run.json file. It doesn't do this.
- 21. FIXED (core) - ftc rm-node rpi3-1 - cannot remove node rpi3-1
- get message "global name 'install_nodes' is not defined"
- 23. FIXED (core) - ftc rm-node rpi3-1 - has no effect
- 24. FIXED (core) - ftc rm-jobs --remove-logs Bench.default.Benchmark.hackbench
- gives weird message
Test failures by board [edit section]
Here are testplan_default failures for 2018-12-26bbb [edit section]
Summary:# ftc list-runs --where "start_time>4 days ago,status!=PASS,board=bbb" Runs on this host: Test Started host:board Status --------------------------- ---------------------- ------------------------------ ------ Benchmark.Interbench 2018-12-26_18:00:01 timslab:bbb ERROR Benchmark.OpenSSL 2018-12-26_18:50:36 timslab:bbb SKIP Benchmark.cyclictest 2018-12-26_18:51:25 timslab:bbb FAIL Benchmark.gtkperf 2018-12-26_19:01:27 timslab:bbb SKIP Benchmark.x11perf 2018-12-26_18:49:56 timslab:bbb SKIP Functional.LTP 2018-12-26_19:07:47 timslab:bbb FAIL Functional.LTP 2018-12-26_21:20:47 timslab:bbb FAIL Functional.OpenSSL 2018-12-26_18:56:31 timslab:bbb FAIL Functional.aiostress 2018-12-26_23:06:06 timslab:bbb FAIL Functional.fontconfig 2018-12-26_19:01:14 timslab:bbb FAIL Functional.ft2demos 2018-12-26_19:00:25 timslab:bbb ERROR
Analysis:
- FIXED (test) bbb.default.Benchmark.Interbench - ERROR - timeout of 40m reported as SUCCESS
- reported as success in Jenkins and ERROR in run.json
- DEFER (test, board) bbb.default.Benchmark.Interbench - ERROR - test hangs!! (not new)
- hangs in Audio test (not new, but should be resolved)
last data reported from test is: --- Benchmarking simulated cpu of Audio in the presence of simulated --- Load Latency +/- SD (ms) Max Latency % Desired CPU % Deadlines Met None 0.1 +/- 0.1 0.3 100 100 Video 0.1 +/- 0.1 0.2 100 100 X 0.1 +/- 0.1 1.3 100 100 Burn 0.0 +/- 0.0 0.1 100 100 Write calloc: Cannot allocate memory 0.1 +/- 0.3 6.6 100 100
- DEFER (test,toolchain) bbb.default.Benchmark.OpenSSL - build failed - missing zconf.h from zlib (not new)
arm-linux-gnueabihf-gcc -I.. -I../.. -I../asn1 -I../evp -I../../include \ --sysroot=/ -DZLIB_SHARED -DZLIB -DOPENSSL_THREADS -c -o c_zlib.o c_zlib.c In file included from c_zlib.c:25:0: /usr/include/zlib.h:34:19: fatal error: zconf.h: No such file or directory #include "zconf.h" ^ compilation terminated.
- CANDIDATE (board) bbb.default.Benchmark.cyclictest - FAIL - board doesn't have RT kernel (expected)
- max_latency doesn't pass threshold test, for 1000ms reference value
- fix would be to provide threshold for non-RT board
- add bbb.Benchmark.cyclictest.criteria.json
- DEFER (test?,toolchain?) bbb.default.Benchmark.gtkperf - build failure (not new)
- build failed - can't build test
- last message: checking for gtk+-2.0 gdk-2.0
- build failed - can't build test
- DEFER (toolchain) bbb.default.Benchmark.x11perf - build failure
- build failed - cannot find -lXext -lXrender -lX11 -lXmu
- DEFER (test) bbb.default.Functional.glib - build failure - swp in Thumb mode not supported (not new)
- error was:
/bin/sh ../libtool --tag=CC --mode=compile arm-linux-gnueabihf-gcc -DHAVE_CONFIG_H -I. -I.. \ -I.. -DG_LOG_DOMAIN=\"GLib\" -DG_DISABLE_CAST_CHECKS -DG_DISABLE_DEPRECATED -DGLIB_COMPILATION \ -DPCRE_STATIC -DG_DISABLE_SINGLE_INCLUDES -pthread -g -O2 -Wall -MT gatomic.lo -MD -MP \ -MF .deps/gatomic.Tpo -c -o gatomic.lo gatomic.c libtool: compile: arm-linux-gnueabihf-gcc -DHAVE_CONFIG_H -I. -I.. -I.. -DG_LOG_DOMAIN=\"GLib\" \ -DG_DISABLE_CAST_CHECKS -DG_DISABLE_DEPRECATED -DGLIB_COMPILATION -DPCRE_STATIC \ -DG_DISABLE_SINGLE_INCLUDES -pthread -g -O2 -Wall -MT gatomic.lo -MD -MP -MF .deps/gatomic.Tpo \ -c gatomic.c -fPIC -DPIC -o .libs/gatomic.o /tmp/ccTcovfi.s: Assembler messages: /tmp/ccTcovfi.s:65: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode /tmp/ccTcovfi.s:85: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode /tmp/ccTcovfi.s:188: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode /tmp/ccTcovfi.s:208: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode /tmp/ccTcovfi.s:310: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode /tmp/ccTcovfi.s:330: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode /tmp/ccTcovfi.s:432: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode /tmp/ccTcovfi.s:452: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode Makefile:994: recipe for target 'gatomic.lo' failed
- RESEARCH (test) bbb.default.Functional.LTP - FAIL - see details below
- DEFER (test) bbb.default.Functional.OpenSSL - FAIL - missing zconf.h (not new)
- FIXED (board) bbb.default.Functional.aiostress - FAIL - can't load libaio.so.1 on board (not new)
- should be reported as ERROR (failure to run test program)
- FIXED (board) bbb.default.Functional.aiostress - FAIL - can't load libaio.so.1 on board (not new)
- should use dependency check for package libaio (not sure it will work with non-Debian)
- should resolve with a dependency on libaio.so.1 on the board
- does this test apply, if libaio.so is not present on the board?? - NO
- libaio.so.1 is located in bbb:/lib/arm-linux-gnueabihf/libaio.so.1
- /lib/arm-linux-gnueabihf system shared lib path is in: /etc/ld.so.conf.d/arm-linux-gneuabihf.conf
- can't build without libaio-dev in the SDK (libaio.h)
- aio functions are supposed to be provided by C library
- removed '-laio' from build line
- can be resolved with 'apt-get install libaio1' on the board
- FIXED (board) bbb.default.Functional.fontconfig - FAIL - missing fc-list (expected)
- added assert_has_program to fuego_test.sh
- DEFER bbb.default.Functional.ft2demos - ERROR - missing X libs (not new)
libtool: link: arm-linux-gnueabihf-gcc -Wl,-L -Wl,//usr/lib -Wl,-lXau -Wl,-lXdmcp --sysroot / \ -o /fuego-rw/buildzone/bbb.default.Functional.ft2demos-debian-armhf/ft2demos-2.3.6/bin/.libs/ftlint \ /fuego-rw/buildzone/bbb.default.Functional.ft2demos-debian-armhf/ft2demos-2.3.6/obj/ftlint.o \ /fuego-rw/buildzone/bbb.default.Functional.ft2demos-debian-armhf/ft2demos-2.3.6/obj/common.o \ ../freetype-2.3.6/objs/.libs/libfreetype.so -lm /usr/lib/gcc-cross/arm-linux-gnueabihf/6/../../../../arm-linux-gnueabihf/bin/ld: cannot find -lXau /usr/lib/gcc-cross/arm-linux-gnueabihf/6/../../../../arm-linux-gnueabihf/bin/ld: cannot find -lXdmcp collect2: error: ld returned 1 exit status Makefile:426: recipe for target '/fuego-rw/buildzone/bbb.default.Functional.ft2demos-debian-armhf/ft2demos-2.3.6/bin/ftlint' failed make: *** [/fuego-rw/buildzone/bbb.default.Functional.ft2demos-debian-armhf/ft2demos-2.3.6/bin/ftlint] Error 1
- DEFER (test, toolchain) - bbb.default.Functional.glib - problem with thumb mode instructions
libtool: compile: arm-linux-gnueabihf-gcc -DHAVE_CONFIG_H -I. -I.. -I.. \ -DG_LOG_DOMAIN=\"GLib\" -DG_DISABLE_CAST_CHECKS -DG_DISABLE_DEPRECATED -DGLIB_COMPILATION \ -DPCRE_STATIC -DG_DISABLE_SINGLE_INCLUDES -pthread -g -O2 -Wall -MT gatomic.lo -MD -MP \ -MF .deps/gatomic.Tpo -c gatomic.c -fPIC -DPIC -o .libs/gatomic.o /tmp/ccTcovfi.s: Assembler messages: /tmp/ccTcovfi.s:65: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode /tmp/ccTcovfi.s:85: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode /tmp/ccTcovfi.s:188: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode /tmp/ccTcovfi.s:208: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode /tmp/ccTcovfi.s:310: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode /tmp/ccTcovfi.s:330: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode /tmp/ccTcovfi.s:432: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode /tmp/ccTcovfi.s:452: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode Makefile:994: recipe for target 'gatomic.lo' failed make[4]: *** [gatomic.lo] Error 1
bbb LTP results [edit section]
# ftc gen-report --where "board=bbb,test=LTP,tguid:result!=PASS,start_time>9 days ago" | sort -k 5 board : bbb kernel : 4.4.88-ti-r125 report_date : 2019-01-04_20:20:37 test : LTP timestamp : 2018-12-26_21:20:47 -------------------------------------------------------------------------------------------------------------------------------- ====================================================================== test_name spec board timestamp tguid tguid:result Functional.LTP default bbb 2018-12-26_21:20:47 sem_init.sem_init_7-1 SKIP Functional.LTP default bbb 2018-12-26_21:20:47 syscalls FAIL Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.add_key03 FAIL Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.add_key04 FAIL Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.eventfd01 FAIL Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.fanotify06 FAIL Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.fcntl35 FAIL Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.fcntl35_64 FAIL Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.getrusage03 FAIL Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.io_cancel01 FAIL Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.io_destroy01 FAIL Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.io_getevents01 FAIL Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.io_setup01 FAIL Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.io_submit01 FAIL Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.keyctl02 FAIL Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.keyctl03 FAIL Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.keyctl04 FAIL Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.keyctl05 FAIL Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.keyctl06 FAIL Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.keyctl07 FAIL Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.msgctl10 FAIL Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.msgctl11 FAIL Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.request_key01 FAIL Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.request_key02 FAIL Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.request_key03 FAIL
min1 [edit section]
Summary:$ ftc list-runs --where "start_time>yesterday,status!=PASS,board=min1" Test Started host:board Status Benchmark.Java 2018-12-26_18:56:19 timslab:min1 ERROR Benchmark.OpenSSL 2018-12-26_19:36:18 timslab:min1 ERROR Benchmark.cyclictest 2018-12-26_19:51:52 timslab:min1 FAIL Benchmark.signaltest 2018-12-26_19:55:13 timslab:min1 FAIL Functional.LTP 2018-12-26_18:00:42 timslab:min1 FAIL Functional.glib 2018-12-26_18:57:06 timslab:min1 FAIL Benchmark.GLMark 2018-12-26_18:55:04 timslab:min1 SKIP Benchmark.x11perf 2018-12-26_18:50:09 timslab:min1 SKIP --------------------------- ---------------------- ------------------------------ ------
Analysis:
- DEFER (toolchain) min1.default.Benchmark.GLMark - ERROR - failed to build
- missing glew.h file for build
- should have been reported as an error
- FIXED (test) min1.default.Benchmark.Java - ERROR - (board does not have java command (expected)
- changed to 'assert_has_program java'
- FIXED (core) min1.default.Benchmark.OpenSSL - interrupted at 15 minutes (according to timeout value), returned SUCCESS to Jenkins
- changed so that ERROR is returned on timeouts
- need to change timeout higher
- CANDIDATE (board) min1.default.Benchmark.cyclictest - FAIL (board does not have RT kernel) (expected)
- max latency over threshold
- should be higher threshold?
- add non-RT threshold??
- max latency over threshold
- CANDIDATE (board) min1.default.Benchmark.signaltest - FAIL (board does not have RT kernel) (expected)
- max latency over threshold
- should be higher threshold?
- max latency over threshold
- DEFER (test?) min1.default.Benchmark.x11perf - SKIP
- x11perf: unable to open display ':0"
- should have been reported as an error
- RESEARCH (test) min1.default.Functional.LTP - FAIL - 10 failures in syscalls, 1 sem_init SKIP (was 9 failures in 1.3 - see details below)
- DEFER (test?) min1.default.Functional.glib - FAIL
- 'threadtests' does not report pass or failure. Test appears to terminate prematurely? (see testlog.txt)
- there are many GLib-CRITICAL reports for "Source ID ## was not found when attempting to remove it"
- returns exit code 143
min1 LTP results [edit section]
# ftc gen-report --where "board=min1,test=LTP,tguid:result!=PASS"
======================================================================
**** Fuego Test Report ****
test : LTP
board : min1
kernel : 4.15.0-33-generic
timestamp : 2018-12-26_18:00:42
report_date : 2018-12-28_19:42:54
======================================================================
--------------------------------------------------------------------------------------------------------------------------------
test_name spec board timestamp tguid tguid:result
--------------------------------------------------------------------------------------------------------------------------------
Functional.LTP default min1 2018-12-26_18:00:42 syscalls FAIL
Functional.LTP default min1 2018-12-26_18:00:42 syscalls.io_getevents01 FAIL
Functional.LTP default min1 2018-12-26_18:00:42 syscalls.fanotify06 FAIL
Functional.LTP default min1 2018-12-26_18:00:42 syscalls.io_submit01 FAIL
Functional.LTP default min1 2018-12-26_18:00:42 syscalls.io_cancel01 FAIL
Functional.LTP default min1 2018-12-26_18:00:42 syscalls.msgctl11 FAIL
Functional.LTP default min1 2018-12-26_18:00:42 syscalls.eventfd01 FAIL
Functional.LTP default min1 2018-12-26_18:00:42 syscalls.open10 FAIL
Functional.LTP default min1 2018-12-26_18:00:42 syscalls.creat08 FAIL
Functional.LTP default min1 2018-12-26_18:00:42 syscalls.io_setup01 FAIL
Functional.LTP default min1 2018-12-26_18:00:42 syscalls.io_destroy01 FAIL
Functional.LTP default min1 2018-12-26_18:00:42 sem_init.sem_init_7-1 SKIP
--------------------------------------------------------------------------------------------------------------------------------
Different from 1.3:
* RESEARCH min1.default.Functional.LTP.syscalls.open10 - FAIL
* RESEARCH min1.default.Functional.LTP.syscalls.creat08 - FAIL
test that were run in 1.3 but not 1.4 release testing:
- min1.default.Functional.aiostress
- min1.default.Functional.kernel_build
ren1 [edit section]
Summary:# ftc list-runs --where "board=ren1,start_time>yesterday,status!=PASS" Runs on this host: Test Started host:board Status --------------------------- ---------------------- ------------------------------ ------ Benchmark.Dhrystone 2018-12-28_21:11:40 timslab:ren1 FAIL Benchmark.GLMark 2018-12-28_21:34:14 timslab:ren1 SKIP Benchmark.Java 2018-12-28_21:38:54 timslab:ren1 ERROR Benchmark.gtkperf 2018-12-28_21:09:41 timslab:ren1 SKIP Benchmark.signaltest 2018-12-28_21:10:53 timslab:ren1 FAIL Benchmark.x11perf 2018-12-28_21:58:42 timslab:ren1 SKIP Functional.LTP 2018-12-28_21:58:50 timslab:ren1 FAIL Functional.aiostress 2018-12-28_21:34:10 timslab:ren1 ERROR Functional.crashme 2018-12-28_21:31:30 timslab:ren1 FAIL Functional.ft2demos 2018-12-28_20:47:19 timslab:ren1 ERROR Functional.glib 2018-12-28_20:45:28 timslab:ren1 FAIL
Analysis:
- DEFER (test, board) ren1.default.Benchmark.Dhrystone - FAIL (needs 100M spec)
- DEFER (test, toolchain) ren1.default.Benchmark.GLMark - SKIP (build failure - missing SDL.h)
- FIXED (board) ren1.default.Benchmark.Java - ERROR (missing 'java' on board) (expected)
- DEFER ren1.default.Benchmark.gtkperf - SKIP
- (deploy error) sh: gdk-pixbuf-query-loaders: command not found
- DEFER ren1.default.Benchmark.signaltest - FAIL
- DEFER ren1.default.Benchmark.x11perf - SKIP
- RESEARCH ren1.default.Functional.LTP - FAIL - see details below
- DEFER ren1.default.Functional.aiostress - ERROR (build error) (not new)
- aiostress.c:58:20: fatal error: libaio.h: No such file or directory compilation terminated.
- DEFER ren1.default.Functional.crashme - FAIL (not new)
- results did not satisfy the threshold
- log_compare "0 ... 3000" (but test result was "0 ... 507")
- results did not satisfy the threshold
- FIXED ren1.default.Functional.ft2demos - ERROR (build error)
- /fuego-core/engine/tests/Functional.ft2demos/fuego_test.sh: line 16: build_error: command not found
- DEFER ren1.default.Functional.ft2demos - ERROR
- libtool: link: cannot find the library `=/usr/lib/libxcb.la' or unhandled argument `=/usr/lib/libxcb.la' Makefile:470: recipe for target '/fuego-rw/buildzone/ren1.default.Functional.ft2demos-poky-aarch64/ft2demos-2.3.6/bin/ftview' failed - make: *** [/fuego-rw/buildzone/ren1.default.Functional.ft2demos-poky-aarch64/ft2demos-2.3.6/bin/ftview] Error 1
- DEFER ren1.default.Functional.glib - FAIL (build error) (not new, same as 1.3)
- 'threadtests' does not report pass or failure. Test appears to terminate prematurely? (see testlog.txt)
- there are many GLib-CRITICAL reports for "Source ID ## was not found when attempting to remove it"
- returns exit code 143
- RESEARCH ren1.default.Functional.kernel_build - FAIL, build error on assembly instruction
- problem with head.S assembly and poky arm64 toolchain (as)
AS arch/arm64/kernel/head.o arch/arm64/kernel/head.S: Assembler messages: arch/arm64/kernel/head.S:824: Error: missing ')' arch/arm64/kernel/head.S:824: Error: missing ')' arch/arm64/kernel/head.S:824: Error: missing ')' arch/arm64/kernel/head.S:824: Error: unexpected characters following instruction at operand 2 -- `mov x2,#(2)|(2U<<(8))' scripts/Makefile.build:391: recipe for target 'arch/arm64/kernel/head.o' failed make[1]: *** [arch/arm64/kernel/head.o] Error 1 Makefile:1055: recipe for target 'arch/arm64/kernel' failed make: *** [arch/arm64/kernel] Error 2
ren1 LTP results [edit section]
# ftc gen-report --where "board=ren1,test=LTP,tguid:result!=PASS"
======================================================================
**** Fuego Test Report ****
test : LTP
board : ren1
kernel : 4.9.0-yocto-standard
timestamp : 2018-12-26_21:38:12
report_date : 2018-12-28_23:12:46
======================================================================
------- - (had an error about missing program 'build_error', but that was fixed:
- /fuego-core/engine/tests/Functional.ft2demos/fuego_test.sh: line 16: build_error: command not found)
-------------------------------------------------------------------------------------------------------------------------
test_name spec board timestamp tguid tguid:result
--------------------------------------------------------------------------------------------------------------------------------
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.cve-2017-6951 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.prot_hsymlinks FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.renameat202 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.add_key04 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.add_key02 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.keyctl08 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.keyctl03 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.keyctl02 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.keyctl05 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.keyctl04 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.keyctl07 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.keyctl06 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.msgctl09 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.msgctl08 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.fanotify06 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.utime02 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.mknod03 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.abort01 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.recvmsg01 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.kill11 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.msgctl10 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.msgctl11 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.request_key02 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.open10 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.renameat201 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.creat08 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.add_key03 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.setsockopt02 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.cve-2017-5669 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.sendmsg01 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.request_key01 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.request_key03 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.setpriority01 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.utime01 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 sem_init.sem_init_7-1 SKIP
--------------------------------------------------------------------------------------------------------------------------------
Changed from 1.3 release testing were:
- DEFER ren1.default.Functional.LTP.syscalls.add_key02 - failed in 1.4
- IGNORE ren1.default.Functional.LTP.syscalls.fanotify08 - worked now (but failed in 1.3)
- IGNORE ren1.default.Functional.LTP.syscalls.fanotify07 - worked now (but failed in 1.3)
- IGNORE ren1.default.Functional.LTP.syscalls.perf_event_open02 - worked now (but failed in 1.3)
- RESEARCH ren1.default.Functional.LTP.syscalls.mknod03 - failed in 1.4
- RESEARCH ren1.default.Functional.LTP.syscalls.creat08 - failed in 1.4
- DEFER, HIGH (post 1.4) ren1.default.Functional.LTP.syscalls.x (for x in: add_key0[1234], keyctl0[2345678], request_key0[123]) (14 tests)
- add these tests to 'should reboot after test' list for LTP
- these are probably xfail?? for creating zombie processes
Also, during the test, several processes (add_key, keyctl, request_key) were stuck in uninterruptible sleep:
$ ps auxwww (trimmed)
24350 root 2860 S /usr/sbin/dropbear -i -r /etc/dropbear/dropbear_rsa_host_key -B
24351 root 2908 S sh -c { cd /home/fuego/fuego.Functional.LTP; TESTS="syscalls "; PTSTESTS="SEM "; \
RTTESTS=""; . ./ltp_target_run.sh; echo $? > /tmp/22722-26579; } 2>&1 | tee /home/fuego/fuego.Functional.LTP/Functional.LTP.log
24352 root 2908 S sh -c { cd /home/fuego/fuego.Functional.LTP; TESTS="syscalls "; PTSTESTS="SEM "; \
RTTESTS=""; . ./ltp_target_run.sh; echo $? > /tmp/22722-26579; } 2>&1 | tee /home/fuego/fuego.Functional.LTP/Functional.LTP.log
24353 root 1760 S tee /home/fuego/fuego.Functional.LTP/Functional.LTP.log
24377 root 3568 S {runltp} /bin/sh ./runltp -C /home/fuego/fuego.Functional.LTP/result/syscalls/failed.log \
-l /home/fuego/fuego.Functional.LTP/result/syscalls/result.log \
-o /home/fuego/fuego.Functional.LTP/result/syscalls/output.log \
-d /home/fuego/fuego.Functional.LTP/tmp -S ./skiplist.txt -f syscalls
24548 root 1888 S /home/fuego/fuego.Functional.LTP/bin/ltp-pan -e -S -a 24377 -n 24377 \
-f /home/fuego/fuego.Functional.LTP/tmp/ltp-T8KMbYu8JQ/alltests \
-l /home/fuego/fuego.Functional.LTP/result/syscalls/result.log \
-o /home/fuego/fuego.Functional.LTP/result/syscalls/output.log \
-C /home/fuego/fuego.Functional.LTP/result/syscalls/failed.log \
-T /home/fuego/fuego.Functional.LTP/output/LTP_RUN_ON-output.log.tconf
24692 root 1820 D add_key01
24694 root 1820 D add_key02
24696 root 2296 D add_key03
24698 root 1824 D add_key04
24753 root 0 SW [kworker/u12:2]
26884 root 2004 D keyctl02
26886 root 1820 D keyctl03
26888 root 1820 D keyctl04
26890 root 1824 D keyctl05
26892 root 1820 D keyctl06
26895 root 1824 D keyctl07
26897 root 1820 D keyctl08
27708 root 1820 D request_key01
27710 root 1820 D request_key02
27712 root 1824 D request_key03
27714 root 1820 D cve-2017-6951
32620 root 0 SW [kworker/u12:0]
rpi3-2 [edit section]
Summary:cascade caused problems
Analysis:
- DEFER rpi3-2.default.Benchmark.GLMark - SKIP - build error, missing glew.h
- compilation problem with cpp code - missing GL header
n file included from vector.h:4:0,
from vector.cpp:1:
oglsdl.h:5:21: fatal error: GL/glew.h: No such file or directory
#include <GL/glew.h>
^
compilation terminated.
- DEFER - rpi3-2.default.Benchmark.Interbench - times out (hangs)
- hangs in Audio test (well, I stopped it)
- DEFER - rpi3-2.default.Benchmark.bonnie - FAIL (processing failure)
- bug in parser for invalid values (probably ++++)
'===== doing fuego phase: processing =====
Reading current values from /fuego-rw/logs/Benchmark.bonnie/rpi3-2.default.1.1/testlog.txt
Bonnie++ raw results: ['1.97', '1.97', 'rpi3-2', '1', '1546056554', '32M', '', '126', '98', '9574', '6',
'9197', '4', '613', '99', '+++++', '+++', '357.2', '5', '16', '', '', '', '', '4154', '19', '+++++', '+++',
'4582', '29', '3820', '17', '+++++', '+++', '7129', '45', '120ms', '344us', '568us', '18744us', '108us',
'20117us', '59893us', '3679us', '5008us', '566us', '175us', '895us']
Traceback (most recent call last):
File "/fuego-core/engine/tests/Benchmark.bonnie/parser.py", line 36, in <module>
measurements["Sequential_Output.PerChr"].append({"name": "speed", "measure" : float(results[2])})
ValueError: could not convert string to float: rpi3-2
ERROR: results did not satisfy the threshold
- DEFER rpi3-2.default.Benchmark.gtkperf - SKIP build error
- problem in configure - checking for gtk+-2.0 gdk-2.0 ..
- DEFER rpi3-2.default.Functional.glib - ERROR - build error
- compiling gatomic, swp instructions in Thumb mode
- FIXED (board, spec?) - rpi3-2.default.Functional.kernel_build - ERROR (no bzImage, but zImage is present)
- no bzImage, but zImage is present
- kernel make always defaults to building 'zImage' if the target is not recognized 'make foo' makes a zImage.
- fixed by adding FUNCTIONAL_KERNEL_BUILD_TARGET="zImage" to rpi3-2.board
- this is fragile, as the spec value overrides 'target'. But by default no 'target' is specified.
- would be better to use "TARGET_KERNEL_IMAGE" in board, and if specified, override the target from the spec?
- FIXED (core) - rpi3-2.default.Functional.kernel_build -
- got this this from generic_parser.py:
#### doing fuego phase: processing (from signal handler) #####
Processing kernel build log
log_compare: pattern '[ \t]*Kernel: arch/. arch/.. arch/.gitignore is ready' \
found 2 times (expected greater or equal than 1)
Traceback (most recent call last):
File "/fuego-core/engine/scripts/generic_parser.py", line 18, in <module>
value = "PASS" if int(RETURN_VALUE) == 0 else "FAIL"
ValueError: invalid literal for int() with base 10: ''
Overview [edit section]
root@tlinux:/# # ftc list-runs --where "start_time>yesterday,status!=PASS" | sort -k 3 Test Started host:board Status Benchmark.Interbench 2018-12-26_18:00:01 timslab:bbb ERROR Functional.LTP 2018-12-26_21:20:47 timslab:bbb ERROR Functional.ft2demos 2018-12-26_19:00:25 timslab:bbb ERROR Benchmark.cyclictest 2018-12-26_18:51:25 timslab:bbb FAIL Functional.LTP 2018-12-26_19:07:47 timslab:bbb FAIL Functional.OpenSSL 2018-12-26_18:56:31 timslab:bbb FAIL Functional.fontconfig 2018-12-26_19:01:14 timslab:bbb FAIL Benchmark.OpenSSL 2018-12-26_18:50:36 timslab:bbb SKIP Benchmark.gtkperf 2018-12-26_19:01:27 timslab:bbb SKIP Benchmark.x11perf 2018-12-26_18:49:56 timslab:bbb SKIP Benchmark.Java 2018-12-26_18:56:19 timslab:min1 ERROR Benchmark.OpenSSL 2018-12-26_19:36:18 timslab:min1 ERROR Benchmark.cyclictest 2018-12-26_19:51:52 timslab:min1 FAIL Benchmark.signaltest 2018-12-26_19:55:13 timslab:min1 FAIL Functional.LTP 2018-12-26_18:00:42 timslab:min1 FAIL Functional.glib 2018-12-26_18:57:06 timslab:min1 FAIL Benchmark.GLMark 2018-12-26_18:55:04 timslab:min1 SKIP Benchmark.x11perf 2018-12-26_18:50:09 timslab:min1 SKIP Functional.LTP 2018-12-26_21:38:12 timslab:ren1 ERROR Functional.netperf 2018-12-26_18:16:13 timslab:ren1 ERROR Benchmark.signaltest 2018-12-26_18:15:59 timslab:ren1 FAIL Functional.glib 2018-12-26_18:33:38 timslab:ren1 FAIL Benchmark.Whetstone 2018-12-26_19:09:31 timslab:rpi3-2 ERROR Benchmark.tiobench 2018-12-26_19:08:28 timslab:rpi3-2 ERROR Functional.ft2demos 2018-12-26_18:50:43 timslab:rpi3-2 ERROR Functional.kernel_build 2018-12-26_18:38:15 timslab:rpi3-2 ERROR Functional.linus_stress 2018-12-26_18:43:44 timslab:rpi3-2 ERROR Benchmark.signaltest 2018-12-26_19:40:41 timslab:rpi3-2 FAIL Benchmark.GLMark 2018-12-26_19:35:22 timslab:rpi3-2 SKIP Benchmark.OpenSSL 2018-12-26_19:17:55 timslab:rpi3-2 SKIP --------------------------- ---------------------- ------------------------------ ------
rpi3-1 LTP testcase problems [edit section]
FIXTHIS - these are from 1.3 release
root@tlinux:/# ftc gen-report --where "board=rpi3-1,test=LTP,tguid:result!=PASS,start_time>2 days ago"
where start_time value is '1.525060103e+12'
======================================================================
**** Fuego Test Report ****
test : LTP
board : rpi3-1
kernel : 4.9.41-v7+
timestamp : 2018-05-01_01:02:42
report_date : 2018-05-02_03:48:23
======================================================================
--------------------------------------------------------------------------------------------------------------------------------
test_name spec board timestamp tguid tguid:result
--------------------------------------------------------------------------------------------------------------------------------
Functional.LTP default rpi3-1 2018-05-01_01:02:42 syscalls FAIL
Functional.LTP default rpi3-1 2018-05-01_01:02:42 syscalls.add_key04 FAIL
Functional.LTP default rpi3-1 2018-05-01_01:02:42 syscalls.keyctl03 FAIL
Functional.LTP default rpi3-1 2018-05-01_01:02:42 syscalls.keyctl02 FAIL
Functional.LTP default rpi3-1 2018-05-01_01:02:42 syscalls.keyctl05 FAIL
Functional.LTP default rpi3-1 2018-05-01_01:02:42 syscalls.keyctl04 FAIL
Functional.LTP default rpi3-1 2018-05-01_01:02:42 syscalls.keyctl07 FAIL
Functional.LTP default rpi3-1 2018-05-01_01:02:42 syscalls.keyctl06 FAIL
Functional.LTP default rpi3-1 2018-05-01_01:02:42 syscalls.io_getevents01 FAIL
Functional.LTP default rpi3-1 2018-05-01_01:02:42 syscalls.io_submit01 FAIL
Functional.LTP default rpi3-1 2018-05-01_01:02:42 syscalls.fcntl36 FAIL
Functional.LTP default rpi3-1 2018-05-01_01:02:42 syscalls.io_cancel01 FAIL
Functional.LTP default rpi3-1 2018-05-01_01:02:42 syscalls.msgctl11 FAIL
Functional.LTP default rpi3-1 2018-05-01_01:02:42 syscalls.eventfd01 FAIL
Functional.LTP default rpi3-1 2018-05-01_01:02:42 syscalls.request_key02 FAIL
Functional.LTP default rpi3-1 2018-05-01_01:02:42 syscalls.io_setup01 FAIL
Functional.LTP default rpi3-1 2018-05-01_01:02:42 syscalls.io_destroy01 FAIL
Functional.LTP default rpi3-1 2018-05-01_01:02:42 syscalls.add_key03 FAIL
Functional.LTP default rpi3-1 2018-05-01_01:02:42 syscalls.request_key01 FAIL
Functional.LTP default rpi3-1 2018-05-01_01:02:42 syscalls.request_key03 FAIL
Functional.LTP default rpi3-1 2018-05-01_01:02:42 sem_init.sem_init_7-1 SKIP
--------------------------------------------------------------------------------------------------------------------------------
table of current test status [edit section]
See Release_test_results
triage data [edit section]
- candidates
- ren1.default.Functional.fuego_test_minimal
- missing spec.json
- fuego-test.default.Functional.fuego_release_test
- urllib3 python module dependency problem
- ren1.default.Functional.aiostress
- generic_parser python exception
- bbb.default.Benchmark.cyclictest (and min1, and signaltest)
- add board-specific, config-specific, distro-specific criteria
- rpi3-2.default.Functional.kernel_build
- wrong kernel image name
- bbb.default.Functional.aiostress
- can't load libaio.so.1
- maybe add dependency check on library (can you do aio without lib?)
- ren1.default.Functional.fuego_test_minimal
- research
- rpi3-2.default.Functional.fuego_test_variables
- if requires fuego-test board, then add dependency check to test
- min1.default.Functional.LTP.syscalls.open10
- see if it's xfail (expected for this kernel release)
- add to board-specific skiplist?
- min1.default.Functional.LTP.syscalls.creat08
- ren1.default.Functional.kernel_build
- see if it's a real kernel bug that need a patch upstream
- depends on whether poky ren1 toolchain is deprecated upstream
- rpi3-2.default.Functional.fuego_test_variables