Release 1.4 testing - 2018-12-26 in 'raw' format
{{TableOfContents}}
Here are some notes on testing the 1.4 release (using fuego-1211-container,
with a rolling master branch through December 2018 and January 2019):
= to do =
- categorize all bugs
- likely location: core, test, board, toolchain, lab, unknown
- severity: high, medium, low
- time to fix: short, long
- priority: 5=high, 1=low, 0=defer
Resolution key:
- CANDIDATE - is a candidate for fixing this release
- RESEARCH - needs more research to categorize and prioritize
- TRANSIENT - problem went away with subsequent test and no change in Fuego
- LOCAL (lab,board) - problem was with setup, config, board status, board, or toolchain, and issue can be ignored
- IGNORE - ignore (not a problem)
- FIXED - problem fixed with change in Fuego
- DEFER - problem exists, but will not be fixed in this release
= uncategorized issues list =
- 1. FIXED (core) '''bbb.default.Benchmark.Interbench''' - timeout is reported as PASS in Jenkins
- see build_number 1 - test hung and it took 40 minutes
- 2. DEFER (test) LTP runs for too long with nothing showing on the console. Jenkins should show the current testcase being run, or something.
- would require output from fuego_test.sh
- 3. FIXED (core) '''min1.default.Benchmark.OpenSSL''' - took 15 minutes (which timed out) but is reported as PASS in Jenkins
- 4. DEFER (core) Jenkins doesn't show different color for ERROR (timeout or abort)
- 5. FIXED (core) main.sh may return success even when the test was interrupted
- e.g. when terminated with SIGTERM, to allow post-test to run and results are only partial
- 6. DEFER (core) '''bbb.default.Benchmark.cyclictest''' it's not easy to find the threshold value. It's not in run.json. It's on the chart, but with multiple boards it's hard to isolate and find
- could add checkboxes for series.
- could add reference value to run.json
- could copy reference.log or reference.json to log directory
- could have parser add reference check failure to console log
- 7. DEFER (core) '''ftc rm-jobs rpi3-1.*.*.*''' - can't do rm-jobs outside the container! (annoying, but whatever)
- required due to possible interactive confirmation
- should be able to rm-jobs without being inside the container
- should be able to do interactive commands even when forwarding to the container ftc
- 8. FIXED (core) '''min1.default.Benchmark.OpenSSL''' - has timeout of 15 minutes (should be default of 40, from what I can see)
- was from 15m default timeout in testplan_smoketest.json
- 9. RESEARCH (test) '''min1.default.Functional.LTP''' - new failure for open10
- 10. RESEARCH (test)''' min1.default.Functional.LTP''' - new failure for creat08
- 11. DEFER (core?) need to have test indicate whether error code indicates error or failure
- example: x11perf - non-zero = error (not failure)
- 12. FIXED (core) it was hard to add a default timeout to testplan_fuego_tests, because I couldn't find the schema for testplans very easily. I didn't know the variable name.
- added wiki page [[Testplan Reference]]
- 13. FIXED (core) testplans usb, mmc and sata could have used 'default_spec' to save lines
- 14. FIXED (core) '''fuego-test.default.Functional.fuego_dependencies'''
- testcase 5 NEED_KCONFIG=CONFIG_PRINTK=y
- problem finding /boot/config-$(uname -r) for need_KCONFIG
- 15. FIXED (core) '''ren1.default.Functional.fuego_test_minimal'''
- fails because of missing spec.json in ftc
- 16. CANDIDATE, HIGH (core) '''fuego-test.default.Functional.fuego_release_test'''
- problem with Python 3 dependencies in docker project (requests requires urllib3, which is missing dependencies?)
{{{#!YellowBox
= ==== doing fuego phase: run =====
-------------------------------------------------
/usr/local/lib/python3.4/dist-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.9.1) or chardet (3.0.4) doesn't match a supported version!
RequestsDependencyWarning)
Traceback (most recent call last):
File "/fuego-core/engine/tests/Functional.fuego_release_test/test_run.py", line 15, in <module>
import docker
File "/usr/local/lib/python3.4/dist-packages/docker/__init__.py", line 2, in <module>
from .api import APIClient
File "/usr/local/lib/python3.4/dist-packages/docker/api/__init__.py", line 2, in <module>
from .client import APIClient
File "/usr/local/lib/python3.4/dist-packages/docker/api/client.py", line 5, in <module>
import requests
File "/usr/local/lib/python3.4/dist-packages/requests/__init__.py", line 105, in <module>
from urllib3.exceptions import DependencyWarning
ImportError: cannot import name 'DependencyWarning'
}}}
- 16. RESEARCH (unknown) '''rpi3-2.default.Functional.fuego_test_variables'''
- testcases 2,3, and 9
- 17. DEFER (board) '''rpi3-2.default.Functional.fuego_dependencies''' - - can't find kconfig
- missing /proc/config.gz or /boot/config-4.9.41-v7+ (but has /lib/modules.../configs.ko)
- 18. DEFER (test) '''rpi3-2.default.Functional.fuego_sleep''' - timed out
- timeout from testplan_fuego_tests is too short
- 19. DEFER (core) test that fails pre-check is shown as 'not run' in Jenkins
- 20. DEFER (core) test that fails pre-check has no result code (listed as None) in 'ftc gen-report'
- 21. FIXED (core) '''ren1.default.Functional.aiostress''' - (post-processing) Invalid literal for base 10 in generic_parser
- changed to a try here, and convert to ERROR in except clause
- or figure out why RETURN_VALUE is missing!!
- it's missing because:
- the build error caused early exit
- RETURN_VALUE was never set (by report, et al.)
- this test doesn't have a test_processing function
{{{#!YellowBox
##### doing fuego phase: processing (from signal handler) #####
WARNING: program returned exit code
Log evaluation may be invalid
Traceback (most recent call last):
File "/fuego-core/engine/scripts/generic_parser.py", line 18, in <module>
value = "PASS" if int(RETURN_VALUE) == 0 else "FAIL"
ValueError: invalid literal for int() with base 10: ''
ERROR: results did not satisfy the threshold
}}}
- 22. DEFER (test) '''ren1.default.Functional.LTP_one_test.syscalls.add_key_02''' - timeout is not converted to Fuego ERROR
- the LTP parser should convert a timed-out test in LTP into a Fuego ERROR in the run.json file. It doesn't do this.
- 21. FIXED (core) - '''ftc rm-node rpi3-1''' - cannot remove node rpi3-1
- get message "global name 'install_nodes' is not defined"
- 23. FIXED (core) - '''ftc rm-node rpi3-1''' - has no effect
- 24. FIXED (core) - '''ftc rm-jobs --remove-logs Bench.default.Benchmark.hackbench'''
- gives weird message
= Test failures by board =
Here are testplan_default failures for 2018-12-26
== bbb ==
Summary:
{{{
# ftc list-runs --where "start_time>4 days ago,status!=PASS,board=bbb"
Runs on this host:
Test Started host:board Status
--------------------------- ---------------------- ------------------------------ ------
Benchmark.Interbench 2018-12-26_18:00:01 timslab:bbb ERROR
Benchmark.OpenSSL 2018-12-26_18:50:36 timslab:bbb SKIP
Benchmark.cyclictest 2018-12-26_18:51:25 timslab:bbb FAIL
Benchmark.gtkperf 2018-12-26_19:01:27 timslab:bbb SKIP
Benchmark.x11perf 2018-12-26_18:49:56 timslab:bbb SKIP
Functional.LTP 2018-12-26_19:07:47 timslab:bbb FAIL
Functional.LTP 2018-12-26_21:20:47 timslab:bbb FAIL
Functional.OpenSSL 2018-12-26_18:56:31 timslab:bbb FAIL
Functional.aiostress 2018-12-26_23:06:06 timslab:bbb FAIL
Functional.fontconfig 2018-12-26_19:01:14 timslab:bbb FAIL
Functional.ft2demos 2018-12-26_19:00:25 timslab:bbb ERROR
}}}
Analysis:
- FIXED (test) bbb.default.Benchmark.Interbench - ERROR - timeout of 40m reported as SUCCESS
- reported as success in Jenkins and ERROR in run.json
- DEFER (test, board) bbb.default.Benchmark.Interbench - ERROR - test hangs!! (not new)
- hangs in Audio test (not new, but should be resolved)
{{{#!YellowBox
last data reported from test is:
--- Benchmarking simulated cpu of Audio in the presence of simulated ---
Load Latency +/- SD (ms) Max Latency % Desired CPU % Deadlines Met
None 0.1 +/- 0.1 0.3 100 100
Video 0.1 +/- 0.1 0.2 100 100
X 0.1 +/- 0.1 1.3 100 100
Burn 0.0 +/- 0.0 0.1 100 100
Write
calloc: Cannot allocate memory
0.1 +/- 0.3 6.6 100 100
}}}
- DEFER (test,toolchain) '''bbb.default.Benchmark.OpenSSL''' - build failed - missing zconf.h from zlib (not new)
{{{#!YellowBox
arm-linux-gnueabihf-gcc -I.. -I../.. -I../asn1 -I../evp -I../../include \
--sysroot=/ -DZLIB_SHARED -DZLIB -DOPENSSL_THREADS -c -o c_zlib.o c_zlib.c
In file included from c_zlib.c:25:0:
/usr/include/zlib.h:34:19: fatal error: zconf.h: No such file or directory
#include "zconf.h"
^
compilation terminated.
}}}
- CANDIDATE (board) '''bbb.default.Benchmark.cyclictest''' - FAIL - board doesn't have RT kernel (expected)
- max_latency doesn't pass threshold test, for 1000ms reference value
- fix would be to provide threshold for non-RT board
- add bbb.Benchmark.cyclictest.criteria.json
- DEFER (test?,toolchain?) '''bbb.default.Benchmark.gtkperf''' - build failure (not new)
- build failed - can't build test
- last message: checking for gtk+-2.0 gdk-2.0
- DEFER (toolchain) '''bbb.default.Benchmark.x11perf''' - build failure
- build failed - cannot find -lXext -lXrender -lX11 -lXmu
- DEFER (test) '''bbb.default.Functional.glib''' - build failure - swp in Thumb mode not supported (not new)
- error was:
{{{#!YellowBox
/bin/sh ../libtool --tag=CC --mode=compile arm-linux-gnueabihf-gcc -DHAVE_CONFIG_H -I. -I.. \
-I.. -DG_LOG_DOMAIN=\"GLib\" -DG_DISABLE_CAST_CHECKS -DG_DISABLE_DEPRECATED -DGLIB_COMPILATION \
-DPCRE_STATIC -DG_DISABLE_SINGLE_INCLUDES -pthread -g -O2 -Wall -MT gatomic.lo -MD -MP \
-MF .deps/gatomic.Tpo -c -o gatomic.lo gatomic.c
libtool: compile: arm-linux-gnueabihf-gcc -DHAVE_CONFIG_H -I. -I.. -I.. -DG_LOG_DOMAIN=\"GLib\" \
-DG_DISABLE_CAST_CHECKS -DG_DISABLE_DEPRECATED -DGLIB_COMPILATION -DPCRE_STATIC \
-DG_DISABLE_SINGLE_INCLUDES -pthread -g -O2 -Wall -MT gatomic.lo -MD -MP -MF .deps/gatomic.Tpo \
-c gatomic.c -fPIC -DPIC -o .libs/gatomic.o
/tmp/ccTcovfi.s: Assembler messages:
/tmp/ccTcovfi.s:65: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
/tmp/ccTcovfi.s:85: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
/tmp/ccTcovfi.s:188: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
/tmp/ccTcovfi.s:208: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
/tmp/ccTcovfi.s:310: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
/tmp/ccTcovfi.s:330: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
/tmp/ccTcovfi.s:432: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
/tmp/ccTcovfi.s:452: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
Makefile:994: recipe for target 'gatomic.lo' failed
}}}
- RESEARCH (test) '''bbb.default.Functional.LTP''' - FAIL - see details below
- DEFER (test) '''bbb.default.Functional.OpenSSL''' - FAIL - missing zconf.h (not new)
- FIXED (board) '''bbb.default.Functional.aiostress''' - FAIL - can't load libaio.so.1 on board (not new)
- should be reported as ERROR (failure to run test program)
- FIXED (board) '''bbb.default.Functional.aiostress''' - FAIL - can't load libaio.so.1 on board (not new)
- should use dependency check for package libaio (not sure it will work with non-Debian)
- should resolve with a dependency on libaio.so.1 on the board
- does this test apply, if libaio.so is not present on the board?? - NO
- libaio.so.1 is located in bbb:/lib/arm-linux-gnueabihf/libaio.so.1
- /lib/arm-linux-gnueabihf system shared lib path is in: /etc/ld.so.conf.d/arm-linux-gneuabihf.conf
- can't build without libaio-dev in the SDK (libaio.h)
- aio functions are supposed to be provided by C library
- removed '-laio' from build line
- can be resolved with 'apt-get install libaio1' on the board
- FIXED (board) '''bbb.default.Functional.fontconfig''' - FAIL - missing fc-list (expected)
- added assert_has_program to fuego_test.sh
- DEFER bbb.default.Functional.ft2demos - ERROR - missing X libs (not new)
{{{#!YellowBox
libtool: link: arm-linux-gnueabihf-gcc -Wl,-L -Wl,//usr/lib -Wl,-lXau -Wl,-lXdmcp --sysroot / \
-o /fuego-rw/buildzone/bbb.default.Functional.ft2demos-debian-armhf/ft2demos-2.3.6/bin/.libs/ftlint \
/fuego-rw/buildzone/bbb.default.Functional.ft2demos-debian-armhf/ft2demos-2.3.6/obj/ftlint.o \
/fuego-rw/buildzone/bbb.default.Functional.ft2demos-debian-armhf/ft2demos-2.3.6/obj/common.o \
../freetype-2.3.6/objs/.libs/libfreetype.so -lm
/usr/lib/gcc-cross/arm-linux-gnueabihf/6/../../../../arm-linux-gnueabihf/bin/ld: cannot find -lXau
/usr/lib/gcc-cross/arm-linux-gnueabihf/6/../../../../arm-linux-gnueabihf/bin/ld: cannot find -lXdmcp
collect2: error: ld returned 1 exit status
Makefile:426: recipe for target '/fuego-rw/buildzone/bbb.default.Functional.ft2demos-debian-armhf/ft2demos-2.3.6/bin/ftlint' failed
make: *** [/fuego-rw/buildzone/bbb.default.Functional.ft2demos-debian-armhf/ft2demos-2.3.6/bin/ftlint] Error 1
}}}
- DEFER (test, toolchain) - '''bbb.default.Functional.glib''' - problem with thumb mode instructions
{{{#!YellowBox
libtool: compile: arm-linux-gnueabihf-gcc -DHAVE_CONFIG_H -I. -I.. -I.. \
-DG_LOG_DOMAIN=\"GLib\" -DG_DISABLE_CAST_CHECKS -DG_DISABLE_DEPRECATED -DGLIB_COMPILATION \
-DPCRE_STATIC -DG_DISABLE_SINGLE_INCLUDES -pthread -g -O2 -Wall -MT gatomic.lo -MD -MP \
-MF .deps/gatomic.Tpo -c gatomic.c -fPIC -DPIC -o .libs/gatomic.o
/tmp/ccTcovfi.s: Assembler messages:
/tmp/ccTcovfi.s:65: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
/tmp/ccTcovfi.s:85: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
/tmp/ccTcovfi.s:188: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
/tmp/ccTcovfi.s:208: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
/tmp/ccTcovfi.s:310: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
/tmp/ccTcovfi.s:330: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
/tmp/ccTcovfi.s:432: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
/tmp/ccTcovfi.s:452: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
Makefile:994: recipe for target 'gatomic.lo' failed
make[4]: *** [gatomic.lo] Error 1
}}}
=== bbb LTP results ===
{{{
# ftc gen-report --where "board=bbb,test=LTP,tguid:result!=PASS,start_time>9 days ago" | sort -k 5
board : bbb
kernel : 4.4.88-ti-r125
report_date : 2019-01-04_20:20:37
test : LTP
timestamp : 2018-12-26_21:20:47
--------------------------------------------------------------------------------------------------------------------------------
======================================================================
test_name spec board timestamp tguid tguid:result
Functional.LTP default bbb 2018-12-26_21:20:47 sem_init.sem_init_7-1 SKIP
Functional.LTP default bbb 2018-12-26_21:20:47 syscalls FAIL
Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.add_key03 FAIL
Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.add_key04 FAIL
Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.eventfd01 FAIL
Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.fanotify06 FAIL
Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.fcntl35 FAIL
Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.fcntl35_64 FAIL
Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.getrusage03 FAIL
Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.io_cancel01 FAIL
Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.io_destroy01 FAIL
Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.io_getevents01 FAIL
Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.io_setup01 FAIL
Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.io_submit01 FAIL
Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.keyctl02 FAIL
Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.keyctl03 FAIL
Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.keyctl04 FAIL
Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.keyctl05 FAIL
Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.keyctl06 FAIL
Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.keyctl07 FAIL
Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.msgctl10 FAIL
Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.msgctl11 FAIL
Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.request_key01 FAIL
Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.request_key02 FAIL
Functional.LTP default bbb 2018-12-26_21:20:47 syscalls.request_key03 FAIL
}}}
== min1 ==
Summary:
{{{
$ ftc list-runs --where "start_time>yesterday,status!=PASS,board=min1"
Test Started host:board Status
Benchmark.Java 2018-12-26_18:56:19 timslab:min1 ERROR
Benchmark.OpenSSL 2018-12-26_19:36:18 timslab:min1 ERROR
Benchmark.cyclictest 2018-12-26_19:51:52 timslab:min1 FAIL
Benchmark.signaltest 2018-12-26_19:55:13 timslab:min1 FAIL
Functional.LTP 2018-12-26_18:00:42 timslab:min1 FAIL
Functional.glib 2018-12-26_18:57:06 timslab:min1 FAIL
Benchmark.GLMark 2018-12-26_18:55:04 timslab:min1 SKIP
Benchmark.x11perf 2018-12-26_18:50:09 timslab:min1 SKIP
--------------------------- ---------------------- ------------------------------ ------
}}}
Analysis:
- DEFER (toolchain) '''min1.default.Benchmark.GLMark''' - ERROR - failed to build
- missing glew.h file for build
- should have been reported as an error
- FIXED (test) '''min1.default.Benchmark.Java''' - ERROR - (board does not have java command (expected)
- changed to 'assert_has_program java'
- FIXED (core) '''min1.default.Benchmark.OpenSSL''' - interrupted at 15 minutes (according to timeout value), returned SUCCESS to Jenkins
- changed so that ERROR is returned on timeouts
- need to change timeout higher
- CANDIDATE (board) '''min1.default.Benchmark.cyclictest''' - FAIL (board does not have RT kernel) (expected)
- max latency over threshold
- should be higher threshold?
- add non-RT threshold??
- CANDIDATE (board) '''min1.default.Benchmark.signaltest''' - FAIL (board does not have RT kernel) (expected)
- max latency over threshold
- should be higher threshold?
- DEFER (test?) '''min1.default.Benchmark.x11perf''' - SKIP
- x11perf: unable to open display ':0"
- should have been reported as an error
- RESEARCH (test) '''min1.default.Functional.LTP''' - FAIL - 10 failures in syscalls, 1 sem_init SKIP (was 9 failures in 1.3 - see details below)
- DEFER (test?) '''min1.default.Functional.glib''' - FAIL
- 'threadtests' does not report pass or failure. Test appears to terminate prematurely? (see testlog.txt)
- there are many GLib-CRITICAL reports for "Source ID ## was not found when attempting to remove it"
- returns exit code 143
=== min1 LTP results ===
{{{
# ftc gen-report --where "board=min1,test=LTP,tguid:result!=PASS"
======================================================================
**** Fuego Test Report ****
test : LTP
board : min1
kernel : 4.15.0-33-generic
timestamp : 2018-12-26_18:00:42
report_date : 2018-12-28_19:42:54
======================================================================
--------------------------------------------------------------------------------------------------------------------------------
test_name spec board timestamp tguid tguid:result
--------------------------------------------------------------------------------------------------------------------------------
Functional.LTP default min1 2018-12-26_18:00:42 syscalls FAIL
Functional.LTP default min1 2018-12-26_18:00:42 syscalls.io_getevents01 FAIL
Functional.LTP default min1 2018-12-26_18:00:42 syscalls.fanotify06 FAIL
Functional.LTP default min1 2018-12-26_18:00:42 syscalls.io_submit01 FAIL
Functional.LTP default min1 2018-12-26_18:00:42 syscalls.io_cancel01 FAIL
Functional.LTP default min1 2018-12-26_18:00:42 syscalls.msgctl11 FAIL
Functional.LTP default min1 2018-12-26_18:00:42 syscalls.eventfd01 FAIL
Functional.LTP default min1 2018-12-26_18:00:42 syscalls.open10 FAIL
Functional.LTP default min1 2018-12-26_18:00:42 syscalls.creat08 FAIL
Functional.LTP default min1 2018-12-26_18:00:42 syscalls.io_setup01 FAIL
Functional.LTP default min1 2018-12-26_18:00:42 syscalls.io_destroy01 FAIL
Functional.LTP default min1 2018-12-26_18:00:42 sem_init.sem_init_7-1 SKIP
--------------------------------------------------------------------------------------------------------------------------------
Different from 1.3:
* RESEARCH min1.default.Functional.LTP.syscalls.open10 - FAIL
* RESEARCH min1.default.Functional.LTP.syscalls.creat08 - FAIL
}}}
test that were run in 1.3 but not 1.4 release testing:
- min1.default.Functional.aiostress
- min1.default.Functional.kernel_build
== ren1 ==
Summary:
{{{
# ftc list-runs --where "board=ren1,start_time>yesterday,status!=PASS"
Runs on this host:
Test Started host:board Status
--------------------------- ---------------------- ------------------------------ ------
Benchmark.Dhrystone 2018-12-28_21:11:40 timslab:ren1 FAIL
Benchmark.GLMark 2018-12-28_21:34:14 timslab:ren1 SKIP
Benchmark.Java 2018-12-28_21:38:54 timslab:ren1 ERROR
Benchmark.gtkperf 2018-12-28_21:09:41 timslab:ren1 SKIP
Benchmark.signaltest 2018-12-28_21:10:53 timslab:ren1 FAIL
Benchmark.x11perf 2018-12-28_21:58:42 timslab:ren1 SKIP
Functional.LTP 2018-12-28_21:58:50 timslab:ren1 FAIL
Functional.aiostress 2018-12-28_21:34:10 timslab:ren1 ERROR
Functional.crashme 2018-12-28_21:31:30 timslab:ren1 FAIL
Functional.ft2demos 2018-12-28_20:47:19 timslab:ren1 ERROR
Functional.glib 2018-12-28_20:45:28 timslab:ren1 FAIL
}}}
Analysis:
- DEFER (test, board) '''ren1.default.Benchmark.Dhrystone''' - FAIL (needs 100M spec)
- DEFER (test, toolchain) '''ren1.default.Benchmark.GLMark''' - SKIP (build failure - missing SDL.h)
- FIXED (board) '''ren1.default.Benchmark.Java''' - ERROR (missing 'java' on board) (expected)
- DEFER '''ren1.default.Benchmark.gtkperf''' - SKIP
- (deploy error) sh: gdk-pixbuf-query-loaders: command not found
- DEFER '''ren1.default.Benchmark.signaltest''' - FAIL
- DEFER '''ren1.default.Benchmark.x11perf''' - SKIP
- RESEARCH '''ren1.default.Functional.LTP''' - FAIL - see details below
- DEFER '''ren1.default.Functional.aiostress''' - ERROR (build error) (not new)
- aiostress.c:58:20: fatal error: libaio.h: No such file or directory compilation terminated.
- DEFER '''ren1.default.Functional.crashme''' - FAIL (not new)
- results did not satisfy the threshold
- log_compare "0 ... 3000" (but test result was "0 ... 507")
- FIXED '''ren1.default.Functional.ft2demos''' - ERROR (build error)
- /fuego-core/engine/tests/Functional.ft2demos/fuego_test.sh: line 16: build_error: command not found
- DEFER '''ren1.default.Functional.ft2demos''' - ERROR
{{{#!YellowBox
- libtool: link: cannot find the library `=/usr/lib/libxcb.la' or unhandled argument `=/usr/lib/libxcb.la'
Makefile:470: recipe for target '/fuego-rw/buildzone/ren1.default.Functional.ft2demos-poky-aarch64/ft2demos-2.3.6/bin/ftview' failed
- make: *** [/fuego-rw/buildzone/ren1.default.Functional.ft2demos-poky-aarch64/ft2demos-2.3.6/bin/ftview] Error 1
}}}
- DEFER '''ren1.default.Functional.glib''' - FAIL (build error) (not new, same as 1.3)
- 'threadtests' does not report pass or failure. Test appears to terminate prematurely? (see testlog.txt)
- there are many GLib-CRITICAL reports for "Source ID ## was not found when attempting to remove it"
- returns exit code 143
- RESEARCH '''ren1.default.Functional.kernel_build''' - FAIL, build error on assembly instruction
- problem with head.S assembly and poky arm64 toolchain (as)
{{{#!YellowBox
AS arch/arm64/kernel/head.o
arch/arm64/kernel/head.S: Assembler messages:
arch/arm64/kernel/head.S:824: Error: missing ')'
arch/arm64/kernel/head.S:824: Error: missing ')'
arch/arm64/kernel/head.S:824: Error: missing ')'
arch/arm64/kernel/head.S:824: Error: unexpected characters following instruction at operand 2 -- `mov x2,#(2)|(2U<<(8))'
scripts/Makefile.build:391: recipe for target 'arch/arm64/kernel/head.o' failed
make[1]: *** [arch/arm64/kernel/head.o] Error 1
Makefile:1055: recipe for target 'arch/arm64/kernel' failed
make: *** [arch/arm64/kernel] Error 2
}}}
=== ren1 LTP results ===
{{{
# ftc gen-report --where "board=ren1,test=LTP,tguid:result!=PASS"
======================================================================
**** Fuego Test Report ****
test : LTP
board : ren1
kernel : 4.9.0-yocto-standard
timestamp : 2018-12-26_21:38:12
report_date : 2018-12-28_23:12:46
======================================================================
------- - (had an error about missing program 'build_error', but that was fixed:
- /fuego-core/engine/tests/Functional.ft2demos/fuego_test.sh: line 16: build_error: command not found)
-------------------------------------------------------------------------------------------------------------------------
test_name spec board timestamp tguid tguid:result
--------------------------------------------------------------------------------------------------------------------------------
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.cve-2017-6951 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.prot_hsymlinks FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.renameat202 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.add_key04 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.add_key02 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.keyctl08 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.keyctl03 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.keyctl02 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.keyctl05 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.keyctl04 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.keyctl07 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.keyctl06 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.msgctl09 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.msgctl08 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.fanotify06 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.utime02 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.mknod03 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.abort01 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.recvmsg01 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.kill11 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.msgctl10 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.msgctl11 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.request_key02 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.open10 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.renameat201 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.creat08 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.add_key03 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.setsockopt02 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.cve-2017-5669 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.sendmsg01 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.request_key01 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.request_key03 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.setpriority01 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 syscalls.utime01 FAIL
Functional.LTP default ren1 2018-12-26_21:38:12 sem_init.sem_init_7-1 SKIP
--------------------------------------------------------------------------------------------------------------------------------
}}}
Changed from 1.3 release testing were:
- DEFER ren1.default.Functional.LTP.syscalls.add_key02 - failed in 1.4
- IGNORE ren1.default.Functional.LTP.syscalls.fanotify08 - worked now (but failed in 1.3)
- IGNORE ren1.default.Functional.LTP.syscalls.fanotify07 - worked now (but failed in 1.3)
- IGNORE ren1.default.Functional.LTP.syscalls.perf_event_open02 - worked now (but failed in 1.3)
- RESEARCH ren1.default.Functional.LTP.syscalls.mknod03 - failed in 1.4
- RESEARCH ren1.default.Functional.LTP.syscalls.creat08 - failed in 1.4
- DEFER, HIGH (post 1.4) ren1.default.Functional.LTP.syscalls.x (for x in: add_key0[1234], keyctl0[2345678], request_key0[123]) (14 tests)
- add these tests to 'should reboot after test' list for LTP
- these are probably xfail?? for creating zombie processes
Also, during the test, several processes (add_key, keyctl, request_key) were stuck in uninterruptible sleep:
{{{
$ ps auxwww (trimmed)
24350 root 2860 S /usr/sbin/dropbear -i -r /etc/dropbear/dropbear_rsa_host_key -B
24351 root 2908 S sh -c { cd /home/fuego/fuego.Functional.LTP; TESTS="syscalls "; PTSTESTS="SEM "; \
RTTESTS=""; . ./ltp_target_run.sh; echo $? > /tmp/22722-26579; } 2>&1 | tee /home/fuego/fuego.Functional.LTP/Functional.LTP.log
24352 root 2908 S sh -c { cd /home/fuego/fuego.Functional.LTP; TESTS="syscalls "; PTSTESTS="SEM "; \
RTTESTS=""; . ./ltp_target_run.sh; echo $? > /tmp/22722-26579; } 2>&1 | tee /home/fuego/fuego.Functional.LTP/Functional.LTP.log
24353 root 1760 S tee /home/fuego/fuego.Functional.LTP/Functional.LTP.log
24377 root 3568 S {runltp} /bin/sh ./runltp -C /home/fuego/fuego.Functional.LTP/result/syscalls/failed.log \
-l /home/fuego/fuego.Functional.LTP/result/syscalls/result.log \
-o /home/fuego/fuego.Functional.LTP/result/syscalls/output.log \
-d /home/fuego/fuego.Functional.LTP/tmp -S ./skiplist.txt -f syscalls
24548 root 1888 S /home/fuego/fuego.Functional.LTP/bin/ltp-pan -e -S -a 24377 -n 24377 \
-f /home/fuego/fuego.Functional.LTP/tmp/ltp-T8KMbYu8JQ/alltests \
-l /home/fuego/fuego.Functional.LTP/result/syscalls/result.log \
-o /home/fuego/fuego.Functional.LTP/result/syscalls/output.log \
-C /home/fuego/fuego.Functional.LTP/result/syscalls/failed.log \
-T /home/fuego/fuego.Functional.LTP/output/LTP_RUN_ON-output.log.tconf
24692 root 1820 D add_key01
24694 root 1820 D add_key02
24696 root 2296 D add_key03
24698 root 1824 D add_key04
24753 root 0 SW [kworker/u12:2]
26884 root 2004 D keyctl02
26886 root 1820 D keyctl03
26888 root 1820 D keyctl04
26890 root 1824 D keyctl05
26892 root 1820 D keyctl06
26895 root 1824 D keyctl07
26897 root 1820 D keyctl08
27708 root 1820 D request_key01
27710 root 1820 D request_key02
27712 root 1824 D request_key03
27714 root 1820 D cve-2017-6951
32620 root 0 SW [kworker/u12:0]
}}}
== rpi3-2 ==
Summary:
{{{
cascade caused problems
}}}
Analysis:
- DEFER '''rpi3-2.default.Benchmark.GLMark''' - SKIP - build error, missing glew.h
- compilation problem with cpp code - missing GL header
{{{
n file included from vector.h:4:0,
from vector.cpp:1:
oglsdl.h:5:21: fatal error: GL/glew.h: No such file or directory
#include <GL/glew.h>
^
compilation terminated.
}}}
- DEFER - '''rpi3-2.default.Benchmark.Interbench''' - times out (hangs)
- hangs in Audio test (well, I stopped it)
- DEFER - '''rpi3-2.default.Benchmark.bonnie''' - FAIL (processing failure)
- bug in parser for invalid values (probably ++++)
{{{
'===== doing fuego phase: processing =====
Reading current values from /fuego-rw/logs/Benchmark.bonnie/rpi3-2.default.1.1/testlog.txt
Bonnie++ raw results: ['1.97', '1.97', 'rpi3-2', '1', '1546056554', '32M', '', '126', '98', '9574', '6',
'9197', '4', '613', '99', '+++++', '+++', '357.2', '5', '16', '', '', '', '', '4154', '19', '+++++', '+++',
'4582', '29', '3820', '17', '+++++', '+++', '7129', '45', '120ms', '344us', '568us', '18744us', '108us',
'20117us', '59893us', '3679us', '5008us', '566us', '175us', '895us']
Traceback (most recent call last):
File "/fuego-core/engine/tests/Benchmark.bonnie/parser.py", line 36, in <module>
measurements["Sequential_Output.PerChr"].append({"name": "speed", "measure" : float(results[2])})
ValueError: could not convert string to float: rpi3-2
ERROR: results did not satisfy the threshold
}}}
- DEFER '''rpi3-2.default.Benchmark.gtkperf''' - SKIP build error
- problem in configure - checking for gtk+-2.0 gdk-2.0 ..
- DEFER '''rpi3-2.default.Functional.glib''' - ERROR - build error
- compiling gatomic, swp instructions in Thumb mode
- FIXED (board, spec?) - '''rpi3-2.default.Functional.kernel_build''' - ERROR (no bzImage, but zImage is present)
- no bzImage, but zImage is present
- kernel make always defaults to building 'zImage' if the target is not recognized 'make foo' makes a zImage.
- fixed by adding FUNCTIONAL_KERNEL_BUILD_TARGET="zImage" to rpi3-2.board
- this is fragile, as the spec value overrides 'target'. But by default no 'target' is specified.
- would be better to use "TARGET_KERNEL_IMAGE" in board, and if specified, override the target from the spec?
- FIXED (core) - '''rpi3-2.default.Functional.kernel_build''' -
- got this this from generic_parser.py:
{{{#!YellowBox
#### doing fuego phase: processing (from signal handler) #####
Processing kernel build log
log_compare: pattern '[ \t]*Kernel: arch/. arch/.. arch/.gitignore is ready' \
found 2 times (expected greater or equal than 1)
Traceback (most recent call last):
File "/fuego-core/engine/scripts/generic_parser.py", line 18, in <module>
value = "PASS" if int(RETURN_VALUE) == 0 else "FAIL"
ValueError: invalid literal for int() with base 10: ''
}}}
= Overview =
{{{
root@tlinux:/# # ftc list-runs --where "start_time>yesterday,status!=PASS" | sort -k 3
Test Started host:board Status
Benchmark.Interbench 2018-12-26_18:00:01 timslab:bbb ERROR
Functional.LTP 2018-12-26_21:20:47 timslab:bbb ERROR
Functional.ft2demos 2018-12-26_19:00:25 timslab:bbb ERROR
Benchmark.cyclictest 2018-12-26_18:51:25 timslab:bbb FAIL
Functional.LTP 2018-12-26_19:07:47 timslab:bbb FAIL
Functional.OpenSSL 2018-12-26_18:56:31 timslab:bbb FAIL
Functional.fontconfig 2018-12-26_19:01:14 timslab:bbb FAIL
Benchmark.OpenSSL 2018-12-26_18:50:36 timslab:bbb SKIP
Benchmark.gtkperf 2018-12-26_19:01:27 timslab:bbb SKIP
Benchmark.x11perf 2018-12-26_18:49:56 timslab:bbb SKIP
Benchmark.Java 2018-12-26_18:56:19 timslab:min1 ERROR
Benchmark.OpenSSL 2018-12-26_19:36:18 timslab:min1 ERROR
Benchmark.cyclictest 2018-12-26_19:51:52 timslab:min1 FAIL
Benchmark.signaltest 2018-12-26_19:55:13 timslab:min1 FAIL
Functional.LTP 2018-12-26_18:00:42 timslab:min1 FAIL
Functional.glib 2018-12-26_18:57:06 timslab:min1 FAIL
Benchmark.GLMark 2018-12-26_18:55:04 timslab:min1 SKIP
Benchmark.x11perf 2018-12-26_18:50:09 timslab:min1 SKIP
Functional.LTP 2018-12-26_21:38:12 timslab:ren1 ERROR
Functional.netperf 2018-12-26_18:16:13 timslab:ren1 ERROR
Benchmark.signaltest 2018-12-26_18:15:59 timslab:ren1 FAIL
Functional.glib 2018-12-26_18:33:38 timslab:ren1 FAIL
Benchmark.Whetstone 2018-12-26_19:09:31 timslab:rpi3-2 ERROR
Benchmark.tiobench 2018-12-26_19:08:28 timslab:rpi3-2 ERROR
Functional.ft2demos 2018-12-26_18:50:43 timslab:rpi3-2 ERROR
Functional.kernel_build 2018-12-26_18:38:15 timslab:rpi3-2 ERROR
Functional.linus_stress 2018-12-26_18:43:44 timslab:rpi3-2 ERROR
Benchmark.signaltest 2018-12-26_19:40:41 timslab:rpi3-2 FAIL
Benchmark.GLMark 2018-12-26_19:35:22 timslab:rpi3-2 SKIP
Benchmark.OpenSSL 2018-12-26_19:17:55 timslab:rpi3-2 SKIP
--------------------------- ---------------------- ------------------------------ ------
}}}
== rpi3-1 LTP testcase problems ==
FIXTHIS - these are from 1.3 release
{{{
root@tlinux:/# ftc gen-report --where "board=rpi3-1,test=LTP,tguid:result!=PASS,start_time>2 days ago"
where start_time value is '1.525060103e+12'
======================================================================
**** Fuego Test Report ****
test : LTP
board : rpi3-1
kernel : 4.9.41-v7+
timestamp : 2018-05-01_01:02:42
report_date : 2018-05-02_03:48:23
======================================================================
--------------------------------------------------------------------------------------------------------------------------------
test_name spec board timestamp tguid tguid:result
--------------------------------------------------------------------------------------------------------------------------------
Functional.LTP default rpi3-1 2018-05-01_01:02:42 syscalls FAIL
Functional.LTP default rpi3-1 2018-05-01_01:02:42 syscalls.add_key04 FAIL
Functional.LTP default rpi3-1 2018-05-01_01:02:42 syscalls.keyctl03 FAIL
Functional.LTP default rpi3-1 2018-05-01_01:02:42 syscalls.keyctl02 FAIL
Functional.LTP default rpi3-1 2018-05-01_01:02:42 syscalls.keyctl05 FAIL
Functional.LTP default rpi3-1 2018-05-01_01:02:42 syscalls.keyctl04 FAIL
Functional.LTP default rpi3-1 2018-05-01_01:02:42 syscalls.keyctl07 FAIL
Functional.LTP default rpi3-1 2018-05-01_01:02:42 syscalls.keyctl06 FAIL
Functional.LTP default rpi3-1 2018-05-01_01:02:42 syscalls.io_getevents01 FAIL
Functional.LTP default rpi3-1 2018-05-01_01:02:42 syscalls.io_submit01 FAIL
Functional.LTP default rpi3-1 2018-05-01_01:02:42 syscalls.fcntl36 FAIL
Functional.LTP default rpi3-1 2018-05-01_01:02:42 syscalls.io_cancel01 FAIL
Functional.LTP default rpi3-1 2018-05-01_01:02:42 syscalls.msgctl11 FAIL
Functional.LTP default rpi3-1 2018-05-01_01:02:42 syscalls.eventfd01 FAIL
Functional.LTP default rpi3-1 2018-05-01_01:02:42 syscalls.request_key02 FAIL
Functional.LTP default rpi3-1 2018-05-01_01:02:42 syscalls.io_setup01 FAIL
Functional.LTP default rpi3-1 2018-05-01_01:02:42 syscalls.io_destroy01 FAIL
Functional.LTP default rpi3-1 2018-05-01_01:02:42 syscalls.add_key03 FAIL
Functional.LTP default rpi3-1 2018-05-01_01:02:42 syscalls.request_key01 FAIL
Functional.LTP default rpi3-1 2018-05-01_01:02:42 syscalls.request_key03 FAIL
Functional.LTP default rpi3-1 2018-05-01_01:02:42 sem_init.sem_init_7-1 SKIP
--------------------------------------------------------------------------------------------------------------------------------
}}}
= table of current test status =
See [[Release_test_results]]
= triage data =
- candidates
- ren1.default.Functional.fuego_test_minimal
- missing spec.json
- fuego-test.default.Functional.fuego_release_test
- urllib3 python module dependency problem
- ren1.default.Functional.aiostress
- generic_parser python exception
- bbb.default.Benchmark.cyclictest (and min1, and signaltest)
- add board-specific, config-specific, distro-specific criteria
- rpi3-2.default.Functional.kernel_build
- wrong kernel image name
- bbb.default.Functional.aiostress
- can't load libaio.so.1
- maybe add dependency check on library (can you do aio without lib?)
- research
- rpi3-2.default.Functional.fuego_test_variables
- if requires fuego-test board, then add dependency check to test
- min1.default.Functional.LTP.syscalls.open10
- see if it's xfail (expected for this kernel release)
- add to board-specific skiplist?
- min1.default.Functional.LTP.syscalls.creat08
- ren1.default.Functional.kernel_build
- see if it's a real kernel bug that need a patch upstream
- depends on whether poky ren1 toolchain is deprecated upstream