|
Here are some notes on testing the 1.4 release (using fuego-1211-container,
with a rolling master branch through December 2018 and January 2019):
|
{{TableOfContents}}
Here are some notes on testing the 1.4 release (using fuego-1211-container,
with a rolling master branch through December 2018 and January 2019):
|
|
See an overview issues table at Release test results
|
See an overview issues table at [[Release test results]]
|
|
|
= to do =
- categorize all bugs
- likely location: core, test, board, toolchain, lab, unknown
- severity: high, medium, low
- time to fix: short, long
- priority: 5=high, 1=low, 0=defer
|
|
Resolution key:
- CANDIDATE - is a candidate for fixing this release
- RESEARCH - needs more research to categorize and prioritize
- TRANSIENT - problem went away with subsequent test and no change in Fuego
- LOCAL (lab,board) - problem was with setup, config, board status, board, or toolchain, and issue can be ignored
- IGNORE - ignore (not a problem)
- FIXED - problem fixed with change in Fuego
- DEFER - problem exists, but will not be fixed in this release
|
Resolution key:
- CANDIDATE - is a candidate for fixing this release
- RESEARCH - needs more research to categorize and prioritize
- TRANSIENT - problem went away with subsequent test and no change in Fuego
- LOCAL (lab,board) - problem was with setup, config, board status, board, or toolchain, and issue can be ignored
- IGNORE - ignore (not a problem)
- FIXED - problem fixed with change in Fuego
- DEFER - problem exists, but will not be fixed in this release
|
|
http://fuegotest.org/wiki/Release_1.5_testing_-_2019-05-02#deferred_from_1.4_release
|
= uncategorized issues list =
== deferred from 1.4 release ==
See http://fuegotest.org/wiki/Release_1.5_testing_-_2019-05-02#deferred_from_1.4_release
|
|
|
== new errors ==
- xx. (distro) '''docker.default.Functional.fuego_board_check''' - missing /sbin/reboot
- /sbin/reboot is present on host as a symlink to /bin/systemctl
- this is expected
- xx. Can't easily navigate to LTP individual test logs in Jenkins interface
|
|
Benchmark.GLMark - build failure on multiple platforms (missing glew.h, missing libSDL)
|
= old errors =
- [[Benchmark.GLMark]] - build failure on multiple platforms (missing glew.h, missing libSDL)
|
|
|
= Errors by board =
|
|
|
== bbb board ==
- (old) '''bbb.default.Benchmark.GLMark''' - missing GL/glew.h
- (old) '''bbb.default.Benchmark.Interbench''' - run error: hangs in Audio test
- (old) '''bbb.default.Benchmark.gtkperf''' - build error: configure error: checking for gtk+-2.0 gdk-2.0
- (old) '''bbb.default.Benchmark.X11perf''' - missing X libs
- (RESEARCH) (old) '''bbb.default.Functional.glib''' - missing gettext lib
|
|
# ftc list-runs --where "start_time>4 days ago,status!=PASS,board=bbb"
|
# ftc list-runs --where "start_time>4 days ago,status!=PASS,board=bbb"
|
Runs on this host:
Test Status
--------------------------- - ------
Benchmark.Dhrystone ERROR - fixed
Benchmark.GLMark SKIP - build error: GL/glew.h: No such file or directory
Benchmark.Interbench ERROR - run error: hangs in Audio test
Benchmark.Java SKIP - no java: OK
Benchmark.gtkperf SKIP - build error: configure error: checking for gtk+-2.0 gdk-2.0 ...
Benchmark.netpipe FAIL - numbers below thresholds (size-262144.throughput = 37.54 which is not "gt 400")
Benchmark.x11perf SKIP - missing X libs
Functional.LTP FAIL - multiple failures (hangs)
Functional.fontconfig SKIP - pre_check error: missing 'fc-list': OK
Functional.ft2demos SKIP - build error: cannot find Xau, Xdmcp libs
Functional.fuego_abort SKIP - run error: aborted = OK
Functional.fuego_compare_reports FAIL - not configured (missing baseline results file): OK
Functional.fuego_ftc_test FAIL - (not relevant to this board)
Functional.fuego_lint FAIL - (not relevant to this board)
Functional.fuego_release_test FAIL - (not relevant to this board)
Functional.fuego_slow_log FAIL - (not relevant to this board)
Functional.fuego_test_variables FAIL - (not relevant to this board)
Functional.glib SKIP - build error: missing gettext, configure error.
Functional.hello_world FAIL - expected (spec=fail)
}}}
|
{{{
Runs on this host:
Test Status
--------------------------- - ------
Benchmark.Dhrystone ERROR - fixed
Benchmark.GLMark SKIP - build error: GL/glew.h: No such file or directory
Benchmark.Interbench ERROR - run error: hangs in Audio test
Benchmark.Java SKIP - no java: OK
Benchmark.gtkperf SKIP - build error: configure error: checking for gtk+-2.0 gdk-2.0 ...
Benchmark.netpipe FAIL - numbers below thresholds (size-262144.throughput = 37.54 which is not "gt 400")
Benchmark.x11perf SKIP - missing X libs
Functional.LTP FAIL - multiple failures (hangs)
Functional.fontconfig SKIP - pre_check error: missing 'fc-list': OK
Functional.ft2demos SKIP - build error: cannot find Xau, Xdmcp libs
Functional.fuego_abort SKIP - run error: aborted = OK
Functional.fuego_compare_reports FAIL - not configured (missing baseline results file): OK
Functional.fuego_ftc_test FAIL - (not relevant to this board)
Functional.fuego_lint FAIL - (not relevant to this board)
Functional.fuego_release_test FAIL - (not relevant to this board)
Functional.fuego_slow_log FAIL - (not relevant to this board)
Functional.fuego_test_variables FAIL - (not relevant to this board)
Functional.glib SKIP - build error: missing gettext, configure error.
Functional.hello_world FAIL - expected (spec=fail)
}}}
|
|
|
== min1 board ==
Here are the tests that had issues:
{{{
# ftc list-runs --where board=min1,status!=PASS
Runs on this host:
Test Status
--------------------------- ------
Benchmark.GLMark SKIP - build errror, missing GL/glew.h (old)
Benchmark.cyclictest FAIL - non-rt kernel
Benchmark.dbench4 FAIL - (new) - failed to connect to board after reboot
Benchmark.gtkperf SKIP - build error: configure error: checking for gtk+-2.0 gdk-2.0 (old)
Benchmark.signaltest FAIL - non-rt kernel
Benchmark.x11perf SKIP - runtime: unable to open display ':0
Functional.LTP FAIL - see below
Functional.glib SKIP - build error: missing gettext support (RESEARCH)
}}}
|
|
----
|
|
Here are the LTP non-PASS results:
{{{
# ftc gen-report --where board=min1,test=LTP,tguid:result!=PASS
|
Here are the LTP non-PASS results:
{{{
# ftc gen-report --where board=min1,test=LTP,tguid:result!=PASS
|
|
}}}
|
}}}
|
|
---
Here is the reason for each LTP failure (with some comments)
{{{
------------------------------------
tguid tguid:result
------------------------------------
syscalls.fanotify06 FAIL - segfault
syscalls.msgctl11 FAIL - queue mismatch
syscalls.open10 FAIL - setgid not set
syscalls.creat08 FAIL - setgid not set
sem_init.sem_init_7-1 SKIP - ??? (RESEARCH)
------------------------------------
}}}
|
---
Here is the reason for each LTP failure (with some comments)
{{{
------------------------------------
tguid tguid:result
------------------------------------
syscalls.fanotify06 FAIL - segfault
syscalls.msgctl11 FAIL - queue mismatch
syscalls.open10 FAIL - setgid not set
syscalls.creat08 FAIL - setgid not set
sem_init.sem_init_7-1 SKIP - ??? (RESEARCH)
------------------------------------
}}}
|
|
|
== ren1 board ==
|
|
# ftc list-runs --where board=ren1,status!=PASS
|
# ftc list-runs --where board=ren1,status!=PASS
|
Runs on this host:
Test Status
--------------------------- ------
Benchmark.Dhrystone FAIL - time too small (expected - but could fix test)
Benchmark.GLMark SKIP - missing SDL/SDL.h (old)
Benchmark.Java SKIP - pre_test: missing java (expected, ok)
Benchmark.gtkperf SKIP - gdk-pixbuf-query-loaders: command not found
Benchmark.reboot SKIP - runtime error: didn't capture reboot (RESEARCH)
Benchmark.signaltest FAIL - non-RT kernel
Benchmark.x11perf SKIP - build error: no package xmuu
Functional.LTP FAIL - see below
Functional.aiostress SKIP - pre_test: no /lib/*/libaio (ok?)
Functional.crashme FAIL - 0 .. 507, but expected 3000
Functional.ft2demos SKIP - build error: missing /usr/lib/libxcb.la (RESEARCH)
Functional.glib SKIP - missing gettext support (CANDIDATE)
Functional.kernel_build FAIL - missing "Kernel: arch/.* is ready". bzImage, no such file or directory
}}}
|
{{{
Runs on this host:
Test Status
--------------------------- ------
Benchmark.Dhrystone FAIL - time too small (expected - but could fix test)
Benchmark.GLMark SKIP - missing SDL/SDL.h (old)
Benchmark.Java SKIP - pre_test: missing java (expected, ok)
Benchmark.gtkperf SKIP - gdk-pixbuf-query-loaders: command not found
Benchmark.reboot SKIP - runtime error: didn't capture reboot (RESEARCH)
Benchmark.signaltest FAIL - non-RT kernel
Benchmark.x11perf SKIP - build error: no package xmuu
Functional.LTP FAIL - see below
Functional.aiostress SKIP - pre_test: no /lib/*/libaio (ok?)
Functional.crashme FAIL - 0 .. 507, but expected 3000
Functional.ft2demos SKIP - build error: missing /usr/lib/libxcb.la (RESEARCH)
Functional.glib SKIP - missing gettext support (CANDIDATE)
Functional.kernel_build FAIL - missing "Kernel: arch/.* is ready". bzImage, no such file or directory
}}}
|
|
----
|
|
Here is the LTP report for non-PASS tests:
|
Here is the LTP report for non-PASS tests:
|
|
# ftc gen-report --where board=ren1,test=LTP,tguid:result!=PASS
|
# ftc gen-report --where board=ren1,test=LTP,tguid:result!=PASS
|
======================================================================
**** Fuego Test Report ****
test : LTP
board : ren1
kernel : 4.9.0-yocto-standard
timestamp : 2019-08-14_06:38:07
report_date : 2019-08-14_20:34:55
======================================================================
----------------------------------------------------------------------------------------
test_name spec board timestamp tguid tguid:result
----------------------------------------------------------------------------------------
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.cve-2017-6951 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.prot_hsymlinks FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.renameat202 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.add_key04 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.add_key02 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.keyctl08 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.keyctl03 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.keyctl02 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.keyctl05 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.keyctl04 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.keyctl07 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.keyctl06 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.msgctl09 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.msgctl08 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.fanotify06 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.utime02 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.mknod03 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.abort01 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.kill11 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.msgctl10 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.msgctl11 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.request_key02 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.open10 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.renameat201 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.creat08 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.add_key03 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.setsockopt02 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.cve-2017-5669 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.sendmsg01 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.request_key01 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.request_key03 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.setpriority01 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.utime01 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 sem_init.sem_init_7-1 SKIP
----------------------------------------------------------------------------------------
}}}
|
{{{
======================================================================
**** Fuego Test Report ****
test : LTP
board : ren1
kernel : 4.9.0-yocto-standard
timestamp : 2019-08-14_06:38:07
report_date : 2019-08-14_20:34:55
======================================================================
----------------------------------------------------------------------------------------
test_name spec board timestamp tguid tguid:result
----------------------------------------------------------------------------------------
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.cve-2017-6951 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.prot_hsymlinks FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.renameat202 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.add_key04 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.add_key02 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.keyctl08 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.keyctl03 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.keyctl02 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.keyctl05 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.keyctl04 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.keyctl07 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.keyctl06 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.msgctl09 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.msgctl08 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.fanotify06 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.utime02 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.mknod03 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.abort01 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.kill11 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.msgctl10 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.msgctl11 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.request_key02 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.open10 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.renameat201 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.creat08 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.add_key03 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.setsockopt02 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.cve-2017-5669 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.sendmsg01 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.request_key01 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.request_key03 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.setpriority01 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 syscalls.utime01 FAIL
Functional.LTP default ren1 2019-08-14_06:38:07 sem_init.sem_init_7-1 SKIP
----------------------------------------------------------------------------------------
}}}
|
|
----
|
|
Here are LTP details:
|
Here are LTP details:
|
|
To get sorted testcase names:
* $ ftc gen-report --where board=ren1,test=LTP,tguid:result!=PASS --fields tguid,tguid:result | sort
|
To get sorted testcase names:
* $ ftc gen-report --where board=ren1,test=LTP,tguid:result!=PASS --fields tguid,tguid:result | sort
|
--------------------------------------
tguid tguid:result
--------------------------------------
sem_init.sem_init_7-1 SKIP
syscalls.abort01 FAIL - invalid argument
syscalls.add_key02 FAIL - EINVAL
syscalls.add_key03 FAIL - create keyring for another user
syscalls.add_key04 FAIL - SIGSEGV
syscalls.creat08 FAIL - setgid should be set
syscalls.cve-2017-5669 FAIL - mapped VM addr in first 64Kb
syscalls.cve-2017-6951 FAIL - timeout requesting dead key
syscalls.fanotify06 FAIL - SIGSEGV
syscalls.keyctl02 FAIL - timeout
syscalls.keyctl03 FAIL - timeout
syscalls.keyctl04 FAIL - timeout
syscalls.keyctl05 FAIL - timeout
syscalls.keyctl06 FAIL - timeout
syscalls.keyctl07 FAIL - timeout
syscalls.keyctl08 FAIL - timeout
syscalls.kill11 FAIL - invalid argument to setrlimit
syscalls.mknod03 FAIL - setgid bit not set
syscalls.msgctl08 FAIL - ipcs not found
syscalls.msgctl09 FAIL - ipcs not found
syscalls.msgctl10 FAIL - ipcs not found
syscalls.msgctl11 FAIL - ipcs not found
syscalls.open10 FAIL - setgid bit not set
syscalls.prot_hsymlinks FAIL - failed to run cmd: useradd hsym
syscalls.renameat201 FAIL - Invalid argument
syscalls.renameat202 FAIL - Invalid argument
syscalls.request_key01 FAIL - timeout
syscalls.request_key02 FAIL - timeout
syscalls.request_key03 FAIL - timeout
syscalls.sendmsg01 FAIL - ip/ifconfig not found
syscalls.setpriority01 FAIL - useradd exited with non-zero code
syscalls.setsockopt02 FAIL - allowed bad tp_sizeof_priv value
syscalls.utime01 FAIL - tmp_file access and mod times not set
syscalls.utime02 FAIL - tmp_file access and mod times not set
--------------------------------------
}}}
|
{{{
--------------------------------------
tguid tguid:result
--------------------------------------
sem_init.sem_init_7-1 SKIP
syscalls.abort01 FAIL - invalid argument
syscalls.add_key02 FAIL - EINVAL
syscalls.add_key03 FAIL - create keyring for another user
syscalls.add_key04 FAIL - SIGSEGV
syscalls.creat08 FAIL - setgid should be set
syscalls.cve-2017-5669 FAIL - mapped VM addr in first 64Kb
syscalls.cve-2017-6951 FAIL - timeout requesting dead key
syscalls.fanotify06 FAIL - SIGSEGV
syscalls.keyctl02 FAIL - timeout
syscalls.keyctl03 FAIL - timeout
syscalls.keyctl04 FAIL - timeout
syscalls.keyctl05 FAIL - timeout
syscalls.keyctl06 FAIL - timeout
syscalls.keyctl07 FAIL - timeout
syscalls.keyctl08 FAIL - timeout
syscalls.kill11 FAIL - invalid argument to setrlimit
syscalls.mknod03 FAIL - setgid bit not set
syscalls.msgctl08 FAIL - ipcs not found
syscalls.msgctl09 FAIL - ipcs not found
syscalls.msgctl10 FAIL - ipcs not found
syscalls.msgctl11 FAIL - ipcs not found
syscalls.open10 FAIL - setgid bit not set
syscalls.prot_hsymlinks FAIL - failed to run cmd: useradd hsym
syscalls.renameat201 FAIL - Invalid argument
syscalls.renameat202 FAIL - Invalid argument
syscalls.request_key01 FAIL - timeout
syscalls.request_key02 FAIL - timeout
syscalls.request_key03 FAIL - timeout
syscalls.sendmsg01 FAIL - ip/ifconfig not found
syscalls.setpriority01 FAIL - useradd exited with non-zero code
syscalls.setsockopt02 FAIL - allowed bad tp_sizeof_priv value
syscalls.utime01 FAIL - tmp_file access and mod times not set
syscalls.utime02 FAIL - tmp_file access and mod times not set
--------------------------------------
}}}
|
|
|
== fuego-test board ==
- (CANDIDATE) '''fuego-test.default.Functional.fuego_abort''' - reports SKIP, but should be OK (behavior is correct)
- should turn this test into a nested test, that checks for correct messages in the console log
- (CANDIDATE) '''fuego-test.default.fuego_board_check''' - missing /sbin/reboot in container
- '''fuego-test.default.Functional.fuego_ftc_test''' - testcase 6 (ftc query-board differences)
- '''fuego-test.default.Functional.fuego_lint''' - expected failures
- '''fuego-test.default.Functional.fuego_release_test''' -
- '''fuego-test.default.Functional.fuego_slow_log''' -
|
|
# ftc list-runs --where board=fuego-test,status!=PASS
|
# ftc list-runs --where board=fuego-test,status!=PASS
|
Runs on this host:
Test Status
--------------------------- ------
Functional.fuego_abort SKIP - OK (CANDIDATE)
Functional.fuego_board_check FAIL - missing /sbin/reboot
Functional.fuego_compare_reports FAIL - missing baseline
Functional.fuego_ftc_test FAIL - (FIXED) testcase 3 - ftc help, testcase 6 - quoting of variables
Functional.fuego_lint FAIL - 559 errors in ftc (ok for now)
Functional.fuego_release_test FAIL - (CANDIDATE) password is required
Functional.fuego_slow_log FAIL - (FIXED) syntax error near unexpected token done
}}}
|
{{{
Runs on this host:
Test Status
--------------------------- ------
Functional.fuego_abort SKIP - OK (CANDIDATE)
Functional.fuego_board_check FAIL - missing /sbin/reboot
Functional.fuego_compare_reports FAIL - missing baseline
Functional.fuego_ftc_test FAIL - (FIXED) testcase 3 - ftc help, testcase 6 - quoting of variables
Functional.fuego_lint FAIL - 559 errors in ftc (ok for now)
Functional.fuego_release_test FAIL - (CANDIDATE) password is required
Functional.fuego_slow_log FAIL - (FIXED) syntax error near unexpected token done
}}}
|