Release_1.5_testing_-_2019-08-13 

Fuego wiki

Login or create account

Release 1.5 testing - 2019-08-13

Here are some notes on testing the 1.4 release (using fuego-1211-container, with a rolling master branch through December 2018 and January 2019):

See an overview issues table at Release test results

to do [edit section]

  • categorize all bugs
    • likely location: core, test, board, toolchain, lab, unknown
    • severity: high, medium, low
    • time to fix: short, long
    • priority: 5=high, 1=low, 0=defer

Resolution key:

  • CANDIDATE - is a candidate for fixing this release
  • RESEARCH - needs more research to categorize and prioritize
  • TRANSIENT - problem went away with subsequent test and no change in Fuego
  • LOCAL (lab,board) - problem was with setup, config, board status, board, or toolchain, and issue can be ignored
  • IGNORE - ignore (not a problem)
  • FIXED - problem fixed with change in Fuego
  • DEFER - problem exists, but will not be fixed in this release

uncategorized issues list [edit section]

deferred from 1.4 release [edit section]

See http://fuegotest.org/wiki/Release_1.5_testing_-_2019-05-02#deferred_from_1.4_release

new errors [edit section]

  • xx. (distro) docker.default.Functional.fuego_board_check - missing /sbin/reboot
    • /sbin/reboot is present on host as a symlink to /bin/systemctl
    • this is expected
  • xx. Can't easily navigate to LTP individual test logs in Jenkins interface

old errors [edit section]

  • Benchmark.GLMark - build failure on multiple platforms (missing glew.h, missing libSDL)

Errors by board [edit section]

bbb board [edit section]

  • (old) bbb.default.Benchmark.GLMark - missing GL/glew.h
  • (old) bbb.default.Benchmark.Interbench - run error: hangs in Audio test
  • (old) bbb.default.Benchmark.gtkperf - build error: configure error: checking for gtk+-2.0 gdk-2.0
  • (old) bbb.default.Benchmark.X11perf - missing X libs
  • (RESEARCH) (old) bbb.default.Functional.glib - missing gettext lib

# ftc list-runs --where "start_time>4 days ago,status!=PASS,board=bbb"

Runs on this host:
  Test                           Status
  --------------------------- -  ------
  Benchmark.Dhrystone            ERROR - fixed
  Benchmark.GLMark               SKIP - build error: GL/glew.h: No such file or directory
  Benchmark.Interbench           ERROR - run error: hangs in Audio test
  Benchmark.Java                 SKIP - no java: OK
  Benchmark.gtkperf              SKIP - build error: configure error: checking for gtk+-2.0 gdk-2.0 ...
  Benchmark.netpipe              FAIL - numbers below thresholds (size-262144.throughput = 37.54 which is not "gt 400")
  Benchmark.x11perf              SKIP - missing X libs
  Functional.LTP                 FAIL - multiple failures (hangs)
  Functional.fontconfig          SKIP - pre_check error: missing 'fc-list': OK
  Functional.ft2demos            SKIP - build error: cannot find Xau, Xdmcp libs
  Functional.fuego_abort         SKIP - run error: aborted = OK
  Functional.fuego_compare_reports FAIL - not configured (missing baseline results file): OK
  Functional.fuego_ftc_test        FAIL - (not relevant to this board)
  Functional.fuego_lint            FAIL - (not relevant to this board)
  Functional.fuego_release_test    FAIL - (not relevant to this board)
  Functional.fuego_slow_log        FAIL - (not relevant to this board)
  Functional.fuego_test_variables  FAIL - (not relevant to this board)
  Functional.glib                  SKIP - build error: missing gettext, configure error.
  Functional.hello_world           FAIL - expected (spec=fail)

min1 board [edit section]

Here are the tests that had issues:
# ftc list-runs --where board=min1,status!=PASS
Runs on this host:
  Test                         Status
  ---------------------------  ------
  Benchmark.GLMark             SKIP - build errror, missing GL/glew.h (old)
  Benchmark.cyclictest         FAIL - non-rt kernel
  Benchmark.dbench4            FAIL - (new) - failed to connect to board after reboot
  Benchmark.gtkperf            SKIP - build error: configure error: checking for gtk+-2.0 gdk-2.0 (old)
  Benchmark.signaltest         FAIL - non-rt kernel
  Benchmark.x11perf            SKIP - runtime: unable to open display ':0
  Functional.LTP               FAIL - see below
  Functional.glib              SKIP - build error: missing gettext support (RESEARCH)


Here are the LTP non-PASS results:

# ftc gen-report --where board=min1,test=LTP,tguid:result!=PASS

--- Here is the reason for each LTP failure (with some comments)

------------------------------------
 tguid                 tguid:result
------------------------------------
 syscalls.fanotify06   FAIL  - segfault
 syscalls.msgctl11     FAIL  - queue mismatch
 syscalls.open10       FAIL  - setgid not set
 syscalls.creat08      FAIL  - setgid not set
 sem_init.sem_init_7-1 SKIP  - ??? (RESEARCH)
------------------------------------

ren1 board [edit section]

# ftc list-runs --where board=ren1,status!=PASS

Runs on this host:
  Test                        Status
  --------------------------- ------
  Benchmark.Dhrystone         FAIL - time too small (expected - but could fix test)
  Benchmark.GLMark            SKIP - missing SDL/SDL.h (old)
  Benchmark.Java              SKIP - pre_test: missing java (expected, ok)
  Benchmark.gtkperf           SKIP - gdk-pixbuf-query-loaders: command not found
  Benchmark.reboot            SKIP - runtime error: didn't capture reboot (RESEARCH)
  Benchmark.signaltest        FAIL - non-RT kernel
  Benchmark.x11perf           SKIP - build error: no package xmuu
  Functional.LTP              FAIL - see below
  Functional.aiostress        SKIP - pre_test: no /lib/*/libaio (ok?)
  Functional.crashme          FAIL - 0 .. 507, but expected 3000
  Functional.ft2demos         SKIP - build error: missing /usr/lib/libxcb.la (RESEARCH)
  Functional.glib             SKIP - missing gettext support (CANDIDATE)
  Functional.kernel_build     FAIL - missing "Kernel: arch/.* is ready". bzImage, no such file or directory


Here is the LTP report for non-PASS tests:

# ftc gen-report --where board=ren1,test=LTP,tguid:result!=PASS

======================================================================
           **** Fuego Test Report ****
  test                : LTP
  board               : ren1
  kernel              : 4.9.0-yocto-standard
  timestamp           : 2019-08-14_06:38:07
  report_date         : 2019-08-14_20:34:55
======================================================================
----------------------------------------------------------------------------------------
  test_name      spec    board timestamp           tguid                   tguid:result
----------------------------------------------------------------------------------------
  Functional.LTP default ren1  2019-08-14_06:38:07 syscalls                FAIL
  Functional.LTP default ren1  2019-08-14_06:38:07 syscalls.cve-2017-6951  FAIL
  Functional.LTP default ren1  2019-08-14_06:38:07 syscalls.prot_hsymlinks FAIL
  Functional.LTP default ren1  2019-08-14_06:38:07 syscalls.renameat202    FAIL
  Functional.LTP default ren1  2019-08-14_06:38:07 syscalls.add_key04      FAIL
  Functional.LTP default ren1  2019-08-14_06:38:07 syscalls.add_key02      FAIL
  Functional.LTP default ren1  2019-08-14_06:38:07 syscalls.keyctl08       FAIL
  Functional.LTP default ren1  2019-08-14_06:38:07 syscalls.keyctl03       FAIL
  Functional.LTP default ren1  2019-08-14_06:38:07 syscalls.keyctl02       FAIL
  Functional.LTP default ren1  2019-08-14_06:38:07 syscalls.keyctl05       FAIL
  Functional.LTP default ren1  2019-08-14_06:38:07 syscalls.keyctl04       FAIL
  Functional.LTP default ren1  2019-08-14_06:38:07 syscalls.keyctl07       FAIL
  Functional.LTP default ren1  2019-08-14_06:38:07 syscalls.keyctl06       FAIL
  Functional.LTP default ren1  2019-08-14_06:38:07 syscalls.msgctl09       FAIL
  Functional.LTP default ren1  2019-08-14_06:38:07 syscalls.msgctl08       FAIL
  Functional.LTP default ren1  2019-08-14_06:38:07 syscalls.fanotify06     FAIL
  Functional.LTP default ren1  2019-08-14_06:38:07 syscalls.utime02        FAIL
  Functional.LTP default ren1  2019-08-14_06:38:07 syscalls.mknod03        FAIL
  Functional.LTP default ren1  2019-08-14_06:38:07 syscalls.abort01        FAIL
  Functional.LTP default ren1  2019-08-14_06:38:07 syscalls.kill11         FAIL
  Functional.LTP default ren1  2019-08-14_06:38:07 syscalls.msgctl10       FAIL
  Functional.LTP default ren1  2019-08-14_06:38:07 syscalls.msgctl11       FAIL
  Functional.LTP default ren1  2019-08-14_06:38:07 syscalls.request_key02  FAIL
  Functional.LTP default ren1  2019-08-14_06:38:07 syscalls.open10         FAIL
  Functional.LTP default ren1  2019-08-14_06:38:07 syscalls.renameat201    FAIL
  Functional.LTP default ren1  2019-08-14_06:38:07 syscalls.creat08        FAIL
  Functional.LTP default ren1  2019-08-14_06:38:07 syscalls.add_key03      FAIL
  Functional.LTP default ren1  2019-08-14_06:38:07 syscalls.setsockopt02   FAIL
  Functional.LTP default ren1  2019-08-14_06:38:07 syscalls.cve-2017-5669  FAIL
  Functional.LTP default ren1  2019-08-14_06:38:07 syscalls.sendmsg01      FAIL
  Functional.LTP default ren1  2019-08-14_06:38:07 syscalls.request_key01  FAIL
  Functional.LTP default ren1  2019-08-14_06:38:07 syscalls.request_key03  FAIL
  Functional.LTP default ren1  2019-08-14_06:38:07 syscalls.setpriority01  FAIL
  Functional.LTP default ren1  2019-08-14_06:38:07 syscalls.utime01        FAIL
  Functional.LTP default ren1  2019-08-14_06:38:07 sem_init.sem_init_7-1   SKIP
----------------------------------------------------------------------------------------


Here are LTP details:

To get sorted testcase names:

  • $ ftc gen-report --where board=ren1,test=LTP,tguid:result!=PASS --fields tguid,tguid:result | sort

--------------------------------------
 tguid                   tguid:result
--------------------------------------
 sem_init.sem_init_7-1   SKIP
 syscalls.abort01        FAIL - invalid argument
 syscalls.add_key02      FAIL - EINVAL
 syscalls.add_key03      FAIL - create keyring for another user
 syscalls.add_key04      FAIL - SIGSEGV
 syscalls.creat08        FAIL - setgid should be set
 syscalls.cve-2017-5669  FAIL - mapped VM addr in first 64Kb
 syscalls.cve-2017-6951  FAIL - timeout requesting dead key
 syscalls.fanotify06     FAIL - SIGSEGV
 syscalls.keyctl02       FAIL - timeout
 syscalls.keyctl03       FAIL - timeout
 syscalls.keyctl04       FAIL - timeout
 syscalls.keyctl05       FAIL - timeout
 syscalls.keyctl06       FAIL - timeout
 syscalls.keyctl07       FAIL - timeout
 syscalls.keyctl08       FAIL - timeout
 syscalls.kill11         FAIL - invalid argument to setrlimit
 syscalls.mknod03        FAIL - setgid bit not set
 syscalls.msgctl08       FAIL - ipcs not found
 syscalls.msgctl09       FAIL - ipcs not found
 syscalls.msgctl10       FAIL - ipcs not found
 syscalls.msgctl11       FAIL - ipcs not found
 syscalls.open10         FAIL - setgid bit not set
 syscalls.prot_hsymlinks FAIL - failed to run cmd: useradd hsym
 syscalls.renameat201    FAIL - Invalid argument
 syscalls.renameat202    FAIL - Invalid argument
 syscalls.request_key01  FAIL - timeout
 syscalls.request_key02  FAIL - timeout
 syscalls.request_key03  FAIL - timeout
 syscalls.sendmsg01      FAIL - ip/ifconfig not found
 syscalls.setpriority01  FAIL - useradd exited with non-zero code
 syscalls.setsockopt02   FAIL - allowed bad tp_sizeof_priv value
 syscalls.utime01        FAIL - tmp_file access and mod times not set
 syscalls.utime02        FAIL - tmp_file access and mod times not set
--------------------------------------

fuego-test board [edit section]

  • (CANDIDATE) fuego-test.default.Functional.fuego_abort - reports SKIP, but should be OK (behavior is correct)
    • should turn this test into a nested test, that checks for correct messages in the console log
  • (CANDIDATE) fuego-test.default.fuego_board_check - missing /sbin/reboot in container
  • fuego-test.default.Functional.fuego_ftc_test - testcase 6 (ftc query-board differences)
  • fuego-test.default.Functional.fuego_lint - expected failures
  • fuego-test.default.Functional.fuego_release_test -
  • fuego-test.default.Functional.fuego_slow_log -

# ftc list-runs --where board=fuego-test,status!=PASS

Runs on this host:
  Test                          Status
  ---------------------------   ------
  Functional.fuego_abort        SKIP - OK (CANDIDATE)
  Functional.fuego_board_check  FAIL - missing /sbin/reboot
  Functional.fuego_compare_reports FAIL - missing baseline
  Functional.fuego_ftc_test     FAIL - (FIXED) testcase 3 - ftc help, testcase 6 - quoting of variables
  Functional.fuego_lint         FAIL - 559 errors in ftc (ok for now)
  Functional.fuego_release_test FAIL - (CANDIDATE) password is required
  Functional.fuego_slow_log     FAIL - (FIXED) syntax error near unexpected token done

TBWiki engine 1.8.3 by Tim Bird