Release_1.4_testing_-_2018-12-26 

Fuego wiki

Login or create account

Release 1.4 testing - 2018-12-26

Here are some notes on testing the 1.4 release (using fuego-1211-container, with a rolling master branch through December 2018 and January 2019):

to do [edit section]

  • categorize all bugs
    • likely location: core, test, board, toolchain, lab, unknown
    • severity: high, medium, low
    • time to fix: short, long
    • priority: 5=high, 1=low, 0=defer

Resolution key:

  • CANDIDATE - is a candidate for fixing this release
  • RESEARCH - needs more research to categorize and prioritize
  • TRANSIENT - problem went away with subsequent test and no change in Fuego
  • LOCAL (lab,board) - problem was with setup, config, board status, board, or toolchain, and issue can be ignored
  • IGNORE - ignore (not a problem)
  • FIXED - problem fixed with change in Fuego
  • DEFER - problem exists, but will not be fixed in this release

uncategorized issues list [edit section]

  • 1. FIXED (core) bbb.default.Benchmark.Interbench - timeout is reported as PASS in Jenkins
    • see build_number 1 - test hung and it took 40 minutes
  • 2. DEFER (test) LTP runs for too long with nothing showing on the console. Jenkins should show the current testcase being run, or something.
    • would require output from fuego_test.sh
  • 3. FIXED (core) min1.default.Benchmark.OpenSSL - took 15 minutes (which timed out) but is reported as PASS in Jenkins
  • 4. DEFER (core) Jenkins doesn't show different color for ERROR (timeout or abort)
  • 5. FIXED (core) main.sh may return success even when the test was interrupted
    • e.g. when terminated with SIGTERM, to allow post-test to run and results are only partial
  • 6. DEFER (core) bbb.default.Benchmark.cyclictest it's not easy to find the threshold value. It's not in run.json. It's on the chart, but with multiple boards it's hard to isolate and find
    • could add checkboxes for series.
    • could add reference value to run.json
    • could copy reference.log or reference.json to log directory
    • could have parser add reference check failure to console log
  • 7. DEFER (core) ftc rm-jobs rpi3-1.*.*.* - can't do rm-jobs outside the container! (annoying, but whatever)
    • required due to possible interactive confirmation
    • should be able to rm-jobs without being inside the container
    • should be able to do interactive commands even when forwarding to the container ftc
  • 8. FIXED (core) min1.default.Benchmark.OpenSSL - has timeout of 15 minutes (should be default of 40, from what I can see)
    • was from 15m default timeout in testplan_smoketest.json
  • 9. RESEARCH (test) min1.default.Functional.LTP - new failure for open10
  • 10. RESEARCH (test) min1.default.Functional.LTP - new failure for creat08
  • 11. DEFER (core?) need to have test indicate whether error code indicates error or failure
    • example: x11perf - non-zero = error (not failure)
  • 12. FIXED (core) it was hard to add a default timeout to testplan_fuego_tests, because I couldn't find the schema for testplans very easily. I didn't know the variable name.
  • 13. FIXED (core) testplans usb, mmc and sata could have used 'default_spec' to save lines
  • 14. FIXED (core) fuego-test.default.Functional.fuego_dependencies
    • testcase 5 NEED_KCONFIG=CONFIG_PRINTK=y
    • problem finding /boot/config-$(uname -r) for need_KCONFIG
  • 15. FIXED (core) ren1.default.Functional.fuego_test_minimal
    • fails because of missing spec.json in ftc
  • 16. CANDIDATE, HIGH (core) fuego-test.default.Functional.fuego_release_test
    • problem with Python 3 dependencies in docker project (requests requires urllib3, which is missing dependencies?)

     = ==== doing fuego phase: run =====
    -------------------------------------------------
    /usr/local/lib/python3.4/dist-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.9.1) or chardet (3.0.4) doesn't match a supported version!
      RequestsDependencyWarning)
    Traceback (most recent call last):
      File "/fuego-core/engine/tests/Functional.fuego_release_test/test_run.py", line 15, in <module>
        import docker
      File "/usr/local/lib/python3.4/dist-packages/docker/__init__.py", line 2, in <module>
        from .api import APIClient
      File "/usr/local/lib/python3.4/dist-packages/docker/api/__init__.py", line 2, in <module>
        from .client import APIClient
      File "/usr/local/lib/python3.4/dist-packages/docker/api/client.py", line 5, in <module>
        import requests
      File "/usr/local/lib/python3.4/dist-packages/requests/__init__.py", line 105, in <module>
        from urllib3.exceptions import DependencyWarning
    ImportError: cannot import name 'DependencyWarning'

  • 16. RESEARCH (unknown) rpi3-2.default.Functional.fuego_test_variables
    • testcases 2,3, and 9
  • 17. DEFER (board) rpi3-2.default.Functional.fuego_dependencies - - can't find kconfig
    • missing /proc/config.gz or /boot/config-4.9.41-v7+ (but has /lib/modules.../configs.ko)
  • 18. DEFER (test) rpi3-2.default.Functional.fuego_sleep - timed out
    • timeout from testplan_fuego_tests is too short
  • 19. DEFER (core) test that fails pre-check is shown as 'not run' in Jenkins
  • 20. DEFER (core) test that fails pre-check has no result code (listed as None) in 'ftc gen-report'
  • 21. FIXED (core) ren1.default.Functional.aiostress - (post-processing) Invalid literal for base 10 in generic_parser
    • changed to a try here, and convert to ERROR in except clause
    • or figure out why RETURN_VALUE is missing!!
      • it's missing because:
        • the build error caused early exit
        • RETURN_VALUE was never set (by report, et al.)
        • this test doesn't have a test_processing function

    ##### doing fuego phase: processing (from signal handler) #####
    WARNING: program returned exit code
    Log evaluation may be invalid
    Traceback (most recent call last):
      File "/fuego-core/engine/scripts/generic_parser.py", line 18, in <module>
        value = "PASS" if int(RETURN_VALUE) == 0 else "FAIL"
    ValueError: invalid literal for int() with base 10: ''
    ERROR: results did not satisfy the threshold

  • 22. DEFER (test) ren1.default.Functional.LTP_one_test.syscalls.add_key_02 - timeout is not converted to Fuego ERROR
    • the LTP parser should convert a timed-out test in LTP into a Fuego ERROR in the run.json file. It doesn't do this.
  • 21. FIXED (core) - ftc rm-node rpi3-1 - cannot remove node rpi3-1
    • get message "global name 'install_nodes' is not defined"
  • 23. FIXED (core) - ftc rm-node rpi3-1 - has no effect
  • 24. FIXED (core) - ftc rm-jobs --remove-logs Bench.default.Benchmark.hackbench
    • gives weird message

Test failures by board [edit section]

Here are testplan_default failures for 2018-12-26

bbb [edit section]

Summary:
# ftc list-runs --where "start_time>4 days ago,status!=PASS,board=bbb"
Runs on this host:
  Test                        Started                host:board                     Status
  --------------------------- ---------------------- ------------------------------ ------
  Benchmark.Interbench        2018-12-26_18:00:01    timslab:bbb                    ERROR
  Benchmark.OpenSSL           2018-12-26_18:50:36    timslab:bbb                    SKIP
  Benchmark.cyclictest        2018-12-26_18:51:25    timslab:bbb                    FAIL
  Benchmark.gtkperf           2018-12-26_19:01:27    timslab:bbb                    SKIP
  Benchmark.x11perf           2018-12-26_18:49:56    timslab:bbb                    SKIP
  Functional.LTP              2018-12-26_19:07:47    timslab:bbb                    FAIL
  Functional.LTP              2018-12-26_21:20:47    timslab:bbb                    FAIL
  Functional.OpenSSL          2018-12-26_18:56:31    timslab:bbb                    FAIL
  Functional.aiostress        2018-12-26_23:06:06    timslab:bbb                    FAIL
  Functional.fontconfig       2018-12-26_19:01:14    timslab:bbb                    FAIL
  Functional.ft2demos         2018-12-26_19:00:25    timslab:bbb                    ERROR

Analysis:

  • FIXED (test) bbb.default.Benchmark.Interbench - ERROR - timeout of 40m reported as SUCCESS
    • reported as success in Jenkins and ERROR in run.json
  • DEFER (test, board) bbb.default.Benchmark.Interbench - ERROR - test hangs!! (not new)
    • hangs in Audio test (not new, but should be resolved)

    last data reported from test is:
    --- Benchmarking simulated cpu of Audio in the presence of simulated ---
    Load	Latency +/- SD (ms)  Max Latency   % Desired CPU  % Deadlines Met
    None	   0.1 +/- 0.1        0.3		 100	        100
    Video	   0.1 +/- 0.1        0.2		 100	        100
    X	   0.1 +/- 0.1        1.3		 100	        100
    Burn	   0.0 +/- 0.0        0.1		 100	        100
    Write
    calloc: Cannot allocate memory
       0.1 +/- 0.3        6.6		 100	        100

  • DEFER (test,toolchain) bbb.default.Benchmark.OpenSSL - build failed - missing zconf.h from zlib (not new)
      arm-linux-gnueabihf-gcc -I.. -I../.. -I../asn1 -I../evp -I../../include  \
         --sysroot=/ -DZLIB_SHARED -DZLIB -DOPENSSL_THREADS     -c -o c_zlib.o c_zlib.c
      In file included from c_zlib.c:25:0:
      /usr/include/zlib.h:34:19: fatal error: zconf.h: No such file or directory
       #include "zconf.h"
                         ^
      compilation terminated.

  • CANDIDATE (board) bbb.default.Benchmark.cyclictest - FAIL - board doesn't have RT kernel (expected)
    • max_latency doesn't pass threshold test, for 1000ms reference value
    • fix would be to provide threshold for non-RT board
      • add bbb.Benchmark.cyclictest.criteria.json
  • DEFER (test?,toolchain?) bbb.default.Benchmark.gtkperf - build failure (not new)
    • build failed - can't build test
      • last message: checking for gtk+-2.0 gdk-2.0
  • DEFER (toolchain) bbb.default.Benchmark.x11perf - build failure
    • build failed - cannot find -lXext -lXrender -lX11 -lXmu
  • DEFER (test) bbb.default.Functional.glib - build failure - swp in Thumb mode not supported (not new)
    • error was:

    /bin/sh ../libtool  --tag=CC   --mode=compile arm-linux-gnueabihf-gcc -DHAVE_CONFIG_H -I. -I.. \
       -I..  -DG_LOG_DOMAIN=\"GLib\" -DG_DISABLE_CAST_CHECKS -DG_DISABLE_DEPRECATED -DGLIB_COMPILATION \
       -DPCRE_STATIC -DG_DISABLE_SINGLE_INCLUDES -pthread  -g -O2 -Wall -MT gatomic.lo -MD -MP \
       -MF .deps/gatomic.Tpo -c -o gatomic.lo gatomic.c
    libtool: compile:  arm-linux-gnueabihf-gcc -DHAVE_CONFIG_H -I. -I.. -I.. -DG_LOG_DOMAIN=\"GLib\" \
      -DG_DISABLE_CAST_CHECKS -DG_DISABLE_DEPRECATED -DGLIB_COMPILATION -DPCRE_STATIC \
      -DG_DISABLE_SINGLE_INCLUDES -pthread -g -O2 -Wall -MT gatomic.lo -MD -MP -MF .deps/gatomic.Tpo \
      -c gatomic.c  -fPIC -DPIC -o .libs/gatomic.o
    /tmp/ccTcovfi.s: Assembler messages:
    /tmp/ccTcovfi.s:65: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
    /tmp/ccTcovfi.s:85: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
    /tmp/ccTcovfi.s:188: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
    /tmp/ccTcovfi.s:208: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
    /tmp/ccTcovfi.s:310: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
    /tmp/ccTcovfi.s:330: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
    /tmp/ccTcovfi.s:432: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
    /tmp/ccTcovfi.s:452: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
    Makefile:994: recipe for target 'gatomic.lo' failed

  • RESEARCH (test) bbb.default.Functional.LTP - FAIL - see details below
  • DEFER (test) bbb.default.Functional.OpenSSL - FAIL - missing zconf.h (not new)
  • FIXED (board) bbb.default.Functional.aiostress - FAIL - can't load libaio.so.1 on board (not new)
    • should be reported as ERROR (failure to run test program)
  • FIXED (board) bbb.default.Functional.aiostress - FAIL - can't load libaio.so.1 on board (not new)
    • should use dependency check for package libaio (not sure it will work with non-Debian)
    • should resolve with a dependency on libaio.so.1 on the board
      • does this test apply, if libaio.so is not present on the board?? - NO
      • libaio.so.1 is located in bbb:/lib/arm-linux-gnueabihf/libaio.so.1
      • /lib/arm-linux-gnueabihf system shared lib path is in: /etc/ld.so.conf.d/arm-linux-gneuabihf.conf
    • can't build without libaio-dev in the SDK (libaio.h)
      • aio functions are supposed to be provided by C library
    • removed '-laio' from build line
    • can be resolved with 'apt-get install libaio1' on the board
  • FIXED (board) bbb.default.Functional.fontconfig - FAIL - missing fc-list (expected)
    • added assert_has_program to fuego_test.sh
  • DEFER bbb.default.Functional.ft2demos - ERROR - missing X libs (not new)

    libtool: link: arm-linux-gnueabihf-gcc -Wl,-L -Wl,//usr/lib -Wl,-lXau -Wl,-lXdmcp --sysroot / \
      -o /fuego-rw/buildzone/bbb.default.Functional.ft2demos-debian-armhf/ft2demos-2.3.6/bin/.libs/ftlint \
      /fuego-rw/buildzone/bbb.default.Functional.ft2demos-debian-armhf/ft2demos-2.3.6/obj/ftlint.o \
      /fuego-rw/buildzone/bbb.default.Functional.ft2demos-debian-armhf/ft2demos-2.3.6/obj/common.o  \
      ../freetype-2.3.6/objs/.libs/libfreetype.so -lm
    /usr/lib/gcc-cross/arm-linux-gnueabihf/6/../../../../arm-linux-gnueabihf/bin/ld: cannot find -lXau
    /usr/lib/gcc-cross/arm-linux-gnueabihf/6/../../../../arm-linux-gnueabihf/bin/ld: cannot find -lXdmcp
    collect2: error: ld returned 1 exit status
    Makefile:426: recipe for target '/fuego-rw/buildzone/bbb.default.Functional.ft2demos-debian-armhf/ft2demos-2.3.6/bin/ftlint' failed
    make: *** [/fuego-rw/buildzone/bbb.default.Functional.ft2demos-debian-armhf/ft2demos-2.3.6/bin/ftlint] Error 1

  • DEFER (test, toolchain) - bbb.default.Functional.glib - problem with thumb mode instructions

    libtool: compile:  arm-linux-gnueabihf-gcc -DHAVE_CONFIG_H -I. -I.. -I.. \
      -DG_LOG_DOMAIN=\"GLib\" -DG_DISABLE_CAST_CHECKS -DG_DISABLE_DEPRECATED -DGLIB_COMPILATION \
      -DPCRE_STATIC -DG_DISABLE_SINGLE_INCLUDES -pthread -g -O2 -Wall -MT gatomic.lo -MD -MP \
      -MF .deps/gatomic.Tpo -c gatomic.c  -fPIC -DPIC -o .libs/gatomic.o
    /tmp/ccTcovfi.s: Assembler messages:
    /tmp/ccTcovfi.s:65: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
    /tmp/ccTcovfi.s:85: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
    /tmp/ccTcovfi.s:188: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
    /tmp/ccTcovfi.s:208: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
    /tmp/ccTcovfi.s:310: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
    /tmp/ccTcovfi.s:330: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
    /tmp/ccTcovfi.s:432: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
    /tmp/ccTcovfi.s:452: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
    Makefile:994: recipe for target 'gatomic.lo' failed
    make[4]: *** [gatomic.lo] Error 1

bbb LTP results [edit section]

# ftc gen-report --where "board=bbb,test=LTP,tguid:result!=PASS,start_time>9 days ago" | sort -k 5

  board               : bbb
  kernel              : 4.4.88-ti-r125
  report_date         : 2019-01-04_20:20:37
  test                : LTP
  timestamp           : 2018-12-26_21:20:47
--------------------------------------------------------------------------------------------------------------------------------
======================================================================
  test_name            spec                 board                timestamp            tguid                tguid:result
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  sem_init.sem_init_7-1 SKIP
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls             FAIL
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.add_key03   FAIL
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.add_key04   FAIL
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.eventfd01   FAIL
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.fanotify06  FAIL
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.fcntl35     FAIL
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.fcntl35_64  FAIL
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.getrusage03 FAIL
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.io_cancel01 FAIL
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.io_destroy01 FAIL
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.io_getevents01 FAIL
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.io_setup01  FAIL
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.io_submit01 FAIL
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.keyctl02    FAIL
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.keyctl03    FAIL
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.keyctl04    FAIL
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.keyctl05    FAIL
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.keyctl06    FAIL
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.keyctl07    FAIL
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.msgctl10    FAIL
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.msgctl11    FAIL
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.request_key01 FAIL
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.request_key02 FAIL
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.request_key03 FAIL

min1 [edit section]

Summary:
$ ftc list-runs --where "start_time>yesterday,status!=PASS,board=min1"
  Test                        Started                host:board                     Status
  Benchmark.Java              2018-12-26_18:56:19    timslab:min1                   ERROR
  Benchmark.OpenSSL           2018-12-26_19:36:18    timslab:min1                   ERROR
  Benchmark.cyclictest        2018-12-26_19:51:52    timslab:min1                   FAIL
  Benchmark.signaltest        2018-12-26_19:55:13    timslab:min1                   FAIL
  Functional.LTP              2018-12-26_18:00:42    timslab:min1                   FAIL
  Functional.glib             2018-12-26_18:57:06    timslab:min1                   FAIL
  Benchmark.GLMark            2018-12-26_18:55:04    timslab:min1                   SKIP
  Benchmark.x11perf           2018-12-26_18:50:09    timslab:min1                   SKIP
  --------------------------- ---------------------- ------------------------------ ------

Analysis:

  • DEFER (toolchain) min1.default.Benchmark.GLMark - ERROR - failed to build
    • missing glew.h file for build
    • should have been reported as an error
  • FIXED (test) min1.default.Benchmark.Java - ERROR - (board does not have java command (expected)
    • changed to 'assert_has_program java'
  • FIXED (core) min1.default.Benchmark.OpenSSL - interrupted at 15 minutes (according to timeout value), returned SUCCESS to Jenkins
    • changed so that ERROR is returned on timeouts
    • need to change timeout higher
  • CANDIDATE (board) min1.default.Benchmark.cyclictest - FAIL (board does not have RT kernel) (expected)
    • max latency over threshold
      • should be higher threshold?
    • add non-RT threshold??
  • CANDIDATE (board) min1.default.Benchmark.signaltest - FAIL (board does not have RT kernel) (expected)
    • max latency over threshold
      • should be higher threshold?
  • DEFER (test?) min1.default.Benchmark.x11perf - SKIP
    • x11perf: unable to open display ':0"
    • should have been reported as an error
  • RESEARCH (test) min1.default.Functional.LTP - FAIL - 10 failures in syscalls, 1 sem_init SKIP (was 9 failures in 1.3 - see details below)
  • DEFER (test?) min1.default.Functional.glib - FAIL
    • 'threadtests' does not report pass or failure. Test appears to terminate prematurely? (see testlog.txt)
    • there are many GLib-CRITICAL reports for "Source ID ## was not found when attempting to remove it"
    • returns exit code 143

min1 LTP results [edit section]

# ftc gen-report --where "board=min1,test=LTP,tguid:result!=PASS"
======================================================================
           **** Fuego Test Report ****
  test                : LTP
  board               : min1
  kernel              : 4.15.0-33-generic
  timestamp           : 2018-12-26_18:00:42
  report_date         : 2018-12-28_19:42:54
======================================================================
--------------------------------------------------------------------------------------------------------------------------------
  test_name            spec                 board                timestamp            tguid                tguid:result
--------------------------------------------------------------------------------------------------------------------------------
  Functional.LTP       default              min1                 2018-12-26_18:00:42  syscalls             FAIL
  Functional.LTP       default              min1                 2018-12-26_18:00:42  syscalls.io_getevents01 FAIL
  Functional.LTP       default              min1                 2018-12-26_18:00:42  syscalls.fanotify06  FAIL
  Functional.LTP       default              min1                 2018-12-26_18:00:42  syscalls.io_submit01 FAIL
  Functional.LTP       default              min1                 2018-12-26_18:00:42  syscalls.io_cancel01 FAIL
  Functional.LTP       default              min1                 2018-12-26_18:00:42  syscalls.msgctl11    FAIL
  Functional.LTP       default              min1                 2018-12-26_18:00:42  syscalls.eventfd01   FAIL
  Functional.LTP       default              min1                 2018-12-26_18:00:42  syscalls.open10      FAIL
  Functional.LTP       default              min1                 2018-12-26_18:00:42  syscalls.creat08     FAIL
  Functional.LTP       default              min1                 2018-12-26_18:00:42  syscalls.io_setup01  FAIL
  Functional.LTP       default              min1                 2018-12-26_18:00:42  syscalls.io_destroy01 FAIL
  Functional.LTP       default              min1                 2018-12-26_18:00:42  sem_init.sem_init_7-1 SKIP
--------------------------------------------------------------------------------------------------------------------------------

Different from 1.3:
 * RESEARCH min1.default.Functional.LTP.syscalls.open10 - FAIL
 * RESEARCH min1.default.Functional.LTP.syscalls.creat08 - FAIL

test that were run in 1.3 but not 1.4 release testing:

  • min1.default.Functional.aiostress
  • min1.default.Functional.kernel_build

ren1 [edit section]

Summary:
# ftc list-runs --where "board=ren1,start_time>yesterday,status!=PASS"
Runs on this host:
  Test                        Started                host:board                     Status
  --------------------------- ---------------------- ------------------------------ ------
  Benchmark.Dhrystone         2018-12-28_21:11:40    timslab:ren1                   FAIL
  Benchmark.GLMark            2018-12-28_21:34:14    timslab:ren1                   SKIP
  Benchmark.Java              2018-12-28_21:38:54    timslab:ren1                   ERROR
  Benchmark.gtkperf           2018-12-28_21:09:41    timslab:ren1                   SKIP
  Benchmark.signaltest        2018-12-28_21:10:53    timslab:ren1                   FAIL
  Benchmark.x11perf           2018-12-28_21:58:42    timslab:ren1                   SKIP
  Functional.LTP              2018-12-28_21:58:50    timslab:ren1                   FAIL
  Functional.aiostress        2018-12-28_21:34:10    timslab:ren1                   ERROR
  Functional.crashme          2018-12-28_21:31:30    timslab:ren1                   FAIL
  Functional.ft2demos         2018-12-28_20:47:19    timslab:ren1                   ERROR
  Functional.glib             2018-12-28_20:45:28    timslab:ren1                   FAIL

Analysis:

  • DEFER (test, board) ren1.default.Benchmark.Dhrystone - FAIL (needs 100M spec)
  • DEFER (test, toolchain) ren1.default.Benchmark.GLMark - SKIP (build failure - missing SDL.h)
  • FIXED (board) ren1.default.Benchmark.Java - ERROR (missing 'java' on board) (expected)
  • DEFER ren1.default.Benchmark.gtkperf - SKIP
    • (deploy error) sh: gdk-pixbuf-query-loaders: command not found
  • DEFER ren1.default.Benchmark.signaltest - FAIL
  • DEFER ren1.default.Benchmark.x11perf - SKIP
  • RESEARCH ren1.default.Functional.LTP - FAIL - see details below
  • DEFER ren1.default.Functional.aiostress - ERROR (build error) (not new)
    • aiostress.c:58:20: fatal error: libaio.h: No such file or directory compilation terminated.
  • DEFER ren1.default.Functional.crashme - FAIL (not new)
    • results did not satisfy the threshold
      • log_compare "0 ... 3000" (but test result was "0 ... 507")
  • FIXED ren1.default.Functional.ft2demos - ERROR (build error)
    • /fuego-core/engine/tests/Functional.ft2demos/fuego_test.sh: line 16: build_error: command not found
  • DEFER ren1.default.Functional.ft2demos - ERROR
           - libtool: link: cannot find the library `=/usr/lib/libxcb.la' or unhandled argument `=/usr/lib/libxcb.la'
      Makefile:470: recipe for target '/fuego-rw/buildzone/ren1.default.Functional.ft2demos-poky-aarch64/ft2demos-2.3.6/bin/ftview' failed
         - make: *** [/fuego-rw/buildzone/ren1.default.Functional.ft2demos-poky-aarch64/ft2demos-2.3.6/bin/ftview] Error 1

  • DEFER ren1.default.Functional.glib - FAIL (build error) (not new, same as 1.3)
    • 'threadtests' does not report pass or failure. Test appears to terminate prematurely? (see testlog.txt)
    • there are many GLib-CRITICAL reports for "Source ID ## was not found when attempting to remove it"
    • returns exit code 143

  • RESEARCH ren1.default.Functional.kernel_build - FAIL, build error on assembly instruction
    • problem with head.S assembly and poky arm64 toolchain (as)

      AS      arch/arm64/kernel/head.o
    arch/arm64/kernel/head.S: Assembler messages:
    arch/arm64/kernel/head.S:824: Error: missing ')'
    arch/arm64/kernel/head.S:824: Error: missing ')'
    arch/arm64/kernel/head.S:824: Error: missing ')'
    arch/arm64/kernel/head.S:824: Error: unexpected characters following instruction at operand 2 -- `mov x2,#(2)|(2U<<(8))'
    scripts/Makefile.build:391: recipe for target 'arch/arm64/kernel/head.o' failed
    make[1]: *** [arch/arm64/kernel/head.o] Error 1
    Makefile:1055: recipe for target 'arch/arm64/kernel' failed
    make: *** [arch/arm64/kernel] Error 2

ren1 LTP results [edit section]

# ftc gen-report --where "board=ren1,test=LTP,tguid:result!=PASS"
======================================================================
           **** Fuego Test Report ****
  test                : LTP
  board               : ren1
  kernel              : 4.9.0-yocto-standard
  timestamp           : 2018-12-26_21:38:12
  report_date         : 2018-12-28_23:12:46
======================================================================
-------   - (had an error about missing program 'build_error', but that was fixed:
      - /fuego-core/engine/tests/Functional.ft2demos/fuego_test.sh: line 16: build_error: command not found)
-------------------------------------------------------------------------------------------------------------------------
  test_name            spec                 board                timestamp            tguid                tguid:result
--------------------------------------------------------------------------------------------------------------------------------
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls             FAIL
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.cve-2017-6951 FAIL
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.prot_hsymlinks FAIL
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.renameat202 FAIL
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.add_key04   FAIL
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.add_key02   FAIL
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.keyctl08    FAIL
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.keyctl03    FAIL
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.keyctl02    FAIL
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.keyctl05    FAIL
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.keyctl04    FAIL
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.keyctl07    FAIL
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.keyctl06    FAIL
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.msgctl09    FAIL
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.msgctl08    FAIL
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.fanotify06  FAIL
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.utime02     FAIL
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.mknod03     FAIL
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.abort01     FAIL
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.recvmsg01   FAIL
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.kill11      FAIL
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.msgctl10    FAIL
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.msgctl11    FAIL
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.request_key02 FAIL
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.open10      FAIL
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.renameat201 FAIL
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.creat08     FAIL
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.add_key03   FAIL
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.setsockopt02 FAIL
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.cve-2017-5669 FAIL
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.sendmsg01   FAIL
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.request_key01 FAIL
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.request_key03 FAIL
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.setpriority01 FAIL
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.utime01     FAIL
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  sem_init.sem_init_7-1 SKIP
--------------------------------------------------------------------------------------------------------------------------------

Changed from 1.3 release testing were:

  • DEFER ren1.default.Functional.LTP.syscalls.add_key02 - failed in 1.4
  • IGNORE ren1.default.Functional.LTP.syscalls.fanotify08 - worked now (but failed in 1.3)
  • IGNORE ren1.default.Functional.LTP.syscalls.fanotify07 - worked now (but failed in 1.3)
  • IGNORE ren1.default.Functional.LTP.syscalls.perf_event_open02 - worked now (but failed in 1.3)
  • RESEARCH ren1.default.Functional.LTP.syscalls.mknod03 - failed in 1.4
  • RESEARCH ren1.default.Functional.LTP.syscalls.creat08 - failed in 1.4

  • DEFER, HIGH (post 1.4) ren1.default.Functional.LTP.syscalls.x (for x in: add_key0[1234], keyctl0[2345678], request_key0[123]) (14 tests)
    • add these tests to 'should reboot after test' list for LTP
    • these are probably xfail?? for creating zombie processes

Also, during the test, several processes (add_key, keyctl, request_key) were stuck in uninterruptible sleep:

$ ps auxwww (trimmed)
24350 root      2860 S    /usr/sbin/dropbear -i -r /etc/dropbear/dropbear_rsa_host_key -B
24351 root      2908 S    sh -c { cd /home/fuego/fuego.Functional.LTP; TESTS="syscalls "; PTSTESTS="SEM "; \
  RTTESTS=""; . ./ltp_target_run.sh; echo $? > /tmp/22722-26579; } 2>&1 | tee /home/fuego/fuego.Functional.LTP/Functional.LTP.log
24352 root      2908 S    sh -c { cd /home/fuego/fuego.Functional.LTP; TESTS="syscalls "; PTSTESTS="SEM "; \
  RTTESTS=""; . ./ltp_target_run.sh; echo $? > /tmp/22722-26579; } 2>&1 | tee /home/fuego/fuego.Functional.LTP/Functional.LTP.log
24353 root      1760 S    tee /home/fuego/fuego.Functional.LTP/Functional.LTP.log
24377 root      3568 S    {runltp} /bin/sh ./runltp -C /home/fuego/fuego.Functional.LTP/result/syscalls/failed.log \
  -l /home/fuego/fuego.Functional.LTP/result/syscalls/result.log \
  -o /home/fuego/fuego.Functional.LTP/result/syscalls/output.log \
  -d /home/fuego/fuego.Functional.LTP/tmp -S ./skiplist.txt -f syscalls
24548 root      1888 S    /home/fuego/fuego.Functional.LTP/bin/ltp-pan -e -S -a 24377 -n 24377 \
  -f /home/fuego/fuego.Functional.LTP/tmp/ltp-T8KMbYu8JQ/alltests \
  -l /home/fuego/fuego.Functional.LTP/result/syscalls/result.log \
  -o /home/fuego/fuego.Functional.LTP/result/syscalls/output.log \
  -C /home/fuego/fuego.Functional.LTP/result/syscalls/failed.log \
  -T /home/fuego/fuego.Functional.LTP/output/LTP_RUN_ON-output.log.tconf
24692 root      1820 D    add_key01
24694 root      1820 D    add_key02
24696 root      2296 D    add_key03
24698 root      1824 D    add_key04
24753 root         0 SW   [kworker/u12:2]
26884 root      2004 D    keyctl02
26886 root      1820 D    keyctl03
26888 root      1820 D    keyctl04
26890 root      1824 D    keyctl05
26892 root      1820 D    keyctl06
26895 root      1824 D    keyctl07
26897 root      1820 D    keyctl08
27708 root      1820 D    request_key01
27710 root      1820 D    request_key02
27712 root      1824 D    request_key03
27714 root      1820 D    cve-2017-6951
32620 root         0 SW   [kworker/u12:0]

rpi3-2 [edit section]

Summary:
cascade caused problems

Analysis:

  • DEFER rpi3-2.default.Benchmark.GLMark - SKIP - build error, missing glew.h
    • compilation problem with cpp code - missing GL header

n file included from vector.h:4:0,
                 from vector.cpp:1:
oglsdl.h:5:21: fatal error: GL/glew.h: No such file or directory
 #include <GL/glew.h>
                     ^
compilation terminated.
  • DEFER - rpi3-2.default.Benchmark.Interbench - times out (hangs)
    • hangs in Audio test (well, I stopped it)

  • DEFER - rpi3-2.default.Benchmark.bonnie - FAIL (processing failure)
    • bug in parser for invalid values (probably ++++)

'===== doing fuego phase: processing =====
Reading current values from /fuego-rw/logs/Benchmark.bonnie/rpi3-2.default.1.1/testlog.txt

Bonnie++ raw results: ['1.97', '1.97', 'rpi3-2', '1', '1546056554', '32M', '', '126', '98', '9574', '6',
   '9197', '4', '613', '99', '+++++', '+++', '357.2', '5', '16', '', '', '', '', '4154', '19', '+++++', '+++',
   '4582', '29', '3820', '17', '+++++', '+++', '7129', '45', '120ms', '344us', '568us', '18744us', '108us',
   '20117us', '59893us', '3679us', '5008us', '566us', '175us', '895us']
Traceback (most recent call last):
  File "/fuego-core/engine/tests/Benchmark.bonnie/parser.py", line 36, in <module>
    measurements["Sequential_Output.PerChr"].append({"name": "speed", "measure" : float(results[2])})
ValueError: could not convert string to float: rpi3-2
ERROR: results did not satisfy the threshold

  • DEFER rpi3-2.default.Benchmark.gtkperf - SKIP build error
    • problem in configure - checking for gtk+-2.0 gdk-2.0 ..
  • DEFER rpi3-2.default.Functional.glib - ERROR - build error
    • compiling gatomic, swp instructions in Thumb mode
  • FIXED (board, spec?) - rpi3-2.default.Functional.kernel_build - ERROR (no bzImage, but zImage is present)
    • no bzImage, but zImage is present
    • kernel make always defaults to building 'zImage' if the target is not recognized 'make foo' makes a zImage.
    • fixed by adding FUNCTIONAL_KERNEL_BUILD_TARGET="zImage" to rpi3-2.board
      • this is fragile, as the spec value overrides 'target'. But by default no 'target' is specified.
      • would be better to use "TARGET_KERNEL_IMAGE" in board, and if specified, override the target from the spec?

  • FIXED (core) - rpi3-2.default.Functional.kernel_build -
    • got this this from generic_parser.py:

    #### doing fuego phase: processing (from signal handler) #####
    Processing kernel build log
    log_compare: pattern '[ \t]*Kernel: arch/. arch/.. arch/.gitignore is ready' \
      found 2 times (expected greater or equal than 1)
    Traceback (most recent call last):
      File "/fuego-core/engine/scripts/generic_parser.py", line 18, in <module>
        value = "PASS" if int(RETURN_VALUE) == 0 else "FAIL"
    ValueError: invalid literal for int() with base 10: ''

Overview [edit section]

  root@tlinux:/# # ftc list-runs --where "start_time>yesterday,status!=PASS" | sort -k 3
  Test                        Started                host:board                     Status
  Benchmark.Interbench        2018-12-26_18:00:01    timslab:bbb                    ERROR
  Functional.LTP              2018-12-26_21:20:47    timslab:bbb                    ERROR
  Functional.ft2demos         2018-12-26_19:00:25    timslab:bbb                    ERROR
  Benchmark.cyclictest        2018-12-26_18:51:25    timslab:bbb                    FAIL
  Functional.LTP              2018-12-26_19:07:47    timslab:bbb                    FAIL
  Functional.OpenSSL          2018-12-26_18:56:31    timslab:bbb                    FAIL
  Functional.fontconfig       2018-12-26_19:01:14    timslab:bbb                    FAIL
  Benchmark.OpenSSL           2018-12-26_18:50:36    timslab:bbb                    SKIP
  Benchmark.gtkperf           2018-12-26_19:01:27    timslab:bbb                    SKIP
  Benchmark.x11perf           2018-12-26_18:49:56    timslab:bbb                    SKIP

  Benchmark.Java              2018-12-26_18:56:19    timslab:min1                   ERROR
  Benchmark.OpenSSL           2018-12-26_19:36:18    timslab:min1                   ERROR
  Benchmark.cyclictest        2018-12-26_19:51:52    timslab:min1                   FAIL
  Benchmark.signaltest        2018-12-26_19:55:13    timslab:min1                   FAIL
  Functional.LTP              2018-12-26_18:00:42    timslab:min1                   FAIL
  Functional.glib             2018-12-26_18:57:06    timslab:min1                   FAIL
  Benchmark.GLMark            2018-12-26_18:55:04    timslab:min1                   SKIP
  Benchmark.x11perf           2018-12-26_18:50:09    timslab:min1                   SKIP

  Functional.LTP              2018-12-26_21:38:12    timslab:ren1                   ERROR
  Functional.netperf          2018-12-26_18:16:13    timslab:ren1                   ERROR
  Benchmark.signaltest        2018-12-26_18:15:59    timslab:ren1                   FAIL
  Functional.glib             2018-12-26_18:33:38    timslab:ren1                   FAIL

  Benchmark.Whetstone         2018-12-26_19:09:31    timslab:rpi3-2                 ERROR
  Benchmark.tiobench          2018-12-26_19:08:28    timslab:rpi3-2                 ERROR
  Functional.ft2demos         2018-12-26_18:50:43    timslab:rpi3-2                 ERROR
  Functional.kernel_build     2018-12-26_18:38:15    timslab:rpi3-2                 ERROR
  Functional.linus_stress     2018-12-26_18:43:44    timslab:rpi3-2                 ERROR
  Benchmark.signaltest        2018-12-26_19:40:41    timslab:rpi3-2                 FAIL
  Benchmark.GLMark            2018-12-26_19:35:22    timslab:rpi3-2                 SKIP
  Benchmark.OpenSSL           2018-12-26_19:17:55    timslab:rpi3-2                 SKIP
  --------------------------- ---------------------- ------------------------------ ------

rpi3-1 LTP testcase problems [edit section]

FIXTHIS - these are from 1.3 release
root@tlinux:/# ftc gen-report --where "board=rpi3-1,test=LTP,tguid:result!=PASS,start_time>2 days ago"
where start_time value is '1.525060103e+12'
======================================================================
           **** Fuego Test Report ****
  test                : LTP
  board               : rpi3-1
  kernel              : 4.9.41-v7+
  timestamp           : 2018-05-01_01:02:42
  report_date         : 2018-05-02_03:48:23
======================================================================
--------------------------------------------------------------------------------------------------------------------------------
  test_name            spec                 board                timestamp            tguid                tguid:result
--------------------------------------------------------------------------------------------------------------------------------
  Functional.LTP       default              rpi3-1               2018-05-01_01:02:42  syscalls             FAIL
  Functional.LTP       default              rpi3-1               2018-05-01_01:02:42  syscalls.add_key04   FAIL
  Functional.LTP       default              rpi3-1               2018-05-01_01:02:42  syscalls.keyctl03    FAIL
  Functional.LTP       default              rpi3-1               2018-05-01_01:02:42  syscalls.keyctl02    FAIL
  Functional.LTP       default              rpi3-1               2018-05-01_01:02:42  syscalls.keyctl05    FAIL
  Functional.LTP       default              rpi3-1               2018-05-01_01:02:42  syscalls.keyctl04    FAIL
  Functional.LTP       default              rpi3-1               2018-05-01_01:02:42  syscalls.keyctl07    FAIL
  Functional.LTP       default              rpi3-1               2018-05-01_01:02:42  syscalls.keyctl06    FAIL
  Functional.LTP       default              rpi3-1               2018-05-01_01:02:42  syscalls.io_getevents01 FAIL
  Functional.LTP       default              rpi3-1               2018-05-01_01:02:42  syscalls.io_submit01 FAIL
  Functional.LTP       default              rpi3-1               2018-05-01_01:02:42  syscalls.fcntl36     FAIL
  Functional.LTP       default              rpi3-1               2018-05-01_01:02:42  syscalls.io_cancel01 FAIL
  Functional.LTP       default              rpi3-1               2018-05-01_01:02:42  syscalls.msgctl11    FAIL
  Functional.LTP       default              rpi3-1               2018-05-01_01:02:42  syscalls.eventfd01   FAIL
  Functional.LTP       default              rpi3-1               2018-05-01_01:02:42  syscalls.request_key02 FAIL
  Functional.LTP       default              rpi3-1               2018-05-01_01:02:42  syscalls.io_setup01  FAIL
  Functional.LTP       default              rpi3-1               2018-05-01_01:02:42  syscalls.io_destroy01 FAIL
  Functional.LTP       default              rpi3-1               2018-05-01_01:02:42  syscalls.add_key03   FAIL
  Functional.LTP       default              rpi3-1               2018-05-01_01:02:42  syscalls.request_key01 FAIL
  Functional.LTP       default              rpi3-1               2018-05-01_01:02:42  syscalls.request_key03 FAIL
  Functional.LTP       default              rpi3-1               2018-05-01_01:02:42  sem_init.sem_init_7-1 SKIP
--------------------------------------------------------------------------------------------------------------------------------

table of current test status [edit section]

See Release_test_results

triage data [edit section]

  • candidates
    • ren1.default.Functional.fuego_test_minimal
      • missing spec.json
    • fuego-test.default.Functional.fuego_release_test
      • urllib3 python module dependency problem
    • ren1.default.Functional.aiostress
      • generic_parser python exception
    • bbb.default.Benchmark.cyclictest (and min1, and signaltest)
      • add board-specific, config-specific, distro-specific criteria
    • rpi3-2.default.Functional.kernel_build
      • wrong kernel image name
    • bbb.default.Functional.aiostress
      • can't load libaio.so.1
      • maybe add dependency check on library (can you do aio without lib?)
  • research
    • rpi3-2.default.Functional.fuego_test_variables
      • if requires fuego-test board, then add dependency check to test
    • min1.default.Functional.LTP.syscalls.open10
      • see if it's xfail (expected for this kernel release)
      • add to board-specific skiplist?
    • min1.default.Functional.LTP.syscalls.creat08
    • ren1.default.Functional.kernel_build
      • see if it's a real kernel bug that need a patch upstream
      • depends on whether poky ren1 toolchain is deprecated upstream

TBWiki engine 1.8.2 by Tim Bird