Parser_module_API >> Release_1.5_testing_-_2019-05-02 

Fuego wiki

Login or create account

Release 1.5 testing - 2019-05-02

Here are some notes on testing the 1.4 release (using fuego-1211-container, with a rolling master branch through December 2018 and January 2019):

See an overview issues table at Release test results

to do [edit section]

  • review bugs labeled as FIXED in Release 1.4 testing - 2018-12-26
  • categorize all bugs
    • likely location: core, test, board, toolchain, lab, unknown
    • severity: high, medium, low
    • time to fix: short, long
    • priority: 5=high, 1=low, 0=defer

Resolution key:

  • CANDIDATE - is a candidate for fixing this release
  • RESEARCH - needs more research to categorize and prioritize
  • TRANSIENT - problem went away with subsequent test and no change in Fuego
  • LOCAL (lab,board) - problem was with setup, config, board status, board, or toolchain, and issue can be ignored
  • IGNORE - ignore (not a problem)
  • FIXED - problem fixed with change in Fuego
  • DEFER - problem exists, but will not be fixed in this release

uncategorized issues list [edit section]

deferred from 1.4 release [edit section]

  • 1. (test) LTP runs for too long with nothing showing on the console. Jenkins should show the current testcase being run, or something.
    • would require output from fuego_test.sh
  • 4. (core) Jenkins doesn't show different color for ERROR (timeout or abort)
  • 6. (core) bbb.default.Benchmark.cyclictest it's not easy to find the threshold value. It's not in run.json. It's on the chart, but with multiple boards it's hard to isolate and find
    • could add checkboxes for series.
    • could add reference value to run.json
    • could copy reference.log or reference.json to log directory
    • could have parser add reference check failure to console log
  • 7. (core) ftc rm-jobs rpi3-1.*.*.* - can't do rm-jobs outside the container! (annoying, but whatever)
    • required due to possible interactive confirmation
    • should be able to rm-jobs without being inside the container
    • should be able to do interactive commands even when forwarding to the container ftc
  • 9. RESEARCH (test) min1.default.Functional.LTP - new failure for open10
  • 10. RESEARCH (test) min1.default.Functional.LTP - new failure for creat08
  • 11. (core?) need to have test indicate whether error code indicates error or failure
    • example: x11perf - non-zero = error (not failure)
  • 16. CANDIDATE, HIGH (core) fuego-test.default.Functional.fuego_release_test
    • problem with Python 3 dependencies in docker project (requests requires urllib3, which is missing dependencies?)

    ...
     = ==== doing fuego phase: run =====
    -------------------------------------------------
    /usr/local/lib/python3.4/dist-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.9.1) or chardet (3.0.4) doesn't match a supported version!
      RequestsDependencyWarning)
    Traceback (most recent call last):
      File "/fuego-core/engine/tests/Functional.fuego_release_test/test_run.py", line 15, in <module>
        import docker
      File "/usr/local/lib/python3.4/dist-packages/docker/__init__.py", line 2, in <module>
        from .api import APIClient
      File "/usr/local/lib/python3.4/dist-packages/docker/api/__init__.py", line 2, in <module>
        from .client import APIClient
      File "/usr/local/lib/python3.4/dist-packages/docker/api/client.py", line 5, in <module>
        import requests
      File "/usr/local/lib/python3.4/dist-packages/requests/__init__.py", line 105, in <module>
        from urllib3.exceptions import DependencyWarning
    ImportError: cannot import name 'DependencyWarning'

  • 17. (board) rpi3-2.default.Functional.fuego_dependencies - - can't find kconfig
    • missing /proc/config.gz or /boot/config-4.9.41-v7+ (but has /lib/modules.../configs.ko)
  • 18. (test) rpi3-2.default.Functional.fuego_sleep - timed out
    • timeout from testplan_fuego_tests is too short
  • 19. (core) test that fails pre-check is shown as 'not run' in Jenkins
  • 20. (core) test that fails pre-check has no result code (listed as None) in 'ftc gen-report'
  • 22. (test) ren1.default.Functional.LTP_one_test.syscalls.add_key_02 - timeout is not converted to Fuego ERROR
    • the LTP parser should convert a timed-out test in LTP into a Fuego ERROR in the run.json file. It doesn't do this.

new errors [edit section]

  • xx. batch testcase names end in underscore in underscore
    • log_this should not put trailing space in hostlog.txt
    • or, have parser trim trailing whitespace (regex?) or python code?
  • xx. (core) docker.default.Functional.batch_nested - nested batch test has invalid job page link in hostlog.txt
    • shown as "Info:", when it should be '2' (see run 1)
        Running test Functional.batch_hello
        Log dir: logs/Functional.batch_hello/docker.Info:.Info:.Info:
        Job page: http://localhost:8090/fuego/job/docker.Info:.Functional.batch_hello/Info:/
        [[nested-4]] ok 2 - Functional.batch_hello
  • xx. (distro) docker.default.Functional.fuego_board_check - missing /sbin/reboot
    • /sbin/reboot is present on host as a symlink to /bin/systemctl
  • xx. ren1.default.Functional.hello_world (run 3) - problem importing filelock
    • python traceback in prepare_chart_data.py trying to import module 'filelock'
    • problem is in poky-aarch64-tools.sh - re-declaration of ORIG_PATH
  • xx. ftc query-board -b fuego_test - python error

root@timdesk:/fuego-rw# ftc query-board -b ftc-test Information for board: ftc-test

ARCHITECTURE : "arm" EXPAT_SUBTEST_COUNT_NEG : "41" EXPAT_SUBTEST_COUNT_POS : "1769" FUEGO_HOME : "/home/a" Traceback (most recent call last): File "/usr/local/bin/ftc", line 5275, in <module> main() File "/usr/local/bin/ftc", line 5260, in main do_query_board(bvars, options) File "/usr/local/bin/ftc", line 4203, in do_query_board if value.find('\n') == -1: AttributeError: 'list' object has no attribute 'find'

[edit section]

log_compare: pattern 'SUCCESS' found 1 times (expected greater or equal than 1) Traceback (most recent call last): File "/fuego-core/scripts/generic_parser.py", line 30, in <module> sys.exit(plib.process(results)) File "/fuego-core/scripts/parser/common.py", line 701, in process from prepare_chart_data import store_flat_results, make_chart_data File "/fuego-core/scripts/parser/prepare_chart_data.py", line 36, in <module> from filelock import FileLock ImportError: No module named filelock
  • need to update all -tools.sh files in ~/work/fuego-reference
  • need to make sure following are reviewed:
    • ./aarch64-tools.sh:ORIG_PATH=$PATH
    • ./debian-armhf-tools.sh:ORIG_PATH=$PATH
    • ./poky-qemuarm-tools.sh:ORIG_PATH=$PATH
    • ./lager-tools.sh:ORIG_PATH=$PATH
    • ./x86_64-tools.sh:ORIG_PATH=$PATH
    • ./debian-armel-tools.sh:ORIG_PATH=$PATH
  • xx. Can't easily navigate to LTP individual test logs in Jenkins interface

    old errors [edit section]

    • Benchmark.GLMark - build failure on multiple platforms (missing glew.h, missing libSDL)

    Errors by board [edit section]

    # ftc list-runs --where "start_time>4 days ago,status!=PASS,board=bbb"

    bbb board [edit section]

    • (old) bbb.default.Benchmark.GLMark - missing GL/glew.h
    • (old) bbb.default.Benchmark.Interbench - run error: hangs in Audio test
    • bbb.default.Benchmark.gtkperf - build error: configure error: checking for gtk+-2.0 gdk-2.0
    • bbb.default.Benchmark.X11perf - missing X libs
    • (RESEARCH) bbb.default.Functional.glib - missing gettext lib

    Runs on this host:
      Test                           Status
      --------------------------- -  ------
      Benchmark.Dhrystone            ERROR - fixed
      Benchmark.GLMark               SKIP - build error: GL/glew.h: No such file or directory
      Benchmark.Interbench           ERROR - run error: hangs in Audio test
      Benchmark.Java                 SKIP - no java: OK
      Benchmark.gtkperf              SKIP - build error: configure error: checking for gtk+-2.0 gdk-2.0 ...
      Benchmark.netpipe              SKIP - missing NPtcp in host container, missing netpipe-tcp in container
      Benchmark.x11perf              SKIP - missing X libs
      Functional.LTP                 FAIL - multiple failures
      Functional.bc                  ERROR
      Functional.fontconfig          SKIP
      Functional.ft2demos            SKIP
      Functional.fuego_abort         SKIP
      Functional.fuego_compare_reports FAIL - not configured
      Functional.fuego_ftc_test        FAIL
      Functional.fuego_lint            FAIL
      Functional.fuego_release_test    FAIL
      Functional.fuego_slow_log        FAIL
      Functional.fuego_test_variables  FAIL - expected
      Functional.glib                  SKIP
      Functional.hello_world           FAIL - expected (spec=fail)
    

    • CANDIDATE bbb.default.Benchmark.netpipe' - run error: missing NPtcp in docker container
      • /usr/bin/NPtcp is provided by Debian package netpipe-tcp
      • + add netpipe-tcp to Dockerfile

    min1 board [edit section]

    Here are the tests that had issues:
    # ftc list-runs --where board=min1,status!=PASS
    Runs on this host:
      Test                         Status
      ---------------------------  ------
      Benchmark.GLMark             SKIP - build errror, missing GL/glew.h (old)
      Benchmark.cyclictest         FAIL - non-rt kernel
      Benchmark.ffsb               SKIP - current utilization = 1844660474359607984128.00% (RESEARCH)
      Benchmark.gtkperf            SKIP - build error: configure error: checking for gtk+-2.0 gdk-2.0 (old)
      Benchmark.signaltest         FAIL - non-rt kernel
      Benchmark.x11perf            SKIP - runtime: unable to open display ':0
      Functional.LTP               FAIL - see below
      Functional.glib              SKIP - build error: missing gettext support (RESEARCH)
    


    Here are the LTP non-PASS results:

    # ftc gen-report --where board=min1,test=LTP,tguid:result!=PASS
    ======================================================================
               **** Fuego Test Report ****
      test                : LTP
      board               : min1
      kernel              : 4.15.0-33-generic
      timestamp           : 2019-05-03_12:57:07
      report_date         : 2019-05-08_00:30:28
    ======================================================================
    --------------------------------------------------------------------------------------
      test_name      spec    board timestamp           tguid                 tguid:result
    --------------------------------------------------------------------------------------
      Functional.LTP default min1  2019-05-03_12:57:07 syscalls              FAIL
      Functional.LTP default min1  2019-05-03_12:57:07 syscalls.fanotify06   FAIL
      Functional.LTP default min1  2019-05-03_12:57:07 syscalls.msgctl11     FAIL
      Functional.LTP default min1  2019-05-03_12:57:07 syscalls.open10       FAIL
      Functional.LTP default min1  2019-05-03_12:57:07 syscalls.creat08      FAIL
      Functional.LTP default min1  2019-05-03_12:57:07 sem_init.sem_init_7-1 SKIP
    --------------------------------------------------------------------------------------
    

    --- Here is the reason for each LTP failure (with some comments)

    ------------------------------------
     tguid                 tguid:result
    ------------------------------------
     syscalls.fanotify06   FAIL  - segfault
     syscalls.msgctl11     FAIL  - queue mismatch
     syscalls.open10       FAIL  - setgid not set
     syscalls.creat08      FAIL  - setgid not set
     sem_init.sem_init_7-1 SKIP  - ??? (RESEARCH)
    ------------------------------------
    

    ren1 board [edit section]

    Runs on this host:
      Test                        Status
      --------------------------- ------
      Benchmark.Dhrystone         FAIL - time too small (expected - but could fix test)
      Benchmark.GLMark            SKIP - missing SDL/SDL.h (old)
      Benchmark.Java              SKIP - pre_test: missing java (expected, ok)
      Benchmark.gtkperf           SKIP - gdk-pixbuf-query-loaders: command not found
      Benchmark.reboot            SKIP - runtime error: didn't capture reboot (RESEARCH)
      Benchmark.signaltest        FAIL - non-RT kernel
      Benchmark.x11perf           SKIP - build error: no package xmuu
      Functional.LTP              FAIL - see below
      Functional.aiostress        SKIP - pre_test: no /lib/*/libaio (ok?)
      Functional.crashme          FAIL - 0 .. 507, but expected 3000
      Functional.ft2demos         SKIP - build error: missing /usr/lib/libxcb.la (RESEARCH)
      Functional.glib             SKIP - missing gettext support (CANDIDATE)
      Functional.kernel_build     FAIL - missing "Kernel: arch/.* is ready". bzImage, no such file or directory
    


    Here is the LTP report for non-PASS tests:

    ======================================================================
               **** Fuego Test Report ****
      test                : LTP
      board               : ren1
      kernel              : 4.9.0-yocto-standard
      timestamp           : 2019-05-03_13:41:21
      report_date         : 2019-05-08_00:39:12
    ======================================================================
    ----------------------------------------------------------------------------------------
      test_name      spec    board timestamp           tguid                   tguid:result
    ----------------------------------------------------------------------------------------
      Functional.LTP default ren1  2019-05-03_13:41:21 syscalls                FAIL
      Functional.LTP default ren1  2019-05-03_13:41:21 syscalls.cve-2017-6951  FAIL
      Functional.LTP default ren1  2019-05-03_13:41:21 syscalls.prot_hsymlinks FAIL
      Functional.LTP default ren1  2019-05-03_13:41:21 syscalls.renameat202    FAIL
      Functional.LTP default ren1  2019-05-03_13:41:21 syscalls.add_key04      FAIL
      Functional.LTP default ren1  2019-05-03_13:41:21 syscalls.add_key02      FAIL
      Functional.LTP default ren1  2019-05-03_13:41:21 syscalls.keyctl08       FAIL
      Functional.LTP default ren1  2019-05-03_13:41:21 syscalls.keyctl03       FAIL
      Functional.LTP default ren1  2019-05-03_13:41:21 syscalls.keyctl02       FAIL
      Functional.LTP default ren1  2019-05-03_13:41:21 syscalls.keyctl05       FAIL
      Functional.LTP default ren1  2019-05-03_13:41:21 syscalls.keyctl04       FAIL
      Functional.LTP default ren1  2019-05-03_13:41:21 syscalls.keyctl07       FAIL
      Functional.LTP default ren1  2019-05-03_13:41:21 syscalls.keyctl06       FAIL
      Functional.LTP default ren1  2019-05-03_13:41:21 syscalls.msgctl09       FAIL
      Functional.LTP default ren1  2019-05-03_13:41:21 syscalls.msgctl08       FAIL
      Functional.LTP default ren1  2019-05-03_13:41:21 syscalls.fanotify06     FAIL
      Functional.LTP default ren1  2019-05-03_13:41:21 syscalls.utime02        FAIL
      Functional.LTP default ren1  2019-05-03_13:41:21 syscalls.mknod03        FAIL
      Functional.LTP default ren1  2019-05-03_13:41:21 syscalls.abort01        FAIL
      Functional.LTP default ren1  2019-05-03_13:41:21 syscalls.kill11         FAIL
      Functional.LTP default ren1  2019-05-03_13:41:21 syscalls.msgctl10       FAIL
      Functional.LTP default ren1  2019-05-03_13:41:21 syscalls.msgctl11       FAIL
      Functional.LTP default ren1  2019-05-03_13:41:21 syscalls.request_key02  FAIL
      Functional.LTP default ren1  2019-05-03_13:41:21 syscalls.open10         FAIL
      Functional.LTP default ren1  2019-05-03_13:41:21 syscalls.renameat201    FAIL
      Functional.LTP default ren1  2019-05-03_13:41:21 syscalls.creat08        FAIL
      Functional.LTP default ren1  2019-05-03_13:41:21 syscalls.add_key03      FAIL
      Functional.LTP default ren1  2019-05-03_13:41:21 syscalls.setsockopt02   FAIL
      Functional.LTP default ren1  2019-05-03_13:41:21 syscalls.cve-2017-5669  FAIL
      Functional.LTP default ren1  2019-05-03_13:41:21 syscalls.sendmsg01      FAIL
      Functional.LTP default ren1  2019-05-03_13:41:21 syscalls.request_key01  FAIL
      Functional.LTP default ren1  2019-05-03_13:41:21 syscalls.request_key03  FAIL
      Functional.LTP default ren1  2019-05-03_13:41:21 syscalls.setpriority01  FAIL
      Functional.LTP default ren1  2019-05-03_13:41:21 syscalls.utime01        FAIL
      Functional.LTP default ren1  2019-05-03_13:41:21 sem_init.sem_init_7-1   SKIP
    ----------------------------------------------------------------------------------------
    


    Here are LTP details:

    --------------------------------------
     tguid                   tguid:result
    --------------------------------------
    sem_init.sem_init_7-1   SKIP
     syscalls.abort01        FAIL - invalid argument
     syscalls.add_key02      FAIL - EINVAL
     syscalls.add_key03      FAIL - create keyring for another user
     syscalls.add_key04      FAIL - SIGSEGV
     syscalls.creat08        FAIL - setgid should be set
     syscalls.cve-2017-5669  FAIL - mapped VM addr in first 64Kb
     syscalls.cve-2017-6951  FAIL - timeout requesting dead key
     syscalls.fanotify06     FAIL - SIGSEGV
     syscalls.keyctl02       FAIL - timeout
     syscalls.keyctl03       FAIL - timeout
     syscalls.keyctl04       FAIL - timeout
     syscalls.keyctl05       FAIL - timeout
     syscalls.keyctl06       FAIL - timeout
     syscalls.keyctl07       FAIL - timeout
     syscalls.keyctl08       FAIL - timeout
     syscalls.kill11         FAIL - invalid argument to setrlimit
     syscalls.mknod03        FAIL - setgid bit not set
     syscalls.msgctl08       FAIL - ipcs not found
     syscalls.msgctl09       FAIL - ipcs not found
     syscalls.msgctl10       FAIL - ipcs not found
     syscalls.msgctl11       FAIL - ipcs not found
     syscalls.open10         FAIL - setgid bit not set
     syscalls.prot_hsymlinks FAIL - failed to run cmd: useradd hsym
     syscalls.renameat201    FAIL - Invalid argument
     syscalls.renameat202    FAIL - Invalid argument
     syscalls.request_key01  FAIL - timeout
     syscalls.request_key02  FAIL - timeout
     syscalls.request_key03  FAIL - timeout
     syscalls.sendmsg01      FAIL - ip/ifconfig not found
     syscalls.setpriority01  FAIL - useradd exited with non-zero code
     syscalls.setsockopt02   FAIL - allowed bad tp_sizeof_priv value
     syscalls.utime01        FAIL - tmp_file access and mod times not set
     syscalls.utime02        FAIL - tmp_file access and mod times not set
    


    fuego-test board [edit section]

    • (CANDIDATE) fuego-test.default.Functional.fuego_abort - reports SKIP, but should be OK (behavior is correct)
      • should turn this test into a nested test, that checks for correct messages in the console log
    • (CANDIDATE) fuego-test.default.fuego_board_check - missing /sbin/reboot in container
    • fuego-test.default.Functional.fuego_ftc_test - testcase 6 (ftc query-board differences)
    • fuego-test.default.Functional.fuego_lint - expected failures
    • fuego-test.default.Functional.fuego_release_test -
    • fuego-test.default.Functional.fuego_slow_log -
      Runs on this host:
        Test                          Status
        ---------------------------   ------
        Functional.fuego_abort        SKIP - OK (CANDIDATE)
        Functional.fuego_board_check  FAIL - missing /sbin/reboot
        Functional.fuego_ftc_test     FAIL - testcase 6, addition of ssh2serial, misc.
        Functional.fuego_lint         FAIL
        Functional.fuego_release_test FAIL
        Functional.fuego_slow_log     FAIL
      

  • TBWiki engine 1.8.3 by Tim Bird