{{TableOfContents}}
Here are some notes on testing the 1.4 release (using fuego-1211-container,
with a rolling master branch through December 2018 and January 2019):

to do [edit section]

= to do =
 - categorize all bugs
   - likely location: core, test, board, toolchain, lab, unknown
   - severity: high, medium, low
   - time to fix: short, long
   - priority: 5=high, 1=low, 0=defer

Resolution key:
 - CANDIDATE - is a candidate for fixing this release
 - RESEARCH - needs more research to categorize and prioritize
 - TRANSIENT - problem went away with subsequent test and no change in Fuego
 - LOCAL (lab,board) - problem was with setup, config, board status, board, or toolchain, and issue can be ignored
 - IGNORE - ignore (not a problem)
 - FIXED - problem fixed with change in Fuego
 - DEFER - problem exists, but will not be fixed in this release

uncategorized issues list [edit section]

= uncategorized issues list =
 - 1. FIXED (core) '''bbb.default.Benchmark.Interbench''' -  timeout is reported as PASS in Jenkins
   - see build_number 1 - test hung and it took 40 minutes
 - 2. DEFER (test) LTP runs for too long with nothing showing on the console.  Jenkins should show the current testcase being run, or something.
   - would require output from fuego_test.sh
 - 3. FIXED (core) '''min1.default.Benchmark.OpenSSL''' - took 15 minutes (which timed out) but is reported as PASS in Jenkins
 - 4. DEFER (core) Jenkins doesn't show different color for ERROR (timeout or abort)
 - 5. FIXED (core) main.sh may return success even when the test was interrupted
   - e.g. when terminated with SIGTERM, to allow post-test to run and results are only partial
 - 6. DEFER (core) '''bbb.default.Benchmark.cyclictest''' it's not easy to find the threshold value.  It's not in run.json.  It's on the chart, but with multiple boards it's hard to isolate and find
   - could add checkboxes for series.
   - could add reference value to run.json
   - could copy reference.log or reference.json to log directory
   - could have parser add reference check failure to console log
 - 7. DEFER (core) '''ftc rm-jobs rpi3-1.*.*.*''' - can't do rm-jobs outside the container! (annoying, but whatever)
    - required due to possible interactive confirmation
    - should be able to rm-jobs without being inside the container
    - should be able to do interactive commands even when forwarding to the container ftc
 - 8. FIXED (core) '''min1.default.Benchmark.OpenSSL''' - has timeout of 15 minutes (should be default of 40, from what I can see)
   - was from 15m default timeout in testplan_smoketest.json
 - 9. RESEARCH (test) '''min1.default.Functional.LTP''' - new failure for open10
 - 10. RESEARCH (test)''' min1.default.Functional.LTP''' - new failure for creat08
 - 11. DEFER (core?) need to have test indicate whether error code indicates error or failure
   - example: x11perf - non-zero = error (not failure)
 - 12. FIXED (core) it was hard to add a default timeout to testplan_fuego_tests, because I couldn't find the schema for testplans very easily.  I didn't know the variable name.
   - added wiki page [[Testplan Reference]]
 - 13. FIXED (core) testplans usb, mmc and sata could have used 'default_spec' to save lines
 - 14. FIXED (core) '''fuego-test.default.Functional.fuego_dependencies'''
    - testcase 5 NEED_KCONFIG=CONFIG_PRINTK=y
    - problem finding /boot/config-$(uname -r) for need_KCONFIG
 - 15. FIXED (core) '''ren1.default.Functional.fuego_test_minimal'''
    - fails because of missing spec.json in ftc
 - 16. CANDIDATE, HIGH (core) '''fuego-test.default.Functional.fuego_release_test'''
    - problem with Python 3 dependencies in docker project (requests requires urllib3, which is missing dependencies?)

{{{#!YellowBox
 = ==== doing fuego phase: run =====
-------------------------------------------------
/usr/local/lib/python3.4/dist-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.9.1) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)
Traceback (most recent call last):
  File "/fuego-core/engine/tests/Functional.fuego_release_test/test_run.py", line 15, in <module>
    import docker
  File "/usr/local/lib/python3.4/dist-packages/docker/__init__.py", line 2, in <module>
    from .api import APIClient
  File "/usr/local/lib/python3.4/dist-packages/docker/api/__init__.py", line 2, in <module>
    from .client import APIClient
  File "/usr/local/lib/python3.4/dist-packages/docker/api/client.py", line 5, in <module>
    import requests
  File "/usr/local/lib/python3.4/dist-packages/requests/__init__.py", line 105, in <module>
    from urllib3.exceptions import DependencyWarning
ImportError: cannot import name 'DependencyWarning'
}}}

• 16. RESEARCH (unknown) rpi3-2.default.Functional.fuego_test_variables - testcases 2,3, and 9 - 17. DEFER (board) rpi3-2.default.Functional.fuego_dependencies - - can't find kconfig - missing /proc/config.gz or /boot/config-4.9.41-v7+ (but has /lib/modules.../configs.ko) - 18. DEFER (test) rpi3-2.default.Functional.fuego_sleep - timed out - timeout from testplan_fuego_tests is too short - 19. DEFER (core) test that fails pre-check is shown as 'not run' in Jenkins - 20. DEFER (core) test that fails pre-check has no result code (listed as None) in 'ftc gen-report' - 21. FIXED (core) ren1.default.Functional.aiostress - (post-processing) Invalid literal for base 10 in generic_parser - changed to a try here, and convert to ERROR in except clause - or figure out why RETURN_VALUE is missing!! - it's missing because: - the build error caused early exit - RETURN_VALUE was never set (by report, et al.) - this test doesn't have a test_processing function

 - 16. RESEARCH (unknown) '''rpi3-2.default.Functional.fuego_test_variables'''
    - testcases 2,3, and 9
 - 17. DEFER (board) '''rpi3-2.default.Functional.fuego_dependencies''' - - can't find kconfig
    - missing /proc/config.gz or /boot/config-4.9.41-v7+ (but has /lib/modules.../configs.ko)
 - 18. DEFER (test) '''rpi3-2.default.Functional.fuego_sleep''' - timed out
    - timeout from testplan_fuego_tests is too short
 - 19. DEFER (core) test that fails pre-check is shown as 'not run' in Jenkins
 - 20. DEFER (core) test that fails pre-check has no result code (listed as None) in 'ftc gen-report'
 - 21. FIXED (core) '''ren1.default.Functional.aiostress''' - (post-processing) Invalid literal for base 10 in generic_parser
    - changed to a try here, and convert to ERROR in except clause
    - or figure out why RETURN_VALUE is missing!!
      - it's missing because:
        - the build error caused early exit
        - RETURN_VALUE was never set (by report, et al.)
        - this test doesn't have a test_processing function

{{{#!YellowBox
##### doing fuego phase: processing (from signal handler) #####
WARNING: program returned exit code 
Log evaluation may be invalid
Traceback (most recent call last):
  File "/fuego-core/engine/scripts/generic_parser.py", line 18, in <module>
    value = "PASS" if int(RETURN_VALUE) == 0 else "FAIL"
ValueError: invalid literal for int() with base 10: ''
ERROR: results did not satisfy the threshold
}}}

• 22. DEFER (test) ren1.default.Functional.LTP_one_test.syscalls.add_key_02 - timeout is not converted to Fuego ERROR - the LTP parser should convert a timed-out test in LTP into a Fuego ERROR in the run.json file. It doesn't do this. - 21. FIXED (core) - ftc rm-node rpi3-1 - cannot remove node rpi3-1 - get message "global name 'install_nodes' is not defined" - 23. FIXED (core) - ftc rm-node rpi3-1 - has no effect - 24. FIXED (core) - ftc rm-jobs --remove-logs Bench.default.Benchmark.hackbench - gives weird message

 - 22. DEFER (test) '''ren1.default.Functional.LTP_one_test.syscalls.add_key_02''' - timeout is not converted to Fuego ERROR
    - the LTP parser should convert a timed-out test in LTP into a Fuego ERROR in the run.json file.  It doesn't do this.
 - 21. FIXED (core) - '''ftc rm-node rpi3-1''' - cannot remove node rpi3-1
   -  get message "global name 'install_nodes' is not defined"
 - 23. FIXED (core) - '''ftc rm-node rpi3-1''' - has no effect
 - 24. FIXED (core) - '''ftc rm-jobs --remove-logs Bench.default.Benchmark.hackbench'''
   - gives weird message

Test failures by board [edit section]

= Test failures by board =
Here are testplan_default failures for 2018-12-26
== bbb ==
Summary:
{{{
# ftc list-runs --where "start_time>4 days ago,status!=PASS,board=bbb"
Runs on this host:
  Test                        Started                host:board                     Status
  --------------------------- ---------------------- ------------------------------ ------
  Benchmark.Interbench        2018-12-26_18:00:01    timslab:bbb                    ERROR 
  Benchmark.OpenSSL           2018-12-26_18:50:36    timslab:bbb                    SKIP  
  Benchmark.cyclictest        2018-12-26_18:51:25    timslab:bbb                    FAIL  
  Benchmark.gtkperf           2018-12-26_19:01:27    timslab:bbb                    SKIP  
  Benchmark.x11perf           2018-12-26_18:49:56    timslab:bbb                    SKIP  
  Functional.LTP              2018-12-26_19:07:47    timslab:bbb                    FAIL  
  Functional.LTP              2018-12-26_21:20:47    timslab:bbb                    FAIL  
  Functional.OpenSSL          2018-12-26_18:56:31    timslab:bbb                    FAIL  
  Functional.aiostress        2018-12-26_23:06:06    timslab:bbb                    FAIL  
  Functional.fontconfig       2018-12-26_19:01:14    timslab:bbb                    FAIL  
  Functional.ft2demos         2018-12-26_19:00:25    timslab:bbb                    ERROR 
}}}

Analysis:
 - FIXED (test) bbb.default.Benchmark.Interbench - ERROR - timeout of 40m reported as SUCCESS
   - reported as success in Jenkins and ERROR in run.json
 - DEFER (test, board) bbb.default.Benchmark.Interbench - ERROR - test hangs!! (not new)
   - hangs in Audio test (not new, but should be resolved)

{{{#!YellowBox
last data reported from test is:
--- Benchmarking simulated cpu of Audio in the presence of simulated ---
Load	Latency +/- SD (ms)  Max Latency   % Desired CPU  % Deadlines Met
None	   0.1 +/- 0.1        0.3		 100	        100
Video	   0.1 +/- 0.1        0.2		 100	        100
X	   0.1 +/- 0.1        1.3		 100	        100
Burn	   0.0 +/- 0.0        0.1		 100	        100
Write	
calloc: Cannot allocate memory
   0.1 +/- 0.3        6.6		 100	        100
}}}

• DEFER (test,toolchain) bbb.default.Benchmark.OpenSSL - build failed - missing zconf.h from zlib (not new) {{{#!YellowBox arm-linux-gnueabihf-gcc -I.. -I../.. -I../asn1 -I../evp -I../../include \ --sysroot=/ -DZLIB_SHARED -DZLIB -DOPENSSL_THREADS -c -o c_zlib.o c_zlib.c In file included from c_zlib.c:25:0: /usr/include/zlib.h:34:19: fatal error: zconf.h: No such file or directory #include "zconf.h" ^ compilation terminated. }}}

 - DEFER (test,toolchain) '''bbb.default.Benchmark.OpenSSL''' - build failed - missing zconf.h from zlib (not new)
{{{#!YellowBox
arm-linux-gnueabihf-gcc -I.. -I../.. -I../asn1 -I../evp -I../../include  \
   --sysroot=/ -DZLIB_SHARED -DZLIB -DOPENSSL_THREADS     -c -o c_zlib.o c_zlib.c
In file included from c_zlib.c:25:0:
/usr/include/zlib.h:34:19: fatal error: zconf.h: No such file or directory
 #include "zconf.h"
                   ^
compilation terminated.
}}}

• CANDIDATE (board) bbb.default.Benchmark.cyclictest - FAIL - board doesn't have RT kernel (expected) - max_latency doesn't pass threshold test, for 1000ms reference value - fix would be to provide threshold for non-RT board - add bbb.Benchmark.cyclictest.criteria.json - DEFER (test?,toolchain?) bbb.default.Benchmark.gtkperf - build failure (not new) - build failed - can't build test - last message: checking for gtk+-2.0 gdk-2.0 - DEFER (toolchain) bbb.default.Benchmark.x11perf - build failure - build failed - cannot find -lXext -lXrender -lX11 -lXmu - DEFER (test) bbb.default.Functional.glib - build failure - swp in Thumb mode not supported (not new) - error was:

 - CANDIDATE (board) '''bbb.default.Benchmark.cyclictest''' - FAIL - board doesn't have RT kernel (expected)
   - max_latency doesn't pass threshold test, for 1000ms reference value
   - fix would be to provide threshold for non-RT board
     - add bbb.Benchmark.cyclictest.criteria.json
 - DEFER (test?,toolchain?) '''bbb.default.Benchmark.gtkperf''' - build failure (not new)
   - build failed - can't build test
     - last message: checking for gtk+-2.0 gdk-2.0
 - DEFER (toolchain) '''bbb.default.Benchmark.x11perf''' - build failure
   - build failed - cannot find -lXext -lXrender -lX11 -lXmu
 - DEFER (test) '''bbb.default.Functional.glib''' - build failure - swp in Thumb mode not supported (not new)
   - error was:

{{{#!YellowBox
/bin/sh ../libtool  --tag=CC   --mode=compile arm-linux-gnueabihf-gcc -DHAVE_CONFIG_H -I. -I.. \
   -I..  -DG_LOG_DOMAIN=\"GLib\" -DG_DISABLE_CAST_CHECKS -DG_DISABLE_DEPRECATED -DGLIB_COMPILATION \
   -DPCRE_STATIC -DG_DISABLE_SINGLE_INCLUDES -pthread  -g -O2 -Wall -MT gatomic.lo -MD -MP \
   -MF .deps/gatomic.Tpo -c -o gatomic.lo gatomic.c
libtool: compile:  arm-linux-gnueabihf-gcc -DHAVE_CONFIG_H -I. -I.. -I.. -DG_LOG_DOMAIN=\"GLib\" \
  -DG_DISABLE_CAST_CHECKS -DG_DISABLE_DEPRECATED -DGLIB_COMPILATION -DPCRE_STATIC \
  -DG_DISABLE_SINGLE_INCLUDES -pthread -g -O2 -Wall -MT gatomic.lo -MD -MP -MF .deps/gatomic.Tpo \
  -c gatomic.c  -fPIC -DPIC -o .libs/gatomic.o
/tmp/ccTcovfi.s: Assembler messages:
/tmp/ccTcovfi.s:65: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
/tmp/ccTcovfi.s:85: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
/tmp/ccTcovfi.s:188: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
/tmp/ccTcovfi.s:208: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
/tmp/ccTcovfi.s:310: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
/tmp/ccTcovfi.s:330: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
/tmp/ccTcovfi.s:432: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
/tmp/ccTcovfi.s:452: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
Makefile:994: recipe for target 'gatomic.lo' failed
}}}

• RESEARCH (test) bbb.default.Functional.LTP - FAIL - see details below - DEFER (test) bbb.default.Functional.OpenSSL - FAIL - missing zconf.h (not new) - FIXED (board) bbb.default.Functional.aiostress - FAIL - can't load libaio.so.1 on board (not new) - should be reported as ERROR (failure to run test program) - FIXED (board) bbb.default.Functional.aiostress - FAIL - can't load libaio.so.1 on board (not new) - should use dependency check for package libaio (not sure it will work with non-Debian) - should resolve with a dependency on libaio.so.1 on the board - does this test apply, if libaio.so is not present on the board?? - NO - libaio.so.1 is located in bbb:/lib/arm-linux-gnueabihf/libaio.so.1 - /lib/arm-linux-gnueabihf system shared lib path is in: /etc/ld.so.conf.d/arm-linux-gneuabihf.conf - can't build without libaio-dev in the SDK (libaio.h) - aio functions are supposed to be provided by C library - removed '-laio' from build line - can be resolved with 'apt-get install libaio1' on the board - FIXED (board) bbb.default.Functional.fontconfig - FAIL - missing fc-list (expected) - added assert_has_program to fuego_test.sh - DEFER bbb.default.Functional.ft2demos - ERROR - missing X libs (not new)

 - RESEARCH (test) '''bbb.default.Functional.LTP''' - FAIL - see details below
 - DEFER (test) '''bbb.default.Functional.OpenSSL'''  - FAIL - missing zconf.h (not new)
 - FIXED (board) '''bbb.default.Functional.aiostress''' - FAIL - can't load libaio.so.1 on board (not new)
   - should be reported as ERROR (failure to run test program)
 - FIXED (board) '''bbb.default.Functional.aiostress''' - FAIL - can't load libaio.so.1 on board (not new)
   - should use dependency check for package libaio (not sure it will work with non-Debian)
   - should resolve with a dependency on libaio.so.1 on the board
     - does this test apply, if libaio.so is not present on the board?? - NO
     - libaio.so.1 is located in bbb:/lib/arm-linux-gnueabihf/libaio.so.1
     - /lib/arm-linux-gnueabihf system shared lib path is in: /etc/ld.so.conf.d/arm-linux-gneuabihf.conf
   - can't build without libaio-dev in the SDK (libaio.h)
     - aio functions are supposed to be provided by C library
   - removed '-laio' from build line
   - can be resolved with 'apt-get install libaio1' on the board
 - FIXED (board) '''bbb.default.Functional.fontconfig''' - FAIL - missing fc-list (expected)
   - added assert_has_program to fuego_test.sh
 - DEFER bbb.default.Functional.ft2demos - ERROR - missing X libs (not new)

{{{#!YellowBox
libtool: link: arm-linux-gnueabihf-gcc -Wl,-L -Wl,//usr/lib -Wl,-lXau -Wl,-lXdmcp --sysroot / \
  -o /fuego-rw/buildzone/bbb.default.Functional.ft2demos-debian-armhf/ft2demos-2.3.6/bin/.libs/ftlint \
  /fuego-rw/buildzone/bbb.default.Functional.ft2demos-debian-armhf/ft2demos-2.3.6/obj/ftlint.o \
  /fuego-rw/buildzone/bbb.default.Functional.ft2demos-debian-armhf/ft2demos-2.3.6/obj/common.o  \
  ../freetype-2.3.6/objs/.libs/libfreetype.so -lm
/usr/lib/gcc-cross/arm-linux-gnueabihf/6/../../../../arm-linux-gnueabihf/bin/ld: cannot find -lXau
/usr/lib/gcc-cross/arm-linux-gnueabihf/6/../../../../arm-linux-gnueabihf/bin/ld: cannot find -lXdmcp
collect2: error: ld returned 1 exit status
Makefile:426: recipe for target '/fuego-rw/buildzone/bbb.default.Functional.ft2demos-debian-armhf/ft2demos-2.3.6/bin/ftlint' failed
make: *** [/fuego-rw/buildzone/bbb.default.Functional.ft2demos-debian-armhf/ft2demos-2.3.6/bin/ftlint] Error 1
}}}

• DEFER (test, toolchain) - bbb.default.Functional.glib - problem with thumb mode instructions

 - DEFER (test, toolchain) - '''bbb.default.Functional.glib''' - problem with thumb mode instructions

{{{#!YellowBox
libtool: compile:  arm-linux-gnueabihf-gcc -DHAVE_CONFIG_H -I. -I.. -I.. \
  -DG_LOG_DOMAIN=\"GLib\" -DG_DISABLE_CAST_CHECKS -DG_DISABLE_DEPRECATED -DGLIB_COMPILATION \
  -DPCRE_STATIC -DG_DISABLE_SINGLE_INCLUDES -pthread -g -O2 -Wall -MT gatomic.lo -MD -MP \
  -MF .deps/gatomic.Tpo -c gatomic.c  -fPIC -DPIC -o .libs/gatomic.o
/tmp/ccTcovfi.s: Assembler messages:
/tmp/ccTcovfi.s:65: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
/tmp/ccTcovfi.s:85: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
/tmp/ccTcovfi.s:188: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
/tmp/ccTcovfi.s:208: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
/tmp/ccTcovfi.s:310: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
/tmp/ccTcovfi.s:330: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
/tmp/ccTcovfi.s:432: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
/tmp/ccTcovfi.s:452: Error: selected processor does not support `swp r3,r5,[r4]' in Thumb mode
Makefile:994: recipe for target 'gatomic.lo' failed
make[4]: *** [gatomic.lo] Error 1
}}}

bbb LTP results [edit section]

=== bbb LTP results ===
{{{
# ftc gen-report --where "board=bbb,test=LTP,tguid:result!=PASS,start_time>9 days ago" | sort -k 5

  board               : bbb
  kernel              : 4.4.88-ti-r125
  report_date         : 2019-01-04_20:20:37
  test                : LTP
  timestamp           : 2018-12-26_21:20:47
--------------------------------------------------------------------------------------------------------------------------------
======================================================================
  test_name            spec                 board                timestamp            tguid                tguid:result         
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  sem_init.sem_init_7-1 SKIP                 
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls             FAIL                 
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.add_key03   FAIL                 
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.add_key04   FAIL                 
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.eventfd01   FAIL                 
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.fanotify06  FAIL                 
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.fcntl35     FAIL                 
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.fcntl35_64  FAIL                 
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.getrusage03 FAIL                 
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.io_cancel01 FAIL                 
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.io_destroy01 FAIL                 
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.io_getevents01 FAIL                 
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.io_setup01  FAIL                 
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.io_submit01 FAIL                 
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.keyctl02    FAIL                 
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.keyctl03    FAIL                 
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.keyctl04    FAIL                 
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.keyctl05    FAIL                 
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.keyctl06    FAIL                 
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.keyctl07    FAIL                 
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.msgctl10    FAIL                 
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.msgctl11    FAIL                 
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.request_key01 FAIL                 
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.request_key02 FAIL                 
  Functional.LTP       default              bbb                  2018-12-26_21:20:47  syscalls.request_key03 FAIL               
}}}

min1 [edit section]

== min1 ==
Summary:
{{{
$ ftc list-runs --where "start_time>yesterday,status!=PASS,board=min1"
  Test                        Started                host:board                     Status
  Benchmark.Java              2018-12-26_18:56:19    timslab:min1                   ERROR 
  Benchmark.OpenSSL           2018-12-26_19:36:18    timslab:min1                   ERROR 
  Benchmark.cyclictest        2018-12-26_19:51:52    timslab:min1                   FAIL  
  Benchmark.signaltest        2018-12-26_19:55:13    timslab:min1                   FAIL  
  Functional.LTP              2018-12-26_18:00:42    timslab:min1                   FAIL
  Functional.glib             2018-12-26_18:57:06    timslab:min1                   FAIL  
  Benchmark.GLMark            2018-12-26_18:55:04    timslab:min1                   SKIP  
  Benchmark.x11perf           2018-12-26_18:50:09    timslab:min1                   SKIP  
  --------------------------- ---------------------- ------------------------------ ------
}}}

Analysis:
 - DEFER (toolchain) '''min1.default.Benchmark.GLMark''' - ERROR - failed to build
   - missing glew.h file for build
   - should have been reported as an error
 - FIXED (test) '''min1.default.Benchmark.Java''' - ERROR - (board does not have java command (expected)
   - changed to 'assert_has_program java'
 - FIXED (core) '''min1.default.Benchmark.OpenSSL''' - interrupted at 15 minutes (according to timeout value), returned SUCCESS to Jenkins
   - changed so that ERROR is returned on timeouts
   - need to change timeout higher
 - CANDIDATE (board) '''min1.default.Benchmark.cyclictest''' - FAIL (board does not have RT kernel) (expected)
   - max latency over threshold
     - should be higher threshold?
   - add non-RT threshold??
 - CANDIDATE (board) '''min1.default.Benchmark.signaltest''' - FAIL (board does not have RT kernel) (expected)
   - max latency over threshold
     - should be higher threshold?
 - DEFER (test?) '''min1.default.Benchmark.x11perf''' - SKIP
   - x11perf: unable to open display ':0"
   - should have been reported as an error
 - RESEARCH (test) '''min1.default.Functional.LTP''' - FAIL - 10 failures in syscalls, 1 sem_init SKIP (was 9 failures in 1.3 - see details below)
 - DEFER (test?) '''min1.default.Functional.glib''' - FAIL
    - 'threadtests' does not report pass or failure.  Test appears to terminate prematurely? (see testlog.txt)
    - there are many GLib-CRITICAL reports for "Source ID ## was not found when attempting to remove it"
    - returns exit code 143

min1 LTP results [edit section]

=== min1 LTP results ===
{{{
# ftc gen-report --where "board=min1,test=LTP,tguid:result!=PASS"
======================================================================
           **** Fuego Test Report ****
  test                : LTP
  board               : min1
  kernel              : 4.15.0-33-generic
  timestamp           : 2018-12-26_18:00:42
  report_date         : 2018-12-28_19:42:54
======================================================================
--------------------------------------------------------------------------------------------------------------------------------
  test_name            spec                 board                timestamp            tguid                tguid:result         
--------------------------------------------------------------------------------------------------------------------------------
  Functional.LTP       default              min1                 2018-12-26_18:00:42  syscalls             FAIL                 
  Functional.LTP       default              min1                 2018-12-26_18:00:42  syscalls.io_getevents01 FAIL                 
  Functional.LTP       default              min1                 2018-12-26_18:00:42  syscalls.fanotify06  FAIL                 
  Functional.LTP       default              min1                 2018-12-26_18:00:42  syscalls.io_submit01 FAIL                 
  Functional.LTP       default              min1                 2018-12-26_18:00:42  syscalls.io_cancel01 FAIL                 
  Functional.LTP       default              min1                 2018-12-26_18:00:42  syscalls.msgctl11    FAIL                 
  Functional.LTP       default              min1                 2018-12-26_18:00:42  syscalls.eventfd01   FAIL                 
  Functional.LTP       default              min1                 2018-12-26_18:00:42  syscalls.open10      FAIL                 
  Functional.LTP       default              min1                 2018-12-26_18:00:42  syscalls.creat08     FAIL                 
  Functional.LTP       default              min1                 2018-12-26_18:00:42  syscalls.io_setup01  FAIL                 
  Functional.LTP       default              min1                 2018-12-26_18:00:42  syscalls.io_destroy01 FAIL                 
  Functional.LTP       default              min1                 2018-12-26_18:00:42  sem_init.sem_init_7-1 SKIP                 
--------------------------------------------------------------------------------------------------------------------------------

Different from 1.3:
 * RESEARCH min1.default.Functional.LTP.syscalls.open10 - FAIL
 * RESEARCH min1.default.Functional.LTP.syscalls.creat08 - FAIL

}}}

test that were run in 1.3 but not 1.4 release testing:
 - min1.default.Functional.aiostress
 - min1.default.Functional.kernel_build

ren1 [edit section]

== ren1 ==
Summary:
{{{
# ftc list-runs --where "board=ren1,start_time>yesterday,status!=PASS"
Runs on this host:
  Test                        Started                host:board                     Status
  --------------------------- ---------------------- ------------------------------ ------
  Benchmark.Dhrystone         2018-12-28_21:11:40    timslab:ren1                   FAIL  
  Benchmark.GLMark            2018-12-28_21:34:14    timslab:ren1                   SKIP  
  Benchmark.Java              2018-12-28_21:38:54    timslab:ren1                   ERROR 
  Benchmark.gtkperf           2018-12-28_21:09:41    timslab:ren1                   SKIP  
  Benchmark.signaltest        2018-12-28_21:10:53    timslab:ren1                   FAIL  
  Benchmark.x11perf           2018-12-28_21:58:42    timslab:ren1                   SKIP  
  Functional.LTP              2018-12-28_21:58:50    timslab:ren1                   FAIL  
  Functional.aiostress        2018-12-28_21:34:10    timslab:ren1                   ERROR 
  Functional.crashme          2018-12-28_21:31:30    timslab:ren1                   FAIL  
  Functional.ft2demos         2018-12-28_20:47:19    timslab:ren1                   ERROR 
  Functional.glib             2018-12-28_20:45:28    timslab:ren1                   FAIL  
}}}

Analysis:
  - DEFER (test, board) '''ren1.default.Benchmark.Dhrystone''' - FAIL  (needs 100M spec)
  - DEFER (test, toolchain) '''ren1.default.Benchmark.GLMark''' - SKIP (build failure - missing SDL.h)
  - FIXED (board) '''ren1.default.Benchmark.Java'''  - ERROR (missing 'java' on board) (expected) 
  - DEFER '''ren1.default.Benchmark.gtkperf''' - SKIP
    - (deploy error) sh: gdk-pixbuf-query-loaders: command not found
  - DEFER '''ren1.default.Benchmark.signaltest''' - FAIL  
  - DEFER '''ren1.default.Benchmark.x11perf''' - SKIP  
  - RESEARCH '''ren1.default.Functional.LTP''' - FAIL  - see details below
  - DEFER '''ren1.default.Functional.aiostress''' - ERROR (build error) (not new)
    - aiostress.c:58:20: fatal error: libaio.h: No such file or directory compilation terminated.
  - DEFER '''ren1.default.Functional.crashme''' - FAIL (not new)
     - results did not satisfy the threshold
       - log_compare "0 ... 3000"   (but test result was "0 ...   507")
  - FIXED '''ren1.default.Functional.ft2demos''' - ERROR (build error)
    - /fuego-core/engine/tests/Functional.ft2demos/fuego_test.sh: line 16: build_error: command not found
  - DEFER '''ren1.default.Functional.ft2demos''' - ERROR 
{{{#!YellowBox
     - libtool: link: cannot find the library `=/usr/lib/libxcb.la' or unhandled argument `=/usr/lib/libxcb.la'
Makefile:470: recipe for target '/fuego-rw/buildzone/ren1.default.Functional.ft2demos-poky-aarch64/ft2demos-2.3.6/bin/ftview' failed
   - make: *** [/fuego-rw/buildzone/ren1.default.Functional.ft2demos-poky-aarch64/ft2demos-2.3.6/bin/ftview] Error 1
}}}

• DEFER ren1.default.Functional.glib - FAIL (build error) (not new, same as 1.3) - 'threadtests' does not report pass or failure. Test appears to terminate prematurely? (see testlog.txt) - there are many GLib-CRITICAL reports for "Source ID ## was not found when attempting to remove it" - returns exit code 143

  - DEFER '''ren1.default.Functional.glib''' - FAIL (build error) (not new, same as 1.3)
    - 'threadtests' does not report pass or failure.  Test appears to terminate prematurely? (see testlog.txt)
    - there are many GLib-CRITICAL reports for "Source ID ## was not found when attempting to remove it"
    - returns exit code 143

• RESEARCH ren1.default.Functional.kernel_build - FAIL, build error on assembly instruction - problem with head.S assembly and poky arm64 toolchain (as)

  - RESEARCH '''ren1.default.Functional.kernel_build''' - FAIL, build error on assembly instruction
    - problem with head.S assembly and poky arm64 toolchain (as)

{{{#!YellowBox
  AS      arch/arm64/kernel/head.o
arch/arm64/kernel/head.S: Assembler messages:
arch/arm64/kernel/head.S:824: Error: missing ')'
arch/arm64/kernel/head.S:824: Error: missing ')'
arch/arm64/kernel/head.S:824: Error: missing ')'
arch/arm64/kernel/head.S:824: Error: unexpected characters following instruction at operand 2 -- `mov x2,#(2)|(2U<<(8))'
scripts/Makefile.build:391: recipe for target 'arch/arm64/kernel/head.o' failed
make[1]: *** [arch/arm64/kernel/head.o] Error 1
Makefile:1055: recipe for target 'arch/arm64/kernel' failed
make: *** [arch/arm64/kernel] Error 2
}}}

ren1 LTP results [edit section]

=== ren1 LTP results ===
{{{
# ftc gen-report --where "board=ren1,test=LTP,tguid:result!=PASS"
======================================================================
           **** Fuego Test Report ****
  test                : LTP
  board               : ren1
  kernel              : 4.9.0-yocto-standard
  timestamp           : 2018-12-26_21:38:12
  report_date         : 2018-12-28_23:12:46
======================================================================
-------   - (had an error about missing program 'build_error', but that was fixed: 
      - /fuego-core/engine/tests/Functional.ft2demos/fuego_test.sh: line 16: build_error: command not found)
-------------------------------------------------------------------------------------------------------------------------
  test_name            spec                 board                timestamp            tguid                tguid:result         
--------------------------------------------------------------------------------------------------------------------------------
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls             FAIL                 
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.cve-2017-6951 FAIL                 
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.prot_hsymlinks FAIL                 
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.renameat202 FAIL                 
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.add_key04   FAIL                 
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.add_key02   FAIL                 
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.keyctl08    FAIL                 
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.keyctl03    FAIL                 
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.keyctl02    FAIL                 
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.keyctl05    FAIL                 
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.keyctl04    FAIL                 
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.keyctl07    FAIL                 
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.keyctl06    FAIL                 
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.msgctl09    FAIL                 
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.msgctl08    FAIL                 
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.fanotify06  FAIL                 
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.utime02     FAIL                 
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.mknod03     FAIL                 
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.abort01     FAIL                 
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.recvmsg01   FAIL                 
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.kill11      FAIL                 
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.msgctl10    FAIL                 
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.msgctl11    FAIL                 
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.request_key02 FAIL                 
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.open10      FAIL                 
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.renameat201 FAIL                 
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.creat08     FAIL                 
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.add_key03   FAIL                 
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.setsockopt02 FAIL                 
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.cve-2017-5669 FAIL                 
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.sendmsg01   FAIL                 
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.request_key01 FAIL                 
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.request_key03 FAIL                 
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.setpriority01 FAIL                 
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  syscalls.utime01     FAIL                 
  Functional.LTP       default              ren1                 2018-12-26_21:38:12  sem_init.sem_init_7-1 SKIP                 
--------------------------------------------------------------------------------------------------------------------------------
}}}

Changed from 1.3 release testing were:
 - DEFER ren1.default.Functional.LTP.syscalls.add_key02 - failed in 1.4
 - IGNORE ren1.default.Functional.LTP.syscalls.fanotify08 - worked now (but failed in 1.3)
 - IGNORE ren1.default.Functional.LTP.syscalls.fanotify07 - worked now (but failed in 1.3)
 - IGNORE ren1.default.Functional.LTP.syscalls.perf_event_open02 - worked now (but failed in 1.3)
 - RESEARCH ren1.default.Functional.LTP.syscalls.mknod03 - failed in 1.4
 - RESEARCH ren1.default.Functional.LTP.syscalls.creat08 - failed in 1.4

• DEFER, HIGH (post 1.4) ren1.default.Functional.LTP.syscalls.x (for x in: add_key0[1234], keyctl0[2345678], request_key0[123]) (14 tests) - add these tests to 'should reboot after test' list for LTP - these are probably xfail?? for creating zombie processes

 - DEFER, HIGH (post 1.4) ren1.default.Functional.LTP.syscalls.x (for x in: add_key0[1234], keyctl0[2345678], request_key0[123]) (14 tests)
    - add these tests to 'should reboot after test' list for LTP
    - these are probably xfail?? for creating zombie processes

Also, during the test, several processes (add_key, keyctl, request_key) were stuck in uninterruptible sleep:
{{{
$ ps auxwww (trimmed)
24350 root      2860 S    /usr/sbin/dropbear -i -r /etc/dropbear/dropbear_rsa_host_key -B
24351 root      2908 S    sh -c { cd /home/fuego/fuego.Functional.LTP; TESTS="syscalls "; PTSTESTS="SEM "; \
  RTTESTS=""; . ./ltp_target_run.sh; echo $? > /tmp/22722-26579; } 2>&1 | tee /home/fuego/fuego.Functional.LTP/Functional.LTP.log
24352 root      2908 S    sh -c { cd /home/fuego/fuego.Functional.LTP; TESTS="syscalls "; PTSTESTS="SEM "; \
  RTTESTS=""; . ./ltp_target_run.sh; echo $? > /tmp/22722-26579; } 2>&1 | tee /home/fuego/fuego.Functional.LTP/Functional.LTP.log
24353 root      1760 S    tee /home/fuego/fuego.Functional.LTP/Functional.LTP.log
24377 root      3568 S    {runltp} /bin/sh ./runltp -C /home/fuego/fuego.Functional.LTP/result/syscalls/failed.log \
  -l /home/fuego/fuego.Functional.LTP/result/syscalls/result.log \
  -o /home/fuego/fuego.Functional.LTP/result/syscalls/output.log \
  -d /home/fuego/fuego.Functional.LTP/tmp -S ./skiplist.txt -f syscalls
24548 root      1888 S    /home/fuego/fuego.Functional.LTP/bin/ltp-pan -e -S -a 24377 -n 24377 \
  -f /home/fuego/fuego.Functional.LTP/tmp/ltp-T8KMbYu8JQ/alltests \
  -l /home/fuego/fuego.Functional.LTP/result/syscalls/result.log \
  -o /home/fuego/fuego.Functional.LTP/result/syscalls/output.log \
  -C /home/fuego/fuego.Functional.LTP/result/syscalls/failed.log \
  -T /home/fuego/fuego.Functional.LTP/output/LTP_RUN_ON-output.log.tconf
24692 root      1820 D    add_key01
24694 root      1820 D    add_key02
24696 root      2296 D    add_key03
24698 root      1824 D    add_key04
24753 root         0 SW   [kworker/u12:2]
26884 root      2004 D    keyctl02
26886 root      1820 D    keyctl03
26888 root      1820 D    keyctl04
26890 root      1824 D    keyctl05
26892 root      1820 D    keyctl06
26895 root      1824 D    keyctl07
26897 root      1820 D    keyctl08
27708 root      1820 D    request_key01
27710 root      1820 D    request_key02
27712 root      1824 D    request_key03
27714 root      1820 D    cve-2017-6951
32620 root         0 SW   [kworker/u12:0]
}}}

rpi3-2 [edit section]

== rpi3-2 ==
Summary:
{{{
cascade caused problems
}}}

Analysis:
 - DEFER '''rpi3-2.default.Benchmark.GLMark''' - SKIP - build error, missing glew.h
   - compilation problem with cpp code - missing GL header

n file included from vector.h:4:0,
                 from vector.cpp:1:
oglsdl.h:5:21: fatal error: GL/glew.h: No such file or directory
 #include <GL/glew.h>
                     ^
compilation terminated.
}}}
 - DEFER - '''rpi3-2.default.Benchmark.Interbench''' - times out (hangs)
   - hangs in Audio test (well, I stopped it)

{{{
n file included from vector.h:4:0,
                 from vector.cpp:1:
oglsdl.h:5:21: fatal error: GL/glew.h: No such file or directory
 #include <GL/glew.h>
                     ^
compilation terminated.
}}}
 - DEFER - '''rpi3-2.default.Benchmark.Interbench''' - times out (hangs)
   - hangs in Audio test (well, I stopped it)

• DEFER - rpi3-2.default.Benchmark.bonnie - FAIL (processing failure) - bug in parser for invalid values (probably ++++)

 - DEFER - '''rpi3-2.default.Benchmark.bonnie''' - FAIL (processing failure)
   - bug in parser for invalid values (probably ++++)

'===== doing fuego phase: processing =====
Reading current values from /fuego-rw/logs/Benchmark.bonnie/rpi3-2.default.1.1/testlog.txt

{{{
'===== doing fuego phase: processing =====
Reading current values from /fuego-rw/logs/Benchmark.bonnie/rpi3-2.default.1.1/testlog.txt

Bonnie++ raw results: ['1.97', '1.97', 'rpi3-2', '1', '1546056554', '32M', '', '126', '98', '9574', '6',
   '9197', '4', '613', '99', '+++++', '+++', '357.2', '5', '16', '', '', '', '', '4154', '19', '+++++', '+++',
   '4582', '29', '3820', '17', '+++++', '+++', '7129', '45', '120ms', '344us', '568us', '18744us', '108us',
   '20117us', '59893us', '3679us', '5008us', '566us', '175us', '895us']
Traceback (most recent call last):
  File "/fuego-core/engine/tests/Benchmark.bonnie/parser.py", line 36, in <module>
    measurements["Sequential_Output.PerChr"].append({"name": "speed", "measure" : float(results[2])})
ValueError: could not convert string to float: rpi3-2
ERROR: results did not satisfy the threshold
}}}

• DEFER rpi3-2.default.Benchmark.gtkperf - SKIP build error - problem in configure - checking for gtk+-2.0 gdk-2.0 .. - DEFER rpi3-2.default.Functional.glib - ERROR - build error - compiling gatomic, swp instructions in Thumb mode - FIXED (board, spec?) - rpi3-2.default.Functional.kernel_build - ERROR (no bzImage, but zImage is present) - no bzImage, but zImage is present - kernel make always defaults to building 'zImage' if the target is not recognized 'make foo' makes a zImage. - fixed by adding FUNCTIONAL_KERNEL_BUILD_TARGET="zImage" to rpi3-2.board - this is fragile, as the spec value overrides 'target'. But by default no 'target' is specified. - would be better to use "TARGET_KERNEL_IMAGE" in board, and if specified, override the target from the spec?

 - DEFER '''rpi3-2.default.Benchmark.gtkperf''' - SKIP build error
   - problem in configure - checking for gtk+-2.0 gdk-2.0 ..
 - DEFER '''rpi3-2.default.Functional.glib''' - ERROR - build error
   - compiling gatomic, swp instructions in Thumb mode
 - FIXED (board, spec?) - '''rpi3-2.default.Functional.kernel_build''' - ERROR (no bzImage, but zImage is present)
   - no bzImage, but zImage is present
   - kernel make always defaults to building 'zImage' if the target is not recognized 'make foo' makes a zImage.
   - fixed by adding FUNCTIONAL_KERNEL_BUILD_TARGET="zImage" to rpi3-2.board
     - this is fragile, as the spec value overrides 'target'.  But by default no 'target' is specified.
     - would be better to use "TARGET_KERNEL_IMAGE" in board, and if specified, override the target from the spec?

• FIXED (core) - rpi3-2.default.Functional.kernel_build - - got this this from generic_parser.py:

 - FIXED (core) - '''rpi3-2.default.Functional.kernel_build''' -
   - got this this from generic_parser.py:

{{{#!YellowBox
#### doing fuego phase: processing (from signal handler) #####
Processing kernel build log
log_compare: pattern '[ \t]*Kernel: arch/. arch/.. arch/.gitignore is ready' \
  found 2 times (expected greater or equal than 1)
Traceback (most recent call last):
  File "/fuego-core/engine/scripts/generic_parser.py", line 18, in <module>
    value = "PASS" if int(RETURN_VALUE) == 0 else "FAIL"
ValueError: invalid literal for int() with base 10: ''
}}}

Overview [edit section]

= Overview =

  root@tlinux:/# # ftc list-runs --where "start_time>yesterday,status!=PASS" | sort -k 3
  Test                        Started                host:board                     Status
  Benchmark.Interbench        2018-12-26_18:00:01    timslab:bbb                    ERROR 
  Functional.LTP              2018-12-26_21:20:47    timslab:bbb                    ERROR 
  Functional.ft2demos         2018-12-26_19:00:25    timslab:bbb                    ERROR 
  Benchmark.cyclictest        2018-12-26_18:51:25    timslab:bbb                    FAIL  
  Functional.LTP              2018-12-26_19:07:47    timslab:bbb                    FAIL  
  Functional.OpenSSL          2018-12-26_18:56:31    timslab:bbb                    FAIL  
  Functional.fontconfig       2018-12-26_19:01:14    timslab:bbb                    FAIL  
  Benchmark.OpenSSL           2018-12-26_18:50:36    timslab:bbb                    SKIP  
  Benchmark.gtkperf           2018-12-26_19:01:27    timslab:bbb                    SKIP  
  Benchmark.x11perf           2018-12-26_18:49:56    timslab:bbb                    SKIP

{{{
  root@tlinux:/# # ftc list-runs --where "start_time>yesterday,status!=PASS" | sort -k 3
  Test                        Started                host:board                     Status
  Benchmark.Interbench        2018-12-26_18:00:01    timslab:bbb                    ERROR 
  Functional.LTP              2018-12-26_21:20:47    timslab:bbb                    ERROR 
  Functional.ft2demos         2018-12-26_19:00:25    timslab:bbb                    ERROR 
  Benchmark.cyclictest        2018-12-26_18:51:25    timslab:bbb                    FAIL  
  Functional.LTP              2018-12-26_19:07:47    timslab:bbb                    FAIL  
  Functional.OpenSSL          2018-12-26_18:56:31    timslab:bbb                    FAIL  
  Functional.fontconfig       2018-12-26_19:01:14    timslab:bbb                    FAIL  
  Benchmark.OpenSSL           2018-12-26_18:50:36    timslab:bbb                    SKIP  
  Benchmark.gtkperf           2018-12-26_19:01:27    timslab:bbb                    SKIP  
  Benchmark.x11perf           2018-12-26_18:49:56    timslab:bbb                    SKIP  

  Benchmark.Java              2018-12-26_18:56:19    timslab:min1                   ERROR 
  Benchmark.OpenSSL           2018-12-26_19:36:18    timslab:min1                   ERROR 
  Benchmark.cyclictest        2018-12-26_19:51:52    timslab:min1                   FAIL  
  Benchmark.signaltest        2018-12-26_19:55:13    timslab:min1                   FAIL  
  Functional.LTP              2018-12-26_18:00:42    timslab:min1                   FAIL  
  Functional.glib             2018-12-26_18:57:06    timslab:min1                   FAIL  
  Benchmark.GLMark            2018-12-26_18:55:04    timslab:min1                   SKIP  
  Benchmark.x11perf           2018-12-26_18:50:09    timslab:min1                   SKIP

  Functional.LTP              2018-12-26_21:38:12    timslab:ren1                   ERROR 
  Functional.netperf          2018-12-26_18:16:13    timslab:ren1                   ERROR 
  Benchmark.signaltest        2018-12-26_18:15:59    timslab:ren1                   FAIL  
  Functional.glib             2018-12-26_18:33:38    timslab:ren1                   FAIL  

  Benchmark.Whetstone         2018-12-26_19:09:31    timslab:rpi3-2                 ERROR 
  Benchmark.tiobench          2018-12-26_19:08:28    timslab:rpi3-2                 ERROR 
  Functional.ft2demos         2018-12-26_18:50:43    timslab:rpi3-2                 ERROR 
  Functional.kernel_build     2018-12-26_18:38:15    timslab:rpi3-2                 ERROR 
  Functional.linus_stress     2018-12-26_18:43:44    timslab:rpi3-2                 ERROR 
  Benchmark.signaltest        2018-12-26_19:40:41    timslab:rpi3-2                 FAIL  
  Benchmark.GLMark            2018-12-26_19:35:22    timslab:rpi3-2                 SKIP  
  Benchmark.OpenSSL           2018-12-26_19:17:55    timslab:rpi3-2                 SKIP  
  --------------------------- ---------------------- ------------------------------ ------

}}}

rpi3-1 LTP testcase problems [edit section]

== rpi3-1 LTP testcase problems ==
FIXTHIS - these are from 1.3 release
{{{
root@tlinux:/# ftc gen-report --where "board=rpi3-1,test=LTP,tguid:result!=PASS,start_time>2 days ago"
where start_time value is '1.525060103e+12'
======================================================================
           **** Fuego Test Report ****
  test                : LTP
  board               : rpi3-1
  kernel              : 4.9.41-v7+
  timestamp           : 2018-05-01_01:02:42
  report_date         : 2018-05-02_03:48:23
======================================================================
--------------------------------------------------------------------------------------------------------------------------------
  test_name            spec                 board                timestamp            tguid                tguid:result         
--------------------------------------------------------------------------------------------------------------------------------
  Functional.LTP       default              rpi3-1               2018-05-01_01:02:42  syscalls             FAIL                 
  Functional.LTP       default              rpi3-1               2018-05-01_01:02:42  syscalls.add_key04   FAIL                 
  Functional.LTP       default              rpi3-1               2018-05-01_01:02:42  syscalls.keyctl03    FAIL                 
  Functional.LTP       default              rpi3-1               2018-05-01_01:02:42  syscalls.keyctl02    FAIL                 
  Functional.LTP       default              rpi3-1               2018-05-01_01:02:42  syscalls.keyctl05    FAIL                 
  Functional.LTP       default              rpi3-1               2018-05-01_01:02:42  syscalls.keyctl04    FAIL                 
  Functional.LTP       default              rpi3-1               2018-05-01_01:02:42  syscalls.keyctl07    FAIL                 
  Functional.LTP       default              rpi3-1               2018-05-01_01:02:42  syscalls.keyctl06    FAIL                 
  Functional.LTP       default              rpi3-1               2018-05-01_01:02:42  syscalls.io_getevents01 FAIL                 
  Functional.LTP       default              rpi3-1               2018-05-01_01:02:42  syscalls.io_submit01 FAIL                 
  Functional.LTP       default              rpi3-1               2018-05-01_01:02:42  syscalls.fcntl36     FAIL                 
  Functional.LTP       default              rpi3-1               2018-05-01_01:02:42  syscalls.io_cancel01 FAIL                 
  Functional.LTP       default              rpi3-1               2018-05-01_01:02:42  syscalls.msgctl11    FAIL                 
  Functional.LTP       default              rpi3-1               2018-05-01_01:02:42  syscalls.eventfd01   FAIL                 
  Functional.LTP       default              rpi3-1               2018-05-01_01:02:42  syscalls.request_key02 FAIL                 
  Functional.LTP       default              rpi3-1               2018-05-01_01:02:42  syscalls.io_setup01  FAIL                 
  Functional.LTP       default              rpi3-1               2018-05-01_01:02:42  syscalls.io_destroy01 FAIL                 
  Functional.LTP       default              rpi3-1               2018-05-01_01:02:42  syscalls.add_key03   FAIL                 
  Functional.LTP       default              rpi3-1               2018-05-01_01:02:42  syscalls.request_key01 FAIL                 
  Functional.LTP       default              rpi3-1               2018-05-01_01:02:42  syscalls.request_key03 FAIL                 
  Functional.LTP       default              rpi3-1               2018-05-01_01:02:42  sem_init.sem_init_7-1 SKIP                 
--------------------------------------------------------------------------------------------------------------------------------
}}}

table of current test status [edit section]

= table of current test status =
See [[Release_test_results]]

triage data [edit section]

= triage data =
     - candidates
       - ren1.default.Functional.fuego_test_minimal
         - missing spec.json
       - fuego-test.default.Functional.fuego_release_test
         - urllib3 python module dependency problem
       - ren1.default.Functional.aiostress
         - generic_parser python exception
       - bbb.default.Benchmark.cyclictest  (and min1, and signaltest)
         - add board-specific, config-specific, distro-specific criteria
       - rpi3-2.default.Functional.kernel_build
         - wrong kernel image name
       - bbb.default.Functional.aiostress
         - can't load libaio.so.1
         - maybe add dependency check on library (can you do aio without lib?)
     - research
       - rpi3-2.default.Functional.fuego_test_variables
         - if requires fuego-test board, then add dependency check to test
       - min1.default.Functional.LTP.syscalls.open10
         - see if it's xfail (expected for this kernel release)
         - add to board-specific skiplist?
       - min1.default.Functional.LTP.syscalls.creat08
       - ren1.default.Functional.kernel_build
         - see if it's a real kernel bug that need a patch upstream
         - depends on whether poky ren1 toolchain is deprecated upstream