{{TableOfContents}}
Here are some notes on testing the 1.6 release, using the next branch
(with new docker Debian version (bullseye) and Jenkins version (2.414.1)

The container is: fuego-322-container, corresponding to directory:
/home/tbird/work/fuego/next-2024-03-22/fuego

See an overview issues table at [[Release test results]]

to do [edit section]

= to do =
 - categorize all bugs
   - likely location: core, test, board, toolchain, lab, unknown
   - severity: high, medium, low
   - time to fix: short, long
   - priority: 5=high, 1=low, 0=defer

Resolution key:
 - CANDIDATE - is a candidate for fixing this release
 - RESEARCH - needs more research to categorize and prioritize
 - TRANSIENT - problem went away with subsequent test and no change in Fuego
 - LOCAL (lab,board) - problem was with setup, config, board status, board, or toolchain, and issue can be ignored
 - IGNORE - ignore (not a problem)
 - FIXED - problem fixed with change in Fuego
 - DEFER - problem exists, but will not be fixed in this release

uncategorized issues list [edit section]

= uncategorized issues list =
== deferred from 1.4 release ==
See http://fuegotest.org/wiki/Release_1.5_testing_-_2019-05-02#deferred_from_1.4_release

new errors [edit section]

== new errors ==
 - xx. (distro) '''docker.default.Functional.fuego_board_check''' - missing /sbin/reboot
    - /sbin/reboot is present on host as a symlink to /bin/systemctl
    - this is expected
 - xx. Can't easily navigate to LTP individual test logs in Jenkins interface

old errors [edit section]

= old errors =
 - [[Benchmark.GLMark]] - build failure on multiple platforms (missing glew.h, missing libSDL)

Errors by test [edit section]

= Errors by test =
 - Whetstone - (run phase, bbb & min1) libm.so.6: version `GLIBC_2.29' not found - probable toolchain problem? (toolchain too new for board)?
 - min1.default.Functional.netperf - build error
 0 Dhrystone - (run phase, rpi3-2) bad intepreter (error is 'dhrystone not found'), but the file is present.  Interpreter is /lib/ld-linux-armhf.so.3,
   but rpi3-2 has ld-linux /lib/ld-linux-aarch64.so.1
   - did I upgrade the distro on rpi3-2 to 64-bit?

low priority:
 - Dhrystone - (processing, docker & ren1) not enough loops (using spec '500M' works)

Errors by board [edit section]

= Errors by board =

bbb board [edit section]

== bbb board ==
# ftc list-runs --where "start_time>4 days ago,status!=PASS,board=bbb"

.. add data here ...
}}}

{{{
.. add data here ...
}}}

min1 board [edit section]

== min1 board ==
Here are the tests that had issues:

• min1.default.Benchmark.Whetstone - libm.so.6 version GLIBC_2.29 not found

 - min1.default.Benchmark.Whetstone - libm.so.6 version GLIBC_2.29 not found

# ftc list-runs --where board=min1,status!=PASS
... add data here...
}}}

{{{
# ftc list-runs --where board=min1,status!=PASS
... add data here...
}}}

----

Here are the LTP non-PASS results:
{{{
# ftc gen-report --where board=min1,test=LTP,tguid:result!=PASS
... add data here ...
}}}

---
Here is the reason for each LTP failure (with some comments)
{{{
UPDATE this table!!
------------------------------------
 tguid                 tguid:result 
------------------------------------
 syscalls.fanotify06   FAIL  - segfault       
 syscalls.msgctl11     FAIL  - queue mismatch       
 syscalls.open10       FAIL  - setgid not set
 syscalls.creat08      FAIL  - setgid not set        
 sem_init.sem_init_7-1 SKIP  - ??? (RESEARCH)       
------------------------------------
}}}

ren1 board [edit section]

== ren1 board ==

# ftc list-runs --where board=ren1,status!=PASS

..add data here ..
}}}

{{{
..add data here ..
}}}

----

Here is the LTP report for non-PASS tests:

# ftc gen-report --where board=ren1,test=LTP,tguid:result!=PASS

.. add data here ..
}}}

{{{
.. add data here ..
}}}

----

Here are LTP details:

To get sorted testcase names:
 * $ ftc gen-report --where board=ren1,test=LTP,tguid:result!=PASS --fields tguid,tguid:result | sort

.. add data here ..
}}}

{{{
.. add data here ..
}}}

fuego-test board [edit section]

== fuego-test board ==
UPDATE THIS DATA
 - (CANDIDATE) '''fuego-test.default.Functional.fuego_abort''' - reports SKIP, but should be OK (behavior is correct)
   - should turn this test into a nested test, that checks for correct messages in the console log
 - (CANDIDATE) '''fuego-test.default.fuego_board_check''' - missing /sbin/reboot in container
 - '''fuego-test.default.Functional.fuego_ftc_test''' - testcase 6 (ftc query-board differences)
 - '''fuego-test.default.Functional.fuego_lint''' - expected failures
 - '''fuego-test.default.Functional.fuego_release_test''' - 
 - '''fuego-test.default.Functional.fuego_slow_log''' - 

# ftc list-runs --where board=fuego-test,status!=PASS

UPDATE THIS DATA
Runs on this host:
  Test                          Status
  ---------------------------   ------
  Functional.fuego_abort        SKIP - OK (CANDIDATE)
  Functional.fuego_board_check  FAIL - missing /sbin/reboot
  Functional.fuego_compare_reports FAIL - missing baseline
  Functional.fuego_ftc_test     FAIL - (FIXED) testcase 3 - ftc help, testcase 6 - quoting of variables
  Functional.fuego_lint         FAIL - 559 errors in ftc (ok for now)
  Functional.fuego_release_test FAIL - (CANDIDATE) password is required
  Functional.fuego_slow_log     FAIL - (FIXED) syntax error near unexpected token done
}}}

{{{
UPDATE THIS DATA
Runs on this host:
  Test                          Status
  ---------------------------   ------
  Functional.fuego_abort        SKIP - OK (CANDIDATE)
  Functional.fuego_board_check  FAIL - missing /sbin/reboot
  Functional.fuego_compare_reports FAIL - missing baseline
  Functional.fuego_ftc_test     FAIL - (FIXED) testcase 3 - ftc help, testcase 6 - quoting of variables
  Functional.fuego_lint         FAIL - 559 errors in ftc (ok for now)
  Functional.fuego_release_test FAIL - (CANDIDATE) password is required
  Functional.fuego_slow_log     FAIL - (FIXED) syntax error near unexpected token done
}}}