Release 1.6 To Do

Contents:

priority
blocking bugs
candidate features/bugfixes list
release testing
item review
fix bugs
priorities for this release
future planned/priorities

deferred this release

previously deferred things

deferred from 1.5
deferred from 1.4
deferred from 1.3

Here are things to do for the 1.6 release:

priority [edit section]

(not done) update Release_1.6_Notes from git log
- see git log
(not done) Add items from release notes to documentation
decide major feature list for this release
- (not done) check which items on this list were completed
- finalize basic fserver features for this release
  - (done, may not be complete) add binary test packages to fserver
  - run test, get result, add to local visualization
- support board dependency cache
- add kselftest test
- fix LTP visualization bugs
- separate LTP posix test from main LTP test
- add a serial port monitor
- support provisioning
(not done) add major features
(not done) triage test results
- (not done) see Release 1.6 testing - 2023-xx-xx
(not done) find blocking bugs
(not done) fix blocking bugs
(not done) finish Release 1.6 Notes

blocking bugs [edit section]

Results table bugs:
- docker.bc-by2.Functional.bc job has results table title: docker-Functional.bc-default
- docker.bc-by2.Functional.bc job has results table with links to .../Functional.bc/docker.default.6.6/testlog.txt (completely wrong)

Jenkins errors:
- Data in older format - with error message:
  - NoClassDefFoundError: hudson/matrix/MatrixAggregatable
- Dependency error with plugins

candidate features/bugfixes list [edit section]

finish Release 1.6 Notes
- check ftc log for ftc improvements
- check fuego-core log for fuego-core improvements
- check fuego git log for fuego improvements
upgrade jenkins
- requires a new IPC from ftc
  - probably requires setting sshkey, and setting up a default 'fuego' user
finish features, or use existing features:
- batch tests
  - convert remaining testplans to batch tests:
    - testplan_agl.json
    - testplan_docker.json
    - testplan_fuego_*.json
    - testplan_usb_filesystem.json
    - testplan_mmc_filesystem
    - testplan_reboot.json
    - testplan_sata_filesystem.json
    - testplan_sata.json
    - testplan_usb_filesystem.json
  - add per-board criteria files for batch_default
    - eg. have bbb use fail_ok_list for GLMark, Interbench, etc.
  - add batch test skiplist mechanism
    - have run_test check for board skiplist, and skip test
      - need TAP13 syntax for a SKIP result
  - fix bug with link to sub-tests with different specs
- Fuego server (fserver)
  - (done) ftc list-tests --remote
    - (done) install latest fserver software on fuegotest.org
      - version is 0.6.0 (git top=18c8080)
- upstream serio improvements to output flow (don't buffer output on command)
  - check for discussion from March with Frank that stalled
make new batch tests (customize these in a sensible way):
- testplan_lava.json
- testplan_ltsi.json
- testplan_lts.json
Documentation:
- (not done) document new features
  - (not done) remote test (server)
- (not done) document new tests
  - try: echo "engine/tests/*" | xargs git-when-introduced.sh | sort -k 2 -V
  - try: for t in tests/* ; do git-when-introduced.sh engine/$t ; done | sort -k 2 -V

release testing [edit section]

Do release testing
- default test plan:
  - test bbb
  - test docker
  - test min1
  - test ren1
  - test rpi3-2
  - test localhost
  - test virtual host
- fuego test plan:
  - docker
  - fuego-test
  - ftc-test
  - bbb
  - min1
  - ren1
  - rpi3-2
triage test results:
- see Release 1.6 testing - 2019-xx-xx
- prioritize bugs
fix priority bugs

item review [edit section]

General:
- review items on the Fuego_Issues_List
- review items on the Fuego_Wiki_Fixthis_List
- review items on CodeFixthisList
- follow release check list on Release Process

fix bugs [edit section]

fix bugs:

DEFER ftc add-node help doesn't work (ftc add-nodes help does)
- make help work with singular command names

DEFER beaglebone-serial Functional.fuego_board_check has error in pre_test and then hangs:

Logged in as user root
echo Starting test beaglebone
ne-serial.default.Functional.fuego_board_check >> /tmp/fuego.Functional.fuego_boa
ard_check/fake-syslog' ; r=$?; echo '
ERROR: invalid literal for int() with base 10: '[ [&&serio_cmd_done&&]]'

batch bugs:
- ftc add-jobs -b bbb -t Functional.batch_hello from outside container, with container not running, shows "Import error: jenkins module not found", when it's actually a problem accessing the container? -

priorities for this release [edit section]

future planned/priorities [edit section]

support interactive run-test (with interactive creation of board file)

deferred this release [edit section]

previously deferred things [edit section]

Here is a list of previously deferred things that are still outstanding:

deferred from 1.5 [edit section]

fix LTP lack of console output during test
use nested test mechanisms:
- 'action' tests, to do: make_cache, make_report
- add parallel testing infrastructure (pipelines):
  - add monitor actions
  - add stress actions
- add provisioning actions
support remote test
- test scheduling?? (no)

deferred from 1.4 [edit section]

(1) full fserver support
- document how to use fserver
- how to use the existing fserver at fuegotest
- how to set up your own fserver

deferred from 1.3 [edit section]

add checkbox and replot to flot plots? (maybe not)
support fuego release self-test (probably defer to 1.4 release)
create pre-built docker image
more flake8 testing
- add check of parsing code to Functional.fuego_lint
add backfire kernel module build
parse LTP error descriptions
process rst into html for testcases, and have it appear in Jenkins interface
support board reboot (via BOARD_CONTROL) with another tool
- support lava, labgrid, r4d, or boruta (Pawel's) DUT control software?
support board provisioning (via BOARD_CONTROL)
- LTS provisioning support
remove or convert all chart_configs to proper files
ftc set-criteria ...
convert documentation to reStructureText
- create new user guide
- convert wiki architecture guide
- convert wiki developer reference materials