Contents: priority blocking bugs general list release testing item review fix bugs priorities for this release future planned/priorities deferred this release things done previously deferred things deferred from 1.3

Here are things to do for the 1.5 release:

priority [edit section]

• fix blocking bugs
• (done) finish Release 1.5 Notes
• triage test results
  • (done) see Release 1.5 testing - 2019-05-02
  • (done) see Release 1.5 testing - 2019-08-13
• copy files from fuego-1.5 wiki that are different

blocking bugs [edit section]

• Results table bugs:
  • docker.bc-by2.Functional.bc job has results table title: docker-Functional.bc-default
  • docker.bc-by2.Functional.bc job has results table with links to .../Functional.bc/docker.default.6.6/testlog.txt (completely wrong)

• (FIXED) min1.default.Functional.LTP does not show min1-Functional.LTP results table (but it shows ren1 results table)
  • ltp_process.py not getting called, due to ORIG_PATH handling

• Jenkins errors:
  • Data in older format - with error message:
    • NoClassDefFoundError: hudson/matrix/MatrixAggregatable
  • Dependency error with plugins

general list [edit section]

• (1) finish Release 1.5 Notes
• finish features:
  • batch tests
    • convert remaining testplans to batch tests:
      • testplan_agl.json
      • (done) testplan_bc_*.json
      • (done) testplan_default.json
      • testplan_docker.json
      • testplan_fuego_*.json
      • (done) testplan_hello_world_*.json
      • testplan_lava.json
      • testplan_ltsi.json
      • testplan_lts.json
      • testplan_mmc_filesystem.json
      • testplan_reboot.json
      • testplan_sata_filesystem.json
      • testplan_sata.json
      • (done) testplan_smoketest.json
      • testplan_usb_filesystem.json
    • add per-board criteria files for batch_default
      • eg. have bbb use fail_ok_list for GLMark, Interbench, etc.
    • add batch test skiplist mechanism
      • have run_test check for board skiplist, and skip test
        • need TAP13 syntax for a SKIP result
    • fix bug with link to sub-tests with different specs
  • (done) squad backend prototype
  • (done) user-configurable Jenkins port
  • Fuego server (fserver)
    • ftc list-tests --remote
      • install latest fserver software on fuegotest.org
  • serio improvements to output flow (don't buffer output on command)
    • check for discussion from March with Frank that stalled
• Documentation:
  • document batch tests
    • (done) document run_test
    • (done) document allocate_next_batch_id
    • (done) creating jobs for batch tests
    • links in the batch results to job pages
    • generating reports for batch test results
  • document ftc put-run squad
    • configuration for squad server
  • document user configurable port
  • document default port 8090
  • document new tests
    • try: echo "engine/tests/*" | xargs git-when-introduced.sh | sort -k 2 -V
    • try: for t in tests/* ; do git-when-introduced.sh engine/$t ; done | sort -k 2 -V

• Functional.linaro

  tests/Functional.fuego_batch1/fuego_test.sh: v1.4.0-21-g3dc481c
  tests/Functional.linaro/fuego_test.sh: v1.4.0-36-g6f5b706
  tests/Functional.nscd/fuego_test.sh: v1.4.0-41-g6781800
  tests/Functional.module_init_tools/fuego_test.sh: v1.4.0-43-g38cfbfb
  tests/Functional.batch_default/fuego_test.sh: v1.4.0-62-gfbfdfff
  tests/Functional.ipmi/fuego_test.sh: v1.4.0-65-gcee2f9c
  tests/Functional.openhpid/fuego_test.sh: v1.4.0-66-g2ab8b5c
  tests/Functional.openct/fuego_test.sh: v1.4.0-67-ga2ff574
  tests/Functional.brctl/fuego_test.sh: v1.4.0-68-gbd32eaa
  tests/Functional.multipathd/fuego_test.sh: v1.4.0-69-g8d1cf2e
  tests/Functional.ptest/fuego_test.sh: v1.4.0-71-g4cc8d3f
  tests/Functional.libxml/fuego_test.sh: v1.4.0-82-g019fc83
  tests/Functional.batch_bc/fuego_test.sh: v1.4.0-88-gbbfa30c
  tests/Functional.iperf3_server/fuego_test.sh: v1.4.0-89-g2dd180a
  tests/Functional.batch_smoketest/fuego_test.sh: v1.4.0-93-g82e2474
  tests/Functional.batch_hello/fuego_test.sh: v1.4.0-98-gab4a9ec
  
  

remote test (server)

release testing [edit section]

• Do release testing
  • default test plan:
    • test bbb
    • test docker
    • test min1
    • test ren1
    • test rpi3-2
    • test localhost
    • test virtual host
  • fuego test plan:
    • docker
    • fuego-test
    • ftc-test
    • bbb
    • min1
    • ren1
    • rpi3-2
• triage test results:
  • see Release 1.5 testing - 2019-05-02
  • prioritize bugs
• fix priority bugs

item review [edit section]

• General:
  • review items on the Fuego_Issues_List
  • review items on the Fuego_Wiki_Fixthis_List
  • review items on CodeFixthisList
  • follow release check list on Release Process

fix bugs [edit section]

• fix bugs:
  • DEFER ftc add-node help doesn't work (ftc add-nodes help does)
    • make help work with singular command names
  • DEFER beaglebone-serial Functional.fuego_board_check has error in pre_test and then hangs:

    Logged in as user root
    echo Starting test beaglebone
    ne-serial.default.Functional.fuego_board_check >> /tmp/fuego.Functional.fuego_boa
    ard_check/fake-syslog' ; r=$?; echo '
    ERROR: invalid literal for int() with base 10: '[ [&&serio_cmd_done&&]]'
    

• batch bugs:
  • ftc add-jobs -b bbb -t Functional.batch_hello from outside container, with container not running, shows "Import error: jenkins module not found", when it's actually a problem accessing the container? -

• things to commit

priorities for this release [edit section]

future planned/priorities [edit section]

• support squad back-end
• support interactive run-test (with interactive creation of board file)

deferred this release [edit section]

• depend on nested test mechanisms:
  • 'action' tests, to do: make_cache, make_report
  • monitor tests (needs parallel testing infrastructure)
  • stress tests (needs parallel testing infrastructure)
• support remote test
  • test scheduling?? (no)
• (1) full fserver support
  • document how to use fserver
  • how to use the existing fserver at fuegotest
  • how to set up your own fserver

things done [edit section]

previously deferred things [edit section]

Here is a list of previously deferred things that are still outstanding:

deferred from 1.3 [edit section]

• add checkbox and replot to flot plots? (maybe not)
• support fuego release self-test (probably defer to 1.4 release)
• create pre-built docker image
• more flake8 testing
  • add check of parsing code to Functional.fuego_lint
• add backfire kernel module build
• parse LTP error descriptions
• process rst into html for testcases, and have it appear in Jenkins interface
• support board reboot (via BOARD_CONTROL) with another tool
  • support lava, labgrid, r4d, or boruta (Pawel's) DUT control software?
• support board provisioning (via BOARD_CONTROL)
  • LTS provisioning support
• remove or convert all chart_configs to proper files
• ftc set-criteria ...
• convert documentation to reStructureText
  • create new user guide
  • convert wiki architecture guide
  • convert wiki developer reference materials