Fuego wiki

Login or create account

Release 1.5 To Do

Here are things to do for the 1.5 release:

priority [edit section]

blocking bugs [edit section]

  • Results table bugs:
    • docker.bc-by2.Functional.bc job has results table title: docker-Functional.bc-default
    • docker.bc-by2.Functional.bc job has results table with links to .../Functional.bc/docker.default.6.6/testlog.txt (completely wrong)

  • (FIXED) min1.default.Functional.LTP does not show min1-Functional.LTP results table (but it shows ren1 results table)
    • not getting called, due to ORIG_PATH handling

  • Jenkins errors:
    • Data in older format - with error message:
      • NoClassDefFoundError: hudson/matrix/MatrixAggregatable
    • Dependency error with plugins

general list [edit section]

  • (1) finish Release 1.5 Notes
  • finish features:
    • batch tests
      • convert remaining testplans to batch tests:
        • testplan_agl.json
        • (done) testplan_bc_*.json
        • (done) testplan_default.json
        • testplan_docker.json
        • testplan_fuego_*.json
        • (done) testplan_hello_world_*.json
        • testplan_lava.json
        • testplan_ltsi.json
        • testplan_lts.json
        • testplan_mmc_filesystem.json
        • testplan_reboot.json
        • testplan_sata_filesystem.json
        • testplan_sata.json
        • (done) testplan_smoketest.json
        • testplan_usb_filesystem.json
      • add per-board criteria files for batch_default
        • eg. have bbb use fail_ok_list for GLMark, Interbench, etc.
      • add batch test skiplist mechanism
        • have run_test check for board skiplist, and skip test
          • need TAP13 syntax for a SKIP result
      • fix bug with link to sub-tests with different specs
    • (done) squad backend prototype
    • (done) user-configurable Jenkins port
    • Fuego server (fserver)
      • ftc list-tests --remote
        • install latest fserver software on
    • serio improvements to output flow (don't buffer output on command)
      • check for discussion from March with Frank that stalled
  • Documentation:
    • document batch tests
      • (done) document run_test
      • (done) document allocate_next_batch_id
      • (done) creating jobs for batch tests
      • links in the batch results to job pages
      • generating reports for batch test results
    • document ftc put-run squad
      • configuration for squad server
    • document user configurable port
    • document default port 8090
    • document new tests
      • try: echo "engine/tests/*" | xargs | sort -k 2 -V
      • try: for t in tests/* ; do engine/$t ; done | sort -k 2 -V

  • Functional.linaro
    tests/Functional.fuego_batch1/ v1.4.0-21-g3dc481c
    tests/Functional.linaro/ v1.4.0-36-g6f5b706
    tests/Functional.nscd/ v1.4.0-41-g6781800
    tests/Functional.module_init_tools/ v1.4.0-43-g38cfbfb
    tests/Functional.batch_default/ v1.4.0-62-gfbfdfff
    tests/Functional.ipmi/ v1.4.0-65-gcee2f9c
    tests/Functional.openhpid/ v1.4.0-66-g2ab8b5c
    tests/Functional.openct/ v1.4.0-67-ga2ff574
    tests/Functional.brctl/ v1.4.0-68-gbd32eaa
    tests/Functional.multipathd/ v1.4.0-69-g8d1cf2e
    tests/Functional.ptest/ v1.4.0-71-g4cc8d3f
    tests/Functional.libxml/ v1.4.0-82-g019fc83
    tests/Functional.batch_bc/ v1.4.0-88-gbbfa30c
    tests/Functional.iperf3_server/ v1.4.0-89-g2dd180a
    tests/Functional.batch_smoketest/ v1.4.0-93-g82e2474
    tests/Functional.batch_hello/ v1.4.0-98-gab4a9ec
  • remote test (server)

    release testing [edit section]

    • Do release testing
      • default test plan:
        • test bbb
        • test docker
        • test min1
        • test ren1
        • test rpi3-2
        • test localhost
        • test virtual host
      • fuego test plan:
        • docker
        • fuego-test
        • ftc-test
        • bbb
        • min1
        • ren1
        • rpi3-2
    • triage test results:
    • fix priority bugs

    item review [edit section]

    fix bugs [edit section]

    • fix bugs:
      • DEFER ftc add-node help doesn't work (ftc add-nodes help does)
        • make help work with singular command names
      • DEFER beaglebone-serial Functional.fuego_board_check has error in pre_test and then hangs:
        Logged in as user root
        echo Starting test beaglebone
        ne-serial.default.Functional.fuego_board_check >> /tmp/fuego.Functional.fuego_boa
        ard_check/fake-syslog' ; r=$?; echo '
        ERROR: invalid literal for int() with base 10: '[ [&&serio_cmd_done&&]]'

    • batch bugs:
      • ftc add-jobs -b bbb -t Functional.batch_hello from outside container, with container not running, shows "Import error: jenkins module not found", when it's actually a problem accessing the container? -

    • things to commit

    priorities for this release [edit section]

    future planned/priorities [edit section]

    • support squad back-end
    • support interactive run-test (with interactive creation of board file)

    deferred this release [edit section]

    • depend on nested test mechanisms:
      • 'action' tests, to do: make_cache, make_report
      • monitor tests (needs parallel testing infrastructure)
      • stress tests (needs parallel testing infrastructure)
    • support remote test
      • test scheduling?? (no)
    • (1) full fserver support
      • document how to use fserver
      • how to use the existing fserver at fuegotest
      • how to set up your own fserver

    things done [edit section]

    previously deferred things [edit section]

    Here is a list of previously deferred things that are still outstanding:

    deferred from 1.3 [edit section]

    • add checkbox and replot to flot plots? (maybe not)
    • support fuego release self-test (probably defer to 1.4 release)
    • create pre-built docker image
    • more flake8 testing
      • add check of parsing code to Functional.fuego_lint
    • add backfire kernel module build
    • parse LTP error descriptions
    • process rst into html for testcases, and have it appear in Jenkins interface
    • support board reboot (via BOARD_CONTROL) with another tool
      • support lava, labgrid, r4d, or boruta (Pawel's) DUT control software?
    • support board provisioning (via BOARD_CONTROL)
      • LTS provisioning support
    • remove or convert all chart_configs to proper files
    • ftc set-criteria ...
    • convert documentation to reStructureText
      • create new user guide
      • convert wiki architecture guide
      • convert wiki developer reference materials

  • TBWiki engine 1.8.3 by Tim Bird