Remote testing proposal

Contents:

Introduction
Tasks
Instructions for configuring ppt lab for remote execution
Running poll_requests.sh
Bugs encountered during testing

Bugs fixed during testing
Debug and implementation notes

Desired operational flow
To Do for remote testing features/fixes work
Features that work
NOTES from calls

2020-10-08/09 call

Resources

Introduction [edit section]

This proposal is to set up some prototype remote boards or labs, to test the following functionality in Fuego:

fserver - using instance on birdcloud.org
ftc remote operations:
- ftc list-boards -r (and -a) [<attr=value> ...]
- ftc put-request
- ftc put-test
- ftc put-binary-package
- ftc list-requests
- ftc run-request
- ftc rm-request
- ftc put-run
- ftc put-lab (*implement if needed*)
- ftc put-board (*implement if needed*)
- ftc query-request
- ftc get-run
- ftc install-test

This is not the same thing as:

Working_with_remote_boards

Tasks [edit section]

(done) decide on a board - raspberry pi
(done?) decide on a distribution - yocto project?
(done) set up board at PathPartnerTech - already set up
(done) decide which test(s) to use - start with Functional.batch_smoketest
- PathPartnerTech - will test local smoketest run to see if it works
(done) configure Fuego installation at PathPartnerTech to execute poll_requests.sh
- (done) Tim to write up instructions for executing poll_requests.sh (on this page)
set up hardware for remote hardware reboot?
- PathPartnerTech to work on solution for hardware reboot for Rpi board
set up periodic job at timslab to execute tests
- Tim will set up periodic job in his lab
resolve issues that come up
- specifically look for issues in the following areas:
  - (done) does fserver need to support board and lab objects required for system to work
    - yes - added to fserver
  - security issues - only allowing trusted users and tests
  - networking issues (does it work with a proxy)
  - do binary packages work OK
(optional) Have PathPartnerTech set up an fserver instance and test operations on it themselves
- (done) Tim will provide instructions for setting this up
have PathPartnerTech download mainline kernel so that periodic tests can catch new bugs.

Instructions for configuring ppt lab for remote execution [edit section]

For an overview of fserver operation, see Using Fuego with fserver

Here are detailed instructions:

name your lab "pptlab" - or let me know what lab name you choose
name your board "rpi" - or let me know what board name you choose

Configure your local copy of Fuego by editing fuego.conf:

vi fuego-ro/conf/fuego.conf (and edit or add these lines)
- host_name=pptlab
- server_domain=birdcloud.org/cgi-bin

Configure your board with the name "rpi"

mv fuego-ro/boards/<your-board-name>.board fuego-ro/boards/rpi.board

Make sure you can access the instance of fserver running on birdcloud.org:

ftc list-boards -r

You should see:

Boards on the Fuego server:
   board-pptlab:rpi.json
   board-timslab:bbb.json
   board-timslab:min1.json

Running poll_requests.sh [edit section]

NOTE: I fixed a bug in poll_requests.sh, and pushed it to master on August 26, 2020
Please pull the latest fuego-core master branch to get the bugfix.

After adding the settings to fuego.conf, inside the Fuego docker container, run poll_requests.sh. Please 'tee' the output to a log file, so we can diagnose any errors that are encountered.

 $ /fuego-core/scripts/poll_requests.sh 2>&1 | tee "/fuego-rw/logs/poll-log-$(date -Iseconds).txt"

Let me know when you have done this, and I will start sending jobs.

Bugs encountered during testing [edit section]

1. ftc put-request doesn't use -b or -t to specify board and test
- this is confusing, as the syntax for put-request is different from the syntax for run-test
2. when test was '-t', an error was generated, but the request was not marked as done on the server
- Error was: "Error: Test -t not found"
  - need "ftc put-request" to handle error, and mark test as done
    - do_run_test needs to avoid doing error_out(), or poll_requests needs to handle error_out case
      - this error_out is in get_test_arg()
  - need to push some kind of run.json result with the error
there's no timeout in the request json
- need a test timeout, and a request timeout

Bugs fixed during testing [edit section]

poll_requests pulling jobs for wrong lab
- was due to "ftc list-requests" not using host=<my-host> filtering when filter of state=pending was used
invalid request filenames (extra .json extension) on fserver site
- fserver now validates that board is registered with server before allowing request
used new configuration names (server_type and server_domain) in poll_requests messages
added code to extract run data on put-run operation
added start_time and done_time to request object schema (in ftc)

Debug and implementation notes [edit section]

request object differences
- the fields submitted for a request are not the same as the request.json on the server. This means that the server has added or modified the object (which is at odds with the policy that the server is a "dumb object store").
- who is in charge of the json schema for a request object (the client or the fserver)?

Desired operational flow [edit section]

0.1 define a new test or modify an existing test
0.2 package test
0.3 put test package on server
0.4 put binary test package on server
1 tim (in timslab) puts request for test on server
2 pptlab server detects test for a board
3 pptlab executes test
4 pptlab puts run on server from test
5 timslab waits for result
- ttc run-test -r should wait for result, and then download result from server, put it into a local location, and display the consolelog
  - should have a remote-lab timeout
6 timslab download results
7 if waiting, timslab ftc shows results after test completes
8 timslab puts results into /fuego-rw/log area
- add --extract option to "ftc get-run"
9 timslab puts results into jenkins, if there's a local proxy node for the remote board
- add --jenkins-put to "ftc get-run"??
1.1 (optional) jenkins (in timslab) puts request for test on server
- should be easy to configure with a proxy node and put-request integrated into 'ftc run-test'

To Do for remote testing features/fixes work [edit section]

support 'ftc put-board -b <board>'
- fserver already supports 'put_board' operation
support 'ftc run-test -b <remote_board>'
- does ftc put-request <remote_board>, watches status on server, waits for ready
  - watches status on server
    - waits for test to be 'done'
  - downloads run
  - extracts it
  - (optionally) installs data into proxy jobs/nodes in local Jenkins
  - displays console log
add timeout to fserver to mark test as done (with an error) some time after it started
- (done) add start time to request (json) -
  - (done) add time when dest host marked test as 'running'
- (done) now have: request_time, start_time, done_time
add test version to request (json)
- test_name is not sufficient
add architecture and toolchain to board (json)
create jenkins proxy node in client, for a remote fuego node
- create jobs and builds for proxy node, so user can see results locally for remotely executed tests
add command line "ftc rm-request" (or 'delete-request')
add web interface to delete objects

Features that work [edit section]

test runs that were remotely requested show up in server host Jenkins interface

NOTES from calls [edit section]

2020-10-08/09 call [edit section]

PathPartner updated jenkins to 2.235.3
- did the update manually, but have a Dockerfile with changes
- have a list of plugins they thought would be good to add to pipeline list
- had to create a link to java-cli.jar in order for ftc to see it
- have not tested ftc outside the container
PathPartner created pipeline jobs for Jenkins to do kernel checkout, build and provisioning, then trigger Fuego jobs
- [thought: would be good to save kernel_id for kcidb use]
- Tim would like to examine the pipeline jobs *

Bugs:

current version of Jenkins in Fuego doesn't support modern features (specifically pipelines)
ppt requests are missing run data on fserver.py
some requests on fserver are left as running, with no resolution
Timslab is not processing jobs from fserver regularly
user name and lab name are not set on ppt-submitted requests on fserver

Action items:

Tim to add 'add-job' to poll-requests.sh if request is for a test that doesn't have a job
- generate job-id, if not 'ftc list-jobs | grep job-id' then ttc add-job
Tim to look at adding fserver request timeout
- already added start time to request
Tim to look at having poll-requests submit a non-Fuego Jenkins job
- can I request a pipeline job start?
  - can ftc start a pipeline job?
- do these jobs get listed by 'ftc list-jobs'?
(done) ppt to send notes about Jenkins update
- including Dockerfile changes, and list of plugins they added
(done) ppt to add user name to fuego.conf so I can tell who the requestor is
(done) ppt to send pipeline files for Tim to review

Resources [edit section]

https://birdcloud.org/cgi-bin/fserver.py
https://elinux.org/Test_Hardware#Power_control_2
- for hardware for web control of power to boards