LabControl project

(done) show a list of boards registered with the server
- show the status of a board
- reboot a board
show a list of resources registered with the server
(done) support 'lc list boards'
support 'lc query board {board} [[criteria1] ...]
support 'lc board {board} info'
support 'lc board {board} status'
support 'lc board {board} reboot'
support 'lc board {board} reserve [timeout]'
support 'lc board {board} provision'
support 'lc resource netperf status'
support 'lc resource netperf reserve [timeout]'
support 'lc resource netperf release'
support 'lc resource serial1 status'
support 'lc resource serial1 query [criteria]'
support 'lc resource powermonitor status'
support 'lc resource powermonitor start'
support 'lc resource powermonitor stop'
support 'lc resource powermonitor getlog'
support 'lc resource videograbber status'
support 'lc resource videograbber start'
support 'lc resource videograbber stream' - provide live feed of video feed
support 'lc resource videograbber stop'
support 'lc resource videograbber download <dest_path>' - get saved video file

use ttc for operations on timdesk for my lab
- put bbb in LabControl
- put min1 in LabControl

Components [edit section]

It consists of 2 main elements:

a LabControl server, which stored configuration information, and performs actions to control objects in a board farm
- this presents a user interface for lab status and control
- this presents a REST interface for lc to interact with
- this stores board and resource configuration information, in the form of json files on the server
- the server can perform operations:
  - locally, either directly or via lc
  - remotely, by storing a request which a remote lc retrieves and performs
a command line utility (called 'lc'), which is used in multiple ways:
- for humans to configure the system
- for humans to query the system
- for the automated test system to reserve resources
- for the automated test system to initiate actions with resources in the lab
- for tests to detect resources and connection points in the lab
- for the test system to convey information about board status to the system

Envisioned Scenarios [edit section]

Using command line [edit section]

reboot a board [edit section]

Options:

lc reboot board {board}
lc board {board} power off
lc board {board} power on

Operation at the server:

find power_control resource connected to board
send power control command to resource
collect log from operation, and save

Operation at the client: If synchronous operation:

send command to server
wait for completion
show log and status

check board status using CLI [edit section]

lc board {board} power status

register new board with server [edit section]

lc board register <{board.json}

Using web interface [edit section]

reboot a board using interface [edit section]

Present interface for reboot on board page.

When user clicks link (or button?), cause operation on server:

Operation at the server:

(see steps for power control initiated by client)

check board status using web interface [edit section]

Show board status on board page

Use AJAX to update the status in real time. Status is gathered from lab/board/resources at time of request.

multiplexing [edit section]

lc command can read resource configuration from local files lcserver can call lc command for resource configuration lcserver can call lc command to perform operations lcserver can call arbitrary local processes to perform operations lcserver stores it's own configuration data in json files

farm and tests using LabControl [edit section]

test system gets address of host side of serial port for a serial test
test system gets address of SUT side of serial port for a serial test

CLI operation [edit section]

query
update
reserve
release
register
connect
configure
start
stop

Architecture [edit section]

web server has:
- definitions for boards

Notes and questions [edit section]

where is the "business logic" of the lab - is it embedded in the server or in the CLI, or in the systems that use these?
would be nice to be able to "collapse the system" down to not needing web interaction, in the degenerate case (a single system managing the lab, or running the tests)
- it is possible to make lcserver.py into a python module for lc??
  - could possibly collapse network part, and just transfer request from lc to lcserver.py (and response from lcserver.py to lc) directly. They're both just text transfers.

Comparison with Board Farm Cloud implementation [edit section]

BFC rest api [edit section]

See file:BFC-TAS_REST_APIs.pdf

Some examples:

Object ^ Category ^ Verb ^ REST path ^ HTTP Method ^ Notes ^

Device Power Control on /api/v0.2/devices/<device_name>/power/on/ GET .

Device Power Control off /api/v0.2/devices/<device_name>/power/off/ GET .

Device Power Control reboot /api/v0.2/devices/<device_name>/power/reboot/ GET .

Device Power Control status /api/v0.2/devices/<device_name>/power/ GET .

Device Power Control user defined command /api/v0.2/devices/<device_name>/power/<user-defined>/ GET .

Device Access Control Active /api/v0.2/devices/ GET .

Device Access Control My /api/v0.2/devices/mine/ GET .

Device Access Control allocate /api/v0.2/devices/<device_name>/assign GET .

Device Access Control release /api/v0.2/devices/<device_name>/release GET .

Device Access Control release force /api/v0.2/devices/<device_name>/release/force/ GET .

Device Info info /api/v0.2/devices/<device_name>/ GET .

Device Shell Access run /api/v0.2/devices/<device_name>/run/serial/ GET Why is this separate from ssh?

Device Shell Access run /api/v0.2/devices/<device_name>/run/ssh/ GET Why is this separate from serial?

Device File Management upload /api/v0.2/devices/<device_name>/upload/serial/ PUT Why is this separate from ssh?

Device File Management upload /api/v0.2/devices/<device_name>/upload/ssh/ PUT Why is this separate from serial?

Device File Management download /api/v0.2/devices/<device_name>/download/serial/ GET Why is this separate from ssh?

Device File Management download /api/v0.2/devices/<device_name>/download/ssh/ GET Why is this separate from serial?

Device Port Forwarding add /api/v0.2/devices/<device_name>/portfw/ssh/ POST .

Device Port Forwarding remove /api/v0.2/devices/<device_name>/portfw/ssh/ DELETE .

Device Port Forwarding restore /api/v0.2/devices/<device_name>/portfw/ssh/ PUT .

Device Web Cam restartfeed /api/v0.2/devices/<device_name>/webcam/restartfeed/ GET Path doesn't look right in document (missing v0.2 element)

Device Image Capture capture /api/v0.2/devices/<device_name>/image/capture/ GET Path doesn't look right in document (missing v0.2 element)

Zombie & IOCX Hotplugs on /api/v0.2/devices/<device_name>/hotplugs/<hotplug_number(1-4)/on/ PUT .

Zombie & IOCX Hotplugs off /api/v0.2/devices/<device_name>/hotplugs/<hotplug_number(1-4)/off/ PUT .

Zombie & IOCX Hotplugs status /api/v0.2/devices/<device_name>/hotplugs/<hotplug_number(1-4)/ GET .

Zombie & IOCX Hotplugs switch /api/v0.2/devices/<device_name>/hotplugs/<hotplug_number(1-4)/switch/ PUT .

Zombie & IOCX GPIO read /api/v0.2/devices/<device_name>/gpio/read/<gpio_pin_pattern>/<gpio_pin_data>/ GET .

Zombie & IOCX GPIO write /api/v0.2/devices/<device_name>/gpio/write/<gpio_pin_pattern>/<gpio_pin_data>/ PUT .

Console Serial Console start /api/v0.2/devices/<device_name>/console/serial/start/ GET Why is serial hardcoded here?

Console Serial Console stop /api/v0.2/devices/<device_name>/console/serial/stop/ GET Why is serial hardcoded here?

Console Serial Console restart /api/v0.2/devices/<device_name>/console/serial/restart/ GET Why is serial hardcoded here?

Console Serial Console isactive /api/v0.2/devices/<device_name>/console/serial/isactive/ GET Why is serial hardcoded here?

Query

BFC CLI api [edit section]

BFC uses a command line tool called 'ebf' Here are some of the ebf command-line options:

Category ^ Command line ^ Description ^ Notes ^

Devices ebf list devices List available devices .

Devices ebf <device_name> status Print Device information - available/allocated to someone .

Devices ebf <device_name> allocate Assign Device to user .

Devices ebf <device_name> release Release Device .

Devices ebf <device_name> release force Release Device assigned to another user .

Devices ebf <device_name> mydevices List all Devices assigned to me ttc has no correlary

Devices ebf <device_name> info Show Device information (such as power commands, Zombie info, IOCX connected status, AV streaming availability, etc) .

Devices ebf <device_name> run <command> Run command on Device and return command output .

Devices ebf <device_name> upload <src_path> <device dest path> Upload file to powered-up device .

Devices ebf <device_name> download <device file path> Download file from powered-up device No local destination path?

Console ebf <device_name> console Access Device's serial console .

Console ebf <device_name> logs Download Device's console logs ttc has no correlary

Power ebf <device_name> power on|off|reboot|user-defined-command Power control .

Power ebf <device_name> power status Device's current power status .

Hotplug Control ebf <device_name> hotplug <hotplug_number> on|off|status|switch Control hotplugs (IOCX) Similar to SDB relays

GPIO ebf <device_name> gpio <command> <gpio_pin_pattern> <gpio_pin_data> Control GPIO to/from board .

Query

ttc cli [edit section]

Category ^	Command line ^	Description ^	Notes ^
Device management	ttc list	List targets	.
Device management	ttc <target> status	Show target status (power, network, can execute command?, reservation)	.
Device management	ttc <target> reserve	Reserve target	.
Device management	ttc <target> release	Release target reservation	.
Device management	ttc <target> release -f	For release of reservation (held by another user)	.
Device management	ttc <target> info	Show information about a target configuration	.
Command execution	ttc <target> run <command>	Run command and return command output	.
File transfer	ttc <target> cp <src_path> target:<dest_path>	Copy a file to the target	.
File transfer	ttc <target> cp target:<src_path> <dest_path>	Copy a file from the target	.
Console	ttc <target> console	Access target console	Usually serial
Interactive login	ttc <target> login	Access non-console target login shell	Usually ssh (but can be adb)
Power	ttc <target> on\|off\|reboot\|reset	Control power (/reset) for target	.
Power	ttc <target> pos	Get target power status (power-on-status)	.
Build	ttc <target> get_kernel	Get source for kernel for target	.
Build	ttc <target> get_config	Get kernel configuration for target	.
Build	ttc <target> set_config <config_entry>	Set kernel configuration option(s) for target	.
Build	ttc <target> kbuild	Build kernel for target	.
Build	ttc <target> mbuild	Build modules for target	.
Build	ttc <target> fsbuild	Build rootfs for target	.
Provisioning	ttc <target> kinstall	Install kernel where target will boot to it	.
Provisioning	ttc <target> minstall	Install modules where target will use them	.
Provisioning	ttc <target> fsinstall	Install root filesystem where target will boot to it	.

Query

Example [edit section]

GPIO test using wget to access rest api [edit section]

Here are some lines from Sample-GPIO-Bit-IO-Test.sh:

WRITE_SET=$(wget -qO- http://$BFC_TAS_IP/api/$BOARD_NAME/gpio/write_mode/1)
RESET=$(wget -qO- http://$BFC_TAS_IP/api/$BOARD_NAME/gpio/write_byte/255/0)

WRITE_1=$(wget -qO- http://$BFC_TAS_IP/api/$BOARD_NAME/gpio/write_bit/1/1)
PARSED_WRITE=$( echo $WRITE_1 | grep -o [0-9] )
if [ $(( $PARSED_WRITE % 2 )) = 1 ]; then
        echo ""
        echo "*** Successfully Wrote 1 to GPIO1 ***"
else
        echo ""
        echo "[FAIL] Write to GPIO1 failed, please check the hardware connection and parameters"
fi

This seems to only get data from the IOCX board gpio pins, and not from the DUT. I'm not sure how this test works.

serial rx test using lc [edit section]

This is a draft for brainstorming purposes:

function noabort() {
    set +e
    eval "$@"
    RET=$?
    set -e
    return $RET
}

function cmd() {
    lc $BOARD run $@
}

# this is overly simplistic - doesn't handle multiple sources
function get() {
    lc $BOARD download $1 $2
}

function test_pre_check() {
    assert_define BAUDRATE_LIST
    assert_define SERIAL_RX_DUT_SERIAL_DEV

    # check that there's a serial device connected
    SERIAL_TX_LAB_SERIAL_DEV=$(lc $BOARD connection serial tx)
    if -z "$SERIAL_TX_LAB_SERIAL_DEV" ; then
        abort "Board $BOARD has no configured serial tx endpoint device in the lab"
    fi
}

# This test executes on the board, and works with the opposite endpoint
# in the lab to test the serial port on the board (with the board receiving)

function test_run {
    local LAB_DEV=SERIAL_TX_LAB_SERIAL_DEV
    local DUT_DEV=SERIAL_RX_DUT_SERIAL_DEV
    local TEST_DATA="This is a test ASCII string"

    echo -n "$TEST_DATA" > expected
    for RATE in $BAUDRATE_LIST ; do
        # set tx serial rate
        # stty -F $LAB_DEV $RATE raw -echo -echoe -echok
        lc $BOARD serial set_baud_rate $RATE

        # set up to receive for either TEST_DATA.length or 10 sec.
        cmd "stty -F $DUT_DEV $RATE raw -echo -echoe -echok min ${#TEST_DATA} time 100"
        cmd "nohup cat $DUT_DEV > /tmp/received &"

        # cmd seems to be returning slightly before the cat starts reading
        sleep 0.3

        # send the data from the lab device
        lc $BOARD serial tx $LAB_DEV "$TEST_DATA"

        # kill "cat" in case that the communication failed.
        noabort "cmd \"sync ; killall cat\""
        get /tmp/received received
        log_this "diff received expected && echo TEST-$RATE OK || echo TEST-$RATE FAILED"
    done
}

function test_cleanup {
    noabort "cmd \"rm /tmp/received ; rm /tmp/expected\""
}

function test_processing {
    log_compare "$TESTDIR" "$(echo "$FUNCTIONAL_SERIAL_RX_BAUDRATES" | wc -w)" "^TEST.*OK" "p"
}

Interface between farm and tests [edit section]

test-config file [edit section]

It is proposed that there be a file called test-config located in: /etc or in /usr/test on the device under test. The file would consist of a series of lines that indicated the values for variables used by tests.

A line would consist of an all-uppercase-name, followed by an equals sign ("="), and a value statement. The value statement could be a literal value with simple static value definition, or it could be a command that could be used to obtain the value.

If a value starts with $(, then any tool using the file would execute it as a command and use the stdout from the command as the value for the variable.

Items may be quoted (e.g. to preserve spaces). Leading and trailing quotes are not part of the value.

See /etc/os-release for an example of a file with similar format (without the command execution part).

NAME="beaglebone_4"
NETPERF_SERVER_IP=10.0.1.1
#NETPERF_SERVER_IP=$(lc resource netperf info ip_addr)
POWER_MONITOR=$(lc board resource_connection power_monitor)

The value of using a file, is that it can be staged to a board in multiple ways, to accommodate a variety of testing configurations, and it is very lightweight. For a particular rootfs, the file could be installed with just the definitions required for a single test. A simple shell script could parse the file and retrieve values. But even in its simple form it has sufficient indirection to allow a test to obtain the values it needs to perform tests.

This essentially is an on-device BOARD file.

Need for a registry [edit section]

There will need to be a registry to hold the names of the items that are used as resource identifiers for the system.

Proposed initial identifiers [edit section]

Right now, this is just a collection of names culled from different places or made up wholesale:

NETPERF_SERVER_ADDRESS
UART_RX

Notes [edit section]

Notes from call on 2020-09-17 [edit section]

would like to demonstrate 4 different tests with lab resources off the DUT
- netperf - needs to be allocated to board for test duration
- serial - needs control of endpoint off the device (baud rate, tx, rx)
- gpio - needs control of endpoint off the device (set mode, write value)
- power monitor - need to capture data from resource during the test
  - need start/stop/collect data

TimeSys to provide name, company biography for co-speaker for OSSEU board farm API talk

short-term goal:
- have Fuego execute a GPIO test from Tim's Fuego host to a board in the TimeSys BFC lab, using the rest API
- deadline: Sep 30
stretch goals:
- have Fuego execute a GPIO test from Fuego host to a board in Tim's Fuego lab
- have BFC execute a GPIO test from BFC host to a board in Tim's Fuego lab
- execute the gpio test locally on a board in the BFC lab
- execute the gpio test locally on a board in the Fuego lab

'run', 'upload', and 'download' actions shouldn't need to specify transport (serial or ssh)

resources may either be multiplexed, or have a fixed connection to a board
- system should be able to list all resources connected to a board
- system should be able to assign a multiplexed resource to a board for the duration of a test
- the test should not need to know the resource_id for connected resources
  - once the resource assignment is made (either statically or dynamically) the API should allow referencing the resource via the board name
    - this is how power apis work - you reference the board, not the PDU

many resources are not multiplexed, but are statically assigned to a board
- some resources, such as a netperf server are automatically multiplexed by the nature of their hardware
some resources may have multiple instances (e.g. gpio pins, serial endpoints) that have to be selected for the test

thoughts on boards/resources/connections [edit section]

actions for resources can be sent to the board, if there's only one resource of that type connected to a board
- example: pdu actions: power, on, off, status
the labcontrol server keeps track of connections

First test - proof of concept [edit section]

gpio test from Fuego host to BFC board
- See Functional.gpio
  - create test.yaml
  - create test.spec
  - create fuego_test.sh
    - dependencies: wget
- add BFC board to Fuego's lab
  - decide which transport to use (native or wrapped by ttc)
  - add LAB_SERVER to board file
- test itself communicates directly with bfc/lc server using wget

Object ^	Category ^	Verb ^	REST path ^	HTTP Method ^	Notes ^
Device	Power Control	on	/api/v0.2/devices/<device_name>/power/on/	GET	.
Device	Power Control	off	/api/v0.2/devices/<device_name>/power/off/	GET	.
Device	Power Control	reboot	/api/v0.2/devices/<device_name>/power/reboot/	GET	.
Device	Power Control	status	/api/v0.2/devices/<device_name>/power/	GET	.
Device	Power Control	user defined command	/api/v0.2/devices/<device_name>/power/<user-defined>/	GET	.
Device	Access Control	Active	/api/v0.2/devices/	GET	.
Device	Access Control	My	/api/v0.2/devices/mine/	GET	.
Device	Access Control	allocate	/api/v0.2/devices/<device_name>/assign	GET	.
Device	Access Control	release	/api/v0.2/devices/<device_name>/release	GET	.
Device	Access Control	release force	/api/v0.2/devices/<device_name>/release/force/	GET	.
Device	Info	info	/api/v0.2/devices/<device_name>/	GET	.
Device	Shell Access	run	/api/v0.2/devices/<device_name>/run/serial/	GET	Why is this separate from ssh?
Device	Shell Access	run	/api/v0.2/devices/<device_name>/run/ssh/	GET	Why is this separate from serial?
Device	File Management	upload	/api/v0.2/devices/<device_name>/upload/serial/	PUT	Why is this separate from ssh?
Device	File Management	upload	/api/v0.2/devices/<device_name>/upload/ssh/	PUT	Why is this separate from serial?
Device	File Management	download	/api/v0.2/devices/<device_name>/download/serial/	GET	Why is this separate from ssh?
Device	File Management	download	/api/v0.2/devices/<device_name>/download/ssh/	GET	Why is this separate from serial?
Device	Port Forwarding	add	/api/v0.2/devices/<device_name>/portfw/ssh/	POST	.
Device	Port Forwarding	remove	/api/v0.2/devices/<device_name>/portfw/ssh/	DELETE	.
Device	Port Forwarding	restore	/api/v0.2/devices/<device_name>/portfw/ssh/	PUT	.
Device	Web Cam	restartfeed	/api/v0.2/devices/<device_name>/webcam/restartfeed/	GET	Path doesn't look right in document (missing v0.2 element)
Device	Image Capture	capture	/api/v0.2/devices/<device_name>/image/capture/	GET	Path doesn't look right in document (missing v0.2 element)
Zombie & IOCX	Hotplugs	on	/api/v0.2/devices/<device_name>/hotplugs/<hotplug_number(1-4)/on/	PUT	.
Zombie & IOCX	Hotplugs	off	/api/v0.2/devices/<device_name>/hotplugs/<hotplug_number(1-4)/off/	PUT	.
Zombie & IOCX	Hotplugs	status	/api/v0.2/devices/<device_name>/hotplugs/<hotplug_number(1-4)/	GET	.
Zombie & IOCX	Hotplugs	switch	/api/v0.2/devices/<device_name>/hotplugs/<hotplug_number(1-4)/switch/	PUT	.
Zombie & IOCX	GPIO	read	/api/v0.2/devices/<device_name>/gpio/read/<gpio_pin_pattern>/<gpio_pin_data>/	GET	.
Zombie & IOCX	GPIO	write	/api/v0.2/devices/<device_name>/gpio/write/<gpio_pin_pattern>/<gpio_pin_data>/	PUT	.
Console	Serial Console	start	/api/v0.2/devices/<device_name>/console/serial/start/	GET	Why is serial hardcoded here?
Console	Serial Console	stop	/api/v0.2/devices/<device_name>/console/serial/stop/	GET	Why is serial hardcoded here?
Console	Serial Console	restart	/api/v0.2/devices/<device_name>/console/serial/restart/	GET	Why is serial hardcoded here?
Console	Serial Console	isactive	/api/v0.2/devices/<device_name>/console/serial/isactive/	GET	Why is serial hardcoded here?