LabControl project in 'raw' format
{{TableOfContents}}
LabControl is a system for managing and using equipment in a test lab.
= Collaborators =
The following people are working on this project:
* Tim Bird - Sony
* Harish Bansal - TimeSys
* Atul Bansal - TimeSys
* Ankit Gupta - TimeSys
* Joel Mathis - TimeSys
== Resources ==
Here is the mailing list address for discussion:
* bf-rest-api@lists.timesys.com
= to do =
* (done) show a list of boards registered with the server
* show the status of a board
* reboot a board
* show a list of resources registered with the server
* (done) support 'lc list boards'
* support 'lc query board {board} [[criteria1] ...]
* support 'lc board {board} info'
* support 'lc board {board} status'
* support 'lc board {board} reboot'
* support 'lc board {board} reserve [timeout]'
* support 'lc board {board} provision'
* support 'lc resource netperf status'
* support 'lc resource netperf reserve [timeout]'
* support 'lc resource netperf release'
* support 'lc resource serial1 status'
* support 'lc resource serial1 query [criteria]'
* support 'lc resource powermonitor status'
* support 'lc resource powermonitor start'
* support 'lc resource powermonitor stop'
* support 'lc resource powermonitor getlog'
* support 'lc resource videograbber status'
* support 'lc resource videograbber start'
* support 'lc resource videograbber stream' - provide live feed of video feed
* support 'lc resource videograbber stop'
* support 'lc resource videograbber download <dest_path>' - get saved video file
* use ttc for operations on timdesk for my lab
* put bbb in LabControl
* put min1 in LabControl
= Components =
It consists of 2 main elements:
* a LabControl server, which stored configuration information, and performs
actions to control objects in a board farm
* this presents a user interface for lab status and control
* this presents a REST interface for lc to interact with
* this stores board and resource configuration information, in the form
of json files on the server
* the server can perform operations:
* locally, either directly or via lc
* remotely, by storing a request which a remote lc retrieves and performs
* a command line utility (called 'lc'), which is used in multiple ways:
* for humans to configure the system
* for humans to query the system
* for the automated test system to reserve resources
* for the automated test system to initiate actions with resources in the lab
* for tests to detect resources and connection points in the lab
* for the test system to convey information about board status to the system
= Envisioned Scenarios =
== Using command line ==
=== reboot a board ===
Options:
* lc reboot board {board}
* lc board {board} power off
* lc board {board} power on
Operation at the server:
* find power_control resource connected to board
* send power control command to resource
* collect log from operation, and save
Operation at the client:
If synchronous operation:
* send command to server
* wait for completion
* show log and status
== check board status using CLI ==
* lc board {board} power status
== register new board with server ==
* lc board register <{board.json}
== Using web interface ==
=== reboot a board using interface ===
Present interface for reboot on board page.
When user clicks link (or button?), cause operation on server:
Operation at the server:
* (see steps for power control initiated by client)
== check board status using web interface ===
Show board status on board page
Use AJAX to update the status in real time.
Status is gathered from lab/board/resources at time of request.
= multiplexing =
lc command can read resource configuration from local files
lcserver can call lc command for resource configuration
lcserver can call lc command to perform operations
lcserver can call arbitrary local processes to perform operations
lcserver stores it's own configuration data in json files
= farm and tests using LabControl =
* test system gets address of host side of serial port for a serial test
* test system gets address of SUT side of serial port for a serial test
= CLI operation =
* query
* update
* reserve
* release
* register
* connect
* configure
* start
* stop
= Architecture =
* web server has:
* definitions for boards
= Notes and questions =
* where is the "business logic" of the lab - is it embedded in the server
or in the CLI, or in the systems that use these?
* would be nice to be able to "collapse the system" down to not
needing web interaction, in the degenerate case (a single system managing
the lab, or running the tests)
* it is possible to make lcserver.py into a python module for lc??
* could possibly collapse network part, and just transfer request
from lc to lcserver.py (and response from lcserver.py to lc)
directly. They're both just text transfers.
= Comparison with Board Farm Cloud implementation =
== BFC rest api ==
See [[file:BFC-TAS_REST_APIs.pdf]]
Some examples:
{{{#!Table
||Object||Category||Verb||REST path||HTTP Method||Notes||
||Device||Power Control ||on ||/api/v0.2/devices/<device_name>/power/on/ ||GET||.||
||Device||Power Control ||off ||/api/v0.2/devices/<device_name>/power/off/ ||GET||.||
||Device||Power Control ||reboot ||/api/v0.2/devices/<device_name>/power/reboot/ ||GET||.||
||Device||Power Control ||status ||/api/v0.2/devices/<device_name>/power/ ||GET||.||
||Device||Power Control ||user defined command||/api/v0.2/devices/<device_name>/power/<user-defined>/||GET||.||
||Device||Access Control ||Active ||/api/v0.2/devices/ ||GET||.||
||Device||Access Control ||My ||/api/v0.2/devices/mine/ ||GET||.||
||Device||Access Control ||allocate||/api/v0.2/devices/<device_name>/assign ||GET||.||
||Device||Access Control ||release ||/api/v0.2/devices/<device_name>/release ||GET||.||
||Device||Access Control ||release force||/api/v0.2/devices/<device_name>/release/force/||GET||.||
||Device||Info ||info ||/api/v0.2/devices/<device_name>/ ||GET||.||
||Device||Shell Access ||run ||/api/v0.2/devices/<device_name>/run/serial/ ||GET||Why is this separate from ssh?||
||Device||Shell Access ||run ||/api/v0.2/devices/<device_name>/run/ssh/ ||GET||Why is this separate from serial?||
||Device||File Management||upload ||/api/v0.2/devices/<device_name>/upload/serial/ ||PUT||Why is this separate from ssh?||
||Device||File Management||upload ||/api/v0.2/devices/<device_name>/upload/ssh/ ||PUT||Why is this separate from serial?||
||Device||File Management||download||/api/v0.2/devices/<device_name>/download/serial/||GET||Why is this separate from ssh?||
||Device||File Management||download||/api/v0.2/devices/<device_name>/download/ssh/ ||GET||Why is this separate from serial?||
||Device||Port Forwarding||add ||/api/v0.2/devices/<device_name>/portfw/ssh/ ||POST||.||
||Device||Port Forwarding||remove ||/api/v0.2/devices/<device_name>/portfw/ssh/ ||DELETE||.||
||Device||Port Forwarding||restore ||/api/v0.2/devices/<device_name>/portfw/ssh/ ||PUT||.||
||Device||Web Cam ||restartfeed||/api/v0.2/devices/<device_name>/webcam/restartfeed/||GET||Path doesn't look right in document (missing v0.2 element)||
||Device||Image Capture ||capture ||/api/v0.2/devices/<device_name>/image/capture/ ||GET||Path doesn't look right in document (missing v0.2 element)||
||Zombie & IOCX||Hotplugs||on ||/api/v0.2/devices/<device_name>/hotplugs/<hotplug_number(1-4)/on/ ||PUT||.||
||Zombie & IOCX||Hotplugs||off ||/api/v0.2/devices/<device_name>/hotplugs/<hotplug_number(1-4)/off/ ||PUT||.||
||Zombie & IOCX||Hotplugs||status ||/api/v0.2/devices/<device_name>/hotplugs/<hotplug_number(1-4)/ ||GET||.||
||Zombie & IOCX||Hotplugs||switch ||/api/v0.2/devices/<device_name>/hotplugs/<hotplug_number(1-4)/switch/||PUT||.||
||Zombie & IOCX||GPIO ||read ||/api/v0.2/devices/<device_name>/gpio/read/<gpio_pin_pattern>/<gpio_pin_data>/||GET||.||
||Zombie & IOCX||GPIO ||write ||/api/v0.2/devices/<device_name>/gpio/write/<gpio_pin_pattern>/<gpio_pin_data>/||PUT||.||
||Console ||Serial Console||start||/api/v0.2/devices/<device_name>/console/serial/start/ ||GET||Why is serial hardcoded here?||
||Console ||Serial Console||stop||/api/v0.2/devices/<device_name>/console/serial/stop/ ||GET||Why is serial hardcoded here?||
||Console ||Serial Console||restart||/api/v0.2/devices/<device_name>/console/serial/restart/ ||GET||Why is serial hardcoded here?||
||Console ||Serial Console||isactive||/api/v0.2/devices/<device_name>/console/serial/isactive/ ||GET||Why is serial hardcoded here?||
}}}
== BFC CLI api ==
BFC uses a command line tool called 'ebf' Here are some of the ebf command-line options:
{{{#!Table
||Category||Command line ||Description ||Notes||
||Devices ||ebf list devices ||List available devices ||.||
||Devices ||ebf <device_name> status ||Print Device information - available/allocated to someone||.||
||Devices ||ebf <device_name> allocate ||Assign Device to user ||.||
||Devices ||ebf <device_name> release ||Release Device ||.||
||Devices ||ebf <device_name> release force ||Release Device assigned to another user||.||
||Devices ||ebf <device_name> mydevices ||List all Devices assigned to me ||ttc has no correlary||
||Devices ||ebf <device_name> info ||Show Device information (such as power commands, Zombie info, IOCX connected status, AV streaming availability, etc)||.||
||Devices ||ebf <device_name> run <command> ||Run command on Device and return command output||.||
||Devices ||ebf <device_name> upload <src_path> <device dest path>||Upload file to powered-up device||.||
||Devices ||ebf <device_name> download <device file path>||Download file from powered-up device||No local destination path?||
||Console ||ebf <device_name> console ||Access Device's serial console ||.||
||Console ||ebf <device_name> logs ||Download Device's console logs ||ttc has no correlary||
||Power ||ebf <device_name> power on|off|reboot|user-defined-command||Power control||.||
||Power ||ebf <device_name> power status ||Device's current power status ||.||
||Hotplug Control||ebf <device_name> hotplug <hotplug_number> on|off|status|switch||Control hotplugs (IOCX)||Similar to SDB relays||
||GPIO ||ebf <device_name> gpio <command> <gpio_pin_pattern> <gpio_pin_data>||Control GPIO to/from board||.||
}}}
== ttc cli ==
{{{#!Table
||Category ||Command line ||Description||Notes||
||Device management ||ttc list ||List targets||.||
||Device management ||ttc <target> status ||Show target status (power, network, can execute command?, reservation)||.||
||Device management ||ttc <target> reserve ||Reserve target||.||
||Device management ||ttc <target> release ||Release target reservation||.||
||Device management ||ttc <target> release -f ||For release of reservation (held by another user)||.||
||Device management ||ttc <target> info ||Show information about a target configuration||.||
||Command execution ||ttc <target> run <command>||Run command and return command output||.||
||File transfer ||ttc <target> cp <src_path> target:<dest_path> ||Copy a file to the target ||.||
||File transfer ||ttc <target> cp target:<src_path> <dest_path> ||Copy a file from the target ||.||
||Console ||ttc <target> console ||Access target console ||Usually serial||
||Interactive login ||ttc <target> login ||Access non-console target login shell||Usually ssh (but can be adb)||
||Power ||ttc <target> on|off|reboot|reset||Control power (/reset) for target||.||
||Power ||ttc <target> pos ||Get target power status (power-on-status)||.||
||Build ||ttc <target> get_kernel ||Get source for kernel for target||.||
||Build ||ttc <target> get_config ||Get kernel configuration for target||.||
||Build ||ttc <target> set_config <config_entry> ||Set kernel configuration option(s) for target||.||
||Build ||ttc <target> kbuild ||Build kernel for target||.||
||Build ||ttc <target> mbuild ||Build modules for target||.||
||Build ||ttc <target> fsbuild ||Build rootfs for target||.||
||Provisioning ||ttc <target> kinstall ||Install kernel where target will boot to it||.||
||Provisioning ||ttc <target> minstall ||Install modules where target will use them||.||
||Provisioning ||ttc <target> fsinstall ||Install root filesystem where target will boot to it||.||
}}}
= Example =
== GPIO test using wget to access rest api ==
Here are some lines from Sample-GPIO-Bit-IO-Test.sh:
{{{#!YellowBox
WRITE_SET=$(wget -qO- http://$BFC_TAS_IP/api/$BOARD_NAME/gpio/write_mode/1)
RESET=$(wget -qO- http://$BFC_TAS_IP/api/$BOARD_NAME/gpio/write_byte/255/0)
WRITE_1=$(wget -qO- http://$BFC_TAS_IP/api/$BOARD_NAME/gpio/write_bit/1/1)
PARSED_WRITE=$( echo $WRITE_1 | grep -o [0-9] )
if [ $(( $PARSED_WRITE % 2 )) = 1 ]; then
echo ""
echo "*** Successfully Wrote 1 to GPIO1 ***"
else
echo ""
echo "[FAIL] Write to GPIO1 failed, please check the hardware connection and parameters"
fi
}}}
This seems to only get data from the IOCX board gpio pins, and not from the
DUT. I'm not sure how this test works.
== serial rx test using lc ==
This is a draft for brainstorming purposes:
{{{#!YellowBox
function noabort() {
set +e
eval "$@"
RET=$?
set -e
return $RET
}
function cmd() {
lc $BOARD run $@
}
# this is overly simplistic - doesn't handle multiple sources
function get() {
lc $BOARD download $1 $2
}
function test_pre_check() {
assert_define BAUDRATE_LIST
assert_define SERIAL_RX_DUT_SERIAL_DEV
# check that there's a serial device connected
SERIAL_TX_LAB_SERIAL_DEV=$(lc $BOARD connection serial tx)
if -z "$SERIAL_TX_LAB_SERIAL_DEV" ; then
abort "Board $BOARD has no configured serial tx endpoint device in the lab"
fi
}
# This test executes on the board, and works with the opposite endpoint
# in the lab to test the serial port on the board (with the board receiving)
function test_run {
local LAB_DEV=SERIAL_TX_LAB_SERIAL_DEV
local DUT_DEV=SERIAL_RX_DUT_SERIAL_DEV
local TEST_DATA="This is a test ASCII string"
echo -n "$TEST_DATA" > expected
for RATE in $BAUDRATE_LIST ; do
# set tx serial rate
# stty -F $LAB_DEV $RATE raw -echo -echoe -echok
lc $BOARD serial set_baud_rate $RATE
# set up to receive for either TEST_DATA.length or 10 sec.
cmd "stty -F $DUT_DEV $RATE raw -echo -echoe -echok min ${#TEST_DATA} time 100"
cmd "nohup cat $DUT_DEV > /tmp/received &"
# cmd seems to be returning slightly before the cat starts reading
sleep 0.3
# send the data from the lab device
lc $BOARD serial tx $LAB_DEV "$TEST_DATA"
# kill "cat" in case that the communication failed.
noabort "cmd \"sync ; killall cat\""
get /tmp/received received
log_this "diff received expected && echo TEST-$RATE OK || echo TEST-$RATE FAILED"
done
}
function test_cleanup {
noabort "cmd \"rm /tmp/received ; rm /tmp/expected\""
}
function test_processing {
log_compare "$TESTDIR" "$(echo "$FUNCTIONAL_SERIAL_RX_BAUDRATES" | wc -w)" "^TEST.*OK" "p"
}
}}}
= Interface between farm and tests =
== test-config file ==
It is proposed that there be a file called ``test-config`` located in: ``/etc``
or in ``/usr/test`` on the device under test. The file would consist of a series
of lines that indicated the values for variables used by tests.
A line would consist of an all-uppercase-name, followed by an equals sign
("="), and a value statement. The value statement could be a literal
value with simple static value definition, or it could be a command
that could be used to obtain the value.
If a value starts with $(, then any tool using the file would execute it as a command and use the stdout from the command as the value for the variable.
Items may be quoted (e.g. to preserve spaces). Leading and trailing
quotes are not part of the value.
See /etc/os-release for an example of a file with similar format (without the
command execution part).
{{{
NAME="beaglebone_4"
NETPERF_SERVER_IP=10.0.1.1
#NETPERF_SERVER_IP=$(lc resource netperf info ip_addr)
POWER_MONITOR=$(lc board resource_connection power_monitor)
}}}
The value of using a file, is that it can be staged to a board in multiple
ways, to accommodate a variety of testing configurations, and it is very
lightweight. For a particular rootfs, the file could be installed with just
the definitions required for a single test. A simple shell script could
parse the file and retrieve values. But even in its simple form it has
sufficient indirection to allow a test to obtain the values it needs to
perform tests.
This essentially is an on-device BOARD file.
== Need for a registry ==
There will need to be a registry to hold the names of the items that are used
as resource identifiers for the system.
== Proposed initial identifiers ==
Right now, this is just a collection of names culled from different places
or made up wholesale:
* NETPERF_SERVER_ADDRESS
* UART_RX
= Notes =
== Notes from call on 2020-09-17 ==
* would like to demonstrate 4 different tests with lab resources off the DUT
* netperf - needs to be allocated to board for test duration
* serial - needs control of endpoint off the device (baud rate, tx, rx)
* gpio - needs control of endpoint off the device (set mode, write value)
* power monitor - need to capture data from resource during the test
- need start/stop/collect data
* TimeSys to provide name, company biography for co-speaker for OSSEU board farm API talk
* short-term goal:
* have Fuego execute a GPIO test from Tim's Fuego host to a board in the TimeSys BFC lab, using the rest API
* deadline: Sep 30
* stretch goals:
* have Fuego execute a GPIO test from Fuego host to a board in Tim's Fuego lab
* have BFC execute a GPIO test from BFC host to a board in Tim's Fuego lab
* execute the gpio test locally on a board in the BFC lab
* execute the gpio test locally on a board in the Fuego lab
* 'run', 'upload', and 'download' actions shouldn't need to specify transport (serial or ssh)
* resources may either be multiplexed, or have a fixed connection to a board
* system should be able to list all resources connected to a board
* system should be able to assign a multiplexed resource to a board for the duration of a test
* the test should not need to know the resource_id for connected resources
* once the resource assignment is made (either statically or dynamically) the API should allow referencing the resource via the board name
* this is how power apis work - you reference the board, not the PDU
* many resources are not multiplexed, but are statically assigned to a board
* some resources, such as a netperf server are automatically multiplexed by the nature of their hardware
* some resources may have multiple instances (e.g. gpio pins, serial endpoints) that have to be selected for the test
== thoughts on boards/resources/connections ==
* actions for resources can be sent to the board, if there's only one resource of that type connected to a board
* example: pdu actions: power, on, off, status
* the labcontrol server keeps track of connections
== First test - proof of concept ==
* gpio test from Fuego host to BFC board
* See Functional.gpio
* create test.yaml
* create test.spec
* create fuego_test.sh
* dependencies: wget
* add BFC board to Fuego's lab
* decide which transport to use (native or wrapped by ttc)
* add LAB_SERVER to board file
* test itself communicates directly with bfc/lc server using wget