Test Stack Layers

Here is some information on standards for test stack layers.

Different people have different approaches to the different sub-tasks for an automated testing stack.

= Andrew Murray (the Farm stack) = Andrew posted some thoughts on different layers (using a farm metaphor) Here is the original message: https://lists.yoctoproject.org/pipermail/automated-testing/2017-November/000134.html

Grass
The Grass - This is the bottom layer which provides the bare minimum software abstraction to the fundamental capabilities of the farm. This consists of physical devices and suitable drivers for them. The interface to higher layers is of the language of turn power port 2 on, bridge relay 5 on, etc. The goal here is for us to be able pick up some hardware off the shelf (e.g. a APC power switch) and to have a 'driver' already available from this community that fits into this view of the world. In order to achieve this it would be necessary to define categories of devices (power, relay, serial), the verbs (power cycle, power off, etc) and a way of addressing physical devices. I believe the LabGrid stack has a good model in this respect.

Cow
The Cow - This is the middle layer which uses the bottom layer and provides an abstraction of a board, this is where boards are defined and related to the physical devices. This layer would manage exclusive access to boards and self-tests. Users of this layer could find out what boards are available (access control), what their capabilities are and access to those capabilities. The language used here may be turn board on, get serial port, press BOOT switch, etc. The goal here is that we can work as a community to create definition of boards - a bit like device tree at a high physical level.

Dairy
The Dairy - This is the top layer which uses the middle layer. This is the application layer, applications can focus on their value-add which may be running tests (LAVA) or providing remote access to boards (perhaps with additional features such as 'workspaces' with prebuilt and predeployed software images).

= Pawel Wieczorrk (SLAV stack) = From Pawel's presentation in ELC 2018: [[Media:Piece_of_cake.pdf | PDF]] (See also ELC_2018_Presentations for a link to the video.

In the SLAV stack, there is a DUT controller board for each DUT. They use the Muxpi as the DUT controller board.

Dryad
Manages single DUT
 * Fully aware of its capabilities
 * Requires only two interfaces
 * Power supply
 * Network connection (Ethernet)

A command line tool called 'stm' is used to multiplex an SDCard between the DUT and the test server. stm is also used to turn power on and off to a board.

A command line tool called 'fota' is used to ...?

Boruta
Manages access to multiple boards in a farm. Dryad farm management system
 * Schedules requests
 * Priority
 * Device groups
 * Delayed access
 * Provides convenient access to selected Dryad

This supports reservations for both the automated test system, and the

Boruta: * matches requirements * prepares environment * sets up tunnel
 * allocates boards (reserves them?)

Weles
Welese is a lightweight testing framework, built on top of Boruta:

It provides a LAVA-like interface.

It has a YAML job definition that has actions executed on the DUT.
 * Deploy
 * Boot
 * Test
 * Collect logs and data

Weles does the following:
 * parse the YAML job definition
 * collect assets
 * request DUT
 * perform tests

Perun
Perun does OS image testing


 * schedules verification (based on new set of OS images)

Actions:
 * Crawl URL
 * report changes
 * submit weles jobs
 * collect artifacts
 * interpret results

= Tim Bird (Fuego stack) = This section may get long, but here goes:

Fuego has several distinct areas if execution:

board layer

 * defines how to control board hardware
 * verb is hardware_reboot
 * also: board_setup, board_teardown
 * defined how to access board functions (system logger)
 * defines SDK for board

transport layer
Defined in board file:
 * defines how to communicate with board (e.g. ssh, serial)
 * verbs are put, get, cmd
 * also: transport_connect, transport_disconnect

test layer

 * Tests are defined with shell scripts
 * Tests are built from source
 * the 'ftc' tool is used to execute a test, which consists of several phases:
 * pre_check - check that DUT has required attributes and is ready (matching)
 * build - build the test software
 * deploy - deploy the test software
 * run - execute the test
 * post_process - retrieve results
 * log_compare - analyze results

user interface
Jenkins is used for job/board scheduling

what's left to the user?

 * detecting that the software under test has changed
 * initiating the test (trigger)
 * building the software under test
 * deploying the software under test to the DUT (provisioning)

= Brainstorming = Stack elements:
 * Board control
 * Device under test
 * connections
 * report generator
 * job dispatcher
 * device database (has information about devices)
 * Test
 * Result
 * change detector
 * artifact storage
 * Presentation service
 * logs
 * run artifacts (logs, traces, results)
 * test artifacts (source, binary, meta-data)

objects:
 * test
 * test results
 * text
 * history
 * tables
 * charts
 * results database
 * board (node)
 * host
 * job
 * run (build)
 * request (run request)
 * connection/channel
 * facility
 * user
 * interface (results presentation interface)
 * scheduler
 * administrator

= QA operations =
 * detect something has changed
 * build software
 * install it
 * test it
 * determine if a result matters
 * (previously) establish a baseline behavior that is acceptable
 * match current behavior with baseline behavior
 * save results
 * examine results
 * report problems
 * find problems
 * fix problems

= uncategorized list of actions =
 * reboot the DUT
 * power on to DUT
 * power off to DUT
 * read serial console of DUT
 * build test software
 * generate test data
 * generate test variations
 * build software under test
 * build kernel under test
 * build rootfs under test
 * install software under test (provision)
 * install kernel (update tftp, switch sdcard, install to /boot, etc.)
 * install rootfs (update nfs area, write to sdcard, write to partitions, etc.)
 * switch sdcard to host
 * switch sdcard to DUT
 * deploy test
 * install test software
 * collect log from test / send log from test
 * parse log from test
 * present test results to user (web interface)
 * present table of past results
 * present chart of past results
 * detect software under test has changed
 * schedule test for DUT
 * reserve DUT
 * match job request with DUT
 * remove test from DUT
 * determine pass/fail from test log
 * determine if metric is over/under threshold
 * determine if required tests passed (allow individual test cases to be ignored)
 * detect if DUT is working
 * store test artifacts
 * store test results
 * notify user of test failure
 * notify user of board failure
 * bisect software under test
 * press/hold button on DUT
 * customize the test's expected value
 * customize the pass criteria for a test
 * customize a benchmark's threshold
 * customize the 'ignore result' list for a test
 * customize the pass count for a test
 * customize the parameters to a test
 * add an testcase to the skip list for a test
 * create a variation of a test
 * specify different parameters to a test (in Fuego, the spec, or the board, or dynamic test vars)
 * edit a test variation

Here are a bunch of miscellaneous ones:
 * update expected results for a test
 * monitor power while test is running
 * validate data from test
 * disconnect/reconnect power during test
 * disconnect/reconnect bus (USB, CAN, i2C, etc.) during test
 * disconnect/reconnect network during test
 * change battery load during test
 * monitor temparature during test
 * capture video during test
 * capture audio during test
 * capture trace during test
 * control user interface during test
 * install host software needed for test
 * set up network server for test

= tool operations = Operations supported by different tools:

ttc

 * console - access the target serial console
 * cp - Copy files to or from the target.
 * fsbuild - Build root filesystem for use on target.
 * fsinstall - Install root filesystem for use on target.
 * get_config - Install kernel config for target in the $KBUILD_OUTPUT directory
 * get_kernel - Install kernel sources for target in the $KERNEL_SRC directory
 * info - Show information about a target.
 * kbuild - Build kernel from source.
 * kinstall - Install kernel for use on target.
 * list - Show a list of available targets.
 * login - access a network login on the target.
 * off - Turn off target board.
 * on - Turn on target board.
 * pos - Show power status of target board.
 * reboot - Reboot target board.
 * release - Release a reservation of a target.
 * reserve - Reserve a target for use.
 * reset - Reset target board.
 * rm - Remove files from the target.
 * run - Run a command on the target.
 * set_config - Set one or more individual config options
 * setenv - Prepare environment for building for target.
 * status - Show status of target, including reservations. (not implemented yet)
 * wait_for - Wait for a condition to be true.

labgrid-client

 * monitor - Monitor events from the coordinator
 * resources - List available resources
 * places - List available places
 * show - Ahow a place and related resources
 * create - Add a new place
 * delete - Delete an existing place
 * acquire - Acquire a place
 * release - Release a place
 * env - Generate a labgrid environement for a place
 * power - Change or get a place's power status
 * power get
 * power on
 * power off
 * power status
 * console - Connect to the console
 * fastboot - Run fastboot
 * bootstrap - Start a bootloader
 * io - Interact with Onewire devices