Test Execution Notes

This page has notes about aspects of the execution control for different systems that perform test execution.

Here are some of the elements of a the test execution API:
 * test API - what functions can the test call?
 * file operations
 * service operations (start, stop)
 * log access
 * results output functions
 * host/target abstraction - is the test executed from host or directly on target
 * DUT preparation/control: - this might be in a separate standard
 * kernel installation / provisioning
 * console access
 * log access
 * test activation - how is the test activated or started?
 * e.g. installed and activated as part of boot
 * e.g. a command is executed as a program?
 * e.g. is the test built into the kernel and activated via some trigger mechanism (e.g. via /proc or /sys or an ioctl)?
 * test retrieval, build, deployment
 * test execution:
 * test name, or standard for test invocation name
 * ex: 'make test' - a standard make target name?
 * runtest.sh? - a standard shell script name?
 * .sh - a standard shell script name, based on test meta-data?
 * test phases?
 * test selection / skiplists - how are test selected for inclusion or exclusion in a test run
 * test variables - how does the test control execution
 * make variables
 * environment variables
 * command line arguments

Does this include test scheduling? (no), but it does include test selection.

Does this include test building (no) - see Test Building notes

= board-local test APIs =

test script names and locations
Every test is in a directory

APIs
Provided by board library: fuego_board_function_lib.sh

Is a library of shell functions.

board-side functions:
 * set_init_manager
 * detect_logger_service
 * exec_service_on_target
 * detect_active_eth_device
 * get_service_status

host-side functions: See http://fuegotest.org/wiki/Test_Script_APIs

test variables
Provided in environment variables, prefixed by test name:
 * ex: FUNCTIONAL_HELLO_ARGS
 * ex:

Required minimum Linux command set
Tools builtin or on the PATH
 * cat, df, find, free, grep, head, logger, logread, mkdir, mount, ps, rm ,rmdir, sync, tail, tee, touch, true, umount, uname, uptime, xargs, [

Required command arguments:
 * mkdir -p, rm -rf, grep -f,

Tools at specific paths:
 * /sbin/reboot, /bin/true, /sbin/route

Data files:
 * /var/log/messages, /var/log/syslog
 * /proc/interrupts, /proc/drop_caches, /proc/oom_score_adj

(optional)
 * proc/config.gz

Require arguments:

kselftest

 * test selection via TARGETS make variable
 * test skiplist support via SKIP_TARGETS make variable
 * patch submitted 9/24/2019 - need to check mainline status - see https://lore.kernel.org/linux-kselftest/20190925132421.23572-1-cristian.marussi@arm.com/

LTP

 * test selection via 'scenario' or 'runtest' files (located in /runtest)
 * test skiplist support via runltp -S option
 * ex: runltp ... -S skiplist.txt ...
 * test command line argument control is via the runtest file

= Standards =

Makefile targets

 * Makefile targets for software testing:
 * make test
 * make check