Test Execution Notes

This page has notes about aspects of the execution control for different systems that perform test execution.

Here are some of the elements of a the test execution API:

• test API - what functions can the test call?
  • file operations
  • service operations (start, stop)
  • log access
  • results output functions
• host/target abstraction - is the test executed from host or directly on target
• DUT preparation/control: - this might be in a separate standard
  • kernel installation / provisioning
  • console access
  • log access
  • test activation - how is the test activated or started?
    • e.g. installed and activated as part of boot
    • e.g. a command is executed as a program?
    • e.g. is the test built into the kernel and activated via some trigger mechanism (e.g. via /proc or /sys or an ioctl)?
• test retrieval, build, deployment
• test execution:
  • test name, or standard for test invocation name
  • ex: 'make test' - a standard make target name?
  • runtest.sh? - a standard shell script name?
  • <testname>.sh - a standard shell script name, based on test meta-data?
• test phases?
• test selection / skiplists - how are test selected for inclusion or exclusion in a test run
• test variables - how does the test control execution
  • make variables
  • environment variables
  • command line arguments

Does this include test scheduling? (no), but it does include test selection.

Does this include test building (no) - see Test Building notes

Notes by test system

Fuego

test script names and locations

Every test is in a directory

APIs

Provided by board library: fuego_board_function_lib.sh

Is a library of shell functions.

board-side functions:

• set_init_manager()
• detect_logger_service()
• exec_service_on_target()
• detect_active_eth_device()
• get_service_status()

host-side functions: See http://fuegotest.org/wiki/Test_Script_APIs

test variables

Provided in environment variables, prefixed by test name:

• ex: FUNCTIONAL_HELLO_ARGS
• ex:

Required minimum Linux command set

Tools builtin or on the PATH

• cat, df, find, free, grep, head, logger, logread, mkdir, mount, ps, rm ,rmdir, sync, tail, tee, touch, true, umount, uname, uptime, xargs, [

Required command arguments:

• mkdir -p, rm -rf, grep -f,

Tools at specific paths:

• /sbin/reboot, /bin/true, /sbin/route

Data files:

• /var/log/messages, /var/log/syslog
• /proc/interrupts, /proc/drop_caches, /proc/oom_score_adj

(optional)

• proc/config.gz

Require arguments:

kselftest

• test selection via TARGETS make variable
• test skiplist support via SKIP_TARGETS make variable
  • patch submitted 9/24/2019 - need to check mainline status - see https://lore.kernel.org/linux-kselftest/20190925132421.23572-1-cristian.marussi@arm.com/

Location of tests

• source of tests is in: <kernel_repository>/tools/testing/selftests/<area>
• test binaries are placed in: $INSTALL_PATH/kselftest
  • they are organized by test area, which corresponds to the directory in which they reside in the kernel source repsoitory

test executable names

There is no standard, but the names appear to be obtainable with 'make -s -C tools/testing/selftests/<area> emit_tests'

A script is generated which runs all the test, sourcing 'kselftest/runner.sh' and calling all the test names with the function 'run_many', generating TAP output based on the return code for each test.

LTP

• test selection via 'scenario' or 'runtest' files (located in <topdir>/runtest)
• test skiplist support via runltp -S option
  • ex: runltp ... -S skiplist.txt ...
• test command line argument control is via the runtest file

Standards

Makefile targets

• Makefile targets for software testing:
  • make test
  • make check

sources

If a program uses Makefile to build and install the software under test, then it should provide the following Makefile targets, to support testing of the software.

"make test"

or

"make check"

See https://www.gnu.org/software/make/manual/make.html#Goals (section "9.2 Arguments to Specify the Goals")

See also: https://www.gnu.org/software/make/manual/make.html#Standard-Targets