Test Dependencies

From eLinux.org
Revision as of 10:46, 22 August 2019 by Tim Bird (talk | contribs) (LTP)
Jump to: navigation, search

This page is for organizing information about test dependencies, which are part of a test's Test Definition

Action Items

  • research kselftest dependency expressions
    • does kselftest only support config fragments?
    • how does its internal testrunner use this?
  • record information about the LTP docparse system
  • research and document LKFT test dependencies

Description

A test dependency is an expression of a requirement or constraint for a test.

A test dependency can serve multiple purposes:

  • to filter the list of tests that apply to a target
  • to allow a test scheduler to select or locate an appropriate target for a test
    • or, to prevent a test from being associated with (or run on) a particular target or set of targets
  • to indicate that some action needs to occur before the test is run (part of test setup)
    • to indicate that some resources needs to be claimed before test start and released on test exit
    • to indicate tests or steps that must be performed before this test
  • to document what the test is related to (the services it needs, or the bug it checks for)

Examples

  • needs root permissions in order to run
  • needs a particular kconfig option configuration
  • needs a package installed on the target
  • needs a program on the target
  • needs a library in the SDK for the target (for the build phase)

From LTP docparse README.md document:

  • Test needs at least 1GB or RAM.
  • Test needs a block device at least 512MB in size
  • Test needs a numa machine with two memory nodes and at least 300 free pages on each node
  • Test needs i2c eeprom connected on a i2c bus
  • Test needs two serial ports connected via null-modem cable

Random notes

  • to process these, the testrunner code is going to need to associate the dependencies with the test (name), which may force us to solve the 'tguid' problem.
  • this is most likely part of:
    • interface C (between test definition repository and test manager), or
    • interface E (between test manager and test scheduler)

Expressions in different systems

Fuego

See [1]

Some Fuego dependency statements are declarative, and some are imperative.

Fuego has a test_pre_check phase, in which dependencies are checked, and in which arbitrary code can be executed to check that required pre-requisites for the test are met.

Fuego stores dependencies in the fuego_test.sh file.

Declarative dependencies can be placed anywhere in the file. Imperative dependencies (pre_check code) is placed in the function test_pre_check() in fuego_test.sh.

examples

  • NEED_ROOT=1
  • NEED_MEMORY=50M
  • NEED_FREE_STORAGE=100M
  • NEED_FREE_STORAGE="100M $BOARD_TESTDIR"
  • NEED_KCONFIG="CONFIG_PRINTK=y"
  • NEED_KCONFIG="CONFIG_LOCKDEP_SUPPORT=n"
  • NEED_PROGRAM=expect
  • assert_has_program expect
  • assert_define "LINK_NETMASK"
  • assert_define BENCHMARK_WHETSTONE_LOOPS
  • is_on_target liblwip.so LIB_LWIP /lib:/usr/lib:/usr/local/lib
  • is_on_target_path time PROGRAM_TIME
  • is_on_sdk libjpeg.so LIBJPEG /lib:/usr/lib/:/usr/local/lib:/usr/lib/$ARCH-linux-*/:/usr/lib/$TOOLCHAIN-linux-*/

notes

Yes, Fuego has both declarative (NEED_*) and imperative (is_on_target) dependency statements.

LTP

Declared as fields in a test program's tst_test structure.

examples

  • needs_root = 1
  • needs_tmpdir = 1
  • needs_checkpoints = 1

new docparse system

LTP developers are working on a new system to add more dependency data to the C files, and parse it out into a json file, for communication with external test runners.

See https://github.com/metan-ucw/ltp/tree/master/docparse

0-day

examples

  • need_modules: true
  • need_memory: 8G
  • need_cpu: 4
  • need_x: true
  • need_kconfig: CONFIG_MD_RAID456
  • need_kconfig: CONFIG_FTRACE=y
  • need_kconfig:
  • avoid_nfs: 1
  • need_kconfig (list of strings)
need_kconfig:
- CONFIG_SUSPEND=y
- CONFIG_FTRACE=y
- CONFIG_FUNCTION_TRACER=y
need_kconfig:
- CONFIG_BLK_DEV_LOOP
- CONFIG_CAN=m
- CONFIG_CAN_RAW=m
- CONFIG_CAN_VCAN=m
- CONFIG_IPV6_VTI=m
- CONFIG_MINIX_FS=m


NOTES

There seems to be some kind of calculated need_kconfig: This is from lkp-test/include/fs1/OTHERS <%=

   case ___
   when /cifs|cramfs|logfs|squashfs/
       "need_kconfig: CONFIG_#{___.upcase}"
   else
       "need_kconfig: CONFIG_#{___.upcase}_FS"
   end

%>