Test definition survey

From eLinux.org
Revision as of 20:33, 17 January 2019 by Tim Bird (talk | contribs) (Fuego)
Jump to: navigation, search

Here is a list of test definition fields, attributes, file formats, operations, instructions, functions, etc. (I won't know what even what they consist of until I see them).

This is the object in your test system that "defines" what a test is. It likely has meta-data about the program to run, how to get the program started, maybe what things are required for the program to run, how the results should be interpreted, etc.

Survey - link to test definitions

Here is a link to test definitions in different systems: One simple, one characteristic, and link to repository containing lots of them:


fuego files:

  • fuego_test.sh
  • spec.json
  • parser.py - has the testlog parser for this test
  • criteria.json - has the pass criteria for this test
  • test.yaml - has meta-data for this test
  • chart_config.json - has charting configuration
  • reference.json - has units for test results
  • docs - directory of test and testcase documentation
  • (test program source) - tarball, or repository reference for test program source
  • (patches against test program source) - changes for test program source

jenkins files:

  • config.xml (job file) - Jenkins job description for a test ((board, spec, test) combination)


  • config.xml::actions
  • config.xml::descriptions
  • config.xml::keepDependencies
  • config.xml::scm
  • config.xml::assignedNode - tag for which board or set of boards can run this job
  • config.xml::canRoam
  • config.xml::disabled
  • config.xml::blockBuildWhenDownstreamBuilding
  • config.xml::blockBuildWhenUpstreamBuilding
  • config.xml::triggers
  • config.xml::concurrentBuild
  • config.xml::customWorkspace
  • config.xml::builders
  • config.xml::hudson.tasks.Shell:command - Fuego command to run (includes board, spec, timeout, flags, and test)
  • config.xml::publishers
  • config.xml::flotile.FlotPlublisher
  • config.xml::hudson.plugins.descriptionSetter.DescriptionSetterPublisher(:regexp,:regexpForFailed,:description,:descriptionForFailed,:setForMatrix)
  • config.xml::buildWrappers
  • fuego_test.sh::NEED_* - 'need' variables for declarative dependency checks
  • fuego_test.sh::tarball - program source reference (can be local tarball or remote tarball, or url?)
  • fuego_test.sh::test_pre_check - (optional) shell function to test dependencies and pre-conditions
  • fuego_test.sh::test_build - shell function to build test program source
  • fuego_test.sh::test_deploy - shell function to put test program on target board
  • fuego_test.sh::test_run - shell function to run test program on the target board
  • fuego_test.sh::test_snapshot - (optional) shell function to gather machine status
  • fuego_test.sh::test_fetch_results - (optional) shell function to gather results and logs from target board
  • fuego_test.sh::test_processing - shell function to determine result
  • spec.json::testName - name of the test
  • spec.json::specs - list of test specs (variants)
  • spec.json::specs[<specname>].xxx - arbitrary test variables for the indicated test spec
  • spec.json::specs[<specname>].skiplist - list of testcases to skip
  • spec.json::specs[<specname>].extra_success_links - links for Jenkins display on test success
  • spec.json::specs[<specname>].extra_fail_links - links for Jenkins display on test failure
  • parser.py - python code to parse the testlog from (test_run::report(), report_live(), and log_this()) calls
  • reference.json::test_sets::name
  • reference.json::test_sets::test_cases - list of test cases in this test_set
  • reference.json::test_sets::test_cases::name
  • reference.json::test_sets::test_cases::measurements - list of measurements in this test case
  • reference.json::test_sets::test_cases::measurements::name
  • reference.json::test_sets::test_cases::measurements::unit
  • criteria.json::schema_version
  • criteria.json::criteria - list of results pass criteria
  • criteria.json::criteria::tguid - test globally unique identifier for this criteria
  • criteria.json::criteria::reference - reference condition for this criteria
  • criteria.json::criteria::reference::value - reference value(s) for this criteria
  • criteria.json::criteria::reference::operator - operator for this condition (eq, le, lt, ge, gt, bt, ne)
  • criteria.json::criteria::min_pass
  • criteria.json::criteria::max_fail
  • criteria.json::criteria::fail_ok_list
  • criteria.json::criteria::must_pass_list
  • test.yaml::


For the 'files' part, each test in test-definitions is stored in a separate directory. The directory has to contain at least the YAML file that is compliant with LAVA test definition. We have a sanity check script (validate.py) that is executed on any pull request. This ensures all files pushed to the repository are compliant. Usual practice is that test directory contains a test script (shell script). Script is responsible for installing dependencies, running tests and parsing results. There is no mandatory format there but test-definitions provides a library of functions that help writing test scripts. There are libraries for 'linux' and 'android'. We also host directory for manual tests and simple executor for them but in the context of automated testing these are irrelevant.

Yocto Project

An "on target" test of the compiler:

(same directory has simple python/perl tests and so on)

http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/lib/oeqa/files (for the test files for context, they're just hello world examples)

This is a "selftest" for the "devtool" command that is part of the overall build system, its a bit more complex with shared functions and tests for each of devtool's subcommands.

This has all the test code and core test definitions. Test definitions are in cases directories under "manual", "runtime", "sdk" and "selftest" directories.