Difference between revisions of "Test definition survey"

From eLinux.org
Jump to: navigation, search
(fix bullets)
(add YP and LAVA responses)
Line 29: Line 29:
 
fields:
 
fields:
 
* (to be filled out)
 
* (to be filled out)
 +
 +
== LAVA ==
 +
* (simple) busybox: https://github.com/Linaro/test-definitions/tree/master/automated/linux/busybox
 +
* (characteristic) libhugetblfs: https://github.com/Linaro/test-definitions/tree/master/automated/linux/libhugetlbfs
 +
* (repository) https://github.com/Linaro/test-definitions/
 +
 +
For the 'files' part, each test in test-definitions is stored in a
 +
separate directory. The directory has to contain at least the YAML
 +
file that is compliant with LAVA test definition. We have a sanity
 +
check script (validate.py) that is executed on any pull request. This
 +
ensures all files pushed to the repository are compliant. Usual
 +
practice is that test directory contains a test script (shell script).
 +
Script is responsible for installing dependencies, running tests and
 +
parsing results. There is no mandatory format there but
 +
test-definitions provides a library of functions that help writing
 +
test scripts. There are libraries for 'linux' and 'android'. We also
 +
host directory for manual tests and simple executor for them but in
 +
the context of automated testing these are irrelevant.
 +
 +
 +
== Yocto Project ==
 +
An "on target" test of the compiler:
 +
 +
* http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/lib/oeqa/runtime/cases/gcc.py
 +
(same directory has simple python/perl tests and so on)
 +
 +
http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/lib/oeqa/files
 +
(for the test files for context, they're just hello world examples)
 +
 +
 +
* (characteristic) devtool:  http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/lib/oeqa/selftest/cases/devtool.py
 +
This is a "selftest" for the "devtool" command that is part of the
 +
overall build system, its a bit more complex with shared functions and
 +
tests for each of devtool's subcommands.
 +
 +
* (repository) http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/lib/oeqa
 +
This has all the test code and core test definitions. Test definitions are in
 +
cases directories under "manual", "runtime", "sdk" and "selftest"
 +
directories.
 +
 +
*  bitback-selftest at: http://git.yoctoproject.org/cgit.cgi/poky/tree/bitbake/lib/bb/tests

Revision as of 11:00, 11 January 2019

Here is a list of test definition fields, attributes, file formats, operations, instructions, functions, etc. (I won't know what even what they consist of until I see them).

This is the object in your test system that "defines" what a test is. It likely has meta-data about the program to run, how to get the program started, maybe what things are required for the program to run, how the results should be interpreted, etc.

Survey - link to test definitions

Here is a link to test definitions in different systems: One simple, one characteristic, and link to repository containing lots of them:

Fuego

files:

  • fuego_test.sh
  • spec.json
  • parser.py
  • criteria.json
  • test.yaml
  • chart_config.json
  • reference.json
  • docs
  • (test program source)

fields:

  • (to be filled out)

LAVA

For the 'files' part, each test in test-definitions is stored in a separate directory. The directory has to contain at least the YAML file that is compliant with LAVA test definition. We have a sanity check script (validate.py) that is executed on any pull request. This ensures all files pushed to the repository are compliant. Usual practice is that test directory contains a test script (shell script). Script is responsible for installing dependencies, running tests and parsing results. There is no mandatory format there but test-definitions provides a library of functions that help writing test scripts. There are libraries for 'linux' and 'android'. We also host directory for manual tests and simple executor for them but in the context of automated testing these are irrelevant.


Yocto Project

An "on target" test of the compiler:

(same directory has simple python/perl tests and so on)

http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/lib/oeqa/files (for the test files for context, they're just hello world examples)


This is a "selftest" for the "devtool" command that is part of the overall build system, its a bit more complex with shared functions and tests for each of devtool's subcommands.

This has all the test code and core test definitions. Test definitions are in cases directories under "manual", "runtime", "sdk" and "selftest" directories.