Test definition survey
Here is a list of test definition fields, attributes, file formats, operations, instructions, functions, etc. (I won't know what even what they consist of until I see them).
This is the object in your test system that "defines" what a test is. It likely has meta-data about the program to run, how to get the program started, maybe what things are required for the program to run, how the results should be interpreted, etc.
Here is a link to test definitions in different systems: One simple, one characteristic, and link to repository containing lots of them:
- (simple) Functional.hello_world: https://bitbucket.org/tbird20d/fuego-core/src/master/engine/tests/Functional.hello_world/
- (characteristic) Benchmark.bonnie: https://bitbucket.org/tbird20d/fuego-core/src/master/engine/tests/Benchmark.bonnie/
- (repository) https://bitbucket.org/tbird20d/fuego-core/src/master/engine/tests/
- (test program source)
- (to be filled out)
- (simple) busybox: https://github.com/Linaro/test-definitions/tree/master/automated/linux/busybox
- (characteristic) libhugetblfs: https://github.com/Linaro/test-definitions/tree/master/automated/linux/libhugetlbfs
- (repository) https://github.com/Linaro/test-definitions/
For the 'files' part, each test in test-definitions is stored in a separate directory. The directory has to contain at least the YAML file that is compliant with LAVA test definition. We have a sanity check script (validate.py) that is executed on any pull request. This ensures all files pushed to the repository are compliant. Usual practice is that test directory contains a test script (shell script). Script is responsible for installing dependencies, running tests and parsing results. There is no mandatory format there but test-definitions provides a library of functions that help writing test scripts. There are libraries for 'linux' and 'android'. We also host directory for manual tests and simple executor for them but in the context of automated testing these are irrelevant.
An "on target" test of the compiler:
(same directory has simple python/perl tests and so on)
http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/lib/oeqa/files (for the test files for context, they're just hello world examples)
- (characteristic) devtool: http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/lib/oeqa/selftest/cases/devtool.py
This is a "selftest" for the "devtool" command that is part of the overall build system, its a bit more complex with shared functions and tests for each of devtool's subcommands.
This has all the test code and core test definitions. Test definitions are in cases directories under "manual", "runtime", "sdk" and "selftest" directories.
- bitback-selftest at: http://git.yoctoproject.org/cgit.cgi/poky/tree/bitbake/lib/bb/tests