Test Glossary

This is a list of terms that are used by various test system in the Linux ecosystem, and by the Test Standards that are discussed and proposed on this site.

= Terms =
 * Actual Value - the value that was seen for an operation performed by a test
 * Bisection - automatic testing of SUT variations to find the source of a problem
 * Boot - to start the target from an off state. The end of 'boot' is the point of time when a test can be started, which will be different depending on whether a test is testing the bootloader, the kernel or some user-space software.
 * Build artifact - item created during build of the software under test
 * Build manager - a machine or process that performs builds of the software under test. Also known as a build server, when it is a machine.
 * Continuous Integration Loop - the process of automatically testing software as it is integrated. Usually, a commit to a source repository (the trigger) will cause a build, deploy, instantiation and test of the software, with notification of results to interested parties. Also "CI Loop".
 * Dependency - indicates a pre-requisite that must be filled in order for a test to run
 * Some examples: test requires root access, test needs 100 meg of memory, some program or package must be installed, etc.
 * Device Under Test (DUT) - the product, board or device that is being tested. It consists of hardware and software under test. (See also 'board', 'target')
 * This glossary uses 'target' as the more generic term for a thing that hosts the system being tested (for example, it could be a Virtual Machine)
 * Deploy - put the test program or system software on the target
 * Expected value - the value that was expected for an operation performed by a test
 * Lab - a collection of resources for testing one or more targets (also 'board farm')
 * Log - one of the run artifacts - output from the test program or test framework
 * Log Parsing - extracting information from a log into a machine-processable format (possibly into a common format)
 * Monitor (noun) - a program or process to watch some attribute (e.g. power) while the test is running
 * A monitor could be running either on or off the target, and it could be collecting data from either on or off the target.
 * Notification - communication based on results of test (triggered by results and including results)
 * Pass criteria - set of constraints indicating pass/fail conditions for a test
 * PDU - Power Distribution Unit - a piece of hardware used to control power to one or more DUTs (used by LAVA)
 * Report generation - collecting run data and putting it into a formatted output
 * Request (noun) - a request to execute a test
 * Result - the status indicated by a test - pass/fail (or something else) for a Run
 * Results query - Selection and filtering of data from runs, to find patterns
 * Run (noun) - an execution instance of a test (in Jenkins, a build)
 * Run artifact - item created during a run of the test program
 * Serial console - the Linux kernel console connected over a serial connection
 * Often, the kernel serial console is the most reliable way to receive test output data. This is particularly true if the kernel is the software under test and the test is a boot test which might fail part-way through the boot.
 * Software under test - the software being tested
 * System under test - The complete system being tested, including hardware and software. See Target, DUT.
 * Target - The board, device or virtual machine in which tests are performed. Usually this is thought of to consist of both the Device Under Test and the accompanying Software Under Test.
 * Target Manager - a process or machine that controls and schedules a target (see also DUT controller, DUT supervisor, DUT scheduler)
 * This entity does things like booting the target, controlling networking to the target, moving data to and from the target, installing software on the target, and providing access to a target's serial console.
 * Test agent - software running on the DUT that assists in test operations (e.g. test deployment, execution, log gathering, debugging
 * One example would be 'adb', for Android-based systems)
 * Test definition - meta-data and software that comprise a particular test
 * Test plan - the definition and meta-data for a group of tests run together (e.g. sequentially on the same target). This always includes the list of tests, but may also include per-test data such as timeout, test variables or reporting/notification instructions.
 * Test program - a script or binary on the target that performs the test
 * Test scheduler - program for scheduling tests (selecting a target for a test, reserving it, releasing it)
 * Test setup - refers to the sequence of operations required to prepare the target or the lab environment for the test
 * This might include other major operations, such as deploying the system software to the target.
 * Test software - source and/or binary that implements the test
 * Transport (noun) - the method of communicating and transferring data between the test system and the target
 * Trigger (noun) - an event that causes the CI loop to start
 * Variant - arguments or data that affect the execution and output of a test (e.g. test program command line; Fuego calls this a 'spec')
 * Visualization - allowing the viewing of test artifacts, in aggregated form (e.g. multiple runs plotted in a single diagram)

Contested or confusing terms

 * Device Under Test - was confusing to enterprise testers and those using virtual machines. Argument was that they have no hardware under test, and that "Device" is confusing, and is treated as just another resource for the test.
 * Deploy - put the test program or system software on the target. Putting the system software on the target is much different from putting the test program on the target, so usage of "deploy" is ambiguous.

Candidate terms
be a better fit. Otherwise 'device types', 'groups' or 'classes' come to mind.
 * Board - the board hardware under test. See also Device Under Test and Target.
 * Feature - an attribute of a DUT or SUT or test environment that can be used to match tests. Requiring a DUT to have a particular feature could be a test dependency.
 * this is called 'feature' by labgrid
 * this is called a 'device tag' by LAVA
 * this was called a 'capability' by JTA (the precursor to Fuego)
 * Device type - The name of a set of DUTs that have identical or similar features, such that any one of them can be used to run a test (used by LAVA)
 * Tim Bird's comment: Some examples would be good. I'm not sure I like this.  Is there a related term for the set of boards that have a particular type?  (e.g. something that refers to the set of boards, rather than the characteristics of the set?)  Maybe board pool or target pool?
 * Jan Lubbe's comment: We may want to have the ability to put one DUT into several (overlapping) groups, so having 'labels' like the Jenkins executors may
 * Interactive DUT access - the ability to take a board out of automated testing service, for use in interactive testing or debugging sessions (or for some other reason.)
 * other possible terms: 'target-offlining'? 'target reservation'?

Rejected terms
Note: reason for rejection is in italics.


 * Test Profile - same thing as Test Definition. Test Profile is used by Phoronix Test Suite.
 * DUT Supervisor - provides connection to the DUT and abstraction for DUT management actions (e.g. dut_boot, dut_login, dut_exec, dut_copyfrom, dut_copyto commands - additional software on NanoPi) (used by SLAV). This glossary uses "Target manager" instead.
 * DUT controller - program and hardware for controlling a DUT (reboot, provision, etc.). Confusing because of DUT confusion.  Subsumed in "Target Manager"
 * DUT scheduler - program for managing access to a DUT (take online/offline, make available for interactive use) Confusing because of DUT confusion. Subsumed in "Target Manager".
 * Provision (verb) - arrange the DUT and the lab environment (including other external hardware) for a test
 * This may include installing the SUT to the device under test and booting the DUT.
 * Not enough people used this for it to be useful for communication.

= External Test System Glossaries =
 * Fuego: http://fuegotest.org/wiki/Glossary
 * LAVA: https://validation.linaro.org/static/docs/v2/glossary.html

Generic terminology lists

 * https://www.softwaretestinghelp.com/software-testing-terms-complete-glossary/
 * Here's one that overly comprehensive: The International Software Testing Qualification Board Glossary