LAVA survey response

= LAVA survey response = LAVA survey response provided by Matt Hart Additional information provided by Neil Williams and Milosz Wasilewski.

Survey Questions

 * What is the name of your test framework? LAVA

Which of the aspects below of the CI loop does your test framework perform?

Lab/Board Farm, Scheduler, DUT Control (Deploy, Provision, Test, Collect Results)

Does your test framework:

source code access

 * access source code repositories for the software under test? No
 * access source code repositories for the test software? Yes, all the common VCS
 * include the source for the test software? No
 * provide interfaces for developers to perform code reviews? No
 * detect that the software under test has a new version? No, most people would use jenkins or similar
 * if so, how? (e.g. polling a repository, a git hook, scanning a mail list, etc.)
 * detect that the test software has a new version?

test definitions
Does your test system:  With a few exceptions, the test definitions are in public git repositories and the URL of each test definition is in the test job definition. LAVA records the git commit hash of the version used for each specific test job within the results metadata of the test job.
 * have a test definition repository? LAVA does not come with tests, however Linaro does maintain a set of job definitions
 * Where are they?

 It depends. Some are wrappers for existing test operations, like LTP and kselftest. Test definitions are being made more and more portable (so that the operations within the test can be reproduced more easily outside LAVA). The test should check if LAVA helpers are available and continue without them where possible.
 * Are these tests generalizes or specific to Linaro test projects or labs?

Some test definitions expect to find peripheral hardware attached to a DUT which would mean that the test would fail when run on a similar device without the peripheral. The test job definition will contain tags which determine which devices can be scheduled for those test jobs. Otherwise, most test definitions will work outside Linaro labs.

To understand the test definitions better, start with an existing test job using a device which is familiar to you and follow the links from the test job definition. There are a lot of permutations and variations, just browsing the git repositories would get confusing / boring. Linaro also has a lot of AOSP (Android Open Source Project) related test jobs and those are typically more complex than tests for operating systems like Yocto, OE or Debian.

Here is the repository of test definitions: https://git.linaro.org/qa/test-definitions.git

Tests can be run standalone (as test scripts on DUT) or in LAVA. There is a README: https://git.linaro.org/qa/test-definitions.git/tree/automated/README There is also a standalone executor called test-runner.py: https://git.linaro.org/qa/test-definitions.git/tree/automated/utils/test-runner.py

The main idea is to have a standard way of starting/executing tests and unified way of reporting results.

Does your test definition include:
 * what data format or language is used for the test definitions? YAML
 * source code (or source code location)? Yes
 * dependency information? Yes
 * execution instructions? Yes
 * command line variants? Yes
 * environment variants? Yes
 * setup instructions? Yes
 * cleanup instructions? Yes
 * if anything else, please describe:

Does your test system:
 * provide a set of existing tests? No
 * if so, how many?

build management
Does your test system: As long as the deployed OS has enough support to do the build, a test can build on the DUT itself. This is useful for small utilities or where the test program needs to know something specific about the device/kernel. Some test jobs need to run for hours, so a few minutes at the front compiling a small part of the test isn't a problem.
 * build the software under test (e.g. the kernel)? No
 * build the test software? Sometimes, quite a lot of LAVA users will build the test software on the device they are testing before executing it

LAVA also supports running tests in an LXC which can then use USB or TCP/IP to communicate with the DUT - for operating systems which do not directly support POSIX on the DUT and which provide tools to interact in this way, e.g. AOSP. The power management team regularly build a small utility in the LXC to communicate with an energy probe which is attached to the DUT. This use case is becoming more common. See https://validation.linaro.org/static/docs/v2/deploy-lxc.html

Building auxiliary software like this is typically done at the start of one of the test actions as a setup phase. LAVA supports raising an exception to kill the test job early if this setup stage fails (e.g. due to a build error) to avoid producing spurious results from a broken setup phase.

Other operations can be done during DUT deployment to add the LAVA test helpers to the rootfs.

See https://github.com/ivoire/Artifactorial
 * build other software (such as the distro, libraries, firmware)? 'No
 * support cross-compilation? No
 * require a toolchain or build system for the SUT? Yes
 * require a toolchain or build system for the test software? No, it can be built on the device, though pre-built is obviously faster
 * come with pre-built toolchains? No
 * store the build artifacts for generated software? No, however pushing to external storage is supported (Artifactorial)

It is implemented as a service for Linaro here: https://archive.validation.linaro.org/

Note, however, that the same functionality can be used to push to any third party service, it doesn't have to be Artifactorial. Any service with which the DUT can authenticate and transfer data will work. The URL returned can be stored as a test reference in the test job results.

Tim's note: run artifacts are pushed to storage by the DUT.


 * in what format is the build metadata stored (e.g. json)?
 * are the build artifacts stored as raw files or in a database?
 * if a database, what database?

Test scheduling/management
Does your test system:
 * check that dependencies are met before a test is run? Yes
 * schedule the test for the DUT? Yes
 * select an appropriate individual DUT based on SUT or test attributes? Yes
 * reserve the DUT? Yes
 * release the DUT? Yes
 * install the software under test to the DUT? Yes
 * install required packages before a test is run? Yes
 * require particular bootloader on the DUT? (e.g. grub, uboot, etc.) Yes
 * deploy the test program to the DUT? Yes
 * prepare the test environment on the DUT? Yes
 * start a monitor (another process to collect data) on the DUT? No
 * start a monitor on external equipment? No
 * initiate the test on the DUT? Yes
 * clean up the test environment on the DUT? Yes

DUT control
Does your test system:
 * store board configuration data? Yes
 * in what format? YAML, rendered from Jinja2
 * store external equipment configuration data? Yes
 * in what format? YAML, rendered from Jinja2
 * power cycle the DUT? Yes
 * monitor the power usage during a run? Possibly, there is basic support for ARM energy probes
 * gather a kernel trace during a run? Yes
 * claim other hardware resources or machines (other than the DUT) for use during a test? Yes, and can claim other DUT in a multi-node job
 * reserve a board for interactive use (ie remove it from automated testing)? Not directly, but users use LAVA to provision a device and hand over SSH access
 * provide a web-based control interface for the lab? Yes
 * provide a CLI control interface for the lab? Yes

Run artifact handling
Does your test system: ''' Live test result reporting is a key feature. It ensures that a test job produces some results even if the device locks up part of the way through the test. '''
 * store run artifacts No
 * in what format?
 * put the run meta-data in a database? Yes
 * if so, which database? LAVA results database, postgres
 * parse the test logs for results? Yes, during the run 
 * convert data from test logs into a unified format? Yes
 * if so, what is the format? Stored in database, can be fetched as YAML
 * evaluate pass criteria for a test (e.g. ignored results, counts or thresholds)? Yes
 * do you have a common set of result name? Yes
 * if so, what are they? Pass, Fail, Skip, Unknown
 * How is run data collected from the DUT? Parsed from the DUT serial output on the fly

LAVA essentially scans the serial console output for results token. A unique start and end string must be pre-defined and a Python regular expression is provided by the test writer to parse the test results which must appear between those strings.
 * How is run data collected from external equipment? External equipment is considered another DUT
 * Is external equipment data parsed? Same as other DUT

User interface
Does your test system:
 * have a visualization system? Yes
 * show build artifacts to users? No
 * show run artifacts to users? Yes, if pushed to external storage a link can be put in the results
 * do you have a common set of result colors? No
 * if so, what are they? N/A
 * generate reports for test runs? No
 * notify users of test results by e-mail? Only job status
 * can you query (aggregate and filter) the build meta-data? No
 * can you query (aggregate and filter) the run meta-data? Yes, if stored as a result
 * what language or data format is used for online results presentation? (e.g. HTML, Javascript, xml, etc.) Javascript
 * what language or data format is used for reports? (e.g. PDF, excel, etc.) N/A
 * does your test system have a CLI control tool? Yes
 * what is it called? LAVACLI

Languages:
Examples: json, python, yaml, C, javascript, etc. What languages or data formats is the user required to learn? (as opposed to those used internally)
 * what is the base language of your test framework core? Python


 * YAML for writing a job definition and device description
 * Shell script for writing a test definition

Can a user do the following with your test framework:

 * manually request that a test be executed (independent of a CI trigger)? 'Yes
 * see the results of recent tests? Yes
 * set the pass criteria for a test? Yes, would require editing the job
 * set the threshold value for a benchmark test? Yes, would require editing the job
 * set the list of testcase results to ignore? Yes, would require editing the job
 * provide a rating for a test? (e.g. give it 4 stars out of 5) No
 * customize a test? Yes, would require editing the job
 * alter the command line for the test program? Yes, would require editing the job
 * alter the environment of the test program? Yes, would require editing the job
 * specify to skip a testcase? Yes, would require editing the job
 * set a new expected value for a test? Yes, would require editing the job
 * edit the test program source? Yes, would require editing the job
 * customize the notification criteria? Yes
 * customize the notification mechanism (eg. e-mail, text) Yes
 * generate a custom report for a set of runs? Yes
 * save the report parameters to generate the same report in the future? Yes

Requirements
Does your test framework: Currently, testing devices without a shell precludes interactive commands or DUT-side operations initiated by LAVA, but some form of limited interaction is being scoped for a future release of LAVA.
 * require minimum software on the DUT? Bootloader
 * If so, what? POSIX shell for most DUT, however IOT devices are supported without a shell

In the case of an IOT device without a shell, the test initiation is baked into the boot sequence for the board. The IoT board typically provides an interface to deploy a new boot file, gets reset and executes the boot file immediately. Test jobs often end in under a minute.


 * require minimum hardware on the DUT (e.g. memory) Serial port
 * require agent software on the DUT? (e.g. extra software besides production software) No
 * If so, what agent?
 * is there optional agent software or libraries for the DUT? No
 * require external hardware in your labs? Power control is required for most DUT types

APIS
Does your test framework:
 * use existing APIs or data formats to interact within itself, or with 3rd-party modules? ZMQ (see http://zeromq.org/)
 * have a published API for any of its sub-module interactions (any of the lines in the diagram)? No
 * Please provide a link or links to the APIs?

Sorry - this is kind of open-ended... Are they:
 * What is the nature of the APIs you currently use?
 * RPCs?
 * Unix-style? (command line invocation, while grabbing sub-tool output)
 * compiled libraries?
 * interpreter modules or libraries?
 * web-based APIs?
 * something else?


 * ZMQ is used between LAVA workers (dispatchers) and the master, to schedule jobs
 * ZMQ is kind of like message-based network sockets (Note by Tim)
 * XML-RPC is for users to submit jobs, access results, and control the lab

Test results and other data can be extracted through web-based APIs as YAML etc. There is also a limited REST API.

Relationship to other software:

 * what major components does your test framework use? Jenkins, Squad
 * does your test framework interoperate with other test frameworks or software?
 * which ones? A common LAVA setup is Jenkins to create the builds, LAVA to execute the tests, and Squad/KernelCI to consume the results

Overview
Please list your major components here:


 * LAVA Server - (DUT Scheduler) - UI, Results storage, Device configuration files, Job scheduling, User interaction
 * LAVA Dispatcher - (DUT Controller) - Device interaction (deploy, boot, execute tests), Device Power Control, Test results Parsing

Glossary
Also, please suggest any terms or concepts that are missing.

PDU - Power Distribution Unit, however has become a standard term for automation power control.

= Additional Data =