LKFT survey response

= LKFT survey response = LKFT survey response provided by Milosz Wasilewski

Survey Questions

 * What is the name of your test framework? Linux Kernel Functional Testing (LKFT)

Does your test framework:

source code access

 * access source code repositories for the software under test? yes
 * access source code repositories for the test software? yes
 * include the source for the test software? no
 * provide interfaces for developers to perform code reviews? no
 * detect that the software under test has a new version? yes
 * if so, how? (e.g. polling a repository, a git hook, scanning a mail list, etc.) jenkins trigger on repository change
 * detect that the test software has a new version? no - use always the latest one or leave it to the user

test definitions
Does your test system:
 * have a test definition repository? yes
 * if so, what data format or language is used (e.g. yaml, json, shell script) yaml - wraps around shell/python scripts or executables

Does your test definition include:
 * source code (or source code location)? yes - both
 * dependency information? yes/no - test scripts have to make sure that dependencies are met, but it's not enforced
 * execution instructions? yes
 * command line variants? no
 * environment variants? yes
 * setup instructions? leave it to user - test setup is partially performed by execution engine (LAVA) and partially by test scripts
 * cleanup instructions? no
 * if anything else, please describe:

Does your test system:
 * provide a set of existing tests? yes
 * if so, how many? Hard to say. Currently we're running LTP, kselftest and libhugetlbfs, but there are a couple more tests available.

build management
Does your test system:
 * build the software under test (e.g. the kernel)? yes
 * build the test software? yes
 * build other software (such as the distro, libraries, firmware)? yes
 * support cross-compilation? yes
 * require a toolchain or build system for the SUT?


 * require a toolchain or build system for the test software?
 * come with pre-built toolchains?
 * store the build artifacts for generated software? yes
 * in what format is the build metadata stored (e.g. json)? json, key-value properties files
 * are the build artifacts stored as raw files or in a database? raw files
 * if a database, what database?

Test scheduling/management
Does your test system: LKFT uses LAVA for execution. Lava reboots between tests, so the slate is clean. From my experience not many test suites cleans after themselves. LAVA doesn't clean up after itself. However, even rebooting doesn't always guarantee clean start of the following tests. 
 * check that dependencies are met before a test is run? yes - as a part of test job setup
 * schedule the test for the DUT? yes
 * select an appropriate individual DUT based on SUT or test attributes? yes
 * reserve the DUT? yes
 * release the DUT? yes
 * install the software under test to the DUT? yes
 * install required packages before a test is run? leave to user - dependency installation can be part of a test execution
 * require particular bootloader on the DUT? (e.g. grub, uboot, etc.) no
 * deploy the test program to the DUT? yes
 * prepare the test environment on the DUT? yes
 * start a monitor (another process to collect data) on the DUT? no - test monitoring happens over serial connection to DUT
 * start a monitor on external equipment? yes
 * initiate the test on the DUT? yes
 * clean up the test environment on the DUT? no

DUT control
Does your test system:
 * store board configuration data? yes
 * in what format? jinja2 - part of LAVA configuration
 * store external equipment configuration data? yes
 * in what format? jinja2 - part of LAVA configuration
 * power cycle the DUT? yes
 * monitor the power usage during a run? no
 * gather a kernel trace during a run? no
 * claim other hardware resources or machines (other than the DUT) for use during a test? leave to user - can be configured as part of a test job
 * reserve a board for interactive use (ie remove it from automated testing)? leave to user - requires ssh access to DUT
 * provide a web-based control interface for the lab? yes
 * provide a CLI control interface for the lab? yes

Run artifact handling
Does your test system: 'xfail' is assigned to the test case that was executed properly and produced a 'fail' result. Than during result post-processing we check whether such failure is marked as 'known issue' (in database). If it is, then the result is changed to xfail. I know this isn't the correct usage of this state. IIRC xfail should simply be equal to 'pass' because we deliberately trigger the condition that is considered 'failed' for some reason. Anyway, LKFT uses xfail to mark failures that we know about but don't want to deal with right now. It is meant to make the distinction between 'known' failure and 'regression' which is 'unexpected failure'.
 * store run artifacts yes
 * in what format? leave to user - uploaded files are generated during the test, which is controlled by user
 * put the run meta-data in a database? yes
 * if so, which database? postgresql
 * parse the test logs for results? yes
 * convert data from test logs into a unified format? no
 * if so, what is the format? n/a
 * evaluate pass criteria for a test (e.g. ignored results, counts or thresholds)? yes
 * do you have a common set of result names: (e.g. pass, fail, skip, etc.) yes
 * if so, what are they? pass, fail, skip, xfail (for ignored failed test)


 * How is run data collected from the DUT? pushing from DUT over serial line
 * How is run data collected from external equipment? same as for DUT - any external equipment is considered a DUT
 * Is external equipment data parsed? yes

User interface
Does your test system:
 * have a visualization system? yes
 * show build artifacts to users? no
 * show run artifacts to users? yes
 * do you have a common set of result colors? yes
 * if so, what are they? red - fail, green - pass, yellow - skip, blue - xfail, grey - total
 * generate reports for test runs? yes
 * notify users of test results by e-mail? yes


 * can you query (aggregate and filter) the build meta-data? no
 * can you query (aggregate and filter) the run meta-data? no


 * what language or data format is used for online results presentation? (e.g. HTML, Javascript, xml, etc.) HTML and JSON (REST API)
 * what language or data format is used for reports? (e.g. PDF, excel, etc.) HTML, txt


 * does your test system have a CLI control tool? yes
 * what is it called? lavacli

Languages:
Examples: json, python, yaml, C, javascript, etc.
 * what is the base language of your test framework core? python

What languages or data formats is the user required to learn? yaml - as test definition description for LAVA

Can a user do the following with your test framework:

 * manually request that a test be executed (independent of a CI trigger)? yes
 * see the results of recent tests? yes
 * set the pass criteria for a test? no
 * set the threshold value for a benchmark test? not yet (WiP)
 * set the list of testcase results to ignore? yes
 * provide a rating for a test? (e.g. give it 4 stars out of 5) no
 * customize a test? yes
 * alter the command line for the test program? no
 * alter the environment of the test program? no
 * specify to skip a testcase? yes
 * set a new expected value for a test? no
 * edit the test program source? yes
 * customize the notification criteria? no
 * customize the notification mechanism (eg. e-mail, text) yes
 * generate a custom report for a set of runs? yes
 * save the report parameters to generate the same report in the future? yes

Requirements
Does your test framework:
 * require minimum software on the DUT? yes
 * if so, what? POSIX shell
 * require minimum hardware on the DUT (e.g. memory) yes
 * If so, what?  1) boot on power, 2) serial line with unique ID, 3) 'not-crazy' bootloader

An example of a board that doesn't do boot-on-power correctly is the BeagleBoard X15 from TI. It requires button press to start booting. We had to modify the boards and add relays to work around that.

'serial line with unique ID' means something that has unique id in /dev/serial/by-id. Since LAVA talks to devices using serial line and there can be multiple devices connected to the same host, serial ID number has to be unique.

Some examples of 'crazy' bootloaders are the "UEFI menu". LAVA can (barely) deal with it. All ncurses style menus are 'no no'; color codes in serial output are also impossible to work with.


 * require agent software on the DUT? yes
 * If so, what agent? lava helper scripts (LAVA overlay) that gets installed during setup phase; adb for android based testing
 * is there optional agent software or libraries for the DUT? no
 * require external hardware in your labs? yes - power relays, managed USB hubs

APIS
Does your test framework:
 * use existing APIs or data formats to interact within itself, or with 3rd-party modules? yes
 * have a published API for any of its sub-module interactions (any of the lines in the diagram)? yes
 * Please provide a link or links to the APIs?
 * LAVA XML-RPC: https://validation.linaro.org/api/help/
 * LAVA ZMQ: didn't find any docs, sorry :(
 * Jenkins API: https://ci.linaro.org/api/
 * SQUAD REST API: https://qa-reports.linaro.org/api/

Sorry - this is kind of open-ended... Are they:
 * What is the nature of the APIs you currently use?
 * RPCs? no
 * Unix-style? (command line invocation, while grabbing sub-tool output) no
 * compiled libraries? no
 * interpreter modules or libraries? no
 * web-based APIs? yes
 * something else?

Relationship to other software:

 * what major components does your test framework use?


 * Jenkins - for building SUT
 * LAVA - for test scheduling and execution
 * SQUAD - for result collection, aggregation and reporting
 * test-definitons.git - for test description, skip lists and known issues
 * does your test framework interoperate with other test frameworks or software? leave to user
 * which ones? 3rd party test suites can be used however they usually require wrapper scripts

Overview
Please list the major components of your test system.

Please list your major components here:
 * Jenkins - for building SUT
 * LAVA - for test scheduling and execution
 * SQUAD - for result collection, aggregation and reporting
 * test-definitons.git - for test description, skip lists and known issues

= Additional Data =