Ktest survey response

= ktest survey response = ktest survey response provided by Steven Rostedt

Survey Questions

 * What is the name of your test framework? ktest

Does your test framework:

source code access

 * access source code repositories for the software under test? yes
 * access source code repositories for the test software? yes
 * include the source for the test software? no
 * provide interfaces for developers to perform code reviews? no
 * detect that the software under test has a new version? no
 * if so, how? n/a
 * detect that the test software has a new version? no

test definitions
Does your test system:
 * have a test definition repository? no
 * if so, what data format or language is used? n/a

Does your test definition include:
 * source code (or source code location)? yes
 * dependency information? no
 * execution instructions? yes
 * command line variants? yes
 * environment variants? yes
 * setup instructions? yes
 * cleanup instructions? no
 * if anything else, please describe:

Does your test system:
 * provide a set of existing tests? yes
 * if so, how many? 4

build management
Does your test system:
 * build the software under test (e.g. the kernel)? yes
 * build the test software? yes, but only if a script provided does so
 * build other software (such as the distro, libraries, firmware)? yes, but only if a script provided does so
 * support cross-compilation? yes
 * require a toolchain or build system for the SUT? yes
 * require a toolchain or build system for the test software? no
 * come with pre-built toolchains? no
 * store the build artifacts for generated software? no
 * in what format is the build metadata stored (e.g. json)? shell scripts
 * are the build artifacts stored as raw files or in a database? raw
 * if a database, what database?

Test scheduling/management
Does your test system:
 * check that dependencies are met before a test is run? no
 * schedule the test for the DUT? no
 * select an appropriate individual DUT based on SUT or test attributes? no
 * reserve the DUT? no
 * release the DUT? no
 * install the software under test to the DUT? yes
 * install required packages before a test is run? yes, only if script is provided
 * require particular bootloader on the DUT? Works with grub, sys/extlinux, tftboot (anything that a script can provide)
 * deploy the test program to the DUT? yes
 * prepare the test environment on the DUT? no
 * start a monitor (another process to collect data) on the DUT? yes
 * start a monitor on external equipment? yes
 * initiate the test on the DUT? yes
 * clean up the test environment on the DUT? yes

DUT control
Does your test system:
 * store board configuration data? no
 * in what format?
 * store external equipment configuration data? no
 * in what format?
 * power cycle the DUT? yes
 * monitor the power usage during a run? no
 * gather a kernel trace during a run? yes
 * claim other hardware resources or machines (other than the DUT) for use during a test? no
 * reserve a board for interactive use (ie remove it from automated testing)? no
 * provide a web-based control interface for the lab? no
 * provide a CLI control interface for the lab? no

Run artifact handling
Does your test system:
 * store run artifacts yes
 * in what format? raw format
 * put the run meta-data in a database? yes
 * if so, which database? file system
 * parse the test logs for results? yes
 * convert data from test logs into a unified format? no
 * if so, what is the format? n/a
 * evaluate pass criteria for a test (e.g. ignored results, counts or thresholds)? yes
 * do you have a common set of result names: (e.g. pass, fail, skip, etc.) yes
 * if so, what are they? PASS, FAIL
 * How is run data collected from the DUT? monitoring consoles
 * How is run data collected from external equipment?
 * Is external equipment data parsed? monitoring consoles

User interface
Does your test system:
 * have a visualization system? no
 * show build artifacts to users? no
 * show run artifacts to users? no
 * do you have a common set of result colors? no
 * if so, what are they? n/a
 * generate reports for test runs? yes
 * notify users of test results by e-mail? yes (recently added feature)
 * can you query (aggregate and filter) the build meta-data? yes, with grep
 * can you query (aggregate and filter) the run meta-data? yes, with grep
 * what language or data format is used for online results presentation? plain text
 * what language or data format is used for reports? plain text
 * does your test system have a CLI control tool? no
 * what is it called? n/a

Languages:

 * what is the base language of your test framework core? Perl
 * What languages or data formats is the user required to learn? bash

Can a user do the following with your test framework:

 * manually request that a test be executed (independent of a CI trigger)? yes
 * see the results of recent tests? yes (if they save the log)
 * set the pass criteria for a test? yes
 * set the threshold value for a benchmark test? yes
 * set the list of testcase results to ignore? sorta
 * provide a rating for a test? (e.g. give it 4 stars out of 5) no
 * customize a test? yes
 * alter the command line for the test program? yes
 * alter the environment of the test program? set
 * specify to skip a testcase? yes
 * set a new expected value for a test? yes
 * edit the test program source? yes
 * customize the notification criteria? yes
 * customize the notification mechanism (eg. e-mail, text) yes
 * generate a custom report for a set of runs? no
 * save the report parameters to generate the same report in the future? yes

Requirements
Does your test framework:
 * require minimum software on the DUT? yes
 * if so, what? Usually sshd (to load kernels on the target, and install for reboot)
 * require minimum hardware on the DUT (e.g. memory) yes
 * if so, what?
 * require agent software on the DUT? (e.g. extra software besides production software) yes
 * If so, what agent? sshd (but if a target boots from an external source, the externals source can have the code to load)
 * is there optional agent software or libraries for the DUT? no
 * require external hardware in your labs? no idea what that means

APIS
Does your test framework:
 * use existing APIs or data formats to interact within itself, or with 3rd-party modules? no
 * have a published API for any of its sub-module interactions (any of the lines in the diagram)? no
 * Please provide a link or links to the APIs?


 * What is the nature of the APIs you currently use? Mostly a config file where you can define pretty much all commands with a shell script.

Relationship to other software:

 * what major components does your test framework use? make, gcc, build tools. ssh, scp, grub, install tools, etc.
 * does your test framework interoperate with other test frameworks or software? no (but it can, via shell scripts)
 * which ones?

Overview
Please list your major components here:


 * Build - builds a kernel (can check for warnings)
 * Install - installs the kernel 
 * Boot - boots the kernel (checking for errors)
 * Test - runs test described by shell scripts

= Additional Data =