Ci-rt survey response

= ci-rt survey response = ci-rt survey response provided by Manuel Traut

r4d is no test-system. It's just a tool for managing test-racks and control them by libvirt.

I filled out the survey for 'ci-rt' the testsystem used by RT_PREEMPT developers to test different kernels with different configs on various hardwares. The ci-rt testsystem uses r4d to control the hardware.

= diagram missing element = If you have an element that is not featured in this diagram, please let us know.

Dynamically creating test jobs based on a test-description.

Survey Questions

 * What is the name of your test framework? ci-rt (ci-rt.linutronix.de)

Does your test framework:

source code access

 * access source code repositories for the software under test? yes, via git
 * access source code repositories for the test software? yes, via git
 * include the source for the test software? test-scripts can be part of the test-description, binaries (e.g. cyclictest) needs to be installed on the DUT.
 * provide interfaces for developers to perform code reviews? no
 * detect that the software under test has a new version? yes
 * if so, how? (e.g. polling a repository, a git hook, scanning a mail list, etc.) git hook.
 * detect that the test software has a new version? the test-description includes a git-ref of the test-software to be used.

test definitions
Does your test system:
 * have a test definition repository?
 * if so, what data format or language is used (e.g. yaml, json, shell script) Yes, it's a folder/file layout, the files are either lists that refer to other files, key-value pairs, kernel-patches, or kconfig overlays and kernel configs.

Does your test definition include:
 * source code (or source code location)? No
 * dependency information? No
 * execution instructions? Yes, you can specify if a kernel variant should be deployed to a target and run e.g. cyclictest with different loads; or if it should be just bootet on a target, or just compiled.
 * command line variants? Yes, for kernel params and cyclictest.
 * environment variants? Yes, for kernel compile.
 * setup instructions? No.
 * cleanup instructions? No.
 * if anything else, please describe: ci-rt is able to build and test different variants of kernels based on different kernel configs, overlays and architectures.

Does your test system:
 * provide a set of existing tests?
 * if so, how many? kernel compile, kernel boot, cyclictest, generictest

build management
Does your test system:
 * build the software under test (e.g. the kernel)? Yes and analyze compiler warnings/errors.
 * build the test software? No.
 * build other software (such as the distro, libraries, firmware)? No.
 * support cross-compilation? Yes.
 * require a toolchain or build system for the SUT? Yes, toolchains.
 * require a toolchain or build system for the test software? N/A.
 * come with pre-built toolchains? No. We recommend to use the toolchains that are in Debian/stretch.
 * store the build artifacts for generated software? ci-rt generates a compile script per kernel variant. This is stored.
 * in what format is the build metadata stored (e.g. json)? Debian packages.
 * are the build artifacts stored as raw files or in a database? raw-files. Additionally artifacts that are needed to reproduce the tests without ci-rt are stored in a DB that is accesible by a web-application.
 * if a database, what database? postgresql with row-based authentication.

Test scheduling/management
Does your test system:
 * check that dependencies are met before a test is run? There is a sanity check for the test-description. Also the requirements on the compile boxes (cross-toolchains) and on the target (cyclictest, ..) are checked.
 * schedule the test for the DUT? Yes.
 * select an appropriate individual DUT based on SUT or test attributes? Yes.
 * reserve the DUT? Yes.
 * release the DUT? Yes.
 * install the software under test to the DUT? Yes.
 * install required packages before a test is run? No.
 * require particular bootloader on the DUT? (e.g. grub, uboot, etc.) No.
 * deploy the test program to the DUT? Yes. (some scripts)
 * prepare the test environment on the DUT? Yes.
 * start a monitor (another process to collect data) on the DUT? No.
 * start a monitor on external equipment? No.
 * initiate the test on the DUT? Yes.
 * clean up the test environment on the DUT? Yes.

DUT control
The DUT control system in ci-rt is r4d.

Does your test system:
 * store board configuration data? Yes.
 * in what format? R4D
 * store external equipment configuration data?
 * in what format? No.
 * power cycle the DUT? Yes.
 * monitor the power usage during a run? No.
 * gather a kernel trace during a run? Yes.
 * claim other hardware resources or machines (other than the DUT) for use during a test? No.
 * reserve a board for interactive use (ie remove it from automated testing)? Yes.
 * provide a web-based control interface for the lab? Yes, via Jenkins
 * provide a CLI control interface for the lab? Yes, via Jenkins

Run artifact handling
Does your test system:
 * store run artifacts: yes
 * in what format? junit-xml
 * put the run meta-data in a database? yes
 * if so, which database? postgresql
 * parse the test logs for results? yes - kernel-compile, kernel-boot
 * convert data from test logs into a unified format? yes
 * if so, what is the format? junit-xml
 * evaluate pass criteria for a test (e.g. ignored results, counts or thresholds)?
 * do you have a common set of result names: (e.g. pass, fail, skip, etc.)
 * if so, what are they? cyclictest threshold, no common name.
 * How is run data collected from the DUT?
 * e.g. by pushing from the DUT, or pulling from a server? by Jenkins stashes.
 * How is run data collected from external equipment? serial bootlog is recorded via r4d/libvirt
 * Is external equipment data parsed? Yes, serial bootlog is analyzed.

User interface
Does your test system:
 * have a visualization system? Yes, tomcat hosted web-application.
 * show build artifacts to users? Yes.
 * show run artifacts to users? Yes.
 * do you have a common set of result colors? yes
 * if so, what are they? green / red
 * generate reports for test runs? Yes.
 * notify users of test results by e-mail? Yes. And Administrators about problems with the build-infrastructure. E.g. target is not booting.
 * can you query (aggregate and filter) the build meta-data? No.
 * can you query (aggregate and filter) the run meta-data? No. Only graphical compare of cyclictest results.
 * what language or data format is used for online results presentation? (e.g. HTML, Javascript, xml, etc.) HTML, jsp
 * what language or data format is used for reports? (e.g. PDF, excel, etc.) txt, junit-xml
 * does your test system have a CLI control tool? yes
 * what is it called? jenkins-cli

Languages:
Examples: json, python, yaml, C, javascript, etc.
 * what is the base language of your test framework core? groovy
 * What languages or data formats is the user required to learn? the format of the ci-rt test-description

Can a user do the following with your test framework:

 * manually request that a test be executed (independent of a CI trigger)? yes
 * see the results of recent tests? yes
 * set the pass criteria for a test? yes, for cyclictest
 * set the threshold value for a benchmark test? yes, for cyclictest
 * set the list of testcase results to ignore? no
 * provide a rating for a test? (e.g. give it 4 stars out of 5) no
 * customize a test? yes
 * alter the command line for the test program? yes
 * alter the environment of the test program? yes
 * specify to skip a testcase? yes
 * set a new expected value for a test? yes
 * edit the test program source? no
 * customize the notification criteria? no
 * customize the notification mechanism (eg. e-mail, text) no
 * generate a custom report for a set of runs? no
 * save the report parameters to generate the same report in the future? no

Requirements
Does your test framework:
 * require minimum software on the DUT? yes
 * If so, what? (e.g. POSIX shell or some other interpreter, specific libraries, command line tools, etc.) openjdk8
 * require minimum hardware on the DUT (e.g. memory) no
 * require agent software on the DUT? (e.g. extra software besides production software) no
 * If so, what agent?
 * is there optional agent software or libraries for the DUT? no
 * require external hardware in your labs? Serialserver, Powercontrol

APIS
Does your test framework:
 * use existing APIs or data formats to interact within itself, or with 3rd-party modules? junit-xml
 * have a published API for any of its sub-module interactions (any of the lines in the diagram)?
 * Please provide a link or links to the APIs?


 * DUT Control:
 * R4D    - https://github.com/ci-rt/r4d
 * libvirt - https://github.com/ci-rt/libvirt-debian

Sorry - this is kind of open-ended... Are they:
 * What is the nature of the APIs you currently use?
 * RPCs? SOAP
 * Unix-style? (command line invocation, while grabbing sub-tool output)
 * compiled libraries?
 * interpreter modules or libraries?
 * web-based APIs?
 * something else?

Relationship to other software:

 * what major components does your test framework use? Jenkins, R4D, libvirt, tomcat, postgresql


 * does your test framework interoperate with other test frameworks or software? no
 * which ones?

Overview
Please list the major components of your test system.


 * Jenkins - job trigger, visualization, notification
 * ci-rt-lib (jenkins, groovy lib) - variant management, test scheduling, kernel build, kernel deploy, log/result retrieval
 * libvirt/r4d - access serial console of DUT and power control DUTs
 * webapplication - visualization, provide test results public (without having jenkins accesible)
 * pyjutest - run commands and produce junit-xml including stdout/err return value, runtime

= Additional Data =