Opentest survey response

Opentest survey response, provided by Carlos Hernandez

Survey Questions

 * What is the name of your test framework? Opentest

Which of the aspects of the CI loop does your test framework perform?

Does your test framework:

source code access

 * overview: Opentest test provides various command line interfaces to facilitate integration with CI loops. Jenkins is used to detect code changes and trigger builds.
 * access source code repositories for the software under test? provided by Jenkins
 * access source code repositories for the test software? provided by Jenkins
 * include the source for the test software? yes
 * provide interfaces for developers to perform code reviews? No. Mainly kernel (email-based) flow. Each component defines its own review mechanism (email, bitbucket PR, etc.)
 * detect that the software under test has a new version? provided by Jenkins
 * if so, how? (e.g. polling a repository, a git hook, scanning a mail list, etc.) Jenkinks jobs (called internally auto-merge) poll every hour for code changes . Auto-merge runs static checks, build and boot tests before merging into integration branch
 * detect that the test software has a new version? Provided by the user, by default points to master latest but user can specify branch/commit to use

test definitions
Does your test system:
 * have a test definition repository? yes, Testlink
 * if so, what data format or language is used (e.g. yaml, json, shell script) It uses mysql db to store test definitions and they can be exported/imported to/from xml

Does your test definition include: Does your test system:
 * source code (or source code location)? yes, relative path is included
 * dependency information? yes, it is called capabilities in opentest
 * execution instructions? yes
 * command line variants? yes
 * environment variants? no
 * setup instructions? yes
 * cleanup instructions? yes
 * if anything else, please describe:
 * Test Execution Engine type (Lava, Vatf, etc.)
 * Hardware requirements: Type of HW assets and its capabilities
 * provide a set of existing tests? yes
 * if so, how many? around 5000 for Linux. Please note that for us ltp's syscall is just 1 test.

build management
Does your test system:
 * build the software under test (e.g. the kernel)? yes, in one use case via Launcher component. But usually it is handled out-of-band by Jenkins
 * build the test software? handled out-of-band by Jenkins
 * build other software (such as the distro, libraries, firmware)? handled out-of-band by Jenkins
 * support cross-compilation? handled out-of-band by Jenkins
 * require a toolchain or build system for the SUT? yes, handled out-of-band by Jenkins
 * require a toolchain or build system for the test software? when required handled out-of-band by Jenkins
 * come with pre-built toolchains? no
 * store the build artifacts for generated software? software assets are stored in external file servers, only link references are stored. Files are also stored in cache temporarily by test framework.
 * in what format is the build metadata stored (e.g. json)? postgresql
 * are the build artifacts stored as raw files or in a database? raw files in a file server
 * if a database, what database?

Test scheduling/management
Does your test system:
 * check that dependencies are met before a test is run? yes
 * schedule the test for the DUT? yes
 * select an appropriate individual DUT based on SUT or test attributes? yes
 * reserve the DUT? yes
 * release the DUT? yes
 * install the software under test to the DUT? yes
 * install required packages before a test is run? yes
 * require particular bootloader on the DUT? (e.g. grub, uboot, etc.) No. test script logic defines boot mechanism. Linux test cases currently use uboot but the framework makes no assumptions, it is totally handled by test script logic.
 * deploy the test program to the DUT? yes
 * prepare the test environment on the DUT? yes (nfs, tftp, etc.)
 * start a monitor (another process to collect data) on the DUT? it is controlled by test script. Linux test scripts do provide an option to monitor/collect cpu and mem usage
 * start a monitor on external equipment? drivers are provided to control external test equipment, e.g. multimeters, video capture cards, etc.
 * initiate the test on the DUT? yes
 * clean up the test environment on the DUT? yes

DUT control
Does your test system:
 * store board configuration data? yes
 * in what format? ruby file known as bench file.
 * store external equipment configuration data? yes
 * in what format? same bench file as above. It describes all equipment available in that setup (Test Execution Engine in Opentest, similar to Lava Dispatcher in Lava)
 * power cycle the DUT? yes
 * monitor the power usage during a run? yes, it is controlled by the test script.
 * gather a kernel trace during a run? yes
 * claim other hardware resources or machines (other than the DUT) for use during a test? yes
 * reserve a board for interactive use (ie remove it from automated testing)? no
 * provide a web-based control interface for the lab? no, web interface is only to monitor. registration is handle by TEEs automatically on power up.
 * provide a CLI control interface for the lab? yes

Run artifact handling
Does your test system:
 * store run artifacts? yes
 * in what format? logs are stored in file servers. results are in xml and in mysql.
 * put the run meta-data in a database? yes
 * if so, which database? mysql
 * parse the test logs for results? controlled by test script logic, api to set result provided to the test script.
 * convert data from test logs into a unified format? yes, xslt translation files are used to marshall into Opentest result format
 * if so, what is the format? xml
 * evaluate pass criteria for a test (e.g. ignored results, counts or thresholds)? yes, controlled by test script
 * do you have a common set of result names: (e.g. pass, fail, skip, etc.) yes
 * if so, what are they? pass, fail, skip, not-run, blocked


 * How is run data collected from the DUT? It can be done different ways, it is controlled by test script, but usually data is collected from serial connection directly.
 * e.g. by pushing from the DUT, or pulling from a server? see above
 * How is run data collected from external equipment? Base Driver classes are provided to control external equipment. The classes takes care of storing traces usually collected from connection interface, e.g. serial, telnet, etc.
 * Is external equipment data parsed? yes, it is controlled by test script

User interface
Does your test system:
 * have a visualization system? yes
 * show build artifacts to users? yes
 * show run artifacts to users? yes
 * do you have a common set of result colors? yes
 * if so, what are they? pass (green), fail (red), skip (yellow), block (blue), not-run (gray)
 * generate reports for test runs? yes
 * notify users of test results by e-mail? yes


 * can you query (aggregate and filter) the build meta-data? yes, can be done different ways, Testlink, Django and Elasticsearch
 * can you query (aggregate and filter) the run meta-data? yes, can be done different ways, Testlink, Django and Elasticsearch


 * what language or data format is used for online results presentation? (e.g. HTML, Javascript, xml, etc.) HTML
 * what language or data format is used for reports? (e.g. PDF, excel, etc.) HTML, excel, word, openoffice


 * does your test system have a CLI control tool? yes, both user and admin CLI
 * what is it called? Opentest CLI (user), staf CLI (admin)'''

Languages:
Examples: json, python, yaml, C, javascript, etc.
 * what is the base language of your test framework core? Core is Java. Test scripts vary depending on Test Execution Engine, Ruby (VATF), Python (STP), yaml/bash (Lava). In general, any test execution engine can be pluggin if it can be executed from the command line.

What languages or data formats is the user required to learn? Whatever language is used by the test execution engine he/she wants to use.

Can a user do the following with your test framework:

 * manually request that a test be executed (independent of a CI trigger)? yes
 * see the results of recent tests? yes
 * set the pass criteria for a test? yes
 * set the threshold value for a benchmark test? yes, comparison against historical mean can also be used.
 * set the list of testcase results to ignore? yes
 * provide a rating for a test? (e.g. give it 4 stars out of 5) no
 * customize a test? yes
 * alter the command line for the test program? yes
 * alter the environment of the test program? yes
 * specify to skip a testcase? yes
 * set a new expected value for a test? yes, a new version can be defined
 * edit the test program source? not from the test framework directly.
 * customize the notification criteria? no. notifications can be turn on/off but always in same format
 * customize the notification mechanism (eg. e-mail, text) no
 * generate a custom report for a set of runs? yes
 * save the report parameters to generate the same report in the future? no

Requirements
Does your test framework:
 * require minimum software on the DUT? no
 * require minimum hardware on the DUT (e.g. memory)? yes
 * If so, what? (e.g. POSIX shell or some other interpreter, specific libraries, command line tools, etc.) An interface to load software images and control the DUT is required. Usually a serial console is used for this purpose but it could be done in a different way.
 * require agent software on the DUT? (e.g. extra software besides production software) no
 * If so, what agent? no
 * is there optional agent software or libraries for the DUT? no
 * require external hardware in your labs? depends on the test case but usually power and serial are required by the test scripts. However, they are not a framework requirement.

APIS
Does your test framework:
 * use existing APIs or data formats to interact within itself, or with 3rd-party modules? yes, functionallity is exposed as STAF services. It is also possibly to use XML-RPC to trigger test and obtain test results.
 * have a published API for any of its sub-module interactions (any of the lines in the diagram)? yes
 * Please provide a link or links to the APIs? http://arago-project.org/wiki/index.php/OpentestArchitecture


 * What is the nature of the APIs you currently use?
 * async or sync RPC implemented as staf messages, see http://staf.sourceforge.net/current/STAFUG.htm for more information. XML-RPC is used for high-level tasks, see https://github.com/TestLinkOpenSourceTRMS/testlink-code/blob/testlink_1_9/lib/api/xmlrpc/v1/xmlrpc.class.php for more details.

Relationship to other software:

 * what major components does your test framework use (e.g. Jenkins, Mondo DB, Squad, Lava, etc.)
 * does your test framework interoperate with other test frameworks or software? yes
 * which ones? Jenkins, Staf, Testlink, Lava

Overview
Please list your major components here:  
 * Jenkins - Trigger build/test jobs
 * Testlink (Test Management System) - Store test definitions, test results, test reports
 * Test Master Controller - Schedule tests, manage hw resources, manage test jobs
 * Build Execution Engine - Fetch or build software assets. In practice only fecth BEEs are used.
 * Test Execution Engine - Control DUT and run test logic.

= Additional Data =
 * Architecture information and diagrams: http://arago-project.org/wiki/index.php/OpentestArchitecture

Here is a diagram of the OpenTest Ecosystem: