Difference between revisions of "Buildbot survey response"
(formating) |
|||
(7 intermediate revisions by 3 users not shown) | |||
Line 1: | Line 1: | ||
= Buildbot survey response = | = Buildbot survey response = | ||
− | Buildbot survey response provided by Alice Ferrazzi | + | Buildbot survey response provided by Alice Ferrazzi, updates Krzysztof Kozlowski |
Line 34: | Line 34: | ||
Does your test system: | Does your test system: | ||
− | * provide a set of existing tests? | + | * provide a set of existing tests? '''no''' |
** if so, how many? | ** if so, how many? | ||
Line 46: | Line 46: | ||
* require a toolchain or build system for the SUT? '''yes''' | * require a toolchain or build system for the SUT? '''yes''' | ||
* require a toolchain or build system for the test software? '''no''' | * require a toolchain or build system for the test software? '''no''' | ||
− | * come with pre-built toolchains? ''' | + | * come with pre-built toolchains? '''no''' |
− | * store the build artifacts for generated software? | + | * store the build artifacts for generated software? '''yes''' |
− | ** in what format is the build metadata stored (e.g. json)? | + | ** in what format is the build metadata stored (e.g. json)? '''internal''' |
− | ** are the build artifacts stored as raw files or in a database? | + | ** are the build artifacts stored as raw files or in a database? '''raw files''' |
*** if a database, what database? '''everything supported by sqlalchemy''' | *** if a database, what database? '''everything supported by sqlalchemy''' | ||
Line 56: | Line 56: | ||
* check that dependencies are met before a test is run? | * check that dependencies are met before a test is run? | ||
* schedule the test for the DUT? '''yes''' | * schedule the test for the DUT? '''yes''' | ||
− | ** select an appropriate individual DUT based on SUT or test attributes? | + | ** select an appropriate individual DUT based on SUT or test attributes? '''yes''' (can be configured) |
** reserve the DUT? '''yes''' | ** reserve the DUT? '''yes''' | ||
** release the DUT? '''yes''' | ** release the DUT? '''yes''' | ||
− | * install the software under test to the DUT? | + | * install the software under test to the DUT? '''yes/no''' (can be configured) |
− | * install required packages before a test is run? '''yes/ | + | * install required packages before a test is run? '''yes/no''' (can be configured) |
− | * require particular bootloader on the DUT? (e.g. grub, uboot, etc.) ''' | + | * require particular bootloader on the DUT? (e.g. grub, uboot, etc.) '''no''' (depends on the setups, e.g. whether with QEMU or, real hardware) |
− | * deploy the test program to the DUT? | + | * deploy the test program to the DUT? '''yes/no''' (can be configured) |
− | * prepare the test environment on the DUT? | + | * prepare the test environment on the DUT? '''yes/no''' (can be configured) |
− | * start a monitor (another process to collect data) on the DUT? | + | * start a monitor (another process to collect data) on the DUT? '''yes/no''' (can be configured) |
− | * start a monitor on external equipment? | + | * start a monitor on external equipment? '''yes/no''' (can be configured) |
* initiate the test on the DUT? '''yes''' | * initiate the test on the DUT? '''yes''' | ||
* clean up the test environment on the DUT? '''yes''' | * clean up the test environment on the DUT? '''yes''' | ||
Line 71: | Line 71: | ||
==== DUT control ==== | ==== DUT control ==== | ||
Does your test system: | Does your test system: | ||
− | * store board configuration data? | + | * store board configuration data? '''no''' |
** in what format? | ** in what format? | ||
− | * store external equipment configuration data? | + | * store external equipment configuration data? '''no''' |
** in what format? | ** in what format? | ||
− | * power cycle the DUT? | + | * power cycle the DUT? '''no''' |
− | * monitor the power usage during a run? | + | * monitor the power usage during a run? '''no''' |
− | * gather a kernel trace during a run? | + | * gather a kernel trace during a run? '''no''' |
− | * claim other hardware resources or machines (other than the DUT) for use during a test? | + | * claim other hardware resources or machines (other than the DUT) for use during a test? '''no''' |
− | * reserve a board for interactive use (ie remove it from automated testing)? | + | * reserve a board for interactive use (ie remove it from automated testing)? '''no''' |
− | * provide a web-based control interface for the lab? '''Buildbot web interface''' | + | * provide a web-based control interface for the lab? '''Buildbot web interface''' (limited, only stop test, pause the worker) |
* provide a CLI control interface for the lab? '''yes, Buildbot-worker''' | * provide a CLI control interface for the lab? '''yes, Buildbot-worker''' | ||
− | |||
− | |||
==== Run artifact handling ==== | ==== Run artifact handling ==== | ||
Line 96: | Line 94: | ||
* evaluate pass criteria for a test (e.g. ignored results, counts or thresholds)? '''yes''' | * evaluate pass criteria for a test (e.g. ignored results, counts or thresholds)? '''yes''' | ||
* do you have a common set of result names: (e.g. pass, fail, skip, etc.) '''yes''' | * do you have a common set of result names: (e.g. pass, fail, skip, etc.) '''yes''' | ||
− | ** if so, what are they? ''' | + | ** if so, what are they? ''' [http://docs.buildbot.net/latest/developer/results.html Buildbot test results]''' |
* How is run data collected from the DUT? | * How is run data collected from the DUT? | ||
Line 103: | Line 101: | ||
* How is run data collected from external equipment? | * How is run data collected from external equipment? | ||
− | * Is external equipment data parsed? | + | * Is external equipment data parsed? |
==== User interface ==== | ==== User interface ==== | ||
Line 122: | Line 120: | ||
* does your test system have a CLI control tool? '''yes''' | * does your test system have a CLI control tool? '''yes''' | ||
− | ** what is it called? '''buildbot | + | ** what is it called? '''buildbot''' |
==== Languages: ==== | ==== Languages: ==== | ||
Line 133: | Line 131: | ||
* see the results of recent tests? '''yes''' | * see the results of recent tests? '''yes''' | ||
* set the pass criteria for a test? '''yes''' | * set the pass criteria for a test? '''yes''' | ||
− | ** set the threshold value for a benchmark test? ''' | + | ** set the threshold value for a benchmark test? '''yes/no''' (can be configured) |
− | ** set the list of testcase results to ignore? ''' | + | ** set the list of testcase results to ignore? '''yes/no''' (can be configured) |
* provide a rating for a test? (e.g. give it 4 stars out of 5) '''no''' | * provide a rating for a test? (e.g. give it 4 stars out of 5) '''no''' | ||
* customize a test? '''yes''' | * customize a test? '''yes''' | ||
Line 153: | Line 151: | ||
** If so, what? | ** If so, what? | ||
− | * require agent software on the DUT? '''yes''' | + | * require agent software on the DUT? '''yes/no''' |
− | ** If so, what agent? '''buildbot-worker''' | + | ** If so, what agent? '''buildbot-worker or any access (UART, SSH) if buildbot-worker is not running on DUT''' |
* is there optional agent software or libraries for the DUT? '''no''' | * is there optional agent software or libraries for the DUT? '''no''' | ||
* require external hardware in your labs? '''no''' | * require external hardware in your labs? '''no''' | ||
Line 166: | Line 164: | ||
* What is the nature of the APIs you currently use? | * What is the nature of the APIs you currently use? | ||
Are they: | Are they: | ||
− | ** RPCs? | + | ** RPCs? '''No''' |
− | ** Unix-style? | + | ** Unix-style? '''No''' |
− | ** compiled libraries? | + | ** compiled libraries? '''No''' |
− | ** interpreter modules or libraries? | + | ** interpreter modules or libraries? '''yes (python API)''' |
− | ** web-based APIs? | + | ** web-based APIs? '''yes (REST API and websocket event queue system)''' |
− | ** something else? | + | ** something else? '''No''' |
==== Relationship to other software: ==== | ==== Relationship to other software: ==== | ||
− | * what major components does your test framework use? '''Buildbot''' | + | * what major components does your test framework use? '''Buildbot''', '''Python''' |
− | |||
− | |||
== Overview == | == Overview == |
Latest revision as of 06:57, 18 October 2018
Contents
- 1 Buildbot survey response
- 1.1 Survey Questions
- 1.1.1 source code access
- 1.1.2 test definitions
- 1.1.3 build management
- 1.1.4 Test scheduling/management
- 1.1.5 DUT control
- 1.1.6 Run artifact handling
- 1.1.7 User interface
- 1.1.8 Languages:
- 1.1.9 Can a user do the following with your test framework:
- 1.1.10 Requirements
- 1.1.11 APIS
- 1.1.12 Relationship to other software:
- 1.2 Overview
- 1.1 Survey Questions
- 2 Additional Data
Buildbot survey response
Buildbot survey response provided by Alice Ferrazzi, updates Krzysztof Kozlowski
Survey Questions
- What is the name of your test framework? Buildbot
Does your test framework:
source code access
- access source code repositories for the software under test? yes
- access source code repositories for the test software? yes
- include the source for the test software? yes
- provide interfaces for developers to perform code reviews? yes
- detect that the software under test has a new version? yes
- if so, how? git polling
- detect that the test software has a new version? experimental
test definitions
Does your test system:
- have a test definition repository? yes
- if so, what data format or language is used? Python/Bash
Does your test definition include:
- source code (or source code location)? yes. See https://github.com/buildbot/buildbot/
- dependency information? 'yes
- execution instructions? yes
- command line variants? yes
- environment variants? yes
- setup instructions? yes
- cleanup instructions? yes
- if anything else, please describe:
Does your test system:
- provide a set of existing tests? no
- if so, how many?
build management
Does your test system:
- build the software under test (e.g. the kernel)? yes
- build the test software? yes
- build other software (such as the distro, libraries, firmware)? yes
- support cross-compilation? 'yes
- require a toolchain or build system for the SUT? yes
- require a toolchain or build system for the test software? no
- come with pre-built toolchains? no
- store the build artifacts for generated software? yes
- in what format is the build metadata stored (e.g. json)? internal
- are the build artifacts stored as raw files or in a database? raw files
- if a database, what database? everything supported by sqlalchemy
Test scheduling/management
Does your test system:
- check that dependencies are met before a test is run?
- schedule the test for the DUT? yes
- select an appropriate individual DUT based on SUT or test attributes? yes (can be configured)
- reserve the DUT? yes
- release the DUT? yes
- install the software under test to the DUT? yes/no (can be configured)
- install required packages before a test is run? yes/no (can be configured)
- require particular bootloader on the DUT? (e.g. grub, uboot, etc.) no (depends on the setups, e.g. whether with QEMU or, real hardware)
- deploy the test program to the DUT? yes/no (can be configured)
- prepare the test environment on the DUT? yes/no (can be configured)
- start a monitor (another process to collect data) on the DUT? yes/no (can be configured)
- start a monitor on external equipment? yes/no (can be configured)
- initiate the test on the DUT? yes
- clean up the test environment on the DUT? yes
DUT control
Does your test system:
- store board configuration data? no
- in what format?
- store external equipment configuration data? no
- in what format?
- power cycle the DUT? no
- monitor the power usage during a run? no
- gather a kernel trace during a run? no
- claim other hardware resources or machines (other than the DUT) for use during a test? no
- reserve a board for interactive use (ie remove it from automated testing)? no
- provide a web-based control interface for the lab? Buildbot web interface (limited, only stop test, pause the worker)
- provide a CLI control interface for the lab? yes, Buildbot-worker
Run artifact handling
Does your test system:
- store run artifacts yes
- in what format? raw files
- put the run meta-data in a database? yes
- if so, which database? anything supported by sqlalchemy
- parse the test logs for results? yes
- convert data from test logs into a unified format? no
- if so, what is the format?
- evaluate pass criteria for a test (e.g. ignored results, counts or thresholds)? yes
- do you have a common set of result names: (e.g. pass, fail, skip, etc.) yes
- if so, what are they? Buildbot test results
- How is run data collected from the DUT?
Buildbot-worker/Buildbot master
- e.g. by pushing from the DUT, or pulling from a server?
- How is run data collected from external equipment?
- Is external equipment data parsed?
User interface
Does your test system:
- have a visualization system? yes
- show build artifacts to users? yes
- show run artifacts to users? yes
- do you have a common set of result colors? yes
- if so, what are they? Buildbot test results
- generate reports for test runs? yes
- notify users of test results by e-mail? yes
- can you query (aggregate and filter) the build meta-data? yes
- can you query (aggregate and filter) the run meta-data? yes
- what language or data format is used for online results presentation? custom html/javascript app
- what language or data format is used for reports? Depend from the report tool used
- does your test system have a CLI control tool? yes
- what is it called? buildbot
Languages:
- what is the base language of your test framework core? Python
- What languages or data formats is the user required to learn? Python
Can a user do the following with your test framework:
- Can a user manually request that a test be executed? yes
- see the results of recent tests? yes
- set the pass criteria for a test? yes
- set the threshold value for a benchmark test? yes/no (can be configured)
- set the list of testcase results to ignore? yes/no (can be configured)
- provide a rating for a test? (e.g. give it 4 stars out of 5) no
- customize a test? yes
- alter the command line for the test program? no
- alter the environment of the test program? yes
- specify to skip a testcase? yes
- set a new expected value for a test? yes
- edit the test program source? yes
- customize the notification criteria? yes
- customize the notification mechanism (eg. e-mail, text) yes
- generate a custom report for a set of runs? yes
- save the report parameters to generate the same report in the future? yes
Requirements
Does your test framework:
- require minimum software on the DUT?
- require minimum hardware on the DUT (e.g. memory)
- If so, what?
- require agent software on the DUT? yes/no
- If so, what agent? buildbot-worker or any access (UART, SSH) if buildbot-worker is not running on DUT
- is there optional agent software or libraries for the DUT? no
- require external hardware in your labs? no
APIS
Does your test framework:
- use existing APIs or data formats to interact within itself, or with 3rd-party modules? yes [1]
- have a published API for any of its sub-module interactions (any of the lines in the diagram)? yes
- Please provide a link or links to the APIs? [2]
- What is the nature of the APIs you currently use?
Are they:
- RPCs? No
- Unix-style? No
- compiled libraries? No
- interpreter modules or libraries? yes (python API)
- web-based APIs? yes (REST API and websocket event queue system)
- something else? No
Relationship to other software:
- what major components does your test framework use? Buildbot, Python
Overview
Please list your major components here:
- Build/Test Management (Buildbot)
- Test Scheduling (Buildbot)
- Results Management (Buildbot)
- View/Interact (Buildbot)