Syzbot survey response

= syzbot survey response = syzbox survey response provided by Dmitry Vyukov

syzbot is probably somewhat different from most other test systems (automatically generates test cases, only monitors kernel crashes, currently uses only VMs), so I found some questions hard to answer, but I tried to do my best. Hope you will find this new perspective useful too.

Survey Questions

 * What is the name of your test framework?

syzbot

Which of the aspects below of the CI loop does your test framework perform?

Does your test framework:

source code access

 * access source code repositories for the software under test? yes
 * access source code repositories for the test software? yes
 * include the source for the test software? yes
 * provide interfaces for developers to perform code reviews? no
 * detect that the software under test has a new version? yes
 * if so, how? polling git trees every N hours
 * detect that the test software has a new version? yes, polling git tree every N hours

test definitions
Does your test system:
 * have a test definition repository? yes
 * if so, what data format or language is used own declarative format for kernel interfaces

Does your test definition include: syscall/socket proto is not implemented, required /dev file is not present'''
 * source code (or source code location)? no
 * dependency information? '''no, but most is inferred, e.g.
 * execution instructions? no, all tests are equivalent
 * command line variants? no
 * environment variants? no
 * setup instructions? no
 * cleanup instructions? no
 * if anything else, please describe: tests are generated randomly from kernel interface descriptions

Does your test system:
 * provide a set of existing tests? interesting tests are automatically detected and persisted across runs
 * if so, how many? 37000

build management
Does your test system:
 * build the software under test (e.g. the kernel)? yes
 * build the test software? yes
 * build other software (such as the distro, libraries, firmware)? no, uses prepackaged images/compilers/etc
 * support cross-compilation? yes
 * require a toolchain or build system for the SUT? yes
 * require a toolchain or build system for the test software? yes
 * come with pre-built toolchains? no
 * store the build artifacts for generated software? no
 * in what format is the build metadata stored (e.g. json)? database table
 * are the build artifacts stored as raw files or in a database? build artifacts are not stored, metadata in database
 * if a database, what database? Google Cloud Datastore

Test scheduling/management
Does your test system:
 * check that dependencies are met before a test is run? mostly yes, but it does not really matter as we only care about crashing kernel
 * schedule the test for the DUT? yes, but DUTs are cloud VMs
 * select an appropriate individual DUT based on SUT or test attributes? no
 * reserve the DUT? yes, but these are just cloud VMs
 * release the DUT? yes, but these are just cloud VMs
 * install the software under test to the DUT? yes
 * install required packages before a test is run? no, no dependencies
 * require particular bootloader on the DUT? (e.g. grub, uboot, etc.) no
 * deploy the test program to the DUT? yes
 * prepare the test environment on the DUT? yes
 * start a monitor (another process to collect data) on the DUT? no
 * start a monitor on external equipment? yes, using console output
 * initiate the test on the DUT? yes
 * clean up the test environment on the DUT? yes

DUT control
Does your test system:
 * store board configuration data? no
 * in what format?
 * store external equipment configuration data? no
 * in what format?
 * power cycle the DUT? yes
 * monitor the power usage during a run? no
 * gather a kernel trace during a run? no
 * claim other hardware resources or machines (other than the DUT) for use during a test? no
 * reserve a board for interactive use (ie remove it from automated testing)? no
 * provide a web-based control interface for the lab? no
 * provide a CLI control interface for the lab? no

Run artifact handling
Does your test system:
 * store run artifacts no
 * in what format?
 * put the run meta-data in a database? no
 * if so, which database?
 * parse the test logs for results? yes, console output for crashes
 * convert data from test logs into a unified format? yes
 * if so, what is the format? database: crash title, text, console output, maintainer emails
 * evaluate pass criteria for a test (e.g. ignored results, counts or thresholds)? no
 * do you have a common set of result names: (e.g. pass, fail, skip, etc.) no
 * if so, what are they?


 * How is run data collected from the DUT? capturing of console output + test driver output via ssh
 * e.g. by pushing from the DUT, or pulling from a server?
 * How is run data collected from external equipment? no
 * Is external equipment data parsed? no

User interface
Does your test system:
 * have a visualization system? yes
 * show build artifacts to users? no
 * show run artifacts to users? yes
 * do you have a common set of result colors? no
 * if so, what are they?
 * generate reports for test runs? yes
 * notify users of test results by e-mail? yes, +integration with bug tracking systems


 * can you query (aggregate and filter) the build meta-data? no
 * can you query (aggregate and filter) the run meta-data? no


 * what language or data format is used for online results presentation? HTML
 * what language or data format is used for reports? text/plain


 * does your test system have a CLI control tool? no, but it has email-based controls
 * what is it called? custom

Languages:

 * what is the base language of your test framework core? Go

What languages or data formats is the user required to learn? in-house declarative language (as opposed to those used internally)

Can a user do the following with your test framework:
in general syzbot can be integrated with any bug tracking system'''
 * manually request that a test be executed (independent of a CI trigger)? yes, we support patch testing for reported bugs
 * see the results of recent tests? not applicable
 * set the pass criteria for a test? no
 * set the threshold value for a benchmark test? not applicable
 * set the list of testcase results to ignore? no
 * provide a rating for a test? (e.g. give it 4 stars out of 5) no
 * customize a test? no
 * alter the command line for the test program? not applicable
 * alter the environment of the test program? not applicable
 * specify to skip a testcase? no
 * set a new expected value for a test? no
 * edit the test program source? yes
 * customize the notification criteria? no
 * customize the notification mechanism (eg. e-mail, text) '''maybe,
 * generate a custom report for a set of runs? not applicable
 * save the report parameters to generate the same report in the future? not applicable

Requirements
Does your test framework: libraries, command line tools, etc.) sshd production software) yes, but it's copied automatically
 * require minimum software on the DUT? yes
 * require minimum hardware on the DUT (e.g. memory) no
 * If so, what? (e.g. POSIX shell or some other interpreter, specific
 * require agent software on the DUT? (e.g. extra software besides
 * If so, what agent? custom agent: generates tests, setups test env, talks to master machine
 * is there optional agent software or libraries for the DUT? no
 * require external hardware in your labs? no

APIS
Does your test framework: the lines in the diagram)? yes
 * use existing APIs or data formats to interact within itself, or with 3rd-party modules? yes
 * have a published API for any of its sub-module interactions (any of
 * Please provide a link or links to the APIs?

syzbot extensively uses Google Cloud APIs, in particular: GCE machine management Appengine Datastore APIs Cloud Storage

Within the test farm syzbot uses custom RPC APIs to exchange test programs and metadata: https://github.com_google/syzkaller/blob/master/pkg/rpctype/rpctype.go

Web/API application uses HTTPS/JSON APIs to upload crash info, build metadata, request patch testing, talk to bug tracking systems: https://github.com/google/syzkaller/blob/master/dashboard/dashapi/dashapi.go

Sorry - this is kind of open-ended... Are they:
 * What is the nature of the APIs you currently use?
 * RPCs?
 * Unix-style? (command line invocation, while grabbing sub-tool output)
 * compiled libraries?
 * interpreter modules or libraries?
 * web-based APIs?
 * something else?

Binary RPCs over TCP and JSON RPCs over HTTPS

Relationship to other software:

 * what major components does your test framework use (e.g. Jenkins, Mondo DB, Squad, Lava, etc.) Google Cloud Platform
 * does your test framework interoperate with other test frameworks or software? no
 * which ones?

Overview
Please list the major components of your test system.

Please list your major components here:


 * Dashboard: AppEngine application, renders web UI, sends/receives emails, talks to bug tracking systems, manages persistent database, serves APIs from other parts of the system (uploading of crashes, build metadata, etc).
 * syz-ci: continuous SUT (kernel) and test software (syzkaller) poll, build, update; uploads build metadata to dashboard; starts syz-manager instances; there are several syz-ci instances.
 * syz-manager: manages test machines; starts syz-fuzzer on test machines; monitors console output of test machines; uploads crash info to dashboard; talks to syz-hub.
 * syz-hub: allows several syz-manager's to exchange interesting test cases.
 * syz-fuzzer: runs on the test machine; generates and executes test cases; sends new interesting test cases to syz-manager.
 * syz-executor: executes/interpretes test cases (the only part of the system in C++).

= Additional Data =
 * video: Syzbot and the Tale of a Thousand Kernel Bugs - Linux Security Summit 2018
 * slides: PDF