Syzbot survey response

= syzbot survey response = syzbox survey response provided by Dmitry Vyukov

syzbot is probably somewhat different from most other test systems (automatically generates test cases, only monitors kernel crashes, currently uses only VMs), so I found some questions hard to answer, but I tried to do my best. Hope you will find this new perspective useful too.

Survey Questions

 * What is the name of your test framework?

syzbot

Which of the aspects below of the CI loop does your test framework perform?

Does your test framework:

source code access

 * access source code repositories for the software under test? yes
 * access source code repositories for the test software? yes
 * include the source for the test software? yes
 * provide interfaces for developers to perform code reviews? no
 * detect that the software under test has a new version? yes
 * if so, how? polling git trees every N hours
 * detect that the test software has a new version? yes, polling git tree every N hours

test definitions
Does your test system:
 * have a test definition repository? yes
 * if so, what data format or language is used own declarative format for kernel interfaces

Does your test definition include: syscall/socket proto is not implemented, required /dev file is not present'''
 * source code (or source code location)? no
 * dependency information? '''no, but most is inferred, e.g.
 * execution instructions? no, all tests are equivalent
 * command line variants? no
 * environment variants? no
 * setup instructions? no
 * cleanup instructions? no
 * if anything else, please describe: tests are generated randomly from kernel interface descriptions

Does your test system: That's a lot of tests, but they are not tests in classical meaning, because we only see if kernel crashes on them or not. But we know that they all are unique in some sense.
 * provide a set of existing tests? interesting tests are automatically detected and persisted across runs
 * if so, how many? 37000

I suspect this test base gives the highest kernel coverage out there. It allows to cover 1M+ LOC in tens of minutes. We don't mind sharing it.

build management
Does your test system:
 * build the software under test (e.g. the kernel)? yes
 * build the test software? yes
 * build other software (such as the distro, libraries, firmware)? no, uses prepackaged images/compilers/etc
 * support cross-compilation? yes
 * require a toolchain or build system for the SUT? yes
 * require a toolchain or build system for the test software? yes
 * come with pre-built toolchains? no
 * store the build artifacts for generated software? no
 * in what format is the build metadata stored (e.g. json)? database table
 * are the build artifacts stored as raw files or in a database? build artifacts are not stored, metadata in database
 * if a database, what database? Google Cloud Datastore

Test scheduling/management
Does your test system:
 * check that dependencies are met before a test is run? mostly yes, but it does not really matter as we only care about crashing kernel
 * schedule the test for the DUT? yes, but DUTs are cloud VMs
 * select an appropriate individual DUT based on SUT or test attributes? no
 * reserve the DUT? yes, but these are just cloud VMs
 * release the DUT? yes, but these are just cloud VMs
 * install the software under test to the DUT? yes
 * install required packages before a test is run? no, no dependencies
 * require particular bootloader on the DUT? (e.g. grub, uboot, etc.) no
 * deploy the test program to the DUT? yes
 * prepare the test environment on the DUT? yes
 * start a monitor (another process to collect data) on the DUT? no
 * start a monitor on external equipment? yes, using console output
 * initiate the test on the DUT? yes
 * clean up the test environment on the DUT? yes

DUT control
Does your test system:
 * store board configuration data? no
 * in what format?
 * store external equipment configuration data? no
 * in what format?
 * power cycle the DUT? yes
 * monitor the power usage during a run? no
 * gather a kernel trace during a run? no
 * claim other hardware resources or machines (other than the DUT) for use during a test? no
 * reserve a board for interactive use (ie remove it from automated testing)? no
 * provide a web-based control interface for the lab? no
 * provide a CLI control interface for the lab? no

Run artifact handling
Does your test system:
 * store run artifacts no
 * in what format?
 * put the run meta-data in a database? no
 * if so, which database?
 * parse the test logs for results? yes, console output for crashes
 * convert data from test logs into a unified format? yes
 * if so, what is the format? database: crash title, text, console output, maintainer emails
 * evaluate pass criteria for a test (e.g. ignored results, counts or thresholds)? no
 * do you have a common set of result names: (e.g. pass, fail, skip, etc.) no
 * if so, what are they?


 * How is run data collected from the DUT? capturing of console output + test driver output via ssh
 * e.g. by pushing from the DUT, or pulling from a server?
 * How is run data collected from external equipment? no
 * Is external equipment data parsed? no

User interface
Does your test system:
 * have a visualization system? yes
 * show build artifacts to users? no
 * show run artifacts to users? yes
 * do you have a common set of result colors? no
 * if so, what are they?
 * generate reports for test runs? yes
 * notify users of test results by e-mail? yes, +integration with bug tracking systems


 * can you query (aggregate and filter) the build meta-data? no
 * can you query (aggregate and filter) the run meta-data? no


 * what language or data format is used for online results presentation? HTML
 * what language or data format is used for reports? text/plain


 * does your test system have a CLI control tool? no, but it has email-based controls
 * what is it called? custom

Languages:

 * what is the base language of your test framework core? Go

What languages or data formats is the user required to learn? in-house declarative language (as opposed to those used internally)

Can a user do the following with your test framework:
in general syzbot can be integrated with any bug tracking system'''
 * manually request that a test be executed (independent of a CI trigger)? yes, we support patch testing for reported bugs
 * see the results of recent tests? not applicable
 * set the pass criteria for a test? no
 * set the threshold value for a benchmark test? not applicable
 * set the list of testcase results to ignore? no
 * provide a rating for a test? (e.g. give it 4 stars out of 5) no
 * customize a test? no
 * alter the command line for the test program? not applicable
 * alter the environment of the test program? not applicable
 * specify to skip a testcase? no
 * set a new expected value for a test? no
 * edit the test program source? yes
 * customize the notification criteria? no
 * customize the notification mechanism (eg. e-mail, text) '''maybe,
 * generate a custom report for a set of runs? not applicable
 * save the report parameters to generate the same report in the future? not applicable

Requirements
Does your test framework: libraries, command line tools, etc.) sshd production software) yes, but it's copied automatically
 * require minimum software on the DUT? yes
 * require minimum hardware on the DUT (e.g. memory) no
 * If so, what? (e.g. POSIX shell or some other interpreter, specific
 * require agent software on the DUT? (e.g. extra software besides
 * If so, what agent? custom agent: generates tests, setups test env, talks to master machine
 * is there optional agent software or libraries for the DUT? no
 * require external hardware in your labs? no

APIS
Does your test framework: the lines in the diagram)? yes
 * use existing APIs or data formats to interact within itself, or with 3rd-party modules? yes
 * have a published API for any of its sub-module interactions (any of
 * Please provide a link or links to the APIs?

syzbot extensively uses Google Cloud APIs, in particular: GCE machine management Appengine Datastore APIs Cloud Storage

Within the test farm syzbot uses custom RPC APIs to exchange test programs and metadata: https://github.com_google/syzkaller/blob/master/pkg/rpctype/rpctype.go

Web/API application uses HTTPS/JSON APIs to upload crash info, build metadata, request patch testing, talk to bug tracking systems: https://github.com/google/syzkaller/blob/master/dashboard/dashapi/dashapi.go

Sorry - this is kind of open-ended... Are they:
 * What is the nature of the APIs you currently use?
 * RPCs?
 * Unix-style? (command line invocation, while grabbing sub-tool output)
 * compiled libraries?
 * interpreter modules or libraries?
 * web-based APIs?
 * something else?

Binary RPCs over TCP and JSON RPCs over HTTPS

Relationship to other software:

 * what major components does your test framework use (e.g. Jenkins, Mondo DB, Squad, Lava, etc.) Google Cloud Platform
 * does your test framework interoperate with other test frameworks or software? no
 * which ones?

Overview
Please list the major components of your test system.

Please list your major components here:


 * Dashboard: AppEngine application, renders web UI, sends/receives emails, talks to bug tracking systems, manages persistent database, serves APIs from other parts of the system (uploading of crashes, build metadata, etc).
 * syz-ci: continuous SUT (kernel) and test software (syzkaller) poll, build, update; uploads build metadata to dashboard; starts syz-manager instances; there are several syz-ci instances.
 * syz-manager: manages test machines; starts syz-fuzzer on test machines; monitors console output of test machines; uploads crash info to dashboard; talks to syz-hub.
 * syz-hub: allows several syz-manager's to exchange interesting test cases.
 * syz-fuzzer: runs on the test machine; generates and executes test cases; sends new interesting test cases to syz-manager.
 * syz-executor: executes/interpretes test cases (the only part of the system in C++).

= Additional Data =
 * video: Syzbot and the Tale of a Thousand Kernel Bugs - Linux Security Summit 2018
 * slides: PDF