Testing

This page has lists of resources for Linux testing.

= Presentations from events =

LPC 2016 testing mini-summit (Nov 2016)
The 2016 Linux Plumbers Conference had a mini-summit on Testing and Fuzzing, as well as other sessions dedicated to open source testing issues. Here are some links related to that event: collect presentation links above
 * Testing and Fuzzing agenda
 * etherpad from the session
 * presentations:
 * Testing Kernels in the Cloud - by Theodore Ts'o
 * All About the perf_fuzzer - by Vince Weaver
 * Applying mutation testing for testing RCU: Progress report - by Iftekhar Ahmed
 * Integrating and running all the kernel tests - by Yannick Brosseau
 * [[Media:Linux-Kernel-Testing-Where-are-we.pdf|Linux Kernel Testing: Where are we? (PDF)]] - by Guenter Roeck
 * Syzkaller: Future development - by Dmitry Vyukov
 * Trinity: Next directions - by Dave Jones
 * Increase Test Coverage in a Linux-based OS (Google doc) - By Guillermo Ponce and Victor Rodriguez
 * BOF notes

Japan Jamboree 58-1 (Oct 2016)

 * presentations:
 * [[Media:Introduction-to-Fuego-JJ58-1.pdf|Fuego: Introduction, Status, and Future directions (PDF)]] - by Tim Bird, Sony
 * Case Study (1) - by Kusakabe-san, Fujitsu TEN
 * [[Media:Fuego-jamboree-oct-2016.pdf|Fuego LTP Improvements (PDF)]] - by Danial Sangorrin, Toshiba]
 * [[Media:ELCE-2016-Continuous-Integration-and-Autotest-Environment-using-Fuego_Master.pdf|Continuous Integration and Autotest Environment using Fuego (PDF)]](from ELCE 2016) - by Kengo Ibe, Kotaro Hashimoto, Mitsubishi Electric
 * [[Media:The_Test_Activities_In_Renesas_Using_Fuego_Current.pdf|Current State of the Test Activities in Renesas Using Fuego (PDF)]] - by Masao Nomura, Renesas Electronics Corporation

ELC Europe 2016 (Oct 2016)

 * test frameworks:
 * [[Media:Brown.pdf|KernelCI needs you! (PDF)]] - by Mark Brown, Linaro, and Kevin Hilman, BayLibre
 * [[Media:Continuous_Integration_and_Autotest_Environment_Using_Fuego.pdf|Continuous Integration and Autotest Environment Using Fuego (PDF)]] - by Kenji Tadano & Kengo Ibe, Mitsubishi Electric


 * board farms:
 * [[Media:Herd_Your_Boards.pdf|Herd Your Baords, Become a Farmer]] by Geert Uytterhoeven, Glider bvba
 * [[Media:Automated_Testing_Laboratory_for_Embedded_Linux_Distributions.pdf|Automated Testing Laboratory for Embedded Linux Distributions (PDF)]] - by Pawel Wieczorek, Samsung R&D Institute Poland
 * [[Media:Building_a_Bards_Farm.pdf|Building a Board Farm: Continuous Integration and Remote Control (PDF)]] - by Antoine Tenart & Quentin Schulz, Free Electrons

Previous ELC and ELCE talks on testing

 * ELCE 2010 - [[Media:Embedded_software_testing.pdf|Embedded Software Testing (PDF)]] - by Arnout Vandecappelle

= Testing Standards = Start collecting info for testing standards here:
 * https://tools.ietf.org/html/rfc2119 - ietf MUST, SHALL, MAY, etc. wording standards

Things we'd like to standardize in open source testing:
 * test nomenclature (test glossary)
 * how to specify test pre-requisites
 * ex: assert_define ENV_VAR_NAME
 * ex: kernel_config
 * test API
 * host/target abstraction
 * kernel installation
 * file operations
 * console access
 * command execution
 * test retrieval, build, deployment
 * test execution:
 * ex: 'make test'
 * test log output format
 * counts
 * subtest results
 * Candidate formats:
 * https://testanything.org/
 * test package format
 * meta-data for each test
 * test results
 * baseline expected results for particular tests on particular platforms
 * what tests can be skipped etc.