Automated Testing Summit 2018

This is a public planning page for the Automated Testing Summit

= 2018 meeting = Coordinated by: Tim Bird and Kevin Hilman

Action items:
 * Finalize sessions, presentations, and discussion topics
 * Coordinate registration, signage and badging with ELCE/OSSE

= Date and Venue =
 * Location: Edinburgh, Scotland
 * Venue: in the Edinburgh International Conference Centre
 * Date: October 25, 2018


 * co-located (following) Embedded Linux Conference Europe and Open Source Summit Europe
 * ELCE and OSSEU run Oct 22-24
 * see https://events.linuxfoundation.org/events/elc-openiot-europe-2018/
 * You can use the Linux Foundation discounts for room blocks at certain hotels in Edinburgh. See the venue-travel page for ELCE for more information.


 * Room: to be announced
 * Time: 9:00 am to 5:00 pm

Lunch will be included. Attendance is by invitation and is free-of-charge.

Sponsorship is provided by the Core Embedded Linux Project of the Linux Foundation.

= Attendees =

wanted attendees

 * Pawel Wieczorek (Samsung) - SLAV
 * Kevin Hilman (BayLibre) - KernelCI, lab-in-a-box
 * Milosz Wasilewski (Linaro) - LKFT
 * Matt Hart (Linaro) - LAVA, upstream-focused testing, kernelci
 * Neil Willliams (Linaro) - LAVA
 * Jan Lübbe (Pengutronix) - Labgrid
 * Tim Bird (Sony) - Fuego, ttc
 * Andrew Murray (Witekio) - started this initiative
 * Daniel Sangorrin (Toshiba) - Fuego/CIP
 * Khiem Nguyen (Renesas) - LTSI testing
 * Guenter Roeck - (Google) - ChromeOS testing, kerneltests.org
 * Fengguang Wu - (Intel) - 0 day test service
 * Agustin Benito Bethencourt (CodeThink) - board@desk testing for CIP (presentation)
 * Shuah Khan - (Samsung) - kselftest
 * Greg KH (Linux Foundation) - LTS maintainer
 * Michael Larabel (Phoronix, phoronic-test-suites, phoromatic, openbenchmark.org, linuxbenchmark.com) - Phoronix Test Suites
 * Cyril Hrubis (SUSE) - LTP
 * more people from industry
 * (comcast? - ask Comcast folks if they are interested)

possible invitees

 * Yannick Brosseau (Facebook) -
 * Dmitry Vyokov (Google) - KASAN, syskaller testing
 * Kenji Tadano (Mitsubishi)
 * Kengo Ibe (Mitsubishi)


 * someone from yocto? (who?)
 * someone from buildroot? (Thomas Petazzoni?) (is the buildroot community interested in this?)
 * someone from U-Boot? (Tom Rini/Stephen Warren/Simon Glass)?
 * Sjoerd Simons (Collabora) - kernelci contributor
 * Guillaume Tucker (Collabora) - kernelci contributor

interested in attending
Please add your preferred co-located conference (ELCE, Plumbers, or something else), if you have one.


 * Geert Uytterhoeven - general board farm (ELCE or FOSDEM)
 * Kieran Bingham - general board farm
 * Jan-Simon Möller - general board farm, especially LAVA & KernelCI
 * Marco Cavallini (Koan) - general board farm
 * Chris Fiege (Pengutronix) - board farm hardware, special development tools (USB-SD-Mux, ...)
 * Michal Simek (Xilinx) - general board farm
 * Punnaiah Choudary Kalluri (Xilinx) - general board farm
 * Neil Willliams (Linaro) - LAVA
 * Tim Bird (Sony) - Fuego, ttc (either ELCE or Plumbers is OK)
 * Carlos Hernandez (Texas Instruments) - manages board farm at TI
 * Otavio Salvador (O.S. Systems) - (prefers ELCE)
 * Jan Lübbe (Pengutronix) - labgrid (prefers just after ELCE)
 * Milosz Wasilewski (Linaro) - LKFT (ELCE or FOSDEM are fine) - can cover LKFT if Tom can't make it
 * Muhammad Shafique (Mentor) - general board farm, LAVA & KernelCI
 * Mirza Krak (Endian Technologies AB) - (ELCE or FOSDEM)
 * Tim Orling (Yocto Project) - (ELCE, possibly Plumbers)
 * Matt Hart (Linaro) - board farm, PDUDaemon, LAVA and KernelCI
 * Rouven Czerwinski (Pengutronix) - labgrid (prefers just after ELCE)
 * Mark Brown (Linaro) - Board farm stuff, kernelci, consuming results
 * Alan Bennett (Open Source Foundries) - (prefers ELCE)
 * Tyler Baker (Open Source Foundries)
 * Manuel Traut (Linuxtronix) - r4d, jenkins, ci-rt
 * Anna-Maria Gleixner (Linuxtronix) - r4d
 * Zhenfeng Zhao (Wind River) - (either ELCE or Plumbers is fine)
 * Shuah Khan (Samsung) - kselftest (Linaro connect?)
 * Steve McIntyre (Linaro) - LAVA (any)
 * Matteo Vit (Starware Design) - general board farm, hardware (ELCE)
 * Dan Rue (Linaro) - Linaro LKFT
 * Ryan S Arnold (Linaro) - LAVA & LKFT (either ELCE, LPC, or Linaro Connect)
 * Roland Hieber (Pengutronix)
 * Eystein Stenberg (Mender.io) (either ELCE or Plumbers)
 * Nobuhiro Iwamatsu (Cybertrust Japan / CIP / Debian / U-Boot) - (prefers ELCE)
 * Alice Ferrazzi (Cybertrust Japan / Gentoo Kernel Project Leader) (either ELCE or Plumbers) kselftest contributor / Gentoo kernel CI
 * Marc Kleine-Budde (Pengutronix) - (prefers just after ELCE)
 * Tsugikazu Shibata (LTSI) - (prefers ELCE)
 * Hirotaka Motai (Mitsubishi Electric) - (prefers ELCE)
 * Li Xiaoming (Fujitsu) - (prefers ELCE)

= Pre-meeting work = See Test Stack Layers

Tim is working on a glossary and survey. Kevin has a high-level CI-Loop diagram for discussion.

See Test Stack Survey for the work in progress.

= Agenda (brainstorming) = Here is some brainstorming on an agenda...


 * board farm survey (lightning talks)
 * what are people using?
 * what works?
 * what's missing?
 * board discovery/lab introspection?
 * layers and interfaces
 * what layers are supported, needed?
 * any way to leverage/separate/isolate existing software?
 * what tests need to be supported?
 * boot-time
 * run-time
 * package-based (package unit tests)
 * driver (hardware specific?)
 * requiring specialized hardware external to board (e.g. canbus simulator, hdmi frame-grabber)
 * multinode
 * how to allocate/schedule multiple pieces of equipment for a test (e.g. 2 or more nodes for a network test)
 * results reporting
 * centralized server and API to it (kernelCI json?)
 * sharing tests
 * how to define standards
 * de-facto only? (dominant project? (cough, LAVA))
 * documents?
 * survey of existing projects, and what pieces they focus on (or don't)