Automated Testing Summit 2018

This is a public planning page for the Automated Testing Summit

= 2018 meeting = Coordinated by: Tim Bird and Kevin Hilman

Mailing list
Discussions on this topic have started at on the "Automated Testing" mailing list of the Yocto Project
 * See https://lists.yoctoproject.org/listinfo/automated-testing

Action items from meeting
Action items:
 * Tim: refine glossary with feedback from summit
 * Tim: Create glossary wiki page (separate from survey)
 * modify CI reference diagram with discussed changes (Kevin?)
 * Tim: Finalize survey results (mostly done, see Test_Stack_Survey)
 * Start working on pdudaemon as central location for power control abstraction for DUT control driver interface (who to do this?)
 * Tim Bird: Create a test Definition survey (to collect Test Definition fields)
 * Tim Bird: Send test phases, as a way of starting discussion on Test Execution API
 * Tim Orling: create pdudaemon debian package?
 * "result format" survey needed soon
 * (done) create a page for test suite links - see Test Systems
 * page created, but not fully populated yet
 * (done) create a page for automated testing overview - see Automated Testing
 * create a reference for experimenting with a test framework: hello_test on a beaglebone
 * document describing how something works on each framework
 * Kevin: create an automated test project in the Linux Foundation
 * collect Run Artifact fields (for possible RA standard) (unassigned)
 * Tim Bird: arrange for sessions and meetings at ELCE 2019
 * Chris Fiege: create "Design for Testing" document aimed at board hardware designers

= Date and Venue =
 * Location: Edinburgh, Scotland
 * Venue: in the Edinburgh International Conference Centre
 * Date: October 25, 2018


 * co-located (following) Embedded Linux Conference Europe and Open Source Summit Europe
 * ELCE and OSSEU run Oct 22-24
 * see https://events.linuxfoundation.org/events/elc-openiot-europe-2018/
 * You can use the Linux Foundation discounts for room blocks at certain hotels in Edinburgh. See the venue-travel page for ELCE for more information.
 * Room: Ochil Suite 1-3, Level 1, EICC
 * Time: 9:00 am to 5:00 pm

Lunch will be included. Attendance is by invitation and is free-of-charge.

Sponsorship was provided by:
 * the Core Embedded Linux Project of the Linux Foundation
 * Linaro
 * DENSO TEN

= Summit Artifacts =

Minutes
See ATS 2018 Minutes

Presentation
Here are Tim's slides from the event: [[Media:ATS2018-Intro-and-Discussion-Topics.pdf | PDF]]

Video
A video link will be provided shortly.

= Attendees = This was a closed, invitation-only event.

For this event, Kevin and I decided to keep it to a small focused group. Linaro sponsored recording the presentations and discussion, and the videos will be made available after the summit. I apologize if you would like to attend but have not been invited. We are hopeful that we can put together a public automated testing event in the future.

Invitations were sent out previously, and representatives from the following projects have agreed to come to the event.
 * 0-day
 * Fuego
 * Gentoo CI system
 * Buildbot
 * Jenkins
 * KernelCI
 * kerneltests.org
 * Kselftest
 * ktest
 * LAVA
 * Labgrid
 * LKFT
 * LTP
 * Opentest
 * Phoronix Test Suite
 * ptest
 * R4D
 * SLAV
 * syzkaller/syzbot
 * tbot
 * Xilinux testing
 * Yocto project (oeqa and ptest)

= Pre-meeting work = Tim and Kevin worked on a glossary and survey, and CI loop diagram for discussion.

See Test Stack Survey for the work in progress.

Eventually, we'd like to fill out the information on: Test Stack Layers

= Schedule =

Stuff not fit into schedule yet: *

= Agenda (brainstorming) = Here is some brainstorming on an agenda...


 * board farm survey (lightning talks)
 * what are people using?
 * what works?
 * what's missing?
 * board discovery/lab introspection?
 * layers and interfaces
 * what layers are supported, needed?
 * any way to leverage/separate/isolate existing software?
 * what tests need to be supported?
 * boot-time
 * run-time
 * package-based (package unit tests)
 * driver (hardware specific?)
 * requiring specialized hardware external to board (e.g. canbus simulator, hdmi frame-grabber)
 * multinode
 * how to allocate/schedule multiple pieces of equipment for a test (e.g. 2 or more nodes for a network test)
 * results reporting
 * centralized server and API to it (kernelCI json?)
 * sharing tests
 * how to define standards
 * de-facto only? (dominant project? (cough, LAVA))
 * documents?
 * survey of existing projects, and what pieces they focus on (or don't)