Automated Testing Summit
This is a public planning page for the Automated Testing Summit
Coordinated by: Tim Bird and Kevin Hilman
Discussions on this topic have started at on the "Automated Testing" mailing list of the Yocto Project
Action items from meeting
- (done) Tim: refine glossary with feedback from summit
- (done) Tim: Create glossary wiki page (separate from survey)
- modify CI reference diagram with discussed changes (Kevin?)
- (done) Tim: Finalize survey results (mostly done, see Test_Stack_Survey#Responses)
- Start working on pdudaemon as central location for power control abstraction for DUT control driver interface (who to do this?)
- Tim Bird: Create a test Definition survey (to collect Test Definition fields)
- Tim Bird: Send test phases, as a way of starting discussion on Test Execution API
- Tim Orling: create pdudaemon debian package?
- "result format" survey needed soon
- (done) create a page for test suite links - see Test Systems
- page created, but not fully populated yet
- (done) create a page for automated testing overview - see Automated Testing
- create a reference for experimenting with a test framework: hello_test on a beaglebone
- document describing how something works on each framework
- Kevin: create an automated test project in the Linux Foundation
- collect Run Artifact fields (for possible RA standard) (unassigned)
- Tim Bird: arrange for sessions and meetings at ELCE 2019
- started: I have some information from Angela at the Linux Foundation about the cost of a co-located side meeting
- Chris Fiege: create "Design for Testing" document aimed at board hardware designers
Date and Venue
- Location: Edinburgh, Scotland
- Venue: in the Edinburgh International Conference Centre
- Date: October 25, 2018
- co-located (following) Embedded Linux Conference Europe and Open Source Summit Europe
- ELCE and OSSEU run Oct 22-24
- see https://events.linuxfoundation.org/events/elc-openiot-europe-2018/
- You can use the Linux Foundation discounts for room blocks at certain hotels in Edinburgh. See the venue-travel page for ELCE for more information.
- Room: Ochil Suite 1-3, Level 1, EICC
- Time: 9:00 am to 5:00 pm
Lunch will be included. Attendance is by invitation and is free-of-charge.
Sponsorship was provided by:
See ATS 2018 Minutes
Here are Tim's slides from the event: PDF
Videos are available at: https://www.linaro.org/events/attended/ats-2018/
Also, Linaro has them on their YouTube channel at: https://www.youtube.com/playlist?list=PLKZSArYQptsPGftWXdlTI7vPCc42FXksD
Many thanks to Linaro as the sponsor for recording and processing the video for the event!
Here is an article describing the report Kevin Hillman made of the summit. He made this report a few weeks later, at Plumbers Conference 2019.
The article was written by Jake Edge.
This was a closed, invitation-only event.
For this event, Kevin and I decided to keep it to a small focused group. Linaro sponsored recording the presentations and discussion, and the videos will be made available after the summit. I apologize if you would like to attend but have not been invited. We are hopeful that we can put together a public automated testing event in the future.
Invitations were sent out previously, and representatives from the following projects have agreed to come to the event.
- Gentoo CI system
- Phoronix Test Suite
- Xilinux testing
- Yocto project (oeqa and ptest)
Tim and Kevin worked on a glossary and survey, and CI loop diagram for discussion.
See Test Stack Survey for the work in progress.
Eventually, we'd like to fill out the information on: Test Stack Layers
|Time||Topic||Presenter or discussion leader||Slides|
|9:00-9:10||Welcome and Introduction||Tim and Kevin||no slides|
|9:10-9:40||Vision and problem definition||Tim||?|
|9:40-10:40||Glossary and Diagram discussion||Tim and Kevin||.|
|11:00-12:30||test definition, build artifacts, execution API (E)||.||.|
|1:00-2:00||Brainstorming embedded Linux Requirements with CELP members||Tim (discussion leader)||.|
|2:00-2:50||run artifacts, standardized results format, parsers||.||.|
|3:10-4:30||Farm standards - DUT control drivers, board definitions, discoverability, protocols||.||.|
|4:30-5:00||wrap-up||Tim and Kevin||.|
Stuff not fit into schedule yet:
Here is some brainstorming on an agenda...
- board farm survey (lightning talks)
- what are people using?
- what works?
- what's missing?
- board discovery/lab introspection?
- layers and interfaces
- what layers are supported, needed?
- any way to leverage/separate/isolate existing software?
- what tests need to be supported?
- package-based (package unit tests)
- driver (hardware specific?)
- requiring specialized hardware external to board (e.g. canbus simulator, hdmi frame-grabber)
- how to allocate/schedule multiple pieces of equipment for a test (e.g. 2 or more nodes for a network test)
- results reporting
- centralized server and API to it (kernelCI json?)
- sharing tests
- how to define standards
- de-facto only? (dominant project? (cough, LAVA))
- survey of existing projects, and what pieces they focus on (or don't)