Difference between revisions of "Automated Testing Summit 2018"

From eLinux.org
Jump to: navigation, search
(Action items from meeting)
 
(42 intermediate revisions by 2 users not shown)
Line 4: Line 4:
 
Coordinated by: Tim Bird and Kevin Hilman
 
Coordinated by: Tim Bird and Kevin Hilman
  
 +
== Mailing list ==
 +
Discussions on this topic have started at on the "Automated Testing" mailing list of the Yocto Project
 +
* See https://lists.yoctoproject.org/listinfo/automated-testing
 +
 +
== Action items from meeting ==
 
Action items:
 
Action items:
* Finalize sessions, presentations, and discussion topics
+
* (done) Tim: refine glossary with feedback from summit
* Coordinate registration, signage and badging with ELCE/OSSE
+
** (done) Tim: Create glossary wiki page (separate from survey)
 +
* modify CI reference diagram with discussed changes (Kevin?)
 +
* (done) Tim: Finalize survey results (mostly done, see [[Test_Stack_Survey#Responses]])
 +
* Start working on pdudaemon as central location for power control abstraction for DUT control driver interface (who to do this?)
 +
* (done) Tim Bird: Create a test Definition survey (to collect Test Definition fields)
 +
* Tim Bird: Send test phases, as a way of starting discussion on Test Execution API
 +
* Tim Orling: create pdudaemon debian package?
 +
* "result format" survey needed soon
 +
* (done) create a page for test suite links - see [[Test Systems]]
 +
** page created, but not fully populated yet
 +
* (done) create a page for automated testing overview - see [[Automated Testing]]
 +
* create a reference for experimenting with a test framework: hello_test on a beaglebone
 +
** document describing how something works on each framework
 +
* Kevin: create an automated test project in the Linux Foundation
 +
* collect Run Artifact fields (for possible RA standard) (unassigned)
 +
* (started) Tim Bird: arrange for sessions and meetings at ELCE 2019
 +
** I have some information from Angela at the Linux Foundation about the cost of a co-located side meeting
 +
** see [[Automated Testing Summit 2019]]
 +
* Chris Fiege: create "Design for Testing" document aimed at board hardware designers
  
 
= Date and Venue  =
 
= Date and Venue  =
Line 17: Line 40:
 
** see https://events.linuxfoundation.org/events/elc-openiot-europe-2018/
 
** see https://events.linuxfoundation.org/events/elc-openiot-europe-2018/
 
** You can use the Linux Foundation discounts for room blocks at certain hotels in Edinburgh.  See [https://events.linuxfoundation.org/events/elc-openiot-europe-2018/attend/venue-travel/ the venue-travel page] for ELCE for more information.
 
** You can use the Linux Foundation discounts for room blocks at certain hotels in Edinburgh.  See [https://events.linuxfoundation.org/events/elc-openiot-europe-2018/attend/venue-travel/ the venue-travel page] for ELCE for more information.
 
+
* Room: Ochil Suite 1-3, Level 1, EICC
* Room: ''to be announced''
 
 
* Time: 9:00 am to 5:00 pm
 
* Time: 9:00 am to 5:00 pm
  
 
Lunch will be included.  Attendance is by invitation and is free-of-charge.
 
Lunch will be included.  Attendance is by invitation and is free-of-charge.
  
Sponsorship is provided by the
+
Sponsorship was provided by:
[https://wiki.linuxfoundation.org/celp/start Core Embedded Linux Project]
+
* the [https://wiki.linuxfoundation.org/celp/start Core Embedded Linux Project] of the Linux Foundation
of the Linux Foundation.
+
* [https://linaro.org Linaro]
 +
* [http://www.denso-ten.com/ DENSO TEN]
 +
 
 +
= Summit Artifacts =
 +
== Minutes ==
 +
See [[ATS 2018 Minutes]]
 +
 
 +
== Presentation ==
 +
Here are Tim's slides from the event: [[Media:ATS2018-Intro-and-Discussion-Topics.pdf | PDF]]
 +
 
 +
== Video ==
 +
Videos are available at: https://www.linaro.org/events/attended/ats-2018/
 +
 
 +
Also, Linaro has them on their YouTube channel at: https://www.youtube.com/playlist?list=PLKZSArYQptsPGftWXdlTI7vPCc42FXksD
 +
 
 +
Many thanks to Linaro as the sponsor for recording and processing the video for the event!
 +
 
 +
== LWN.net report ==
 +
Here is an article describing the report Kevin Hillman made of the summit.  He made
 +
this report a few weeks later, at Plumbers Conference 2019.
 +
 
 +
The article was written by Jake Edge.
 +
 
 +
See https://lwn.net/Articles/771782/
  
 
= Attendees =
 
= Attendees =
 +
This was a closed, invitation-only event.
  
== wanted attendees ==
+
For this event, Kevin and I decided to keep it to a small focused group.
* Pawel Wieczorek (Samsung) - SLAV
+
Linaro sponsored recording the presentations and discussion, and the videos
* Kevin Hilman (BayLibre) - KernelCI, lab-in-a-box
+
will be made available after the summit. I apologize if you would like to
* Milosz Wasilewski (Linaro) - LKFT
+
attend but have not been invited. We are hopeful that we can put
* Matt Hart (Linaro) - LAVA, upstream-focused testing, kernelci
+
together a public automated testing event in the future.
* [[User:Codehelp|Neil Willliams]] (Linaro) - LAVA
 
* Jan Lübbe (Pengutronix) - Labgrid
 
* Tim Bird (Sony) - Fuego, ttc
 
* Andrew Murray (Witekio) - started this initiative
 
* Daniel Sangorrin (Toshiba) - Fuego/CIP
 
* [[User:Khiem.renesas|Khiem Nguyen]] (Renesas) - LTSI testing
 
* Guenter Roeck - (Google) - ChromeOS testing, kerneltests.org
 
* Fengguang Wu - (Intel) - 0 day test service
 
* Agustin Benito Bethencourt (CodeThink) - board@desk testing for CIP ([https://wiki.linuxfoundation.org/_media/civilinfrastructureplatform/testing_at_cip__b_d.pdf presentation])
 
* Shuah Khan - (Samsung) - kselftest
 
* Greg KH (Linux Foundation) - LTS maintainer
 
* Michael Larabel (Phoronix, phoronic-test-suites, phoromatic, openbenchmark.org, linuxbenchmark.com) - Phoronix Test Suites
 
* Cyril Hrubis (SUSE) - LTP
 
* more people from industry
 
** (comcast? - ask Comcast folks if they are interested)
 
  
== possible invitees ==
+
Invitations were sent out previously, and representatives from the following projects have agreed to come to the event.
* Yannick Brosseau (Facebook) -
+
* 0-day
* Dmitry Vyokov (Google) - KASAN, syskaller testing
+
* Fuego
* Kenji Tadano (Mitsubishi)
+
* Gentoo CI system
* Kengo Ibe (Mitsubishi)
+
* Buildbot
 +
* Jenkins
 +
* KernelCI
 +
* kerneltests.org
 +
* Kselftest
 +
* ktest
 +
* LAVA
 +
* Labgrid
 +
* LKFT
 +
* LTP
 +
* Opentest
 +
* Phoronix Test Suite
 +
* ptest
 +
* R4D
 +
* SLAV
 +
* syzkaller/syzbot
 +
* tbot
 +
* Xilinux testing
 +
* Yocto project (oeqa and ptest)
  
* someone from yocto? (who?)
+
= Pre-meeting work =
* someone from buildroot? (Thomas Petazzoni?)  (is the buildroot community interested in this?)
+
Tim and Kevin worked on a glossary and survey, and CI loop diagram for discussion.
* someone from U-Boot? (Tom Rini/Stephen Warren/Simon Glass)?
 
* Sjoerd Simons (Collabora) - kernelci contributor
 
* Guillaume Tucker (Collabora) - kernelci contributor
 
  
== interested in attending ==
+
See [[Test Stack Survey]] for the work in progress.
Please add your preferred co-located conference (ELCE, Plumbers, or something else), if you have one.
 
  
* Geert Uytterhoeven - general board farm (ELCE or FOSDEM)
+
Eventually, we'd like to fill out the information on: [[Test Stack Layers]]
* Kieran Bingham - general board farm
 
* Jan-Simon Möller - general board farm, especially LAVA & KernelCI
 
* Marco Cavallini (Koan) - general board farm
 
* Chris Fiege (Pengutronix) - board farm hardware, special development tools (USB-SD-Mux, ...)
 
* Michal Simek (Xilinx) - general board farm
 
* Punnaiah Choudary Kalluri (Xilinx) - general board farm
 
* [[User:Codehelp|Neil Willliams]] (Linaro) - LAVA
 
* Tim Bird (Sony) - Fuego, ttc (either ELCE or Plumbers is OK)
 
* Carlos Hernandez (Texas Instruments) - manages board farm at TI
 
* Otavio Salvador (O.S. Systems) - (prefers ELCE)
 
* Jan Lübbe (Pengutronix) - labgrid (prefers just after ELCE)
 
* Milosz Wasilewski (Linaro) - LKFT (ELCE or FOSDEM are fine) - can cover LKFT if Tom can't make it
 
* Muhammad Shafique (Mentor) - general board farm, LAVA & KernelCI
 
* Mirza Krak (Endian Technologies AB) - (ELCE or FOSDEM)
 
* Tim Orling (Yocto Project) - (ELCE, possibly Plumbers)
 
* Matt Hart (Linaro) - board farm, PDUDaemon, LAVA and KernelCI
 
* Rouven Czerwinski (Pengutronix) - labgrid (prefers just after ELCE)
 
* Mark Brown (Linaro) - Board farm stuff, kernelci, consuming results
 
* Alan Bennett (Open Source Foundries) - (prefers ELCE)
 
* Tyler Baker (Open Source Foundries)
 
* Manuel Traut (Linuxtronix) - r4d, jenkins, ci-rt
 
* Anna-Maria Gleixner (Linuxtronix) - r4d
 
* Zhenfeng Zhao (Wind River) - (either ELCE or Plumbers is fine)
 
* Shuah Khan (Samsung) - kselftest (Linaro connect?)
 
* Steve McIntyre (Linaro) - LAVA (any)
 
* Matteo Vit (Starware Design) - general board farm, hardware (ELCE)
 
* [[User:drue|Dan Rue]] (Linaro) - Linaro LKFT
 
* Ryan S Arnold (Linaro) - LAVA & LKFT (either ELCE, LPC, or Linaro Connect)
 
* Roland Hieber (Pengutronix)
 
* Eystein Stenberg (Mender.io) (either ELCE or Plumbers)
 
* Nobuhiro Iwamatsu (Cybertrust Japan / CIP / Debian / U-Boot) - (prefers ELCE)
 
* Alice Ferrazzi (Cybertrust Japan / Gentoo Kernel Project Leader) (either ELCE or Plumbers) kselftest contributor / Gentoo kernel CI
 
* Marc Kleine-Budde (Pengutronix) - (prefers just after ELCE)
 
* Tsugikazu Shibata (LTSI) - (prefers ELCE)
 
* Hirotaka Motai (Mitsubishi Electric) - (prefers ELCE)
 
* Li Xiaoming (Fujitsu) - (prefers ELCE)
 
  
= Pre-meeting work =
+
= Schedule =
See [[Test Stack Layers]]
 
  
Tim is working on a glossary and survey. Kevin has a high-level CI-Loop diagram for discussion.
+
{| class="wikitable"
 +
|+ ATS 2018 Schedule
 +
|-
 +
! Time
 +
! Topic
 +
! Presenter or discussion leader
 +
! Slides
 +
|-
 +
|9:00-9:10||Welcome and Introduction||Tim and Kevin||''no slides''
 +
|-
 +
|9:10-9:40||Vision and problem definition||Tim||?
 +
|-
 +
|9:40-10:40||Glossary and Diagram discussion||Tim and Kevin||.
 +
|- style="background:lightblue"
 +
| 10:40-11:00 || '''BREAK''' || ''n/a'' ||.
 +
|-
 +
|11:00-12:30||test definition, build artifacts, execution API (E)||.||.
 +
|- style="background:lightblue"
 +
|12:30-2:00||'''LUNCH BREAK'''||''n/a''||.
 +
|- style="background:lightblue"
 +
|1:00-2:00||''Brainstorming embedded Linux Requirements with CELP members''||Tim (discussion leader)||.
 +
|-
 +
|2:00-2:50||run artifacts, standardized results format, parsers||.||.
 +
|- style="background:lightblue"
 +
|2:50-3:10||'''BREAK'''||''n/a''||.
 +
|-
 +
|3:10-4:30||Farm standards - DUT control drivers, board definitions, discoverability, protocols||.||.
 +
|-
 +
|4:30-5:00||wrap-up||Tim and Kevin||.
 +
|}
  
See [[Test Stack Survey]] for the work in progress.
+
Stuff not fit into schedule yet:
 +
*
  
 
= Agenda (brainstorming) =
 
= Agenda (brainstorming) =

Latest revision as of 14:17, 26 April 2019

This is a public planning page for the Automated Testing Summit

2018 meeting

Coordinated by: Tim Bird and Kevin Hilman

Mailing list

Discussions on this topic have started at on the "Automated Testing" mailing list of the Yocto Project

Action items from meeting

Action items:

  • (done) Tim: refine glossary with feedback from summit
    • (done) Tim: Create glossary wiki page (separate from survey)
  • modify CI reference diagram with discussed changes (Kevin?)
  • (done) Tim: Finalize survey results (mostly done, see Test_Stack_Survey#Responses)
  • Start working on pdudaemon as central location for power control abstraction for DUT control driver interface (who to do this?)
  • (done) Tim Bird: Create a test Definition survey (to collect Test Definition fields)
  • Tim Bird: Send test phases, as a way of starting discussion on Test Execution API
  • Tim Orling: create pdudaemon debian package?
  • "result format" survey needed soon
  • (done) create a page for test suite links - see Test Systems
    • page created, but not fully populated yet
  • (done) create a page for automated testing overview - see Automated Testing
  • create a reference for experimenting with a test framework: hello_test on a beaglebone
    • document describing how something works on each framework
  • Kevin: create an automated test project in the Linux Foundation
  • collect Run Artifact fields (for possible RA standard) (unassigned)
  • (started) Tim Bird: arrange for sessions and meetings at ELCE 2019
  • Chris Fiege: create "Design for Testing" document aimed at board hardware designers

Date and Venue

  • Location: Edinburgh, Scotland
  • Venue: in the Edinburgh International Conference Centre
  • Date: October 25, 2018

Lunch will be included. Attendance is by invitation and is free-of-charge.

Sponsorship was provided by:

Summit Artifacts

Minutes

See ATS 2018 Minutes

Presentation

Here are Tim's slides from the event: PDF

Video

Videos are available at: https://www.linaro.org/events/attended/ats-2018/

Also, Linaro has them on their YouTube channel at: https://www.youtube.com/playlist?list=PLKZSArYQptsPGftWXdlTI7vPCc42FXksD

Many thanks to Linaro as the sponsor for recording and processing the video for the event!

LWN.net report

Here is an article describing the report Kevin Hillman made of the summit. He made this report a few weeks later, at Plumbers Conference 2019.

The article was written by Jake Edge.

See https://lwn.net/Articles/771782/

Attendees

This was a closed, invitation-only event.

For this event, Kevin and I decided to keep it to a small focused group. Linaro sponsored recording the presentations and discussion, and the videos will be made available after the summit. I apologize if you would like to attend but have not been invited. We are hopeful that we can put together a public automated testing event in the future.

Invitations were sent out previously, and representatives from the following projects have agreed to come to the event.

  • 0-day
  • Fuego
  • Gentoo CI system
  • Buildbot
  • Jenkins
  • KernelCI
  • kerneltests.org
  • Kselftest
  • ktest
  • LAVA
  • Labgrid
  • LKFT
  • LTP
  • Opentest
  • Phoronix Test Suite
  • ptest
  • R4D
  • SLAV
  • syzkaller/syzbot
  • tbot
  • Xilinux testing
  • Yocto project (oeqa and ptest)

Pre-meeting work

Tim and Kevin worked on a glossary and survey, and CI loop diagram for discussion.

See Test Stack Survey for the work in progress.

Eventually, we'd like to fill out the information on: Test Stack Layers

Schedule

ATS 2018 Schedule
Time Topic Presenter or discussion leader Slides
9:00-9:10 Welcome and Introduction Tim and Kevin no slides
9:10-9:40 Vision and problem definition Tim ?
9:40-10:40 Glossary and Diagram discussion Tim and Kevin .
10:40-11:00 BREAK n/a .
11:00-12:30 test definition, build artifacts, execution API (E) . .
12:30-2:00 LUNCH BREAK n/a .
1:00-2:00 Brainstorming embedded Linux Requirements with CELP members Tim (discussion leader) .
2:00-2:50 run artifacts, standardized results format, parsers . .
2:50-3:10 BREAK n/a .
3:10-4:30 Farm standards - DUT control drivers, board definitions, discoverability, protocols . .
4:30-5:00 wrap-up Tim and Kevin .

Stuff not fit into schedule yet:

*

Agenda (brainstorming)

Here is some brainstorming on an agenda...

  • board farm survey (lightning talks)
    • what are people using?
    • what works?
    • what's missing?
      • board discovery/lab introspection?
  • layers and interfaces
    • what layers are supported, needed?
    • any way to leverage/separate/isolate existing software?
  • what tests need to be supported?
    • boot-time
    • run-time
    • package-based (package unit tests)
    • driver (hardware specific?)
      • requiring specialized hardware external to board (e.g. canbus simulator, hdmi frame-grabber)
    • multinode
      • how to allocate/schedule multiple pieces of equipment for a test (e.g. 2 or more nodes for a network test)
  • results reporting
    • centralized server and API to it (kernelCI json?)
  • sharing tests
  • how to define standards
    • de-facto only? (dominant project? (cough, LAVA))
    • documents?
  • survey of existing projects, and what pieces they focus on (or don't)