Difference between revisions of "Automated Testing Summit 2019"

From eLinux.org
Jump to: navigation, search
(Presentations)
(Presentations: add Jan-Simon's presentation)
 
(13 intermediate revisions by 3 users not shown)
Line 79: Line 79:
  
 
= Summit Artifacts =
 
= Summit Artifacts =
''documents and presentations''
+
== Minutes and News coverage ==
* for presentations - see below
+
See [[ATS2019-Minutes]]
 +
 
 +
Jake Edge wrote an article about ATS 2019 for LWN.net.
 +
See https://lwn.net/Articles/804050/
  
 
== Key decisions ==
 
== Key decisions ==
Line 100: Line 103:
 
== Action items from meeting ==
 
== Action items from meeting ==
 
Action items:
 
Action items:
* post slides to this page
+
* post slides to this page (all but one present)
* send minutes
+
* send minutes (done)
  
 
== Presentations ==
 
== Presentations ==
Line 108: Line 111:
 
! Session !! Presenter !! Slides
 
! Session !! Presenter !! Slides
 
|- bgcolor="#a0c0c0"
 
|- bgcolor="#a0c0c0"
| colspan="3" align="center" | Status updates and collaboration issues
+
| colspan="3" align="center" | Status updates
 
|-
 
|-
| Update on previous conferences, KernelCI update || Kevin Hilman || [[File:ATS_2019.pdf]]
+
| The bugs are too fast, and why we can't catch them (and a report on recent meetups) || Kevin Hilman || [[File:ATS_2019.pdf]]
 
|-
 
|-
| LKFT status update || Milosz Wasilewski || <need file>
+
| LKFT status update || Milosz Wasilewski || [[File:2019_ATS_LKFT_status.pdf]]
 
|-
 
|-
 
| Fuego Status Update|| Tim Bird || [[File:ATS2019-Fuego-Status-Update.pdf]]
 
| Fuego Status Update|| Tim Bird || [[File:ATS2019-Fuego-Status-Update.pdf]]
 
|-
 
|-
| CKI Status Update || Veronika Kabatova || <need file>
+
| KernelCI Status Update || Kevin Hilman || [[File:ATS_2019.pdf]] (starting at slide 15)
 
|-
 
|-
| SLAV: Test Stack Abstraction Layers || Paweł Wieczorek || [[File:Test_stack_abstraction_layers.pdf]]
+
| CKI Status Update || Veronika Kabatova || [[File:cki-overview-ats.pdf]]
 +
|-
 +
| SLAV Status Update || Paweł Wieczorek || [[File:SLAV_Status_Update.pdf]]
 +
|- bgcolor="#a0c0c0"
 +
| colspan="3" align="center" | Collaboration philosophy and issues
 
|-
 
|-
| Open Testing Philosophy || Kevin Hilman || <need file>?
+
| Open Testing Philosophy || Kevin Hilman || [[File:ATS_2019.pdf]] (starting at slide 23)
 
|-  
 
|-  
 
| Working together to Build a Modular CI Ecosystem|| Tim Bird || [[File:ATS2019- buiding-a-modular-ecosystem.pdf]]
 
| Working together to Build a Modular CI Ecosystem|| Tim Bird || [[File:ATS2019- buiding-a-modular-ecosystem.pdf]]
Line 126: Line 133:
 
| colspan="3"  align="center" | Board/LAB management track  
 
| colspan="3"  align="center" | Board/LAB management track  
 
|-
 
|-
| Labgrid - real world examples || Jan Lübbe || <need file>
+
| Labgrid - Real World Examples || Jan Lübbe || [[File:ATS2019-jlu-labgrid.pdf]]
 
|-
 
|-
 
| New Ways Out of the Struggle of Testing Embedded Devices || Chris Fiege || [[File:ATS2019-cfi-embedded-testing_handout.pdf]]
 
| New Ways Out of the Struggle of Testing Embedded Devices || Chris Fiege || [[File:ATS2019-cfi-embedded-testing_handout.pdf]]
Line 139: Line 146:
 
|-
 
|-
 
| Test Metadata Extraction|| Cyril Hrubis || [[File:Test metadata extraction.pdf]]
 
| Test Metadata Extraction|| Cyril Hrubis || [[File:Test metadata extraction.pdf]]
 +
|-
 +
| Test Plan Templating in LAVA || Milosz Wasilewski || [[File:Testjob templating for LAVA.pdf]]
 
|-
 
|-
 
| A Survey of Open Source Test Definitions || Tim Bird || [[File:ATS2019-survey-of-test-definitions.pdf]]
 
| A Survey of Open Source Test Definitions || Tim Bird || [[File:ATS2019-survey-of-test-definitions.pdf]]
Line 144: Line 153:
 
| colspan="3"  align="center" | QA systems and issues
 
| colspan="3"  align="center" | QA systems and issues
 
|-
 
|-
| How AGL Tests its distros and what challenges we face || Jan-Simon Möller || <need file>
+
| How AGL Tests its distros and what challenges we face || Jan-Simon Möller || [[File:How AGL tests its distros and what challenges we face.pdf]]
 
|-
 
|-
 
| A Guide to CIP Testing || Chris Paterson & Michael Adler || [[File:2019-10-31 ATS-Lyon CIP Testing Overview v3.pdf]]
 
| A Guide to CIP Testing || Chris Paterson & Michael Adler || [[File:2019-10-31 ATS-Lyon CIP Testing Overview v3.pdf]]
 
|-
 
|-
 
|}
 
|}

Latest revision as of 00:58, 21 November 2019

This is a public planning page for the Automated Testing Summit

Please see: https://events.linuxfoundation.org/events/ats-2019/

2019 meeting

Coordinated by: Tim Bird This is now a full-blow Linux Foundation event, with sponsors, a CFP, program and everything.

See the Linux Foundation Event page for details.

Mailing list

Inter-project discussions on Automated Testing occur on the "Automated Testing" mailing list of the Yocto Project

Date and Venue

  • Location: Lyon, France
  • Venue: Lyon Convention Center, 50 Quai Charles de Gaulle
  • Date: October 31, 2019
  • Rooms: Pasteur Lounge and Rhone 3AB
  • Time: 9:00 am to 17:15 (5:15 pm)

Attendance is by open to the public, and we are projecting the attendee fee to be $250, with this fee waived for speakers.

Summit Pre-planning

Here is a list of action items to do to prepare for the summit:

  • organize program committee (done)
  • get sponsors (done)
    • create sponsorship prospectus (done)
    • contact candidate sponsors (done)
  • hold "Call for Proposals" (done)
    • need to determine CFP dates (done)
  • determine agenda (done)
    • organize program committee (done)
  • advertise for attendees (done)
  • reserve venue, catering, AV rentals (done)
  • create signage, web site, registration system (done)
  • hold event (done)

For private planning information, see the private planning wiki page

Call for Presentations

  • CFP Open: Tuesday, May 28
  • CFP Close: Friday, August 13
  • Notifications Out: September 2
  • Schedule Out: September 4
  • Slides Due: October 24, 2019
  • Event Date: October 31, 2019

Suggested Topics

  • Board Farms and Test Labs - hardware, software, useful practices
  • Layers and Interfaces - components of your automated CI loop and protocols between them
  • Results Reporting - results aggregation, visualization, notifications, etc.
  • Defining standards - propose or discuss conventions for test definitions, results formats, layer interoperability (e.g. protocols), etc.
  • Existing Projects - describe your test system or framework and any challenges you overcame or that are still outstanding
  • Scalability - supporting big data, or handling distributed builds or tests
  • Ease of use - making testing and test development accessible to non-QA folk (developers, end-users)

Pre-meeting work

Here's what Tim thinks we need to do:

  • Refine the test stack reference diagram
  • Organize committees for specific APIs
    • designate a head for each committee

Sponsors

Sponsorship was provided by:

  • Sony - Platinum sponsor
  • Core Embedded Linux Project (of the Linux Foundation) - Gold sponsor
  • Linaro - Bronze sponsor
  • Civil Infrastructure Platform - Bronze sponsor

Schedule

See https://ats19.sched.com/

Summit Artifacts

Minutes and News coverage

See ATS2019-Minutes

Jake Edge wrote an article about ATS 2019 for LWN.net. See https://lwn.net/Articles/804050/

Key decisions

  • all systems to make a kcidb client (upload results)
  • use LTP metadata format as initial standard
    • add meta-data convert to kselftest (Tim)
  • build a prototype to move boards between LAVA and labgrid (Jan)
  • want to collect more information for "Hardware Design for Testing"
  • get something working for board management layer for poweron/poweroff/serial-console
  • continue work on test definition common schema and central repository (Tim)
  • add information about hardware you find useful, or have designed please add to Board_Farm
  • focus on meeting at Plumbers next year
  • continue using automated-testing@lists.yoctoproject.org

Action items from meeting

Action items:

  • post slides to this page (all but one present)
  • send minutes (done)

Presentations

Session Presenter Slides
Status updates
The bugs are too fast, and why we can't catch them (and a report on recent meetups) Kevin Hilman File:ATS 2019.pdf
LKFT status update Milosz Wasilewski File:2019 ATS LKFT status.pdf
Fuego Status Update Tim Bird File:ATS2019-Fuego-Status-Update.pdf
KernelCI Status Update Kevin Hilman File:ATS 2019.pdf (starting at slide 15)
CKI Status Update Veronika Kabatova File:Cki-overview-ats.pdf
SLAV Status Update Paweł Wieczorek File:SLAV Status Update.pdf
Collaboration philosophy and issues
Open Testing Philosophy Kevin Hilman File:ATS 2019.pdf (starting at slide 23)
Working together to Build a Modular CI Ecosystem Tim Bird File:ATS2019- buiding-a-modular-ecosystem.pdf
Board/LAB management track
Labgrid - Real World Examples Jan Lübbe File:ATS2019-jlu-labgrid.pdf
New Ways Out of the Struggle of Testing Embedded Devices Chris Fiege File:ATS2019-cfi-embedded-testing handout.pdf
Beaker Project: Automated Testing at Red Hat Tomas Klohna Beaker at Slides.com File:ATS19 - Beaker Project - Automated Testing at Red Hat.pdf
SLAV: Test Stack Abstraction Layers Paweł Wieczorek File:Test stack abstraction layers.pdf
Test systems/ Test definitions track
Bootloader testing in LAVA Remi Duraffort File:ATS-Lyon-Bootloader-testing-in-LAVA.pdf
Test Metadata Extraction Cyril Hrubis File:Test metadata extraction.pdf
Test Plan Templating in LAVA Milosz Wasilewski File:Testjob templating for LAVA.pdf
A Survey of Open Source Test Definitions Tim Bird File:ATS2019-survey-of-test-definitions.pdf
QA systems and issues
How AGL Tests its distros and what challenges we face Jan-Simon Möller File:How AGL tests its distros and what challenges we face.pdf
A Guide to CIP Testing Chris Paterson & Michael Adler File:2019-10-31 ATS-Lyon CIP Testing Overview v3.pdf