Difference between revisions of "Automated Testing Summit 2019"
(→Presentations: add Jan-Simon's presentation) |
|||
(52 intermediate revisions by 9 users not shown) | |||
Line 1: | Line 1: | ||
This is a public planning page for the Automated Testing Summit | This is a public planning page for the Automated Testing Summit | ||
+ | |||
+ | Please see: https://events.linuxfoundation.org/events/ats-2019/ | ||
= 2019 meeting = | = 2019 meeting = | ||
Coordinated by: Tim Bird | Coordinated by: Tim Bird | ||
+ | This is now a full-blow Linux Foundation event, with sponsors, a CFP, program and everything. | ||
+ | |||
+ | See the [https://events.linuxfoundation.org/events/ats-2019/ Linux Foundation Event page] for details. | ||
== Mailing list == | == Mailing list == | ||
Line 12: | Line 17: | ||
* Venue: Lyon Convention Center, 50 Quai Charles de Gaulle | * Venue: Lyon Convention Center, 50 Quai Charles de Gaulle | ||
* Date: October 31, 2019 | * Date: October 31, 2019 | ||
− | * Rooms: | + | * Rooms: Pasteur Lounge and Rhone 3AB |
− | * Time: | + | * Time: 9:00 am to 17:15 (5:15 pm) |
** co-located (following) Embedded Linux Conference Europe and Open Source Summit Europe | ** co-located (following) Embedded Linux Conference Europe and Open Source Summit Europe | ||
Line 24: | Line 29: | ||
= Summit Pre-planning = | = Summit Pre-planning = | ||
Here is a list of action items to do to prepare for the summit: | Here is a list of action items to do to prepare for the summit: | ||
− | + | * organize program committee (''done'') | |
− | + | * get sponsors (''done'') | |
− | + | ** create sponsorship prospectus (''done'') | |
− | + | ** contact candidate sponsors (''done'') | |
− | + | * hold "Call for Proposals" (''done'') | |
− | + | ** need to determine CFP dates (''done'') | |
+ | * determine agenda (''done'') | ||
+ | ** organize program committee (''done'') | ||
+ | * advertise for attendees (''done'') | ||
+ | * reserve venue, catering, AV rentals (''done'') | ||
+ | * create signage, web site, registration system (''done'') | ||
+ | * hold event (''done'') | ||
+ | |||
+ | For private planning information, see [http://embeddedlinuxconference.com/cgi-bin/elcwiki.cgi/ATS_Planning the private planning wiki page] | ||
− | Sponsorship | + | == Call for Presentations == |
− | * ''( | + | * CFP Open: Tuesday, May 28 |
+ | * CFP Close: Friday, August 13 | ||
+ | * Notifications Out: September 2 | ||
+ | * Schedule Out: September 4 | ||
+ | * Slides Due: October 24, 2019 | ||
+ | * Event Date: October 31, 2019 | ||
+ | |||
+ | Suggested Topics | ||
+ | |||
+ | * Board Farms and Test Labs - hardware, software, useful practices | ||
+ | * Layers and Interfaces - components of your automated CI loop and protocols between them | ||
+ | * Results Reporting - results aggregation, visualization, notifications, etc. | ||
+ | * Defining standards - propose or discuss conventions for test definitions, results formats, layer interoperability (e.g. protocols), etc. | ||
+ | * Existing Projects - describe your test system or framework and any challenges you overcame or that are still outstanding | ||
+ | * Scalability - supporting big data, or handling distributed builds or tests | ||
+ | * Ease of use - making testing and test development accessible to non-QA folk (developers, end-users) | ||
+ | |||
+ | = Pre-meeting work = | ||
+ | Here's what Tim thinks we need to do: | ||
+ | * Refine the test stack reference diagram | ||
+ | * Organize committees for specific APIs | ||
+ | ** designate a head for each committee | ||
+ | |||
+ | = Sponsors = | ||
+ | Sponsorship was provided by: | ||
+ | * '''Sony''' - Platinum sponsor | ||
+ | * '''Core Embedded Linux Project''' (of the Linux Foundation) - Gold sponsor | ||
+ | * '''Linaro''' - Bronze sponsor | ||
+ | * '''Civil Infrastructure Platform''' - Bronze sponsor | ||
= Schedule = | = Schedule = | ||
− | + | See https://ats19.sched.com/ | |
= Summit Artifacts = | = Summit Artifacts = | ||
− | + | == Minutes and News coverage == | |
+ | See [[ATS2019-Minutes]] | ||
+ | |||
+ | Jake Edge wrote an article about ATS 2019 for LWN.net. | ||
+ | See https://lwn.net/Articles/804050/ | ||
+ | |||
+ | == Key decisions == | ||
+ | * all systems to make a kcidb client (upload results) | ||
+ | ** extend the kcidb schema | ||
+ | ** https://github.com/kernelci/kcidb | ||
+ | ** priority (over test definition unification work) | ||
+ | * use LTP metadata format as initial standard | ||
+ | ** add meta-data convert to kselftest (Tim) | ||
+ | * build a prototype to move boards between LAVA and labgrid (Jan) | ||
+ | * want to collect more information for "Hardware Design for Testing" | ||
+ | ** Contributions are Welcome: https://github.com/SmithChart/Designing-for-Automated-Testing/ | ||
+ | * get something working for board management layer for poweron/poweroff/serial-console | ||
+ | * continue work on test definition common schema and central repository (Tim) | ||
+ | * add information about hardware you find useful, or have designed please add to [[Board_Farm]] | ||
+ | * focus on meeting at Plumbers next year | ||
+ | * continue using automated-testing@lists.yoctoproject.org | ||
== Action items from meeting == | == Action items from meeting == | ||
Action items: | Action items: | ||
+ | * post slides to this page (all but one present) | ||
+ | * send minutes (done) | ||
== Presentations == | == Presentations == | ||
− | + | {| class="wikitable" | |
− | == | + | |- |
− | + | ! Session !! Presenter !! Slides | |
− | + | |- bgcolor="#a0c0c0" | |
− | = | + | | colspan="3" align="center" | Status updates |
+ | |- | ||
+ | | The bugs are too fast, and why we can't catch them (and a report on recent meetups) || Kevin Hilman || [[File:ATS_2019.pdf]] | ||
+ | |- | ||
+ | | LKFT status update || Milosz Wasilewski || [[File:2019_ATS_LKFT_status.pdf]] | ||
+ | |- | ||
+ | | Fuego Status Update|| Tim Bird || [[File:ATS2019-Fuego-Status-Update.pdf]] | ||
+ | |- | ||
+ | | KernelCI Status Update || Kevin Hilman || [[File:ATS_2019.pdf]] (starting at slide 15) | ||
+ | |- | ||
+ | | CKI Status Update || Veronika Kabatova || [[File:cki-overview-ats.pdf]] | ||
+ | |- | ||
+ | | SLAV Status Update || Paweł Wieczorek || [[File:SLAV_Status_Update.pdf]] | ||
+ | |- bgcolor="#a0c0c0" | ||
+ | | colspan="3" align="center" | Collaboration philosophy and issues | ||
+ | |- | ||
+ | | Open Testing Philosophy || Kevin Hilman || [[File:ATS_2019.pdf]] (starting at slide 23) | ||
+ | |- | ||
+ | | Working together to Build a Modular CI Ecosystem|| Tim Bird || [[File:ATS2019- buiding-a-modular-ecosystem.pdf]] | ||
+ | |- bgcolor="#a0c0c0" | ||
+ | | colspan="3" align="center" | Board/LAB management track | ||
+ | |- | ||
+ | | Labgrid - Real World Examples || Jan Lübbe || [[File:ATS2019-jlu-labgrid.pdf]] | ||
+ | |- | ||
+ | | New Ways Out of the Struggle of Testing Embedded Devices || Chris Fiege || [[File:ATS2019-cfi-embedded-testing_handout.pdf]] | ||
+ | |- | ||
+ | | Beaker Project: Automated Testing at Red Hat|| Tomas Klohna || [https://redhat.slides.com/tklohna/deck?token=soUy6UhB Beaker at Slides.com] [[File:ATS19 - Beaker Project - Automated Testing at Red Hat.pdf]] | ||
+ | |- | ||
+ | | SLAV: Test Stack Abstraction Layers || Paweł Wieczorek || [[File:Test_stack_abstraction_layers.pdf]] | ||
+ | |- bgcolor="#a0c0c0" | ||
+ | | colspan="3" align="center" | Test systems/ Test definitions track | ||
+ | |- | ||
+ | | Bootloader testing in LAVA || Remi Duraffort || [[File:ATS-Lyon-Bootloader-testing-in-LAVA.pdf]] | ||
+ | |- | ||
+ | | Test Metadata Extraction|| Cyril Hrubis || [[File:Test metadata extraction.pdf]] | ||
+ | |- | ||
+ | | Test Plan Templating in LAVA || Milosz Wasilewski || [[File:Testjob templating for LAVA.pdf]] | ||
+ | |- | ||
+ | | A Survey of Open Source Test Definitions || Tim Bird || [[File:ATS2019-survey-of-test-definitions.pdf]] | ||
+ | |- bgcolor="#a0c0c0" | ||
+ | | colspan="3" align="center" | QA systems and issues | ||
+ | |- | ||
+ | | How AGL Tests its distros and what challenges we face || Jan-Simon Möller || [[File:How AGL tests its distros and what challenges we face.pdf]] | ||
+ | |- | ||
+ | | A Guide to CIP Testing || Chris Paterson & Michael Adler || [[File:2019-10-31 ATS-Lyon CIP Testing Overview v3.pdf]] | ||
+ | |- | ||
+ | |} |
Revision as of 01:58, 21 November 2019
This is a public planning page for the Automated Testing Summit
Please see: https://events.linuxfoundation.org/events/ats-2019/
Contents
2019 meeting
Coordinated by: Tim Bird This is now a full-blow Linux Foundation event, with sponsors, a CFP, program and everything.
See the Linux Foundation Event page for details.
Mailing list
Inter-project discussions on Automated Testing occur on the "Automated Testing" mailing list of the Yocto Project
Date and Venue
- Location: Lyon, France
- Venue: Lyon Convention Center, 50 Quai Charles de Gaulle
- Date: October 31, 2019
- Rooms: Pasteur Lounge and Rhone 3AB
- Time: 9:00 am to 17:15 (5:15 pm)
- co-located (following) Embedded Linux Conference Europe and Open Source Summit Europe
- ELCE and OSSEU run Oct 28-30
- see https://events.linuxfoundation.org/events/embedded-linux-conference-europe-2019/
Attendance is by open to the public, and we are projecting the attendee fee to be $250, with this fee waived for speakers.
Summit Pre-planning
Here is a list of action items to do to prepare for the summit:
- organize program committee (done)
- get sponsors (done)
- create sponsorship prospectus (done)
- contact candidate sponsors (done)
- hold "Call for Proposals" (done)
- need to determine CFP dates (done)
- determine agenda (done)
- organize program committee (done)
- advertise for attendees (done)
- reserve venue, catering, AV rentals (done)
- create signage, web site, registration system (done)
- hold event (done)
For private planning information, see the private planning wiki page
Call for Presentations
- CFP Open: Tuesday, May 28
- CFP Close: Friday, August 13
- Notifications Out: September 2
- Schedule Out: September 4
- Slides Due: October 24, 2019
- Event Date: October 31, 2019
Suggested Topics
- Board Farms and Test Labs - hardware, software, useful practices
- Layers and Interfaces - components of your automated CI loop and protocols between them
- Results Reporting - results aggregation, visualization, notifications, etc.
- Defining standards - propose or discuss conventions for test definitions, results formats, layer interoperability (e.g. protocols), etc.
- Existing Projects - describe your test system or framework and any challenges you overcame or that are still outstanding
- Scalability - supporting big data, or handling distributed builds or tests
- Ease of use - making testing and test development accessible to non-QA folk (developers, end-users)
Pre-meeting work
Here's what Tim thinks we need to do:
- Refine the test stack reference diagram
- Organize committees for specific APIs
- designate a head for each committee
Sponsors
Sponsorship was provided by:
- Sony - Platinum sponsor
- Core Embedded Linux Project (of the Linux Foundation) - Gold sponsor
- Linaro - Bronze sponsor
- Civil Infrastructure Platform - Bronze sponsor
Schedule
Summit Artifacts
Minutes and News coverage
See ATS2019-Minutes
Jake Edge wrote an article about ATS 2019 for LWN.net. See https://lwn.net/Articles/804050/
Key decisions
- all systems to make a kcidb client (upload results)
- extend the kcidb schema
- https://github.com/kernelci/kcidb
- priority (over test definition unification work)
- use LTP metadata format as initial standard
- add meta-data convert to kselftest (Tim)
- build a prototype to move boards between LAVA and labgrid (Jan)
- want to collect more information for "Hardware Design for Testing"
- Contributions are Welcome: https://github.com/SmithChart/Designing-for-Automated-Testing/
- get something working for board management layer for poweron/poweroff/serial-console
- continue work on test definition common schema and central repository (Tim)
- add information about hardware you find useful, or have designed please add to Board_Farm
- focus on meeting at Plumbers next year
- continue using automated-testing@lists.yoctoproject.org
Action items from meeting
Action items:
- post slides to this page (all but one present)
- send minutes (done)
Presentations
Session | Presenter | Slides |
---|---|---|
Status updates | ||
The bugs are too fast, and why we can't catch them (and a report on recent meetups) | Kevin Hilman | File:ATS 2019.pdf |
LKFT status update | Milosz Wasilewski | File:2019 ATS LKFT status.pdf |
Fuego Status Update | Tim Bird | File:ATS2019-Fuego-Status-Update.pdf |
KernelCI Status Update | Kevin Hilman | File:ATS 2019.pdf (starting at slide 15) |
CKI Status Update | Veronika Kabatova | File:Cki-overview-ats.pdf |
SLAV Status Update | Paweł Wieczorek | File:SLAV Status Update.pdf |
Collaboration philosophy and issues | ||
Open Testing Philosophy | Kevin Hilman | File:ATS 2019.pdf (starting at slide 23) |
Working together to Build a Modular CI Ecosystem | Tim Bird | File:ATS2019- buiding-a-modular-ecosystem.pdf |
Board/LAB management track | ||
Labgrid - Real World Examples | Jan Lübbe | File:ATS2019-jlu-labgrid.pdf |
New Ways Out of the Struggle of Testing Embedded Devices | Chris Fiege | File:ATS2019-cfi-embedded-testing handout.pdf |
Beaker Project: Automated Testing at Red Hat | Tomas Klohna | Beaker at Slides.com File:ATS19 - Beaker Project - Automated Testing at Red Hat.pdf |
SLAV: Test Stack Abstraction Layers | Paweł Wieczorek | File:Test stack abstraction layers.pdf |
Test systems/ Test definitions track | ||
Bootloader testing in LAVA | Remi Duraffort | File:ATS-Lyon-Bootloader-testing-in-LAVA.pdf |
Test Metadata Extraction | Cyril Hrubis | File:Test metadata extraction.pdf |
Test Plan Templating in LAVA | Milosz Wasilewski | File:Testjob templating for LAVA.pdf |
A Survey of Open Source Test Definitions | Tim Bird | File:ATS2019-survey-of-test-definitions.pdf |
QA systems and issues | ||
How AGL Tests its distros and what challenges we face | Jan-Simon Möller | File:How AGL tests its distros and what challenges we face.pdf |
A Guide to CIP Testing | Chris Paterson & Michael Adler | File:2019-10-31 ATS-Lyon CIP Testing Overview v3.pdf |