https://elinux.org/api.php?action=feedcontributions&user=Tklohna&feedformat=atomeLinux.org - User contributions [en]2024-03-29T05:53:14ZUser contributionsMediaWiki 1.31.0https://elinux.org/index.php?title=Automated_Testing_Summit_2019&diff=503556Automated Testing Summit 20192019-11-05T20:31:12Z<p>Tklohna: /* Presentations */</p>
<hr />
<div>This is a public planning page for the Automated Testing Summit<br />
<br />
Please see: https://events.linuxfoundation.org/events/ats-2019/<br />
<br />
= 2019 meeting =<br />
Coordinated by: Tim Bird<br />
This is now a full-blow Linux Foundation event, with sponsors, a CFP, program and everything.<br />
<br />
See the [https://events.linuxfoundation.org/events/ats-2019/ Linux Foundation Event page] for details.<br />
<br />
== Mailing list ==<br />
Inter-project discussions on Automated Testing occur on the "Automated Testing" mailing list of the Yocto Project<br />
* See https://lists.yoctoproject.org/listinfo/automated-testing<br />
<br />
= Date and Venue =<br />
* Location: Lyon, France<br />
* Venue: Lyon Convention Center, 50 Quai Charles de Gaulle<br />
* Date: October 31, 2019<br />
* Rooms: Pasteur Lounge and Rhone 3AB<br />
* Time: 9:00 am to 17:15 (5:15 pm)<br />
<br />
** co-located (following) Embedded Linux Conference Europe and Open Source Summit Europe<br />
** ELCE and OSSEU run Oct 28-30<br />
** see https://events.linuxfoundation.org/events/embedded-linux-conference-europe-2019/<br />
<br />
Attendance is by open to the public, and we are projecting the attendee fee to be $250, with this<br />
fee waived for speakers.<br />
<br />
= Summit Pre-planning =<br />
Here is a list of action items to do to prepare for the summit:<br />
* organize program committee (''done'')<br />
* get sponsors (''done'')<br />
** create sponsorship prospectus (''done'')<br />
** contact candidate sponsors (''done'')<br />
* hold "Call for Proposals" (''done'')<br />
** need to determine CFP dates (''done'')<br />
* determine agenda (''done'')<br />
** organize program committee (''done'')<br />
* advertise for attendees (''done'')<br />
* reserve venue, catering, AV rentals (''done'')<br />
* create signage, web site, registration system (''done'')<br />
* hold event (''done'')<br />
<br />
For private planning information, see [http://embeddedlinuxconference.com/cgi-bin/elcwiki.cgi/ATS_Planning the private planning wiki page]<br />
<br />
== Call for Presentations ==<br />
* CFP Open: Tuesday, May 28<br />
* CFP Close: Friday, August 13<br />
* Notifications Out: September 2<br />
* Schedule Out: September 4<br />
* Slides Due: October 24, 2019<br />
* Event Date: October 31, 2019<br />
<br />
Suggested Topics<br />
<br />
* Board Farms and Test Labs - hardware, software, useful practices<br />
* Layers and Interfaces - components of your automated CI loop and protocols between them<br />
* Results Reporting - results aggregation, visualization, notifications, etc.<br />
* Defining standards - propose or discuss conventions for test definitions, results formats, layer interoperability (e.g. protocols), etc.<br />
* Existing Projects - describe your test system or framework and any challenges you overcame or that are still outstanding<br />
* Scalability - supporting big data, or handling distributed builds or tests<br />
* Ease of use - making testing and test development accessible to non-QA folk (developers, end-users)<br />
<br />
= Pre-meeting work =<br />
Here's what Tim thinks we need to do:<br />
* Refine the test stack reference diagram<br />
* Organize committees for specific APIs<br />
** designate a head for each committee<br />
<br />
= Sponsors =<br />
Sponsorship was provided by:<br />
* '''Sony''' - Platinum sponsor<br />
* '''Core Embedded Linux Project''' (of the Linux Foundation) - Gold sponsor<br />
* '''Linaro''' - Bronze sponsor<br />
* '''Civil Infrastructure Platform''' - Bronze sponsor<br />
<br />
= Schedule =<br />
See https://ats19.sched.com/<br />
<br />
= Summit Artifacts =<br />
''documents and presentations''<br />
* for presentations - see below<br />
<br />
== Key decisions ==<br />
* all systems to make a kcidb client (upload results)<br />
** extend the kcidb schema<br />
** https://github.com/kernelci/kcidb<br />
** priority (over test definition unification work)<br />
* use LTP metadata format as initial standard<br />
** add meta-data convert to kselftest (Tim)<br />
* build a prototype to move boards between LAVA and labgrid (Jan)<br />
* want to collect more information for "Hardware Design for Testing"<br />
** Contributions are Welcome: https://github.com/SmithChart/Designing-for-Automated-Testing/<br />
* get something working for board management layer for poweron/poweroff/serial-console<br />
* continue work on test definition common schema and central repository (Tim)<br />
* add information about hardware you find useful, or have designed please add to [[Board_Farm]]<br />
* focus on meeting at Plumbers next year<br />
* continue using automated-testing@lists.yoctoproject.org<br />
<br />
== Action items from meeting ==<br />
Action items:<br />
* post slides to this page<br />
* send minutes<br />
<br />
== Presentations ==<br />
{| class="wikitable"<br />
|-<br />
! Session !! Presenter !! Slides<br />
|-<br />
| A Guide to CIP Testing || Chris Paterson & Michael Adler || [[File:2019-10-31 ATS-Lyon CIP Testing Overview v3.pdf]]<br />
|-<br />
| Update on previous conferences, KernelCI update || Kevin Hilman || [[File:ATS_2019.pdf]]<br />
|-<br />
| Bootloader testing in LAVA || Remi Duraffort || [[File:ATS-Lyon-Bootloader-testing-in-LAVA.pdf]]<br />
|-<br />
| New Ways Out of the Struggle of Testing Embedded Devices || Chris Fiege || [[File:ATS2019-cfi-embedded-testing_handout.pdf]]<br />
|-<br />
| SLAV: Test Stack Abstraction Layers || Paweł Wieczorek || [[File:Test_stack_abstraction_layers.pdf]]<br />
|-<br />
| A Survey of Open Source Test Definitions || Tim Bird || [[File:ATS2019-survey-of-test-definitions.pdf]]<br />
|-<br />
|Working together to Build a Modular CI Ecosystem|| Tim Bird || [[File:ATS2019- buiding-a-modular-ecosystem.pdf]]<br />
|-<br />
|Fuego Status Update|| Tim Bird || [[File:ATS2019-Fuego-Status-Update.pdf]]<br />
|-<br />
|Test Metadata Extraction|| Cyril Hrubis || [[File:Test metadata extraction.pdf]]<br />
|-<br />
|Beaker Project: Automated Testing at Red Hat|| Tomas Klohna || [https://redhat.slides.com/tklohna/deck?token=soUy6UhB Beaker at Slides.com] [[File:ATS19 - Beaker Project - Automated Testing at Red Hat.pdf]]<br />
|-<br />
|}</div>Tklohnahttps://elinux.org/index.php?title=File:ATS19_-_Beaker_Project_-_Automated_Testing_at_Red_Hat.pdf&diff=503551File:ATS19 - Beaker Project - Automated Testing at Red Hat.pdf2019-11-05T20:26:30Z<p>Tklohna: </p>
<hr />
<div></div>Tklohnahttps://elinux.org/index.php?title=Automated_Testing_Summit_2019&diff=503546Automated Testing Summit 20192019-11-05T20:23:58Z<p>Tklohna: /* Presentations */</p>
<hr />
<div>This is a public planning page for the Automated Testing Summit<br />
<br />
Please see: https://events.linuxfoundation.org/events/ats-2019/<br />
<br />
= 2019 meeting =<br />
Coordinated by: Tim Bird<br />
This is now a full-blow Linux Foundation event, with sponsors, a CFP, program and everything.<br />
<br />
See the [https://events.linuxfoundation.org/events/ats-2019/ Linux Foundation Event page] for details.<br />
<br />
== Mailing list ==<br />
Inter-project discussions on Automated Testing occur on the "Automated Testing" mailing list of the Yocto Project<br />
* See https://lists.yoctoproject.org/listinfo/automated-testing<br />
<br />
= Date and Venue =<br />
* Location: Lyon, France<br />
* Venue: Lyon Convention Center, 50 Quai Charles de Gaulle<br />
* Date: October 31, 2019<br />
* Rooms: Pasteur Lounge and Rhone 3AB<br />
* Time: 9:00 am to 17:15 (5:15 pm)<br />
<br />
** co-located (following) Embedded Linux Conference Europe and Open Source Summit Europe<br />
** ELCE and OSSEU run Oct 28-30<br />
** see https://events.linuxfoundation.org/events/embedded-linux-conference-europe-2019/<br />
<br />
Attendance is by open to the public, and we are projecting the attendee fee to be $250, with this<br />
fee waived for speakers.<br />
<br />
= Summit Pre-planning =<br />
Here is a list of action items to do to prepare for the summit:<br />
* organize program committee (''done'')<br />
* get sponsors (''done'')<br />
** create sponsorship prospectus (''done'')<br />
** contact candidate sponsors (''done'')<br />
* hold "Call for Proposals" (''done'')<br />
** need to determine CFP dates (''done'')<br />
* determine agenda (''done'')<br />
** organize program committee (''done'')<br />
* advertise for attendees (''done'')<br />
* reserve venue, catering, AV rentals (''done'')<br />
* create signage, web site, registration system (''done'')<br />
* hold event (''done'')<br />
<br />
For private planning information, see [http://embeddedlinuxconference.com/cgi-bin/elcwiki.cgi/ATS_Planning the private planning wiki page]<br />
<br />
== Call for Presentations ==<br />
* CFP Open: Tuesday, May 28<br />
* CFP Close: Friday, August 13<br />
* Notifications Out: September 2<br />
* Schedule Out: September 4<br />
* Slides Due: October 24, 2019<br />
* Event Date: October 31, 2019<br />
<br />
Suggested Topics<br />
<br />
* Board Farms and Test Labs - hardware, software, useful practices<br />
* Layers and Interfaces - components of your automated CI loop and protocols between them<br />
* Results Reporting - results aggregation, visualization, notifications, etc.<br />
* Defining standards - propose or discuss conventions for test definitions, results formats, layer interoperability (e.g. protocols), etc.<br />
* Existing Projects - describe your test system or framework and any challenges you overcame or that are still outstanding<br />
* Scalability - supporting big data, or handling distributed builds or tests<br />
* Ease of use - making testing and test development accessible to non-QA folk (developers, end-users)<br />
<br />
= Pre-meeting work =<br />
Here's what Tim thinks we need to do:<br />
* Refine the test stack reference diagram<br />
* Organize committees for specific APIs<br />
** designate a head for each committee<br />
<br />
= Sponsors =<br />
Sponsorship was provided by:<br />
* '''Sony''' - Platinum sponsor<br />
* '''Core Embedded Linux Project''' (of the Linux Foundation) - Gold sponsor<br />
* '''Linaro''' - Bronze sponsor<br />
* '''Civil Infrastructure Platform''' - Bronze sponsor<br />
<br />
= Schedule =<br />
See https://ats19.sched.com/<br />
<br />
= Summit Artifacts =<br />
''documents and presentations''<br />
* for presentations - see below<br />
<br />
== Key decisions ==<br />
* all systems to make a kcidb client (upload results)<br />
** extend the kcidb schema<br />
** https://github.com/kernelci/kcidb<br />
** priority (over test definition unification work)<br />
* use LTP metadata format as initial standard<br />
** add meta-data convert to kselftest (Tim)<br />
* build a prototype to move boards between LAVA and labgrid (Jan)<br />
* want to collect more information for "Hardware Design for Testing"<br />
** Contributions are Welcome: https://github.com/SmithChart/Designing-for-Automated-Testing/<br />
* get something working for board management layer for poweron/poweroff/serial-console<br />
* continue work on test definition common schema and central repository (Tim)<br />
* add information about hardware you find useful, or have designed please add to [[Board_Farm]]<br />
* focus on meeting at Plumbers next year<br />
* continue using automated-testing@lists.yoctoproject.org<br />
<br />
== Action items from meeting ==<br />
Action items:<br />
* post slides to this page<br />
* send minutes<br />
<br />
== Presentations ==<br />
{| class="wikitable"<br />
|-<br />
! Session !! Presenter !! Slides<br />
|-<br />
| A Guide to CIP Testing || Chris Paterson & Michael Adler || [[File:2019-10-31 ATS-Lyon CIP Testing Overview v3.pdf]]<br />
|-<br />
| Update on previous conferences, KernelCI update || Kevin Hilman || [[File:ATS_2019.pdf]]<br />
|-<br />
| Bootloader testing in LAVA || Remi Duraffort || [[File:ATS-Lyon-Bootloader-testing-in-LAVA.pdf]]<br />
|-<br />
| New Ways Out of the Struggle of Testing Embedded Devices || Chris Fiege || [[File:ATS2019-cfi-embedded-testing_handout.pdf]]<br />
|-<br />
| SLAV: Test Stack Abstraction Layers || Paweł Wieczorek || [[File:Test_stack_abstraction_layers.pdf]]<br />
|-<br />
| A Survey of Open Source Test Definitions || Tim Bird || [[File:ATS2019-survey-of-test-definitions.pdf]]<br />
|-<br />
|Working together to Build a Modular CI Ecosystem|| Tim Bird || [[File:ATS2019- buiding-a-modular-ecosystem.pdf]]<br />
|-<br />
|Fuego Status Update|| Tim Bird || [[File:ATS2019-Fuego-Status-Update.pdf]]<br />
|-<br />
|Test Metadata Extraction|| Cyril Hrubis || [[File:Test metadata extraction.pdf]]<br />
|-<br />
|Beaker Project: Automated Testing at Red Hat|| Tomas Klohna || [https://redhat.slides.com/tklohna/deck?token=soUy6UhB Beaker at Slides.com]<br />
|-<br />
|}</div>Tklohna