Automated Testing microsummit Linaro Connect BKK19

From eLinux.org
Jump to: navigation, search

Microsummit Description

Linaro invited Tim Bird and Daniel Sangorrin from the Fuego project to Linaro Connect 2019 in Bangkok. At thie event, a "Testing microsummit" meeting was organized by Dan Rue.


Linaro Connect[1] was held April 1-5, 2019 in Bangkok. There were a lot of sessions (listed below) around Linux testing, LAVA, and Fuego.

We also held an impromptu micro-summit (minutes below), where we all locked ourselves in a room together for the day and discussed LAVA, Fuego, test definitions, and yes, pdudaemon.

Finally, Maria Högberg (cc'd) has graciously arranged for a monthly meeting that we can use to coordinate across projects throughout the year - please send her an email if you'd like to be added. The first one is scheduled for Thursday, May 9th at 13:00 UTC.

Linaro Connect 2019 BBK - sessions on testing

Selected Sessions (videos should be available within a week or two at the referenced URLs):

Micro-Summit Minutes

Date: Wednesday, April 4, 2019

Attendees

  • Daniel Sangorrin (Toshiba)
  • Tim Bird (Sony)
  • Carlos Hernandez (TI)
  • Kat Cosgrove (jfrog)
  • Naresh Kamboju
  • Antonio Tercerio
  • Daniel Diaz
  • Charles Oliveira
  • Chase Qi
  • Milosz Wasilewski
  • Stevan Radakovic
  • Dave Pigott
  • Maria Högbergadding
  • Luca Di Stefano
  • Steve McIntyre
  • Matt Hart
  • Remi Duraffort
  • Fathi Boudra
  • Anders Roxell
  • Dan Rue

PDU API

  • Wider community agreement on standardizing on pdudaemon
  • Linaro lab doesn’t actually use pdudaemon
  • Running pdudaemon @ Linaro
    • [Dave] Queueing vs running immediately
    • [matt] we can just add it to pdu daemon if this is a problem
    • [Matt] added in a PR
  • Dave will write a proposal for power control API
    • There will be an abstraction layer for power control to turn a device on or off.
  • [Dave] What about pressing buttons, turning on/off usb ports, etc
    • [Tim] let’s not deal with composition right now, and only components
    • [Carlos] we do need to solve the problem… once we have an interface for power, we need a similar interface for relays.
    • Combinations of things may still make it complicated. For example, hold a button while pressing power.
  • Core pdu api actions: on, off, status
  • Core relay api actions: on, off, ???

USB control

  • LAVA lab uses cambrionix USB hubs
  • Also use good usb shielded cables
  • Sony uses an open hardware board per DUT

Fuego Architecture

  • Tim gave a subset of his Thursday talk; see his Thursday talk for the full details.
  • Jenkins based
  • Test execution system
    • Steps broken into discrete phases like build, deploy, run, process, etc.
    • Board and platform management are abstracted
  • Not serial console based; typically uses SSH on a board that’s already provisioned (imaged)
  • Contains functional and benchmark tests
  • Fuego is a linux distribution, distributed as a container, self contained
  • Testing driven from host, using ssh connection to DUT
  • Is container privileged?
    • [tim] yes
    • [dave] Dynamic device detection is possible without privileged mode
  • Designed for embedded linux testing
    • IoT possible
  • By default, builds tests, though LTP has a way of checking to see if it’s on disk and skipping the build
    • Can also just skip the build phase so long as the build is in the location that is expected on target
  • [long discussion about skip lists, known issues, and (lack of) test documentation]
    • Fuego-core has skip logic inside the test folder (ltp for example)
    • Test-definitions has similar
    • Additionally, lkft uses known issues in SQUAD to annotate known failures

LAVA Architecture

  • Remi gave a brief LAVA architecture overview
  • Components:
    • Lava-server
      • Apache2, lava-server-gunicorn, postgresql, lava-logs, lava-master
    • Lava-dispatcher
      • Lava-slave, lava-run, devices under test
  • One lava-server, multiple dispatchers, multiple DUTs per dispatcher
  • Lava-run is ephemeral process that communicates with DUT during a job run
  • Device-type defines e.g. a raspberry pi
  • Device defines an instance of a raspberry pi, with specifics about which port its plugged into, what serial port is it, etc.
  • [daniel s] can i skip the deploy?
    • Yes, it’s supported
  • [daniel s] parsing in the dispatcher, where is that done?
    • [steve] we recommend parsing gets done on DUT
    • [remi] interestedin junit, tap13 parsing in LAVA so that a parser on DUT isn’t necessary
    • [remi] goal is to not have to parse on DUT and do as much as we can on dispatcher
  • Multinode jobs, used for networking, controlling lab hardware using lxc, etc
  • Remi mentioned a feature to run a container as a part of a test run, which could help with jobs that are more complex, and eliminate a lot of existing multi-node jobs (making it easier)

Running Fuego tests in Lava

  • Fuego operates a lot like an "interactive" LAVA test
  • Are several solutions here: One that seems good is to use the new feature to run a container as part of a test.

Running Linaro tests in Fuego

  • Fuego already has Functional.linaro, which runs a Linaro test using testrunner.

Test Definitions

Tim Bird proposing `run-test.sh` as a naming convention for the name of the test program for each package

  • In ptest, you can build a test package, which you can install and execute.
  • More controversially, why don’t you rename all the <testname>.sh scripts to run-test.sh
  • Tim Bird proposing a standard variable name for the location of kernel config path
    • LTP implements ‘KCONFIG_PATH’

How do we decide such standards?

  • Proposal @ Automated testing list
  • Tap13 is just a website. Maybe we need something similar to publish these things on

Tim Bird on Test Names

  • Standardizing on test names would be nice so that we can compare across projects
  • Related to test case definition discussion
  • A good first step is to investigate how tests are named by various projects, starting with LTP.
    • How different are test case names for LTP?

Carlos Hernandez - Test case definition standards (proposal) Provide everything needed to run any test from any test framework.

  • TGUID
  • Test Name
  • Description
  • Test Execution Engine (e.g. LAVA, Fuego, VATF)
  • Test script path

Next Steps

  • Agreement to start a public monthly meeting. Maria will organize.

Notes

Linaro graciously sponsored Tim and Daniel's attendance at LCC BKK19.