Board Provisioning Notes

This page has notes about "board provisioning", which is the process by which the software under test gets installed onto or made available to the board under test.

The board provisioning layer is responsible for instantiating the software on the device under test. This is a critical element of test automation. The wide variety of execution models, firmwares and hardware configurations, make it a challenge to create a generalized provisioning system. Essentially, the different variables or choices involved in provisioning make for an N-dimensional matrix of possibilities for solving the problem.

There are several different elements which might be involved in provisioning.
 * firmware or bootloader
 * kernel image
 * rootfs image
 * board power control
 * board button control
 * bus or network

It is very often the case that a particular lab or provisioning framework will require some specific element to be fixed. That is, for a particular dimension, the lab or framework does not provide any degrees of freedom, or they limit the degrees of freedom. This makes it hard to compare systems, or integrate them into a single framework.

= provisioning styles =

network-based

 * pxe-based boot
 * apply power, and firmware loads kernel image from network (using pxe)
 * presumably the rootfs also comes from the network (e.g. using NFS)


 * tftp-based boot
 * apply power, and firmware loads kernel image from network (using tftp)
 * presumably the rootfs also comes from the network (e.g. using NFS)

bus-based

 * android
 * uses fastboot to provide RAM-based kernel image
 * alternatively, uses fastboot to write new kernel and filesystem images to the device
 * many consumer products require hardware control (button hold or press during boot) to enter fastboot mode

flash based
In this scenario, the boot images are stored in flash, and the firmware can store new images to the board.

The firmware might support an update protocol, like DFU (Device Firmware Updgrade) or fastboot (see above). More commonly, in a lab setting, u-boot commands are entered to transfer images to the target device and store them into their appropriate locations in the board's flash memory. This requires that the test automation system have console access to U-boot (which usually means access to the serial console).

storage based
Many boards store their boot materials and images on sd cards. For these boards, there are multiple ways to put the images on the storage.

One automated solution involves multiplexing the storage between a host and the device, so that the host can write new images, and then make the images available to the device for software execution.

Another automated solution involves booting the machine into a "safe" mode (known to work), to use to transfer new images to the storage, and then re-booting into the software under test.

Many boards that do this use grub or u-boot, and a Linux kernel as the software for provisioning or re-imaging the board.

storage muxing
Another method of provisioning a board involves directly re-writing new software onto the storage device of the board, without the intervention of the software on the board. Many development board run their software from SDCards, and indeed the manual method of re-provisioning is to remove the SDcard from the product, reflash it with the new images using another machine, and reinsert the SDCard into the product.

There is hardware, called an SDMux, that allows one to automate these manual steps, by switching access to the SDCard storage between the device under test and a host machine.

update-controlled imaging
Many boards (especially those used in end-user products) use an over-the-air (OTA) update procedure as the primary way to apply new software to the device.

Often, many Consumer Electronics Products, support the ability to perform a local update using an update image that is contained on a USB stick.

In this case, the software that actually performs the update is built in to the product, and can be activated via means of holding a special button, or accessing an "update" menu on the device itself.

In the case of routers, it is often the case that an update is triggered using the web interface of the device to install new firmware.

Sometimes the update image is in an undocumented or encrypted format, and it may use cryptographic materials and techniques to only allow authorized users to update the device.

In this case, the primary obstacles to automation are:
 * the ability to get the product into "update mode"
 * the knowledge to create a compatible update image
 * and possibly the cryptographic keys required to make an image the device would accept
 * access to an OTA server that the machine would talk to
 * and possibly the cryptographic keys required for the machine to trust the server
 * ability to insert a USB device into the product

= issues =
 * power control - ability to reboot the board (or instantiate a virtual machine)
 * every provisioning system requires instantiating new software, which means booting with a different image
 * some provisioning systems require booting multiple times - once to enter a provisioning mode, and once to boot the software under test
 * button control - ability to press or hold buttons while a board boots
 * some hardware requires entering a special mode for provisioning, that is only accessible by holding buttons down or pressing buttons during boot
 * firmware control - ability to navigate menus or make selections in a firmware interface, during the provisioning process
 * requires expect-like functionality in order to make this generalizable
 * this might be done over USB (simulated keyboard), or serial port (using serial console)
 * some firmware is modal, requiring one mode for provisioning and a different mode for booting
 * product control - ability to request update via menu selections or user interface of the product itself
 * could require remote control (IR or bluetooth) interface to access product menus (e.g. for TVs)
 * to generalize this, it might even require reading the screen to navigate prompts (like a human would)
 * could require controlling a web interface (with requisite passwords, interpretation of response HTML, etc.) (e.g. for routers)
 * networking - ability to transfer images to the board during provisioning
 * USB - ability to use USB during provisioning operations
 * this might involve image transfer, firmware control and/or status and error detection
 * serial port - ability to use USB or serial for provisioning operations
 * this might involve image transfer, firmware control and/or status and error detection
 * status and error detection - ability to detect the power, network, bus, hardware, firmware, installation and/or operational status of a device, during provisioning

Special considerations: hardware-less test automation
See Provisioning without hardware support

= what's missing =
 * generic expect
 * hardware control of buttons
 * generic model for controlling firmware menus
 * generic model for controlling product menus
 * generic update format or manifest

= a generic model of provisioning = Here are the steps for a generic provisioning model. This is one where the machine is taken from an unprovisioned state, to one where the device is provisioned and opearting with the new software. In order to be generic, this includes many operations which might not be required for the provisioning of a particular specific system.

Steps: 0. prepare images suitable for an update 0.1. prepare an update manifest 1. transfer images to host transfer area 1.1 (eg. put image into tftp boot area) 1.2 (eg. put rootfs into nfs hosting area) 1.3 (eg. put images on a USB stick) 1.4 (eg. put images on an OTA server) 2. get into provisioning mode 2.1 turn power off 2.2 turn power on 2.3 apply hardware control (button holds or presses) required to enter provisioning mode 2.4 interact with firmware or product software to enter provisioning mode (select menus or provide inputs) 2.5 interact with hardware during provisioning mode (e.g. depress and release buttons) 3. transfer images to device 3.1 transmit image 3.2 detect image transfer complete 4. get into software execution mode 4.1 turn power off 4.2 turn power on 4.3 apply hardware control (button holds or presses) required to enter execution mode 4.4 interact with firmware to enter execution mode (select menus or provide inputs) 4.5 interact with hardware to enter execution mode 4.6 detect execution mode entered