Add support for serial-only targets to Fuego


 * Summary: Add support for serial-only targets to Fuego
 * Proposer: Tim Bird

Description
Currently, the Fuego test system supports targets that have an ssh daemon running on the target. ssh is used for command execution and file transfer operations.

This project will consist of adding support to the Fuego test system for command execution, file transfer operations, and log retrieval over serial console. This feature is particularly intended for targets that have no network or bus connection to the host, and only support a serial console. However it should work on any target that supports a serial console.

It is envisioned that something like zmodem will be used on both host and target side to transfer files over the serial connection, and that command execution will be accomplished by just echoing the command to the shell on the serial console.

There should be no pre-requisite software requirement on the target. That is, this feature should work with any target board that has a kernel and shell and execution environment. It should not be assumed that any particular program (such as lrzsz) is already installed. Thus, the feature may need to 'bootstrap' itself, by building lrzsz (or some other custom program), and transfer that to the target, before it can begin normal operations. Possibly this can be done during the pre-test phase with a special command to instantiate the transport. If the feature does need a target-side program, then it will detect if it is already there before installing one that was built by fuego.

The test functions get, put, and cmd, should work the same with a board that uses the 'serial' transport as one that uses the 'ssh' transport. Also, all internal get, put and cmd operations should work correctly. This means that the serial feature needs to support the same argument order and wildcard expansion as ssh.

where to place code
Fuego already includes stubs for where to place the commands to perform execution and file transfer. Please, in see the file, the functions:
 * fuego-core/engine/overlays/base/base-board.fuegoclass,
 * ov_transport_get
 * ov_transport_put
 * ov_transport_cmd

All of these functions take multiple arguments.

In the case of get, the comment in the source (as of Aug 2016) says that the first argument is the destination, and subsequent arguments are the sources.

Note: This makes no sense - it should be all but last argument are sources, and last argument is the destination

The first argument may contain a wildcard.

I believe the actual get command takes a first argument of source (which is on the target), and subsequent arguments as destination (which should be a single directory or file on the host)

In the case of put, all but the last argument are the sources, and the last argument is the destination.

In the case of cmd, all arguments are passed as a single string to the target to execute.

Deliverables
This task will be complete when the following items are delivered:
 * Appropriate code has been added to base-board.fuegoclass to handle a serial transport option, and the 'serial' transport has been shown to work with a standard board over it's serial console.
 * Supporting programs, such as lrzsz, serio, or custom programs have been added to the Fuego system, and put into the fuego repository, to be installed automatically into the Docker image when it is built.
 * A new test is created (Functional.transport_verify). This test will validate the operation of the transport - in particular it should test whether the 'serial' transport supports correct argument handling (support of wildcards), as well as whether appropriate error handling and reporting (to assist the user in debugging transport problems) is done. This will not be a test of a target feature, but a regression test of the framework itself.
 * Documentation on the fuego wiki has been updated to indicate the presence of the 'serial' transport option.

Details
Here are some detailed steps that would accomplish this task.


 * Add the lrzsz package to DockerFile container build script
 * Detect absence of lrzsz on the target, and put there if necessary, using serio
 * build lrzsz if needed, using the correct PLATFORM for the target board
 * build it as a static executable
 * Make sure serio works to transfer an initial binary to the target
 * add a step in the pre-test phase of the fuego system to perform these operations, and test that they work
 * Add appropriate commands in base-boards.class
 * Fill in the stubs for ov_transport_get/put/cmd
 * Make a test (Functional.transport_verify) to verify that get, put and cmd are working correctly
 * The test should verify use of wildcards in both source and destination, and should check that the behavior of the board's configured TRANSPORT matches the behavior of the 'ssh' transport.
 * (possibly) Add lrzsz package as a test package to Fuego
 * This should allow it to be built for an arbitrary target, using the target SDK installed for Fuego
 * Run the Fuego test suite on a beaglebone, using the serial transport option instead of SSH
 * document the 'serial' transport option in the wiki

alternate methods adb
An alternate method for doing this would be to extend adb to work over a serial console.

ADB already has pull, push, shell and 'shell ', which should accomplish everything needed for the fuego transport sessions.

alternate method direct access
Note that one of the difficulties with piping stuff directly to the console (for ov_transport_cmd) is that you can't tell when the end of the output has been reached. Maybe this could be determined by setting $PS1 (the shell prompt) to something guaranteed to not show up in shell output, like: "fuego-ftc-shell&& $" Then, when looking for the end of the command output, the collector can know it's done when this prompt appears.

The same would be true of cat'ing files from the target for a direct 'get' operation.

Related work

 * see the lrzsz package for Debian and Ubuntu
 * See the lrzsz home page
 * the serio program (see below)
 * LAVA uses ser2net to use telnet over the serial console
 * ser2net runs on target, and provides a telnet session which can be accessed from the host?
 * do you run it on both sides (both host and target)?
 * See the Fuego page on this wiki

Scope
I would expect this to take about 2 weeks, including test time.

Contractor Candidates

 * Lineo Solutions
 * Cogent Embedded

serio tool
You can find this tool on github at: https://github.com/frowand/serio This version has the patches from Andrew Jone's blog entry (see below), plus some additional fixes.

Frank Rowand wrote: (from http://andylinuxblog.blogspot.tw/2014/11/serio-serial-file-transfer-program.html)

Serio: A serial file transfer program without needing z/y/xmodem

In one of my previous posts I talked about using zmodem as a backup/emergency method to transfer files onto an embedded device using the serial console. This technique requires you to have the lrz binary on the target to initiate the transfer. But what happens if you have no way of getting lrz onto the target in the first place? In this post I show you a method of serial transfer requiring no binaries to be on the target, and absolutely no setup on the target-side.

First, check that you have a Python 2.x version installed. If you don't, apt-get or yum install it onto your host OS. Check the version with python --version.

Now get the utility, which is called 'serio'. Get it with hg clone https://code.google.com/p/serio/ Before you go ahead, there are two patches you need to apply (by hand):

Patch 1:

Insert self.last_size = 0 at the location shown, highlighted in green:

if callable(fp.read) and callable(fp.write) and callable(fp.readline) and callable (fp.readlines): self.fp = fp                       self.current = 0 self.showprogress = progress self.last_size = 0

Patch 2:

Get rid of this line, on line 19:

self.s.open

and insert these lines instead:

already_open = self.s.isOpen if not already_open: self.s.open

Update (14 June 2015): I tried this again recently. It seems you may need to use self.serial.isOpen and self.serial.open instead.

That patch above is for people who get the "Port is already open" error. Now you can run the utility:

sudo ./serio -s lrz -d lrz -p /dev/ttyS0

where -s is the source file, -d is the destination, and -p is the serial (console) port. You'll get a nice progress bar.

I recommend putting lrz onto your target (as per my example) so you can do zmodem transfers from here on in.

Help! It transfers, but the executable doesn't work!


 * Check that the echo command on your target supports the "-n" and "-e" options (that's all you need).


 * Try cross-compiling a minimal "hello world" program with your toolchain and try transferring that, as a test. Run it, and if you see "hello world" your settings are probably OK.  See the next point...


 * Open the source code and modify the parameter IO_TIME = .1 to be something greater than .1 (try .3 or .5).


 * In addition to the previous step, try using your toolchain's strip command to reduce the size of the binary.

(end of Andy's blog entry)