Jenkins-based Test Automation

Jenkins-based Test Automation (JTA) is the official test framework for the LTSI project.

Please see the official page for this LTSI test project at:
 * http://ltsi.linuxfoundation.org/ltsi-test-project

Code for the test framework can be downloaded from:
 * https://bitbucket.org/cogentembedded/jta-public/

Fujitsu used the completed test framework, and reported their findings here:
 * http://jp.fujitsu.com/group/fct/downloads/v4/events/exhibit/practice-ltsi-test-framework-introduction-of-ethtool-test-set.pdf

Call to action
Please download and use this framework, and send feedback on it to the LTSI mailing list at:
 * https://lists.linuxfoundation.org/mailman/listinfo/ltsi-dev

Vision
The purpose of JTA is to provide a test framework for testing embedded Linux, that is distributed and allows individuals and organizations to easily run their own tests, and at the same time allows people to share their tests with each other.

Historically, test frameworks for embedded Linux have been difficult to set up, and difficult to extend. In cases where a test program was reasonably self-contained, or test system was not very difficult to extend, the system was not easily applied in cross or embedded environments. Some very full frameworks are either not viewed as processor-neutral (LAVA), and are difficult to set up (LAVA), or are targeted at running tests on a dedicated group of boards or devices (http://kernelci.org/).

The vision of open source in general is one of sharing source code and capabilities, to expand the benefits to all participants in the ecosystem. The best way to achieve this is to have mechanisms to easily use the system, and easily share enhancements to the system, so that all participants can use and build on each others efforts.

The goal of JTA is to provide a framework that any group can install and use themselves, while supporting important features like cross-compilation, host/target test execution, easy test administration. Test administration consists of starting tests (both manually and automatically), viewing test results, and detecting regressions. Ease of use is critical, to allow testers to use tests that are otherwise difficult to individually set up, configure, and interpret the results from. It is also important to make it very easy to share tests (scripts, configuration, results parsing, and regression detection methods).

Some secondary goals of this project are the ability for 3rd parties to initiate or schedule tests on our hardware, and the ability to share our test results with others.

The use of Jenkins as the core of the test framework already supports many of the primary and secondary goals. The purpose of this project is to augment the Jenkins system to support embedded configurations of Linux, and to provide a place for centralized sharing of test configurations and collateral.

Actions Items/Proposed extensions/To-Do list
Here is a list of things that need to be fixed or improved about the current system: (Maybe make this a table if it gets big enough and we need more structure - but use a list for now)


 * The guide is pretty sad - it needs a lot more detail
 * what does it need? see JTA guide notes
 * Need an ubuntu install script - Tim is working on it
 * Use existing Jenkins installation/package, if present
 * need a new name - JTA is OK, but how about:
 * JELTS - Jenkins-based Embedded Linux Test System
 * FELT - Framework for Embedded Linux Testing ("It's smooth")
 * Fuego - no meaning, just a cool name (reference to Tierra Del Fuego, a penguin habitat) (spanish for "fire")
 * It looks like there's already a "fuego" package for linux (C++ libraries for 'Go' programs??)
 * testbook - combination of "test" and "facebook" - already used in usb-testbook - see http://testbook.net/usb-trial.html
 * testbox - combination of "test" and "box" (which is popular for embedded linux tools - busybox, toybox, etc.)
 * add support for serial interface to target boards
 * can execute easily enough, but need get and put files over serial console - look at ymodem?
 * add support for adb interface to target boards
 * this should be trivial for the .board file - but possibly less so for the distro file?)

Bugs?

 * when creating a new board, the environment variable "DISTR" was set to distribs/nologger.dist, which caused a missing "DISTRIB" message
 * it appears that this environment variable name should be DISTRIB
 * in the file /home/jenkins/tests/Benchmark.bc/bc-scripts/bc-device.sh, it sets BC_EXPR1=$1 and also!! BC_EXPR2=$1
 * Shouldn't that be BC_EXPR2=$2 ??
 * in the file /home/jenkins/tests/Benchmark.bc/bc-script.sh, the function test_run has the line "report "cd $JTA_HOME/jta.$TESTDIR; ./bc-device.sh $BENCHMARK_BC_EXPR1 $BENCHMARK_BC_EXPR1"
 * Shouldn't that second parameter be $BENCHMARK_BC_EXPR2 (!!)
 * in my log I was seeing both arguments written at "2+2", although I was using the default bc spec.
 * the sample specs for the BC example seem to have poor names
 * instead of the specs being named bc_exp1 and bc_exp2, they should be bc_mult and bc_add, to describe what they do.
 * this is expecially true since exp1 and exp2 are used inside the script themselves to indicate the first and second expressions to evaluate
 * if you click on "graph" to see the results of the test, then go "back" in the browser, it re-runs the test again.

Artemi's items
Here is a list that Artemi sent me in October, 2015
 * Tim's hardware lab... integration/support of Sony debug board
 * Consider synchronization/update to align with latest versions of Jenkins
 * Update LTP test suite
 * Consider update/synchronize with latest public versions of other test suites (e.g. iperf/netperf and other benchmarks)
 * Split into separate repositories and make tests separate from the framework
 * Consider Docker-based installation (to avoid - "failed to install" setup issues)
 * Qemu target support
 * Consider adding more elegant/straightforward way of running separate test on target, command-line interface