Jenkins-based Test Automation

Jenkins-based Test Automation (JTA) is the official test framework for the LTSI project.

Please see the official page for this LTSI test project at:
 * http://ltsi.linuxfoundation.org/ltsi-test-project

Code for the test framework can be downloaded from:
 * https://bitbucket.org/cogentembedded/jta-public/

Fujitsu used the completed test framework, and reported their findings here:
 * http://jp.fujitsu.com/group/fct/downloads/v4/events/exhibit/practice-ltsi-test-framework-introduction-of-ethtool-test-set.pdf

Call to action
Please download and use this framework, and send feedback on it to the LTSI mailing list at:
 * https://lists.linuxfoundation.org/mailman/listinfo/ltsi-dev

Vision
The purpose of JTA is to provide a test framework for testing embedded Linux, that is distributed and allows individuals and organizations to easily run their own tests, and at the same time allows people to share their tests with each other.

Historically, test frameworks for embedded Linux have been difficult to set up, and difficult to extend. In cases where a test program was reasonably self-contained, or test system was not very difficult to extend, the system was not easily applied in cross or embedded environments. Some very full frameworks are either not viewed as processor-neutral (LAVA), and are difficult to set up (LAVA), or are targeted at running tests on a dedicated group of boards or devices (http://kernelci.org/).

The vision of open source in general is one of sharing source code and capabilities, to expand the benefits to all participants in the ecosystem. The best way to achieve this is to have mechanisms to easily use the system, and easily share enhancements to the system, so that all participants can use and build on each others efforts.

The goal of JTA is to provide a framework that any group can install and use themselves, while supporting important features like cross-compilation, host/target test execution, easy test administration. Test administration consists of starting tests (both manually and automatically), viewing test results, and detecting regressions. Ease of use is critical, to allow testers to use tests that are otherwise difficult to individually set up, configure, and interpret the results from. It is also important to make it very easy to share tests (scripts, configuration, results parsing, and regression detection methods).

Some secondary goals of this project are the ability for 3rd parties to initiate or schedule tests on our hardware, and the ability to share our test results with others.

The use of Jenkins as the core of the test framework already supports many of the primary and secondary goals. The purpose of this project is to augment the Jenkins system to support embedded configurations of Linux, and to provide a place for centralized sharing of test configurations and collateral.

Actions Items/Proposed extensions/To-Do list
Here is a list of things that need to be fixed or improved about the current system:

JTA To Do List