TCF survey response

= TCF Testing survey response = TCF survey response provided by Iñaky Perez-Gonzalez and Tim Orling

Survey Questions
 TCF provides support to automate any kind of steps that a human would do otherwise, typing in the console or issuing command line commands.
 * What is the name of your test framework? TCF - Test Case Framework

TCF does the following: 
 * (server side) provides remote access to DUTs (metadata, power switching, debug bridges, serial console i/o, networking tunnels, flashing support)
 * discovers testcases
 * associates testcases with DUT/groups-of-DUTs where it can run.
 * runs each the combination testcase/group-of-DUTs in parallel and evaluate the testcase steps to determine success; report.

TCF is used internally in Intel for testing Zephyr (http://zephyrproject.org) and are expanding to other areas.

Which of the aspects below of the CI loop does your test framework perform?

 Jenkins (or any other CI agent) is used to trigger, checkout code and launch the testcases using the this framework’s command line, which will (as needed) use remote hardware exposed by the daemon. 

Does your test framework:

source code access

 * access source code repositories for the software under test? No, must be checked out by triggering agent (eg: Jenkins)
 * access source code repositories for the test software? No, must be checked out by triggering agent (eg: Jenkins)
 * include the source for the test software? (if meaning the software needed to test features) NO, that is to be provided by the user depending on what they want to test; convenience libraries are provided by the framework.
 * provide interfaces for developers to perform code reviews? NO, left for gerrit, github, etc -- can provide reports to such systems
 * detect that the software under test has a new version? NO, left to triggering agent (eg: Jenkins)
 * if so, how? n/a
 * detect that the test software has a new version? NO, left to triggering agent (eg: Jenkins) or others

test definitions
Does your test system:


 * have a test definition repository? NO, out of scope

 The framework is not tied to any OS or DUT type for that matter (can be used to test a toaster).

Test definitions can have metadata associated (declarative), it being optional but in most cases needed in one form or another for it to be useful. As well, there is an imperative part for actually implementing the test steps.

Natively the framework takes testcases written as a Python class. Loaders can be written to read testcases in whichever format is needed to provide an API for the framework to access the metadata and execute the imperative steps (in whichever format are written).

Test definitions are thus contained, for example: repository that is checked out by a CI agent such as Jenkins) 
 * in files in the file system (most common, in a source-controlled
 * in some other container that is somehow accessible via a driver
 * if so, what data format or language is used (e.g. yaml, json, shell script)

Does your test definition include:
 * source code (or source code location)? Location to python (or other format) test definition in filesystem
 * dependency information? depends on format test interdependency (run X only if Y has passed) not yet supported
 * execution instructions? YES (optional)
 * command line variants? NO (althought they can be sneaked in via environment vars--not recommended to ensure reproduceability)
 * environment variants? YES
 * setup instructions? YES (optional)
 * cleanup instructions? YES (optional)
 * if anything else, please describe:

 Test cases are defined as a sequence of phases that have to occur and expectations that have to be met; they are all optional or as long as needed.

The phases are configuration, building, target acquisition, deployment, evaluation and cleanup. The steps to implement on each phase are implemented as python methods of a subclass of class tcfl.tc.tc_c (if using the native format) or might be implemented by a driver for an specific format (eg: Zephyr's testcase.yaml loader).

Details: 
 * https://intel.github.io/tcf/doc/07-Architecture.html#the-testcase-finder-and-runner
 * https://intel.github.io/tcf/doc/09-api.html#tcf-s-backbone-test-case-finder-and-runner
 * https://intel.github.io/tcf/doc/09-api.html#tcfl.tc.tc_c

Does your test system:
 * provide a set of existing tests? YES examples for reference and self-test
 * if so, how many? ~27 examples, 32 self-tests (combo of TCF tests and py unittests)

build management
Does your test system:
 * build the software under test (e.g. the kernel)? YES if needed
 * build the test software? YES if needed
 * build other software (such as the distro, libraries, firmware)? YES if needed
 * support cross-compilation? YES
 * require a toolchain or build system for the SUT? n/a (depends on what the testcase wants to do)
 * require a toolchain or build system for the test software? n/a (depends on what the testcase wants to do)
 * come with pre-built toolchains? n/a (depends on what the testcase wants to do)
 * store the build artifacts for generated software? TEMPORARILY while needed, then deleted; testcase can decide to archive it somewere else
 * in what format is the build metadata stored (e.g. json)? n/a (testcase specific)
 * are the build artifacts stored as raw files or in a database? n/a (testcase specific)
 * if a database, what database?

Test scheduling/management
Does your test system:
 * check that dependencies are met before a test is run? UNIMPLEMENTED
 * schedule the test for the DUT? YES
 * select an appropriate individual DUT based on SUT or test attributes? YES
 * reserve the DUT? YES
 * release the DUT? YES
 * install the software under test to the DUT? YES (if required by the testcase)
 * install required packages before a test is run? Testcase specific
 * require particular bootloader on the DUT? (e.g. grub, uboot, etc.) NO. Depends on the DUT setup and driver
 * deploy the test program to the DUT? Depends on the testcase, provides facilities to do so
 * prepare the test environment on the DUT? DUT driver specific and testcase specific.  Provides facilities to reset DUTs to well-known state
 * start a monitor (another process to collect data) on the DUT?

 YES/NO

YES: The DUT might be asked to produce output to its output devices if it does not produce it by default already in order to be able to monitor what it does

The intent is the DUT is not modified or altered at all (to try to follow Heisemberg)

E.g.: a kernel redirects output to a serial console, if we need to see that output for anything (like verifying there are no oops during TC execution). 


 * start a monitor on external equipment?

 Testcase specific

Other equipment is treated as other DUT hardware that can be used and manipulated to accomplish whatever the test needs

E.g.: a temperature sensor that prints temperatures (in F) every five seconds to a serial line can be considered a DUT or ‘target’ attached to the server; the server starts a monitoring process to capture the serial output so the client (test runner) can read it.

When the sensor is placed onto a toaster, the following oversimplified test (python):

... toaster.power.off time.sleep(30) data = temperature_sensor.console.read.split(‘\n’) assert all([int(temp) < 80 for temp in data ]) ...

Would verify that if the toaster is off for 30 seconds, the temperature is below 80F all the time and viceversa, when turned on:

... toaster.power.on time.sleep(30) data = temperature_sensor.console.read.split(‘\n’) assert all([int(temp) > 80 for temp in data ]) ... toaster.power.off 


 * initiate the test on the DUT? YES
 * clean up the test environment on the DUT? NO.  Allows post-analysis; left to tests to create a clean well-known-environment before starting

DUT control
Does your test system:
 * store board configuration data? YES
 * in what format? Defined internally in Python dictionaries, REST exported as JSON
 * store external equipment configuration data? YES, all equipment is treated as a DUT
 * in what format?  same
 * power cycle the DUT? YES
 * monitor the power usage during a run? CAN do if proper equipment is attached and test configures and communicates with it
 * gather a kernel trace during a run? CAN do if test monitors proper outputs
 * claim other hardware resources or machines (other than the DUT) for use during a test? YES.  Testcase declares resources (DUTs) needed and runner will claim them all before manipulating them
 * reserve a board for interactive use (ie remove it from automated testing)? NO.  Single reservation system -- automation can claim another one
 * provide a web-based control interface for the lab? NO. cmdline interfaces; web interface doable as a layer on top
 * provide a CLI control interface for the lab? YES

Run artifact handling
Does your test system:
 * store run artifacts NO. left to trigger layer (eg Jenkins)
 * in what format?
 * put the run meta-data in a database? '''YES.  plugin-based reporting mechanism; current plugins available for Jinja2 (text, Junit, HTML...), MongoDB
 * if so, which database?  MongoDB (plugin)
 * parse the test logs for results? test specific
 * convert data from test logs into a unified format? test specific -- test can choose to parse internal logs and produce reporting using the TCF report API for results and KPIs
 * if so, what is the format?  internal format that gets passed to each reporting plugin that is loaded real time; such plugins will store in whatever native format they support
 * evaluate pass criteria for a test? YES.  test specific code will sample the outputs from the DUTs during test executions for expected, unwelcome or non-expected outputs and determine:

There is no run-artifact store in TCF. This is left to the test definition of each test.

Each test runs on its own thread and the calls to the reporting API all get passed to all the report drivers configured and it is up to them to decide how to store or not the information.

A driver might be saving the console output from multiple DUTs used in the testcase in a unified format (so all can be seen at the same time in a cronograph) while another might be capturing the data to postprocess and decide based on a separate process that this testcase run has to be reported to A or B or C for other operations.

 
 * do you have a common set of result names: YES
 * if so, what are they? PASS, ERRR, FAIL, SKIP, BLCK
 * PASS: all went well
 * FAILure: deterministic resolution of the test failing, eg we multiply 2*2 and yields 5 or power measurements from an attached gauge while doing op X yielded a power consumption outside of the expected band.
 * ERRoR: unexpected negative output (for example, a kernel crash while reading a file)
 * SKIP: the DUTs lack the capabilities needed to run such test) and we could only determine that once they were configured, setup and powered up (vs just looking at the metadata)
 * BLoCK: any problem related to infrastructure that disallowed from carrying the test to completion (eg: network failure communicating with the server, DUT power switch failing to execute a power up command, etc...)

 By any means the DUT can provide output and on the test's needs Currently we actively use: Other interfaces can be added depending on target capabilities </b>
 * How is run data collected from the DUT?
 * serial output
 * network output (depending on DUT and its configuration)
 * JTAGs

 interface and DUT specific: </b>
 * e.g. by pushing from the DUT, or pulling from a server?
 * serial line proxied by the test server
 * network port access direct tunneled through the server
 * JTAG (when DUT provides) proxyed through server
 * How is run data collected from external equipment? Same as DUTs; external equipment can be considered another DUT or part of the DUT and the inputs connected to the server equally (eg: serial ports or network ports)
 * Is external equipment data parsed? test specific

User interface
Does your test system:
 * have a visualization system? NO.  left to external reporting interface (eg: TestRails). Currently also feeding output to a Google Sheet

 See: </b>
 * https://docs.google.com/presentation/d/1R9yEEJrQiyGRD2PUkUbYb_HfMKM2vsgAGFWWMCYkY8A/edit#slide=id.g42b5960348_0_634
 * https://docs.google.com/presentation/d/1R9yEEJrQiyGRD2PUkUbYb_HfMKM2vsgAGFWWMCYkY8A/edit#slide=id.g42b5960348_0_727
 * https://docs.google.com/presentation/d/1R9yEEJrQiyGRD2PUkUbYb_HfMKM2vsgAGFWWMCYkY8A/edit#slide=id.g42b5960348_0_645
 * https://docs.google.com/presentation/d/1R9yEEJrQiyGRD2PUkUbYb_HfMKM2vsgAGFWWMCYkY8A/edit#slide=id.g42b5960348_0_716


 * show build artifacts to users? N/A. left to external reporting/triggering interface (eg: Jenkins)
 * show run artifacts to users? N/A. left to external reporting/triggering interface (eg: Jenkins)
 * do you have a common set of result colors? N/A. report tool issue
 * if so, what are they?


 * generate reports for test runs? YES
 * notify users of test results by e-mail? N/A. left to CI integration; report interface can take a plugin to do it
 * can you query (aggregate and filter) the build meta-data? YES
 * can you query (aggregate and filter) the run meta-data? YES
 * what language or data format is used for online results presentation? N/A.  report tool specific.
 * what language or data format is used for reports? (e.g. PDF, excel, etc.) N/A. report tool specific. Current report plugins provide text, junit, mongoDB storage
 * does your test system have a CLI control tool? YES
 * what is it called? tcf

Languages:
What languages or data formats is the user required to learn?
 * what is the base language of your test framework core? python

 </b>
 * python like simple query for target and testcase filtering
 * formats they project uses that are loaded by TCF (eg: Zephyr's testcase.yaml) or Python to implement Testcases in Python
 * Jinja2 templating for reports

Can a user do the following with your test framework:

 * manually request that a test be executed (independent of a CI trigger)? YES
 * see the results of recent tests? YES
 * set the pass criteria for a test? YES
 * set the threshold value for a benchmark test?  YES (if test supports it)
 * set the list of testcase results to ignore?  NO. but they can be postfiltered
 * provide a rating for a test? (e.g. give it 4 stars out of 5) NO
 * customize a test? YES if access to the test repository
 * alter the command line for the test program?  YES
 * alter the environment of the test program?  YES
 * specify to skip a testcase?  YES
 * set a new expected value for a test?  YES (if test permits it)
 * edit the test program source?  YES (if access to the repository)
 * customize the notification criteria? YES (via client config)
 * customize the notification mechanism (eg. e-mail, text)  YES (via client config)
 * generate a custom report for a set of runs? YES. Jinja2 templating
 * save the report parameters to generate the same report in the future? YES.  Jinja2 templating (not fully completed)

Requirements
Does your test framework:


 * require minimum software on the DUT? NO
 * require minimum hardware on the DUT (e.g. memory) NO
 * If so, what?

 DUT could be a toaster for the framework basic needs; tests are expected to declare in which DUTs they can run based on the interfaces those DUTs provide to interact with them and be able to perform the test.

Most basic tests will always need some kind of output collection from targets, so a computing target will likely provide at least a serial port interface to take output from it. </b>
 * require agent software on the DUT? NO
 * If so, what agent?
 * is there optional agent software or libraries for the DUT? NO
 * require external hardware in your labs? DEPENDS On what the tests need to do with the DUTs (power switch interface?  need a remote controlled power switch, etc...)

APIS
Does your test framework:
 * use existing APIs or data formats to interact within itself, or with 3rd-party modules? OPTIONAL. Can provide drivers to load any existing format
 * have a published API for any of its sub-module interactions? YES
 * Please provide a link or links to the APIs? https://intel.github.io/tcf/doc/09-api.html#module-tcfl.tc
 * What is the nature of the APIs you currently use?

Are they:
 * RPCs? YES
 * Unix-style?
 * compiled libraries?
 * interpreter modules or libraries?
 * web-based APIs?
 * something else?

Relationship to other software:

 * what major components does your test framework use?

 CI workflow, Jenkins triggers, checks out SUT and test repositories, configures TCF reporting to MongoDB and launches TCF on the repositories; collects reports and publishes summaries to Google Sheet and Testrails.
 * https://docs.google.com/presentation/d/1R9yEEJrQiyGRD2PUkUbYb_HfMKM2vsgAGFWWMCYkY8A/edit#slide=id.g42db933815_2_0

Gihub/Gerrit (WIP) workflow for pull request verification: Jenkins triggers, checks out SUT and test repositories, determines based on the change what checks need to be run, launches TCF to run them, and reports back to Github/gerrit.

</b>
 * https://docs.google.com/presentation/d/1R9yEEJrQiyGRD2PUkUbYb_HfMKM2vsgAGFWWMCYkY8A/edit#slide=id.g42db933815_2_32


 * does your test framework interoperate with other test frameworks or software? PENDING.  plans discussed to have the runner interfact with other target servers.  Can load testcases from other frameworks based on loader driver.
 * which ones? Eg: Zephyr's sanitycheck

Overview
Please list the major components of your test system.


 * Server: provides remote access to DUTs, exposes whichever interfaces they provide
 * basic interface: metadata, acquisition/release, tunnels,
 * serial consoles
 * debug interfaces
 * power on/off/cycle
 * imaging/flashing

</b>
 * Client:
 * primitives to access DUTs via server
 * testcase runner

Please list your major components here:
 * python
 * Flask

= Additional Data =
 * The TCF software is available at: http://github.com/intel/tcf

Workflows:


 * Developer https://docs.google.com/presentation/d/1R9yEEJrQiyGRD2PUkUbYb_HfMKM2vsgAGFWWMCYkY8A/edit#slide=id.g42b5960348_0_806


 * CI https://docs.google.com/presentation/d/1R9yEEJrQiyGRD2PUkUbYb_HfMKM2vsgAGFWWMCYkY8A/edit#slide=id.g42db933815_2_0


 * Submission (pull request) verification https://docs.google.com/presentation/d/1R9yEEJrQiyGRD2PUkUbYb_HfMKM2vsgAGFWWMCYkY8A/edit#slide=id.g42db933815_2_32