Tbot survey response

= tbot survey response = tbot survey response provided by Heiko Schocher

''' tbot does commandline automatization. So in principle, you can automate all commandline tasks ... tbot does not differentiate between DUT or a lab PC. For example, compiling a linux kernel (which is a testcase) can be run on the DUT (very good RAM test, if you have mounted the rootfs over NFS) or the testcase "build linux" can be executed on a build PC ... '''

Survey Questions

 * What is the name of your test framework? tbot

Does your test framework:

source code access

 * access source code repositories for the software under test? Possible, yes.
 * access source code repositories for the test software? Not needed, as the testcases are in tbot. But I can think of writing a testcase, which clone more tests from an url ...
 * include the source for the test software? Yes.
 * provide interfaces for developers to perform code reviews? No.
 * detect that the software under test has a new version? No
 * if so, how? (e.g. polling a repository, a git hook, scanning a mail list, etc.)
 * detect that the test software has a new version? no. tbot is a command line tool... the easiest CI is starting a cron job.

test definitions
Does your test system:
 * have a test definition repository? Yes
 * if so, what data format or language is used (e.g. yaml, json, shell script) The testcases are written in python.

Does your test definition include:
 * source code (or source code location)? [1] https://github.com/hsdenx/tbot/tree/master/src/tc
 * dependency information? no. Hmm... in tbot you can call testcases from testcases, so maybe this could be interpreted as dependency information.
 * execution instructions? no.
 * command line variants? not in tbot itself, but possible in testcases
 * environment variants? not in tbot itself, but possible in testcases
 * setup instructions? not in tbot itself, but possible in testcases
 * cleanup instructions? not in tbot itself, but possible in testcases
 * if anything else, please describe: Maybe this fits into "execution instructions", I am unsure.

'''Tbot contains testcases for dumping register content from "start addr" to "end address" (under linux with devmem2 in U-Boot with md command or for the BDI debugger with "md" command) into a file. For example a resulting pinmux file for the BBB:'''

https://github.com/hsdenx/tbot/blob/master/src/files/bbb/am335x_pinmux.reg

'''This file can than be used in another testcase, which reads the registers address from the file, reads the current value from this address on the DUT and compare it with the value in the file. So, if you booting a new linux kernel or U-Boot, you can check all register values, you are interested in, and be sure, your new kernel has the same values as your reference.'''

The file for this would be stored on the machine where you start tbot, as tbot opens this file. You pass a simple filepath to the testcase, where to find this file.

Currently config data between testcases is passed as global variables in "tb.config" ... When I started with tbot, I had in mind to get rid of my shell scripts and a way to combine them.

So, the testcase, which use this file, simply has the path, where to find this file in an variable ... and the testcase caller sets this variable. If you have several files simply make a loop and call the testcase...

Does your test system:
 * provide a set of existing tests?Yes


 * if so, how many? 302 in mainline

hs at xmglap:tbot [master] $ find src/tc/ -type f | wc -l 302

'''There are more from customers, easy to add. Just create new subdirectories in [1]. Name of testcase = filename, filename must be unique.'''

build management
Does your test system: There are testcases, which checkout for example yocto, bitbake it, and install for example the new sd card image on the DUT, check if the new image boots, execute commandline tests on the new booted image (incl. check if the new image is really the new just created image through the /etc/timestamp yocto generates).
 * build the software under test (e.g. the kernel)? yes
 * build the test software? Hmm... tbot itself does not need to build the testcases. But you can write testcases, which build for example a c file ... put it to the target in someway, execute the binary ... and so on ...
 * build other software (such as the distro, libraries, firmware)?
 * support cross-compilation? not task of tbot, but yes. I use tbot for weekly U-Boot mainline tests (incl. automating my maintainers work). See an old video for this tasks on youtube (no sound): https://www.youtube.com/watch?v=zfjpj3DLsx4&feature=youtu.be


 * require a toolchain or build system for the SUT? not required. If you have a way for building the images you need for your DUT, than use tbot only for installing the new images on the DUT and test the new image.
 * require a toolchain or build system for the test software? for tbot you need python 2.7 and python module paramiko must be installed on the device, where you start tbot.You do not need a big workstation, a raspberry pi for example is enough. 
 * come with pre-built toolchains? no
 * store the build artifacts for generated software? possible, if you write a testcases for it.
 * in what format is the build metadata stored (e.g. json)? currently for U-Boot simple copy raw files in a subdirectory if all testcases finished successfully. For Linux / Yocto, the answer is: not yet (at last in mainline testcases)
 * are the build artifacts stored as raw files or in a database? In principle all is possible, if you have a command for this task on the commandline ... write a testcase for it.
 * if a database, what database?

Test scheduling/management
Does your test system:
 * check that dependencies are met before a test is run? If the testcase writer implement such a test, yes. For example, I started to generate an automated generated U-Boot commandline documentation with tbot, there are testcases, which check, if a certain U-Boot command exist. If yes, test it. But this is the task of the testcase writer ...
 * schedule the test for the DUT? no.
 * select an appropriate individual DUT based on SUT or test attributes? no. tbot get passed through commandline arguments, what DUT in which laboratory. Then tbot tries to get the DUT there, if this is not possible end with failure.
 * reserve the DUT? You should have a possibility on the commandline to reserve the DUT, than write a testcase for it.
 * release the DUT? See above
 * install the software under test to the DUT? yes
 * install required packages before a test is run? yes, possible
 * require particular bootloader on the DUT? (e.g. grub, uboot, etc.) no, just a commandline. For example, if your debugger has a commandline you can automate it with tbot.
 * deploy the test program to the DUT?

'''if your testcase needs a commmandline tool, which is not already in the DUT rootfs, yes possible. But the testcase writer is responsible for it, to check this. For example, if you use tbot for testing the device before delivering to a customer, you can write a testcase, which installs all the tools you need on the DUT, than execute all your testcases, at the end delete all your previous installed tools.'''


 * prepare the test environment on the DUT? see above.
 * start a monitor (another process to collect data) on the DUT? You can open as much connections you want in tbot, and execute testcases on this connections, so yes, possible.

For example a CAN bus test:

https://www.youtube.com/watch?v=hl7gI4b9CG8&feature=youtu.be

'''There a connection to a PC where a PEAK can adapter is installed is opened. tbot connects to this PC and now you can write in your testcase commands to this connection and commands on your DUT ...'''


 * start a monitor on external equipment? if this monitor has a commandline, yes.
 * initiate the test on the DUT? yes
 * clean up the test environment on the DUT? yes ... Hmm.. there is no generic way in tbot for all this (yet), but it is possible to do.

DUT control
Does your test system:
 * store board configuration data? yes
 * in what format? in a python file, examples: https://github.com/hsdenx/tbot/tree/master/config
 * store external equipment configuration data? possible
 * in what format? add it for example in https://github.com/hsdenx/tbot/tree/master/src/files and add the files to git ...
 * power cycle the DUT? yes
 * monitor the power usage during a run? no
 * gather a kernel trace during a run? All output on the console is logged into a log file.
 * claim other hardware resources or machines (other than the DUT) for use during a test? yes, if this is doable through a command on a commandline.
 * reserve a board for interactive use (ie remove it from automated testing)? no
 * provide a web-based control interface for the lab? no
 * provide a CLI control interface for the lab? yes, as controlling the lab are also testcases.

Run artifact handling
Does your test system:
 * store run artifacts? no, tbot itself does not do this. But ... There are testcases, which generate with the help of gnuplot images (for example analyse top or latency output). All files which tbot generates are at the end in a results directory.

tbot collects the log output from each testcase and saves it in a big logfile in the results directory. There is no need for each testcase to store it. As a testcase may have variables, also the variables with which the testcase is called are stored in the logfile. If a testcase generates files, yes, there is a "results" directory, where they are stored.

A config variables is provided which contins the path to the results directory.


 * in what format? txt, jpg, pdf, html
 * put the run meta-data in a database? Not easy to answer... tbot itself no. But ...

'''You can write Event-Backends (name is not perfect). tbot calls this backends at end of execution. In a backend, you can convert the collected information tbot collected while running (so called events ... not a really good name) into whatever format you want. An event can be for example:
 * start/end of testcase
 * all characters received/sended on the connections which tbot opens
 * Testcases can generate Events (for example, if a U-Boot version is detected the testcase generate an U-Boot Version Event.

'''For example, there is a backend for a MySQL database, which stores some information the user is interested in into a MySQL database. Another backend creates a statistic image with gnuplot, another a dependency graph with dot tool, or a jenkins backend, which creates a junit.xml for jenkins ...'''

see: http://www.tbot.tools/event.html


 * if so, which database?
 * parse the test logs for results? I do not know, what you mean here. In tbot you start a command on a connection, and your testcase parses the output of the command and decides, if it is good or bad, so yes.
 * convert data from test logs into a unified format? yes, see above.
 * if so, what is the format? http://www.tbot.tools/event.html#demos Others are (hopefully easy) possible.
 * evaluate pass criteria for a test (e.g. ignored results, counts or thresholds)? no. A testcase always comes back with True or False.
 * do you have a common set of result names: (e.g. pass, fail, skip, etc.) No names. True/False only
 * if so, what are they?
 * How is run data collected from the DUT? 'only "console" output is logged. But again ... if you need more, write a testcase for it ... You can start for example tbot instances in parallel. For example:
 * start a testcases over a serial console on the DUT (some time-intensive benchmark).
 * start a second tbot instance, which log into the DUT over ssh and start on this connection a testcase, which monitors the output of the top command and generate at the end a graph of top values you are interested in with gnuplot.


 * e.g. by pushing from the DUT, or pulling from a server?
 * How is run data collected from external equipment? not used yet, but possible
 * Is external equipment data parsed? no. It seems I have for each topic the same answer in my mind ... (write a testcase for it ...)

User interface
Does your test system:
 * have a visualization system? no
 * show build artifacts to users? no
 * show run artifacts to users? no
 * do you have a common set of result colors? no
 * if so, what are they?
 * generate reports for test runs? yes
 * notify users of test results by e-mail?

''' no .. heh, but I thought about sending automated EMails, if tbot found, that a new patch in my patchwork ToDo list does not apply to mainline U-Boot or is not checkpatch clean ...

But I think, this should be a human task ;-) '''

etc.)
 * can you query (aggregate and filter) the build meta-data? no
 * can you query (aggregate and filter) the run meta-data? no
 * what language or data format is used for online results presentation? (e.g. HTML, Javascript, xml,

''' tbot itself does not do this, it is a commandline tool, but it can fill a MySQL database with results ... or deliver results to jenkins, buildbot.'''


 * what language or data format is used for reports? (e.g. PDF, excel, etc.) simple txt log file, if no backend is enabled.
 * does your test system have a CLI control tool? yes
 * what is it called? tbot

Languages:
Examples: json, python, yaml, C, javascript, etc. python
 * what is the base language of your test framework core?

What languages or data formats is the user required to learn? (as opposed to those used internally) python and of course, as tbot does commandline automation the user must know the commands on the commandline.

Can a user do the following with your test framework:

 * manually request that a test be executed (independent of a CI trigger)? yes
 * see the results of recent tests? while running, yes with verbose option.
 * set the pass criteria for a test? Heh... never thought about that, but yes! You can pass through a commandline paramter values of variables used in testcases ... so yes, that could be done.
 * set the threshold value for a benchmark test? see above, yes.
 * set the list of testcase results to ignore? no
 * provide a rating for a test? (e.g. give it 4 stars out of 5) no
 * customize a test?
 * alter the command line for the test program? possible if the testcase uses variables
 * alter the environment of the test program? no
 * specify to skip a testcase? no
 * set a new expected value for a test? through variables passing from command line ... yes, possible.
 * edit the test program source? Hmm.. the testcase is the testprogram ...
 * customize the notification criteria? no
 * customize the notification mechanism (eg. e-mail, text)
 * generate a custom report for a set of runs? no
 * save the report parameters to generate the same report in the future? '''no...

'''Hmm... the hacker in me says yes ... the events tbot collects contains all data ... and they are stored in background into a file ... and you can save this file and put it into tbot with the option "-e" ... so it should generate the same info again ...'''

Requirements
Does your test framework:
 * require minimum software on the DUT? depends on the testcases you want to execute!

'''If you have no software on the DUT, you can call (only BDI yet) testcases, which install with a debugger Software into the DUT. Or for example on imx based devices call:'''

https://github.com/hsdenx/tbot/blob/master/src/tc/linux/tc_linux_imx_usb_loader_boot_uboot.py

for using imx_usb_loader for breathing live into a device ...

But as tbot does commandline automatization, you need at least some sort of a commandline.

etc.) depends on the testcases you want to execute!
 * require minimum hardware on the DUT (e.g. memory) depends on the testcases you want to execute
 * If so, what? (e.g. POSIX shell or some other interpreter, specific libraries, command line tools,
 * require agent software on the DUT? (e.g. extra software besides production software) no
 * If so, what agent?
 * is there optional agent software or libraries for the DUT? no
 * require external hardware in your labs? Yes. You need at least a device to which the DUT is connected. This device is called "lab PC". This can be for example a raspberry pi.

You can of course attach more than one DUT to one lab PC.

tbot connects to this lab PC over ssh (but tbot can of course also run on this lab PC). The lab PC must have a connection to the DUT for the console and it must be possible to power on/off the DUT from the lab PC. Therefore tbot always open 2 connections over ssh to the labPC This is the reason, why tbot uses the paramiko module. You can open as many connections on one ssh connection with paramiko.

This is required for every test. Each test in tbot can call n testcases, where n could be a large number.

Both tasks are from tbots view just calling a testcase!

So, you need to know for your DUT on the lab PCs commandline how to


 * power on/off the DUT:
 * -> write a tbot testcase for it
 * -> use this testcase for powering on/off the DUT. You can config in the lab config file, which testcase is needed for the DUT.


 * Connect to console:
 * -> write a testcase for it and use this testcase for connecting to the DUT. Configure this also in the lab config file.

'''Remark: This must not be necessarily a serial line. Of course, if your bootloader needs serial line and you want to do bootloader tests ... you need a serial line ... but you can use for example ssh testcase for connecting to the DUT.'''

'''Also a more complicated setup (hardware reasons) I have, is possible. There, first I have to ssh from the lab PC to another PC and then start there kermit ... yes, write a testcase for this scenario and use it as "console testcase".'''

More explanation about relationship between host, labPC and DUT:

tbot only runs on the host. It opens an ssh connection to the labPC, which communicates with and controls the DUT.

Now tbot open connections (like streams where you can read/write) to the labPC within the ssh connection. By default tbot opens 2 connections.

On the first it always get the console of the DUT, the second is for controlling tasks like powering on/off the DUT.

Now the tricky part ... if you have for example a linux testcase, say copy file. You pass now this testcase only the connection handle. The testcase itself does not know if it runs on the DUT or on the labPC. So you can use this "copy testcase" on the DUT and on the labPC.

The host only connects to the DUT via the labPC.

APIS
Does your test framework:
 * use existing APIs or data formats to interact within itself, or with 3rd-party modules? no
 * have a published API for any of its sub-module interactions (any of the lines in the diagram)? no
 * Please provide a link or links to the APIs?

Sorry - this is kind of open-ended...

'''... Sorry for my long answers ... they tend to get longer and longer ...'''

Are they:
 * What is the nature of the APIs you currently use?
 * RPCs? no
 * Unix-style? (command line invocation, while grabbing sub-tool output) yes
 * compiled libraries? no
 * interpreter modules or libraries? no
 * web-based APIs? no
 * something else? no

Relationship to other software:

 * what major components does your test framework use (e.g. Jenkins, Mondo DB, Squad, Lava, etc.) none
 * does your test framework interoperate with other test frameworks or software? tbot can start other testframeworks if they have a commandline interface.
 * which ones?

''' until they are commandline based, tbot simply can call them. For example, tbot can call the pytest based framework integrated into U-Boot:

http://git.denx.de/?p=u-boot.git;a=tree;f=test;h=7294ad8a90145a61787d251dcec741ca3a239eb2;hb=9dc8d155d4e88563f572ee79aab758eb4272f3fd

Find an output of it here:

http://xeidos.ddns.net/tbot/id_915/test-log.html

This test was started from tbot, http://xeidos.ddns.net/tests/test_db_auslesen.php#915

and the output stored through the database backend from tbot in a mySQL database... which ends in this html page.

Also I tried to integrate:

https://wiki.yoctoproject.org/wiki/Image_tests

as it is possible to run them from cmdline ... but I failed with:

https://lists.yoctoproject.org/pipermail/yocto/2018-January/039580.html

never got an answer ... so I did not yet finished this ... seems it is time to try again.

So, if other testframeworks can be started from commandline, it is easy to write a tbot testcase for them. Think about you have some testscripts ... simply write a testcase in tbot for them (may the testcase installs them on the DUT and removes them after the test) and start them... '''

Overview
Please list your major components here:

''' Not easy ... '''
 * tbot itself - open connections to lab PC on which you can start testcases
 * backends - convert collected information into formats user is interested in

= Additional Data = A bit more on tbot architecture:

Tim Bird said: > It kind of strikes me as the 'forth' of test systems, where larger functionality > is built out of user-defined smaller units. Let me know if I've misunderstood this. Heiko responded:

Yes, that is the idea.

In tbot are a lot of small testcases, for example generating a randomfile of n bytes, copying a file, tftp a file into ram in U-Boot, making a md5sum ... *all* is a testcase.

Now you can write a testcase which calls other testcases, for example the above mentioned, and than you can flash the image say into a ubi volume (which is also a testcase). Read back the image from the UBI volume (yes, also just a testcase), compare the 2 images ...

Now you have a testcase for testing a UBI Volume... you can call this new testcase now again from a new one, which makes a loop, which calls the ubi volume testcase with different image sizes ... or with other ubi volume names ...

Something like a jigsaw puzzle ...

You end up in a testcase you start with tbot, which will start a lot of testcases ... here a graph from a git bisect:

http://xeidos.ddns.net/tbot/id_127/graph.png

For more information, see http://www.tbot.tools/principles.html