Difference between revisions of "JTA To Do List"

From eLinux.org
Jump to: navigation, search
(move resolved, and resolve the DISTRIB bug)
(correct headings)
Line 53: Line 53:
 
** It should happen after a build.  some of the pre_test stuff could go stale (like dropping caches) during the build on the host, which could be quite long.
 
** It should happen after a build.  some of the pre_test stuff could go stale (like dropping caches) during the build on the host, which could be quite long.
  
=== Bugs? ===
+
== Bugs? ==
  
 
* in the file /home/jenkins/tests/Benchmark.bc/bc-scripts/bc-device.sh, it sets BC_EXPR1=$1 and also!! BC_EXPR2=$1
 
* in the file /home/jenkins/tests/Benchmark.bc/bc-scripts/bc-device.sh, it sets BC_EXPR1=$1 and also!! BC_EXPR2=$1
Line 137: Line 137:
  
  
=== Artemi's items ===
+
== Artemi's items ==
 
Here is a list that Artemi sent me in October, 2015
 
Here is a list that Artemi sent me in October, 2015
 
* Tim's hardware lab... integration/support of Sony debug board
 
* Tim's hardware lab... integration/support of Sony debug board

Revision as of 17:21, 3 March 2016

This is a list of items to be done to improve Jenkins-based_Test_Automation (JTA).

Note: Maybe make this a table if it gets big enough and we need more structure - but use a list for now

Tim's ideas

  • improve the guide
    • The guide is pretty sad - it needs a lot more detail
      • what does it need? see JTA guide notes
        • the guide needs some kind of "hello test" instructions
        • the guide needs detailed instructions for how to integrate yocto and buildroot sdks

  • get a new name - JTA is OK, but how about:
    • JELTS - Jenkins-based Embedded Linux Test System
    • FELT - Framework for Embedded Linux Testing ("It's smooth")
    • Fuego - no meaning, just a cool name (reference to Tierra Del Fuego, a penguin habitat) (spanish for "fire")
      • This is my favorite.
      • It looks like there's already a "fuego" package for linux (C++ libraries for 'Go' programs??)
    • testbook - combination of "test" and "facebook" - already used in usb-testbook - see http://testbook.net/usb-trial.html
    • testbox - combination of "test" and "box" (which is popular for embedded linux tools - busybox, toybox, etc.)

  • add support for serial interface to target boards
    • can execute easily enough, but need get and put files over serial console - look at ymodem?

  • add support for adb interface to target boards
    • this should be trivial for the .board file - but possibly less so for the distro file?)

  • convert documentation to asciidoc
  • pre-install test binaries, for multiple architectures
    • this should lower the barrier to entry
    • there's no need to install toolchains (toolchains are needed for new tests only)
      • there's no need to rebuild the tests constantly

  • test using ttc
    • add target_test as a test
  • write hello test tutorial for qemuarm
  • write hello test tutorial for qemux86 (this is even easier??)

  • submit talk proposal for ELC
  • submit BOF proposal for ELC

  • get rid of extraneous jenkins-isms in the interface
    • eg. "Build a Maven 2/3 project" on the New Job page. - what does this mean?, do I care? It's needless junk in our interface.

  • add test dependencies, and run a sanity-checker before tests (to avoid false negatives)
    • check for kernel config, target configuration features (ability to change kernel, ability to reboot), etc.

  • respond on mailing list within 24 hours.
    • response times from cogent are very bad (over a week, and sometimes not at all)

  • move the project to github? (I don't like the bitbucket hosting)

  • pre_test happens before test_build.
    • It should happen after a build. some of the pre_test stuff could go stale (like dropping caches) during the build on the host, which could be quite long.

Bugs?

  • in the file /home/jenkins/tests/Benchmark.bc/bc-scripts/bc-device.sh, it sets BC_EXPR1=$1 and also!! BC_EXPR2=$1
    • Shouldn't that be BC_EXPR2=$2 ??

  • in the file /home/jenkins/tests/Benchmark.bc/bc-script.sh, the function test_run has the line "report "cd $JTA_HOME/jta.$TESTDIR; ./bc-device.sh $BENCHMARK_BC_EXPR1 $BENCHMARK_BC_EXPR1"
    • Shouldn't that second parameter be $BENCHMARK_BC_EXPR2 (!!)
    • in my log I was seeing both arguments written at "2+2", although I was using the default bc spec.

  • the sample specs for the BC example seem to have poor names
    • instead of the specs being named bc_exp1 and bc_exp2, they should be bc_mult and bc_add, to describe what they do.
    • this is expecially true since exp1 and exp2 are used inside the script themselves to indicate the first and second expressions to evaluate

  • if you click on "graph" to see the results of the test, then go "back" in the browser, it re-runs the test again.

  • When working with beaglebone, I had a test failure with the error
+ sshpass -e ssh -o ServerAliveInterval=30 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectTimeout=15 -p root@10.0.1.33 'mkdir -p /tmp/jta.Benchmark.bc && cd /tmp/jta.Benchmark.bc && cat /var/log/messages > bbb.2016-01-26_18-07-01.6.after'
Bad port 'root@10.0.1.33'
+ abort_job 'Error while ROOTFS_LOGREAD command execution on target'
    • The error comes from a mis-configured SSH_PORT (if empty, the -p parameter uses the value from the next field, which is the

user name and ip address)

    • Should not use -p in sshpass command line if SSH_PORT is empty

  • I tried to enter a periodic line of "/5 * * * *" (without the quotes), and the system gave me an error:
    • Invalid input: "/5 * * * *": line 1:1: unexpected token: /
    • this should be an allowed syntax according to the online help
  • I had to modify the test to cause it to be run on target 'bbb' as a periodic test.
    • I set the periodic line to "5,10,15,20,25,30,35,40,45,50,55 * * * *"
    • When the test ran, it ran on target "template-dev"
    • In order to get it to run on "bbb", I modifed the parameters
      • I renamed "Device" to "Device_queried" - to remove the query operation
        • This was a Dynamic Choice Parameter
      • I added a new text Parameter, called "Device", with value "bbb" and description "target".
    • It now ran the test every 5 minutes on target bbb

  • the time and date in the docker container is UTC
    • I have had the docker container running for at least overnight
    • the current time on my desktop (and the real wall time) is: Tue Jan 26 15:37:58 PST 2016
    • the current time in the docker container is: Tue Jan 26 23:38:16 UTC 2016
    • This is throwing off my periodic tests, which I want to run in my time zone

  • problem running jta container, if user runs another container
    • do "docker run hello-world", then jta-host-scripts/docker-start-container.sh, and it will run the hello-world container
    • this script runs the latest container, instead of the jta container (it uses 'docker ps -l -q' to get the container id
    • some other method should be used to set the CONTAINER_ID

  • editing a target configuration and saving it breaks the link between /var/lib/jenkins/config.xml and /userdata/conf/config.xml
    • before editing the target configuration:
root@tlinux:/home/jenkins# ls -lad /var/lib/jenkins/config.xml
lrwxrwxrwx 1 jenkins root 25 Jan 11 18:09 /var/lib/jenkins/config.xml -> /userdata/conf/config.xml
    • after editing the target configuration
root@tlinux:/home/jenkins# ls -lad /var/lib/jenkins/config.xml
-rw-r--r-- 1 jenkins nogroup 13509 Jan 27 14:51 /var/lib/jenkins/config.xml
    • after this, changes made in the host system to jta-public/userdata/conf/config.xml have no effect, and changes made in the jenkins interface are not reflected in jta-public/userdata/conf/config.xml

  • Benchmark.lmbench2 fails to build - it is missing the 'cr' command, but this appears to be an artifact of the AR variable not being defined.
...
arm-linux-gnueabihf-gcc -O -DRUSAGE -DHAVE_uint=1 -DHAVE_int64_t=1 -DHAVE_DRAND48 -DHAVE_SCHED_SETAFFINITY=1   -c lib_sched.c -o ../bin/arm-linux-gnueabihf/lib_sched.o
/bin/rm -f ../bin/arm-linux-gnueabihf/lmbench.a
cr ../bin/arm-linux-gnueabihf/lmbench.a ../bin/arm-linux-gnueabihf/lib_tcp.o ../bin/arm-linux-gnueabihf/lib_udp.o ../bin/arm-linux-gnueabihf/lib_unix.o ../bin/arm-linux-gnueabihf/lib_timing.o ../bin/arm-linux-gnueabihf/lib_mem.o ../bin/arm-linux-gnueabihf/lib_stats.o ../bin/arm-linux-gnueabihf/lib_debug.o ../bin/arm-linux-gnueabihf/getopt.o ../bin/arm-linux-gnueabihf/lib_sched.o
make[2]: cr: Command not found
Makefile:237: recipe for target '../bin/arm-linux-gnueabihf/lmbench.a' failed
make[2]: *** [../bin/arm-linux-gnueabihf/lmbench.a] Error 127
make[2]: Leaving directory '/userdata/buildzone/Benchmark.lmbench2-qemu-armv7hf/src'
Makefile:114: recipe for target 'lmbench' failed
make[1]: *** [lmbench] Error 2
make[1]: Leaving directory '/userdata/buildzone/Benchmark.lmbench2-qemu-armv7hf/src'
Makefile:20: recipe for target 'build' failed
make: *** [build] Error 2
+++ build_error 'error while building test'
+++ touch build_failed
+++ abort_job 'Build failed: error while building test'
+++ set +x

*** ABORTED ***

JTA error reason: Build failed: error while building test
    • the problem is a missing AR definition in tools.sh for the arm-linux-gnueabihf platform


Artemi's items

Here is a list that Artemi sent me in October, 2015

  • Tim's hardware lab... integration/support of Sony debug board
  • Consider synchronization/update to align with latest versions of Jenkins
  • Update LTP test suite
  • Consider update/synchronize with latest public versions of other test suites (e.g. iperf/netperf and other benchmarks)
  • Split into separate repositories and make tests separate from the framework
  • Consider Docker-based installation (to avoid - "failed to install" setup issues)
  • Qemu target support
  • Consider adding more elegant/straightforward way of running separate test on target, command-line interface

Resolved

  • Benchmark.uptime doesn't show the plot on the test page (it says "NO DATA")
    • I've done something wrong, but I don't know what it is. It's config.xml is pretty close to that of Benchmark.Dhrystone, which works.
    • this was my error. I didn't modify /home/jenkins/logs/tests.info with a line for my benchmark

  • when creating a new board, the environment variable "DISTR" was set to distribs/nologger.dist, which caused a missing "DISTRIB" message
    • it appears that this environment variable name should be DISTRIB
    • in the release I'm testing in January 2016 (jta-core last commit Nov 30 2015, and jta-public last commit Nov 30 2015)
    • I was making userdata/conf/boards/bbb.board (copied from template-dev.board) and neither has either DISTR or DISTRIB.
    • adding DISTRIB="distribs/nologger.dist" to bbb.board still resulted in the error
      • This doesn't go in the board file, it goes in the jenkins target configuration (/userdata/conf/config.xml)
      • need to send a patch for template-dev in this file (does it come from jta-core or jta-public?)