Test meta-data survey

Here is information about the schema (field names, and values formats) used by various build systems for automated testing:

Fuego
Here are the fields with meta-data about a test and it's build and execution environment, for Fuego.


 * format=json, env output, shell script (NAME=value pairs)
 * file=run.json, prolog.sh, machine-snapshot.txt
 * fields (from run.json):
 * duration_ms: integer
 * metadata/attachments/name: string
 * metadata/attachments/path: string (file path on local machine)
 * metadata/batch_id: string (usually string-number, but can be 'none')
 * metadata/board: string
 * metadata/build_number: integer
 * metadata/compiled_on: docker
 * metadata/fuego_core_version: string
 * metadata/fuego_version: string
 * metadata/host_name: string
 * metadata/job_name: string
 * metadata/keep_log: boolean ([true|false])
 * metadata/kernel_version: string (from uname -r)
 * metadata/reboot: boolean ([true|false])
 * metadata/rebuild: boolean ([true|false])
 * metadata/start_time: integer (seconds since the epoch)
 * metadata/target_postcleanup: boolean ([true|false])
 * metadata/target_precleanup: boolean ([true|false])
 * metadata/test_plan: string
 * name: string
 * fields (from prolog.sh)
 * TOOLCHAIN: string (Fuego-specific)
 * ARCHITECTURE: string
 * BOARD_TESTDIR: string
 * fields from machine-snapshot.txt
 * AR: string (archiver program name)
 * ARCH: string
 * AS: string (assembler program name)
 * BUILD_ID: integer
 * BUILD_NUMBER: integer
 * BUILD_TAG: string
 * BUILD_TIMESTAMP: string (ISO 8601 date format)
 * CC: string (C compiler program name)
 * CONFIGURE_FLAGS: string (suitable for use with configure scripts)
 * CPP: string (C++ compiler program name)
 * CROSS_COMPILE: string (cross-compiler program name prefix)
 * CXX: string (C++ compiler program name)
 * CXXCPP: string (C++ preprocessor program name (or invocation string))
 * Device: string (board name)
 * FUEGO_BATCH_ID: string
 * FUEGO_CORE_VERSION: string
 * FUEGO_HOST: string (test host identifier)
 * FUEGO_START_TIME: integer (seconds since the epoch)
 * FUEGO_VERSION: string
 * FWVER: string (kernel version from uname -r)
 * JOB_BUILD_DIR: string
 * JOB_NAME=bbb.default.Functional.hello_world
 * LD: string (linker program name)
 * LDFLAGS: string
 * RANLIB: string (ranlib program name)
 * TESTDIR: string (Fuego test name)
 * TESTSPEC: string (Fuego spec name)
 * TESTSUITE_VERSION: string (md5sum of tarball, or git hash)
 * TEST_FUEGO_RELEASE: integer
 * TEST_VERSION: string
 * TOOLCHAIN: string


 * fields from /etc/os-release
 * PRETTY_NAME: string
 * NAME: string
 * VERSION_ID: string
 * VERSION: string
 * ID: string

Sample: {   "duration_ms": 8796, "metadata": { "attachments": [ {               "name": "devlog", "path": "devlog.txt" },           {                "name": "snapshot", "path": "machine-snapshot.txt" },           {                "name": "syslog.before", "path": "syslog.before.txt" },           {                "name": "syslog.after", "path": "syslog.after.txt" },           {                "name": "testlog", "path": "testlog.txt" {               "name": "build_xml", "path": "build.xml" },           {                "name": "consolelog", "path": "consolelog.txt" },           {                "name": "test_spec", "path": "spec.json" },           {                "name": "prolog", "path": "prolog.sh" }       ],        "batch_id": "none", "board": "bbb", "build_number": "9", "compiled_on": "docker", "fuego_core_version": "v1.5.8", "fuego_version": "v1.5.8", "host_name": "fuegohost", "job_name": "bbb.default.Functional.hello_world", "keep_log": true, "kernel_version": "4.4.155-ti-r155", "reboot": "false", "rebuild": "false", "start_time": "1650578453204", "target_postcleanup": "true", "target_precleanup": "true", "test_plan": "None", "test_spec": "default", "testsuite_version": "45c3c93f8e9dc538ff8ea9e885ab05fa", "timestamp": "2022-04-21T22:00:52+0000", "toolchain": "debian-armhf", "workspace": "/fuego-rw/buildzone" },   "name": "Functional.hello_world", "schema_version": "1.0", "status": "PASS", "test_sets": [ {           "name": "default", "status": "PASS", "test_cases": [ {                   "name": "hello_world", "status": "PASS" }           ]        }    ] }

Other test metadata is stored in prolog.sh (or machine-snapshot.txt)

prolog.sh example: EXPAT_SUBTEST_COUNT_POS="1769" MMC_DEV="/dev/mmcblk0p2" SRV_IP="192.168.7.1" TOOLCHAIN="debian-armhf" EXPAT_SUBTEST_COUNT_NEG="41" BUSYBOX_SKIPLIST="chvt install" LOGIN="root" FUEGO_TARGET_TMP="/home/a" PASSWORD="root" SATA_MP="/mnt/sata" LTP_OPEN_POSIX_SUBTEST_COUNT_POS="1232" BOARD_CONTROL="ttc" SATA_DEV="/dev/sdb1" FUNCTIONAL_KERNEL_BUILD_TARGET="zImage" ARCH_TIMER_STRING="GPTIMER2 at 24000000" MMC_MP="/mnt/mmc" ARCH_TIMER_NAME="gp_timer" TTC_TARGET="bbb" USB_DEV="/dev/sda1" LTP_OPEN_POSIX_SUBTEST_COUNT_NEG="158" SSH_PORT="22" IPADDR="192.168.7.2" ARCH_TIMER_ID="67" USB_MP="/mnt/usb" MAX_REBOOT_RETRIES="20" ARCHITECTURE="arm" BOARD_TESTDIR="/home/fuego" TRANSPORT="ssh" ... S2S_SERLOGIN="python /usr/local/bin/serlogin -e 30 -b $BAUD -d $SERIAL " SERLOGIN="run_python_quiet /usr/local/bin/serlogin -e 30 -b $BAUD -d $SERIAL " SERCP="run_python_quiet /usr/local/bin/sercp -B -b $BAUD -r" SERSH="run_python_quiet /usr/local/bin/sersh -b $BAUD $SERIAL " TTC="ttc" S2S_SERSH="python /usr/local/bin/sersh -b $BAUD $SERIAL " SCP_ARGS="-o ServerAliveInterval=30 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=error -o ConnectTimeout=30 -P $SSH_PORT" S2S_SSH_ARGS="-o ServerAliveInterval=30 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=error -o ConnectTimeout=30 -p $SSH_PORT $S2S_LOGIN@" DEVICE="$IPADDR" PATH="/usr/local/bin:$PATH" S2S_SERCP="python /usr/local/bin/sercp -B -b $BAUD -r" SSH_ARGS="-o ServerAliveInterval=30 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=error -o ConnectTimeout=30 -p $SSH_PORT $LOGIN@" S2S_SCP_ARGS="-o ServerAliveInterval=30 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=error -o ConnectTimeout=30 -P $SSH_PORT"
 * 1) class: base-board
 * 1) class: base-params

FUNCTIONAL_HELLO_WORLD_ARG=""
 * 1) class: base-vars

KCIDB

 * format=json?, database?
 * fields (see below)
 * version/major
 * version/minor

checkouts

 * fields for a checkout
 * id: string
 * origin: string

(from the submitter guide )
 * valid: boolean ([true|false])
 * git_repository_url: string
 * git_commit_hash: string (sha265)
 * patchset_hash: string (sha265)
 * start_time: string (RFC3339 date string)

(from the kcidb interface ):
 * branch: string
 * commit_name: string
 * log: ???

builds

 * fields for a build
 * id: string

(from the submitter guide )
 * checkout_id: string
 * origin: string
 * valid: boolean ([true|false])
 * architecture: string
 * compiler: string
 * start_time: string (RFC3339 date string)

(from the kcidb interface)
 * configuration: string
 * duration: string (hours:minutes:seconds.fractions)
 * comment: string
 * output files: ???
 * config
 * dtbs.tar.xz
 * zImage
 * kselftest.tar.xz
 * build.log
 * build-debug.log
 * modules.tar.xz
 * log excerpt: string (multiline??)

tests

 * fields for a build:
 * id: string
 * build_id: string
 * origin: string

(from the submitter guide )
 * status: string ([ERROR|FAIL|PASS|DONE|SKIP])
 * waived: boolean ([true|false])
 * path: string (unique identifier for the test - see https://github.com/kernelci/kcidb/blob/main/tests.yaml)
 * start_time: string (RFC3339 date string)

(from the kcidb interface)
 * tests: integer (number of testcases?)
 * first start time: ???
 * subnodes: ???
 * comment: string

Each object also supports a "misc" field, which can have arbitrary nested fields (in json) example:
 * misc/build_platform: list of strings
 * misc/kernel_image_size: integer
 * misc/vmlinux_bss_size: integer
 * misc/vmlinux_data_size: integer