Difference between revisions of "Test meta-data survey"
(add link for test identifiers) |
|||
Line 263: | Line 263: | ||
** status: string ([ERROR|FAIL|PASS|DONE|SKIP]) | ** status: string ([ERROR|FAIL|PASS|DONE|SKIP]) | ||
** waived: boolean ([true|false]) | ** waived: boolean ([true|false]) | ||
− | ** path: string | + | ** path: string (unique identifier for the test - see https://github.com/kernelci/kcidb/blob/main/tests.yaml) |
** start_time: string (RFC3339 date string) | ** start_time: string (RFC3339 date string) | ||
Latest revision as of 10:22, 10 March 2023
Here is information about the schema (field names, and values formats) used by various build systems for automated testing:
Contents
Fuego
Here are the fields with meta-data about a test and it's build and execution environment, for Fuego.
- format=json, env output, shell script (NAME=value pairs)
- file=run.json, prolog.sh, machine-snapshot.txt
- fields (from run.json):
- duration_ms: integer
- metadata/attachments/name: string
- metadata/attachments/path: string (file path on local machine)
- metadata/batch_id: string (usually string-number, but can be 'none')
- metadata/board: string
- metadata/build_number: integer
- metadata/compiled_on: docker
- metadata/fuego_core_version: string
- metadata/fuego_version: string
- metadata/host_name: string
- metadata/job_name: string
- metadata/keep_log: boolean ([true|false])
- metadata/kernel_version: string (from uname -r)
- metadata/reboot: boolean ([true|false])
- metadata/rebuild: boolean ([true|false])
- metadata/start_time: integer (seconds since the epoch)
- metadata/target_postcleanup: boolean ([true|false])
- metadata/target_precleanup: boolean ([true|false])
- metadata/test_plan: string
- name: string
- fields (from prolog.sh)
- TOOLCHAIN: string (Fuego-specific)
- ARCHITECTURE: string
- BOARD_TESTDIR: string
- fields from machine-snapshot.txt
- AR: string (archiver program name)
- ARCH: string
- AS: string (assembler program name)
- BUILD_ID: integer
- BUILD_NUMBER: integer
- BUILD_TAG: string
- BUILD_TIMESTAMP: string (ISO 8601 date format)
- CC: string (C compiler program name)
- CONFIGURE_FLAGS: string (suitable for use with configure scripts)
- CPP: string (C++ compiler program name)
- CROSS_COMPILE: string (cross-compiler program name prefix)
- CXX: string (C++ compiler program name)
- CXXCPP: string (C++ preprocessor program name (or invocation string))
- Device: string (board name)
- FUEGO_BATCH_ID: string
- FUEGO_CORE_VERSION: string
- FUEGO_HOST: string (test host identifier)
- FUEGO_START_TIME: integer (seconds since the epoch)
- FUEGO_VERSION: string
- FWVER: string (kernel version from uname -r)
- JOB_BUILD_DIR: string
- JOB_NAME=bbb.default.Functional.hello_world
- LD: string (linker program name)
- LDFLAGS: string
- RANLIB: string (ranlib program name)
- TESTDIR: string (Fuego test name)
- TESTSPEC: string (Fuego spec name)
- TESTSUITE_VERSION: string (md5sum of tarball, or git hash)
- TEST_FUEGO_RELEASE: integer
- TEST_VERSION: string
- TOOLCHAIN: string
- fields from /etc/os-release
- PRETTY_NAME: string
- NAME: string
- VERSION_ID: string
- VERSION: string
- ID: string
Sample:
{ "duration_ms": 8796, "metadata": { "attachments": [ { "name": "devlog", "path": "devlog.txt" }, { "name": "snapshot", "path": "machine-snapshot.txt" }, { "name": "syslog.before", "path": "syslog.before.txt" }, { "name": "syslog.after", "path": "syslog.after.txt" }, { "name": "testlog", "path": "testlog.txt" { "name": "build_xml", "path": "build.xml" }, { "name": "consolelog", "path": "consolelog.txt" }, { "name": "test_spec", "path": "spec.json" }, { "name": "prolog", "path": "prolog.sh" } ], "batch_id": "none", "board": "bbb", "build_number": "9", "compiled_on": "docker", "fuego_core_version": "v1.5.8", "fuego_version": "v1.5.8", "host_name": "fuegohost", "job_name": "bbb.default.Functional.hello_world", "keep_log": true, "kernel_version": "4.4.155-ti-r155", "reboot": "false", "rebuild": "false", "start_time": "1650578453204", "target_postcleanup": "true", "target_precleanup": "true", "test_plan": "None", "test_spec": "default", "testsuite_version": "45c3c93f8e9dc538ff8ea9e885ab05fa", "timestamp": "2022-04-21T22:00:52+0000", "toolchain": "debian-armhf", "workspace": "/fuego-rw/buildzone" }, "name": "Functional.hello_world", "schema_version": "1.0", "status": "PASS", "test_sets": [ { "name": "default", "status": "PASS", "test_cases": [ { "name": "hello_world", "status": "PASS" } ] } ] }
Other test metadata is stored in prolog.sh (or machine-snapshot.txt)
prolog.sh example:
#class: base-board EXPAT_SUBTEST_COUNT_POS="1769" MMC_DEV="/dev/mmcblk0p2" SRV_IP="192.168.7.1" TOOLCHAIN="debian-armhf" EXPAT_SUBTEST_COUNT_NEG="41" BUSYBOX_SKIPLIST="chvt install" LOGIN="root" FUEGO_TARGET_TMP="/home/a" PASSWORD="root" SATA_MP="/mnt/sata" LTP_OPEN_POSIX_SUBTEST_COUNT_POS="1232" BOARD_CONTROL="ttc" SATA_DEV="/dev/sdb1" FUNCTIONAL_KERNEL_BUILD_TARGET="zImage" ARCH_TIMER_STRING="GPTIMER2 at 24000000" MMC_MP="/mnt/mmc" ARCH_TIMER_NAME="gp_timer" TTC_TARGET="bbb" USB_DEV="/dev/sda1" LTP_OPEN_POSIX_SUBTEST_COUNT_NEG="158" SSH_PORT="22" IPADDR="192.168.7.2" ARCH_TIMER_ID="67" USB_MP="/mnt/usb" MAX_REBOOT_RETRIES="20" ARCHITECTURE="arm" BOARD_TESTDIR="/home/fuego" TRANSPORT="ssh" ... #class: base-params S2S_SERLOGIN="python /usr/local/bin/serlogin -e 30 -b $BAUD -d $SERIAL " SERLOGIN="run_python_quiet /usr/local/bin/serlogin -e 30 -b $BAUD -d $SERIAL " SERCP="run_python_quiet /usr/local/bin/sercp -B -b $BAUD -r" SERSH="run_python_quiet /usr/local/bin/sersh -b $BAUD $SERIAL " TTC="ttc" S2S_SERSH="python /usr/local/bin/sersh -b $BAUD $SERIAL " SCP_ARGS="-o ServerAliveInterval=30 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=error -o ConnectTimeout=30 -P $SSH_PORT" S2S_SSH_ARGS="-o ServerAliveInterval=30 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=error -o ConnectTimeout=30 -p $SSH_PORT $S2S_LOGIN@" DEVICE="$IPADDR" PATH="/usr/local/bin:$PATH" S2S_SERCP="python /usr/local/bin/sercp -B -b $BAUD -r" SSH_ARGS="-o ServerAliveInterval=30 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=error -o ConnectTimeout=30 -p $SSH_PORT $LOGIN@" S2S_SCP_ARGS="-o ServerAliveInterval=30 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=error -o ConnectTimeout=30 -P $SSH_PORT" #class: base-vars FUNCTIONAL_HELLO_WORLD_ARG=""
KCIDB
- format=json?, database?
- fields (see below)
- version/major
- version/minor
checkouts
- fields for a checkout
- id: string
- origin: string
(from the submitter guide )
- valid: boolean ([true|false])
- git_repository_url: string
- git_commit_hash: string (sha265)
- patchset_hash: string (sha265)
- start_time: string (RFC3339 date string)
(from the kcidb interface [1] ):
- branch: string
- commit_name: string
- log: ???
builds
- fields for a build
- id: string
(from the submitter guide )
- checkout_id: string
- origin: string
- valid: boolean ([true|false])
- architecture: string
- compiler: string
- start_time: string (RFC3339 date string)
(from the kcidb interface)
- configuration: string
- duration: string (hours:minutes:seconds.fractions)
- comment: string
- output files: ???
- config
- dtbs.tar.xz
- zImage
- kselftest.tar.xz
- build.log
- build-debug.log
- modules.tar.xz
- log excerpt: string (multiline??)
tests
- fields for a build:
- id: string
- build_id: string
- origin: string
(from the submitter guide )
- status: string ([ERROR|FAIL|PASS|DONE|SKIP])
- waived: boolean ([true|false])
- path: string (unique identifier for the test - see https://github.com/kernelci/kcidb/blob/main/tests.yaml)
- start_time: string (RFC3339 date string)
(from the kcidb interface)
- tests: integer (number of testcases?)
- first start time: ???
- subnodes: ???
- comment: string
Each object also supports a "misc" field, which can have arbitrary nested fields (in json) example:
- misc/build_platform: list of strings
- misc/kernel_image_size: integer
- misc/vmlinux_bss_size: integer
- misc/vmlinux_data_size: integer