Difference between revisions of "Test meta-data survey"

From eLinux.org
Jump to: navigation, search
(add link for test identifiers)
 
Line 263: Line 263:
 
** status: string ([ERROR|FAIL|PASS|DONE|SKIP])
 
** status: string ([ERROR|FAIL|PASS|DONE|SKIP])
 
** waived: boolean ([true|false])
 
** waived: boolean ([true|false])
** path: string
+
** path: string (unique identifier for the test - see https://github.com/kernelci/kcidb/blob/main/tests.yaml)
 
** start_time: string (RFC3339 date string)
 
** start_time: string (RFC3339 date string)
  

Latest revision as of 10:22, 10 March 2023

Here is information about the schema (field names, and values formats) used by various build systems for automated testing:

Fuego

Here are the fields with meta-data about a test and it's build and execution environment, for Fuego.

  • format=json, env output, shell script (NAME=value pairs)
  • file=run.json, prolog.sh, machine-snapshot.txt
  • fields (from run.json):
    • duration_ms: integer
    • metadata/attachments/name: string
    • metadata/attachments/path: string (file path on local machine)
    • metadata/batch_id: string (usually string-number, but can be 'none')
    • metadata/board: string
    • metadata/build_number: integer
    • metadata/compiled_on: docker
    • metadata/fuego_core_version: string
    • metadata/fuego_version: string
    • metadata/host_name: string
    • metadata/job_name: string
    • metadata/keep_log: boolean ([true|false])
    • metadata/kernel_version: string (from uname -r)
    • metadata/reboot: boolean ([true|false])
    • metadata/rebuild: boolean ([true|false])
    • metadata/start_time: integer (seconds since the epoch)
    • metadata/target_postcleanup: boolean ([true|false])
    • metadata/target_precleanup: boolean ([true|false])
    • metadata/test_plan: string
    • name: string
  • fields (from prolog.sh)
    • TOOLCHAIN: string (Fuego-specific)
    • ARCHITECTURE: string
    • BOARD_TESTDIR: string
  • fields from machine-snapshot.txt
    • AR: string (archiver program name)
    • ARCH: string
    • AS: string (assembler program name)
    • BUILD_ID: integer
    • BUILD_NUMBER: integer
    • BUILD_TAG: string
    • BUILD_TIMESTAMP: string (ISO 8601 date format)
    • CC: string (C compiler program name)
    • CONFIGURE_FLAGS: string (suitable for use with configure scripts)
    • CPP: string (C++ compiler program name)
    • CROSS_COMPILE: string (cross-compiler program name prefix)
    • CXX: string (C++ compiler program name)
    • CXXCPP: string (C++ preprocessor program name (or invocation string))
    • Device: string (board name)
    • FUEGO_BATCH_ID: string
    • FUEGO_CORE_VERSION: string
    • FUEGO_HOST: string (test host identifier)
    • FUEGO_START_TIME: integer (seconds since the epoch)
    • FUEGO_VERSION: string
    • FWVER: string (kernel version from uname -r)
    • JOB_BUILD_DIR: string
    • JOB_NAME=bbb.default.Functional.hello_world
    • LD: string (linker program name)
    • LDFLAGS: string
    • RANLIB: string (ranlib program name)
    • TESTDIR: string (Fuego test name)
    • TESTSPEC: string (Fuego spec name)
    • TESTSUITE_VERSION: string (md5sum of tarball, or git hash)
    • TEST_FUEGO_RELEASE: integer
    • TEST_VERSION: string
    • TOOLCHAIN: string
  • fields from /etc/os-release
    • PRETTY_NAME: string
    • NAME: string
    • VERSION_ID: string
    • VERSION: string
    • ID: string

Sample:

{
    "duration_ms": 8796,
    "metadata": {
        "attachments": [
            {
                "name": "devlog",
                "path": "devlog.txt"
            },
            {
                "name": "snapshot",
                "path": "machine-snapshot.txt"
            },
            {
                "name": "syslog.before",
                "path": "syslog.before.txt"
            },
            {
                "name": "syslog.after",
                "path": "syslog.after.txt"
            },
            {
                "name": "testlog",
                "path": "testlog.txt"
            {
                "name": "build_xml",
                "path": "build.xml"
            },
            {
                "name": "consolelog",
                "path": "consolelog.txt"
            },
            {
                "name": "test_spec",
                "path": "spec.json"
            },
            {
                "name": "prolog",
                "path": "prolog.sh"
            }
        ],
        "batch_id": "none",
        "board": "bbb",
        "build_number": "9",
        "compiled_on": "docker",
        "fuego_core_version": "v1.5.8",
        "fuego_version": "v1.5.8",
        "host_name": "fuegohost",
        "job_name": "bbb.default.Functional.hello_world",
        "keep_log": true,
        "kernel_version": "4.4.155-ti-r155",
        "reboot": "false",
        "rebuild": "false",
        "start_time": "1650578453204",
        "target_postcleanup": "true",
        "target_precleanup": "true",
        "test_plan": "None",
        "test_spec": "default",
        "testsuite_version": "45c3c93f8e9dc538ff8ea9e885ab05fa",
        "timestamp": "2022-04-21T22:00:52+0000",
        "toolchain": "debian-armhf",
        "workspace": "/fuego-rw/buildzone"
    },
    "name": "Functional.hello_world",
    "schema_version": "1.0",
    "status": "PASS",
    "test_sets": [
        {
            "name": "default",
            "status": "PASS",
            "test_cases": [
                {
                    "name": "hello_world",
                    "status": "PASS"
                }
            ]
        }
    ]
}

Other test metadata is stored in prolog.sh (or machine-snapshot.txt)

prolog.sh example:

#class: base-board
EXPAT_SUBTEST_COUNT_POS="1769"
MMC_DEV="/dev/mmcblk0p2"
SRV_IP="192.168.7.1"
TOOLCHAIN="debian-armhf"
EXPAT_SUBTEST_COUNT_NEG="41"
BUSYBOX_SKIPLIST="chvt install"
LOGIN="root"
FUEGO_TARGET_TMP="/home/a"
PASSWORD="root"
SATA_MP="/mnt/sata"
LTP_OPEN_POSIX_SUBTEST_COUNT_POS="1232"
BOARD_CONTROL="ttc"
SATA_DEV="/dev/sdb1"
FUNCTIONAL_KERNEL_BUILD_TARGET="zImage"
ARCH_TIMER_STRING="GPTIMER2 at 24000000"
MMC_MP="/mnt/mmc"
ARCH_TIMER_NAME="gp_timer"
TTC_TARGET="bbb"
USB_DEV="/dev/sda1"
LTP_OPEN_POSIX_SUBTEST_COUNT_NEG="158"
SSH_PORT="22"
IPADDR="192.168.7.2"
ARCH_TIMER_ID="67"
USB_MP="/mnt/usb"
MAX_REBOOT_RETRIES="20"
ARCHITECTURE="arm"
BOARD_TESTDIR="/home/fuego"
TRANSPORT="ssh"
...
#class: base-params
S2S_SERLOGIN="python /usr/local/bin/serlogin -e 30 -b $BAUD -d $SERIAL "
SERLOGIN="run_python_quiet /usr/local/bin/serlogin -e 30 -b $BAUD -d $SERIAL "
SERCP="run_python_quiet /usr/local/bin/sercp -B -b $BAUD -r"
SERSH="run_python_quiet /usr/local/bin/sersh -b $BAUD $SERIAL "
TTC="ttc"
S2S_SERSH="python /usr/local/bin/sersh -b $BAUD $SERIAL "
SCP_ARGS="-o ServerAliveInterval=30 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=error -o ConnectTimeout=30 -P $SSH_PORT"
S2S_SSH_ARGS="-o ServerAliveInterval=30 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=error -o ConnectTimeout=30 -p $SSH_PORT $S2S_LOGIN@"
DEVICE="$IPADDR"
PATH="/usr/local/bin:$PATH"
S2S_SERCP="python /usr/local/bin/sercp -B -b $BAUD -r"
SSH_ARGS="-o ServerAliveInterval=30 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=error -o ConnectTimeout=30 -p $SSH_PORT $LOGIN@"
S2S_SCP_ARGS="-o ServerAliveInterval=30 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=error -o ConnectTimeout=30 -P $SSH_PORT"

#class: base-vars
FUNCTIONAL_HELLO_WORLD_ARG=""

KCIDB

  • format=json?, database?
  • fields (see below)
    • version/major
    • version/minor

checkouts

  • fields for a checkout
    • id: string
    • origin: string

(from the submitter guide )

    • valid: boolean ([true|false])
    • git_repository_url: string
    • git_commit_hash: string (sha265)
    • patchset_hash: string (sha265)
    • start_time: string (RFC3339 date string)

(from the kcidb interface [1] ):

    • branch: string
    • commit_name: string
    • log: ???

builds

  • fields for a build
    • id: string

(from the submitter guide )

    • checkout_id: string
    • origin: string
    • valid: boolean ([true|false])
    • architecture: string
    • compiler: string
    • start_time: string (RFC3339 date string)

(from the kcidb interface)

    • configuration: string
    • duration: string (hours:minutes:seconds.fractions)
    • comment: string
    • output files: ???
      • config
      • dtbs.tar.xz
      • zImage
      • kselftest.tar.xz
      • build.log
      • build-debug.log
      • modules.tar.xz
    • log excerpt: string (multiline??)

tests

  • fields for a build:
    • id: string
    • build_id: string
    • origin: string

(from the submitter guide )

(from the kcidb interface)

    • tests: integer (number of testcases?)
    • first start time: ???
    • subnodes: ???
    • comment: string

Each object also supports a "misc" field, which can have arbitrary nested fields (in json) example:

    • misc/build_platform: list of strings
    • misc/kernel_image_size: integer
    • misc/vmlinux_bss_size: integer
    • misc/vmlinux_data_size: integer

tuxmake

Yocto Project

kernelci

LAVA

syzbot