Sysbench test definition comparison

This page has a comparison between the test definitions from Fuego and Linaro for the OpenSSL test.

= Differences =
 * Fuego only runs the cpu sub-test
 * Linaro runs multiple sub-tests (cpu, memory, threads, mutex, fileio)


 * Linaro assumes a clean install, that will be replaced on next test
 * Things can be modified (packages installed, and forgotten about)


 * Fuego treats system like final product that is immutable
 * Linaro treats system like development system, that is mutable

building

 * Fuego cross-builds the test software
 * Linaro build the software on-target

Pre-requisites

 * Fuego checks for cross-compiler variables
 * Linaro checks for root account

Alterations

 * Linaro can install packages required by the sysbench build on the board
 * Fuego deploys the test software to the board

Execution

 * Linaro runs each sub-test separately
 * Fuego runs just one of the sysbench tests


 * Factorization of the test is different
 * dependency check, alterations, test execution, parsing are done on board for Linaro
 * dependency check, test execution, parsing are done on the host for Fuego

Parsing

 * Linaro parses the output for each sub-test on the target using awk
 * Fuego parses the combined output on the host using python (parser.py)

Results

 * output is different

Presentation

 * Linaro doesn't include presentation control for the test results in the test

Metadata

 * Fuego specifies author, license gitrepo, for test program
 * Linaro specifies the devices for the test to run on
 * Linaro specifies distros where test can run

= questions: =
 * Linaro install_deps: does this also install the package itself (with the openssl binary)?
 * Linaro: what does send-to-laval.sh do?

= Field comparisons =

= Fuego source (Benchmark.sysbench dir) =

fuego_test.sh
tarball=sysbench-0.4.8.tar.bz2


 * 1) FIXTHIS: should check for libaio in test_pre_check

function test_build { # not needed due to patch removing MALLOC check #echo "ac_cv_func_malloc_o_nonnull=no" > config.cache patch -p0 < $TEST_HOME/malloc_cross_compile_fix.patch patch -p0 < $TEST_HOME/disable_libtool.patch ./autogen.sh

# get updated config.sub and config.guess files, so configure # doesn't reject new toolchains cp /usr/share/misc/config.{sub,guess} config

# use this line if config.cache is required #./configure --build=`./config.guess` --HOST=$HOST CC="$CC" AR="$AR" RANLIB="$RANLIB" --config-cache # FIXTHIS: would be nice to support mysql and other tests as well ./configure --host=$PREFIX --without-mysql make }

function test_deploy { put sysbench/sysbench $BOARD_TESTDIR/fuego.$TESTDIR/ }

function test_run { # tests available are: fileio, cpu, memory, threads, mutex, otlp report "cd $BOARD_TESTDIR/fuego.$TESTDIR; ./sysbench --num-threads=4 --test=cpu --cpu-max-prime=20000 --validate run" }

function test_processing { echo "Need something in test_processing" }

parser.py

 * 1) !/usr/bin/python

import os, re, sys import common as plib

measurements = {} regex_string = "^.*total time:\s+([\d]{1,8}.?[\d]{1,43})s$" matches = plib.parse_log(regex_string)

if matches: measurements['cpu.prime_calculation'] = [{"name": "time", "measure" : float(matches[0])}]

regex_string = "^.*events \(avg/stddev\):\s+([\d]{1,8}.?[\d]{1,43})/.*$" matches = plib.parse_log(regex_string)

if matches: measurements['cpu.thread_fairness'] = [{"name": "events", "measure" : float(matches[0])}]

sys.exit(plib.process(measurements))

spec.json
{   "testName": "Benchmark.sysbench", "specs": { "default": {} } }

test.yaml
fuego_package_version: 1 name: Benchmark.sysbench description: | Measure the performance of system operations

This test only performs the 'cpu' performance test. There are other tests available. license: GPL-2.0 author: Alexy Kopytov maintainer: Tim Bird  version: 0.4.8 fuego_release: 1 type: Benchmark tags: ['system', 'cpu', 'performance'] gitrepo: https://github.com/akopytov/sysbench data_files: - fuego_test.sh   - parser.py    - spec.json - test.yaml - sysbench-0.4.8.tar.bz2 - disable_libtool.patch - malloc_cross_compile_fix.patch

= Linaro source (sysbench dir) =

sysbench.sh

 * 1) !/bin/sh -e


 * 1) SysBench is a modular, cross-platform and multi-threaded benchmark tool.
 * 2) Current features allow to test the following system parameters:
 * 3) * file I/O performance
 * 4) * scheduler performance
 * 5) * memory allocation and transfer speed
 * 6) * POSIX threads implementation performance
 * 7) * database server performance

. ../../lib/sh-test-lib
 * 1) shellcheck disable=SC1091

OUTPUT="$(pwd)/output" RESULT_FILE="${OUTPUT}/result.txt" export RESULT_FILE SKIP_INSTALL="false"

NUM_THREADS="NPROC" TESTS="cpu memory threads mutex fileio"
 * 1) sysbench test parameters.
 * 1) TESTS="cpu memory threads mutex fileio oltp"

usage { echo "usage: $0 [-n ] [-t ] [-s ] 1>&2" exit 1 }

while getopts ":n:t:s:" opt; do   case "${opt}" in        n) NUM_THREADS="${OPTARG}" ;;        t) TESTS="${OPTARG}" ;; s) SKIP_INSTALL="${OPTARG}" ;;       *) usage ;; esac done [ "${NUM_THREADS}" = "NPROC" ] && NUM_THREADS=$(nproc)

install_sysbench { git clone https://github.com/akopytov/sysbench cd sysbench git checkout 0.4 ./autogen.sh   if echo "${TESTS}" | grep "oltp"; then ./configure else ./configure --without-mysql fi   make install cd ../ }

! check_root && error_msg "Please run this script as root." create_out_dir "${OUTPUT}" cd "${OUTPUT}"

if [ "${SKIP_INSTALL}" = "true" ] || [ "${SKIP_INSTALL}" = "True" ]; then info_msg "sysbench installation skipped" else dist_name # shellcheck disable=SC2154 case "${dist}" in       debian|ubuntu)            install_deps "git build-essential automake libtool"            if echo "${TESTS}" | grep "oltp"; then                install_deps "default-libmysqlclient-dev default-mysql-server"                systemctl start mysql            fi            install_sysbench            ;;        fedora|centos) install_deps "git gcc make automake libtool" if echo "${TESTS}" | grep "oltp"; then install_deps "mysql-devel mariadb-server mariadb" systemctl start mariadb fi           install_sysbench ;;       oe-rpb)            # Assume all dependent packages are already installed.            install_sysbench            ;;        *) warn_msg "Unsupported distro: ${dist}! Package installation skipped!" ;;   esac fi
 * 1) Test installation.

sysbench --version
 * 1) Verify test installation.

general_parser { # if $1 is there, let's append to test name in the result file # shellcheck disable=SC2039 local tc="$tc$1" ms=$(grep -m 1 "total time" "${logfile}" | awk '{print substr($NF,1,length($NF)-1)}') add_metric "${tc}-total-time" "pass" "${ms}" "s"

ms=$(grep "total number of events" "${logfile}" | awk '{print $NF}') add_metric "${tc}-total-number-of-events" "pass" "${ms}" "times"

ms=$(grep "total time taken by event execution" "${logfile}" | awk '{print $NF}') add_metric "${tc}-total-time-taken-by-event-execution" "pass" "${ms}" "s"

for i in min avg max approx; do       ms=$(grep -m 1 "$i" "${logfile}" | awk '{print substr($NF,1,length($NF)-2)}') add_metric "${tc}-response-time-$i" "pass" "${ms}" "ms" done

ms=$(grep "events (avg/stddev)" "${logfile}" | awk '{print $NF}') ms_avg=$(echo "${ms}" | awk -F'/' '{print $1}') ms_stddev=$(echo "${ms}" | awk -F'/' '{print $2}') add_metric "${tc}-events-avg" "pass" "${ms_avg}" "times" add_metric "${tc}-events-stddev" "pass" "${ms_stddev}" "times"

ms=$(grep "execution time (avg/stddev)" "${logfile}" | awk '{print $NF}') ms_avg=$(echo "${ms}" | awk -F'/' '{print $1}') ms_stddev=$(echo "${ms}" | awk -F'/' '{print $2}') add_metric "${tc}-execution-time-avg" "pass" "${ms_avg}" "s" add_metric "${tc}-execution-time-stddev" "pass" "${ms_stddev}" "s" }

for tc in ${TESTS}; do   echo info_msg "Running sysbench ${tc} test..." logfile="${OUTPUT}/sysbench-${tc}.txt" case "${tc}" in       percpu)            processor_id="$(awk '/^processor/{print $3}' /proc/cpuinfo)"            for i in ${processor_id}; do                taskset -c "$i" sysbench --num-threads=1 --test=cpu run | tee "${logfile}"                general_parser "$i"            done            ;;        cpu|threads|mutex) sysbench --num-threads="${NUM_THREADS}" --test="${tc}" run | tee "${logfile}" general_parser ;;       memory)            sysbench --num-threads="${NUM_THREADS}" --test=memory run | tee "${logfile}"            general_parser
 * 1) Test run.

ms=$(grep "Operations" "${logfile}" | awk '{print substr($4,2)}') add_metric "${tc}-ops" "pass" "${ms}" "ops"

ms=$(grep "transferred" "${logfile}" | awk '{print substr($4, 2)}') units=$(grep "transferred" "${logfile}" | awk '{print substr($5,1,length($NF)-1)}') add_metric "${tc}-transfer" "pass" "${ms}" "${units}" ;;       fileio)            mkdir fileio && cd fileio            for mode in seqwr seqrewr seqrd rndrd rndwr rndrw; do                tc="fileio-${mode}"                logfile="${OUTPUT}/sysbench-${tc}.txt"                sync                echo 3 > /proc/sys/vm/drop_caches                sleep 5                sysbench --num-threads="${NUM_THREADS}" --test=fileio --file-total-size=2G --file-test-mode="${mode}" prepare                # --file-extra-flags=direct is needed when file size is smaller then RAM.                sysbench --num-threads="${NUM_THREADS}" --test=fileio --file-extra-flags=direct --file-total-size=2G --file-test-mode="${mode}" run | tee "${logfile}"                sysbench --num-threads="${NUM_THREADS}" --test=fileio --file-total-size=2G --file-test-mode="${mode}" cleanup                general_parser

ms=$(grep "transferred" "${logfile}" | awk '{print substr($NF, 2,(length($NF)-8))}') units=$(grep "transferred" "${logfile}" | awk '{print substr($NF,(length($NF)-6),6)}') add_metric "${tc}-transfer" "pass" "${ms}" "${units}"

ms=$(grep "Requests/sec" "${logfile}" | awk '{print $1}') add_metric "${tc}-ops" "pass" "${ms}" "ops" done cd ../ ;;       oltp)            # Use the same passwd as lamp and lemp tests.            mysqladmin -u root password lxmptest  > /dev/null 2>&1 || true            # Delete sysbench in case it exists.            mysql --user='root' --password='lxmptest' -e 'DROP DATABASE sysbench' > /dev/null 2>&1 || true            # Create sysbench database.            mysql --user="root" --password="lxmptest" -e "CREATE DATABASE sysbench"

sysbench --num-threads="${NUM_THREADS}" --test=oltp --db-driver=mysql --oltp-table-size=1000000 --mysql-db=sysbench --mysql-user=root --mysql-password=lxmptest prepare sysbench --num-threads="${NUM_THREADS}" --test=oltp --db-driver=mysql --oltp-table-size=1000000 --mysql-db=sysbench --mysql-user=root --mysql-password=lxmptest run | tee "${logfile}"

# Parse test log. general_parser

for i in "read" write other total; do               ms=$(grep "${i}:" "${logfile}" | awk '{print $NF}') add_metric "${tc}-${i}-queries" "pass" "${ms}" "queries" done

for i in transactions deadlocks "read/write requests" "other operations"; do               ms=$(grep "${i}:" sysbench-oltp.txt | awk '{print substr($(NF-2),2)}') i=$(echo "$i" | sed 's/ /-/g; s|/|-|') add_metric "${tc}-${i}" "pass" "${ms}" "ops" done

# cleanup mysql --user='root' --password='lxmptest' -e 'DROP DATABASE sysbench' ;;   esac done

sysbench.yaml
metadata: name: sysbench format: "Lava-Test-Shell Test Definition 1.0" description: "SysBench is a modular, cross-platform and multi-threaded                 benchmark tool for evaluating OS parameters that are                  important for a system running a database under intensive                  load. Current features allow to test fileio, cpu, memory,                  threads, mutex and oltp." maintainer: - chase.qi@linaro.org os: - debian - ubuntu - fedora - centos - openembedded scope: - performance environment: - lava-test-shell devices: - hi6220-hikey - apq8016-sbc - mustang - moonshot - thunderX - d03 - d05

params: # "${TESTS}" could be one or more of the following tests. # TESTS: cpu memory threads mutex fileio oltp # TESTS: "oltp" TESTS: "percpu cpu memory threads mutex fileio" # Number of threads to run. # Run $(nproc) threads by default. NUM_THREADS: "NPROC" SKIP_INSTALL: "false"

run: steps: - cd ./automated/linux/sysbench/ - ./sysbench.sh -n "${NUM_THREADS}" -t "${TESTS}" -s "${SKIP_INSTALL}" - ../../utils/send-to-lava.sh ./output/result.txt