Sysbench test definition comparison

From eLinux.org
Jump to: navigation, search

This page has a comparison between the test definitions from Fuego and Linaro for the OpenSSL test.

Differences

  • Fuego only runs the cpu sub-test
  • Linaro runs multiple sub-tests (cpu, memory, threads, mutex, fileio)
  • Linaro assumes a clean install, that will be replaced on next test
    • Things can be modified (packages installed, and forgotten about)
  • Fuego treats system like final product that is immutable
  • Linaro treats system like development system, that is mutable

preparation

building

  • Fuego cross-builds the test software
  • Linaro build the software on-target

Pre-requisites

  • Fuego checks for cross-compiler variables
  • Linaro checks for root account

Alterations

  • Linaro can install packages required by the sysbench build on the board
  • Fuego deploys the test software to the board

Execution

  • Linaro runs each sub-test separately
  • Fuego runs just one of the sysbench tests
  • Factorization of the test is different
    • dependency check, alterations, test execution, parsing are done on board for Linaro
    • dependency check, test execution, parsing are done on the host for Fuego

Parsing

  • Linaro parses the output for each sub-test on the target using awk
  • Fuego parses the combined output on the host using python (parser.py)

Results

  • output is different

Presentation

  • Linaro doesn't include presentation control for the test results in the test

Metadata

  • Fuego specifies author, license gitrepo, for test program
  • Linaro specifies the devices for the test to run on
  • Linaro specifies distros where test can run

questions:

  • Linaro install_deps: does this also install the package itself (with the openssl binary)?
  • Linaro: what does send-to-laval.sh do?


Field comparisons

Field items
Fuego Linaro Notes
item use item use
fuego_test.sh:test_pre_check check required test pre-requisites (none in this test) sysbench.sh:! check_root && error_msg check required test pre-requisites Linaro code is inline in test script
fuego_test.sh:test_build cross-build the test program from tar sysbench.sh:install_sysbench download and build the test program -
- - sysbench.sh:install_sysbench install required packages for build Linaro has different build and dependency info per distro, Fuego has no notion of installing auxiliary packages on the board
fuego_test.sh:test_deploy Put test program on the board sysbench.sh:install_sysbench install test program on board (locally)
fuego_test.sh:test_run instructions to execute the test program on the board sysbench.yaml:run:steps: instructions to execute the test program on the board -
parser.py code to parse the test program log sysbench.sh:general_parser and awk lines code to parse the test program log Linaro parsing is done on board
spec.json indicates values for test variables (none for this test) sysbench.sh:NUM_THREADS= indicates values for test variables Linaro options are read on command line of test script
test.yaml:fuego_package indicates type/format of test sysbench.yaml:metadata:format indicates type/format of test -
test.yaml:name name of test sysbench.yaml:metadata:name name of test similar
test.yaml:description description of test sysbench.yaml:metadata:description description of test similar
test.yaml:license/author/version test program information - - Informational data
test.yaml:maintainer Maintainer of this Fuego test sysbench.yaml:metadata:maintainer Maintainer of this Linaro test similar
test.yaml:fuego_release Fuego revision of this test - - -
test.yaml:type type of test sysbench.yaml:metadata:scope type of test? -
- - sysbench.yaml:metadata:os OSes that this test can run on Linaro only?
- - sysbench.yaml:metadata:devices devices that this test can run on Linaro only? (Fuego board selection is done by user when creating jobs for boards?)
test.yaml:tags tags for this test - - Fuego only?
test.yaml:params test variable names, values, options (note: none in this test) sysbench.yaml:params test variable names and values similar
test.yaml:gitrepo upstream git repository for test program - - Fuego only?
test.yaml:data_files manifest used for packaging the test - - Fuego only?

Fuego source (Benchmark.sysbench dir)

fuego_test.sh

tarball=sysbench-0.4.8.tar.bz2

# FIXTHIS: should check for libaio in test_pre_check

function test_build {
    # not needed due to patch removing MALLOC check
    #echo "ac_cv_func_malloc_o_nonnull=no" > config.cache
    patch -p0 < $TEST_HOME/malloc_cross_compile_fix.patch
    patch -p0 < $TEST_HOME/disable_libtool.patch
    ./autogen.sh

    # get updated config.sub and config.guess files, so configure
    # doesn't reject new toolchains
    cp /usr/share/misc/config.{sub,guess} config

    # use this line if config.cache is required
    #./configure --build=`./config.guess` --HOST=$HOST CC="$CC" AR="$AR" RANLIB="$RANLIB" --config-cache
    # FIXTHIS: would be nice to support mysql and other tests as well
    ./configure --host=$PREFIX --without-mysql
    make
}

function test_deploy {
    put sysbench/sysbench $BOARD_TESTDIR/fuego.$TESTDIR/
}

function test_run {
    # tests available are: fileio, cpu, memory, threads, mutex, otlp
    report "cd $BOARD_TESTDIR/fuego.$TESTDIR; ./sysbench --num-threads=4 --test=cpu --cpu-max-prime=20000 --validate run"
}

function test_processing {
    echo "Need something in test_processing"
}

parser.py

#!/usr/bin/python

import os, re, sys
import common as plib

measurements = {}
regex_string = "^.*total time:\s+([\d]{1,8}.?[\d]{1,43})s$"
matches = plib.parse_log(regex_string)

if matches:
    measurements['cpu.prime_calculation'] = [{"name": "time", "measure" : float(matches[0])}]

regex_string = "^.*events \(avg/stddev\):\s+([\d]{1,8}.?[\d]{1,43})/.*$"
matches = plib.parse_log(regex_string)

if matches:
    measurements['cpu.thread_fairness'] = [{"name": "events", "measure" : float(matches[0])}]

sys.exit(plib.process(measurements))

spec.json

{
    "testName": "Benchmark.sysbench",
    "specs": {
        "default": {}
    }
}

test.yaml

fuego_package_version: 1
name: Benchmark.sysbench
description: |
    Measure the performance of system operations

    This test only performs the 'cpu' performance test.
    There are other tests available.
license: GPL-2.0
author: Alexy Kopytov
maintainer: Tim Bird <tim.bird@sony.com>
version: 0.4.8
fuego_release: 1
type: Benchmark
tags: ['system', 'cpu', 'performance']
gitrepo: https://github.com/akopytov/sysbench
data_files:
    - fuego_test.sh
    - parser.py
    - spec.json
    - test.yaml
    - sysbench-0.4.8.tar.bz2
    - disable_libtool.patch
    - malloc_cross_compile_fix.patch

Linaro source (sysbench dir)

sysbench.sh

#!/bin/sh -e

# SysBench is a modular, cross-platform and multi-threaded benchmark tool.
# Current features allow to test the following system parameters:
# * file I/O performance
# * scheduler performance
# * memory allocation and transfer speed
# * POSIX threads implementation performance
# * database server performance

# shellcheck disable=SC1091
. ../../lib/sh-test-lib

OUTPUT="$(pwd)/output"
RESULT_FILE="${OUTPUT}/result.txt"
export RESULT_FILE
SKIP_INSTALL="false"

# sysbench test parameters.
NUM_THREADS="NPROC"
# TESTS="cpu memory threads mutex fileio oltp"
TESTS="cpu memory threads mutex fileio"

usage() {
    echo "usage: $0 [-n <num-threads>] [-t <test>] [-s <true|false>] 1>&2"
    exit 1
}

while getopts ":n:t:s:" opt; do
    case "${opt}" in
        n) NUM_THREADS="${OPTARG}" ;;
        t) TESTS="${OPTARG}" ;;
        s) SKIP_INSTALL="${OPTARG}" ;;
        *) usage ;;
    esac
done
[ "${NUM_THREADS}" = "NPROC" ] && NUM_THREADS=$(nproc)

install_sysbench() {
    git clone https://github.com/akopytov/sysbench
    cd sysbench
    git checkout 0.4
    ./autogen.sh
    if echo "${TESTS}" | grep "oltp"; then
        ./configure
    else
        ./configure --without-mysql
    fi
    make install
    cd ../
}

! check_root && error_msg "Please run this script as root."
create_out_dir "${OUTPUT}"
cd "${OUTPUT}"

# Test installation.
if [ "${SKIP_INSTALL}" = "true" ] || [ "${SKIP_INSTALL}" = "True" ]; then
    info_msg "sysbench installation skipped"
else
    dist_name
    # shellcheck disable=SC2154
    case "${dist}" in
        debian|ubuntu)
            install_deps "git build-essential automake libtool"
            if echo "${TESTS}" | grep "oltp"; then
                install_deps "default-libmysqlclient-dev default-mysql-server"
                systemctl start mysql
            fi
            install_sysbench
            ;;
        fedora|centos)
            install_deps "git gcc make automake libtool"
            if echo "${TESTS}" | grep "oltp"; then
                install_deps "mysql-devel mariadb-server mariadb"
                systemctl start mariadb
            fi
            install_sysbench
            ;;
        oe-rpb)
            # Assume all dependent packages are already installed.
            install_sysbench
            ;;
        *)
            warn_msg "Unsupported distro: ${dist}! Package installation skipped!"
            ;;
    esac
fi

# Verify test installation.
sysbench --version

general_parser() {
    # if $1 is there, let's append to test name in the result file
    # shellcheck disable=SC2039
    local tc="$tc$1"
    ms=$(grep -m 1 "total time" "${logfile}" | awk '{print substr($NF,1,length($NF)-1)}')
    add_metric "${tc}-total-time" "pass" "${ms}" "s"

    ms=$(grep "total number of events" "${logfile}" | awk '{print $NF}')
    add_metric "${tc}-total-number-of-events" "pass" "${ms}" "times"

    ms=$(grep "total time taken by event execution" "${logfile}" | awk '{print $NF}')
    add_metric "${tc}-total-time-taken-by-event-execution" "pass" "${ms}" "s"

    for i in min avg max approx; do
        ms=$(grep -m 1 "$i" "${logfile}" | awk '{print substr($NF,1,length($NF)-2)}')
        add_metric "${tc}-response-time-$i" "pass" "${ms}" "ms"
    done

    ms=$(grep "events (avg/stddev)" "${logfile}" |  awk '{print $NF}')
    ms_avg=$(echo "${ms}" | awk -F'/' '{print $1}')
    ms_stddev=$(echo "${ms}" | awk -F'/' '{print $2}')
    add_metric "${tc}-events-avg" "pass" "${ms_avg}" "times"
    add_metric "${tc}-events-stddev" "pass" "${ms_stddev}" "times"

    ms=$(grep "execution time (avg/stddev)" "${logfile}" |  awk '{print $NF}')
    ms_avg=$(echo "${ms}" | awk -F'/' '{print $1}')
    ms_stddev=$(echo "${ms}" | awk -F'/' '{print $2}')
    add_metric "${tc}-execution-time-avg" "pass" "${ms_avg}" "s"
    add_metric "${tc}-execution-time-stddev" "pass" "${ms_stddev}" "s"
}

# Test run.
for tc in ${TESTS}; do
    echo
    info_msg "Running sysbench ${tc} test..."
    logfile="${OUTPUT}/sysbench-${tc}.txt"
    case "${tc}" in
        percpu)
            processor_id="$(awk '/^processor/{print $3}' /proc/cpuinfo)"
            for i in ${processor_id}; do
                taskset -c "$i" sysbench --num-threads=1 --test=cpu run | tee "${logfile}"
                general_parser "$i"
            done
            ;;
        cpu|threads|mutex)
            sysbench --num-threads="${NUM_THREADS}" --test="${tc}" run | tee "${logfile}"
            general_parser
            ;;
        memory)
            sysbench --num-threads="${NUM_THREADS}" --test=memory run | tee "${logfile}"
            general_parser

            ms=$(grep "Operations" "${logfile}" | awk '{print substr($4,2)}')
            add_metric "${tc}-ops" "pass" "${ms}" "ops"

            ms=$(grep "transferred" "${logfile}" | awk '{print substr($4, 2)}')
            units=$(grep "transferred" "${logfile}" | awk '{print substr($5,1,length($NF)-1)}')
            add_metric "${tc}-transfer" "pass" "${ms}" "${units}"
            ;;
        fileio)
            mkdir fileio && cd fileio
            for mode in seqwr seqrewr seqrd rndrd rndwr rndrw; do
                tc="fileio-${mode}"
                logfile="${OUTPUT}/sysbench-${tc}.txt"
                sync
                echo 3 > /proc/sys/vm/drop_caches
                sleep 5
                sysbench --num-threads="${NUM_THREADS}" --test=fileio --file-total-size=2G --file-test-mode="${mode}" prepare
                # --file-extra-flags=direct is needed when file size is smaller then RAM.
                sysbench --num-threads="${NUM_THREADS}" --test=fileio --file-extra-flags=direct --file-total-size=2G --file-test-mode="${mode}" run | tee "${logfile}"
                sysbench --num-threads="${NUM_THREADS}" --test=fileio --file-total-size=2G --file-test-mode="${mode}" cleanup
                general_parser

                ms=$(grep "transferred" "${logfile}" | awk '{print substr($NF, 2,(length($NF)-8))}')
                units=$(grep "transferred" "${logfile}" | awk '{print substr($NF,(length($NF)-6),6)}')
                add_metric "${tc}-transfer" "pass" "${ms}" "${units}"

                ms=$(grep "Requests/sec" "${logfile}" | awk '{print $1}')
                add_metric "${tc}-ops" "pass" "${ms}" "ops"
            done
            cd ../
            ;;
        oltp)
            # Use the same passwd as lamp and lemp tests.
            mysqladmin -u root password lxmptest  > /dev/null 2>&1 || true
            # Delete sysbench in case it exists.
            mysql --user='root' --password='lxmptest' -e 'DROP DATABASE sysbench' > /dev/null 2>&1 || true
            # Create sysbench database.
            mysql --user="root" --password="lxmptest" -e "CREATE DATABASE sysbench"

            sysbench --num-threads="${NUM_THREADS}" --test=oltp --db-driver=mysql --oltp-table-size=1000000 --mysql-db=sysbench --mysql-user=root --mysql-password=lxmptest prepare
            sysbench --num-threads="${NUM_THREADS}" --test=oltp --db-driver=mysql --oltp-table-size=1000000 --mysql-db=sysbench --mysql-user=root --mysql-password=lxmptest run | tee "${logfile}"

            # Parse test log.
            general_parser

            for i in "read" write other total; do
                ms=$(grep "${i}:" "${logfile}" | awk '{print $NF}')
                add_metric "${tc}-${i}-queries" "pass" "${ms}" "queries"
            done

            for i in transactions deadlocks "read/write requests" "other operations"; do
                ms=$(grep "${i}:" sysbench-oltp.txt | awk '{print substr($(NF-2),2)}')
                i=$(echo "$i" | sed 's/ /-/g; s|/|-|')
                add_metric "${tc}-${i}" "pass" "${ms}" "ops"
            done

            # cleanup
            mysql --user='root' --password='lxmptest' -e 'DROP DATABASE sysbench'
            ;;
    esac
done

sysbench.yaml

metadata:
    name: sysbench
    format: "Lava-Test-Shell Test Definition 1.0"
    description: "SysBench is a modular, cross-platform and multi-threaded
                  benchmark tool for evaluating OS parameters that are
                  important for a system running a database under intensive
                  load. Current features allow to test fileio, cpu, memory,
                  threads, mutex and oltp."
    maintainer:
        - chase.qi@linaro.org
    os:
        - debian
        - ubuntu
        - fedora
        - centos
        - openembedded
    scope:
        - performance
    environment:
        - lava-test-shell
    devices:
        - hi6220-hikey
        - apq8016-sbc
        - mustang
        - moonshot
        - thunderX
        - d03
        - d05

params:
    # "${TESTS}" could be one or more of the following tests.
    # TESTS: cpu memory threads mutex fileio oltp
    # TESTS: "oltp"
    TESTS: "percpu cpu memory threads mutex fileio"
    # Number of threads to run.
    # Run $(nproc) threads by default.
    NUM_THREADS: "NPROC"
    SKIP_INSTALL: "false"

run:
    steps:
        - cd ./automated/linux/sysbench/
        - ./sysbench.sh -n "${NUM_THREADS}" -t "${TESTS}" -s "${SKIP_INSTALL}"
        - ../../utils/send-to-lava.sh ./output/result.txt