Phoronix survey response

= Phoronix Test Suite survey response = Phoronix Test Suite (PTS) survey response provided by Michael Larabel

Survey Questions

 * What is the name of your test framework? Phoronix Test Suite (PTS) / Phoromatic.

Phoromatic is sort of separate technically but is part of the same PTS package. Phoromatic is what provides the client/server processes for those wanting test orchestration across a network or other environments as opposed to just firing off one-time benchmarks on a system.

Does your test framework:

source code access

 * access source code repositories for the software under test? yes - if open source
 * access source code repositories for the test software?
 * include the source for the test software?
 * provide interfaces for developers to perform code reviews? no
 * detect that the software under test has a new version? no
 * if so, how? n/a
 * detect that the test software has a new version? no

The Phoronix Test Suite / Phoromatic accesses the source code of the software under test if the software under test happens to be open-source and intended to be built from source as opposed to relying upon x86/x86_64/ARM release binaries for closed-source projects. The Phoronix Test Suite makes no enforcement of the software under test being open vs. closed-source or if built from source or just relying upon a static binary.

The test profiles (test scripts) most often are open-source and publicly available but the Phoronix Test Suite / Phoromatic also allows for internal/private/behind-the-firewall tests that are not public.

PTS will fetch all of the needed files whether it be source or binaries based upon the information specified in the downloads.xml package from download URLs, finding them on Phoromatic Server LAN caches based on file hashes, or similar caches/repositories.

There are tests by the Phoronix Test Suite (often with a "system/" prefix) where PTS if running say system/blender, PTS will be fetching Blender from the system's package management archive and using that as the basis for the test (or any other system-wide / in PATH Blender). Or there is the 'default' blender test profile where it's downloading the official Blender Linux/Windows packages from Blender3D.org and using those statically defined packages for all benchmarking.

That's similar to how many other tests exist of having options to either use a distribution/system-supplied package or to be building (or downloading) it from upstream project's servers/repositories. Of the open-source tests, PTS will almost always use the source-based tests for maximum architecture support, allowing to test compilers, etc, over using any stock binaries the project might publish. There isn't any cases where say PTS/me is building binaries and then hosting them somewhere and having PTS clients use that. The preference is certainly to be as close to upstream programs as possible.

test definitions
Does your test system:
 * have a test definition repository? yes, for public tests. PTS calls them "profiles"
 * if so, what data format or language is used? XML and bash scripts

Does your test definition include: test-profile.xsd has the following: RequiresRoot, RequiresDisplay, RequiresNetwork, RequiresInternet EnvironmentSize, EnvironmentTestingSize, SystemDependencies, and SupportedArchitectures and SupportedPlatforms.
 * source code (or source code location)? yes, if available, but can be binary also
 * dependency information?

These are used for checks to be sane about whether to run/install a test, warn the user, or not. For example, RequiresRoot can prompt the user for their sudo password, but if PTS is being run in batch/unattended mode, it knows to just error out running that test rather than hanging on a possible sudo prompt.

For graphics tests or those enabling RequiresDisplay, if no display server is running, it will just skip over running those and alert the user.

The SupportedPlatforms / SupportedArchitectures is for indicating if a test will only work on certain OS/archs and to then not advertise these tests to users on unsupported software/hardware.

SystemsDependencies on distributions with reverse-dependency solving capability will figure out what package provides a given header file / binary in order to install it. Such as for the tests that are trying to test the OS-supplied SQLite, Blender, or other common tests.

The Environment tags are mainly for reference purposes so the user can know ahead of time approximate storage requirements as well as a basic check by PTS to alert the user if they don't have enough space (useful when benchmarking on ARM SBCs with small SD cards). 

 There is an extensive system for managing 'external-test-dependencies'

"external dependencies" allows the installation of packages needed for a given test to successfully build/run. A list of supported OS/distributions in that system and their packages can be found @ https://github.com/phoronix-test-suite/phoronix-test-suite/tree/master/pts-core/external-test-dependencies_xml

That is a static list of mapping generic names to various distribution-specific packages that are then passed to the package manager at installation.

Complementary to the 'external dependencies' is also the 'system dependencies' as outlined above where on distributions with reverse dependency solving support, PTS just passes the file/binary to search for and has the package manager spit out the package name rather than having to rely on static lists... But with not all distributions / package managers offering that functionality, these XML based lists are still maintained.

The default behavior is that PTS installs the dependencies for a test as it's needed. PTS will check to see if the necessary files are on the system ahead of test installation otherwise will figure out what dependencies are needed and go ahead and call the package manager to install the files.

PTS currently doesn't attempt to do temporary/removal of packages, simply as to be conservative... To avoid potentially uninstalling some packages a user may have want or already installed and to avoid building a more complicated abstraction layer on top of the package managers for trying to keep tracking of what it should end up uninstalling or not, the most sane approach has just been to install the packages (and the user can see what packages are installed) and if they want to remove those packages later on they can do so.

For a majority of the tests though, just a standard compiler toolchain is needed and not often many exotic packages unless really going for some unique workload with a long list of dependencies. 

These are in the install.sh (or install_linux.sh / install_bsd.sh / install_windows.sh) files. A Git copy of all the tests with their scripts can be found @ https://github.com/phoronix-test-suite/test-profiles 
 * execution instructions? yes

TestSettings:Default:Arguments and TestSettings:Option:Menu:Entry:Name and Value and other items in TestSettings have elements that are about the user interface for controlling and configuring a test.
 * command line variants? yes, as well as interactive test arguments

There are tags are for offering the user options such as what size disk tests to run, what resolution they might want to run a benchmark at, and other tunables. 

PTS passes the options as command line arguments to the test script, but it's also accessible through environment variables.
 * environment variants? yes


 * setup instructions?
 * cleanup instructions?
 * if anything else, please describe: results parser - see below

The test profile meta-data specification is documented here: https://github.com/phoronix-test-suite/phoronix-test-suite/blob/master/pts-core/openbenchmarking.org/schemas/test-profile.xsd

There is also the result parser specification for how test profiles express how to read the "result" out of any log files : https://github.com/phoronix-test-suite/phoronix-test-suite/blob/master/pts-core/openbenchmarking.org/schemas/results-parser.xsd

It's been a pain to get here, but the parser seems to be working out now and working generally fine with all new tests I end up adding... Rarely do I need to do parser changes/additions to PTS these days.

Most other systems seem to use regex or similar, but was trying to make something that is user-friendly for even non-developers to quickly glance at and see what's going on without having to think much about it, etc.

Does your test system: See here: https://github.com/phoronix-test-suite/test-profiles
 * provide a set of existing tests? yes
 * if so, how many? around 400 unique core test profiles

And here are some examples:

Here's like a fairly basic test with smallpt: https://github.com/phoronix-test-suite/test-profiles/tree/master/pts/smallpt-1.1.0

And then like a more complex example with HPC Challenge: https://github.com/phoronix-test-suite/test-profiles/tree/master/pts/hpcc-1.2.4

Or blender as another popular one: https://github.com/phoronix-test-suite/test-profiles/tree/master/pts/blender-1.4.1 </b>

build management
Does your test system:
 * build the software under test (e.g. the kernel)?
 * build the test software?
 * build other software (such as the distro, libraries, firmware)?
 * support cross-compilation?
 * require a toolchain or build system for the SUT?
 * require a toolchain or build system for the test software?
 * come with pre-built toolchains?
 * store the build artifacts for generated software?
 * in what format is the build metadata stored (e.g. json)?
 * are the build artifacts stored as raw files or in a database?
 * if a database, what database?

'''The Phoronix Test Suite / Phoromatic basically provides for when a test is to occur, any pre/post installation and pre/post run hooks to run. From there bash scripts or similar can be run to cater to setting up the environment depending upon whether the user is each time building a new kernel and rebooting / building a fresh compiler / updating the OS from the default OS package set / etc. So it doesn't enforce much of a direct work-flow on users but leaves it up to them to customize the workflow to their environment.

The pre/post installation and run hooks are arbitrary scripts provided by the user.</b>

Test scheduling/management
Does your test system:
 * check that dependencies are met before a test is run? yes


 * schedule the test for the DUT? yes - see below
 * select an appropriate individual DUT based on SUT or test attributes?
 * reserve the DUT?
 * release the DUT?

'''Yes, if going after device targeting there is also support in Phoromatic for group/anything targeting based upon the preferences and devices connected. There is support for scheduling on a timed basis, whenever an API/REST call is made to the Phoromatic server (if wanting to hook it up to a per-commit environment or other arbitrary events, etc.'''


 * install the software under test to the DUT? Yes
 * install required packages before a test is run? Yes
 * require particular bootloader on the DUT? No
 * deploy the test program to the DUT? If it's defined by the test profile or the arbitrary hooks/scripts.
 * prepare the test environment on the DUT? If the user defined it as part of their scripts.
 * start a monitor (another process to collect data) on the DUT? Yes

PTS supports recording perf events, various hardware/software thermal/voltage/fan/power sensors in a separate thread and collecting that with the results.

A lot of this is in the Phodevi system. The Phodevi (Phoronix Device Interface) tries to be like a HAL to abstract as much of the hardware/software details as possible. It's helped in porting to macOS/Windows/Solaris/BSD more easily and with supporting the vast array of different Linux interfaces such as around Linux GPU details has generally been a headache to maintain with many driver-specific queries.</b>


 * start a monitor on external equipment?
 * initiate the test on the DUT? Yes
 * clean up the test environment on the DUT? In terms of uninstalling tests if that is desired and any other actions specified by the user in the -post scripts.

DUT control
Does your test system:
 * store board configuration data? No - this is left up to the user customizing the environment for their test intentions
 * in what format?
 * store external equipment configuration data? No
 * in what format?
 * power cycle the DUT? Built-in is just standard reboot support or from the Phoromatic Server is also the ability to WoL the devices under test.
 * monitor the power usage during a run? Yes if the user enables that setting. By default it supports various IPMI power sensors, WattsUp Pro power meters, and various per-component wattage sensors like on various GPUs.
 * gather a kernel trace during a run? Not by default but can be done via a PTS module.
 * claim other hardware resources or machines (other than the DUT) for use during a test? No
 * reserve a board for interactive use (ie remove it from automated testing)? If a "maintenace mode" button is clicked from the Phoromatic server UI, the board will boot up and block all testing from happening so administrator(s) can do any maintenance or other non-test tasks.
 * provide a web-based control interface for the lab? Yes
 * provide a CLI control interface for the lab? Basic CLI offered at this point as well as an in-development WebSockets based communication interface.

Run artifact handling
Does your test system:
 * store run artifacts yes - raw system logs
 * in what format? bundled .tar.gz or .zip depending on OS
 * put the run meta-data in a database?
 * if so, which database?
 * parse the test logs for results? Yes
 * convert data from test logs into a unified format? Yes
 * if so, what is the format? '''Phoronix Test Suite Result XML format - https://github.com/phoronix-test-suite/phoronix-test-suite/blob/master/pts-core/openbenchmarking.org/schemas/result-file.xsd
 * evaluate pass criteria for a test? Yes

Yes the Phoronix Test Suite can support ignoring first runs or other arbitrary runs, specify behavior for how many runs should be done by default for accurate readings (and dynamically increase the run count if high variance), and also allow besides quantitative results to also handle qualitative results like screenshots from GPU rendering / pass-fail tests and other data inputs.

The pass criteria is part of the parser process, and is defined in the XML discussed above.</b>


 * do you have a common set of result names?
 * if so, what are they?
 * How is run data collected from the DUT? Currently pushing from the DUT but there is a WIP WebSockets implementation.
 * How is run data collected from external equipment? Via the Phoronix Test Suite's Phodevi library (Phoronix Device Interface) that provides much of the OS hardware and software level abstractions.


 * Is external equipment data parsed?

User interface
Does your test system:
 * have a visualization system? Yes
 * show build artifacts to users? Yes
 * show run artifacts to users? Yes
 * do you have a common set of result colors? Unclear.
 * if so, what are they?
 * generate reports for test runs? Yes
 * notify users of test results by e-mail? Yes
 * can you query (aggregate and filter) the build meta-data? Via the CLI but not the web interface currently.
 * can you query (aggregate and filter) the run meta-data? Via the CLI but not the web interface currently.
 * what language or data format is used for online results presentation? HTML, XML, PDF
 * what language or data format is used for reports? PDF, CSV, TXT
 * does your test system have a CLI control tool? yes
 * what is it called? phoronix-test-suite

Languages:

 * what is the base language of your test framework core? 'Bash scripts - programs in other languages may be called Python, PHP, etc) as long as it's listed as an external dependency.

Most of PTS/Phoromatic itself is written in PHP... Started out that way as I had already written graphing code in PHP for Phoronix.com and other components and PTS started growing out organically. And that PHP is pretty much well supported on all architectures/platforms so was an easy base target when starting out. But PHP really isn't used by the test profiles/scripts or other elements as that's all left up to the user. </b>

What languages or data formats is the user required to learn? XML

Can a user do the following with your test framework:

 * manually request that a test be executed? Yes
 * see the results of recent tests? Yes
 * set the pass criteria for a test? Yes
 * set the threshold value for a benchmark test? Yes
 * set the list of testcase results to ignore? Yes
 * provide a rating for a test? (e.g. give it 4 stars out of 5) No
 * customize a test? Yes
 * alter the command line for the test program? Yes
 * alter the environment of the test program? Yes
 * specify to skip a testcase? Yes
 * set a new expected value for a test? No
 * edit the test program source? Yes
 * customize the notification criteria? yes - Basic customizations currently
 * customize the notification mechanism (eg. e-mail, text)
 * generate a custom report for a set of runs? Yes
 * save the report parameters to generate the same report in the future? Yes

Requirements
Does your test framework: PHP is required on both client and server, though just the PHP CLI components and not any LAMP stack.</b>
 * require minimum software on the DUT? yes
 * if so, what?: PHP 5.3+ (ideally PHP7) on Linux, macOS, Windows 10 / Windows Server 2016, BSDs, or Solaris.

PTS runs on the client under test as it's lightweight. It wouldn't be hard to have the PTS client SSH out to boxes, but that currently isn't done and there is no growing need for that with ARM/embedded devices continuing to get more powerful.</b>


 * require minimum hardware on the DUT?
 * If so, what?
 * require agent software on the DUT? No, the Phoronix Test Suite will dynamically pull in any additional dependencies needed by a given test.
 * If so, what agent?
 * is there optional agent software or libraries for the DUT?
 * require external hardware in your labs?

APIS
Does your test framework:
 * use existing APIs or data formats to interact within itself, or with 3rd-party modules?
 * have a published API for any of its sub-module interactions (any of the lines in the diagram)? Yes
 * Please provide a link or links to the APIs? https://github.com/phoronix-test-suite/phoronix-test-suite/tree/master/pts-core/openbenchmarking.org/schemas

Are they:
 * What is the nature of the APIs you currently use?
 * RPCs?
 * Unix-style? (command line invocation, while grabbing sub-tool output)
 * compiled libraries?
 * interpreter modules or libraries?
 * web-based APIs?
 * something else?

Relationship to other software:

 * what major components does your test framework use?
 * does your test framework interoperate with other test frameworks or software?
 * which ones?

Overview
Please list your major components here:   Basically Phoromatic is used for test orchestration across multiple systems, test scheduling, local collecting of test results, and also for serving as a download-cache. The Phoromatic Server can cache all the needed test files and distribute it to the client systems for behind-the-firewall type uses or simply for speeding up test install times and reducing bandwidth. </b> = Additional Data =
 * Phodevi - Phoronix Device Interface library - provides much of the OS hardware and software level abstractions.
 * Phoronix Test Suite client - The main client used for facilitating benchmarking and acting as a client when communicating with a Phoromatic Server for running benchmarks.
 * OpenBenchmark.org - Central "cloud" repository for hosting test profiles/suites independent of the PTS client releases as well as for public results aggregation and data analytics platform.
 * pts_Graph - results visualizer
 * Phoromatic Server -

test-profile schema
Here is the test-profile schema (extracted from test-profile.xsd

"PhoronixTestSuite"> "TestInformation" "Title" type="TitleString" "AppVersion" type="xs:string" "Description" type="DescriptionString" "ResultScale" type="xs:string" "Proportion" type="Proportion" "ResultQuantifier" type="ResultQuantifier" "DisplayFormat" type="DisplayFormat" "SubTitle" type="xs:string" "Executable" type="xs:string" "TimesToRun" type="TimesToRun" "IgnoreRuns" type="xs:string" "InstallationAgreement" type="xs:string" "PreInstallMessage" type="xs:string" "PostInstallMessage" type="xs:string" "TestProfile"> "Version" type="VersionString"> "SupportedPlatforms" type="PlatformString" "SoftwareType" type="SoftwareType" "TestType" type="TestType" "License" type="License" "Status" type="ProfileStatus" "SupportedArchitectures" type="xs:string" "ExternalDependencies" type="xs:string" "Extends" type="TestExtends" "RequiresRoot" type="Boolean" "EnvironmentSize" type="PositiveDecimal" "EnvironmentTestingSize" type="PositiveDecimal" "EstimatedTimePerRun" type="PositiveInteger" "ProjectURL" type="URLString" "RequiresCoreVersionMin" type="CoreVersion" "RequiresCoreVersionMax" type="CoreVersion" "InternalTags" type="xs:string" "Maintainer" type="xs:string" "AllowResultsSharing" type="Boolean" "AutoSaveResults" type="Boolean" "SystemDependencies" type="xs:string" "TestSettings" "Default" "Arguments" type="xs:string" "PostArguments" type="xs:string" "AllowCacheShare" type="Boolean" "MinimumLength" type="PositiveDecimal" "MaximumLength" type="PositiveDecimal" "Option" "DisplayName" type="xs:string"> "Identifier" type="xs:string" "ArgumentPrefix" type="xs:string" "ArgumentPostfix" type="xs:string" "DefaultEntry" type="xs:string" "Menu" "Entry" maxOccurs="unbounded" "Name" type="xs:string" "Value" type="xs:string" "Message" type="xs:string"

results-parser.xsd
Here are fields in the results-parser file:

"PhoronixTestSuite"> "ResultsParser" "OutputTemplate" type="xs:string"> "MatchToTestArguments" type="xs:string" "ResultKey" type="xs:string" "LineHint" type="xs:string" "LineBeforeHint" type="xs:string" "LineAfterHint" type="xs:string" "ResultBeforeString" type="xs:string" "ResultAfterString" type="xs:string" "StripFromResult" type="xs:string" "StripResultPostfix" type="xs:string" "MultiMatch" type="xs:string" "DivideResultBy" type="PositiveInteger" "MultiplyResultBy" type="PositiveInteger" "ResultScale" type="xs:string" "ResultProportion" type="xs:string" "ResultPrecision" type="xs:string" "ArgumentsDescription" type="xs:string" "AppendToArgumentsDescription" type="xs:string" "FileFormat" type="xs:string" "TurnCharsToSpace" type="xs:string" "DeleteOutputBefore" type="xs:string" "DeleteOutputAfter" type="xs:string" "ImageParser" "SourceImage" type="xs:string"> "MatchToTestArguments" type="xs:string" "ImageX" type="PositiveInteger" "ImageY" type="PositiveInteger" "ImageWidth" type="PositiveInteger" "ImageHeight" type="PositiveInteger" "SystemMonitor" "Sensor" type="xs:string"> "PollingFrequency" type="PositiveInteger" "Report" type="MonitorReporting" "ExtraData" "Identifier" type="xs:string">