Difference between revisions of "Test Systems"

From eLinux.org
Jump to: navigation, search
m (Bot (Edward's framework))
 
 
(60 intermediate revisions by 13 users not shown)
Line 1: Line 1:
 
Here is a quick list of different test systems (and test projects) for Linux:
 
Here is a quick list of different test systems (and test projects) for Linux:
  
This information has also been posted to the OSDL test-tools wiki.
+
== Test Projects ==
 +
Clearly, a number of open source projects in this realm exist.  This page intends to collect descriptions and links to these projects
  
See: http://developer.osdl.org/dev/test_tools/index.php/Test_Projects
+
=== Test Automation Tools ===
 +
* 0-day
 +
* [http://test.kernel.org/autotest Autotest]
 +
* [https://avocado-framework.github.io/ Avocado] (Autotest next generation)
 +
* buildbot
 +
* [https://ci-rt.linutronix.de/RT-Test/about.jsp ci-rt] - Linutronix Test Environment [https://github.com/ci-rt/docs GitHub]
 +
* [https://cki-project.org CKI Project - Continuous Kernel Integration @ Red Hat] (code available at [https://github.com/cki-project GitHub] and [https://gitlab.com/cki-project GitLab])
 +
* [http://fuegotest.org/ Fuego Test system] ([https://bitbucket.org/tbird20d/fuego bitbucket repository])
 +
* Gentoo Kernel CI
 +
* [https://github.com/hms-networks/hottest hottest] - inspired by LAVA and Fuego
 +
* [https://jenkins.io/ Jenkins] - Automation server for building, deploying and automating any project
 +
* [https://github.com/mcgrof/kdevops/ kdevops] ([https://people.kernel.org/mcgrof/kdevops-a-devops-framework-for-linux-kernel-development blog description])
 +
* [https://github.com/Zenterio/opensourcelib/tree/master/k2 K2] - Test framework and tools for embedded testing
 +
* [https://kernelci.org/ KernelCI] ([https://github.com/kernelci/kernelci-doc/wiki wiki and docs])
 +
* kerneltests
 +
* [[Ktest]] Automated kernel testing
 +
* [https://github.com/oracle/ktf KTF] Kernel Test Framework
 +
* [https://github.com/labgrid-project/labgrid labgrid] - Embedded board control Python library [https://labgrid.readthedocs.io/en/latest/ Read The Docs]
 +
* [https://lavasoftware.org/ LAVA Software Community Project] - [https://git.lavasoftware.org/ (Community Project GitLab repository)]
 +
* [http://arago-project.org/wiki/index.php/Opentest Opentest]
 +
* [https://www.phoronix-test-suite.com/ Phoronix Test Suite]
 +
* [https://github.com/ci-rt/r4d r4d] - Remote For Device-under-test and (Part of ci-rt)
 +
* [https://testing.108.redhat.com/ Red Hat Test Project] (Dead link)
 +
* [https://github.com/SamsungSLAV/slav Samsung SLAV] - Managing devices, tesframework, workers
 +
* [https://github.com/google/syzkaller syzbot] - Unsupervised coverage-guided kernel fuzzer
 +
* [https://tbot.tools/ tbot] - Testing/automation tool that is focused on usage in embedded development [https://github.com/Rahix/tbot GitHub]
 +
* [https://intel.github.io/tcf/ TCF (Test Case Framework)] - Simplifies the creation and execution of test cases
 +
* Xilinx Test
 +
* [https://wiki.yoctoproject.org/wiki/Ptest Yocto Project] - Includes many test package and test elements (ptest)
 +
* [https://github.com/znailnetem/znail Znail] - RPi network emulator, emulate packet delay, loss, duplication, corruption, rate, override DNS
  
There was a testing summit held at [[Linux World]] in San Francisco in August of 2006.
+
See also https://bottest.wiki.kernel.org/ - which has a list of projects that test the kernel.
The minutes from the meeting are at:
 
http://lists.osdl.org/pipermail/test-tools/2006-August/000000.html
 
  
As a result of the meeting, a mailing list and wiki were created.
+
See also http://www.opensourcetesting.org/category/testing-tools-overview/functional/page/1/
These resources are intended to be used to coordinate the efforts
 
of the disparate testing projects in Linux. The list info is at:
 
https://lists.osdl.org/mailman/listinfo/test-tools
 
  
The wiki is at:
+
=== Test Suites ===
http://developer.osdl.org/dev/test_tools/index.php/Main_Page
 
  
 +
* [https://github.com/osandov/blktests blktests]
 +
* [https://github.com/linux-test-project/ltp Linux Test Project]
 +
* [http://processors.wiki.ti.com/index.php/LTP-DDT LTP-DDT]
 +
* [https://www.kernel.org/doc/html/latest/dev-tools/kselftest.html kselftest] - See notes at [[kselftest Notes]]
 +
* [https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git/ xfstests]
  
== old page content ==
+
=== Static Analysis ===
This content was moved to the OSDL test-tools wiki at:
 
http://developer.osdl.org/dev/test_tools/index.php/Test_Projects
 
  
It is preserved here only temporarily. (TRB - Aug, 2006)
+
* [http://tree.celinuxforum.org/CelfPubWiki/Sparse Sparse]
 +
* [http://smatch.sourceforge.net/ Smatch]
 +
* [http://clang.llvm.org/StaticAnalysis.html clang]
  
ABAT - https://sourceforge.net/projects/abat/
+
== Automated kernel test results ==
- does a download/build/boot regression test for several mainline kernel trees, as soon as new versions are available
+
Here are some locations where automated tests of the Linux kernel can be viewed:
- test results matrix is available here:
 
    - http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/regression_matrix.html
 
  
Linux Test Project
+
=== kernelci ===
- http://ltp.sourceforge.net/
+
https://kernelci.org/job/
  
CELF's Proposed Test Lab
+
=== kerneltests ===
  - [[Open Test Lab]]
+
https://kerneltests.org/waterfall
  
[[K Auto Build]]  - http://armlinux.simtec.co.uk/index.html
+
== Commercial Offerings ==
  - [[KAuto Build]] project for ARM architectures
+
Several vendors offer paid-for services:
  - Article about the site is at: http://www.linuxdevices.com/news/NS4646877354.html
 
  
There was a testing summit held
+
* [https://www.gooddevicelabs.co.uk/canaryqa/ Good Device Labs - CanaryQA]
 +
* [https://www.timesys.com/open-source-embedded/board-farm-cloud/ TimeSys Board Farm Cloud]
  
I wanted to report that there was a testing summit held at
+
== Bug tracking system ==
[[Linux World]] in San Francisco a few weeks ago. I represented
+
 
the CELF test lab effort at the meeting.
+
A '''bug tracking system''' is a software application that is designed to help quality assurance and computer programmers keep track of reported software bugs in their work.
 +
 
 +
Many bug-tracking systems, such as those used by most open source software projects, allow users to enter bug reports directly. Other systems are used only internally in a company or organization doing software development. Typically bug tracking systems are integrated with other software project management applications.
 +
 
 +
Having a bug tracking system is extremely valuable in software development, and they are used extensively by companies developing software products.
 +
 
 +
=== Components ===
 +
 
 +
A major component of a bug tracking system is a database that records facts about known bugs. Facts may include the time a bug was reported, its severity, the erroneous program behavior, and details on how to reproduce the bug; as well as the identity of the person who reported it and any programmers who may be working on fixing it.
 +
 
 +
Typical bug tracking systems support the concept of the life cycle for a bug which is tracked through status assigned to the bug. A bug tracking system should allow administrators to configure permissions based on status, move the bug to another status, or delete the bug. The system should also allow administrators to configure the bug statuses and to what status a bug in a particular status can be moved to.
 +
 
 +
=== Usage ===
 +
 
 +
In a corporate environment, a bug-tracking system may be used to generate reports on the productivity of programmers at fixing bugs. However, this may sometimes yield inaccurate results because different bugs may have different levels of severity and complexity. The severity of a bug may not be directly related to the complexity of fixing the bug. There may be different opinions among the managers and architects.
 +
 
 +
A ''local bug tracker (LBT)'' is usually a computer program used by a team of application support professionals to keep track of issues communicated to software developers. Using an LBT allows support professionals to track bugs in their "own language" and not the "language of the developers." In addition, LBT use allows a team of support professionals to track specific information about users who have called to complain that may not always be needed in the actual development queue (thus, there are two tracking systems when an LBT is in place.)
 +
 
 +
=== External links ===
 +
 
 +
* [http://trac.edgewall.org/ Trac ] - Web-based software project management and bug/issue tracking system.
 +
* [http://www.bugzilla.org/ Bugzilla ] - Bug tracking used by the Mozilla projects. Inherently web-based, written in Perl , and uses MySQL as its database back-end. Open-Source.
 +
* [http://www.bugreport.ir/? Bugreport is a place to publish the security advisories ]
 +
* [http://www.chiark.greenend.org.uk/~sgtatham/bugs.html How to Report Bugs Effectively]

Latest revision as of 01:25, 1 June 2020

Here is a quick list of different test systems (and test projects) for Linux:

Test Projects

Clearly, a number of open source projects in this realm exist. This page intends to collect descriptions and links to these projects

Test Automation Tools

See also https://bottest.wiki.kernel.org/ - which has a list of projects that test the kernel.

See also http://www.opensourcetesting.org/category/testing-tools-overview/functional/page/1/

Test Suites

Static Analysis

Automated kernel test results

Here are some locations where automated tests of the Linux kernel can be viewed:

kernelci

https://kernelci.org/job/

kerneltests

https://kerneltests.org/waterfall

Commercial Offerings

Several vendors offer paid-for services:

Bug tracking system

A bug tracking system is a software application that is designed to help quality assurance and computer programmers keep track of reported software bugs in their work.

Many bug-tracking systems, such as those used by most open source software projects, allow users to enter bug reports directly. Other systems are used only internally in a company or organization doing software development. Typically bug tracking systems are integrated with other software project management applications.

Having a bug tracking system is extremely valuable in software development, and they are used extensively by companies developing software products.

Components

A major component of a bug tracking system is a database that records facts about known bugs. Facts may include the time a bug was reported, its severity, the erroneous program behavior, and details on how to reproduce the bug; as well as the identity of the person who reported it and any programmers who may be working on fixing it.

Typical bug tracking systems support the concept of the life cycle for a bug which is tracked through status assigned to the bug. A bug tracking system should allow administrators to configure permissions based on status, move the bug to another status, or delete the bug. The system should also allow administrators to configure the bug statuses and to what status a bug in a particular status can be moved to.

Usage

In a corporate environment, a bug-tracking system may be used to generate reports on the productivity of programmers at fixing bugs. However, this may sometimes yield inaccurate results because different bugs may have different levels of severity and complexity. The severity of a bug may not be directly related to the complexity of fixing the bug. There may be different opinions among the managers and architects.

A local bug tracker (LBT) is usually a computer program used by a team of application support professionals to keep track of issues communicated to software developers. Using an LBT allows support professionals to track bugs in their "own language" and not the "language of the developers." In addition, LBT use allows a team of support professionals to track specific information about users who have called to complain that may not always be needed in the actual development queue (thus, there are two tracking systems when an LBT is in place.)

External links