New test ideas

LTP system call test (already in Fuego)
LTP posix test suite (already in Fuego)
LSB-FHS - Linux Standard Base Filesystem Hierarchy Standard test
- See https://www.opengroup.org/testing/lsb-fhs/

Things to check in a newly created product [edit section]

After a change to the kernel or software stack (Linux distribution) on a product, the following items should be checked:

interrupts are working (at correct or expected rate)
modules that were expected are loaded correctly
file systems were mounted with correct permissions and attributes
busybox or toybox has all needed sub-commands
shell has needed features (should be part of posix test, but I suspect it's not)
kselftest - for kernel features
libraries that were expected are present

Tests from other distros or projects [edit section]

Some existing Linux distributions or project have their own selection of tests, for the package they include. Here are some candidate project to get test ideas from (if not the tests themselves).

Apertis tests [edit section]

See https://gitlab.apertis.org/pkg/apertis-tests.git

AGL tests [edit section]

See https://git.automotivelinux.org/src/qa-testdefinitions/tree/common/scripts

busybox presence test [edit section]

see https://git.automotivelinux.org/src/qa-testdefinitions/tree/common/scripts/busybox.sh

service availability test [edit section]

https://git.automotivelinux.org/src/qa-testdefinitions/tree/common/scripts/service-check.sh

CIP tests [edit section]

For overall testing strategy, see https://wiki.linuxfoundation.org/civilinfrastructureplatform/ciptesting/centalisedtesting

Tests are at: https://gitlab.com/cip-project/cip-testing/cip-kernel-tests

Tests are at: https://gitlab.com/cip-project/cip-testing/test-definitions/-/tree/master/automated/linux

CKI (RedHat) tests [edit section]

Debian tests [edit section]

Yocto Project tests (ptests?) [edit section]

Tests by technology area or subsystem [edit section]

Security tests [edit section]

OpenVAS [edit section]

check out OpenVAS - a tool that checks for CVEs (from OSSJ)
- see http://www.openvas.org/

cvecheck [edit section]

see https://github.com/clearlinux/cve-check-tool (last updated 5 years ago)
For yocto project, see https://git.yoctoproject.org/poky/tree/meta/classes/cve-check.bbclass

Lynis [edit section]

Lynis is a vulnerability scanner for Linux systems.

see https://github.com/CISOfy/lynis

Memory [edit section]

mmtests [edit section]

mmtests by Mel Gorman

https://github.com/gormanm/mmtests

filesystem [edit section]

xfstests [edit section]

xfstests seems to be the new standard for measuring Linux file system performance. We should include this test in fuego.

See the following for more information:

https://git.kernel.org/cgit/fs/xfs/xfstests-dev.git/
- clone with 'git clone git://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git'
- I think this is the main upstream repository for xfstests (the repository at http://oss.sgi.com/ has been deprecated)
An automated xfstests infrastructure using kvm
- Ted Ts'o's work on automating xfstests
Toward better testing
- Dave Chimmer's report on the status of xfstests at an event in 2014

block layer performance measurement [edit section]

Possibly something simple like 'time dd ...' is useful for catching some things (and it's short).

Here is a post from Linus Walleij about using a simple dd to measure block layer performance. He found a regression of performance using the MQ block layer scheduler, using this.

I got blk-mq running for MMC/SD today and I see a gross performance
regression, from 37 MB/s to 27 MB/s on Ux500 7.38 GB eMMC
with a simple dd test:

BEFORE switching to MQ:

time dd if=/dev/mmcblk3 of=/dev/null bs=1M count=1024
1073741824 bytes (1.0GB) copied, 27.530335 seconds, 37.2MB/s
real    0m 27.54s
user    0m 0.02s
sys     0m 7.56s

AFTER switching to MQ:

time dd if=/dev/mmcblk3 of=/dev/null bs=1M count=1024
1073741824 bytes (1.0GB) copied, 37.170990 seconds, 27.5MB/s
real    0m 37.18s
user    0m 0.02s
sys     0m 7.32s

I will however post my hacky patch as a RFD to the blockdevs and
the block maintainers, along with the numbers and a speculation
about what may be causing it. asynchronous requests (request
pipelining) is one thing, another thing is front/back merge in
the block layer I guess.

Bus testing [edit section]

serial port [edit section]

i2c [edit section]

USB testing [edit section]

CAN bus testing [edit section]

From agl-discussions list Dec 13: https://lists.linuxfoundation.org/pipermail/automotive-discussions/2016-December/003056.html

I'm interested your benchmark amb can data. http://docs.automotivelinux.org/docs/apis_services/en/dev/reference/iotbzh2016/signaling/AGL-AppFW-CAN-Signaling-Benchmark.pdf

I want to test amb d-bus can data benchmark, So can you share your used "can data" and "amd configuration" ?

This test apparently has a CAN packet injector, written by Cogent.

Miscellaneous [edit section]

modetest, ifconfig, ping, rtcwake
inspect /sys/kernel/debug/devices_deferred
From Marek's diagram:
- boot, bash, defer, rtc0, rtc1, fbtest, ping, wifi, bt, alsa, cpu_hp, sleep, sleep2, ping2, reboot, off