|
|
VTS Kselftest Workflow
|
|
|
======================
|
|
|
|
|
|
A Guide to Developing Kselftest for VTS/Android
|
|
|
|
|
|
What is Kselftest?
|
|
|
------------------
|
|
|
|
|
|
Kselftest is a collection of tests included within the Linux kernel repository
|
|
|
at tools/testing/selftests. The tests cover a broad range of areas but total
|
|
|
coverage (as of 4.14 anyway) is very spotty. Many of the tests do not apply to
|
|
|
Android platforms.
|
|
|
|
|
|
As of mid 2018 there are 23 tests in kselftest which run on ARM in VTS.
|
|
|
Almost all of them get executed in both 32-bit and 64-bit mode. There are many
|
|
|
tests which are not enabled because they have dependencies which are not met on
|
|
|
Android platforms.
|
|
|
|
|
|
How is Kselftest Run in VTS?
|
|
|
----------------------------
|
|
|
|
|
|
The Kselftest source is located at external/linux-kselftest in the Android
|
|
|
tree. It is a set of files (tools/testing/selftests/) copied into its own
|
|
|
repository from the Linux kernel repository. This is not an exact mirror of
|
|
|
upstream; there are outstanding changes to kselftest for it to work on Android
|
|
|
which have not yet been pushed upstream (contained in android/patches). In
|
|
|
addition to the kselftest source there is also the VTS wrapper around it. This
|
|
|
is located at test/vts-testcase/kernel/linux_kselftest. Some noteworthy
|
|
|
directories/files:
|
|
|
|
|
|
* `external/linux-kselftest/Android.bp`: Lists build rules for the kselftest modules built in Soong.
|
|
|
* `external/linux-kselftest/android/`: Contains Android-specific files, aside from Android.[bp, mk] at top level.
|
|
|
* `external/linux-kselftest/android/Android.kselftest.mk`: Lists build rules for the kselftest modules built under make.
|
|
|
* `external/linux-kselftest/android/kselftest_test_list.mk`: Lists all tests that will get pulled into VTS - VTS depends on this list.
|
|
|
* `external/linux-kselftest/android/README`: Lists details on why particular tests are not enabled in Android.
|
|
|
* `external/linux-kselftest/tools/testing/selftests`: Kselftest sources.
|
|
|
* `test/vts-testcase/kernel/linux_kselftest/testcases/`: tests to run in VTS kselftest suites are
|
|
|
specified in each of the test XML file, e.g, vts_linux_kselftest_arm_32.xml contains tests for 32 bit ARM platform.
|
|
|
|
|
|
Running Kselftest through atest
|
|
|
-------------------------------
|
|
|
You can run Kselftest tests with atest, which handles all the setup and build steps.
|
|
|
|
|
|
To run all 32 bit Kselftest tests:
|
|
|
* `atest vts_linux_kselftest_arm`
|
|
|
|
|
|
To run all 64 bit Kselftest tests:
|
|
|
* `atest vts_linux_kselftest_arm_64`
|
|
|
|
|
|
To run a single test:
|
|
|
* `atest vts_linux_kselftest_arm_64:timers_valid-adjtimex_arm_64`
|
|
|
|
|
|
Running Kselftest through VTS
|
|
|
-----------------------------
|
|
|
To run VTS kselftest through VTS, it must first be built. VTS is not device specific,
|
|
|
you need not compile it specifically for the device you wish to run it on, assuming
|
|
|
it is the same architecture.
|
|
|
* `. build/envsetup.sh`
|
|
|
* `lunch`
|
|
|
* `make -j vts`
|
|
|
|
|
|
Then open vts-tradefed and run the VTS tests (64 bit ARM platform as an example):
|
|
|
* `vts-tradefed`
|
|
|
* `vts-tf > run vts-kernel -m vts_linux_kselftest_arm_64`
|
|
|
|
|
|
If you regularly work with multiple devices it may be useful to specify the
|
|
|
specific device you wish to run VTS on via the serial number:
|
|
|
* `vts-tf > run vts-kernel -m vts_linux_kselftest_arm_64 -s 000123456789`
|
|
|
|
|
|
Or a specific test:
|
|
|
* `vts-tf > run vts-kernel -m vts_linux_kselftest_arm_64 -t net_socket`
|
|
|
|
|
|
Running Kselftest Directly
|
|
|
--------------------------
|
|
|
|
|
|
Running kselftest tests within VTS can be quite cumbersome, especially if you
|
|
|
are iterating a lot trying to debug something. Build and run kselftest tests
|
|
|
faster by doing
|
|
|
|
|
|
* `external/linux-kselftest$ mma`
|
|
|
* `external/linux-kselftest$ adb sync data`
|
|
|
|
|
|
The test cases will be located at `/data/nativetest{64,}/linux-kselftest/`.
|
|
|
|
|
|
Build just the kselftest test you care about with `make -j kselftest_<testname>` (listed in `external/linux-kselftest/android/kselftest_test_list.mk`).
|
|
|
|
|
|
What Outstanding Issues Exist?
|
|
|
------------------------------
|
|
|
|
|
|
The hotlist for kselftest bugs is
|
|
|
[kselftest-todo](https://buganizer.corp.google.com/hotlists/745928).
|
|
|
|
|
|
When you begin working on a kselftest bug please assign the bug to yourself so
|
|
|
that others know it is being worked on.
|
|
|
|
|
|
Testing x86_64
|
|
|
--------------
|
|
|
|
|
|
It is not advisable to run kselftest tests directly on your host unless you are fully
|
|
|
aware of what the tests will do and are okay with it. These tests may
|
|
|
destabilize your box or cause data loss. If you need to run tests on an x86
|
|
|
platform and are unsure if they are safe you should run them in emulation, in a
|
|
|
virtualized environment, or on a dedicated development x86 platform.
|
|
|
|
|
|
To run Kselftest tests for x86 platform, you can do:
|
|
|
* `atest vts_linux_kselftest_x86`
|
|
|
* `atest vts_linux_kselftest_x86_64`
|
|
|
|
|
|
Sending Fixes Upstream
|
|
|
----------------------
|
|
|
|
|
|
Kselftest is part of the upstream Linux kernel and as such you can use the
|
|
|
get_maintainers.pl script there to see who to send patches to. Here’s an
|
|
|
example using a commit already upstream:
|
|
|
|
|
|
```
|
|
|
smuckle@smuckle:~/repos/linux$ git show 352909b49ba | scripts/get_maintainer.pl
|
|
|
Shuah Khan <shuah@kernel.org> (maintainer:KERNEL SELFTEST FRAMEWORK,commit_signer:2/6=33%,authored:2/6=33%,added_lines:2/6=33%,removed_lines:3/6=50%)
|
|
|
Thomas Gleixner <tglx@linutronix.de> (commit_signer:2/6=33%)
|
|
|
Greg Kroah-Hartman <gregkh@linuxfoundation.org> (commit_signer:1/6=17%,authored:1/6=17%,added_lines:1/6=17%)
|
|
|
"Radim Krčmář" <rkrcmar@redhat.com> (commit_signer:1/6=17%)
|
|
|
Ingo Molnar <mingo@kernel.org> (commit_signer:1/6=17%,commit_signer:1/1=100%)
|
|
|
Andy Lutomirski <luto@kernel.org> (authored:2/6=33%,added_lines:2/6=33%,removed_lines:2/6=33%,authored:1/1=100%,added_lines:500/500=100%)
|
|
|
"Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> (authored:1/6=17%,added_lines:1/6=17%,removed_lines:1/6=17%)
|
|
|
linux-kernel@vger.kernel.org (open list)
|
|
|
linux-kselftest@vger.kernel.org (open list:KERNEL SELFTEST FRAMEWORK)
|
|
|
```
|
|
|
|
|
|
In summary patches should be sent to linux-kselftest@vger.kernel.org and
|
|
|
linux-kernel@vger.kernel.org, Shuah Khan (kselftest maintainer), and anyone
|
|
|
else who has touched the code in question. Standard Linux kernel coding style
|
|
|
and patch rules apply. They should be checkpatch (scripts/checkpatch.pl in the
|
|
|
kernel repository) clean and sent in plain text in canonical patch format. One
|
|
|
easy way to do this is by using git format-patch and git send-email.
|
|
|
|
|
|
Merging Fixes
|
|
|
-------------
|
|
|
|
|
|
When possible please merge fixes upstream first. Then cherrypick the change
|
|
|
onto aosp/master in external/linux-kselftest.
|
|
|
|
|
|
If your change cannot go upstream first for whatever reason, please commit a
|
|
|
patch for it into external/linux-kselftest/android/patches. This allows easier
|
|
|
tracking of the delta with upstream and streamlines upgrades to new kselftest
|
|
|
releases.
|
|
|
|
|
|
Updating Kselftest
|
|
|
------------------
|
|
|
|
|
|
To merge in a new upstream version of kselftest:
|
|
|
1. Do a git merge of the upstream tag with the "ours" policy, dropping all upstream changes. Do not commit yet.
|
|
|
2. Delete tools/testing/selftests and replace it with a copy from the upstream kernel at the same tag as used above.
|
|
|
3. Apply the patches in android/patches/, resolving conflicts as required.
|
|
|
4. Test on all supported kernel versions, ensuring that any tests currently enabled in VTS do not generate new failures.
|
|
|
5. Commit the merge.
|
|
|
|
|
|
If the runtime of kselftest changes significantly be sure to update the runtime-hint and test-timeout parameters to VTS in
|
|
|
test config files under
|
|
|
`test/vts-testcase/kernel/linux_kselftest/testcases/`.
|
|
|
|
|
|
Notes on Individual Tests
|
|
|
-------------------------
|
|
|
|
|
|
### bpf/
|
|
|
|
|
|
The bpf tests depend on the bpf and elf libraries, and several kernel
|
|
|
config options that are not required by Android.
|
|
|
|
|
|
### cpufreq/
|
|
|
|
|
|
Cpufreq is not required by Android.
|
|
|
|
|
|
### cpu-hotplug/
|
|
|
|
|
|
Cpu hotplug is not required by Android.
|
|
|
|
|
|
### filesystems/dnotify_test.c
|
|
|
|
|
|
This test has an endless loop in it. It is not currently run as part of the
|
|
|
kselftest suite.
|
|
|
|
|
|
### firmware/
|
|
|
|
|
|
These tests depend on a kernel module enabled by CONFIG_TEST_FIRMWARE, which is
|
|
|
not required by Android.
|
|
|
|
|
|
### ftrace/
|
|
|
|
|
|
Ftrace is a debug feature and not required by Android.
|
|
|
|
|
|
### gpio/
|
|
|
|
|
|
The gpio test requires a test kernel module and gpiolib, neither of which are
|
|
|
required by Android.
|
|
|
|
|
|
### ipc/
|
|
|
|
|
|
The ipc test requires CONFIG_EXPERT and CONFIG_CHECKPOINT_RESTORE be enabled,
|
|
|
neither of which are required by Android.
|
|
|
|
|
|
### kvm/
|
|
|
|
|
|
KVM is not required by Android.
|
|
|
|
|
|
### lib/
|
|
|
|
|
|
The lib/ tests rely on kernel modules enabled by CONFIG_TEST_PRINTF,
|
|
|
CONFIG_TEST_BITMAP, and CONFIG_PRIME_NUMBERS. None of these are required by
|
|
|
Android.
|
|
|
|
|
|
### locking/
|
|
|
|
|
|
The wait-wound mutex test depends on a kernel module enabled by
|
|
|
CONFIG_WW_MUTEX_SELFTEST which is not required by Android.
|
|
|
|
|
|
### media_tests/
|
|
|
|
|
|
The media_tests depend on /dev/mediaX and /dev/videoX files which may not be
|
|
|
present. They also require manual inspection of dmesg and are not part of
|
|
|
the normal kselftest suite.
|
|
|
|
|
|
### membarrier/
|
|
|
|
|
|
The membarrier test depends on CONFIG_EXPERT and CONFIG_MEMBARRIER being
|
|
|
enabled, neither of which are required by Android.
|
|
|
|
|
|
### memfd/
|
|
|
|
|
|
These tests depend on FUSE which is not present in Android.
|
|
|
|
|
|
### memory-hotplug/
|
|
|
|
|
|
Memory hotplug is not required by Android.
|
|
|
|
|
|
### mount/
|
|
|
|
|
|
The mount test depends on CONFIG_USER_NS which is not required by Android.
|
|
|
|
|
|
### mqueue/
|
|
|
|
|
|
These tests depend on -lrt and -lpopt which are not present in Android.
|
|
|
|
|
|
### net/
|
|
|
|
|
|
The test_bpf.sh test requires a kernel module enabled by CONFIG_TEST_BPF.
|
|
|
The netdevice.sh test operates on an ethernet interface.
|
|
|
The psock_fanout test depends on CONFIG_BPF_SYSCALL which is not required on
|
|
|
Android.
|
|
|
|
|
|
### netfilter/
|
|
|
|
|
|
These tests depend on CONFIG_NET_NS and CONFIG_NF_TABLES_INET, neither of which
|
|
|
are required by Android.
|
|
|
|
|
|
### nsfs/
|
|
|
|
|
|
These tests depend on CONFIG_USER_NS and CONFIG_PID_NS, neither of which are
|
|
|
required by Android.
|
|
|
|
|
|
### ntb/
|
|
|
|
|
|
NTB (non-transparent bridge) is not required by Android.
|
|
|
|
|
|
### pstore/
|
|
|
|
|
|
Pstore is recommended, not required, for Android.
|
|
|
|
|
|
### ptp/
|
|
|
|
|
|
PTP support is not required by Android.
|
|
|
|
|
|
### rseq/
|
|
|
|
|
|
The rseq system call, enabled by CONFIG_RSEQ, is not required by Android.
|
|
|
|
|
|
### sigaltstack/
|
|
|
|
|
|
This is a test for functionality that is not available in bionic (swapcontext()).
|
|
|
|
|
|
### static_keys/
|
|
|
|
|
|
This test depends on a kernel module enabled by CONFIG_TEST_STATIC_KEYS. This
|
|
|
kernel module is not required by Android.
|
|
|
|
|
|
### sync/
|
|
|
|
|
|
These tests verify the legacy sync API which is deprecated on Android.
|
|
|
|
|
|
### sysctl
|
|
|
|
|
|
This test depends on a kernel module enabled by CONFIG_TEST_SYSCTL. This kernel
|
|
|
module is not required by Android.
|
|
|
|
|
|
### timers/
|
|
|
|
|
|
The adjtick test adjusts the system time and takes 2 minutes.
|
|
|
|
|
|
The change_skew test takes too long to run as part of VTS.
|
|
|
|
|
|
The clocksource_switch test takes too long to run as part of VTS and also tries
|
|
|
all clocksources.
|
|
|
|
|
|
The freq_step test relies on clocks having a greater precision than what may be
|
|
|
available.
|
|
|
|
|
|
The leap-a-day test repeatedly modifies the system date in order to test leap
|
|
|
second behavior. This may disrupt the target environment.
|
|
|
|
|
|
The leapcrash test adjusts the system date and takes several minutes to run.
|
|
|
|
|
|
The mqueue_lat test relies on mq_* calls which are not present on Android.
|
|
|
|
|
|
The rtctest_setdate test modifies the RTC's time and date which may disrupt the
|
|
|
target environment.
|
|
|
|
|
|
The set-2038 test repeatedly sets the time in order to test edge cases. This
|
|
|
may disrupt the target enviornment. The test also takes a long time.
|
|
|
|
|
|
The set-tz test sets the timezone which may disrupt the target environment.
|
|
|
|
|
|
The skew_consistency test takes too long to run as part of VTS.
|
|
|
|
|
|
### uevent/
|
|
|
|
|
|
This test depends on CONFIG_USER_NS which is not required by Android.
|
|
|
|
|
|
### user/
|
|
|
|
|
|
This test depends on a kernel module enabled by CONFIG_TEST_USER_COPY. This
|
|
|
kernel module is not required by Android.
|
|
|
|
|
|
### vm/
|
|
|
|
|
|
The hugepage-mmap, hugepage-shm, compaction, and map_hugetlb tests rely on
|
|
|
hugetlbfs/hugetlb page support which is not required by Android.
|
|
|
|
|
|
The userfaultfd test relies on the userfaultfd syscall which is not required by
|
|
|
Android.
|
|
|
|
|
|
### watchdog/
|
|
|
|
|
|
The watchdog test depends on CONFIG_WATCHDOG which is not required by Android.
|
|
|
|
|
|
### zram/
|
|
|
|
|
|
The zram test depends on CONFIG_ZSMALLOC and CONFIG_ZRAM which are not required
|
|
|
by Android.
|
