| 1 | Audio Testing of VirtualBox
|
|---|
| 2 | ===========================
|
|---|
| 3 |
|
|---|
| 4 |
|
|---|
| 5 | Overview / Goal
|
|---|
| 6 | ---------------
|
|---|
| 7 |
|
|---|
| 8 | The goal is to create a flexible testing framework to test the
|
|---|
| 9 | VirtualBox audio stack.
|
|---|
| 10 |
|
|---|
| 11 | It should be runnable with an easy-to-use setup so that also regular users
|
|---|
| 12 | can perform tests on request, without having to install or set up additional
|
|---|
| 13 | dependencies.
|
|---|
| 14 |
|
|---|
| 15 | That framework must be runnable on all host/guest combinations together with all
|
|---|
| 16 | audio drivers ("backends") and device emulations being offered. This makes it a
|
|---|
| 17 | rather big testing matrix which therefore has to be processed in an automated
|
|---|
| 18 | fashion.
|
|---|
| 19 |
|
|---|
| 20 | Additionally it should be flexible enough to add more (custom) tests later on.
|
|---|
| 21 |
|
|---|
| 22 |
|
|---|
| 23 | Operation
|
|---|
| 24 | ---------
|
|---|
| 25 |
|
|---|
| 26 | The framework consists of several components which try to make use as much of
|
|---|
| 27 | the existing audio stack code as possible. This allows the following
|
|---|
| 28 | operation modes:
|
|---|
| 29 |
|
|---|
| 30 | Standalone
|
|---|
| 31 | Playing back / recording audio data (test tones / .WAV files) in a
|
|---|
| 32 | standalone scenario, i.e. no VirtualBox / VMs required). This mode is using
|
|---|
| 33 | VirtualBox' audio (mixing) stack and available backend drivers without the
|
|---|
| 34 | need of VirtualBox being installed.
|
|---|
| 35 |
|
|---|
| 36 | Manual
|
|---|
| 37 | Performing single / multiple tests manually on a local machine.
|
|---|
| 38 | Requires a running and set up test VM.
|
|---|
| 39 |
|
|---|
| 40 | Automated
|
|---|
| 41 | Performs single / multiple tests via the Validation Kit audio test
|
|---|
| 42 | driver and can be triggered via the Validation Kit Test Manager.
|
|---|
| 43 |
|
|---|
| 44 | (Re-)validation of previously ran tests
|
|---|
| 45 | This takes two test sets and runs the validation / analysis on them.
|
|---|
| 46 |
|
|---|
| 47 | Self testing mode
|
|---|
| 48 | Performs standalone self tests to verify / debug the involved components.
|
|---|
| 49 |
|
|---|
| 50 |
|
|---|
| 51 | Components and Terminology
|
|---|
| 52 | --------------------------
|
|---|
| 53 |
|
|---|
| 54 | The following components are in charge for performing the audio tests
|
|---|
| 55 | (depends on the operation mode, see above):
|
|---|
| 56 |
|
|---|
| 57 | - VBoxAudioTest (also known as VKAT, "Validation Kit Audio Test"):
|
|---|
| 58 | A binary which can perform the standalone audio tests mentioned above, as well
|
|---|
| 59 | as acting as the guest and host service(s) when performing manual or automated
|
|---|
| 60 | tests. It also includes the analysis / verification of audio test sets.
|
|---|
| 61 | VKAT also is included in host installations and Guest Additions since
|
|---|
| 62 | VirtualBox 7.0 to give customers and end users the opportunity to test and
|
|---|
| 63 | verify the audio stack.
|
|---|
| 64 |
|
|---|
| 65 | Additional features include:
|
|---|
| 66 | * Automatic probing of audio backends ("--probe-backends")
|
|---|
| 67 | * Manual playback of test tones ("play -t")
|
|---|
| 68 | * Manual playback of .WAV files ("play <WAV-File>")
|
|---|
| 69 | * Manual recording to .WAV files ("recording <WAV-File>")
|
|---|
| 70 | * Manual device enumeration (sub command "enum")
|
|---|
| 71 | * Manual (re-)verification of test sets (sub command "verify")
|
|---|
| 72 | * Self-contained self tests (sub command "selftest")
|
|---|
| 73 |
|
|---|
| 74 | See the syntax help ("--help") for more.
|
|---|
| 75 |
|
|---|
| 76 | - ATS ("Audio Testing Service"): Component which is being used by 1 and the
|
|---|
| 77 | Validation Kit audio driver (backend) to communicate across guest and host
|
|---|
| 78 | boundaries. Currently using a TCP/IP transport layer. Also works with VMs
|
|---|
| 79 | which are configured with NAT networking ("reverse connection").
|
|---|
| 80 |
|
|---|
| 81 | - Validation Kit audio test driver (tdAudioTest.py): Used for integrating and
|
|---|
| 82 | invoking VKAT for manual and automated tests via the Validation Kit framework
|
|---|
| 83 | (Test Manager). Optional. The test driver can be found at [1]_.
|
|---|
| 84 |
|
|---|
| 85 | - Validation Kit audio driver (backend): A dedicated audio backend which
|
|---|
| 86 | communicates with VKAT running on the same host to perform the actual audio
|
|---|
| 87 | tests on a VirtualBox installation. This makes it possible to test the full
|
|---|
| 88 | audio stack on a running VM without any additional / external tools.
|
|---|
| 89 |
|
|---|
| 90 | On guest playback, data will be recorded, on guest recording, data will be
|
|---|
| 91 | injected from the host into the audio stack.
|
|---|
| 92 |
|
|---|
| 93 | - Test sets contain
|
|---|
| 94 | - a test manifest with all information required (vkat_manifest.ini)
|
|---|
| 95 | - the generated / captured audio data (as raw PCM)
|
|---|
| 96 |
|
|---|
| 97 | and are either packed as .tar.gz archives or consist of a dedicated directory
|
|---|
| 98 | per test set.
|
|---|
| 99 |
|
|---|
| 100 | There always must be at least two test sets - one from the host side and one
|
|---|
| 101 | from the guest side - to perform a verification.
|
|---|
| 102 |
|
|---|
| 103 | Each test set contains a test tag so that matching test sets can be
|
|---|
| 104 | identified.
|
|---|
| 105 |
|
|---|
| 106 | The above components are also included in VirtualBox release builds and can be
|
|---|
| 107 | optionally enabled (disabled by default).
|
|---|
| 108 |
|
|---|
| 109 | .. [1] src/VBox/ValidationKit/tests/audio/tdAudioTest.py
|
|---|
| 110 |
|
|---|
| 111 |
|
|---|
| 112 | Setup instructions
|
|---|
| 113 | ------------------
|
|---|
| 114 |
|
|---|
| 115 | - VM needs to be configured to have audio emulation and audio testing enabled
|
|---|
| 116 | (via extra-data, set "VBoxInternal2/Audio/Debug/Enabled" to "true").
|
|---|
| 117 | - Audio input / output for the VM needs to be enabled (depending on the test).
|
|---|
| 118 | - Start VBoxAudioTest on the guest, for example:
|
|---|
| 119 |
|
|---|
| 120 | VBoxAudioTest test --mode guest --tcp-connect-address 10.0.2.2
|
|---|
| 121 |
|
|---|
| 122 | Note: VBoxAudioTest is included with the Guest Additions starting at
|
|---|
| 123 | VirtualBox 7.0.
|
|---|
| 124 | Note: Depending on the VM's networking configuration there might be further
|
|---|
| 125 | steps necessary in order to be able to reach the host from the guest.
|
|---|
| 126 | See the VirtualBox manual for more information.
|
|---|
| 127 |
|
|---|
| 128 |
|
|---|
| 129 | Performing a manual test
|
|---|
| 130 | ------------------------
|
|---|
| 131 |
|
|---|
| 132 | - Follow "Setup instructions".
|
|---|
| 133 | - Start VBoxAudioTest on the host with selected test(s), for example:
|
|---|
| 134 |
|
|---|
| 135 | VBoxAudioTest test --mode host
|
|---|
| 136 |
|
|---|
| 137 | Note: VBoxAudioTest is included with the VirtualBox 7.0 host installers and
|
|---|
| 138 | will be installed by default.
|
|---|
| 139 |
|
|---|
| 140 | - By default the test verification will be done automatically after running the
|
|---|
| 141 | tests.
|
|---|
| 142 |
|
|---|
| 143 |
|
|---|
| 144 | Advanced: Performing manual verification
|
|---|
| 145 | ----------------------------------------
|
|---|
| 146 |
|
|---|
| 147 | VBoxAudioTest can manually be used with the "verify" sub command in order to
|
|---|
| 148 | (re-)verify previously generated test sets. It then will return different exit
|
|---|
| 149 | codes based on the verification result.
|
|---|
| 150 |
|
|---|
| 151 |
|
|---|
| 152 | Advanced: Performing an automated test
|
|---|
| 153 | --------------------------------------
|
|---|
| 154 |
|
|---|
| 155 | - TxS (Test E[x]ecution Service) has to be up and running (part of the
|
|---|
| 156 | Validation Kit) on the guest.
|
|---|
| 157 | - Invoke the tdAudioTest.py test driver, either manually or fully automated
|
|---|
| 158 | via Test Manager.
|
|---|
| 159 |
|
|---|
| 160 |
|
|---|
| 161 | Internals: Workflow for a single test
|
|---|
| 162 | -------------------------------------
|
|---|
| 163 |
|
|---|
| 164 | When a single test is being executed on a running VM, the following (simplified)
|
|---|
| 165 | workflow applies:
|
|---|
| 166 |
|
|---|
| 167 | - VKAT on the host connects to VKAT running on the guest (via ATS, also can be a
|
|---|
| 168 | remote machine in theory).
|
|---|
| 169 | - VKAT on the host connects to Validation Kit audio driver on the host
|
|---|
| 170 | (via ATS, also can be a remote machine in theory).
|
|---|
| 171 | - For example, when doing playback tests, VKAT on the host ...
|
|---|
| 172 | * ... tells the Validation Kit audio driver to start recording
|
|---|
| 173 | guest playback.
|
|---|
| 174 | * ... tells the VKAT on the guest to start playing back audio data.
|
|---|
| 175 | * ... gathers all test data (generated from/by the guest and recorded from
|
|---|
| 176 | the host) as separate test sets.
|
|---|
| 177 | * ... starts verification / analysis of the test sets.
|
|---|
| 178 |
|
|---|
| 179 |
|
|---|
| 180 | Current status / limitations
|
|---|
| 181 | ----------------------------
|
|---|
| 182 |
|
|---|
| 183 | - The following test types are currently implemented:
|
|---|
| 184 | * Test tone (sine wave) playback from the guest
|
|---|
| 185 | * Test tone (sine wave) recording by the guest (injected from the host)
|
|---|
| 186 | - Only the HDA device emulation has been verified so far.
|
|---|
| 187 | - Only the ALSA audio stack on Debian 10 has been verified so far.
|
|---|
| 188 | Note: This is different from PulseAudio using the ALSA plugin!
|
|---|
| 189 |
|
|---|
| 190 |
|
|---|
| 191 | Troubleshooting
|
|---|
| 192 | ---------------
|
|---|
| 193 |
|
|---|
| 194 | - Make sure that audio device emulation is enabled and can be used within the
|
|---|
| 195 | guest. Also, audio input / output has to be enabled, depending on the tests.
|
|---|
| 196 | - Make sure that the guest's VBoxAudioTest's instance can reach the host via
|
|---|
| 197 | the selected transport layer (TCP/IP by default).
|
|---|
| 198 | - Increase the hosts audio logging level
|
|---|
| 199 | (via extra-data, set "VBoxInternal2/Audio/Debug/Level" to "5").
|
|---|
| 200 | - Increase VBoxAudioTest's verbosity level (add "-v", can be specified
|
|---|
| 201 | multiple times).
|
|---|
| 202 | - Check if the VBox release log contains any warnings / errors with the
|
|---|
| 203 | "ValKit:" prefix.
|
|---|
| 204 |
|
|---|
| 205 |
|
|---|
| 206 | :Status: $Id: VBoxAudioValidationKitReadMe.txt 98106 2023-01-17 22:43:07Z vboxsync $
|
|---|
| 207 | :Copyright: Copyright (C) 2021-2023 Oracle Corporation.
|
|---|
