Functional Tests
Functional testing is a pytest
test suite (rooted in functional_tests/
) for
tests that are meant to be run in an isolated environment. Functional testing is
comprised of:
- Native pytest test cases.
- Unit-test like Rust functions.
They are used to test:
- Functionality that cannot work in isolation and depends on external resources (e.g. binaries on the system)
- The more dangerous parts of the codebase, generally things you wouldn't want
to test in your development environment such as:
- Operations that require root access.
- Operations that manipulate disks, RAID arrays, mounts, filesystems, etc.
- Operations that modify the OS in any way.
This document aims to explain the architecture of how functional tests work, and how they are implemented.
Native Python Tests
Native Python tests are pytest tests that are written in Python. They live in
python files in the functional_tests/custom
directory.
Generally, these leverage the vm
fixture, which is an Azure Linux VM that has been
created using virt-deploy
and netlaunch
. The vm
fixture is defined in
conftest.py
.
Rust-Based Tests
Conditional Compilation
Rust-based functional tests should be contained within a module called functional_test
gated by the feature functional-test
. For example:
#[cfg(feature = "functional-test")]
#[cfg_attr(not(test), allow(unused_imports, dead_code))]
mod functional_test {
// tests here
}
Test Case Definition
Each test case should have the proc-macro attribute #[functional_test]
applied
to it. The attribute is defined in pytest_gen
.
This attribute expands roughly to the following:
inventory::submit!{pytest::TestCaseMetadata {
module: module_path!(),
function: #function,
negative: #negative,
feature: #feature,
type_: #test_type,
}}
#[test]
#(#attrs)*
#vis #sig {
#block
}
For example, the following test case:
#[functional_test]
fn test_case() {
// test here
}
Expands to:
inventory::submit!{pytest::TestCaseMetadata {
module: module_path!(), // This will be `crate::module1::moduleN::functional_test`
function: "test_case",
negative: false,
feature: "",
type_: "",
}}
#[test]
fn test_case() {
// test here
}
The key part is the inventory::submit!
macro. This macro comes from the
inventory
crate, which is generally used to register plug-ins. In this case,
the object being registered is a TestCaseMetadata
object, which contains all
the information needed to run the test case.
The inventory
crate creates a global list of all the registered objects, which
can be consumed lated. This happens in the pytest
crate.
Inventory Collection
The pytest
crate contains the definition of the TestCaseMetadata
object, and
thus, also contains the macro inventory::collect!
which is used to collect all
the registered TestCaseMetadata
objects.
The pytest
crate is responsible for iterating over all the submitted test
cases. This happens in the generate_functional_test_manifest()
function, which
is called by Trident's pytest
subcommand. This subcommand is ONLY available
when the pytest-generator
cargo feature is enabled.
Rust-Pytest Interface
To run the functional tests, pytest must be aware of all the test cases that exist within the test binaries.
Rust Export to JSON
The pytest
crate collects all the test cases, and builds a tree representing
rust's modules, and all the test cases within them. This tree is then serialized
into a JSON file called ft.json
, and placed in the functional_tests/
directory.
The underlying structure of the JSON file is:
/// Represents a rust module.
#[derive(Serialize, Default, Debug)]
struct Module {
/// Test cases in this module.
#[serde(skip_serializing_if = "HashMap::is_empty")]
test_cases: HashMap<String, TestCaseInfo>,
/// Submodules of this module.
#[serde(skip_serializing_if = "HashMap::is_empty")]
submodules: HashMap<String, Module>,
}
/// Represents a specific test case.
#[derive(Serialize, Default, Debug)]
struct TestCaseInfo {
/// Pytest markers to apply to this test case.
#[serde(skip_serializing_if = "Vec::is_empty")]
markers: Vec<String>,
}
let mut json_output: HashMap<String, Module> = HashMap::new();
An example of the JSON output is:
{
"osutils": {
"submodules": {
"sfdisk": {
"submodules": {
"functional_test": {
"test_cases": {
"test_get": {
"markers": [
"functional",
"positive",
"helpers"
]
}
}
}
}
},
"container": {
"submodules": {
"functional_test": {
"test_cases": {
"test_get_host_root_path_in_simulated_container": {
"markers": [
"functional",
"positive",
"helpers"
]
},
"test_get_host_root_path_fails_in_simulated_container_without_host_mount": {
"markers": [
"functional",
"negative",
"helpers"
]
},
}
}
}
}
}
}
}
Pytest Collection from JSON
In pytest, we use the hook pytest_collect_file
to look out for the ft.json
file. When we find it, we parse it, and create a FuncTestCollector
object,
which returns one RustModule
object per crate.
The RustModule
object explores the tree and recursively yields more
RustModule
objects, until it reaches the test cases, which it yields as
pytest.Function
objects.
The pytest.Function
objects are partials created from the function
def run_rust_functional_test(vm, crate, module_path, test_case):
"""Runs a rust test on the VM."""
from functional_tests.tools.runner import RunnerTool
testRunner = RunnerTool(vm)
testRunner.run(
crate,
f"{module_path}::{test_case}",
)
All the parameters except for vm
get filled in during collection.
Pytest will then run the run_rust_functional_test
function, which will run the
test case inside of the VM.
Isolated Environment Creation
The functional test pytest suite defines the vm
fixture. This VM is created
using virt-deploy
with a prebuilt Azure Linux 3 image.