Garden Linux

Appearance

Test Framework Architecture ​

This page explains the design and architecture of the Garden Linux test framework, including how tests, plugins, and handlers interact, and how the test distribution system works.

Framework components ​

The Garden Linux test framework is built on pytest and uses a modular architecture with three main components, as defined in ADR-0006 and ADR-0008:

How tests, plugins, and handlers connect ​

The framework uses pytest's plugin system to automatically register fixtures:

  1. Plugins (tests/plugins/) - Provide fixtures for system access
  2. Handlers (tests/handlers/) - Provide fixtures for setup and teardown
  3. Tests (tests/integration/test_*.py) - Use fixtures via dependency injection

Registration: All plugins are automatically registered as pytest fixtures via conftest.py.

Directory structure ​

tests/
├── util/                   # Utility scripts for running tests
│   ├── run.sh              # Main entry point for running tests
│   ├── run_chroot.sh       # Chroot testing environment
│   ├── run_qemu.sh         # QEMU VM testing environment
│   ├── run_cloud.sh        # Cloud provider testing
│   ├── run_oci.sh          # OCI container testing
│   ├── login_qemu.sh       # SSH login to QEMU VM
│   ├── login_cloud.sh      # SSH login to cloud VM
│   └── tf/                 # Terraform configurations for cloud
├── plugins/                # Test plugins
│   └── ...
├── handlers/               # Test handlers
│   └── ...
├── integration/            # Directory for all tests and their categories
│   ├── boot/               # Boot category
│   │   └── test_*.py       # Individual tests
│   ├── core/               # Core category
│   │   └── test_*.py       # Individual tests
│   └── ...

Plugins ​

Plugins are pytest fixtures that handle infrastructure concerns and system interactions.

Purpose: Provide clean Application Programming Interfaces (APIs) for system access (file parsing, service management, and similar tasks).

Usage: Provide pytest fixtures that can be injected into test functions.

Examples: Systemd, Sshd, ShellRunner, KernelModule

Guideline: Handle "how to access" not "what to test".

Plugin design philosophy ​

Plugins focus on infrastructure concerns:

  • File parsing and data access
  • System service management
  • Network connections and sockets
  • Data processing and formatting

Plugins do not contain test logic or assertions. They provide the tools tests need to interact with the system under test.

Handlers ​

Handlers are pytest fixtures that manage test setup and teardown.

Purpose: Setup and teardown of test state (connections, services, environments).

Pattern: Yield fixtures that prepare resources and explicitly clean up after tests.

Usage: Used as pytest fixtures with yield for cleanup.

Examples: service_ssh, service_containerd

Key distinction: Unlike regular fixtures that provide data, handlers manage stateful resources that need explicit cleanup.

Handler responsibilities ​

Handlers must:

  • Save the initial system state before making changes
  • Yield to the test
  • Restore the original state in the teardown phase
  • Only restore what they changed
  • Clean up in reverse order of setup (especially important for dependencies like kernel modules)

Utility functions ​

Utility functions (tests/plugins/utils.py) provide reusable functionality.

Purpose: Helper functions not used directly in tests.

Usage: Used by plugins and handlers.

Examples: equals_ignore_case, parse_etc_file

These functions are building blocks for plugins and handlers, not directly accessible to tests.

Test distribution system ​

The test framework is automatically built and packaged when running tests. The build process creates a self-contained distribution, as specified in ADR-0006, that includes the Python runtime, test framework, and all dependencies.

Build components ​

The build system creates the following artifacts (all under .build/), built for the target architecture (amd64 or arm64):

ArtifactPurposeConsumed by
runtime-<arch>.tar.gzPer-arch Python runtime with dependenciesBuild input only
dist-<arch>.tar.gzTest framework and test filesChroot, OCI, dev sync
dist-<arch>.ext2.rawRaw ext2 filesystem imageQEMU, AWS, OpenStack, Ali
dist-<arch>.ext2.vhdVirtual Hard Disk (VHD) imageAzure
dist-<arch>.ext2.qcow2QCOW2 disk imageAli (alternative)
dist-<arch>.ext2.raw.tar.gzTar file containing disk.rawGoogle Cloud Platform (GCP)
dist-<arch>.ociSingle-arch OCI imageManifest input
dist.ociMultiarch OCI image indexPublished ghcr.io/gardenlinux/test
edk2-qemu-*EFI Development Kit II (EDK2) firmwareQEMU boot

Artifact naming and derivation ​

  • Distribution filenames use Garden Linux architecture naming (amd64, arm64). Kernel architecture names (x86_64, aarch64) only appear where they are required externally, such as Python runtime download URLs and EDK2 firmware filenames.
  • Each per-arch distribution contains only the Python runtime for its own architecture.
  • The multiarch OCI image dist.oci combines the two single-arch dist-<arch>.oci images into one image index.

Build process ​

The build system follows these steps:

  1. Runtime Environment - Downloads standalone Python binaries for the target arch (amd64 or arm64) and installs required packages from requirements.txt
  2. Test Framework - Bundles all test files, plugins, and the test runner script
  3. Distribution - Creates a compressed tar archive, an ext2 disk image with its .vhd/.qcow2/.tar.gz derivatives, and (optionally) an OCI image
  4. Firmware - Downloads EDK2 firmware files for QEMU virtualization

Automatic building ​

The build process runs automatically when you execute ./test:

bash
# Build artifacts are created automatically
./test .build/image.raw

# Or build manually
cd tests
make -f util/build.makefile all
# or inspect available targets
make -f util/build.makefile help

./test reads the target architecture from the artifact's .requirements file and builds only the artifacts required by the selected test type (see the Test CLI Reference).

Distribution structure ​

The built distribution contains:

dist/
├── runtime/
│   └── <arch>/         # Python runtime for this arch only (amd64 or arm64)
├── tests/
│   ├── conftest.py     # Pytest configuration
│   ├── handlers/       # Test handlers
│   ├── integration/    # Tests, grouped by category
│   │   ├── boot/       # Boot category
│   │   ├── core/       # Core category
│   │   └── ...
│   └── plugins/        # Test plugins
└── run_tests           # Test execution script

This self-contained distribution can be:

  • Mounted directly in QEMU VMs
  • Uploaded to cloud instances via SSH
  • Extracted into chroot environments
  • Run in OCI containers

The distribution approach ensures that:

  • Tests run in a consistent Python environment across all platforms
  • No dependencies on the host system's Python installation
  • Tests can run on minimal systems without Python pre-installed
  • The same test code runs identically across all test environments

Design principles ​

The architecture follows these key principles, derived from ADR-0006, ADR-0007, and ADR-0008:

  1. Separation of Concerns - Tests contain assertions, plugins handle infrastructure, handlers manage state
  2. Dependency Injection - Pytest's fixture system provides clean dependency management
  3. Self-Contained - Distribution includes all dependencies (no host system requirements)
  4. Environment Agnostic - Same tests run in chroot, QEMU, cloud, and OCI environments
  5. Extensible - New plugins and handlers can be added without modifying the core framework text

The test framework architecture is based on several key decisions:

EU and German government funding logos

Funded by the European Union – NextGenerationEU.

The views and opinions expressed are solely those of the author(s) and do not necessarily reflect the views of the European Union or the European Commission. Neither the European Union nor the European Commission can be held responsible for them.

Copyright © Linux Foundation Europe.

Test Framework Architecture | Garden Linux is a project of the NeoNephos Foundation. For applicable policies including privacy policy, terms of use and trademark usage guidelines, please see https://linuxfoundation.eu/policies. Linux is a registered trademark of Linus Torvalds.

Building this site is powered by