lisa

A plugin for organizing, analyzing, and selecting tests.

This plugin provides the mark pytest.mark.lisa, aliased as LISA(), for marking up tests metadata beyond that which Pytest provides by default. See the lisa_schema for the expected metadata input.

Tests can be selected through a playbook.yaml file using the criteria schema (using the pytest-playbook plugin). For example:

criteria:
  # Select all Priority 0 tests.
  - priority: 0
  # Run tests with 'smoke' in the name twice.
  - name: smoke
    times: 2
  # Exclude all tests in Area "xdp"
  - area: xdp
    exclude: true

Todo

• Review the criteria schema.

• Provide test metadata statistics via a command-line flag.

• Assert every test has a LISA marker.

Functions

main()	Wrapper function so we can have a lisa binary.
pytest_collection_modifyitems(session, …)	Pytest collection modifyitems hook for modifying the selected items (tests).
pytest_configure(config)	Pytest configure hook to perform initial configuration.
pytest_playbook_schema(schema)	pytest_playbook_schema() hook to update the playbook schema.
pytest_xdist_make_scheduler(config)	pytest-xdist make scheduler hook for implementing a custom scheduler.
validate_mark(mark)	Validate each test’s LISA() parameters.

Classes

LISAScheduling(config[, log])

Implement load scheduling across nodes, but grouping by target parameter.

lisa.LISA = MarkDecorator(mark=Mark(name='lisa', args=(), kwargs={}))

Alias for the Pytest mark lisa.

lisa.main() → None

Wrapper function so we can have a lisa binary.

lisa.pytest_configure(config: Config) → None

Pytest configure hook to perform initial configuration.

We’re registering our custom marker so that it passes --strict-markers.

lisa.pytest_playbook_schema(schema: Dict[Any, Any]) → None

pytest_playbook_schema() hook to update the playbook schema.

lisa.lisa_schema = Schema({Literal("platform", description="The test's intended platform."): <class 'str'>, Literal("category", description="The kind of test this is."): Or('Functional', 'Performance', 'Stress', 'Community', 'Longhaul'), Literal("area", description="The test's area (or 'feature')."): <class 'str'>, Literal("priority", description="The test's priority with 0 being the highest."): Or(0, 1, 2, 3), Optional('tags'): [<class 'str'>], Optional('features'): [<class 'str'>], Optional('reuse'): <class 'bool'>, Optional('count'): And(<class 'int'>, <function <lambda>>)})

Schema used to validate a test’s metadata.

lisa.validate_mark(mark: Mark) → None

Validate each test’s LISA() parameters.

lisa.pytest_collection_modifyitems(session: Session, config: Config, items: List[Item]) → None

Pytest collection modifyitems hook for modifying the selected items (tests).

First we validate all the LISA() marks on the collected tests. Then we parse the given criteria in the playbook to include or exclude tests. We do not care if the platform mismatches because we intend a multiplicative effect where all selected tests in a playbook are run on all the targets.

class lisa.LISAScheduling(config: Config, log=None)

Bases: xdist.scheduler.loadscope.LoadScopeScheduling

Implement load scheduling across nodes, but grouping by target parameter.

This algorithm ensures that all tests which share the same set of parameters (namely the target) will run on the same executor as a single work-unit.

Todo

This essentially confines the targets and one target won’t be spun up multiple times when run in parallel, so we should make this scheduler optional, as an alternative scenario is to spin up multiple near-identical instances of a target in order to run tests in parallel.

This is modeled after the built-in LoadFileScheduling, which also simply subclasses LoadScopeScheduling. See _split_scope for the important part. Note that we can extend this to implement any kind of scheduling algorithm we want.

__init__(config: Config, log=None)

Initialize self. See help(type(self)) for accurate signature.

lisa.pytest_xdist_make_scheduler(config: Config) → LISAScheduling

pytest-xdist make scheduler hook for implementing a custom scheduler.