4. Running Tests - PyRIT Documentation

Testing plays a crucial role in PyRIT development. Ensuring robust tests in PyRIT is crucial for verifying that functionalities are implemented correctly and for preventing unintended alterations to these functionalities when changes are made to PyRIT.

For running PyRIT tests, you need to have pytest package installed, but if you’ve already set up your development environment with uv sync, pytest should be included in that setup.

Running the unit tests¶

Unit tests are the tier you will run most often. Run the full suite through make rather than invoking pytest directly on tests/unit. The target runs in parallel (pytest -n 4 --dist=loadfile), which is several times faster than a serial run:

make unit-test

Note

make is not part of the local dev setup and is often missing on Windows. If you get make: command not found (or 'make' is not recognized), run the equivalent command directly — this is exactly what the target expands to:

uv run -m pytest -n 4 --dist=loadfile tests/unit

The same substitution works for the other targets on this page.

Running a subset while iterating¶

For a narrower run, invoke pytest directly. You can invoke pytest if it’s in your path or via python; either pytest or python -m pytest. For the following examples, we will use pytest.

To run every test under a subdirectory:
```
pytest tests/unit/converter
```
To run tests from a specific file (e.g. test_base64_converter.py), from the PyRIT directory, use:
```
pytest tests/unit/converter/test_base64_converter.py
```
To execute a specific test (test_base64_converter_default) within the test module (test_base64_converter.py),
```
pytest tests/unit/converter/test_base64_converter.py::test_base64_converter_default
```

Running the other test tiers¶

Run those tiers deliberately, through their own targets:

make integration-test
make end-to-end-test
make partner-integration-test

Integration tests additionally require RUN_ALL_TESTS=true and real credentials. See Unit Tests and Integration Tests for details.

Coverage checks¶

make unit-test-cov-xml runs unit tests and enforces 78% overall coverage. make unit-test-diff-cover checks an existing coverage.xml and requires at least 90% coverage on changed executable lines. make diff-cover runs both checks.

Local diff coverage defaults to a two-dot comparison against origin/main. Override the baseline with make unit-test-diff-cover DIFF_COVER_BASE=<revision> (or the same variable with make diff-cover).

For pull requests targeting main, CI tests GitHub’s PR merge commit and sets DIFF_COVER_BASE=HEAD^1, its first parent. This compares the tested merge against the exact target-branch revision it was built from, rather than a moving origin/main. Full checkout history supplies that parent, including for fork PRs; no fetch of the contributor’s branch is needed. This baseline assumes the default PR merge checkout, not an explicit checkout of the PR head. Push, merge-group, and manually dispatched runs still enforce overall coverage only.