Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

4. Running Tests

Testing plays a crucial role in PyRIT development. Ensuring robust tests in PyRIT is crucial for verifying that functionalities are implemented correctly and for preventing unintended alterations to these functionalities when changes are made to PyRIT.

For running PyRIT tests, you need to have pytest package installed, but if you’ve already set up your development environment with uv sync, pytest should be included in that setup.

Running the unit tests

Unit tests are the tier you will run most often. Run the full suite through make rather than invoking pytest directly on tests/unit. The target runs in parallel (pytest -n 4 --dist=loadfile), which is several times faster than a serial run:

make unit-test

make unit-test-junit also writes per-test timings to junit/test-results.xml and prints the 25 slowest test phases. Override the report path with make unit-test-junit JUNIT_XML=junit/custom-results.xml. CI uploads JUnit reports for every OS, Python version, and extras combination, including failed test runs, with those dimensions in each artifact name.

LiteLLM unit tests mock the SDK and do not need tokenizer data or a local HTTP server, including when the litellm or all extra is installed. Real adapter and outgoing-header checks run separately as loopback integration tests.

CodeQL merge-group uploads

CodeQL still analyzes all three languages on PRs, merge groups, main/release pushes, manual runs and its weekly schedule. Merge groups publish the analyzer’s prepared SARIF through the documented code-scanning API so an expired queue ref can be distinguished from a genuine upload failure. Categories and fingerprints are preserved; failed analysis retains CodeQL’s native diagnostic upload. The merge-group helper uses the authenticated client and context supplied by the pinned actions/github-script action, with no separate HTTP client or Python setup.

Only a successful matching-refs response proving that the exact queue ref is absent or points to another SHA permits retirement. A missing-ref upload rejection is rechecked to cover deletion between verification and upload. Authentication, service, processing and analysis failures stay failures. Retirement cancels the verified current CodeQL run, never reports a skipped scan as success, and refuses cancellation when a matrix job has already failed. The CodeQL job needs actions: write for self-cancellation; workflow-wide permissions remain empty. Fork PRs use native CodeQL publishing and GitHub’s read-only fork token policy.

Run the deterministic, offline regression coverage with Node.js installed. The pytest entry point runs the Node scenario tests and checks workflow wiring. It runs node:test in a single Node process, with the shared 300-second correctness watchdog to allow for startup and filesystem delays on busy Windows runners. The watchdog is a hang backstop, not a performance requirement; API calls and polling delays remain mocked.

uv run --frozen pytest tests/unit/build_scripts/test_codeql_merge_group.py -q

The helper’s scenarios can also run directly with node --test-reporter=tap tests/unit/build_scripts/test_codeql_merge_group.cjs.

The documentation JavaScript suites in test_docs_javascript.py also use one Node process per suite and the shared correctness watchdog. Their per-VM execution limits remain unchanged.

The Bash deployment helper in test_code_deployment.py uses the same shared watchdog for subprocess completion. Readiness and rollback tests retain their simulated operation deadlines, polling assertions, and signal exit-code checks.

The seeded-converter worker-thread test also uses the shared watchdog for its two-worker barrier. It still requires distinct random generators and identical seeded draws; worker startup speed is not part of that contract.

Running a subset while iterating

For a narrower run, invoke pytest directly. You can invoke pytest if it’s in your path or via python; either pytest or python -m pytest. For the following examples, we will use pytest.

Correctness tests using the analytics fixture in tests/unit/analytics/ or the analytics_service fixture in tests/unit/backend/ use finite 300-second query budgets to allow for SQL compilation and scheduling delays on busy CI workers. These are deadlock backstops, not expected runtimes. Production deadlines are unchanged. Deadline and cancellation tests use explicit controls and keep their separate timing assertions.

Async correctness tests can use TEST_WATCHDOG_SECONDS from tests/unit/async_utils.py for a finite 300-second deadlock backstop. Tests that start background tasks must release any test-controlled waits and join their tasks before removing mocks or disposing fixtures, including when a watchdog expires. When waiting for a task to signal readiness, also watch for that task finishing so an early exception is not hidden by the readiness timeout. The analytics quick-lane capacity test observes request admission and completion with these shared watchdogs, then releases its reader gate and drains all requests before restoring mocks. Its 503/Retry-After and independent-execution assertions remain unchanged. Use wait_for_event_async to observe readiness and wait_for_completion_async to observe the outcome, including failures after readiness. Observation timeouts do not cancel owned work. Release gates before calling settle_tasks_async in finally.

For held thread workers, release the gate in finally and use drain_tasks_async while patches and fixtures remain active. Unlike settle_tasks_async, this helper does not cancel worker-owning tasks when the observer is cancelled or its watchdog expires: cancelling asyncio.to_thread cannot stop the underlying thread. It waits through repeated observer cancellation and then propagates the original cancellation message. Watchdog expiry explicitly reports incomplete cleanup; it cannot force a worker that never returns to finish.

Use wait_for_event_async to observe the readiness event and its owning task together, so failures before readiness are reported immediately. For a worker held behind a gate, check the owner is still pending after observation; call task.result() first if it has finished so its original exception is preserved. Keep the outer test timeout larger than the combined readiness and cleanup watchdogs, allowing finally to release and drain workers before mocks are removed.

The discovery provider-state test and Copilot cancellation-test startup use the shared TEST_WATCHDOG_SECONDS backstop in tests/unit/async_utils.py. Discovery timeout coverage expires the real cancellation path only after the request starts. Event-controlled analytics expiry tests freeze both deadline clocks so floating-point rounding cannot turn a valid query budget into a failed assertion.

For thread-backed concurrency tests, signal an asyncio.Event from the worker with loop.call_soon_threadsafe(started.set). Do not observe readiness through asyncio.to_thread(threading_event.wait): that observer can queue behind the very worker it is waiting for when the executor is full. Use wait_for_worker_ready_async(started=started, operation=task) from tests/unit/async_utils.py when the worker must remain held during concurrent assertions. It builds on wait_for_event_async but also rejects completion after the readiness signal, propagating worker failure/cancellation immediately.

Cancelling a task wrapping asyncio.to_thread does not join its thread. Prefer releasing the worker and joining the owned operation with drain_tasks_async, which defers caller cancellation until the owned tasks finish. Inspect task results separately to assert operation outcomes. If the operation can finish on cancellation before its thread exits, also observe actual worker completion. Gate timeouts should raise on the shared TEST_WATCHDOG_SECONDS deadlock backstop, not silently release workers after a short elapsed time. Exercise these patterns with constrained executors, including a single worker, and failure/cancellation paths so readiness never depends on spare executor capacity.

File-surface cancellation tests hold an in-flight read at an explicit checkpoint. Cancellation returns before that read is released; the worker may finish that chunk, but must stop before starting another. Wait for worker cleanup before checking final read counts or removing patches, rather than assuming fixed sleeps let the worker settle.

Backend manual-send completion and first-request analytics correctness tests also use the shared watchdog. Manual sends are polled until a terminal snapshot, including finalization. Expiring the test’s wait does not cancel the send; its lifecycle owner still drains pending work before mocks and memory close. Explicit zero-timeout tests check this ownership without requiring the send to finish quickly. The async and repeated manual-send test classes use 600-second whole-test backstops so the 300-second completion watchdog can expire and allow cleanup before pytest stops the test.

The target-type route test starts target metadata discovery, waits for a signal that it has started, and keeps it paused while checking that /api/health responds within 10 seconds. Waiting for that signal does not use an extra worker thread. The waits for discovery to start and for background work to finish use the shared 300-second timeout, so the test fails rather than waiting forever if work gets stuck. This timeout is separate from the health-response requirement. The test runs with one or two worker threads, including failure and cancellation cases. In every case, the request and discovery worker finish before the test restores patched functions and closes the HTTP client.

The scorer-validation cancellation test keeps its five-second observer around scorer draining and native SQLite connection cleanup. ERROR-result persistence is held behind an event until those assertions pass, then awaited separately with the finite completion watchdog. The blocked validation read is not released to make either step finish.

For scenario correctness tests, wait for the owned task to finish, including persistence and scheduler handoff. Use events to control concurrent steps and a finite test watchdog to detect deadlocks, not short per-step latency limits. Catalog queue-admission and configured-estimate independence tests use the existing estimate watchdog fixture. Observe the queued semaphore acquisition before checking that its execution deadline has not started, and verify configured completion while catalog capacity is still occupied. Check each deadline against completed admissions and spy on catalog acquisition during configured work to assert independence directly. Release gates and drain owned requests and service workers in finally, including when a readiness assertion fails. For frontend modal tests, use fake timers and wait for an accessible control on the destination page. A route change does not mean that modal cleanup has restored background accessibility.

Running the other test tiers

Run those tiers deliberately, through their own targets:

make integration-test
make end-to-end-test
make partner-integration-test

Integration tests additionally require RUN_ALL_TESTS=true and real credentials. See Unit Tests and Integration Tests for details.

Coverage checks

make unit-test-cov-xml runs unit tests, enforces 78% overall coverage, and produces the same JUnit timing report. make unit-test-diff-cover checks an existing coverage.xml and requires at least 90% coverage on changed executable lines. make diff-cover runs both checks.

Local diff coverage defaults to a two-dot comparison against origin/main. Override the baseline with make unit-test-diff-cover DIFF_COVER_BASE=<revision> (or the same variable with make diff-cover).

For pull requests targeting main, CI tests GitHub’s PR merge commit and sets DIFF_COVER_BASE=HEAD^1, its first parent. This compares the tested merge against the exact target-branch revision it was built from, rather than a moving origin/main. Full checkout history supplies that parent, including for fork PRs; no fetch of the contributor’s branch is needed. This baseline assumes the default PR merge checkout, not an explicit checkout of the PR head. Push, merge-group, and manually dispatched runs still enforce overall coverage only.