Testing plays a crucial role in PyRIT development. Ensuring robust tests in PyRIT is crucial for verifying that functionalities are implemented correctly and for preventing unintended alterations to these functionalities when changes are made to PyRIT.
For running PyRIT tests, you need to have pytest package installed, but if you’ve already set up your development environment with
uv sync, pytest should be included in that setup.
Running the unit tests¶
Unit tests are the tier you will run most often. Run the full suite through make rather
than invoking pytest directly on tests/unit. The target runs in parallel
(pytest -n 4 --dist=loadfile), which is several times faster than a serial run:
make unit-testmake unit-test-junit also writes per-test timings to junit/test-results.xml and
prints the 25 slowest test phases. Override the report path with
make unit-test-junit JUNIT_XML=junit/custom-results.xml.
CI uploads JUnit reports for every OS, Python version, and extras combination,
including failed test runs, with those dimensions in each artifact name.
LiteLLM unit tests mock the SDK and do not need tokenizer data or a local HTTP
server, including when the litellm or all extra is installed. Real adapter
and outgoing-header checks run separately as
loopback integration tests.
CodeQL merge-group uploads¶
CodeQL still analyzes all three languages on PRs, merge groups, main/release pushes,
manual runs and its weekly schedule. Merge groups publish the analyzer’s prepared
SARIF through the documented code-scanning API so an expired queue ref can be
distinguished from a genuine upload failure. Categories and fingerprints are
preserved; failed analysis retains CodeQL’s native diagnostic upload.
The merge-group helper uses the authenticated client and context supplied by the
pinned actions/github-script action, with no separate HTTP client or Python setup.
Only a successful matching-refs response proving that the exact queue ref is absent
or points to another SHA permits retirement. A missing-ref upload rejection is
rechecked to cover deletion between verification and upload. Authentication,
service, processing and analysis failures stay failures. Retirement cancels the
verified current CodeQL run, never reports a skipped scan as success, and refuses
cancellation when a matrix job has already failed. The CodeQL job needs
actions: write for self-cancellation; workflow-wide permissions remain empty.
Fork PRs use native CodeQL publishing and GitHub’s read-only fork token policy.
Run the deterministic, offline regression coverage with Node.js installed.
The pytest entry point runs the Node scenario tests and checks workflow wiring.
It runs node:test in a single Node process, with the shared 300-second correctness
watchdog to allow for startup and filesystem delays on busy Windows runners.
The watchdog is a hang backstop, not a performance requirement; API calls and
polling delays remain mocked.
uv run --frozen pytest tests/unit/build_scripts/test_codeql_merge_group.py -qThe helper’s scenarios can also run directly with
node --test-reporter=tap tests/unit/build_scripts/test_codeql_merge_group.cjs.
The documentation JavaScript suites in test_docs_javascript.py also use one
Node process per suite and the shared correctness watchdog. Their per-VM
execution limits remain unchanged.
The Bash deployment helper in test_code_deployment.py uses the same shared
watchdog for subprocess completion. Readiness and rollback tests retain their
simulated operation deadlines, polling assertions, and signal exit-code checks.
The seeded-converter worker-thread test also uses the shared watchdog for its two-worker barrier. It still requires distinct random generators and identical seeded draws; worker startup speed is not part of that contract.
Running a subset while iterating¶
For a narrower run, invoke pytest directly. You can invoke pytest if it’s in your path or via python; either pytest or python -m pytest. For the following examples, we will use pytest.
Correctness tests using the analytics fixture in tests/unit/analytics/ or the
analytics_service fixture in tests/unit/backend/ use finite 300-second query
budgets to allow for SQL compilation and scheduling delays on busy
CI workers. These are deadlock backstops, not expected runtimes. Production deadlines
are unchanged. Deadline and cancellation tests use explicit controls and keep their
separate timing assertions.
Async correctness tests can use TEST_WATCHDOG_SECONDS from tests/unit/async_utils.py
for a finite 300-second deadlock backstop. Tests that start background tasks must
release any test-controlled waits and join their tasks before removing mocks or
disposing fixtures, including when a watchdog expires. When waiting for a task to
signal readiness, also watch for that task finishing so an early exception is not
hidden by the readiness timeout.
The analytics quick-lane capacity test observes request admission and completion
with these shared watchdogs, then releases its reader gate and drains all requests
before restoring mocks. Its 503/Retry-After and independent-execution assertions
remain unchanged.
Use wait_for_event_async to observe readiness and wait_for_completion_async to
observe the outcome, including failures after readiness. Observation timeouts do
not cancel owned work. Release gates before calling settle_tasks_async in finally.
For held thread workers, release the gate in finally and use drain_tasks_async
while patches and fixtures remain active. Unlike settle_tasks_async, this helper
does not cancel worker-owning tasks when the observer is cancelled or its watchdog
expires: cancelling asyncio.to_thread cannot stop the underlying thread. It waits
through repeated observer cancellation and then propagates the original cancellation
message. Watchdog expiry explicitly reports incomplete cleanup; it cannot force a
worker that never returns to finish.
Use wait_for_event_async to observe the readiness event and its owning task together,
so failures before readiness are reported immediately. For a worker held behind a
gate, check the owner is still pending after observation; call task.result() first
if it has finished so its original exception is preserved. Keep the outer test timeout
larger than the combined readiness and cleanup watchdogs, allowing finally to release
and drain workers before mocks are removed.
The discovery provider-state test and Copilot cancellation-test startup use the
shared TEST_WATCHDOG_SECONDS backstop in tests/unit/async_utils.py. Discovery
timeout coverage expires the real cancellation path only after the request starts.
Event-controlled analytics expiry tests freeze both deadline clocks so floating-point
rounding cannot turn a valid query budget into a failed assertion.
For thread-backed concurrency tests, signal an asyncio.Event from the worker with
loop.call_soon_threadsafe(started.set). Do not observe readiness through
asyncio.to_thread(threading_event.wait): that observer can queue behind the very
worker it is waiting for when the executor is full. Use
wait_for_worker_ready_async(started=started, operation=task) from
tests/unit/async_utils.py when the worker must remain held during concurrent
assertions. It builds on wait_for_event_async but also rejects completion after
the readiness signal, propagating worker failure/cancellation immediately.
Cancelling a task wrapping asyncio.to_thread does not join its thread. Prefer
releasing the worker and joining the owned operation with drain_tasks_async,
which defers caller cancellation until the owned tasks finish. Inspect task results
separately to assert operation outcomes. If the operation can finish on cancellation
before its thread exits, also observe actual worker completion.
Gate timeouts should raise on the shared TEST_WATCHDOG_SECONDS deadlock backstop,
not silently release workers after a short elapsed time. Exercise these patterns
with constrained executors, including a single worker, and failure/cancellation
paths so readiness never depends on spare executor capacity.
File-surface cancellation tests hold an in-flight read at an explicit checkpoint. Cancellation returns before that read is released; the worker may finish that chunk, but must stop before starting another. Wait for worker cleanup before checking final read counts or removing patches, rather than assuming fixed sleeps let the worker settle.
Backend manual-send completion and first-request analytics correctness tests also use the shared watchdog. Manual sends are polled until a terminal snapshot, including finalization. Expiring the test’s wait does not cancel the send; its lifecycle owner still drains pending work before mocks and memory close. Explicit zero-timeout tests check this ownership without requiring the send to finish quickly. The async and repeated manual-send test classes use 600-second whole-test backstops so the 300-second completion watchdog can expire and allow cleanup before pytest stops the test.
The target-type route test starts target metadata discovery, waits for a signal
that it has started, and keeps it paused while checking that /api/health
responds within 10 seconds. Waiting for that signal does not use an extra worker
thread. The waits for discovery to start and for background work to finish use
the shared 300-second timeout, so the test fails rather than waiting forever if
work gets stuck. This timeout is separate from the health-response requirement.
The test runs with one or two worker threads, including failure and cancellation
cases. In every case, the request and discovery worker finish before the test
restores patched functions and closes the HTTP client.
The scorer-validation cancellation test keeps its five-second observer around scorer draining and native SQLite connection cleanup. ERROR-result persistence is held behind an event until those assertions pass, then awaited separately with the finite completion watchdog. The blocked validation read is not released to make either step finish.
For scenario correctness tests, wait for the owned task to finish, including
persistence and scheduler handoff. Use events to control concurrent steps and a
finite test watchdog to detect deadlocks, not short per-step latency limits.
Catalog queue-admission and configured-estimate independence tests use the existing
estimate watchdog fixture. Observe the queued semaphore acquisition before checking
that its execution deadline has not started, and verify configured completion while
catalog capacity is still occupied. Check each deadline against completed admissions
and spy on catalog acquisition during configured work to assert independence directly.
Release gates and drain owned requests and service workers in finally, including
when a readiness assertion fails.
For frontend modal tests, use fake timers and wait for an accessible control on
the destination page. A route change does not mean that modal cleanup has restored
background accessibility.
To run every test under a subdirectory:
pytest tests/unit/converterTo run tests from a specific file (e.g.
test_base64_converter.py), from the PyRIT directory, use:pytest tests/unit/converter/test_base64_converter.pyTo execute a specific test (
test_base64_converter_default) within the test module (test_base64_converter.py),pytest tests/unit/converter/test_base64_converter.py::test_base64_converter_default
Running the other test tiers¶
Run those tiers deliberately, through their own targets:
make integration-test
make end-to-end-test
make partner-integration-testIntegration tests additionally require RUN_ALL_TESTS=true and real credentials. See
Unit Tests and Integration Tests for details.
Coverage checks¶
make unit-test-cov-xml runs unit tests, enforces 78% overall coverage, and produces
the same JUnit timing report.
make unit-test-diff-cover checks an existing coverage.xml and requires at least
90% coverage on changed executable lines. make diff-cover runs both checks.
Local diff coverage defaults to a two-dot comparison against origin/main.
Override the baseline with make unit-test-diff-cover DIFF_COVER_BASE=<revision>
(or the same variable with make diff-cover).
For pull requests targeting main, CI tests GitHub’s PR merge commit and sets
DIFF_COVER_BASE=HEAD^1, its first parent. This compares the tested merge against
the exact target-branch revision it was built from, rather than a moving
origin/main. Full checkout history supplies that parent, including for fork PRs;
no fetch of the contributor’s branch is needed. This baseline assumes the default
PR merge checkout, not an explicit checkout of the PR head. Push, merge-group,
and manually dispatched runs still enforce overall coverage only.