All new functionality should have unit test coverage. These are found in the tests/unit directory.
Testing is an art to get right! But here are some best practices in terms of unit testing in PyRIT, and some potential concepts to familiarize yourself with as you’re writing these tests.
Make a test that checks one thing and one thing only.
Use
fixturesgenerally, and specifically, if you’re using something across classes, useunit.mocksorintegration.mocks.Memory isolation: Use
sqlite_instancefor a real, isolated SQLite database, orpatch_central_databasewhen patching CentralMemory access.Code coverage and functionality should be checked with unit tests. Notebooks and integration tests should not be relied on for coverage.
MagicMockandAsyncMock: these are the preferred way to mock calls.with patchis acceptable to patch external calls.Don’t write to the actual database, use a
MagicMockfor the memory object or usepatch_central_databaseas the database connection.
Not all of our current tests follow these practices (we’re working on it!) But for some good examples, see test
Async timing and cancellation¶
Use events to coordinate concurrent operations and assert their ordering or concurrency bounds. Timeouts that only prevent a test from hanging should allow for busy CI runners, rather than acting as performance assertions.
For isolation tests, hold one operation at an explicit gate and observe the other operation reaching its intended milestone before releasing that gate. Wait for unrelated persistence or finalization separately, rather than including it in a short deadline for the behavior under test.
When observing an operation’s cancellation or cleanup, use wait_for_completion_async from
unit.async_utils. Unlike asyncio.wait_for, its watchdog does not send another cancellation
request to the operation when the wait expires. Release blocked workers and drain owned tasks
in finally so a failed assertion does not leave background work behind.
Use settle_tasks_async for the final join: it retains the gather, cancels pending work
if its watchdog expires, and drains cancellation cleanup before raising TimeoutError.
That final drain intentionally waits for cooperative cleanup rather than abandoning a task
at a second deadline. An outer test timeout can interrupt this drain; it is a deadlock
backstop, not a guarantee that an uncooperative task settles. A whole-test timeout equal
to an inner watchdog can fire before that watchdog’s expiry handling runs.
Use wait_for_event_async to watch readiness alongside owned tasks, so a task failure
before readiness is reported immediately instead of waiting for an event. Once readiness
is set, the caller observes the task’s outcome separately, even if it has already failed.
For deadline tests, expire a real asyncio.Timeout with reschedule once the operation reaches
the intended pending await. Check the configured timeout arguments, cancellation, cleanup, and
original outcome. This avoids short wall-clock deadlines expiring during unrelated setup.
Windows event loops¶
Windows unit tests keep a native Proactor event loop for each test. Each loop uses
a private TCP socket pair for thread wakeups. Normal closure can leave these
connections in TIME_WAIT, which can exhaust ephemeral ports during the full suite
and fail a later test before its body runs.
The unit-test loop factory uses abortive close only for those two internal sockets when the loop closes. Application sockets, subprocess support, cancellation, and function-scoped loop isolation stay unchanged. Do not share loops between tests or retry loop creation to hide socket allocation failures.
Both coroutine tests and synchronous tests with async fixtures use the factory.
The suite requires pytest-asyncio 1.4 or later and overrides its internal
_asyncio_loop_factory fixture because the public loop-factories hook does not
cover fixture-only runners in 1.4. Regression tests check both paths, so a plugin
upgrade that changes this fixture contract fails explicitly.
SQLite memory fixtures¶
sqlite_instance stays function-scoped. Each test gets a fresh in-memory database and results directory, and its SQLite singleton and CentralMemory registrations are restored afterward. The fixture owns disposal of its memory instance instead of registering process-exit cleanup callbacks for every test.
Declare the memory fixture explicitly even in constructor or identity tests that create targets, scorers, or attacks. Do not rely on another test leaving CentralMemory initialized.
Run uv sync --extra all before validating fixture changes with make unit-test so optional target tests also exercise memory isolation.
To avoid replaying the migration history for every ordinary test, sqlite_template runs the real Alembic migrations and schema check once per pytest session (once in each xdist worker). SQLite’s backup API copies that private, read-only template into each test’s database. Rows, schema changes, temporary tables, and result files are not shared between tests.
Tests of initialization, upgrades, downgrades, schema checks, and reset_database() must still call those production paths explicitly. The fixture does not replace or patch migration or reset APIs.
tests/unit/conftest.py turns off fsync (PRAGMA synchronous=OFF) when a unit test opens a SQLite connection, including file-backed databases in tmp_path. A commit then does not wait for the disk, so a loaded CI disk cannot keep SQLite’s write lock for seconds and push concurrent writers past the busy timeout. Locking and transactions do not change. If another connection holds an exclusive lock at that time, the new connection does not wait and keeps fsync. When a statement fails with SQLITE_BUSY, the error has a note with the time that SQLite waited for the lock. A wait near the busy timeout shows lock contention. A wait near 0 s shows that SQLite did not wait.