Development Guide

Requirements

Name	Installation	Purpose
Python 3.11+	Download	The library is Python-based.
uv	Instructions	uv is used for package management and virtualenv management in Python codebases

Installing dependencies

# Install Python dependencies
uv sync

Generating synthetic queries

Follow these steps to generate synthetic queries using AutoQ:

Set up your project directory:

mkdir -p ./local/autoq_test
cd ./local/autoq_test

Create an input folder and add your input data:
```
mkdir ./input
```
Place your input files inside the ./input directory. To get started, you can use the AP News dataset provided in the datasets folder. To download this example dataset directly into your input folder, run:
```
uv run benchmark-qed data download AP_news input
```
You can also download directly to Azure Blob Storage. See the Datasets documentation for storage options.
Initialize the configuration:
```
uv run benchmark-qed config init autoq .
```
This is the local-filesystem variant.

Alternative blob variant (choose this instead of the local command above; do not run both):
```
uv run benchmark-qed config init autoq . \
    --storage-type blob \
    --container-name my-container \
    --account-url https://<account>.blob.core.windows.net \
    --base-dir autoq_test
```
This command creates two files in the ./autoq_test directory:
- .env: Stores environment variables for the AutoQ pipeline. Open this file and replace <API_KEY> with your OpenAI or Azure API key.
- settings.yaml: Contains pipeline settings. Edit this file as needed for your use case.
The generated settings.yaml includes commented-out sections for configuring Azure Blob Storage as input and output backends. Uncomment and fill in the storage section under input to read data from blob storage, or the output_storage section to write results to blob storage instead of the local filesystem.
Generate synthetic queries:
```
uv run benchmark-qed autoq settings.yaml output
```
This is the local-filesystem variant.

Alternative blob-stored config variant (choose this instead of the local command above; do not run both):
```
uv run benchmark-qed autoq blob://my-container/autoq_test/settings.yaml output \
    --account-url https://<account>.blob.core.windows.net
```
This will process your input data and save the generated queries in the output directory.

By default, AutoQ also generates assertions for data-driven queries. Assertions are testable factual statements that can be used to evaluate answer accuracy. You can configure assertion generation in settings.yaml:
```
assertions:
  max_assertions: 20  # Set to 0 to disable, or null for unlimited
  enable_validation: true  # Enable to filter low-quality assertions (can be slow)
```

Comparing RAG answer pairs

Follow these steps to compare RAG answer pairs using the pairwise scoring pipeline:

Set up your project directory:

mkdir -p ./local/pairwise_test
cd ./local/pairwise_test

Create an input folder and add your question-answer data:
```
mkdir ./input
```
Copy your RAG answer files into the ./input directory. To get started, you can use the example RAG answers available in the example data folder. To download this example dataset directly into your input folder, run:
```
uv run benchmark-qed data download example_answers input
```
You can also download directly to Azure Blob Storage. See the Datasets documentation for storage options.
Create a configuration file for pairwise comparison:
```
uv run benchmark-qed config init autoe_pairwise .
```
This is the local-filesystem variant.

Alternative blob variant (choose this instead of the local command above; do not run both):
```
uv run benchmark-qed config init autoe_pairwise . \
    --storage-type blob \
    --container-name my-container \
    --account-url https://<account>.blob.core.windows.net \
    --base-dir pairwise_test
```
This command creates two files in the ./pairwise_test directory:
- .env: Contains environment variables for the pairwise comparison tests. Open this file and replace <API_KEY> with your OpenAI or Azure API key.
- settings.yaml: Contains pipeline settings, including a persistent stage-aware SQLite cache under .benchmark_qed_cache/differential_pairwise.
The generated settings.yaml includes commented-out input_storage and output_storage sections for configuring Azure Blob Storage backends.

Run the pairwise comparison:

uv run benchmark-qed autoe pairwise-scores settings.yaml output

This is the local-filesystem variant.

Alternative blob-stored config variant (choose this instead of the local command above; do not run both):

uv run benchmark-qed autoe pairwise-scores blob://my-container/pairwise_test/settings.yaml output \
    --account-url https://<account>.blob.core.windows.net

The results will be saved in the output directory.

Comparing RAG answer pairs (differential, extract-and-judge)

The standard pairwise-scores command shows the judge the full text of both answers side by side. LLM judges are known to have biases in this setting — they tend to favor longer answers, richer formatting, or whichever answer appears first, even when the underlying content is not actually better. Counterbalancing the answer order (as the standard pipeline does) cancels position bias, but not length and formatting bias.

The differential-pairwise-scores command reduces those biases with a two-step extract-and-judge method (Censheng's method):

Extract — For each question, the LLM first separates the two answers into three buckets: content that is common to both, content that is unique to answer A, and content that is unique to answer B.
Judge — The judge then compares only the unique content (with the common content supplied as context) on the configured criteria (the standard defaults or any user-defined criteria).

Because the shared material and padding are stripped out before judging, the verdict is driven by each answer's substantive, distinct contribution rather than by its length or presentation. If two answers convey the same information in different amounts of text, they correctly come out as a tie.

Key points:

This is additive — it is a separate command and does not change the existing pairwise-scores behavior.
The criteria are configurable via a criteria field (the standard defaults are used when omitted), and they are judged together in a single call.
The output uses the same schema and significance pipeline as pairwise-scores (win_rates.csv, winrates_sig_tests.csv), so results are directly comparable to the standard method.
Cost trade-off: each comparison makes roughly twice the LLM calls (one extraction step plus one judging step) in exchange for a more bias-resistant verdict.

Follow these steps to run the differential pairwise scoring pipeline:

Set up your project directory:

mkdir -p ./local/differential_pairwise_test
cd ./local/differential_pairwise_test

Create an input folder and add your question-answer data:
```
mkdir ./input
```
Copy your RAG answer files into the ./input directory. To get started, you can use the example RAG answers available in the example data folder. To download this example dataset directly into your input folder, run:
```
uv run benchmark-qed data download example_answers input
```
You can also download directly to Azure Blob Storage. See the Datasets documentation for storage options.
Create a configuration file for differential pairwise comparison:
```
uv run benchmark-qed config init autoe_differential_pairwise .
```
This is the local-filesystem variant.

Alternative blob variant (choose this instead of the local command above; do not run both):
```
uv run benchmark-qed config init autoe_differential_pairwise . \
    --storage-type blob \
    --container-name my-container \
    --account-url https://<account>.blob.core.windows.net \
    --base-dir differential_pairwise_test
```
This command creates two files in the ./differential_pairwise_test directory:
- .env: Contains environment variables for the pairwise comparison tests. Open this file and replace <API_KEY> with your OpenAI or Azure API key.
- settings.yaml: Contains pipeline settings, which you can modify as needed.
The differential command reuses the pairwise configuration structure (base, others, question_sets, trials, llm_config, criteria); when criteria is omitted the standard defaults are used. The generated settings.yaml wires up the extract-and-judge prompts under prompt_config (extract_system_prompt, extract_user_prompt, judge_system_prompt, judge_user_prompt); edit those prompt files under prompts/ or remove the prompt_config section to use the built-in defaults.

The generated cache_config caches extraction and verdict stages independently. A failed judge call can therefore resume from its completed extraction, and changing criteria invalidates verdicts without repeating extraction. Set type: none with storage: null to force both stages to run again.

The generated settings.yaml includes commented-out input_storage and output_storage sections for configuring Azure Blob Storage backends.
Run the differential pairwise comparison:
```
uv run benchmark-qed autoe differential-pairwise-scores settings.yaml output
```
This is the local-filesystem variant.

Alternative blob-stored config variant (choose this instead of the local command above; do not run both):
```
uv run benchmark-qed autoe differential-pairwise-scores blob://my-container/differential_pairwise_test/settings.yaml output \
    --account-url https://<account>.blob.core.windows.net
```
The results will be saved in the output directory. In addition to the per-criterion win scores, each cached comparison CSV also records the extracted common, unique_1, and unique_2 content for inspection.

Inspect the default stage cache with:
```
uv run benchmark-qed cache inspect \
    .benchmark_qed_cache/differential_pairwise/differential_pairwise.sqlite3
```
Existing comparison CSVs are checked before the stage cache. To fully recompute a comparison, use a new output directory (or remove its CSV) and also disable, relocate, or clear the stage cache. See Evaluation Caches for backend and clearing guidance.

Scoring RAG answers against reference answers

Follow these steps to score RAG answers against reference answers using example data from the AP news dataset:

Set up your project directory:

mkdir -p ./local/reference_test
cd ./local/reference_test

Create an input folder and add your data:
```
mkdir ./input
```
Copy your RAG answers and reference answers into the input directory. To get started, you can use the example RAG answers available in the example data folder. To download this example dataset directly into your input folder, run:
```
uv run benchmark-qed data download example_answers input
```
You can also download directly to Azure Blob Storage. See the Datasets documentation for storage options.
Create a configuration file for reference scoring:
```
uv run benchmark-qed config init autoe_reference .
```
This is the local-filesystem variant.

Alternative blob variant (choose this instead of the local command above; do not run both):
```
uv run benchmark-qed config init autoe_reference . \
    --storage-type blob \
    --container-name my-container \
    --account-url https://<account>.blob.core.windows.net \
    --base-dir reference_test
```
This creates two files in the ./reference_test directory:
- .env: Contains environment variables for the reference scoring pipeline. Open this file and replace <API_KEY> with your OpenAI or Azure API key.
- settings.yaml: Contains pipeline settings, including a persistent SQLite judgment cache under .benchmark_qed_cache/reference.
The generated settings.yaml includes commented-out input_storage and output_storage sections for configuring Azure Blob Storage backends. The SQLite cache remains local when blob input/output storage is enabled.
Run the reference scoring:
```
uv run benchmark-qed autoe reference-scores settings.yaml output
```
This is the local-filesystem variant.

Alternative blob-stored config variant (choose this instead of the local command above; do not run both):
```
uv run benchmark-qed autoe reference-scores blob://my-container/reference_test/settings.yaml output \
    --account-url https://<account>.blob.core.windows.net
```
The results will be saved in the output directory.

Successful question/criterion/trial judgments are cached immediately. If a request fails, rerun the same command to reuse completed judgments and evaluate only misses. Inspect the cache with:
```
uv run benchmark-qed cache inspect \
    .benchmark_qed_cache/reference/reference.sqlite3
```
Set cache_config.type: none with storage: null when fresh judgments are required.

Scoring RAG answers against assertions

Follow these steps to evaluate RAG answers against per-question assertions using answer-level (LLM-as-a-judge) evaluation:

Set up your project directory:

mkdir -p ./local/assertion_test
cd ./local/assertion_test

Create an input folder and add your data:
```
mkdir ./input
```
Copy your RAG answers and assertion files into the input directory. To get started, you can use the example RAG answers and generated assertions from the AP news dataset:
```
uv run benchmark-qed data download example_answers input
uv run benchmark-qed data download AP_news input
```
This downloads example answers and assertion files. You can also use your own assertions JSON files or download directly from Azure Blob Storage. See the Datasets documentation for storage options.
Create a configuration file for assertion evaluation:
```
uv run benchmark-qed config init autoe_assertion .
```
This is the local-filesystem variant.

Alternative blob variant (choose this instead of the local command above; do not run both):
```
uv run benchmark-qed config init autoe_assertion . \
    --storage-type blob \
    --container-name my-container \
    --account-url https://<account>.blob.core.windows.net \
    --base-dir assertion_test
```
This command creates two files in the ./assertion_test directory:
- .env: Contains environment variables for the assertion evaluation pipeline. Open this file and replace <API_KEY> with your OpenAI or Azure API key.
- settings.yaml: Contains pipeline settings, which you can modify as needed.
The generated settings.yaml includes commented-out input_storage and output_storage sections for configuring Azure Blob Storage backends.
Run the assertion evaluation:
```
uv run benchmark-qed autoe assertion-scores settings.yaml output
```
This is the local-filesystem variant.

Alternative blob-stored config variant (choose this instead of the local command above; do not run both):
```
uv run benchmark-qed autoe assertion-scores blob://my-container/assertion_test/settings.yaml output \
    --account-url https://<account>.blob.core.windows.net
```
The results will be saved in the output directory, including per-assertion scores and per-question summaries.

The generated single-RAG configuration enables a persistent SQLite cache at .benchmark_qed_cache/assertion/assertion.sqlite3. Each question/assertion/trial judgment is persisted independently, so rerunning after an interruption evaluates only missing work. Inspect it with:
```
uv run benchmark-qed cache inspect \
    .benchmark_qed_cache/assertion/assertion.sqlite3
```
Set cache_config.type: none with storage: null when fresh judgments are required. This answer-level cache does not apply to multi-RAG or hierarchical assertion scoring; chunk-level assertion evaluation uses its own assertion/chunk cache.

Evaluating retrieved chunks against assertions

Chunk-level evaluation scores how well retrieved passages (chunks) support assertions, without waiting for answer synthesis. This is useful for:

Fast retrieval evaluation: Assess retrieval quality before generating complete answers
Efficient iteration: Fix retriever settings without re-running expensive answer generation
Persistent caching: Avoid re-evaluating same (assertion, chunk) pairs across multiple runs

Prerequisites: Where do chunks come from?

Chunks are the retrieved passages from your RAG system. You obtain them by running your retriever (e.g., vector search, BM25, hybrid) on each question. Chunks should include:

The passage text
The chunk index (chunk_id)
The retrieval rank when available

Step-by-step guide

Set up your project directory:

mkdir -p ./local/chunk_assertion_test
cd ./local/chunk_assertion_test

Initialize the chunk-assertion project:

For a local-filesystem project, run:
```
uv run benchmark-qed config init autoe_chunk_assertion .
```
Alternative blob variant (choose this instead of the local command above; do not run both):
```
uv run benchmark-qed config init autoe_chunk_assertion . \
    --storage-type blob \
    --container-name my-container \
    --account-url https://<account>.blob.core.windows.net \
    --base-dir chunk_assertion_test
```
The command creates the project structure before any settings need to be edited:
- settings.yaml: Pipeline configuration with input placeholders
- .env: Environment-variable placeholders for credentials
- input/: Local input directory
- prompts/: Default chunk-assertion judge prompts
Set the required credentials in .env before running the evaluation.
Prepare the retrieval and assertion inputs:

You need two JSON files with matching question_id values:
- A retrieval-results file produced by your retriever. BenchmarkQED evaluates the retrieved chunks, not the synthesized answer.
- An assertions file containing the per-question assertions to check.
To try the workflow with bundled example data, download both kinds of files into the generated input/ directory:
```
uv run benchmark-qed data download example_answers input
```
The examples include matching files for data_local, data_global, and data_linked, with both vector_rag_short_context and vector_rag_long_context retrieval results.

If you use your own retriever, its retrieval-results file must be a JSON array in this form:
```
[
  {
    "question_id": "q1",
    "text": "What is photosynthesis?",
    "context": [
      {
        "chunk_id": "0",
        "text": "Photosynthesis converts sunlight into chemical energy.",
        "rank": 0
      },
      {
        "chunk_id": "5",
        "text": "Chlorophyll captures light energy.",
        "rank": 1
      }
    ]
  }
]
```
Each context item requires chunk_id and text; rank is optional. When every item has a rank, chunks are evaluated in rank order. Otherwise, their existing list order is used. See the chunk-level assertion example in the AutoE notebook for programmatic usage.
Review the generated settings.yaml:

The generated configuration already points to the bundled data_local short-context example:
```
generated:
  name: my_retriever
  retrieval_path: input/vector_rag_short_context/data_local_retrieval_results.json

assertions:
  assertions_path: input/data_local_assertions.json
```
If you downloaded the bundled data in the previous step, these defaults can be used unchanged. To evaluate another example set, change data_local in both paths to data_global or data_linked. To use the long-context retriever, also change vector_rag_short_context to vector_rag_long_context. The question set must match in both files.

Other important generated settings are:
- k_list: Retrieval depths for which coverage metrics are reported
- max_chunks_per_question: Optional cap on evaluated chunks
- cache_config: Cache backend for assertion/chunk judgments
- llm_config: Judge model, provider, credentials, and request settings
- prompt_config: Judge prompt files
The default cache is persistent SQLite:
```
cache_config:
  type: sqlite
  storage:
    type: file
    base_dir: .benchmark_qed_cache/chunk_assertions
  database_name: chunk_assertions.sqlite3
```
Supported cache types are sqlite, json, memory, and none. Existing local chunk_assertions.jsonl caches are imported automatically. Persistent local caches coordinate concurrent processes so only one process judges each uncached assertion/chunk pair.

Cache storage is independent of input_storage and output_storage. A JSON cache with storage.type: file remains local even when inputs and outputs use Azure Blob Storage, and its relative base_dir is resolved from the directory where the command is run. To place JSON entries in blob storage, set cache_config.storage.type: blob and provide its container and credentials explicitly. See Evaluation Caches for local and blob JSON examples.

Run the chunk-level assertion evaluation:

uv run benchmark-qed autoe chunk-assertion-scores settings.yaml output

Alternative blob-stored config variant (choose this instead of the local command above; do not run both):

uv run benchmark-qed autoe chunk-assertion-scores blob://my-container/chunk_assertion_test/settings.yaml output \
    --account-url https://<account>.blob.core.windows.net

Review the outputs and cache:

The output/ directory contains:
- chunk_assertion_results.json: Coverage metrics at each k
- per_query_metrics_*.json: Per-question metrics for paired significance testing
- debug/: Detailed per-question evaluation records
Inspect the default SQLite cache, including namespace counts and active work leases, with:
```
uv run benchmark-qed cache inspect \
    .benchmark_qed_cache/chunk_assertions/chunk_assertions.sqlite3
```
The inspection command applies only to SQLite cache files. To force fresh LLM judgments, set cache_config.type: none and storage: null.

Chunk-level evaluation benefits

Efficient caching: Results cached at (assertion, chunk) granularity using SHA256 content-addressing
Multi-k reporting: Coverage, Strict Coverage, and Coverage Strength metrics at each k value
Reusable cache: Re-run with different k values or retriever configs with zero LLM cost on overlapping chunks
Coverage metrics:
Coverage: % of assertions with full or partial support in top-k chunks
Strict Coverage: % of assertions with full support only
Coverage Strength: Average score across all assertions

For detailed instructions on configuring and running AutoE subcommands, please refer to the AutoE CLI Documentation.

To learn how to use AutoE programmatically, please see the AutoE Notebook Example.

Diving Deeper

To explore the query synthesis workflow in detail, please see the AutoQ CLI Documentation for command-line usage and the AutoQ Notebook Example for a step-by-step programmatic guide.

For a deeper understanding of AutoE evaluation pipelines, please refer to the AutoE CLI Documentation for available commands and the AutoE Notebook Example for hands-on examples.