Development Guide

Requirements

Name	Installation	Purpose
Python 3.11+	Download	The library is Python-based.
uv	Instructions	uv is used for package management and virtualenv management in Python codebases

Installing dependencies

# Install Python dependencies
uv sync

Generating synthetic queries

Follow these steps to generate synthetic queries using AutoQ:

Set up your project directory:

mkdir -p ./local/autoq_test
cd ./local/autoq_test

Create an input folder and add your input data:
```
mkdir ./input
```
Place your input files inside the ./input directory. To get started, you can use the AP News dataset provided in the datasets folder. To download this example dataset directly into your input folder, run:
```
uv run benchmark-qed data download AP_news input
```
You can also download directly to Azure Blob Storage. See the Datasets documentation for storage options.
Initialize the configuration:
```
uv run benchmark-qed config init autoq .
```
This is the local-filesystem variant.

Alternative blob variant (choose this instead of the local command above; do not run both):
```
uv run benchmark-qed config init autoq . \
    --storage-type blob \
    --container-name my-container \
    --account-url https://<account>.blob.core.windows.net \
    --base-dir autoq_test
```
This command creates two files in the ./autoq_test directory: - .env: Stores environment variables for the AutoQ pipeline. Open this file and replace <API_KEY> with your OpenAI or Azure API key. - settings.yaml: Contains pipeline settings. Edit this file as needed for your use case.

The generated settings.yaml includes commented-out sections for configuring Azure Blob Storage as input and output backends. Uncomment and fill in the storage section under input to read data from blob storage, or the output_storage section to write results to blob storage instead of the local filesystem.
Generate synthetic queries:
```
uv run benchmark-qed autoq settings.yaml output
```
This is the local-filesystem variant.

Alternative blob-stored config variant (choose this instead of the local command above; do not run both):
```
uv run benchmark-qed autoq blob://my-container/autoq_test/settings.yaml output \
    --account-url https://<account>.blob.core.windows.net
```
This will process your input data and save the generated queries in the output directory.

By default, AutoQ also generates assertions for data-driven queries. Assertions are testable factual statements that can be used to evaluate answer accuracy. You can configure assertion generation in settings.yaml:
```
assertions:
  max_assertions: 20  # Set to 0 to disable, or null for unlimited
  enable_validation: true  # Enable to filter low-quality assertions (can be slow)
```

Comparing RAG answer pairs

Follow these steps to compare RAG answer pairs using the pairwise scoring pipeline:

Set up your project directory:

mkdir -p ./local/pairwise_test
cd ./local/pairwise_test

Create an input folder and add your question-answer data:
```
mkdir ./input
```
Copy your RAG answer files into the ./input directory. To get started, you can use the example RAG answers available in the example data folder. To download this example dataset directly into your input folder, run:
```
uv run benchmark-qed data download example_answers input
```
You can also download directly to Azure Blob Storage. See the Datasets documentation for storage options.
Create a configuration file for pairwise comparison:
```
uv run benchmark-qed config init autoe_pairwise .
```
This is the local-filesystem variant.

Alternative blob variant (choose this instead of the local command above; do not run both):
```
uv run benchmark-qed config init autoe_pairwise . \
    --storage-type blob \
    --container-name my-container \
    --account-url https://<account>.blob.core.windows.net \
    --base-dir pairwise_test
```
This command creates two files in the ./pairwise_test directory: - .env: Contains environment variables for the pairwise comparison tests. Open this file and replace <API_KEY> with your OpenAI or Azure API key. - settings.yaml: Contains pipeline settings, which you can modify as needed.

The generated settings.yaml includes commented-out input_storage and output_storage sections for configuring Azure Blob Storage backends.

Run the pairwise comparison:

uv run benchmark-qed autoe pairwise-scores settings.yaml output

This is the local-filesystem variant.

Alternative blob-stored config variant (choose this instead of the local command above; do not run both):

uv run benchmark-qed autoe pairwise-scores blob://my-container/pairwise_test/settings.yaml output \
    --account-url https://<account>.blob.core.windows.net

The results will be saved in the output directory.

Comparing RAG answer pairs (differential, extract-and-judge)

The standard pairwise-scores command shows the judge the full text of both answers side by side. LLM judges are known to have biases in this setting — they tend to favor longer answers, richer formatting, or whichever answer appears first, even when the underlying content is not actually better. Counterbalancing the answer order (as the standard pipeline does) cancels position bias, but not length and formatting bias.

The differential-pairwise-scores command reduces those biases with a two-step extract-and-judge method (Censheng's method):

Extract — For each question, the LLM first separates the two answers into three buckets: content that is common to both, content that is unique to answer A, and content that is unique to answer B.
Judge — The judge then compares only the unique content (with the common content supplied as context) on the configured criteria (the standard defaults or any user-defined criteria).

Because the shared material and padding are stripped out before judging, the verdict is driven by each answer's substantive, distinct contribution rather than by its length or presentation. If two answers convey the same information in different amounts of text, they correctly come out as a tie.

Key points:

This is additive — it is a separate command and does not change the existing pairwise-scores behavior.
The criteria are configurable via a criteria field (the standard defaults are used when omitted), and they are judged together in a single call.
The output uses the same schema and significance pipeline as pairwise-scores (win_rates.csv, winrates_sig_tests.csv), so results are directly comparable to the standard method.
Cost trade-off: each comparison makes roughly twice the LLM calls (one extraction step plus one judging step) in exchange for a more bias-resistant verdict.

Follow these steps to run the differential pairwise scoring pipeline:

Set up your project directory:

mkdir -p ./local/differential_pairwise_test
cd ./local/differential_pairwise_test

Create an input folder and add your question-answer data:
```
mkdir ./input
```
Copy your RAG answer files into the ./input directory. To get started, you can use the example RAG answers available in the example data folder. To download this example dataset directly into your input folder, run:
```
uv run benchmark-qed data download example_answers input
```
You can also download directly to Azure Blob Storage. See the Datasets documentation for storage options.
Create a configuration file for differential pairwise comparison:
```
uv run benchmark-qed config init autoe_differential_pairwise .
```
This is the local-filesystem variant.

Alternative blob variant (choose this instead of the local command above; do not run both):
```
uv run benchmark-qed config init autoe_differential_pairwise . \
    --storage-type blob \
    --container-name my-container \
    --account-url https://<account>.blob.core.windows.net \
    --base-dir differential_pairwise_test
```
This command creates two files in the ./differential_pairwise_test directory: - .env: Contains environment variables for the pairwise comparison tests. Open this file and replace <API_KEY> with your OpenAI or Azure API key. - settings.yaml: Contains pipeline settings, which you can modify as needed.

The differential command reuses the pairwise configuration structure (base, others, question_sets, trials, llm_config, criteria); when criteria is omitted the standard defaults are used. The generated settings.yaml wires up the extract-and-judge prompts under prompt_config (extract_system_prompt, extract_user_prompt, judge_system_prompt, judge_user_prompt); edit those prompt files under prompts/ or remove the prompt_config section to use the built-in defaults.

The generated settings.yaml includes commented-out input_storage and output_storage sections for configuring Azure Blob Storage backends.
Run the differential pairwise comparison:
```
uv run benchmark-qed autoe differential-pairwise-scores settings.yaml output
```
This is the local-filesystem variant.

Alternative blob-stored config variant (choose this instead of the local command above; do not run both):
```
uv run benchmark-qed autoe differential-pairwise-scores blob://my-container/differential_pairwise_test/settings.yaml output \
    --account-url https://<account>.blob.core.windows.net
```
The results will be saved in the output directory. In addition to the per-criterion win scores, each cached comparison CSV also records the extracted common, unique_1, and unique_2 content for inspection.

Scoring RAG answers against reference answers

Follow these steps to score RAG answers against reference answers using example data from the AP news dataset:

Set up your project directory:

mkdir -p ./local/reference_test
cd ./local/reference_test

Create an input folder and add your data:
```
mkdir ./input
```
Copy your RAG answers and reference answers into the input directory. To get started, you can use the example RAG answers available in the example data folder. To download this example dataset directly into your input folder, run:
```
uv run benchmark-qed data download example_answers input
```
You can also download directly to Azure Blob Storage. See the Datasets documentation for storage options.
Create a configuration file for reference scoring:
```
uv run benchmark-qed config init autoe_reference .
```
This is the local-filesystem variant.

Alternative blob variant (choose this instead of the local command above; do not run both):
```
uv run benchmark-qed config init autoe_reference . \
    --storage-type blob \
    --container-name my-container \
    --account-url https://<account>.blob.core.windows.net \
    --base-dir reference_test
```
This creates two files in the ./reference_test directory: - .env: Contains environment variables for the reference scoring pipeline. Open this file and replace <API_KEY> with your OpenAI or Azure API key. - settings.yaml: Contains pipeline settings, which you can modify as needed.

The generated settings.yaml includes commented-out input_storage and output_storage sections for configuring Azure Blob Storage backends.

Run the reference scoring:

uv run benchmark-qed autoe reference-scores settings.yaml output

This is the local-filesystem variant.

Alternative blob-stored config variant (choose this instead of the local command above; do not run both):

uv run benchmark-qed autoe reference-scores blob://my-container/reference_test/settings.yaml output \
    --account-url https://<account>.blob.core.windows.net

The results will be saved in the output directory.

Scoring RAG answers against assertions

Follow these steps to evaluate RAG answers against per-question assertions using answer-level (LLM-as-a-judge) evaluation:

Set up your project directory:

mkdir -p ./local/assertion_test
cd ./local/assertion_test

Create an input folder and add your data:
```
mkdir ./input
```
Copy your RAG answers and assertion files into the input directory. To get started, you can use the example RAG answers and generated assertions from the AP news dataset:
```
uv run benchmark-qed data download example_answers input
uv run benchmark-qed data download AP_news input
```
This downloads example answers and assertion files. You can also use your own assertions JSON files or download directly from Azure Blob Storage. See the Datasets documentation for storage options.
Create a configuration file for assertion evaluation:
```
uv run benchmark-qed config init autoe_assertion .
```
This is the local-filesystem variant.

Alternative blob variant (choose this instead of the local command above; do not run both):
```
uv run benchmark-qed config init autoe_assertion . \
    --storage-type blob \
    --container-name my-container \
    --account-url https://<account>.blob.core.windows.net \
    --base-dir assertion_test
```
This command creates two files in the ./assertion_test directory: - .env: Contains environment variables for the assertion evaluation pipeline. Open this file and replace <API_KEY> with your OpenAI or Azure API key. - settings.yaml: Contains pipeline settings, which you can modify as needed.

The generated settings.yaml includes commented-out input_storage and output_storage sections for configuring Azure Blob Storage backends.
Run the assertion evaluation:
```
uv run benchmark-qed autoe assertion-scores settings.yaml output
```
This is the local-filesystem variant.

Alternative blob-stored config variant (choose this instead of the local command above; do not run both):
```
uv run benchmark-qed autoe assertion-scores blob://my-container/assertion_test/settings.yaml output \
    --account-url https://<account>.blob.core.windows.net
```
The results will be saved in the output directory, including per-assertion scores and per-question summaries.

Evaluating retrieved chunks against assertions

Chunk-level evaluation scores how well retrieved passages (chunks) support assertions, without waiting for answer synthesis. This is useful for: - Fast retrieval evaluation: Assess retrieval quality before generating complete answers - Efficient iteration: Fix retriever settings without re-running expensive answer generation - Persistent caching: Avoid re-evaluating same (assertion, chunk) pairs across multiple runs

Prerequisites: Where do chunks come from?

Chunks are the retrieved passages from your RAG system. You obtain them by running your retriever (e.g., vector search, BM25, hybrid) on each question. Chunks should include: - The passage text - The chunk index (chunk_id) - The retrieval rank

You'll provide this data to benchmark-qed in one of two formats (see Step 2 below).

Step-by-step guide

Set up your project directory:

mkdir -p ./local/chunk_assertion_test
cd ./local/chunk_assertion_test

Prepare your input data:
```
mkdir ./input
```
You need two files whose question_ids line up: - Assertion file: The per-question assertions to check. Set via assertions_path in settings.yaml. - Retrieval file: Created by YOUR retrieval system. The tool always evaluates the retrieved chunks (not the synthesized answer) against your assertions. Provide these chunks as a JSON array of RetrievalResult records, set via retrieval_path in settings.yaml.
Note: To try the pipeline end-to-end without wiring up your own retriever, download the bundled example data, which ships matching retrieval-results and assertions files (same question_ids) for both short- and long-context retrievers:
```
uv run benchmark-qed data download example_answers input
```
Then set, for example:
```
retrieval_path: input/vector_rag_short_context/data_local_retrieval_results.json
assertions_path: input/data_local_assertions.json
```
Swap data_local for data_global or data_linked (and pick vector_rag_short_context or vector_rag_long_context) to evaluate a different question set — just keep the retrieval and assertions files on the same set so their question_ids match. These are the same files used by the AutoE notebook example.
Input Format: RetrievalResult JSON file

Create a JSON array (e.g., input/retrieval.json, or any path you prefer) where each record holds a question and its retrieved context. Each context item carries a chunk_id, text, and an optional rank. When rank is present on every item, chunks are evaluated in rank order (top-k semantics); otherwise the chunks are assumed to be already pre-sorted in decreasing order of relevance and list order is used.
```
[
  {
    "question_id": "q1",
    "text": "What is photosynthesis?",
    "context": [
      {
        "chunk_id": "0",
        "text": "Photosynthesis is the process by which plants convert sunlight into chemical energy...",
        "rank": 0
      },
      {
        "chunk_id": "5",
        "text": "In photosynthesis, light energy is captured by chlorophyll molecules..."
      }
    ]
  }
]
```
This is the same schema produced by the retrieval-results examples (e.g. the bundled data_local_retrieval_results.json), so those files work directly with no conversion. See the chunk-level assertion example in the AutoE notebook (notebooks/autoe.ipynb) for end-to-end usage.
Create a configuration file for chunk-level evaluation:
```
uv run benchmark-qed config init autoe_chunk_assertion .
```
This is the local-filesystem variant.

Alternative blob variant (choose this instead of the local command above; do not run both):
```
uv run benchmark-qed config init autoe_chunk_assertion . \
    --storage-type blob \
    --container-name my-container \
    --account-url https://<account>.blob.core.windows.net \
    --base-dir chunk_assertion_test
```
This command creates two files in the ./chunk_assertion_test directory: - .env: Contains environment variables. Open this file and replace <API_KEY> with your OpenAI or Azure API key. - settings.yaml: Contains pipeline settings, including: - retrieval_path: Path to your retrieved chunks JSON (RetrievalResult format above) - k_list: K values to report coverage metrics (e.g., [5, 10, 20, 50]) - cache_dir: Directory for persistent (assertion, chunk) cache

The generated settings.yaml includes commented-out input_storage and output_storage sections for configuring Azure Blob Storage backends.

Update settings.yaml for your data:

The generated template includes placeholder paths. Update them to match your actual file locations:

generated:
  name: my_retriever
  retrieval_path: input/vector_rag_short_context/data_local_retrieval_results.json  # Update to your RetrievalResult file path
assertions:
  assertions_path: input/data_local_assertions.json  # Update to your assertions file path

Run the chunk-level assertion evaluation:
```
uv run benchmark-qed autoe chunk-assertion-scores settings.yaml output
```
This is the local-filesystem variant.

Alternative blob-stored config variant (choose this instead of the local command above; do not run both):
```
uv run benchmark-qed autoe chunk-assertion-scores blob://my-container/chunk_assertion_test/settings.yaml output \
    --account-url https://<account>.blob.core.windows.net
```
The results will be saved in the output directory, including: - chunk_assertion_results.json: Coverage metrics at each k - per_query_metrics_*.json: Per-question metrics for paired significance testing - debug/: Detailed per-question evaluation records

Chunk-level evaluation benefits: - Efficient caching: Results cached at (assertion, chunk) granularity using SHA256 content-addressing - Multi-k reporting: Coverage, Strict Coverage, and Coverage Strength metrics at each k value - Reusable cache: Re-run with different k values or retriever configs with zero LLM cost on overlapping chunks - Coverage metrics: - Coverage: % of assertions with full or partial support in top-k chunks - Strict Coverage: % of assertions with full support only - Coverage Strength: Average score across all assertions

For detailed instructions on configuring and running AutoE subcommands, please refer to the AutoE CLI Documentation.

To learn how to use AutoE programmatically, please see the AutoE Notebook Example.

Diving Deeper

To explore the query synthesis workflow in detail, please see the AutoQ CLI Documentation for command-line usage and the AutoQ Notebook Example for a step-by-step programmatic guide.

For a deeper understanding of AutoE evaluation pipelines, please refer to the AutoE CLI Documentation for available commands and the AutoE Notebook Example for hands-on examples.

Development Guide

Requirements

Installing dependencies

Generating synthetic queries

Comparing RAG answer pairs

Comparing RAG answer pairs (differential, extract-and-judge)

Scoring RAG answers against reference answers

Scoring RAG answers against assertions

Evaluating retrieved chunks against assertions

Prerequisites: Where do chunks come from?

Step-by-step guide

Input Format: RetrievalResult JSON file

Diving Deeper