SkillOpt Documentation & Reproduction Guide

1.1 What is SkillOpt #

SkillOpt is a text-space optimizer that improves a frozen language agent by iteratively editing a natural-language skill document — never the model weights. The skill document is a Markdown file that conditions a target model as it executes tasks. SkillOpt treats this document as the "weights" and runs a training loop that mirrors deep-learning training: rollout (forward pass), reflect (backward pass / gradients), select & apply edits (optimizer step), and a validation gate (accept/reject).

Two roles split every model call:

• Target — executes tasks using the current skill document (the agent being improved).
• Optimizer — analyzes the target's trajectories and proposes edits to the skill document.

The same loop drives six benchmarks out of the box (QA, document QA, embodied agents, math, spreadsheet code generation, and tool-augmented QA).

1.2 Deep-Learning ↔ SkillOpt Analogy #

Every concept below maps to a concrete code construct, so deep-learning intuitions transfer directly to hyperparameter tuning.

1.3 Key Features #

1.4 Repository Layout #

# top level
configs/            # YAML configs (_base_ + per-benchmark)
scripts/            # train.py, eval_only.py CLIs
ckpt/               # packaged reference skills (e.g. gpt5.5_skill.md)
docs/               # this guide + mkdocs sources
skillopt/           # the package
 ├─ config.py        # YAML loading, _base_ inheritance, flatten
 ├─ engine/trainer.py# the training loop (ReflACTTrainer)
 ├─ gradient/        # reflect.py (analyst), aggregate.py (merge)
 ├─ optimizer/       # skill edits, scheduler, clip, slow_update, meta_skill
 ├─ evaluation/      # gate.py (accept/reject logic)
 ├─ model/           # backend clients + routing
 └─ envs/<benchmark>/ # adapter, dataloader, rollout, evaluator, reflect

2.1 Requirements #

• Python ≥ 3.10
• Credentials for at least one model backend (Azure OpenAI, OpenAI-compatible, Anthropic, or a local Qwen server)
• Benchmark datasets are not bundled — prepare your own splits (see §4)

2.2 Install the Package #

Option A — from PyPI:

pip install skillopt

# Optional extras:
pip install skillopt[alfworld]   # ALFWorld benchmark
pip install skillopt[webui]      # Gradio monitoring dashboard
pip install skillopt[claude]     # Claude model backend

Option B — from source (for development):

git clone https://github.com/microsoft/SkillOpt.git
cd SkillOpt
pip install -e .

# Optional extras (install only what you need):
pip install -e ".[alfworld]"   # ALFWorld benchmark
pip install -e ".[claude]"     # Anthropic Claude backend
pip install -e ".[qwen]"       # local Qwen backend
pip install -e ".[webui]"      # monitoring dashboard

# ALFWorld also needs its data assets:
alfworld-download

2.3 Configure Credentials #

Copy the template and fill in whichever backend you will use:

cp .env.example .env
# edit .env, then:
set -a; source .env; set +a

Azure OpenAI (default)

export AZURE_OPENAI_ENDPOINT="https://your-resource.openai.azure.com/"
export AZURE_OPENAI_API_VERSION="2024-12-01-preview"
# Auth option 1 — API key:
export AZURE_OPENAI_API_KEY="your-key"
# Auth option 2 — Azure CLI (no key; recommended on Azure VMs):
export AZURE_OPENAI_AUTH_MODE=azure_cli
# Auth option 3 — Managed Identity:
export AZURE_OPENAI_AUTH_MODE=managed_identity
export AZURE_OPENAI_MANAGED_IDENTITY_CLIENT_ID="your-client-id"

OpenAI-compatible endpoint

export AZURE_OPENAI_ENDPOINT="https://api.openai.com/v1"
export AZURE_OPENAI_API_KEY="sk-..."
export AZURE_OPENAI_AUTH_MODE=openai_compatible

Anthropic Claude / local Qwen

export ANTHROPIC_API_KEY="sk-ant-..."          # claude_chat backend

export QWEN_CHAT_BASE_URL="http://localhost:8000/v1" # local vLLM
export QWEN_CHAT_MODEL="Qwen/Qwen3.5-4B"

2.4 Verify Installation #

python -c "import skillopt; print('SkillOpt ready!')"

3.1 Your First Demo #

What ships in this repo: ready-to-use configs and pretrained skills (ckpt/) for six benchmarks, plus lightweight ID manifests under data/. The manifests pin exactly which examples each split uses but do not contain the example contents — so you materialize the data once before the first run.

Step 1 — materialize the SearchQA splits (one-time; downloads the ~6.5 GB source dataset). The manifest IDs match the key field of the lucadiliello/searchqa dataset:

pip install datasets
python - <<'PY'
import json, os
from datasets import load_dataset

ds = load_dataset("lucadiliello/searchqa")
by_key = {r["key"]: r for split in ds.values() for r in split}

for split in ["train", "val", "test"]:
    ids = json.load(open(f"data/searchqa_id_split/{split}/items.json"))
    items = []
    for x in ids:
        r = by_key[x["id"]]
        items.append({"id": r["key"], "question": r["question"],
                      "context": r["context"], "answers": r["answers"]})
    os.makedirs(f"data/searchqa_split/{split}", exist_ok=True)
    json.dump(items, open(f"data/searchqa_split/{split}/items.json", "w"))
    print(split, len(items))
PY

Step 2 — train (4 epochs × batch 40; see §3.2 for the CLI reference):

python scripts/train.py \
    --config configs/searchqa/default.yaml \
    --split_dir data/searchqa_split \
    --azure_openai_endpoint https://your-resource.openai.azure.com/ \
    --optimizer_model gpt-5.5 \
    --target_model gpt-5.5

Other benchmarks follow the same pattern — materialize from the raw source listed in data/README.md (it documents the lookup key per benchmark), then point --split_dir at the result. The one exception is ALFWorld, whose bundled data/alfworld_path_split works directly: just pip install -e ".[alfworld]" && alfworld-download and set $ALFWORLD_DATA.

To sanity-check your setup without training, evaluate a packaged pretrained skill instead (§3.3 uses ckpt/searchqa/gpt5.5_skill.md), or launch the monitoring WebUI (§8.4).

3.2 Train a Skill #

# Minimal SearchQA run
python scripts/train.py \
    --config configs/searchqa/default.yaml \
    --split_dir /path/to/your/searchqa_split \
    --azure_openai_endpoint https://your-resource.openai.azure.com/ \
    --optimizer_model gpt-5.5 \
    --target_model gpt-5.5

Swap the config for another benchmark (e.g. configs/livemathematicianbench/default.yaml, configs/alfworld/default.yaml). Common CLI arguments:

3.3 Evaluate a Skill #

Evaluate any skill document (a packaged reference skill, or a trained run's best_skill.md) without training:

# Evaluate the packaged GPT-5.5 SearchQA skill on the test split
python scripts/eval_only.py \
  --config configs/searchqa/default.yaml \
  --skill ckpt/searchqa/gpt5.5_skill.md \
  --split valid_unseen \
  --split_dir /path/to/searchqa_split \
  --azure_openai_endpoint https://your-resource.openai.azure.com/

3.4 Output Structure #

outputs/<run_name>/
 ├─ config.json          # flattened runtime config
 ├─ history.json         # per-step training history
 ├─ runtime_state.json   # resume checkpoint
 ├─ best_skill.md        # best validated skill document
 ├─ skills/skill_vXXXX.md# skill snapshot per step
 ├─ steps/step_XXXX/     # per-step artifacts (patches, evals)
 ├─ slow_update/epoch_XX/# slow-update logs & rollouts
 └─ meta_skill/epoch_XX/ # meta-skill logs

3.5 Auto-Resume #

Each completed step persists its state to runtime_state.json and a steps/step_XXXX/ directory. Re-running the same command against the same out_root detects finished work and continues from the last completed step — including epoch-boundary slow-update and meta-skill stages.

4.1 Split Directory Format #

Bringing your own dataset takes three steps: (1) create a split directory with train/ val/ test/ item files in the format below; (2) make sure each item carries the fields the closest existing benchmark adapter expects (§4.2); (3) point --split_dir at it and train with that benchmark's config. If no existing adapter matches your task shape (different rollout or scoring logic), write a new benchmark adapter instead — see §7.2.

With env.split_mode: split_dir (the recommended, deterministic mode), SkillOpt reads a directory containing train/, val/, and test/ subfolders, each holding a JSON array of task items:

data/my_split/
 ├─ train/items.json   # used for rollout (the "train split")
 ├─ val/items.json     # selection split → validation gate (valid_seen)
 └─ test/items.json    # held-out final eval (valid_unseen)

4.2 Item JSON Schema #

Required fields depend on the benchmark; consult skillopt/envs/<benchmark>/dataloader.py for the exact contract. A SearchQA item, for example:

[
  {
    "id":       "unique_item_id",
    "question": "Who wrote the novel ...",
    "context":  "[DOC] relevant passage text ...",
    "answers":  ["expected answer"]
  }
]

4.3 Split Modes #

5.1 The Training Loop #

The loop lives in ReflACTTrainer (skillopt/engine/trainer.py). Each epoch runs a series of optimization steps over rollout batches, then performs two epoch-boundary stages.

for epoch in epochs:
    for step in steps:
        1. Rollout    # target executes a batch of tasks
        2. Reflect    # optimizer analyzes trajectories → edit patches
        3. Aggregate  # hierarchically merge similar patches
        4. Select     # rank & clip edits to the learning rate
        5. Update     # apply patches → candidate skill
        6. Gate       # score on selection split → accept / reject

    # epoch boundary (from epoch 2 onward)
    Slow update   # longitudinal comparison → protected guidance
    Meta skill    # cross-epoch optimizer memory

5.2 The Six Per-Step Stages #

5.3 Validation Gate #

evaluate_gate is a pure decision function. It compares the candidate's selection-set score against the current and best skills:

• accept_new_best — candidate > current and candidate > best → becomes both current and best.
• accept — candidate > current but ≤ best → becomes current only.
• reject — candidate ≤ current → discarded; current/best unchanged.

The comparison metric is configurable via evaluation.gate_metric:

5.4 Slow Update (Momentum) #

At each epoch boundary (from epoch 2), the slow update rolls out both the previous epoch's skill and the current skill on the same sampled tasks, categorizes items (improved / regressed / persistent-fail / stable-success), and asks the optimizer to write a free-form guidance block. This guidance lands in a protected region of the skill that step-level edits cannot touch — only the slow update overwrites it. It is SkillOpt's analogue of momentum, countering cross-epoch forgetting.

Acceptance has two modes, selected by optimizer.slow_update_gate_with_selection:

5.5 Meta Skill (Optimizer Memory) #

The meta skill is optimizer-side memory — it never modifies the target skill document. At the end of each epoch (skipped for epoch 1), the optimizer compares the previous and current epoch's last-step skills on the same sampled tasks and writes a compact, evidence-based reflection on what kind of edits helped or hurt. That memory is then injected as extra context into the next epoch's reflect / merge / learning-rate / ranking stages, so the optimizer accumulates strategy across the run.

5.6 Skill Document Anatomy #

A skill document is plain Markdown. Initial skills can be empty (learn from scratch) or seeded with domain knowledge via env.skill_init. During training the document accrues rules, patterns, and edge-case handling through accepted edit patches. A dedicated protected region holds the slow-update guidance, delimited by HTML-comment markers:

# Question Answering Skill

## Learned rules ...
- When the context contains multiple candidates, prefer ...

<!-- SLOW_UPDATE_START -->
# (epoch-level longitudinal guidance — only the slow update writes here)
<!-- SLOW_UPDATE_END -->

Helpers in optimizer/slow_update.py manage this region: inject_empty_slow_update_field (placeholder at epoch 1), extract_slow_update_field (read), and replace_slow_update_field (overwrite). Step-level edits are blocked from modifying anything inside the markers.

6.1 Configuration System #

Configs are structured YAML with section blocks (model, train, gradient, optimizer, evaluation, env) and _base_ inheritance. A benchmark config inherits the shared defaults and overrides only what differs:

# configs/searchqa/default.yaml
_base_: ../_base_/default.yaml
train:
  train_size: 400
  batch_size: 40
optimizer:
  learning_rate: 4
env:
  name: searchqa
  split_dir: data/searchqa_split

Override any key at the command line without editing files:

python scripts/train.py --config configs/searchqa/default.yaml \
  --cfg-options optimizer.learning_rate=16 optimizer.lr_scheduler=linear

6.2 model.* #

6.3 train.* #

6.4 gradient.* #

6.5 optimizer.* #

6.6 evaluation.* #

6.7 env.* #

7.1 Supported Benchmarks #

Each benchmark is a self-contained module under skillopt/envs/<benchmark>/ with an adapter.py, dataloader.py, rollout.py, and evaluator.py (some add a custom reflect.py). Packaged reference skills live in ckpt/<benchmark>/.

7.2 Add a New Benchmark #

Use skillopt/envs/_template/ as a starting point. At minimum, implement:

1. Dataloader — read your item JSON into the framework's item dicts (dataloader.py).
2. Rollout — run the target on one item with the current skill and return a trajectory + score (rollout.py).
3. Evaluator — score predictions against ground truth (evaluator.py).
4. Adapter — wire the above into the trainer's expected interface and register the env name (adapter.py).

Then add a configs/<name>/default.yaml inheriting _base_/default.yaml and set env.name to your new benchmark.

8.1 Module Map #

8.2 Core Functions #

8.3 CLI Scripts #

scripts/train.py

Runs a full training loop. Required: --config. Override config via --cfg-options section.key=value … or legacy flat flags (--num_epochs, --batch_size, --optimizer_model, --target_model, --lr_scheduler, --edit_budget, --split_dir, …).

scripts/eval_only.py

Evaluates a skill document without training. Required: --config and --skill. Use --split to choose train / valid_seen / valid_unseen / all.

python scripts/eval_only.py \
  --config configs/searchqa/default.yaml \
  --skill outputs/my_run/best_skill.md \
  --split valid_unseen

8.4 WebUI #

An optional Gradio dashboard to configure parameters and monitor runs:

pip install -e ".[webui]"
python -m skillopt_webui.app          # http://localhost:7860
python -m skillopt_webui.app --share  # public share link