| Workflow | Package / command | Use it for |
|---|---|---|
| Research engine | skillopt, skillopt-train, skillopt-eval |
Train and evaluate skill documents on explicit benchmark splits. |
| SkillOpt-Sleep (preview) | skillopt_sleep, skillopt-sleep |
Review supported coding-agent sessions and stage proposed memory/skill updates for human adoption. |
They share the idea of bounded text updates and validation, but they are separate entry points with different configs and safety boundaries. Start with the SkillOpt-Sleep overview before using real session data.
SkillOpt brings the familiar deep-learning training paradigm to agentic prompt optimization:
| Deep Learning | SkillOpt |
|---|---|
| Model weights | Skill document (Markdown) |
| Forward pass | Rollout (target executes tasks) |
| Loss / gradient | Reflect (optimizer produces edit patches) |
| Gradient clipping | Edit selection (learning_rate = max edits) |
| SGD step | Patch application to skill |
| Validation set | Gated evaluation on selection split |
| LR schedule | lr_scheduler: cosine, linear, constant |
| Epochs | Multi-epoch with slow update & meta skill memory |
| Benchmark | Type | Config |
|---|---|---|
| DocVQA | Document QA | configs/docvqa/ |
| ALFWorld | Embodied AI | configs/alfworld/ |
| OfficeQA | Enterprise QA | configs/officeqa/ |
| SearchQA | Open-domain QA | configs/searchqa/ |
| LiveMathematicianBench | Math reasoning | configs/livemathematicianbench/ |
| SpreadsheetBench | Spreadsheet editing | configs/spreadsheetbench/ |
Optimizer and target roles are configured separately. Chat backends include
Azure OpenAI (openai_chat), the provider-neutral
openai_compatible backend, the Claude Code CLI (claude_chat), Qwen, and
MiniMax. Codex and Claude Code exec harnesses are target-only and require
adapter support. Despite its name, claude_chat launches claude -p; it is
not a direct Anthropic API client.
If a provider implements OpenAI Chat Completions, begin with the built-in compatible backend instead of adding a new integration. See Configuration for authentication and per-role overrides.
# Clone and install the research checkout plus the SearchQA data extra
git clone https://github.com/microsoft/SkillOpt.git
cd SkillOpt
python -m pip install -e ".[searchqa]"
# Configure credentials (choose one auth mode in .env)
cp .env.example .env
set -a; source .env; set +a
# Materialize the runnable split from the checked-in ID manifest
python scripts/materialize_searchqa.py
# Train on SearchQA into a predictable output directory
python scripts/train.py \
--config configs/searchqa/default.yaml \
--out_root outputs/searchqa_quickstart
# Evaluate best skill
python scripts/eval_only.py \
--config configs/searchqa/default.yaml \
--skill outputs/searchqa_quickstart/best_skill.md \
--split valid_unseen