Flow - 4x faster than typing
Mini toolseval-guide
A plugin for AI agent evaluation. Plan evals, generate test cases, interpret results for Copilot Studio agents. Grounded in Microsoft's Eval Scenario Library & Triage Playbook.
README
eval-guide
AI agent evaluation toolkit for Copilot Studio. Plan evals, generate test cases, interpret results, and triage failures — from Claude Code or GitHub Copilot.
Grounded in Microsoft's Eval Scenario Library, Triage & Improvement Playbook, Common Evaluation Approaches, and MS Learn agent evaluation documentation.
Install
Claude Code
claude plugin marketplace add microsoft/eval-guide
claude plugin install eval-guide@eval-guideGitHub Copilot
npx skills add microsoft/eval-guideSkills
| Skill | Command | What it does |
|---|---|---|
| Eval Guide | /eval-guide |
Full eval lifecycle — discover, plan, generate, run, interpret. Start here. |
| Eval Suite Planner | /eval-suite-planner |
Populated Eval Suite Template workbook plus an interactive HTML review page for eval sets, methods, gates, human inputs, and grader-validation notes |
| Eval Generator | /eval-generator |
Test cases for single-response and conversation (multi-turn) evaluation modes |
| Eval Result Interpreter | /eval-result-interpreter |
SHIP / ITERATE / BLOCK verdict with root cause classification |
| Eval Triage & Improvement | /eval-triage-and-improvement |
Interactive diagnosis and remediation for failing evals |
| Eval FAQ | /eval-faq |
Methodology questions answered from Microsoft's eval ecosystem |
Quick start
> /eval-guide
Tell me about your agent — what does it do, who uses it, and what does "good" look like?Works the same in both Claude Code and GitHub Copilot.
The toolkit walks you through five operational stages over Microsoft's Practical Guidance on Agent Evaluation — 10-step playbook (the canonical methodology spine, in skills/eval-guide/playbook.md):
| Stage | What happens | Playbook steps | Works without a running agent? |
|---|---|---|---|
| 0. Discover | Articulate what the agent does, what success looks like, the eval objective, the agent's risk tier (5 factors), and the owner | Step 1 | Yes |
| 1. Plan | Scope eval depth by agent architecture; plan capability vs trust & safety eval sets; set pass-rate targets and hard/soft gates; specify human inputs + source→ground-truth map | Steps 1, 4, 5 | Yes |
| 2. Generate & Baseline | Produce capability and trust & safety test-case CSVs (single-response) or conversation blueprints (multi-turn) importable into Copilot Studio; design the regression partition | Steps 2, 3, 8 (design) | Yes |
| 3. Run | Execute the baseline against a live agent | Step 6 | Needs running agent |
| 4. Interpret & Improve | Triage results, classify each failure (eval-setup vs agent-quality), gate-based verdict, design the optimization loop, flag reusable assets | Steps 7, 9, 10 | Needs eval results |
Stages 0-2 work from just an agent description — no running agent required.
Interactive dashboard review
Each stage generates an interactive HTML dashboard served locally in your browser. You review, edit inline, and confirm before the AI proceeds — no more back-and-forth in chat to fix test cases.
Stage complete → Dashboard opens → You review & edit → Confirm → Final artifacts generated| Stage | What you review in the dashboard | What you can edit |
|---|---|---|
| 0. Discover | Agent Vision (purpose, users, knowledge, capabilities, boundaries, success criteria) | All fields inline, add/remove list items |
| 1. Plan | Populated Eval Suite Template workbook plus HTML review page | Edit workbook cells without changing template structure; use the page to review summary, filters, TBDs, and checklist |
| 2. Generate | Test cases per eval set | Edit expected responses, questions, methods, add/remove cases |
| 4. Interpret | Verdict, failure triage, root causes, actions | Reclassify root causes, add comments |
Final deliverables (.docx reports, .csv test sets) are only generated after you confirm via the dashboard.
The dashboard is a standalone HTML file generated by skills/eval-guide/dashboard/serve.py (zero dependencies) and opened directly in your browser — no server required. Feedback auto-saves as you edit via localStorage — if the browser closes, your work is preserved.
Architecture-aware eval scoping
The toolkit automatically scopes evaluation depth based on your agent's architecture:
| Architecture | What gets tested |
|---|---|
| Prompt-level (simple Q&A, no knowledge sources) | Response quality, tone, boundaries, refusal behavior |
| RAG / Knowledge-grounded (has knowledge sources, no tools) | All of the above + retrieval accuracy, grounding, hallucination prevention |
| Agentic (multi-step, tool use, orchestration) | All of the above + tool selection, action correctness, error recovery, task completion |
A simple FAQ bot doesn't need tool-routing tests. A multi-step workflow agent does. The toolkit handles this so you test what actually matters.
Single-response and conversation evaluation
The eval generator supports both modes:
- Single-response — one input, one output. Produces a CSV importable directly into Copilot Studio. Supports all 7 test methods (General Quality, Compare Meaning, Keyword Match, Text Similarity, Exact Match, Capability Use, Custom).
- Conversation (multi-turn) — multi-turn dialogues for agents that handle context-dependent, multi-step tasks. Produces a structured blueprint for creating conversation test sets in Copilot Studio. Supports General Quality, Keyword Match, Capability Use, and Custom.
The skill detects which mode fits your agent and recommends accordingly.
Test data generation strategies
The planner recommends the right test data approach based on agent complexity:
| Approach | Best for |
|---|---|
| Echo | Single-turn Q&A, regression testing, deterministic checks |
| Historical replay | Model change comparisons, per-turn divergence analysis |
| Synthesized personas | Multi-step workflows, persona-dependent behavior, complex scenarios |
Most agents benefit from a hybrid: Echo for fast regression, Synthesized personas for realistic coverage.
What each skill produces
| Skill | Artifacts |
|---|---|
/eval-guide |
Workbook review plus Generate/Interpret dashboards, populated eval-suite workbook (.xlsx), test case CSVs, triage report (.docx) |
/eval-suite-planner |
Populated Eval Suite Template workbook plus interactive HTML review page with registry, gates, TBDs, baseline placeholders, and reusable candidates |
/eval-generator |
Copilot Studio-importable CSV (single-response) or conversation blueprint + .docx report |
/eval-result-interpreter |
SHIP/ITERATE/BLOCK verdict with root cause analysis and pattern detection |
/eval-triage-and-improvement |
Interactive remediation guidance with specific fixes per eval-set failure pattern |
/eval-faq |
Answers grounded in MS Learn, Eval Scenario Library, Triage Playbook |
Enhanced experience with Copilot Studio plugin (Claude Code)
For the full experience — connecting to a live agent, pulling its configuration, and running tests against it — also install the Copilot Studio plugin:
claude plugin marketplace add microsoft/skills-for-copilot-studio
claude plugin install skills-for-copilot-studio@skills-for-copilot-studioWhen both plugins are installed, /eval-guide can:
- Connect to your Copilot Studio agent via
/clone-agentand pull its real topics, knowledge sources, and configuration - Run test cases against the live agent via
/chat-with-agent - Ground the eval plan in what the agent actually does, not just what you describe
Without the Copilot Studio plugin (or when using GitHub Copilot), all skills work in description-based mode — you describe your agent and the skills generate plans and test cases from that description.
Example workflows
"I have an idea for an agent and want to know how to evaluate it"
/eval-guide I'm building an HR policy bot that answers employee questions from our SharePoint knowledge base"I have a plan and need test cases"
/eval-suite-planner Customer support agent handling refund requests, order tracking, and escalation to human agents
/eval-generator"My evals came back and I need to interpret them"
/eval-result-interpreter
> [paste CSV results or attach file]"Some tests are failing and I don't know why"
/eval-triage-and-improvement
> My agent scores 40% on knowledge grounding tests but 90% on general quality"Quick methodology question"
/eval-faq How is evaluating a multi-step workflow different from a simple Q&A agent?Repository structure
eval-guide/
├── .claude-plugin/ # Claude Code plugin configuration
│ ├── plugin.json
│ └── marketplace.json
├── .github/
│ ├── copilot-instructions.md # GitHub Copilot always-on instructions
│ └── prompts/ # GitHub Copilot prompt files
│ ├── eval-guide.prompt.md
│ ├── eval-suite-planner.prompt.md
│ ├── eval-generator.prompt.md
│ ├── eval-result-interpreter.prompt.md
│ ├── eval-triage-and-improvement.prompt.md
│ └── eval-faq.prompt.md
├── bin/ # CLI utilities
│ ├── eval-guide-update-check # Version check (local vs GitHub remote)
│ ├── eval-guide-update-snooze # Snooze upgrade reminders
│ └── eval-guide-update-config # Read/write ~/.eval-guide/config.yaml
├── skills/ # Claude Code skills (SKILL.md format)
│ ├── eval-guide/
│ │ ├── SKILL.md
│ │ ├── scripts/
│ │ │ └── eval-runner.js # Run evals via DirectLine API
│ │ └── dashboard/ # Interactive review dashboards
│ │ ├── serve.py # Python server (zero dependencies)
│ │ └── templates/ # Stage-specific HTML templates
│ │ ├── base.html # Shared layout, CSS, feedback JS
│ │ ├── discover.html # Stage 0: Agent Vision review
│ │ ├── plan.html # Stage 1: Eval plan review
│ │ ├── generate.html # Stage 2: Test cases (editable)
│ │ └── interpret.html# Stage 4: Triage report review
│ ├── eval-suite-planner/
│ ├── eval-generator/
│ ├── eval-result-interpreter/
│ ├── eval-triage-and-improvement/
│ └── eval-faq/
├── AGENTS.md # Agent instructions (GitHub Copilot agent mode + other AI tools)
├── CLAUDE.md # Claude Code project instructions
├── README.md
└── VERSION # Current release version (semver)Methodology
This toolkit is grounded in Microsoft's Practical Guidance on Agent Evaluation — a 10-step playbook. The canonical spine lives in skills/eval-guide/playbook.md: plan the effort (risk tier) → build capability eval sets → build trust & safety eval sets → set pass-rate targets & gates → specify human inputs → run the baseline → iterate to diagnose → regression suite → optimization loop → save reusable assets. Supporting Microsoft sources:
- Eval Scenario Library — 5 business-problem + 9 capability scenario types
- Triage & Improvement Playbook — root cause classification (eval-setup vs agent-quality)
- Common Evaluation Approaches — Echo, Historical Replay, Synthesized Personas; code-based vs LLM-judge graders
- Evaluation Frameworks — scenario validation themes
- MS Learn agent evaluation docs — test methods, comparative testing, rubric-based grading
Contributing
This project welcomes contributions and suggestions. See CONTRIBUTING.md for details.
