1# MCP Server Evaluation Guide
2 
3## Overview
4 
5This document provides guidance on creating comprehensive evaluations for MCP servers. Evaluations test whether LLMs can effectively use your MCP server to answer realistic, complex questions using only the tools provided.
6 
7---
8 
9## Quick Reference
10 
11### Evaluation Requirements
12- Create 10 human-readable questions
13- Questions must be READ-ONLY, INDEPENDENT, NON-DESTRUCTIVE
14- Each question requires multiple tool calls (potentially dozens)
15- Answers must be single, verifiable values
16- Answers must be STABLE (won't change over time)
17 
18### Output Format
19```xml
20<evaluation>
21   <qa_pair>
22      <question>Your question here</question>
23      <answer>Single verifiable answer</answer>
24   </qa_pair>
25</evaluation>
26```
27 
28---
29 
30## Purpose of Evaluations
31 
32The measure of quality of an MCP server is NOT how well or comprehensively the server implements tools, but how well these implementations (input/output schemas, docstrings/descriptions, functionality) enable LLMs with no other context and access ONLY to the MCP servers to answer realistic and difficult questions.
33 
34## Evaluation Overview
35 
36Create 10 human-readable questions requiring ONLY READ-ONLY, INDEPENDENT, NON-DESTRUCTIVE, and IDEMPOTENT operations to answer. Each question should be:
37- Realistic
38- Clear and concise
39- Unambiguous
40- Complex, requiring potentially dozens of tool calls or steps
41- Answerable with a single, verifiable value that you identify in advance
42 
43## Question Guidelines
44 
45### Core Requirements
46 
471. **Questions MUST be independent**
48   - Each question should NOT depend on the answer to any other question
49   - Should not assume prior write operations from processing another question
50 
512. **Questions MUST require ONLY NON-DESTRUCTIVE AND IDEMPOTENT tool use**
52   - Should not instruct or require modifying state to arrive at the correct answer
53 
543. **Questions must be REALISTIC, CLEAR, CONCISE, and COMPLEX**
55   - Must require another LLM to use multiple (potentially dozens of) tools or steps to answer
56 
57### Complexity and Depth
58 
594. **Questions must require deep exploration**
60   - Consider multi-hop questions requiring multiple sub-questions and sequential tool calls
61   - Each step should benefit from information found in previous questions
62 
635. **Questions may require extensive paging**
64   - May need paging through multiple pages of results
65   - May require querying old data (1-2 years out-of-date) to find niche information
66   - The questions must be DIFFICULT
67 
686. **Questions must require deep understanding**
69   - Rather than surface-level knowledge
70   - May pose complex ideas as True/False questions requiring evidence
71   - May use multiple-choice format where LLM must search different hypotheses
72 
737. **Questions must not be solvable with straightforward keyword search**
74   - Do not include specific keywords from the target content
75   - Use synonyms, related concepts, or paraphrases
76   - Require multiple searches, analyzing multiple related items, extracting context, then deriving the answer
77 
78### Tool Testing
79 
808. **Questions should stress-test tool return values**
81   - May elicit tools returning large JSON objects or lists, overwhelming the LLM
82   - Should require understanding multiple modalities of data:
83     - IDs and names
84     - Timestamps and datetimes (months, days, years, seconds)
85     - File IDs, names, extensions, and mimetypes
86     - URLs, GIDs, etc.
87   - Should probe the tool's ability to return all useful forms of data
88 
899. **Questions should MOSTLY reflect real human use cases**
90   - The kinds of information retrieval tasks that HUMANS assisted by an LLM would care about
91 
9210. **Questions may require dozens of tool calls**
93    - This challenges LLMs with limited context
94    - Encourages MCP server tools to reduce information returned
95 
9611. **Include ambiguous questions**
97    - May be ambiguous OR require difficult decisions on which tools to call
98    - Force the LLM to potentially make mistakes or misinterpret
99    - Ensure that despite AMBIGUITY, there is STILL A SINGLE VERIFIABLE ANSWER
100 
101### Stability
102 
10312. **Questions must be designed so the answer DOES NOT CHANGE**
104    - Do not ask questions that rely on "current state" which is dynamic
105    - For example, do not count:
106      - Number of reactions to a post
107      - Number of replies to a thread
108      - Number of members in a channel
109 
11013. **DO NOT let the MCP server RESTRICT the kinds of questions you create**
111    - Create challenging and complex questions
112    - Some may not be solvable with the available MCP server tools
113    - Questions may require specific output formats (datetime vs. epoch time, JSON vs. MARKDOWN)
114    - Questions may require dozens of tool calls to complete
115 
116## Answer Guidelines
117 
118### Verification
119 
1201. **Answers must be VERIFIABLE via direct string comparison**
121   - If the answer can be re-written in many formats, clearly specify the output format in the QUESTION
122   - Examples: "Use YYYY/MM/DD.", "Respond True or False.", "Answer A, B, C, or D and nothing else."
123   - Answer should be a single VERIFIABLE value such as:
124     - User ID, user name, display name, first name, last name
125     - Channel ID, channel name
126     - Message ID, string
127     - URL, title
128     - Numerical quantity
129     - Timestamp, datetime
130     - Boolean (for True/False questions)
131     - Email address, phone number
132     - File ID, file name, file extension
133     - Multiple choice answer
134   - Answers must not require special formatting or complex, structured output
135   - Answer will be verified using DIRECT STRING COMPARISON
136 
137### Readability
138 
1392. **Answers should generally prefer HUMAN-READABLE formats**
140   - Examples: names, first name, last name, datetime, file name, message string, URL, yes/no, true/false, a/b/c/d
141   - Rather than opaque IDs (though IDs are acceptable)
142   - The VAST MAJORITY of answers should be human-readable
143 
144### Stability
145 
1463. **Answers must be STABLE/STATIONARY**
147   - Look at old content (e.g., conversations that have ended, projects that have launched, questions answered)
148   - Create QUESTIONS based on "closed" concepts that will always return the same answer
149   - Questions may ask to consider a fixed time window to insulate from non-stationary answers
150   - Rely on context UNLIKELY to change
151   - Example: if finding a paper name, be SPECIFIC enough so answer is not confused with papers published later
152 
1534. **Answers must be CLEAR and UNAMBIGUOUS**
154   - Questions must be designed so there is a single, clear answer
155   - Answer can be derived from using the MCP server tools
156 
157### Diversity
158 
1595. **Answers must be DIVERSE**
160   - Answer should be a single VERIFIABLE value in diverse modalities and formats
161   - User concept: user ID, user name, display name, first name, last name, email address, phone number
162   - Channel concept: channel ID, channel name, channel topic
163   - Message concept: message ID, message string, timestamp, month, day, year
164 
1656. **Answers must NOT be complex structures**
166   - Not a list of values
167   - Not a complex object
168   - Not a list of IDs or strings
169   - Not natural language text
170   - UNLESS the answer can be straightforwardly verified using DIRECT STRING COMPARISON
171   - And can be realistically reproduced
172   - It should be unlikely that an LLM would return the same list in any other order or format
173 
174## Evaluation Process
175 
176### Step 1: Documentation Inspection
177 
178Read the documentation of the target API to understand:
179- Available endpoints and functionality
180- If ambiguity exists, fetch additional information from the web
181- Parallelize this step AS MUCH AS POSSIBLE
182- Ensure each subagent is ONLY examining documentation from the file system or on the web
183 
184### Step 2: Tool Inspection
185 
186List the tools available in the MCP server:
187- Inspect the MCP server directly
188- Understand input/output schemas, docstrings, and descriptions
189- WITHOUT calling the tools themselves at this stage
190 
191### Step 3: Developing Understanding
192 
193Repeat steps 1 & 2 until you have a good understanding:
194- Iterate multiple times
195- Think about the kinds of tasks you want to create
196- Refine your understanding
197- At NO stage should you READ the code of the MCP server implementation itself
198- Use your intuition and understanding to create reasonable, realistic, but VERY challenging tasks
199 
200### Step 4: Read-Only Content Inspection
201 
202After understanding the API and tools, USE the MCP server tools:
203- Inspect content using READ-ONLY and NON-DESTRUCTIVE operations ONLY
204- Goal: identify specific content (e.g., users, channels, messages, projects, tasks) for creating realistic questions
205- Should NOT call any tools that modify state
206- Will NOT read the code of the MCP server implementation itself
207- Parallelize this step with individual sub-agents pursuing independent explorations
208- Ensure each subagent is only performing READ-ONLY, NON-DESTRUCTIVE, and IDEMPOTENT operations
209- BE CAREFUL: SOME TOOLS may return LOTS OF DATA which would cause you to run out of CONTEXT
210- Make INCREMENTAL, SMALL, AND TARGETED tool calls for exploration
211- In all tool call requests, use the `limit` parameter to limit results (<10)
212- Use pagination
213 
214### Step 5: Task Generation
215 
216After inspecting the content, create 10 human-readable questions:
217- An LLM should be able to answer these with the MCP server
218- Follow all question and answer guidelines above
219 
220## Output Format
221 
222Each QA pair consists of a question and an answer. The output should be an XML file with this structure:
223 
224```xml
225<evaluation>
226   <qa_pair>
227      <question>Find the project created in Q2 2024 with the highest number of completed tasks. What is the project name?</question>
228      <answer>Website Redesign</answer>
229   </qa_pair>
230   <qa_pair>
231      <question>Search for issues labeled as "bug" that were closed in March 2024. Which user closed the most issues? Provide their username.</question>
232      <answer>sarah_dev</answer>
233   </qa_pair>
234   <qa_pair>
235      <question>Look for pull requests that modified files in the /api directory and were merged between January 1 and January 31, 2024. How many different contributors worked on these PRs?</question>
236      <answer>7</answer>
237   </qa_pair>
238   <qa_pair>
239      <question>Find the repository with the most stars that was created before 2023. What is the repository name?</question>
240      <answer>data-pipeline</answer>
241   </qa_pair>
242</evaluation>
243```
244 
245## Evaluation Examples
246 
247### Good Questions
248 
249**Example 1: Multi-hop question requiring deep exploration (GitHub MCP)**
250```xml
251<qa_pair>
252   <question>Find the repository that was archived in Q3 2023 and had previously been the most forked project in the organization. What was the primary programming language used in that repository?</question>
253   <answer>Python</answer>
254</qa_pair>
255```
256 
257This question is good because:
258- Requires multiple searches to find archived repositories
259- Needs to identify which had the most forks before archival
260- Requires examining repository details for the language
261- Answer is a simple, verifiable value
262- Based on historical (closed) data that won't change
263 
264**Example 2: Requires understanding context without keyword matching (Project Management MCP)**
265```xml
266<qa_pair>
267   <question>Locate the initiative focused on improving customer onboarding that was completed in late 2023. The project lead created a retrospective document after completion. What was the lead's role title at that time?</question>
268   <answer>Product Manager</answer>
269</qa_pair>
270```
271 
272This question is good because:
273- Doesn't use specific project name ("initiative focused on improving customer onboarding")
274- Requires finding completed projects from specific timeframe
275- Needs to identify the project lead and their role
276- Requires understanding context from retrospective documents
277- Answer is human-readable and stable
278- Based on completed work (won't change)
279 
280**Example 3: Complex aggregation requiring multiple steps (Issue Tracker MCP)**
281```xml
282<qa_pair>
283   <question>Among all bugs reported in January 2024 that were marked as critical priority, which assignee resolved the highest percentage of their assigned bugs within 48 hours? Provide the assignee's username.</question>
284   <answer>alex_eng</answer>
285</qa_pair>
286```
287 
288This question is good because:
289- Requires filtering bugs by date, priority, and status
290- Needs to group by assignee and calculate resolution rates
291- Requires understanding timestamps to determine 48-hour windows
292- Tests pagination (potentially many bugs to process)
293- Answer is a single username
294- Based on historical data from specific time period
295 
296**Example 4: Requires synthesis across multiple data types (CRM MCP)**
297```xml
298<qa_pair>
299   <question>Find the account that upgraded from the Starter to Enterprise plan in Q4 2023 and had the highest annual contract value. What industry does this account operate in?</question>
300   <answer>Healthcare</answer>
301</qa_pair>
302```
303 
304This question is good because:
305- Requires understanding subscription tier changes
306- Needs to identify upgrade events in specific timeframe
307- Requires comparing contract values
308- Must access account industry information
309- Answer is simple and verifiable
310- Based on completed historical transactions
311 
312### Poor Questions
313 
314**Example 1: Answer changes over time**
315```xml
316<qa_pair>
317   <question>How many open issues are currently assigned to the engineering team?</question>
318   <answer>47</answer>
319</qa_pair>
320```
321 
322This question is poor because:
323- The answer will change as issues are created, closed, or reassigned
324- Not based on stable/stationary data
325- Relies on "current state" which is dynamic
326 
327**Example 2: Too easy with keyword search**
328```xml
329<qa_pair>
330   <question>Find the pull request with title "Add authentication feature" and tell me who created it.</question>
331   <answer>developer123</answer>
332</qa_pair>
333```
334 
335This question is poor because:
336- Can be solved with a straightforward keyword search for exact title
337- Doesn't require deep exploration or understanding
338- No synthesis or analysis needed
339 
340**Example 3: Ambiguous answer format**
341```xml
342<qa_pair>
343   <question>List all the repositories that have Python as their primary language.</question>
344   <answer>repo1, repo2, repo3, data-pipeline, ml-tools</answer>
345</qa_pair>
346```
347 
348This question is poor because:
349- Answer is a list that could be returned in any order
350- Difficult to verify with direct string comparison
351- LLM might format differently (JSON array, comma-separated, newline-separated)
352- Better to ask for a specific aggregate (count) or superlative (most stars)
353 
354## Verification Process
355 
356After creating evaluations:
357 
3581. **Examine the XML file** to understand the schema
3592. **Load each task instruction** and in parallel using the MCP server and tools, identify the correct answer by attempting to solve the task YOURSELF
3603. **Flag any operations** that require WRITE or DESTRUCTIVE operations
3614. **Accumulate all CORRECT answers** and replace any incorrect answers in the document
3625. **Remove any `<qa_pair>`** that require WRITE or DESTRUCTIVE operations
363 
364Remember to parallelize solving tasks to avoid running out of context, then accumulate all answers and make changes to the file at the end.
365 
366## Tips for Creating Quality Evaluations
367 
3681. **Think Hard and Plan Ahead** before generating tasks
3692. **Parallelize Where Opportunity Arises** to speed up the process and manage context
3703. **Focus on Realistic Use Cases** that humans would actually want to accomplish
3714. **Create Challenging Questions** that test the limits of the MCP server's capabilities
3725. **Ensure Stability** by using historical data and closed concepts
3736. **Verify Answers** by solving the questions yourself using the MCP server tools
3747. **Iterate and Refine** based on what you learn during the process
375 
376---
377 
378# Running Evaluations
379 
380After creating your evaluation file, you can use the provided evaluation harness to test your MCP server.
381 
382## Setup
383 
3841. **Install Dependencies**
385 
386   ```bash
387   pip install -r scripts/requirements.txt
388   ```
389 
390   Or install manually:
391   ```bash
392   pip install anthropic mcp
393   ```
394 
3952. **Set API Key**
396 
397   ```bash
398   export ANTHROPIC_API_KEY=your_api_key_here
399   ```
400 
401## Evaluation File Format
402 
403Evaluation files use XML format with `<qa_pair>` elements:
404 
405```xml
406<evaluation>
407   <qa_pair>
408      <question>Find the project created in Q2 2024 with the highest number of completed tasks. What is the project name?</question>
409      <answer>Website Redesign</answer>
410   </qa_pair>
411   <qa_pair>
412      <question>Search for issues labeled as "bug" that were closed in March 2024. Which user closed the most issues? Provide their username.</question>
413      <answer>sarah_dev</answer>
414   </qa_pair>
415</evaluation>
416```
417 
418## Running Evaluations
419 
420The evaluation script (`scripts/evaluation.py`) supports three transport types:
421 
422**Important:**
423- **stdio transport**: The evaluation script automatically launches and manages the MCP server process for you. Do not run the server manually.
424- **sse/http transports**: You must start the MCP server separately before running the evaluation. The script connects to the already-running server at the specified URL.
425 
426### 1. Local STDIO Server
427 
428For locally-run MCP servers (script launches the server automatically):
429 
430```bash
431python scripts/evaluation.py \
432  -t stdio \
433  -c python \
434  -a my_mcp_server.py \
435  evaluation.xml
436```
437 
438With environment variables:
439```bash
440python scripts/evaluation.py \
441  -t stdio \
442  -c python \
443  -a my_mcp_server.py \
444  -e API_KEY=abc123 \
445  -e DEBUG=true \
446  evaluation.xml
447```
448 
449### 2. Server-Sent Events (SSE)
450 
451For SSE-based MCP servers (you must start the server first):
452 
453```bash
454python scripts/evaluation.py \
455  -t sse \
456  -u https://example.com/mcp \
457  -H "Authorization: Bearer token123" \
458  -H "X-Custom-Header: value" \
459  evaluation.xml
460```
461 
462### 3. HTTP (Streamable HTTP)
463 
464For HTTP-based MCP servers (you must start the server first):
465 
466```bash
467python scripts/evaluation.py \
468  -t http \
469  -u https://example.com/mcp \
470  -H "Authorization: Bearer token123" \
471  evaluation.xml
472```
473 
474## Command-Line Options
475 
476```
477usage: evaluation.py [-h] [-t {stdio,sse,http}] [-m MODEL] [-c COMMAND]
478                     [-a ARGS [ARGS ...]] [-e ENV [ENV ...]] [-u URL]
479                     [-H HEADERS [HEADERS ...]] [-o OUTPUT]
480                     eval_file
481 
482positional arguments:
483  eval_file             Path to evaluation XML file
484 
485optional arguments:
486  -h, --help            Show help message
487  -t, --transport       Transport type: stdio, sse, or http (default: stdio)
488  -m, --model           Claude model to use (default: claude-3-7-sonnet-20250219)
489  -o, --output          Output file for report (default: print to stdout)
490 
491stdio options:
492  -c, --command         Command to run MCP server (e.g., python, node)
493  -a, --args            Arguments for the command (e.g., server.py)
494  -e, --env             Environment variables in KEY=VALUE format
495 
496sse/http options:
497  -u, --url             MCP server URL
498  -H, --header          HTTP headers in 'Key: Value' format
499```
500 
501## Output
502 
503The evaluation script generates a detailed report including:
504 
505- **Summary Statistics**:
506  - Accuracy (correct/total)
507  - Average task duration
508  - Average tool calls per task
509  - Total tool calls
510 
511- **Per-Task Results**:
512  - Prompt and expected response
513  - Actual response from the agent
514  - Whether the answer was correct (✅/❌)
515  - Duration and tool call details
516  - Agent's summary of its approach
517  - Agent's feedback on the tools
518 
519### Save Report to File
520 
521```bash
522python scripts/evaluation.py \
523  -t stdio \
524  -c python \
525  -a my_server.py \
526  -o evaluation_report.md \
527  evaluation.xml
528```
529 
530## Complete Example Workflow
531 
532Here's a complete example of creating and running an evaluation:
533 
5341. **Create your evaluation file** (`my_evaluation.xml`):
535 
536```xml
537<evaluation>
538   <qa_pair>
539      <question>Find the user who created the most issues in January 2024. What is their username?</question>
540      <answer>alice_developer</answer>
541   </qa_pair>
542   <qa_pair>
543      <question>Among all pull requests merged in Q1 2024, which repository had the highest number? Provide the repository name.</question>
544      <answer>backend-api</answer>
545   </qa_pair>
546   <qa_pair>
547      <question>Find the project that was completed in December 2023 and had the longest duration from start to finish. How many days did it take?</question>
548      <answer>127</answer>
549   </qa_pair>
550</evaluation>
551```
552 
5532. **Install dependencies**:
554 
555```bash
556pip install -r scripts/requirements.txt
557export ANTHROPIC_API_KEY=your_api_key
558```
559 
5603. **Run evaluation**:
561 
562```bash
563python scripts/evaluation.py \
564  -t stdio \
565  -c python \
566  -a github_mcp_server.py \
567  -e GITHUB_TOKEN=ghp_xxx \
568  -o github_eval_report.md \
569  my_evaluation.xml
570```
571 
5724. **Review the report** in `github_eval_report.md` to:
573   - See which questions passed/failed
574   - Read the agent's feedback on your tools
575   - Identify areas for improvement
576   - Iterate on your MCP server design
577 
578## Troubleshooting
579 
580### Connection Errors
581 
582If you get connection errors:
583- **STDIO**: Verify the command and arguments are correct
584- **SSE/HTTP**: Check the URL is accessible and headers are correct
585- Ensure any required API keys are set in environment variables or headers
586 
587### Low Accuracy
588 
589If many evaluations fail:
590- Review the agent's feedback for each task
591- Check if tool descriptions are clear and comprehensive
592- Verify input parameters are well-documented
593- Consider whether tools return too much or too little data
594- Ensure error messages are actionable
595 
596### Timeout Issues
597 
598If tasks are timing out:
599- Use a more capable model (e.g., `claude-3-7-sonnet-20250219`)
600- Check if tools are returning too much data
601- Verify pagination is working correctly
602- Consider simplifying complex questions