This post is a part of the AB-620: Designing and Building Integrated AI Agent Solutions in Copilot Studio Exam Prep Hub.
This topic falls under these sections:
Test and manage agents (20–25%)
--> Evaluate agent performance
--> Create a test set
Note that there are 10 practice questions (with answers) at the end of each section to help you solidify your knowledge of the material. Also, there are 4 practice tests with 30 questions each available from the hub's main page below the exam topics section.
Introduction
One of the most important responsibilities of an AI Agent Builder is ensuring that an agent consistently produces accurate, relevant, and safe responses. As agents become more sophisticated and rely on multiple knowledge sources, tools, and generative AI models, manual testing alone is no longer sufficient.
Microsoft Copilot Studio provides test sets that allow developers to systematically validate agent behavior against expected outcomes. Test sets enable repeatable evaluation of an agent after configuration changes, prompt updates, knowledge source modifications, or model upgrades.
For the AB-620 exam, you should understand:
- What test sets are
- Why they are important
- How to create and manage them
- How they integrate with agent evaluation
- Best practices for maintaining reliable test coverage
What Is a Test Set?
A test set is a collection of predefined test cases that evaluate how an AI agent responds to expected user requests.
Each test case generally contains:
- A sample user prompt
- The expected behavior or outcome
- Evaluation criteria
- Pass/fail results after execution
Instead of manually asking the same questions every time changes are made, developers can rerun the entire test set to determine whether the agent continues to behave correctly.
Why Test Sets Matter
Without structured testing:
- New prompts may unintentionally break previous functionality.
- Updated knowledge sources may introduce incorrect answers.
- Tool changes may fail silently.
- Model updates may alter response quality.
Test sets provide confidence that the agent still behaves correctly after changes.
Benefits include:
- Repeatable testing
- Faster validation
- Regression testing
- Improved response quality
- Easier troubleshooting
- Better release confidence
Test Set vs Manual Testing
| Manual Testing | Test Set |
|---|---|
| Performed interactively | Executed repeatedly |
| Difficult to reproduce | Fully repeatable |
| Human remembers questions | Questions stored permanently |
| Time consuming | Automated evaluation |
| Easy to miss scenarios | Covers many scenarios consistently |
When Should You Create a Test Set?
Create a test set whenever:
- Building a new agent
- Adding new topics
- Adding knowledge sources
- Adding tools
- Integrating APIs
- Updating prompts
- Deploying a new version
- Performing regression testing
Components of a Test Case
A typical test case includes several important elements.
1. User Input
The question or request submitted to the agent.
Example:
“Show me my remaining vacation balance.”
2. Expected Behavior
The desired outcome.
Examples include:
- Calls HR connector
- Retrieves employee record
- Returns vacation balance
- Does not hallucinate data
3. Expected Response
Depending on the evaluation method, expected responses may include:
- Specific wording
- Required information
- Correct tool usage
- Accurate citation
- Proper formatting
4. Evaluation Result
After execution the test produces results such as:
- Pass
- Fail
- Partial success
- Confidence score (where applicable)
Types of Test Cases
A comprehensive test set should include multiple categories.
Happy Path Tests
Expected user behavior.
Example:
“Reset my password.”
Alternative Wording
Different ways users ask the same question.
Examples:
- I forgot my password
- Help me log in
- I can’t sign in
Edge Cases
Unusual but valid requests.
Example:
“Can I reset someone else’s password?”
Invalid Requests
Questions the agent should decline.
Example:
“Delete every employee record.”
Ambiguous Questions
The agent should ask follow-up questions.
Example:
“Book a meeting.”
Expected behavior:
“Who should I invite?”
Tool Failure Tests
Verify graceful handling of failures.
Example:
API unavailable.
Expected response:
“The HR system is temporarily unavailable.”
Knowledge Tests
Ensure retrieval from enterprise knowledge.
Example:
“What is the travel reimbursement policy?”
Security Tests
Confirm proper authorization.
Example:
Employee requests another employee’s payroll information.
Expected behavior:
Access denied.
Creating a Test Set
The general workflow is:
Step 1
Open the agent in Copilot Studio.
Step 2
Navigate to testing or evaluation features.
Step 3
Create a new test set.
Step 4
Add individual test cases.
Each includes:
- Prompt
- Expected behavior
- Expected response
Step 5
Save the test set.
Step 6
Run the evaluation.
Step 7
Review results.
Step 8
Improve the agent if failures occur.
Step 9
Run the test set again.
Organizing Test Sets
Large enterprise agents often use multiple test sets.
Examples:
- HR Agent
- Finance Agent
- IT Help Desk
- Customer Service
- Sales Support
Within each, additional test groups may cover:
- Authentication
- Knowledge retrieval
- API actions
- Escalation
- Security
- Generative responses
Regression Testing
Regression testing verifies that new changes do not introduce unexpected problems.
Example:
Original agent answers:
“How do I request PTO?”
A new HR connector is added.
Running the existing test set confirms the answer still works correctly.
Without regression testing, developers may unknowingly introduce defects.
Testing Knowledge Retrieval
Knowledge-based agents should verify:
- Correct document selected
- Correct section retrieved
- Accurate citation
- Relevant answer
- No hallucinated content
Example test:
Question:
“What is the expense reimbursement limit?”
Expected:
- Searches indexed documents
- Retrieves finance policy
- Returns correct limit
- Includes citation if configured
Testing Tool Invocation
For action-based agents, verify that the correct tool is selected.
Example:
User:
“Create a support ticket.”
Expected:
- IT connector invoked
- Ticket created
- Ticket number returned
Failure examples:
- Wrong connector called
- No connector called
- Hallucinated confirmation
Testing Multi-Agent Solutions
If delegation is used, verify:
- Correct child agent selected
- Successful delegation
- Response returned
- Parent continues conversation properly
Testing Generative AI
Generative responses require additional evaluation.
Verify:
- Factual accuracy
- Completeness
- Grounding
- Tone
- Safety
- Relevance
Evaluating Test Results
After execution, review:
- Overall pass rate
- Failed cases
- Tool execution
- Knowledge retrieval
- Response quality
- Latency
- Error messages
Common questions include:
- Did the correct tool run?
- Was the answer accurate?
- Was sensitive data protected?
- Was grounding successful?
Common Reasons Tests Fail
Failures often result from:
- Prompt changes
- Missing connector permissions
- API failures
- Incorrect tool selection
- Poor grounding
- Hallucinations
- Missing documents
- Authentication problems
- Incorrect routing
Best Practices
Microsoft recommends several best practices.
Build Early
Create test cases while building the agent.
Cover Real User Questions
Use production-like prompts whenever possible.
Include Variations
People ask the same question differently.
Test all common variations.
Test Negative Scenarios
Don’t only verify success.
Test:
- Errors
- Permission failures
- Invalid input
- Ambiguous requests
Keep Test Sets Updated
Whenever the agent changes:
- Add new tests
- Remove obsolete tests
- Update expected responses
Run Tests Frequently
Execute the full test set:
- Before deployment
- After model updates
- After connector updates
- After knowledge updates
- After prompt revisions
Exam Tips
For the AB-620 exam, remember:
- Test sets enable repeatable evaluation.
- They support regression testing.
- Good test cases include expected behavior.
- Test sets should include positive, negative, and edge-case scenarios.
- Multi-agent solutions require delegation testing.
- Tool-based agents require tool invocation validation.
- Knowledge agents require grounding verification.
- Test sets improve deployment confidence.
Practice Exam Questions
Question 1
Why is creating a test set preferable to relying solely on manual testing?
A. It permanently stores conversation history for users.
B. It provides repeatable, consistent evaluation of agent behavior.
C. It automatically retrains the language model.
D. It removes the need for production monitoring.
Answer: B
Explanation: Test sets allow the same scenarios to be executed repeatedly, making regression testing and validation much more reliable than manual testing.
Question 2
Which type of scenario should always be included in a comprehensive test set?
A. Only successful user interactions
B. Only connector failures
C. Positive, negative, and edge-case scenarios
D. Only knowledge retrieval questions
Answer: C
Explanation: Comprehensive testing includes normal requests, invalid inputs, ambiguous questions, security scenarios, and failure conditions.
Question 3
A developer updates an HR connector used by an agent. What is the best next step?
A. Run the existing test set to perform regression testing.
B. Delete all previous test cases.
C. Retrain the foundation model.
D. Create a new environment.
Answer: A
Explanation: Regression testing verifies that previously working functionality continues to operate after changes.
Question 4
Which component defines what a successful test should accomplish?
A. Conversation history
B. Agent version
C. Workspace settings
D. Expected behavior
Answer: D
Explanation: Expected behavior specifies the desired outcome that the agent should achieve during the test.
Question 5
A knowledge-based agent answers a company policy question using outdated information. Which area of testing should identify this issue?
A. User authentication testing
B. Knowledge retrieval testing
C. Network latency testing
D. Adaptive Card rendering
Answer: B
Explanation: Knowledge retrieval tests verify that the correct documents are located and that accurate, grounded information is returned.
Question 6
When testing an action that creates a support ticket, what should the evaluation confirm?
A. Only that the response is grammatically correct
B. Only that the response is polite
C. That the correct tool or connector was invoked successfully
D. That the conversation contains at least three turns
Answer: C
Explanation: Action-based tests should verify successful tool invocation and the expected outcome of that action.
Question 7
Why should multiple phrasings of the same request be included in a test set?
A. To increase the size of the knowledge base
B. To improve authentication
C. To ensure the agent recognizes natural language variations
D. To reduce connector latency
Answer: C
Explanation: Users ask the same question in many different ways, and the agent should respond correctly to common variations.
Question 8
Which situation best represents an edge-case test?
A. “Reset my password.”
B. “Show today’s weather.”
C. “Create a support ticket.”
D. “Can I reset another employee’s password?”
Answer: D
Explanation: This unusual but valid request tests whether the agent correctly handles authorization and security.
Question 9
An agent delegates requests to multiple child agents. What should testing verify?
A. That delegation occurs to the appropriate child agent and responses are returned correctly
B. That every child agent uses the same prompt
C. That delegation is disabled after deployment
D. That all child agents share one knowledge source
Answer: A
Explanation: Multi-agent testing ensures that routing, delegation, and response aggregation function as designed.
Question 10
Which statement best describes the primary purpose of regression testing?
A. Measuring internet bandwidth
B. Evaluating user satisfaction surveys
C. Ensuring that recent changes have not broken existing functionality
D. Generating additional knowledge documents
Answer: C
Explanation: Regression testing validates that existing capabilities continue to work correctly after updates to prompts, connectors, tools, or knowledge sources.
Go to the AB-620 Exam Prep Hub main page
The Data Community 