AB-620, AI, AI Governance, Azure AI, Microsoft Certification

Create a test set (AB-620 Exam Prep)

 
This post is a part of the AB-620: Designing and Building Integrated AI Agent Solutions in Copilot Studio Exam Prep Hub.
This topic falls under these sections:
Test and manage agents (20–25%)
   --> Evaluate agent performance
      --> Create a test set

Note that there are 10 practice questions (with answers) at the end of each section to help you solidify your knowledge of the material. Also, there are 4 practice tests with 30 questions each available from the hub's main page below the exam topics section.

Introduction

One of the most important responsibilities of an AI Agent Builder is ensuring that an agent consistently produces accurate, relevant, and safe responses. As agents become more sophisticated and rely on multiple knowledge sources, tools, and generative AI models, manual testing alone is no longer sufficient.

Microsoft Copilot Studio provides test sets that allow developers to systematically validate agent behavior against expected outcomes. Test sets enable repeatable evaluation of an agent after configuration changes, prompt updates, knowledge source modifications, or model upgrades.

For the AB-620 exam, you should understand:

  • What test sets are
  • Why they are important
  • How to create and manage them
  • How they integrate with agent evaluation
  • Best practices for maintaining reliable test coverage

What Is a Test Set?

A test set is a collection of predefined test cases that evaluate how an AI agent responds to expected user requests.

Each test case generally contains:

  • A sample user prompt
  • The expected behavior or outcome
  • Evaluation criteria
  • Pass/fail results after execution

Instead of manually asking the same questions every time changes are made, developers can rerun the entire test set to determine whether the agent continues to behave correctly.

Why Test Sets Matter

Without structured testing:

  • New prompts may unintentionally break previous functionality.
  • Updated knowledge sources may introduce incorrect answers.
  • Tool changes may fail silently.
  • Model updates may alter response quality.

Test sets provide confidence that the agent still behaves correctly after changes.

Benefits include:

  • Repeatable testing
  • Faster validation
  • Regression testing
  • Improved response quality
  • Easier troubleshooting
  • Better release confidence

Test Set vs Manual Testing

Manual TestingTest Set
Performed interactivelyExecuted repeatedly
Difficult to reproduceFully repeatable
Human remembers questionsQuestions stored permanently
Time consumingAutomated evaluation
Easy to miss scenariosCovers many scenarios consistently

When Should You Create a Test Set?

Create a test set whenever:

  • Building a new agent
  • Adding new topics
  • Adding knowledge sources
  • Adding tools
  • Integrating APIs
  • Updating prompts
  • Deploying a new version
  • Performing regression testing

Components of a Test Case

A typical test case includes several important elements.

1. User Input

The question or request submitted to the agent.

Example:

“Show me my remaining vacation balance.”

2. Expected Behavior

The desired outcome.

Examples include:

  • Calls HR connector
  • Retrieves employee record
  • Returns vacation balance
  • Does not hallucinate data

3. Expected Response

Depending on the evaluation method, expected responses may include:

  • Specific wording
  • Required information
  • Correct tool usage
  • Accurate citation
  • Proper formatting

4. Evaluation Result

After execution the test produces results such as:

  • Pass
  • Fail
  • Partial success
  • Confidence score (where applicable)

Types of Test Cases

A comprehensive test set should include multiple categories.

Happy Path Tests

Expected user behavior.

Example:

“Reset my password.”

Alternative Wording

Different ways users ask the same question.

Examples:

  • I forgot my password
  • Help me log in
  • I can’t sign in

Edge Cases

Unusual but valid requests.

Example:

“Can I reset someone else’s password?”

Invalid Requests

Questions the agent should decline.

Example:

“Delete every employee record.”

Ambiguous Questions

The agent should ask follow-up questions.

Example:

“Book a meeting.”

Expected behavior:

“Who should I invite?”

Tool Failure Tests

Verify graceful handling of failures.

Example:

API unavailable.

Expected response:

“The HR system is temporarily unavailable.”

Knowledge Tests

Ensure retrieval from enterprise knowledge.

Example:

“What is the travel reimbursement policy?”

Security Tests

Confirm proper authorization.

Example:

Employee requests another employee’s payroll information.

Expected behavior:

Access denied.

Creating a Test Set

The general workflow is:

Step 1

Open the agent in Copilot Studio.

Step 2

Navigate to testing or evaluation features.

Step 3

Create a new test set.

Step 4

Add individual test cases.

Each includes:

  • Prompt
  • Expected behavior
  • Expected response

Step 5

Save the test set.

Step 6

Run the evaluation.

Step 7

Review results.

Step 8

Improve the agent if failures occur.

Step 9

Run the test set again.

Organizing Test Sets

Large enterprise agents often use multiple test sets.

Examples:

  • HR Agent
  • Finance Agent
  • IT Help Desk
  • Customer Service
  • Sales Support

Within each, additional test groups may cover:

  • Authentication
  • Knowledge retrieval
  • API actions
  • Escalation
  • Security
  • Generative responses

Regression Testing

Regression testing verifies that new changes do not introduce unexpected problems.

Example:

Original agent answers:

“How do I request PTO?”

A new HR connector is added.

Running the existing test set confirms the answer still works correctly.

Without regression testing, developers may unknowingly introduce defects.

Testing Knowledge Retrieval

Knowledge-based agents should verify:

  • Correct document selected
  • Correct section retrieved
  • Accurate citation
  • Relevant answer
  • No hallucinated content

Example test:

Question:

“What is the expense reimbursement limit?”

Expected:

  • Searches indexed documents
  • Retrieves finance policy
  • Returns correct limit
  • Includes citation if configured

Testing Tool Invocation

For action-based agents, verify that the correct tool is selected.

Example:

User:

“Create a support ticket.”

Expected:

  • IT connector invoked
  • Ticket created
  • Ticket number returned

Failure examples:

  • Wrong connector called
  • No connector called
  • Hallucinated confirmation

Testing Multi-Agent Solutions

If delegation is used, verify:

  • Correct child agent selected
  • Successful delegation
  • Response returned
  • Parent continues conversation properly

Testing Generative AI

Generative responses require additional evaluation.

Verify:

  • Factual accuracy
  • Completeness
  • Grounding
  • Tone
  • Safety
  • Relevance

Evaluating Test Results

After execution, review:

  • Overall pass rate
  • Failed cases
  • Tool execution
  • Knowledge retrieval
  • Response quality
  • Latency
  • Error messages

Common questions include:

  • Did the correct tool run?
  • Was the answer accurate?
  • Was sensitive data protected?
  • Was grounding successful?

Common Reasons Tests Fail

Failures often result from:

  • Prompt changes
  • Missing connector permissions
  • API failures
  • Incorrect tool selection
  • Poor grounding
  • Hallucinations
  • Missing documents
  • Authentication problems
  • Incorrect routing

Best Practices

Microsoft recommends several best practices.

Build Early

Create test cases while building the agent.

Cover Real User Questions

Use production-like prompts whenever possible.

Include Variations

People ask the same question differently.

Test all common variations.

Test Negative Scenarios

Don’t only verify success.

Test:

  • Errors
  • Permission failures
  • Invalid input
  • Ambiguous requests

Keep Test Sets Updated

Whenever the agent changes:

  • Add new tests
  • Remove obsolete tests
  • Update expected responses

Run Tests Frequently

Execute the full test set:

  • Before deployment
  • After model updates
  • After connector updates
  • After knowledge updates
  • After prompt revisions

Exam Tips

For the AB-620 exam, remember:

  • Test sets enable repeatable evaluation.
  • They support regression testing.
  • Good test cases include expected behavior.
  • Test sets should include positive, negative, and edge-case scenarios.
  • Multi-agent solutions require delegation testing.
  • Tool-based agents require tool invocation validation.
  • Knowledge agents require grounding verification.
  • Test sets improve deployment confidence.

Practice Exam Questions

Question 1

Why is creating a test set preferable to relying solely on manual testing?

A. It permanently stores conversation history for users.

B. It provides repeatable, consistent evaluation of agent behavior.

C. It automatically retrains the language model.

D. It removes the need for production monitoring.

Answer: B

Explanation: Test sets allow the same scenarios to be executed repeatedly, making regression testing and validation much more reliable than manual testing.

Question 2

Which type of scenario should always be included in a comprehensive test set?

A. Only successful user interactions

B. Only connector failures

C. Positive, negative, and edge-case scenarios

D. Only knowledge retrieval questions

Answer: C

Explanation: Comprehensive testing includes normal requests, invalid inputs, ambiguous questions, security scenarios, and failure conditions.

Question 3

A developer updates an HR connector used by an agent. What is the best next step?

A. Run the existing test set to perform regression testing.

B. Delete all previous test cases.

C. Retrain the foundation model.

D. Create a new environment.

Answer: A

Explanation: Regression testing verifies that previously working functionality continues to operate after changes.

Question 4

Which component defines what a successful test should accomplish?

A. Conversation history

B. Agent version

C. Workspace settings

D. Expected behavior

Answer: D

Explanation: Expected behavior specifies the desired outcome that the agent should achieve during the test.

Question 5

A knowledge-based agent answers a company policy question using outdated information. Which area of testing should identify this issue?

A. User authentication testing

B. Knowledge retrieval testing

C. Network latency testing

D. Adaptive Card rendering

Answer: B

Explanation: Knowledge retrieval tests verify that the correct documents are located and that accurate, grounded information is returned.

Question 6

When testing an action that creates a support ticket, what should the evaluation confirm?

A. Only that the response is grammatically correct

B. Only that the response is polite

C. That the correct tool or connector was invoked successfully

D. That the conversation contains at least three turns

Answer: C

Explanation: Action-based tests should verify successful tool invocation and the expected outcome of that action.

Question 7

Why should multiple phrasings of the same request be included in a test set?

A. To increase the size of the knowledge base

B. To improve authentication

C. To ensure the agent recognizes natural language variations

D. To reduce connector latency

Answer: C

Explanation: Users ask the same question in many different ways, and the agent should respond correctly to common variations.

Question 8

Which situation best represents an edge-case test?

A. “Reset my password.”

B. “Show today’s weather.”

C. “Create a support ticket.”

D. “Can I reset another employee’s password?”

Answer: D

Explanation: This unusual but valid request tests whether the agent correctly handles authorization and security.

Question 9

An agent delegates requests to multiple child agents. What should testing verify?

A. That delegation occurs to the appropriate child agent and responses are returned correctly

B. That every child agent uses the same prompt

C. That delegation is disabled after deployment

D. That all child agents share one knowledge source

Answer: A

Explanation: Multi-agent testing ensures that routing, delegation, and response aggregation function as designed.

Question 10

Which statement best describes the primary purpose of regression testing?

A. Measuring internet bandwidth

B. Evaluating user satisfaction surveys

C. Ensuring that recent changes have not broken existing functionality

D. Generating additional knowledge documents

Answer: C

Explanation: Regression testing validates that existing capabilities continue to work correctly after updates to prompts, connectors, tools, or knowledge sources.

Go to the AB-620 Exam Prep Hub main page