AB-620, Agentic AI, AI, Azure AI, Microsoft Certification

Choose an evaluation method (AB-620 Exam Prep)

 
This post is a part of the AB-620: Designing and Building Integrated AI Agent Solutions in Copilot Studio Exam Prep Hub.
This topic falls under these sections:
Test and manage agents (20–25%)
   --> Evaluate agent performance
      --> Choose an evaluation method

Note that there are 10 practice questions (with answers) at the end of each section to help you solidify your knowledge of the material. Also, there are 4 practice tests with 30 questions each available from the hub's main page below the exam topics section.

Introduction

Building an AI agent is only the first step in delivering a successful solution. An equally important responsibility is evaluating whether the agent performs as intended. Microsoft Copilot Studio includes evaluation capabilities that help developers assess the quality, accuracy, safety, and effectiveness of AI-generated responses before an agent is deployed to production.

Selecting the appropriate evaluation method depends on several factors, including:

  • The purpose of the agent
  • Whether the agent is knowledge-based or action-based
  • Whether responses are deterministic or generative
  • The organization’s quality requirements
  • The level of automation desired

For the AB-620 exam, you should understand:

  • Available evaluation methods
  • When to use each method
  • What each method measures
  • How evaluations improve agent quality
  • Best practices for evaluating AI agents

Why Evaluation Is Important

Generative AI systems are probabilistic rather than deterministic. Unlike traditional software that always produces identical output for identical input, AI-generated responses may vary slightly while still being correct.

Evaluation helps determine whether responses are:

  • Accurate
  • Relevant
  • Grounded
  • Complete
  • Safe
  • Helpful
  • Consistent

Without evaluation, organizations risk deploying agents that:

  • Hallucinate facts
  • Provide incomplete answers
  • Use incorrect tools
  • Return outdated information
  • Violate organizational policies

Goals of Agent Evaluation

Evaluation should answer questions such as:

  • Did the agent answer correctly?
  • Was the correct knowledge source used?
  • Was the response grounded?
  • Was the appropriate tool invoked?
  • Was sensitive information protected?
  • Was the response relevant?
  • Did the conversation remain on topic?
  • Did the agent accomplish the user’s goal?

Types of Evaluation Methods

Microsoft Copilot Studio supports multiple evaluation approaches.

The primary categories include:

  • Manual evaluation
  • Automated evaluation
  • AI-assisted evaluation
  • Test set evaluation
  • Human review
  • Continuous monitoring

Each serves a different purpose.

Manual Evaluation

Manual evaluation involves developers or business users interacting directly with the agent.

Typical process:

  1. Ask questions.
  2. Review responses.
  3. Identify problems.
  4. Improve prompts or tools.
  5. Repeat testing.

Advantages

  • Simple
  • Fast for small projects
  • Easy to understand
  • Good during development

Limitations

  • Difficult to scale
  • Subjective
  • Time consuming
  • Not repeatable

Automated Evaluation

Automated evaluation uses predefined test cases to measure agent performance.

Examples include:

  • Running test sets
  • Validating expected responses
  • Measuring pass/fail rates
  • Comparing versions

Benefits include:

  • Repeatability
  • Consistency
  • Speed
  • Regression testing

AI-Assisted Evaluation

AI models can help assess the quality of responses.

Instead of only comparing exact wording, AI can evaluate:

  • Semantic correctness
  • Relevance
  • Helpfulness
  • Completeness
  • Faithfulness to source material

For example:

User asks:

“How do I reset my password?”

The expected response might vary in wording while still being completely correct.

AI-assisted evaluation recognizes that multiple valid responses may exist.

Human Evaluation

Human reviewers examine conversations and determine whether responses meet organizational expectations.

Human reviewers may assess:

  • Tone
  • Accuracy
  • Professionalism
  • Policy compliance
  • User satisfaction

Human evaluation is especially valuable for:

  • Customer service
  • Healthcare
  • Legal
  • Financial services

Test Set Evaluation

A test set contains predefined prompts with expected outcomes.

Running a test set provides:

  • Pass/fail results
  • Quality metrics
  • Regression detection
  • Coverage across scenarios

Test sets are recommended before production deployments.

Continuous Evaluation

Evaluation should continue after deployment.

Production monitoring identifies:

  • New failure patterns
  • Frequently unanswered questions
  • Knowledge gaps
  • Tool failures
  • User frustration

Continuous evaluation supports ongoing improvement.

Evaluation Criteria

Several quality dimensions are commonly evaluated.

1. Correctness

Does the response answer the question accurately?

Example:

User:

“How many vacation days do I have?”

Correct response:

Returns the actual balance from HR.

Incorrect response:

Invents a number.

2. Relevance

Is the response related to the user’s request?

Poor relevance often results from:

  • Incorrect knowledge retrieval
  • Poor prompting
  • Wrong tool selection

3. Groundedness

Groundedness measures whether responses are supported by trusted enterprise data.

Grounded responses:

  • Reference indexed documents
  • Use Azure AI Search
  • Avoid unsupported claims

Ungrounded responses may hallucinate.

4. Completeness

Does the response fully answer the user’s question?

Poor example:

User:

“How do I submit travel expenses?”

Response:

“Use the expense portal.”

Better response:

  • Portal name
  • Required documents
  • Approval workflow
  • Submission deadline

5. Safety

Safety evaluations identify:

  • Harmful content
  • Sensitive information exposure
  • Offensive language
  • Policy violations

Safety is essential for enterprise deployments.

6. Tool Accuracy

If the agent invokes external tools, verify:

  • Correct tool selected
  • Correct parameters supplied
  • Successful execution
  • Correct result returned

7. Conversation Quality

Evaluate whether the conversation flows naturally.

Examples include:

  • Appropriate follow-up questions
  • Context awareness
  • Smooth transitions
  • Helpful clarification requests

Selecting an Evaluation Method

Different scenarios require different evaluation methods.

ScenarioRecommended Evaluation
New prototypeManual testing
Regression testingAutomated test sets
Knowledge retrievalGroundedness evaluation
API actionsTool execution validation
Customer serviceHuman + automated evaluation
Production agentContinuous monitoring
Multi-agent orchestrationDelegation and routing evaluation

Evaluating Knowledge-Based Agents

Knowledge agents should be evaluated for:

  • Correct document retrieval
  • Citation quality
  • Freshness of information
  • Hallucination prevention
  • Accurate summaries

Typical questions include:

  • Did Azure AI Search retrieve the correct content?
  • Was the answer grounded?
  • Was outdated content used?

Evaluating Action-Based Agents

Agents that execute business processes require additional evaluation.

Verify:

  • Tool selection
  • Authentication
  • API success
  • Parameter accuracy
  • Business outcome

Example:

User:

“Create an IT ticket.”

Evaluation checks:

  • Was the ticket created?
  • Was the correct connector called?
  • Was the correct priority assigned?

Evaluating Multi-Agent Solutions

For multi-agent solutions, assess:

  • Proper routing
  • Correct child-agent selection
  • Delegation accuracy
  • Context preservation
  • Final response quality

Failures may occur if:

  • Wrong agent receives the request
  • Delegation loops occur
  • Context is lost between agents

Evaluating Generative Answers

Generative AI introduces additional evaluation dimensions.

Evaluate:

  • Hallucination rate
  • Factual accuracy
  • Grounding quality
  • Readability
  • Tone
  • Completeness
  • Citation quality
  • Confidence

Metrics Used During Evaluation

Organizations often monitor:

  • Pass rate
  • Failure rate
  • Response accuracy
  • Latency
  • Tool success rate
  • Grounding score
  • Hallucination frequency
  • User satisfaction
  • Resolution rate
  • Escalation frequency

Common Evaluation Mistakes

Avoid these common mistakes:

  • Testing only happy-path scenarios
  • Ignoring edge cases
  • Measuring wording instead of meaning
  • Forgetting regression testing
  • Not testing tool failures
  • Ignoring production feedback
  • Using outdated test cases
  • Evaluating only accuracy while ignoring safety

Best Practices

Use Multiple Evaluation Methods

Combine:

  • Manual review
  • Automated testing
  • AI-assisted evaluation
  • Human review

No single method is sufficient for all scenarios.

Create Realistic Test Cases

Use prompts based on actual user behavior instead of artificial examples.

Evaluate Regularly

Run evaluations:

  • Before deployment
  • After prompt changes
  • After connector updates
  • After knowledge updates
  • After model upgrades

Monitor Production

Evaluation should continue after deployment using telemetry, analytics, and user feedback.

Improve Continuously

Use evaluation results to:

  • Refine prompts
  • Improve knowledge sources
  • Fix tools
  • Expand test sets
  • Enhance user experience

Exam Tips

For the AB-620 exam, remember:

  • Different evaluation methods serve different purposes.
  • Automated evaluations support regression testing.
  • AI-assisted evaluations assess semantic quality rather than exact wording.
  • Groundedness is essential for knowledge-based agents.
  • Tool accuracy is critical for action-based agents.
  • Human review remains important for high-risk business scenarios.
  • Evaluation is an ongoing lifecycle activity, not a one-time task.
  • Combining multiple evaluation methods produces the most reliable assessment.

Practice Exam Questions

Question 1

A development team wants to verify that recent prompt changes have not broken existing functionality. Which evaluation method is most appropriate?

A. Automated test set evaluation

B. User satisfaction surveys

C. Manual exploratory testing only

D. Random production monitoring

Answer: A

Explanation: Automated test sets provide repeatable regression testing, allowing teams to verify that previously working scenarios continue to function after changes.

Question 2

Which evaluation criterion determines whether an agent’s response is supported by trusted enterprise data rather than generated from unsupported assumptions?

A. Latency

B. Groundedness

C. Conversation length

D. User engagement

Answer: B

Explanation: Groundedness measures whether responses are based on authoritative data sources, helping reduce hallucinations.

Question 3

A customer service manager wants to assess whether responses are polite, professional, and aligned with company communication standards. Which evaluation method is most appropriate?

A. Automated pass/fail testing

B. API performance testing

C. Human evaluation

D. Network diagnostics

Answer: C

Explanation: Human reviewers are best suited to evaluating tone, professionalism, empathy, and adherence to organizational communication standards.

Question 4

Why is AI-assisted evaluation useful for generative AI responses?

A. It requires every correct answer to match expected wording exactly.

B. It automatically retrains the language model.

C. It eliminates the need for human reviewers.

D. It evaluates semantic correctness even when responses are worded differently.

Answer: D

Explanation: AI-assisted evaluation focuses on meaning and correctness rather than exact text matches, making it well suited for generative responses.

Question 5

Which evaluation criterion confirms that an agent selected the correct connector and completed a requested business action?

A. Tool accuracy

B. Conversation length

C. Groundedness

D. Response formatting

Answer: A

Explanation: Tool accuracy verifies that the appropriate tool was invoked with the correct parameters and that the desired action was completed successfully.

Question 6

Which type of evaluation should continue after an agent is deployed to production?

A. Prototype evaluation only

B. Continuous monitoring and evaluation

C. Initial prompt validation only

D. Installation verification

Answer: B

Explanation: Production monitoring helps identify new issues, emerging user needs, and opportunities for continuous improvement.

Question 7

A developer wants to verify that a knowledge-based agent retrieved the correct document and provided an accurate citation. Which area is being evaluated?

A. Authentication

B. Delegation

C. Knowledge retrieval and groundedness

D. UI rendering

Answer: C

Explanation: Knowledge retrieval evaluations determine whether the correct source was used and whether responses remain grounded in trusted content.

Question 8

What is the primary advantage of automated evaluation compared to manual testing?

A. It permanently stores every user conversation.

B. It guarantees zero hallucinations.

C. It automatically writes new prompts.

D. It provides repeatable, consistent testing across multiple runs.

Answer: D

Explanation: Automated evaluation enables consistent execution of predefined tests, making regression testing reliable and scalable.

Question 9

Which combination provides the most comprehensive assessment of an enterprise AI agent?

A. Manual testing only

B. Human evaluation only

C. Automated testing only

D. A combination of manual, automated, AI-assisted, and human evaluation

Answer: D

Explanation: Each evaluation method measures different aspects of agent quality. Combining them provides the most complete assessment.

Question 10

An evaluation determines that an agent answered the user’s question correctly but omitted several important procedural steps. Which quality criterion needs improvement?

A. Safety

B. Completeness

C. Authentication

D. Latency

Answer: B

Explanation: Completeness measures whether the response fully addresses the user’s request with sufficient detail and context.

Go to the AB-620 Exam Prep Hub main page