AB-620, Agentic AI, AI, Azure AI, Generative AI, Microsoft Certification

Monitor agents by using Application Insights (AB-620 Exam Prep)

 
This post is a part of the AB-620: Designing and Building Integrated AI Agent Solutions in Copilot Studio Exam Prep Hub.
This topic falls under these sections:
Integrate and extend agents in Copilot Studio (40–45%)
   --> Integrate agents with Azure
      --> Monitor agents by using Application Insights

Note that there are 10 practice questions (with answers) at the end of each section to help you solidify your knowledge of the material. Also, there are 4 practice tests with 30 questions each available from the hub's main page below the exam topics section.

Introduction

As AI agents become more sophisticated and business-critical, monitoring their health, performance, reliability, and user interactions becomes essential. An AI agent that responds slowly, generates errors, experiences high failure rates, or consumes excessive resources can negatively impact business operations and user satisfaction.

Microsoft Copilot Studio integrates with Azure Application Insights, a feature of Azure Monitor, to provide comprehensive telemetry, diagnostics, and performance monitoring. Application Insights collects operational data from agents, allowing administrators and developers to observe agent behavior, troubleshoot issues, measure usage, and optimize performance over time.

For the AB-620 exam, you should understand how Application Insights integrates with Copilot Studio, what telemetry it collects, how to analyze monitoring data, and how monitoring supports production AI solutions.

What is Azure Application Insights?

Azure Application Insights is an application performance monitoring (APM) service within Azure Monitor.

It helps organizations:

  • Monitor application availability
  • Track performance
  • Diagnose failures
  • Analyze user behavior
  • Detect anomalies
  • Monitor dependencies
  • Measure response times
  • Identify bottlenecks
  • Generate alerts
  • Improve application reliability

Application Insights provides near real-time visibility into the operational health of applications, including AI-powered agents.

Why Monitor Copilot Studio Agents?

Production AI agents interact with users continuously. Monitoring helps answer questions such as:

  • Is the agent available?
  • Are conversations completing successfully?
  • Are responses taking too long?
  • Are external APIs failing?
  • Which topics are most frequently triggered?
  • Where are users abandoning conversations?
  • Are authentication failures occurring?
  • Are knowledge searches succeeding?
  • Is latency increasing?
  • Are recent deployments causing problems?

Without monitoring, identifying these issues can be difficult.

Monitoring Architecture

A typical monitoring architecture includes:

User
↓
Copilot Studio Agent
↓
Conversation Execution
↓
Telemetry Collection
↓
Application Insights
↓
Azure Monitor
↓
Dashboards
Alerts
Analytics
Logs

Every conversation can generate telemetry that is stored for analysis.

What is Telemetry?

Telemetry is operational data automatically collected from applications.

For Copilot Studio agents, telemetry may include:

  • Conversation start
  • Conversation end
  • User session
  • Topic activation
  • Tool execution
  • API calls
  • Response times
  • Exceptions
  • Authentication events
  • Dependency calls
  • Custom events
  • Prompt execution
  • Generative AI activity
  • User feedback
  • Errors

Telemetry provides the raw information used to monitor system health.

Types of Telemetry

Application Insights collects several categories of telemetry.

Requests

Measures requests processed by the agent.

Examples include:

  • User messages
  • Conversation requests
  • HTTP requests
  • API invocations

Useful metrics include:

  • Duration
  • Success rate
  • Failure rate

Dependencies

Tracks external services called by the agent.

Examples include:

  • REST APIs
  • Azure AI Search
  • Dataverse
  • SQL Database
  • SharePoint
  • Power Platform connectors
  • Azure OpenAI
  • Azure AI Foundry
  • External web services

Dependency tracking helps identify slow or failing external systems.

Exceptions

Captures unexpected errors.

Examples include:

  • Authentication failures
  • Timeout exceptions
  • API failures
  • Missing parameters
  • Invalid requests
  • Permission errors

Developers can use exception details to troubleshoot failures.

Traces

Trace telemetry records detailed execution information.

Examples include:

  • Topic execution
  • Diagnostic messages
  • Workflow progress
  • Variable values
  • Decision branches

Traces are especially useful during debugging.

Events

Custom events capture important business activities.

Examples:

  • Order submitted
  • Employee onboarded
  • Ticket created
  • Payment completed
  • Appointment scheduled

Organizations can define custom events for business-specific monitoring.

Availability

Availability monitoring tests whether an application is reachable.

It can detect:

  • Service outages
  • Connectivity failures
  • Regional problems
  • Downtime

Availability tests help ensure production agents remain accessible.

Metrics Commonly Monitored

Common operational metrics include:

  • Total conversations
  • Active users
  • Average response time
  • Request duration
  • API latency
  • Conversation completion rate
  • Conversation abandonment
  • Error count
  • Exception rate
  • Failed requests
  • CPU utilization (supporting resources)
  • Memory utilization (supporting resources)
  • Dependency performance
  • Token consumption (when available)
  • Cost trends

Integrating Copilot Studio with Application Insights

High-level integration typically includes:

  1. Create an Azure Application Insights resource.
  2. Enable monitoring.
  3. Connect the Copilot Studio environment.
  4. Configure telemetry collection.
  5. Deploy the agent.
  6. Review incoming telemetry.
  7. Build dashboards.
  8. Configure alerts.
  9. Monitor production activity.

Azure Monitor Integration

Application Insights is part of Azure Monitor.

Azure Monitor provides:

  • Centralized monitoring
  • Metrics
  • Log Analytics
  • Alerts
  • Dashboards
  • Workbooks
  • Automation
  • Diagnostic settings

Application Insights contributes telemetry to Azure Monitor, where it can be analyzed alongside other Azure resources.

Log Analytics

Telemetry is stored in Log Analytics, enabling powerful querying using Kusto Query Language (KQL).

Administrators can answer questions such as:

  • Which conversations failed today?
  • Which topics generate the most errors?
  • Which users experience timeouts?
  • What APIs are the slowest?
  • Which connector has the highest latency?
  • How many conversations exceeded five seconds?

Example Monitoring Scenarios

Scenario 1

Users report slow responses.

Application Insights reveals:

  • Average response time increased from 2 seconds to 12 seconds.
  • Azure AI Search dependency latency increased dramatically.

The administrator investigates the search service.

Scenario 2

A new deployment causes failures.

Monitoring identifies:

  • Spike in exceptions.
  • Failed API calls.
  • Authentication errors.

The deployment is rolled back.

Scenario 3

An external REST API becomes unavailable.

Application Insights shows:

  • Dependency failures
  • Timeout exceptions
  • Increased conversation failures

Administrators quickly identify the external dependency rather than blaming Copilot Studio.

Dashboards

Application Insights dashboards visualize operational health.

Typical dashboard components include:

  • Conversation volume
  • Requests per minute
  • Active users
  • Success rate
  • Failure rate
  • Exceptions
  • Response times
  • API latency
  • Dependency health
  • Geographic usage
  • Availability
  • Performance trends

Dashboards allow administrators to monitor systems without manually querying logs.

Alerts

Alerts automatically notify administrators when thresholds are exceeded.

Examples include:

  • Response time exceeds five seconds.
  • Error rate exceeds 3%.
  • Availability drops below 99%.
  • API failures increase suddenly.
  • Authentication failures spike.
  • Conversation completion rate decreases.

Alerts can trigger:

  • Email
  • SMS
  • Microsoft Teams notifications
  • Azure Automation
  • Logic Apps
  • Webhooks

Distributed Tracing

Many enterprise agents call multiple services during a single conversation.

Example:

User
↓
Copilot Studio
↓
Azure AI Search
↓
REST API
↓
Dataverse
↓
Azure AI Foundry
↓
Response

Application Insights correlates these operations into a single end-to-end transaction.

This allows administrators to identify exactly where delays occur.

Correlation IDs

Each conversation can be assigned a correlation ID.

This enables:

  • End-to-end tracing
  • Cross-service diagnostics
  • Root cause analysis
  • Log correlation
  • Easier troubleshooting

Correlation IDs are especially valuable in distributed AI systems.

Monitoring Generative AI Operations

Application Insights can help monitor:

  • Prompt execution
  • Model latency
  • API failures
  • Retrieval operations
  • Tool execution
  • Conversation completion
  • Dependency failures
  • User feedback events

While model-specific metrics may come from Azure AI Foundry or Azure OpenAI, Application Insights provides operational telemetry surrounding those interactions.

Security Considerations

Monitoring should avoid collecting sensitive information.

Best practices include:

  • Avoid storing secrets.
  • Minimize personal information.
  • Mask sensitive values.
  • Follow organizational compliance policies.
  • Apply RBAC to monitoring resources.
  • Encrypt telemetry in transit and at rest.
  • Retain logs according to governance requirements.

Cost Considerations

Application Insights pricing depends largely on:

  • Data ingestion volume
  • Log retention
  • Query frequency
  • Exported telemetry

Organizations should balance monitoring detail with storage costs.

Strategies include:

  • Sample telemetry.
  • Adjust retention periods.
  • Remove unnecessary events.
  • Archive historical logs.

Best Practices

  • Enable monitoring before production deployment.
  • Create dashboards for key performance indicators.
  • Configure proactive alerts.
  • Monitor dependency health.
  • Use distributed tracing.
  • Track conversation completion rates.
  • Review exceptions regularly.
  • Use KQL to investigate issues.
  • Protect sensitive telemetry.
  • Continuously optimize based on monitoring insights.

Common Exam Tips

For the AB-620 exam, remember the following:

  • Application Insights is part of Azure Monitor.
  • It provides application performance monitoring (APM).
  • It collects telemetry from running applications.
  • Telemetry includes requests, dependencies, exceptions, traces, events, and availability data.
  • Dependency monitoring helps diagnose failures in external systems.
  • Log Analytics uses Kusto Query Language (KQL) for querying telemetry.
  • Alerts can automatically notify administrators of operational issues.
  • Distributed tracing correlates activity across multiple services.
  • Correlation IDs enable end-to-end diagnostics.
  • Monitoring supports performance optimization, troubleshooting, and operational reliability.

Practice Exam Questions

Question 1

An administrator wants to determine why users are experiencing slow responses from a Copilot Studio agent. Which Azure service provides detailed performance telemetry for troubleshooting?

A. Azure Storage

B. Azure Application Insights

C. Microsoft Entra ID

D. Azure Key Vault

Answer: B

Explanation: Application Insights collects detailed telemetry such as response times, dependency performance, and exceptions, making it the primary service for diagnosing performance issues.

Question 2

Which type of Application Insights telemetry tracks calls from a Copilot Studio agent to Azure AI Search or external REST APIs?

A. Requests

B. Exceptions

C. Dependencies

D. Availability

Answer: C

Explanation: Dependency telemetry measures calls to external services, databases, connectors, APIs, and Azure resources, allowing administrators to identify slow or failing dependencies.

Question 3

A developer wants to investigate authentication failures generated during agent execution. Which telemetry type should they examine first?

A. Exceptions

B. Availability

C. Metrics

D. Workbooks

Answer: A

Explanation: Authentication failures typically generate exception telemetry, which records detailed information about errors encountered during execution.

Question 4

What is the primary purpose of distributed tracing in Application Insights?

A. Encrypt conversation history

B. Automatically translate telemetry

C. Compress monitoring data

D. Correlate activity across multiple services in a single transaction

Answer: D

Explanation: Distributed tracing connects telemetry from multiple services involved in processing a single request, enabling end-to-end diagnostics.

Question 5

Which language is used to query Application Insights data stored in Log Analytics?

A. T-SQL

B. Power Query M

C. DAX

D. Kusto Query Language (KQL)

Answer: D

Explanation: Log Analytics uses Kusto Query Language (KQL) to query, filter, summarize, and analyze telemetry data.

Question 6

An operations team wants to receive an email whenever an agent’s average response time exceeds five seconds. Which Azure Monitor capability should they configure?

A. Alerts

B. Availability tests

C. Workbooks

D. Sampling

Answer: A

Explanation: Azure Monitor alerts automatically notify administrators when configured thresholds or conditions are met.

Question 7

Which monitoring metric would BEST help determine whether users are abandoning conversations before completion?

A. CPU utilization

B. Conversation completion and abandonment rates

C. Azure subscription quota

D. Virtual machine availability

Answer: B

Explanation: Completion and abandonment metrics directly measure how successfully users finish conversations with the agent.

Question 8

Why are correlation IDs valuable when troubleshooting AI agents?

A. They reduce Azure costs.

B. They increase model accuracy.

C. They link telemetry across multiple services for a single conversation.

D. They automatically encrypt logs.

Answer: C

Explanation: Correlation IDs associate related telemetry from different services, making it easier to trace a request from start to finish.

Question 9

Which best practice helps protect sensitive information when using Application Insights?

A. Store authentication secrets in telemetry for debugging.

B. Collect every possible user input permanently.

C. Disable encryption to improve performance.

D. Mask sensitive data and apply role-based access control (RBAC).

Answer: D

Explanation: Sensitive information should be masked or excluded from telemetry, and access should be restricted using RBAC to support security and compliance.

Question 10

What is the primary benefit of monitoring external dependencies such as Azure AI Search, Dataverse, and REST APIs?

A. It automatically upgrades connectors.

B. It identifies latency and failures occurring outside the Copilot Studio agent itself.

C. It eliminates the need for application logging.

D. It reduces token consumption by language models.

Answer: B

Explanation: Dependency monitoring helps determine whether performance issues or failures originate in external services rather than within the agent, significantly speeding up root cause analysis.

Go to the AB-620 Exam Prep Hub main page