This post is a part of the AB-620: Designing and Building Integrated AI Agent Solutions in Copilot Studio Exam Prep Hub.
This topic falls under these sections:
Integrate and extend agents in Copilot Studio (40–45%)
   --> Add tools to agents
      --> Configure and monitor computer use for an agent

Note that there are 10 practice questions (with answers) at the end of each section to help you solidify your knowledge of the material. Also, there are 4 practice tests with 30 questions each available from the hub's main page below the exam topics section.

Introduction

Many organizations still rely on legacy applications that do not expose REST APIs, Microsoft Power Platform connectors, or Model Context Protocol (MCP) servers. Employees may need to interact with desktop applications, web portals, or line-of-business systems that require clicking buttons, typing into forms, navigating menus, and downloading files.

Computer Use enables AI agents to perform these user interface (UI) interactions by observing and manipulating an application’s graphical interface, much like a human user would.

Rather than integrating through APIs, the agent interacts directly with the application’s user interface.

This capability expands the types of business processes that Copilot Studio agents can automate.

What is Computer Use?

Computer Use is an AI capability that allows an agent to:

• Observe the user interface
• Identify interface elements
• Move the mouse
• Click buttons
• Enter text
• Select menu options
• Scroll pages
• Navigate applications
• Execute repetitive workflows

Instead of calling an API, the agent completes tasks by interacting with the application’s visual interface.

Why Computer Use Exists

Many enterprise applications:

• have no API
• expose limited APIs
• use legacy technologies
• require manual interaction
• contain proprietary interfaces

Examples include:

• Legacy ERP systems
• Internal HR portals
• Desktop accounting software
• Government websites
• Vendor portals
• Older Windows applications

Computer Use provides automation where traditional integrations are unavailable or impractical.

Computer Use vs. API Integration

Exam Tip: Microsoft recommends using APIs, connectors, or MCP servers when available. Computer Use is typically used when no suitable programmatic interface exists.

Typical Computer Use Architecture

User Request
↓
Copilot Studio Agent
↓
Computer Use Tool
↓
AI analyzes screen
↓
Identifies UI elements
↓
Executes mouse/keyboard actions
↓
Application responds
↓
Agent verifies results
↓
Response returned to user

Common Business Scenarios

Computer Use is valuable in situations where employees currently perform repetitive manual tasks.

Invoice Processing

An agent can:

• Open an accounting application
• Enter invoice data
• Select suppliers
• Save records
• Confirm successful submission

Employee Onboarding

The agent can:

• Open HR software
• Create employee records
• Complete forms
• Assign departments
• Generate confirmation numbers

Customer Support

The agent may:

• Open a CRM system
• Search for customers
• Update account information
• Create service tickets
• Retrieve order history

Data Entry

Computer Use can automate:

• Copying information between systems
• Completing repetitive forms
• Updating spreadsheets
• Entering records into legacy databases

Web Portal Automation

Examples include:

• Vendor portals
• Government portals
• Insurance websites
• Banking systems
• Regulatory reporting portals

Computer Use Workflow

A typical execution follows these steps:

1. The user submits a request.
2. The agent determines that Computer Use is required.
3. The application launches (if necessary).
4. The AI observes the current screen.
5. UI elements are identified.
6. The agent performs actions.
7. The application responds.
8. The agent validates the result.
9. The workflow continues or finishes.
10. A response is returned to the user.

How the Agent Understands the Screen

Unlike API integrations, Computer Use relies on visual understanding.

The AI analyzes:

• Buttons
• Text boxes
• Menus
• Tables
• Checkboxes
• Drop-down lists
• Icons
• Dialog boxes
• Navigation controls

This allows it to interact with applications even when source code or APIs are unavailable.

Typical User Actions

A Computer Use agent may perform actions such as:

• Click
• Double-click
• Right-click
• Type text
• Press keyboard shortcuts
• Scroll
• Select menu items
• Drag objects
• Navigate windows
• Confirm dialog boxes
• Upload files
• Download files

Configuring Computer Use

Configuration generally involves:

• Enabling the Computer Use capability
• Selecting or configuring the target environment
• Defining the workflow
• Specifying execution permissions
• Testing interactions
• Publishing the agent

Administrators should verify that the environment meets all prerequisites before deployment.

Designing Reliable Automations

Because UI-based automation depends on visual elements, reliability is critical.

Good designs:

• Follow predictable navigation paths
• Minimize unnecessary clicks
• Use consistent workflows
• Verify intermediate results
• Handle unexpected dialogs
• Include recovery logic

Reliable automation reduces failures caused by interface changes.

Authentication Considerations

Many applications require authentication before automation can begin.

Possible authentication methods include:

• Microsoft Entra ID
• Organizational credentials
• Multi-factor authentication (where supported)
• Session-based authentication
• Single Sign-On (SSO)

Organizations should follow their security policies when storing or accessing credentials. Avoid embedding usernames, passwords, or secrets directly within agent logic.

Permissions

The agent should operate using the principle of least privilege.

Grant only the permissions necessary to complete the intended tasks.

Examples:

• Read-only access when updates are unnecessary
• Department-specific permissions
• Limited application roles
• Restricted administrative privileges

Limiting permissions reduces security risks.

Security Considerations

Computer Use interacts directly with enterprise applications, making security especially important.

Administrators should consider:

• Authentication
• Authorization
• Audit logging
• Data protection
• Session management
• Access reviews
• Conditional access policies
• Secure credential storage

Sensitive Data Handling

Computer Use workflows may encounter:

• Personally identifiable information (PII)
• Financial records
• Medical information
• Customer data
• Employee records

Organizations should:

• Follow compliance requirements
• Minimize unnecessary data exposure
• Log actions appropriately
• Restrict access to sensitive workflows
• Monitor privileged automations

Common Limitations

Computer Use is powerful but has limitations.

Examples include:

UI Changes

If a button moves or is renamed, automation may fail.

Dynamic Pages

Pages that change frequently can reduce reliability.

Pop-up Windows

Unexpected dialogs may interrupt execution.

Performance Delays

Slow applications may require waiting or retry logic.

Unsupported Controls

Some proprietary interface components may be difficult to automate consistently.

When NOT to Use Computer Use

Avoid Computer Use when:

• A REST API is available.
• A Microsoft Power Platform connector exists.
• An MCP server provides direct integration.
• A supported enterprise connector is available.
• A direct database integration is appropriate.

API-based integrations are generally more reliable, scalable, and maintainable than UI automation.

Best Practices

Prefer Native Integrations

Use:

• Connectors
• APIs
• MCP
• Power Automate

before choosing Computer Use.

Keep Workflows Simple

Smaller workflows are easier to maintain and troubleshoot.

Validate Each Step

Confirm that each action succeeds before proceeding.

Handle Unexpected Screens

Prepare for:

• Error messages
• Session timeouts
• Login prompts
• Confirmation dialogs

Use Stable Interfaces

Applications with consistent layouts produce more reliable automations.

Test Regularly

Retest automations after:

• Application upgrades
• UI redesigns
• Security updates
• Browser updates
• Operating system updates

Common Enterprise Use Cases

Organizations commonly use Computer Use for:

• HR onboarding
• Invoice entry
• Insurance claims
• CRM updates
• Legacy ERP automation
• Procurement workflows
• Compliance reporting
• Financial reconciliation
• Customer service operations
• Data migration between systems

Common Exam Mistakes

Candidates often assume that Computer Use is the preferred integration method.

Remember:

• Computer Use is not the first choice.
• APIs and connectors should be used whenever available.
• Computer Use fills the gap when direct integrations are unavailable.

Another common mistake is assuming Computer Use is immune to application changes. Because it relies on the user interface, modifications to screens, layouts, or controls can affect automation reliability.

AB-620 Exam Tips

Remember these key points:

• Computer Use automates interactions through an application’s graphical interface.
• It is intended primarily for systems without suitable APIs or connectors.
• UI automation is generally more fragile than API-based integrations.
• Secure authentication and least-privilege access are essential.
• Validate each interaction to improve reliability.
• Design workflows to tolerate delays and unexpected dialogs.
• Monitor and maintain automations as application interfaces evolve.

Quick Orientation Summary

In the topics above, we explored the fundamentals of Computer Use in Microsoft Copilot Studio, including its purpose, architecture, configuration process, execution model, and how it differs from API-based automation. The topics below focus on monitoring, governance, security, optimization, troubleshooting.

Monitoring Computer Use Sessions

Unlike API tools, Computer Use performs visual interactions with applications. Because of this, monitoring becomes especially important.

Administrators should monitor:

• Session success rates
• Failed execution steps
• Time required to complete tasks
• Screen recognition failures
• Authentication failures
• Unexpected application behavior
• Agent execution history
• Resource consumption
• Retry frequency

Monitoring enables organizations to:

• Detect broken workflows
• Identify application UI changes
• Improve reliability
• Measure automation performance
• Support compliance audits

Execution Logs

Each Computer Use execution produces detailed logs.

Typical information includes:

• Workflow start time
• Workflow completion time
• Individual action history
• Screens visited
• Click locations
• Typed text
• Variables used
• Error messages
• Retry attempts
• Completion status

These logs assist with:

• Troubleshooting
• Performance tuning
• Security investigations
• Compliance reporting

Screenshots and Visual Evidence

Many implementations capture screenshots throughout execution.

Screenshots help identify:

• Missing buttons
• Incorrect pages
• Unexpected pop-ups
• Login failures
• Permission issues
• Validation errors
• UI redesigns

Visual evidence greatly reduces troubleshooting time.

Performance Metrics

Useful metrics include:

Success Rate

Percentage of successful executions.

Example:

• 98 successful runs
• 2 failed runs

Success rate:

98%

Average Completion Time

Tracks workflow efficiency.

Example:

• Average runtime: 22 seconds

If runtime suddenly increases:

• Network latency
• Slow applications
• UI delays
• Infrastructure issues

may be responsible.

Retry Frequency

Measures how often automation must repeat actions.

High retry counts often indicate:

• Unstable interfaces
• Slow page loading
• Timing problems
• UI recognition issues

Failure Categories

Failures should be categorized.

Examples include:

• Authentication failures
• Missing elements
• Timeout errors
• Permission issues
• Application crashes
• Network failures
• Validation errors

This helps prioritize improvements.

Alerts and Notifications

Organizations often configure alerts for:

• Multiple workflow failures
• Authentication problems
• High error rates
• Excessive execution time
• Agent unavailability
• Service interruptions

Early alerts reduce downtime.

Security Best Practices

Computer Use automation may interact with sensitive enterprise applications.

Recommended practices include:

Principle of Least Privilege

Grant only the permissions required.

Avoid:

• Global Administrator
• System Administrator

unless absolutely necessary.

Secure Credential Storage

Never hardcode:

• passwords
• API keys
• connection strings

Instead use:

• secure connections
• credential vaults
• managed identities where applicable

Data Protection

Protect:

• customer records
• financial data
• HR information
• healthcare information

Avoid displaying unnecessary sensitive information during automated sessions.

Network Security

Protect communication through:

• HTTPS
• encrypted connections
• VPNs
• private networking
• firewall policies

Audit Logging

Maintain complete audit trails showing:

• who started automation
• when it ran
• what actions occurred
• whether it succeeded
• data accessed

Governance Considerations

Large organizations should establish governance policies.

Examples include:

Approved Automation Catalog

Document:

• automation purpose
• owner
• business unit
• data sources
• permissions
• dependencies

Change Management

Whenever an application UI changes:

• test automation
• validate workflows
• update selectors
• redeploy safely

Never assume automation continues working after software upgrades.

Environment Separation

Maintain separate environments:

• Development
• Test
• Production

This prevents accidental production disruptions.

Version Control

Maintain versions of:

• Topics
• Flows
• Computer Use configurations
• Prompt changes
• Connectors

Versioning simplifies rollback.

Optimizing Computer Use

Optimization improves reliability.

Recommendations include:

Prefer Stable UI Elements

Avoid selecting:

• moving icons
• temporary banners
• advertisements
• notifications

Instead select:

• permanent buttons
• labeled controls
• predictable navigation

Reduce Unnecessary Clicks

Instead of:

Home
→ Menu
→ Settings
→ Reports
→ Monthly

navigate directly when possible.

Fewer actions reduce failure risk.

Wait for Application Readiness

Do not click immediately after loading.

Allow sufficient time for:

• pages
• dialogs
• data grids
• forms

to finish loading.

Validate Before Continuing

Verify:

• page loaded
• expected button exists
• confirmation displayed

before moving to the next step.

Handle Exceptions

Good automation plans for:

• pop-up windows
• invalid input
• unavailable services
• expired sessions
• disconnected networks

Graceful recovery greatly improves reliability.

Common Troubleshooting Scenarios

Problem

Button cannot be found.

Possible causes:

• UI changed
• page not loaded
• screen resolution changed
• localization differences

Possible solutions:

• retrain selector
• increase wait time
• verify application version

Problem

Automation clicks wrong location.

Possible causes:

• window resized
• scaling changed
• UI redesign

Possible solutions:

• use stable visual anchors
• update automation
• standardize display settings

Problem

Workflow times out.

Possible causes:

• slow network
• server delays
• large reports
• authentication latency

Possible solutions:

• increase timeout
• optimize workflow
• improve infrastructure

Problem

Authentication repeatedly fails.

Possible causes:

• expired credentials
• password changes
• MFA requirements
• permission changes

Possible solutions:

• update credentials
• review authentication policies
• validate permissions

Computer Use vs Traditional Automation

More AB-620 Exam Tips

Remember these key points:

• Computer Use automates graphical user interfaces.
• It should generally be used only when APIs or connectors are unavailable or impractical.
• UI changes can break automation.
• Monitoring execution logs is essential for troubleshooting.
• Apply least-privilege access.
• Separate development, testing, and production environments.
• Validate screen state before performing actions.
• Use retries and exception handling to improve reliability.
• Maintain audit logs for governance and compliance.
• Prefer API-based automation when possible for performance and reliability.

AB-620 Practice Exam Questions

Question 1

A company must automate a legacy desktop application that provides no APIs or connectors. Which capability is the best choice?

A. Azure AI Search

B. Computer Use

C. Adaptive Cards

D. Generative Answers

Answer: B

Explanation:
Computer Use enables an agent to interact directly with a graphical user interface, making it suitable for legacy applications that lack APIs or connectors.

Question 2

Which monitoring metric is most useful for identifying whether an application’s interface has recently changed?

A. Number of licensed users

B. Storage capacity

C. Sudden increase in failed element recognition

D. Number of environments

Answer: C

Explanation:
A sudden rise in element recognition failures often indicates that the application’s user interface has changed, causing automation to fail.

Question 3

An administrator wants to minimize security risks when configuring Computer Use. What is the recommended approach?

A. Assign Global Administrator permissions to every automation account.

B. Store passwords directly in topics.

C. Disable audit logging.

D. Grant only the permissions required for the automation.

Answer: D

Explanation:
Following the principle of least privilege reduces security risks by limiting permissions to only those necessary for the automation.

Question 4

A workflow repeatedly fails because pages have not completely loaded before the next click occurs. Which change would most likely resolve the issue?

A. Reduce timeout values.

B. Disable logging.

C. Add waits or validation that the page has fully loaded before continuing.

D. Increase screen resolution.

Answer: C

Explanation:
Adding waits or verifying that a page is fully loaded helps prevent actions from occurring before the interface is ready.

Question 5

Which scenario is the strongest candidate for Computer Use?

A. Reading information from a well-documented REST API.

B. Querying Azure SQL Database through a connector.

C. Automating a Windows desktop application with no automation interface.

D. Calling a Power Automate flow.

Answer: C

Explanation:
Computer Use is designed for interacting with applications through their graphical interface when APIs or connectors are unavailable.

Question 6

What is the primary reason organizations maintain execution logs for Computer Use sessions?

A. To increase processor speed.

B. To improve internet bandwidth.

C. To provide troubleshooting, auditing, and compliance information.

D. To replace application backups.

Answer: C

Explanation:
Execution logs provide a record of actions, errors, timings, and outcomes that support troubleshooting, auditing, and regulatory compliance.

Question 7

Which practice improves the reliability of Computer Use automations?

A. Clicking elements immediately after opening every page.

B. Selecting temporary notification banners as navigation points.

C. Avoiding validation of page state.

D. Using stable interface elements and reducing unnecessary navigation.

Answer: D

Explanation:
Stable UI elements are less likely to change, and minimizing navigation reduces opportunities for failures.

Question 8

A company deploys Computer Use automations directly into production without testing. What is the greatest risk?

A. Faster execution.

B. Increased automation reliability.

C. Unexpected failures affecting production users.

D. Reduced logging information.

Answer: C

Explanation:
Skipping testing increases the likelihood that defects or UI incompatibilities will disrupt production processes.

Question 9

Which event is most likely to require updates to a Computer Use automation?

A. Increasing storage capacity.

B. A redesign of the target application’s user interface.

C. Adding another Microsoft 365 user.

D. Renaming a Dataverse table unrelated to the workflow.

Answer: B

Explanation:
Computer Use relies on visual interface elements. UI redesigns often require selectors or interaction logic to be updated.

Question 10

Why is API-based automation generally preferred over Computer Use when both options are available?

A. APIs require more manual interaction.

B. APIs always display a graphical interface.

C. APIs are typically faster, more reliable, and less affected by UI changes.

D. APIs cannot return structured data.

Answer: C

Explanation:
API-based automation communicates directly with backend services, avoiding screen interactions and making it more efficient and resilient than UI automation.

Go to the AB-620 Exam Prep Hub main page