This post is a part of the DP-800: Developing AI-Enabled Database Solutions Exam Prep Hub.
This topic falls under these sections:
Implement AI capabilities in database solutions (25–30%)
   --> Design and implement retrieval-augmented generation (RAG)
      --> Send results to a language model

Note that there are 10 practice questions (with answers) at the end of each section to help you solidify your knowledge of the material. Also, there are 4 practice tests with 30 questions each available from the hub's main page below the exam topics section.

Introduction

One of the final steps in a Retrieval-Augmented Generation (RAG) workflow is sending retrieved data to a Large Language Model (LLM). After retrieving relevant information from a SQL database, vector index, or hybrid search system, the application packages the data into a prompt and submits it to an AI model through a REST API. The quality of this process directly affects the accuracy, relevance, security, and efficiency of the generated response.

For the DP-800 exam, you should understand how to prepare retrieved results for language model processing, construct effective prompts, submit requests to AI services, handle responses, and follow best practices for security, performance, and reliability.

Where This Step Fits in a RAG Workflow

A Retrieval-Augmented Generation solution consists of several stages.

User Question
      │
      ▼
Generate Query Embedding
      │
      ▼
Vector or Hybrid Search
      │
      ▼
Retrieve Relevant Documents
      │
      ▼
Prepare Context
      │
      ▼
Build Prompt
      │
      ▼
Send Request to Language Model
      │
      ▼
Receive AI Response
      │
      ▼
Return Answer to User

Sending the retrieved results to the language model is the bridge between the retrieval system and the AI model.

Why Send Retrieved Results?

Large Language Models do not automatically have access to:

• SQL databases
• Internal documentation
• Company policies
• Product catalogs
• Customer records
• Knowledge bases

Instead, developers retrieve the necessary information and include it in the prompt sent to the model.

This process grounds the AI response in trusted, current information.

Components of a Request

A typical request sent to a language model includes several elements.

System Instructions

The system message defines the model’s role and behavior.

Example:

You are an expert SQL database assistant.
Answer only using the supplied context.

System instructions establish the rules the model should follow.

Retrieved Context

The retrieved context contains the information found during vector or hybrid search.

Example:

Document:
Clustered indexes physically store rows according to the index key.

Only relevant context should be included.

User Question

The original user request is included.

Example:

Why do clustered indexes improve query performance?

The language model combines the retrieved context with the user question to generate an answer.

Output Instructions

Developers may specify:

• Response length
• Formatting
• Tone
• JSON output
• Markdown output
• Bullet lists

Example:

Provide a concise answer in three bullet points.

Preparing Retrieved Results

Retrieved documents often require preprocessing before being sent to the model.

Common preprocessing tasks include:

• Removing duplicate documents
• Eliminating irrelevant information
• Trimming excessively long content
• Combining related results
• Filtering unauthorized information
• Formatting structured data as JSON when appropriate

Proper preparation improves both response quality and efficiency.

Selecting Relevant Context

Sending too much information can reduce answer quality and increase cost.

Best practice:

Retrieve only the top-ranking documents.

For example:

Instead of sending:

• 50 documents

Send:

• Top 3–10 highly relevant documents

The exact number depends on the application’s requirements and the model’s context window.

Structuring the Prompt

A well-organized prompt improves response quality.

A common structure is:

System Instructions
Retrieved Context
User Question
Expected Response Format

Example:

You are a SQL expert.
Context:
Clustered indexes physically organize table rows according to the key.
Question:
Why do clustered indexes improve query performance?
Answer only using the provided context.

Sending Structured Data

Sometimes the retrieved information is relational data rather than documents.

Example SQL output:

Instead of sending the table directly, developers often convert it to JSON.

Example:

{
  "Customer":"Contoso",
  "Country":"USA",
  "CreditLimit":50000
}

JSON provides a structured format that AI services process efficiently.

Calling the Language Model

Most AI services expose REST APIs.

A request typically includes:

• HTTPS endpoint
• HTTP POST method
• Authentication
• JSON payload

Conceptually:

Prompt
↓
JSON Request
↓
REST API
↓
Language Model
↓
JSON Response

SQL Server and Azure SQL Database can call supported REST endpoints using the sp_invoke_external_rest_endpoint stored procedure where available.

Processing the Response

Most AI services return JSON.

Example:

{
  "choices":[
    {
      "message":{
        "content":"Clustered indexes improve performance because..."
      }
    }
  ]
}

SQL applications can extract the generated text using JSON functions such as:

• JSON_VALUE
• JSON_QUERY
• OPENJSON

The application can then display, store, or further process the generated response.

Managing Context Windows

Every language model has a maximum context window.

The context window includes:

• System instructions
• Retrieved documents
• User question
• Previous conversation
• Generated response

If too much information is included, requests may fail or important information may be truncated.

Developers should:

• Remove irrelevant content.
• Retrieve fewer documents.
• Summarize long documents.
• Limit prompt size.

Token Usage

Language models process text as tokens.

More retrieved content means:

• More input tokens
• Longer inference time
• Higher API costs
• Increased latency

Reducing unnecessary context improves both performance and cost efficiency.

Security Considerations

Developers should never send sensitive information unnecessarily.

Examples include:

• Passwords
• Authentication secrets
• Personal identifiers
• Confidential financial records
• Protected health information
• Internal security credentials

Before sending data to an external AI service:

• Apply row-level security (RLS).
• Apply column-level security.
• Remove confidential fields.
• Mask sensitive values when appropriate.
• Verify user authorization.

Grounding the Response

One of the primary goals of RAG is grounding.

Grounding means that the model bases its answer on retrieved information rather than relying solely on its internal training.

Example instruction:

Answer only using the supplied documents.
If the answer is unavailable, say you do not know.

This helps reduce hallucinations.

Handling Errors

Common issues include:

Authentication Failures

Examples:

• Expired tokens
• Invalid credentials
• Missing permissions

Network Problems

Examples:

• Endpoint unavailable
• Timeouts
• DNS failures

Rate Limits

AI services may return:

429 Too Many Requests

Applications should implement retry logic using exponential backoff.

Invalid Requests

Examples:

• Malformed JSON
• Missing prompt
• Unsupported parameters

Performance Considerations

Factors affecting performance include:

• Prompt size
• Number of retrieved documents
• Network latency
• AI model size
• Token count
• Response length
• Concurrent requests

Performance can often be improved by:

• Sending fewer documents.
• Using concise prompts.
• Removing duplicate information.
• Optimizing retrieval quality.

Common Mistakes

Sending Irrelevant Documents

The language model may generate inaccurate or confusing responses.

Including Entire Database Records

Large prompts increase token usage and cost.

Poor Prompt Design

Ambiguous instructions often produce inconsistent responses.

Ignoring Security

Sensitive information should never be included unless necessary and authorized.

Missing Grounding Instructions

Without guidance, the model may rely on general knowledge instead of retrieved context.

Best Practices

• Retrieve only the most relevant documents.
• Use clear system instructions.
• Include the user’s original question.
• Organize prompts consistently.
• Limit prompt size to reduce token usage.
• Convert structured data to JSON when appropriate.
• Remove sensitive information before sending requests.
• Validate JSON payloads.
• Monitor latency and token consumption.
• Evaluate AI responses for accuracy and relevance.

Real-World Example

A company stores warranty information in SQL Server.

Workflow:

1. Customer asks:”Is my laptop still under warranty?”
2. SQL retrieves:

Product: X500
Purchase Date: January 10, 2025
Warranty: 2 Years

3. JSON is generated:

{
  "Product":"X500",
  "PurchaseDate":"2025-01-10",
  "Warranty":"2 Years"
}

4. Prompt sent to the language model:

Use the following warranty information:
{
  "Product":"X500",
  "PurchaseDate":"2025-01-10",
  "Warranty":"2 Years"
}
Answer whether the warranty is still valid.

The language model generates a grounded response using the supplied business data.

DP-800 Exam Tips

Remember these key points for the exam:

• Sending results to the language model is the final step before AI response generation in a RAG workflow.
• Retrieved documents should be relevant, concise, and properly formatted.
• System instructions help guide model behavior.
• Structured SQL data is often converted to JSON before being included in prompts.
• Smaller prompts reduce latency and token costs.
• Grounding instructions help reduce hallucinations.
• Responses from AI services are typically returned as JSON.
• Sensitive information should be removed before sending requests to external AI services.

Practice Exam Questions

Question 1

A developer is building a Retrieval-Augmented Generation (RAG) application.

After retrieving relevant documents from a vector search, what is the next logical step?

A. Send the retrieved context to the language model as part of the prompt.

B. Retrain the language model.

C. Rebuild the vector index.

D. Delete duplicate embeddings.

Answer: A

Explanation:
After retrieval, the relevant documents are incorporated into the prompt and sent to the language model so it can generate a grounded response.

Question 2

Why should retrieved documents be included in a prompt sent to a language model?

A. To permanently update the model’s training data.

B. To ground the model’s response using relevant information.

C. To reduce embedding dimensions.

D. To replace vector indexes.

Answer: B

Explanation:
Including retrieved context enables the model to generate responses based on current, authoritative information rather than relying solely on pre-trained knowledge.

Question 3

Which prompt component defines the behavior the language model should follow?

A. Retrieved context

B. User question

C. System instructions

D. JSON response

Answer: C

Explanation:
System instructions establish the role, behavior, and constraints for the language model, such as answering only from the supplied context.

Question 4

A developer sends fifty retrieved documents to a language model, even though only five are relevant.

What is the most likely consequence?

A. Improved grounding accuracy.

B. Reduced API costs.

C. Faster inference.

D. Increased token usage, latency, and potential reduction in response quality.

Answer: D

Explanation:
Including excessive context increases prompt size, consumes more tokens, raises costs, and may dilute the relevance of the information presented to the model.

Question 5

Which format is commonly used to send structured SQL query results to a language model?

A. Binary

B. XML

C. JSON

D. CSV

Answer: C

Explanation:
JSON is the standard format for exchanging structured data with AI services because it is lightweight, hierarchical, and widely supported.

Question 6

What is the primary purpose of grounding instructions such as “Answer only using the supplied context”?

A. Increase the embedding dimension.

B. Reduce hallucinations by limiting the model to retrieved information.

C. Eliminate authentication requirements.

D. Automatically compress prompts.

Answer: B

Explanation:
Grounding instructions encourage the model to base its responses on the retrieved documents instead of relying on unsupported assumptions or prior training.

Question 7

A language model returns its response as JSON.

Which SQL functions can be used to extract the generated answer?

A. MERGE and GROUP BY

B. ROW_NUMBER and RANK

C. STRING_AGG and PIVOT

D. JSON_VALUE, JSON_QUERY, and OPENJSON

Answer: D

Explanation:
SQL Server provides JSON functions that allow applications to parse AI responses and extract specific values from JSON documents.

Question 8

Which security practice is most appropriate before sending retrieved results to an external AI service?

A. Include every available column to maximize context.

B. Remove sensitive or unauthorized information from the retrieved data.

C. Disable row-level security.

D. Send authentication credentials within the prompt.

Answer: B

Explanation:
Only the information necessary for the AI task should be sent. Sensitive data should be removed or masked, and normal security controls should remain in effect.

Question 9

Why is prompt size an important consideration when sending results to a language model?

A. Larger prompts always improve response quality.

B. Prompt size has no effect on AI services.

C. Larger prompts increase token usage, cost, and response latency.

D. Prompt size determines the embedding algorithm.

Answer: C

Explanation:
Every token contributes to processing time and cost. Keeping prompts concise improves performance while reducing API expenses.

Question 10

A company wants an AI assistant to answer questions using current warranty information stored in SQL Server.

Which approach best supports this requirement?

A. Fine-tune the language model every time warranty records change.

B. Store warranty records directly inside the model.

C. Build a RAG workflow that retrieves the current warranty data, formats it appropriately, and sends it to the language model.

D. Disable retrieval and rely only on the model’s training data.

Answer: C

Explanation:
A RAG solution retrieves current business data at query time, formats it (often as JSON), and sends it to the language model, allowing responses to remain accurate without requiring model retraining.

Go to the DP-800 Exam Prep Hub main page

This post is a part of the DP-800: Developing AI-Enabled Database Solutions Exam Prep Hub.
This topic falls under these sections:
Implement AI capabilities in database solutions (25–30%)
   --> Design and implement retrieval-augmented generation (RAG)
      --> Extract language model responses

Note that there are 10 practice questions (with answers) at the end of each section to help you solidify your knowledge of the material. Also, there are 4 practice tests with 30 questions each available from the hub's main page below the exam topics section.

Introduction

After a Large Language Model (LLM) generates a response, database applications must extract the returned content so it can be displayed to users, stored in the database, or used by downstream processes. Because most AI services return results in JSON format, developers must understand how to parse JSON, extract relevant values, handle errors, validate responses, and integrate the output into SQL-based applications.

For the DP-800 exam, you should understand the structure of language model responses, how to extract values using SQL JSON functions, how to handle different response formats, and the best practices for securely and efficiently processing AI-generated output.

Where Response Extraction Fits in a RAG Workflow

Extracting the language model response is one of the final stages in a Retrieval-Augmented Generation (RAG) pipeline.

User Question
      │
      ▼
Retrieve Relevant Documents
      │
      ▼
Build Prompt
      │
      ▼
Send Request to AI Model
      │
      ▼
Receive JSON Response
      │
      ▼
Extract Generated Content
      │
      ▼
Display or Store Results

Without response extraction, the application cannot effectively use the AI-generated answer.

Why AI Responses Are Returned as JSON

Most AI services expose REST APIs.

REST APIs typically exchange data using JSON because it is:

• Lightweight
• Human-readable
• Machine-readable
• Widely supported
• Easy to parse

Whether using Azure AI Foundry models, Azure OpenAI Service, or other AI providers, JSON is the standard response format.

Typical Language Model Response

Although the exact schema varies by provider and API version, chat completion APIs commonly return a structure similar to the following:

{
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "Clustered indexes improve performance because the table rows are stored in key order."
      }
    }
  ]
}

The application typically extracts only the generated text, while ignoring metadata unless it is needed for monitoring or diagnostics.

Common Elements in AI Responses

A language model response may include:

• Generated text
• Response identifier
• Model name
• Completion reason
• Token usage statistics
• Timestamps
• Metadata

Example (simplified):

{
  "id": "chatcmpl-123",
  "model": "gpt-4.1",
  "choices": [
    {
      "message": {
        "content": "Answer text..."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 125,
    "completion_tokens": 38,
    "total_tokens": 163
  }
}

Developers often extract both the generated answer and token usage for logging or cost monitoring.

Parsing JSON in SQL

SQL Server and Azure SQL Database provide built-in JSON functions.

The most commonly used are:

• JSON_VALUE
• JSON_QUERY
• OPENJSON

These functions enable developers to retrieve values from JSON returned by an AI service.

Using JSON_VALUE

JSON_VALUE extracts a single scalar value.

Example:

SELECT JSON_VALUE(@Response,
'$.choices[0].message.content');

Result:

Clustered indexes improve performance because...

This is the most common method for retrieving the generated response.

Using JSON_QUERY

JSON_QUERY extracts JSON objects or arrays.

Example:

SELECT JSON_QUERY(@Response,
'$.choices');

This returns the complete choices array rather than a single value.

Use JSON_QUERY when you need an entire object or array for further processing.

Using OPENJSON

OPENJSON converts JSON into relational rows and columns.

Example:

SELECT *
FROM OPENJSON(@Response, '$.choices');

This is useful when:

• Multiple completions are returned
• Arrays must be processed
• Nested JSON must be flattened

Extracting Token Usage

Many AI services report token consumption.

Example:

{
  "usage": {
      "prompt_tokens":125,
      "completion_tokens":40,
      "total_tokens":165
  }
}

Developers can extract these values.

Example:

SELECT JSON_VALUE(@Response,
'$.usage.total_tokens');

Tracking token usage helps monitor:

• API costs
• Performance
• Resource consumption

Processing Multiple Choices

Some APIs may return multiple candidate responses.

Example:

{
  "choices":[
      {"message":{"content":"Option 1"}},
      {"message":{"content":"Option 2"}}
  ]
}

Developers can use OPENJSON to iterate through the array and select the preferred response.

Storing AI Responses

Generated responses may be:

• Displayed to users
• Saved to SQL tables
• Logged for auditing
• Indexed for future retrieval
• Used by downstream workflows

Example table:

Proper storage supports auditing, analytics, and troubleshooting.

Validating Responses

Applications should validate AI responses before using them.

Check for:

• Missing content
• Empty responses
• Malformed JSON
• Unexpected schema
• API errors

Validation improves application reliability.

Handling API Errors

Not every REST call succeeds.

Possible errors include:

Authentication Failure

Examples:

• Invalid token
• Expired credentials

Network Errors

Examples:

• Timeout
• DNS failure
• Connection failure

Invalid Request

Examples:

• Malformed JSON
• Missing prompt
• Unsupported parameters

Rate Limiting

Example:

429 Too Many Requests

Applications should implement retry logic using exponential backoff where appropriate.

Finish Reasons

Many chat completion APIs include a finish reason.

Examples:

• stop
• length
• content_filter

Meaning:

Applications may use this information to determine whether a response is complete.

Processing Structured Output

Some prompts request JSON output rather than plain text.

Example response:

{
  "summary":"Order shipped.",
  "priority":"High"
}

SQL JSON functions can extract each property individually.

Example:

SELECT JSON_VALUE(@Response,
'$.summary');

Structured outputs are particularly useful for workflow automation.

Security Considerations

When processing AI responses:

• Validate all returned data.
• Do not assume responses are always correct.
• Avoid executing generated SQL without validation.
• Protect sensitive information.
• Log responses securely.
• Apply least-privilege access controls.

Even trusted AI services should be treated as external systems whose outputs require validation.

Performance Considerations

Large responses require:

• More network bandwidth
• More parsing time
• More storage
• More tokens

Developers should:

• Limit response length where appropriate.
• Extract only required fields.
• Avoid storing unnecessary metadata.
• Archive logs according to retention policies.

Common Mistakes

Assuming Every Response Has the Same Schema

Different AI services and API versions may return different JSON structures.

Ignoring Errors

Applications should always check for API failures before attempting to parse the response.

Parsing Entire JSON Documents

Extract only the required values to improve efficiency.

Not Validating Responses

Malformed or incomplete responses should be handled gracefully.

Ignoring Token Usage

Monitoring token consumption helps control costs.

Best Practices

• Parse responses using SQL JSON functions.
• Use JSON_VALUE for scalar values.
• Use JSON_QUERY for objects and arrays.
• Use OPENJSON for arrays and complex JSON.
• Validate response schemas before processing.
• Log errors separately from successful responses.
• Track token usage for monitoring and optimization.
• Limit stored data to what is necessary.
• Handle rate limits and transient failures gracefully.
• Design applications to tolerate API schema changes when possible.

Real-World Example

A customer asks:

“Summarize this support ticket.”

The application:

1. Retrieves ticket information from SQL.
2. Sends it to a language model.
3. Receives:

{
  "choices":[
    {
      "message":{
        "content":"The customer reports intermittent login failures caused by expired authentication tokens."
      }
    }
  ]
}

The application extracts:

SELECT JSON_VALUE(@Response,
'$.choices[0].message.content');

The extracted summary is displayed to the support agent and optionally stored for future reference.

DP-800 Exam Tips

Remember these key points for the exam:

• Most language model APIs return JSON responses.
• JSON_VALUE extracts individual scalar values.
• JSON_QUERY retrieves JSON objects or arrays.
• OPENJSON converts JSON arrays and objects into relational data.
• Applications should validate AI responses before using them.
• Token usage information helps monitor API costs.
• Finish reasons indicate how the model completed generation.
• Handle API errors, rate limits, and malformed responses gracefully.
• Store only the data needed for business purposes.
• AI-generated output should always be treated as data that requires validation before use.

Practice Exam Questions

Question 1

A SQL application receives a JSON response from a language model and needs to extract the generated answer.

Which SQL function is most appropriate for retrieving a single text value?

A. JSON_VALUE

B. JSON_QUERY

C. OPENJSON

D. STRING_SPLIT

Answer: A

Explanation:
JSON_VALUE extracts a single scalar value from a JSON document, making it ideal for retrieving the generated response text.

Question 2

A developer wants to retrieve the entire choices array from a language model response.

Which SQL function should be used?

A. ROW_NUMBER

B. JSON_QUERY

C. MERGE

D. JSON_VALUE

Answer: B

Explanation:
JSON_QUERY returns JSON objects or arrays rather than individual scalar values, making it appropriate for extracting the complete choices array.

Question 3

When is OPENJSON most useful?

A. When extracting a single property value.

B. When converting JSON arrays into relational rows and columns.

C. When generating embeddings.

D. When creating vector indexes.

Answer: B

Explanation:
OPENJSON parses JSON arrays and objects into tabular data that can be queried using SQL.

Question 4

Why should applications validate AI responses before using them?

A. JSON responses are always encrypted.

B. Validation reduces database storage requirements.

C. AI responses may be malformed, incomplete, or contain unexpected structures.

D. Validation automatically reduces token usage.

Answer: C

Explanation:
Applications should verify that responses are valid, complete, and conform to the expected schema before processing them.

Question 5

A developer wants to monitor AI service costs.

Which information should be extracted from the response?

A. The database transaction log.

B. Vector dimensions.

C. Token usage statistics.

D. Query execution plans.

Answer: C

Explanation:
Many AI APIs return token usage information, which is useful for monitoring API consumption and estimating costs.

Question 6

What does a finish reason of stop typically indicate?

A. The request exceeded the maximum token limit.

B. The response was blocked by a content filter.

C. The model completed the response normally.

D. Authentication failed.

Answer: C

Explanation:
A finish reason of stop indicates that the model reached a natural completion point without interruption.

Question 7

A developer receives multiple candidate responses from an AI service.

Which SQL feature is best suited for processing all returned responses?

A. JSON_VALUE

B. OPENJSON

C. GROUP BY

D. FOR JSON AUTO

Answer: B

Explanation:
OPENJSON can iterate through arrays, making it ideal for processing multiple response choices.

Question 8

Which practice best improves the reliability of applications consuming AI responses?

A. Assume every response follows the same JSON schema.

B. Execute AI-generated SQL statements without review.

C. Validate the response structure and handle errors gracefully.

D. Ignore API error messages.

Answer: C

Explanation:
Validating responses and implementing robust error handling help applications remain reliable even when API responses change or errors occur.

Question 9

Why should developers avoid storing unnecessary metadata from AI responses?

A. Metadata prevents JSON parsing.

B. It can increase storage requirements without providing business value.

C. Metadata invalidates embeddings.

D. Metadata reduces retrieval accuracy.

Answer: B

Explanation:
Storing only the required information minimizes storage costs and simplifies downstream processing.

Question 10

A SQL application receives the following JSON:

{
  "choices":[
    {
      "message":{
        "content":"The shipment will arrive tomorrow."
      }
    }
  ]
}

Which value should typically be presented to the end user?

A. The complete JSON document.

B. The choices array.

C. The generated text contained in message.content.

D. The API response identifier.

Answer: C

Explanation:
The value stored in message.content contains the natural-language response generated by the language model and is typically the information displayed to users.

Go to the DP-800 Exam Prep Hub main page