This post is a part of the DP-800: Developing AI-Enabled Database Solutions Exam Prep Hub.
This topic falls under these sections:
Implement AI capabilities in database solutions (25–30%)
   --> Design and implement models and embeddings
      --> Design and implement chunks for embeddings

Note that there are 10 practice questions (with answers) at the end of each section to help you solidify your knowledge of the material. Also, there are 4 practice tests with 30 questions each available from the hub's main page below the exam topics section.

Introduction

One of the most important design decisions when building AI-enabled database solutions is determining how data should be divided before generating embeddings. This process, known as chunking, directly affects the quality of semantic search, Retrieval-Augmented Generation (RAG), recommendation systems, and AI-powered question answering.

Embedding models convert text into numerical vector representations that capture semantic meaning. However, they have input token limits and perform best when the input represents a coherent concept rather than an entire document. Consequently, documents are usually divided into manageable sections called chunks, and an embedding is generated for each chunk individually.

For the DP-800 exam, Microsoft expects candidates to understand how to design chunking strategies, determine appropriate chunk sizes, preserve context through overlap, store chunk metadata, and balance retrieval quality with storage and processing costs.

What Is Chunking?

Chunking is the process of dividing a document or other textual content into smaller units before generating embeddings.

Instead of embedding an entire 100-page document, the document is divided into logical sections such as:

• Chapters
• Sections
• Paragraph groups
• Pages
• Individual articles
• Code blocks
• FAQ entries

Each chunk is embedded independently and stored in a vector database or SQL table.

For example:

Original document

Employee Handbook
    Chapter 1
    Chapter 2
    Chapter 3
    Chapter 4

After chunking:

Chunk 1 → Introduction
Chunk 2 → Employee Benefits
Chunk 3 → Leave Policies
Chunk 4 → Workplace Conduct
Chunk 5 → Remote Work
Chunk 6 → Security Policies

Each chunk receives its own embedding vector.

Why Chunk Documents?

Large Language Models and embedding models have practical limitations.

Reasons for chunking include:

• Token limits
• Improved semantic accuracy
• Faster vector searches
• Reduced hallucinations
• Better retrieval precision
• Lower processing costs
• Easier document updates

Embedding an entire document often produces a vector that represents many unrelated topics simultaneously, reducing search accuracy.

Chunking in a Retrieval-Augmented Generation (RAG) System

A typical RAG workflow follows these steps:

Documents
      │
      ▼
Chunk Documents
      │
      ▼
Generate Embeddings
      │
      ▼
Store Vectors
      │
      ▼
User Query
      │
      ▼
Generate Query Embedding
      │
      ▼
Similarity Search
      │
      ▼
Retrieve Best Chunks
      │
      ▼
Provide Context to LLM
      │
      ▼
Generate Response

Without proper chunking, the retrieved context may be incomplete, overly broad, or irrelevant.

Characteristics of Good Chunks

An effective chunk should be:

• Semantically coherent
• Self-contained
• Small enough for efficient retrieval
• Large enough to preserve context
• Easy to trace back to the original source
• Consistent in formatting

Good chunks represent one primary idea or closely related concepts.

Example:

Good chunk:

Employees may work remotely up to three days each week with manager approval.

Poor chunk:

Employees may work remotely…

(ends halfway through explanation)

Incomplete chunks reduce retrieval quality.

Choosing an Appropriate Chunk Size

There is no universal chunk size.

Instead, chunk size depends on:

• Document type
• User questions
• Embedding model limits
• Retrieval strategy
• Desired context

General guidelines:

Microsoft generally emphasizes semantic chunking over arbitrary character counts.

Fixed-Length Chunking

The simplest strategy divides text into equal-sized pieces.

Example:

Chunk 1
1–500 characters
Chunk 2
501–1000 characters
Chunk 3
1001–1500 characters

Advantages:

• Simple
• Fast
• Easy to automate

Disadvantages:

• Breaks sentences
• Splits ideas
• Reduces semantic quality

Because it ignores meaning, fixed-length chunking is generally less effective than semantic approaches.

Semantic Chunking

Semantic chunking divides documents based on meaning rather than length.

Examples include:

• Headings
• Sections
• Chapters
• Topics
• Complete paragraphs
• Individual procedures
• Entire FAQ entries

Example:

Instead of:

Characters 1–1000
Characters 1001–2000

Use:

Vacation Policy
Medical Leave
Parental Leave
Travel Reimbursement

Semantic chunking produces embeddings that more accurately represent individual concepts.

Sliding Window (Overlapping) Chunking

One common challenge occurs when important information spans the boundary between two chunks.

For example:

Chunk 1
Employees may work remotely
after manager approval.
Chunk 2
After manager approval,
employees must...

The phrase “after manager approval” belongs to both chunks.

To preserve context, systems use overlapping chunks.

Example:

Chunk 1
Sentence 1
Sentence 2
Sentence 3
Sentence 4
Chunk 2
Sentence 3
Sentence 4
Sentence 5
Sentence 6

The overlapping sentences help ensure continuity during retrieval.

Advantages of Overlapping Chunks

Overlap helps:

• Preserve context
• Improve semantic similarity
• Prevent broken explanations
• Improve RAG responses
• Increase retrieval accuracy

Most enterprise RAG systems use overlap rather than completely independent chunks.

Drawbacks of Excessive Overlap

Too much overlap creates problems:

• Duplicate embeddings
• Larger storage requirements
• Longer indexing time
• Increased search costs
• More duplicate search results

The overlap should be sufficient to maintain context without introducing excessive redundancy.

Chunking Structured Data

Not all embeddings originate from unstructured documents.

Structured SQL data can also be chunked.

Examples include:

• Product descriptions
• Customer support articles
• Knowledge base records
• Employee profiles
• Medical summaries

Rather than combining multiple rows, each logical business record is typically embedded separately.

Example:

Products
Laptop A
Laptop B
Laptop C

Each product becomes its own chunk.

Chunking Source Code

Source code should not be chunked arbitrarily.

Preferred chunk boundaries include:

• Functions
• Methods
• Classes
• Modules
• Stored procedures
• SQL scripts

Example:

Instead of:

Characters 1–1000

Use:

CreateCustomer()
DeleteCustomer()
CalculateTax()
GenerateInvoice()

This preserves the logical structure of the code and improves semantic retrieval for developer-focused AI assistants.

Chunk Metadata

Every chunk should include metadata that enables accurate retrieval and traceability.

Common metadata includes:

• Document ID
• Chunk ID
• Source file name
• Page number
• Section heading
• Chunk sequence number
• Creation date
• Last modified date
• Author
• Security classification

Example:

Metadata supports source attribution, document reconstruction, filtering, and citation generation.

Reconstructing Context

Many RAG applications retrieve more than one chunk for a user query.

For example:

Chunk 21
Chunk 22
Chunk 23

Because each chunk contains metadata indicating its position in the document, the application can present adjacent chunks together to provide richer context to the language model.

Testing Chunk Quality

Designing chunking strategies is an iterative process. After implementing a strategy, evaluate it using representative queries and measure retrieval quality.

Common evaluation criteria include:

• Retrieval precision – Are the returned chunks relevant to the query?
• Retrieval recall – Are important chunks consistently found?
• Context completeness – Does each chunk contain enough information to answer the question?
• Duplicate retrieval – Are overlapping chunks causing redundant results?
• Latency – Does the chunking strategy affect search performance?
• Storage efficiency – How much additional storage is required for embeddings and metadata?

Testing with real-world questions often reveals whether chunks are too large, too small, or improperly aligned with document structure.

Best Practices

Microsoft recommends following several best practices when designing chunks for embeddings:

• Prefer semantic chunking over fixed-length splitting whenever possible.
• Keep each chunk focused on a single topic or concept.
• Use overlap to preserve context across chunk boundaries.
• Preserve document structure by chunking at headings, sections, or clauses.
• Store comprehensive metadata with every chunk.
• Choose chunk sizes appropriate for the document type and expected user queries.
• Avoid creating chunks that are so small they lose context or so large they dilute semantic meaning.
• Re-evaluate chunking strategies as documents, embedding models, or application requirements evolve.
• Test retrieval quality with representative workloads before deploying to production.
• Balance retrieval accuracy against storage costs, indexing time, and search performance.

DP-800 Exam Tips

For the DP-800 exam, you should understand how chunk design influences the effectiveness of AI-enabled database solutions. Be prepared to compare fixed-length, semantic, and overlapping chunking strategies and recognize when each is appropriate. Expect scenario-based questions that ask you to optimize retrieval quality, preserve context, reduce hallucinations, or improve the performance of Retrieval-Augmented Generation (RAG) systems. Also know the importance of metadata, chunk boundaries, and testing strategies in building scalable, production-ready embedding solutions.

Practice Exam Questions

Question 1

You are creating a Retrieval-Augmented Generation (RAG) solution for an organization’s HR policy documents. Each document averages 40 pages.

What is the primary reason for dividing the documents into chunks before generating embeddings?

A. SQL databases cannot store documents larger than 1 MB.

B. Embedding models only support numeric data.

C. Smaller chunks improve semantic retrieval by capturing focused context.

D. Chunking compresses documents into fewer tokens.

Answer: C

Explanation:
Embedding models generate vector representations for pieces of text. Smaller, semantically coherent chunks produce embeddings that represent specific concepts, making similarity searches much more accurate than embedding an entire document.

Question 2

A development team is designing chunk sizes for technical manuals.

Which chunking strategy generally produces the best retrieval quality?

A. Split documents into semantically meaningful sections of moderate length

B. One chunk for every document

C. Split every five characters

D. Create one chunk for every paragraph regardless of context

Answer: A

Explanation:
Chunks should balance context and specificity. Semantically meaningful sections preserve enough information for accurate retrieval without becoming overly broad.

Question 3

Why are overlapping chunks commonly used when generating embeddings?

A. To reduce storage requirements

B. To eliminate duplicate vectors

C. To preserve context across chunk boundaries

D. To increase SQL query speed

Answer: C

Explanation:
Without overlap, important information that spans chunk boundaries can be lost. Overlapping text ensures that concepts appearing near boundaries remain available during retrieval.

Question 4

Which factor has the greatest influence on selecting an appropriate chunk size?

A. The SQL Server version

B. The font used in the original document

C. The amount of database memory

D. The type of content and expected user queries

Answer: D

Explanation:
Different document types require different chunk sizes. FAQs, manuals, legal documents, and source code each benefit from different chunking strategies based on how users search for information.

Question 5

A company stores product manuals that include headings, tables, and explanatory paragraphs.

Which chunking strategy is most appropriate?

A. Ignore document structure and split every 1,000 characters.

B. Chunk according to logical document sections while preserving headings.

C. Generate one embedding for every sentence.

D. Combine all manuals into one embedding.

Answer: B

Explanation:
Preserving document structure helps maintain semantic meaning and improves retrieval quality because headings provide valuable contextual information.

Question 6

What is a disadvantage of creating extremely small chunks?

A. Higher SQL licensing costs

B. More accurate semantic understanding

C. Loss of surrounding context during retrieval

D. Lower storage requirements

Answer: C

Explanation:
Very small chunks may not contain enough surrounding information for an LLM to interpret their meaning correctly, resulting in lower-quality retrieval.

Question 7

When implementing chunking for a RAG application, which metadata should typically be stored alongside each embedding?

A. SQL login credentials

B. Original document identifier and chunk position

C. Azure subscription ID

D. CPU utilization statistics

Answer: B

Explanation:
Metadata allows applications to identify the source document, reconstruct surrounding context, provide citations, and retrieve neighboring chunks when generating responses.

Question 8

A legal department requires every retrieved answer to reference complete contractual clauses.

Which chunking strategy is most appropriate?

A. Randomly divide text every 500 characters.

B. Generate one embedding for the entire contract.

C. Chunk by individual words.

D. Chunk according to clause boundaries with slight overlap.

Answer: D

Explanation:
Legal documents rely on clearly defined clauses. Chunking by clause preserves legal meaning while overlap prevents loss of context between adjacent sections.

Question 9

Which statement best describes the relationship between chunk size and retrieval performance?

A. Larger chunks always produce better search results.

B. Smaller chunks always eliminate hallucinations.

C. There is an optimal balance between context size and retrieval precision.

D. Chunk size has no impact on vector search.

Answer: C

Explanation:
Chunks that are too large reduce retrieval precision, while chunks that are too small lose context. Effective systems balance these competing factors.

Question 10

During testing, users report that retrieved passages frequently stop in the middle of explanations.

Which modification would most likely improve retrieval quality?

A. Increase overlap between adjacent chunks.

B. Reduce the embedding vector dimensions.

C. Remove document metadata.

D. Convert embeddings to integers.

Answer: A

Explanation:
Increasing overlap preserves continuity across chunk boundaries, allowing retrieval systems to return more complete passages and improving the quality of generated responses.

Go to the DP-800 Exam Prep Hub main page