This post is a part of the DP-700: Implementing Data Engineering Solutions Using Microsoft Fabric Exam Prep Hub.
This topic falls under these sections:
Ingest and transform data (30–35%)
   --> Ingest and transform streaming data
      --> Process data by using Spark structured streaming

Note that there are 10 practice questions (with answers) at the end of each section to help you solidify your knowledge of the material. Also, there are 2 practice tests with 60 questions each available from the hub's main page below the exam topics section.

Introduction

Modern analytics platforms increasingly require the ability to process data continuously as it arrives rather than waiting for scheduled batch loads. Microsoft Fabric supports this requirement through Spark Structured Streaming, a scalable and fault-tolerant stream processing engine built on Apache Spark.

For the DP-700 exam, you should understand when and how to use Spark Structured Streaming, how it differs from other real-time processing options such as Eventstreams and KQL Querysets, and how to design streaming solutions that write data into OneLake and Delta tables.

Spark Structured Streaming is commonly used when data engineers need to process streaming data with complex transformations, enrichments, joins, aggregations, and machine learning workloads while leveraging the scalability of Spark. (Microsoft Learn)

What Is Spark Structured Streaming?

Spark Structured Streaming is a stream-processing framework built on top of Apache Spark. It treats a continuous stream of incoming data as an unbounded table to which new rows are constantly appended. Developers write code using familiar DataFrame and Spark SQL operations while Spark handles the continuous execution behind the scenes. (Microsoft Learn)

Key characteristics include:

• Near real-time processing
• Fault tolerance
• Automatic recovery
• Horizontal scalability
• Support for complex transformations
• Integration with Delta Lake
• Exactly-once processing capabilities through checkpointing and transaction logs (Microsoft Learn)

How Structured Streaming Works

The processing flow typically follows these steps:

1. Read data from a streaming source.
2. Apply transformations.
3. Write results to a destination.
4. Store checkpoints to track processing progress.
5. Continue processing new data as it arrives.

Common Sources

Spark Structured Streaming supports sources such as:

• Azure Event Hubs
• Apache Kafka
• JSON files
• CSV files
• Parquet files
• Delta tables
• Eventstreams outputs (Microsoft Learn)

Common Destinations

Results can be written to:

• Lakehouse Delta tables
• OneLake storage
• Eventhouses
• Memory sinks for testing
• Other supported storage locations (Microsoft Learn)

Structured Streaming in Microsoft Fabric

Within Microsoft Fabric, Structured Streaming is most commonly implemented through:

• Notebooks
• Spark Job Definitions
• Lakehouses
• Delta tables
• OneLake storage

A typical architecture looks like this:

Azure Event Hub
       |
       v
Spark Structured Streaming
       |
       v
Delta Table in Lakehouse
       |
       v
SQL Analytics Endpoint
       |
       v
Power BI

This pattern enables streaming data to become queryable almost immediately after arrival. (Microsoft Learn)

Structured Streaming vs Batch Processing

Use Batch Processing When:

• Data changes infrequently
• Overnight processing is acceptable
• Real-time insights are unnecessary

Use Structured Streaming When:

• IoT devices generate events continuously
• Fraud detection requires immediate action
• Operational dashboards need live updates
• Telemetry data must be analyzed continuously

Micro-Batch Processing

A common DP-700 exam topic is understanding that Structured Streaming typically uses a micro-batch architecture.

Instead of processing every individual event separately, Spark groups events into small batches and processes them continuously.

Example:

Incoming Events
-----------------
10:00:00 - Event A
10:00:01 - Event B
10:00:02 - Event C
Micro-batch executes
Process A+B+C together

This approach provides:

• Better performance
• Higher throughput
• Easier fault recovery
• Familiar Spark execution model (Microsoft Learn)

Reading Streaming Data

Streaming ingestion begins with:

df = (
    spark.readStream
         .format("eventhubs")
         .load()
)

Important points:

• readStream creates a streaming DataFrame.
• Data remains continuously available.
• Spark automatically detects new events.
• The query remains active until stopped. (Microsoft Learn)

Writing Streaming Data

Streaming results are written using writeStream.

Example:

query = (
    df.writeStream
      .format("delta")
      .outputMode("append")
      .toTable("SalesEvents")
)

Common output modes include:

For Fabric data engineering scenarios, Append mode is most common. (Microsoft Learn)

Delta Lake Integration

One of the most important DP-700 concepts is integrating Structured Streaming with Delta Lake.

Benefits include:

• ACID transactions
• Schema evolution
• Time travel
• Data versioning
• Reliable streaming ingestion

Streaming data can be written directly into Delta tables:

.writeStream
.format("delta")
.toTable("Orders")

This creates a continuously updated Delta table within the Lakehouse. (Microsoft Learn)

Checkpointing

Checkpointing is critical for fault tolerance.

Example:

.option(
    "checkpointLocation",
    "Files/checkpoints/orders"
)

Checkpoints store:

• Processed offsets
• Query progress
• State information

Benefits:

• Prevents duplicate processing
• Enables recovery after failures
• Supports exactly-once processing semantics

A frequent exam scenario involves identifying missing checkpoint configurations as the root cause of duplicate or reprocessed data. (mindmeshacademy.com)

Triggers

Triggers control how often Spark processes incoming data.

Example:

.trigger(
    processingTime="1 minute"
)

Possible trigger strategies:

Larger trigger intervals often improve throughput because more events are processed together. (Microsoft Learn)

Stateful vs Stateless Processing

Stateless Processing

Each event is processed independently.

Examples:

• Filtering
• Column selection
• Simple transformations

stream.filter("temperature > 100")

Stateful Processing

Spark maintains information between batches.

Examples:

• Running totals
• Session windows
• Stream aggregations
• Deduplication

Stateful processing is more powerful but consumes additional memory and storage resources. (jumpstart.fabric.microsoft.com)

Stream Aggregations

Streaming aggregations allow continuous calculations.

Examples:

• Sales totals
• Device counts
• Average temperatures
• Transaction volumes

Example:

stream.groupBy("DeviceID") \
      .count()

This continuously updates counts as new events arrive.

Common Streaming Scenarios in Fabric

IoT Monitoring

Sensors continuously send readings.

Process:

IoT Devices
     |
Event Hub
     |
Spark Structured Streaming
     |
Lakehouse
     |
Power BI Dashboard

Application Telemetry

Applications send logs and metrics continuously.

Use cases:

• Performance monitoring
• Error tracking
• Operational dashboards

Real-Time Business Analytics

Examples include:

• Online sales monitoring
• Inventory tracking
• Customer activity analysis
• Fraud detection

Structured Streaming vs Eventstreams

DP-700 often tests when to use each technology.

Use Eventstreams for simple routing and ingestion.

Use Structured Streaming for advanced engineering workloads. (Microsoft Learn)

Production Best Practices

Use Spark Job Definitions

For production workloads, Microsoft recommends Spark Job Definitions rather than leaving notebooks running continuously. They provide better reliability and restart capabilities. (Microsoft Learn)

Configure Retry Policies

Retry policies allow automatic recovery from infrastructure failures. (Microsoft Learn)

Always Use Checkpoints

Never deploy production streaming jobs without checkpoint locations. (mindmeshacademy.com)

Optimize Partitioning

Appropriate partitioning improves throughput and downstream query performance. (Microsoft Learn)

Monitor Streaming Jobs

Use Fabric Monitoring Hub to monitor:

• Input rate
• Processing rate
• Batch duration
• Streaming query health (Microsoft Learn)

DP-700 Exam Tips

Remember these frequently tested concepts:

• Structured Streaming treats streams as continuously growing tables.
• readStream reads streaming data.
• writeStream writes streaming data.
• Delta tables are common streaming destinations.
• Checkpointing enables fault tolerance.
• Spark Job Definitions are preferred for production streaming workloads.
• Event Hubs is a common streaming source.
• Micro-batch processing is the default execution model.
• Structured Streaming is preferred when complex transformations are required.
• Eventstreams are often preferred for simpler ingestion scenarios.

Practice Exam Questions

Question 1

A company needs to process telemetry data from thousands of IoT devices as soon as it arrives. The solution must perform complex transformations before storing data in a Lakehouse.

Which technology should you choose?

A. Dataflow Gen2
B. Warehouse Stored Procedures
C. Spark Structured Streaming
D. Copy Activity

Correct Answer: C

Explanation: Spark Structured Streaming is designed for continuous data processing and complex transformations on streaming data.

Question 2

What is the primary purpose of a checkpoint location in Structured Streaming?

A. Increase Spark cluster size
B. Store temporary query results
C. Track processing progress and support recovery
D. Compress Delta files

Correct Answer: C

Explanation: Checkpoints store offsets and state information that allow recovery without reprocessing all data.

Question 3

Which method is used to create a streaming DataFrame?

A. readStream()
B. streamRead()
C. loadStreaming()
D. readDelta()

Correct Answer: A

Explanation: readStream() is the Spark API used to create streaming DataFrames.

Question 4

Which destination is most commonly used for Spark Structured Streaming in Microsoft Fabric?

A. Delta table in a Lakehouse
B. Excel workbook
C. Dataflow Gen2
D. Semantic model

Correct Answer: A

Explanation: Delta tables in Lakehouses are the primary streaming storage destination in Fabric.

Question 5

What execution model does Spark Structured Streaming primarily use?

A. Row-by-row execution
B. Continuous SQL polling
C. Micro-batch processing
D. Manual scheduling

Correct Answer: C

Explanation: Structured Streaming processes incoming data as small batches at regular intervals.

Question 6

Which Fabric component is recommended for running production Structured Streaming workloads?

A. Notebook only
B. Dataflow Gen2
C. Pipeline activity
D. Spark Job Definition

Correct Answer: D

Explanation: Spark Job Definitions provide improved reliability, retry policies, and production-grade execution.

Question 7

A streaming job must continuously calculate running totals by customer.

What type of processing is required?

A. Stateless processing
B. Stateful processing
C. Batch processing
D. Snapshot processing

Correct Answer: B

Explanation: Running totals require maintaining state across multiple batches.

Question 8

Which statement about Eventstreams and Structured Streaming is correct?

A. Eventstreams supports more advanced Spark transformations.
B. Structured Streaming is a no-code solution.
C. Structured Streaming supports complex custom code transformations.
D. Eventstreams requires Spark coding.

Correct Answer: C

Explanation: Structured Streaming provides full Spark capabilities and custom coding flexibility.

Question 9

What is the benefit of writing streaming data to Delta tables?

A. Eliminates storage costs
B. Prevents all schema changes
C. Converts data to CSV automatically
D. Provides ACID transactions and reliability

Correct Answer: D

Explanation: Delta Lake provides transactional consistency, schema evolution, and reliable streaming ingestion.

Question 10

A data engineer wants to process incoming events every 60 seconds instead of immediately.

Which feature should be configured?

A. Checkpointing
B. Consumer groups
C. Trigger interval
D. Data partitioning

Correct Answer: C

Explanation: Trigger intervals control how frequently Spark processes incoming streaming data.

Go to the DP-700 Exam Prep Hub main page.

This post is a part of the DP-700: Implementing Data Engineering Solutions Using Microsoft Fabric Exam Prep Hub.
This topic falls under these sections:
Ingest and transform data (30–35%)
   --> Ingest and transform streaming data
      --> Choose an appropriate streaming engine

Note that there are 10 practice questions (with answers) at the end of each section to help you solidify your knowledge of the material. Also, there are 2 practice tests with 60 questions each available from the hub's main page below the exam topics section.

Overview

Modern analytics solutions increasingly rely on the ability to process data as it is generated rather than waiting for scheduled batch loads. Streaming data enables organizations to react to events in near real time, support operational analytics, monitor systems, detect anomalies, and power intelligent applications.

In Microsoft Fabric, selecting the appropriate streaming engine is a critical design decision. The DP-700 exam expects candidates to understand the strengths, limitations, and ideal use cases of the various streaming technologies available in Fabric and to choose the most appropriate option based on business requirements.

This article explores the major streaming engines and technologies within Microsoft Fabric, how they compare, and when to use each one.

What Is Streaming Data?

Streaming data is data that arrives continuously from sources such as:

• IoT devices
• Sensors
• Application logs
• Clickstream events
• Social media feeds
• Financial transactions
• Manufacturing equipment
• Website activity
• Real-time telemetry

Unlike batch processing, where data is collected and processed periodically, streaming systems process data as events arrive.

Common requirements include:

• Low-latency processing
• Real-time dashboards
• Event detection
• Alert generation
• Continuous data ingestion
• Streaming analytics

Streaming Technologies in Microsoft Fabric

The primary streaming technologies that data engineers encounter in Fabric include:

Understanding when to use each is essential for the exam.

Eventstream

What Is Eventstream?

Eventstream is Fabric’s low-code real-time ingestion service.

It captures, transforms, filters, and routes streaming events from multiple sources to multiple destinations.

Think of Eventstream as the ingestion layer of a streaming architecture.

Common Sources

Eventstream can ingest data from:

• Azure Event Hubs
• Kafka endpoints
• Fabric events
• IoT sources
• Real-time telemetry systems
• Custom event producers

Common Destinations

Eventstream can send data to:

• Eventhouse
• KQL Databases
• Lakehouses
• Custom destinations
• Activator

Best Use Cases

Choose Eventstream when:

• Events must be continuously ingested
• Minimal coding is desired
• Data routing is required
• Multiple downstream consumers need the same events
• Building real-time analytics solutions

Exam Tip

If a scenario focuses on ingesting and routing real-time events, Eventstream is usually the best answer.

Eventhouse

What Is Eventhouse?

Eventhouse is a Real-Time Intelligence component optimized for storing and analyzing streaming data.

It is built on Kusto technology and provides:

• High ingestion rates
• Near real-time analytics
• Time-series analysis
• Log analytics
• Event exploration

Key Characteristics

• Optimized for append-only data
• Supports KQL
• Fast query performance
• Near real-time visibility
• Massive scalability

Best Use Cases

Use Eventhouse when:

• Large volumes of events arrive continuously
• Log analytics is required
• Telemetry analysis is needed
• Operational dashboards require low latency

Examples:

• Website activity monitoring
• Application diagnostics
• Manufacturing telemetry
• Security monitoring

KQL Databases

What Is a KQL Database?

A KQL database is the storage and query engine behind many real-time solutions.

It uses Kusto Query Language (KQL) and is highly optimized for:

• Streaming ingestion
• Log analytics
• Time-series data
• Event correlation

Advantages

• Extremely fast analytical queries
• Handles high ingestion volumes
• Rich time-series functions
• Powerful aggregation capabilities

Best Use Cases

Choose KQL databases when:

• Event analysis is the primary objective
• Massive event volumes exist
• Time-based analysis is required
• Operational monitoring is needed

Spark Structured Streaming

What Is Structured Streaming?

Spark Structured Streaming enables continuous processing using Apache Spark.

Unlike Eventstream and Eventhouse, Spark streaming is developer-focused and code-driven.

Supported languages include:

• PySpark
• Scala
• Spark SQL

Capabilities

Spark Structured Streaming supports:

• Complex transformations
• Data enrichment
• Machine learning integration
• Streaming joins
• Stateful processing
• Advanced business logic

Best Use Cases

Choose Spark Structured Streaming when:

• Complex transformations are required
• Large-scale processing is needed
• Machine learning must be integrated
• Events must be joined with reference datasets
• Custom code is acceptable

Examples:

• Fraud detection
• Customer behavior analytics
• Streaming feature engineering
• Predictive maintenance

Exam Tip

If a scenario requires advanced coding and transformation logic, Spark Structured Streaming is often the correct answer.

Real-Time Intelligence

What Is Real-Time Intelligence?

Real-Time Intelligence is Fabric’s complete platform for handling real-time data workloads.

It combines:

• Eventstream
• Eventhouse
• KQL Databases
• Data Activator
• Real-time dashboards

Benefits

Provides:

• End-to-end streaming architecture
• Real-time monitoring
• Event processing
• Alerting
• Operational analytics

Best Use Cases

Use Real-Time Intelligence when an organization needs:

• Comprehensive streaming analytics
• Operational dashboards
• Real-time monitoring
• Event-driven insights

Data Activator

What Is Data Activator?

Data Activator monitors events and automatically takes actions when specified conditions occur.

Examples include:

• Sending emails
• Triggering workflows
• Generating notifications
• Creating alerts

Example

If machine temperature exceeds 90°C:

• Generate an alert
• Notify engineers
• Open a support ticket

Best Use Cases

Choose Data Activator when:

• Business users need alerts
• Event-driven automation is required
• Low-code monitoring is desired

Pipelines Are Not Streaming Engines

A common DP-700 exam trap is confusing pipelines with streaming solutions.

Pipelines:

• Execute scheduled workloads
• Orchestrate activities
• Handle batch data movement

Pipelines do NOT provide continuous event processing.

Appropriate Pipeline Scenarios

• Daily data loads
• Weekly ETL jobs
• Scheduled orchestration
• Batch transformations

Inappropriate Pipeline Scenarios

• Second-by-second monitoring
• Real-time alerts
• Continuous event processing

Selecting the Appropriate Streaming Engine

Scenario 1: IoT Sensor Telemetry

Requirements:

• Millions of sensor events
• Real-time monitoring
• Fast analytics

Best choice:

Eventstream + Eventhouse

Scenario 2: Fraud Detection

Requirements:

• Stream transactions
• Apply advanced business rules
• Perform enrichment joins

Best choice:

Spark Structured Streaming

Scenario 3: Website Log Analysis

Requirements:

• Continuous ingestion
• Fast aggregations
• Time-series analysis

Best choice:

KQL Database/Eventhouse

Scenario 4: Equipment Failure Alerts

Requirements:

• Detect threshold breaches
• Notify operators

Best choice:

Data Activator

Scenario 5: Enterprise Real-Time Analytics Platform

Requirements:

• Complete streaming solution
• Dashboards
• Alerts
• Analytics

Best choice:

Real-Time Intelligence

Comparison of Streaming Engines

DP-700 Exam Tips

Remember these key distinctions:

• Eventstream = ingestion and routing.
• Eventhouse = real-time storage and analytics.
• KQL Database = high-performance event analytics.
• Spark Structured Streaming = advanced code-based processing.
• Data Activator = alerts and automated actions.
• Pipelines = orchestration, not streaming.
• Real-Time Intelligence = end-to-end streaming solution.

Many exam questions focus on matching business requirements to the correct streaming technology.

Practice Exam Questions

Question 1

A company needs to ingest streaming telemetry from thousands of IoT devices and route the data to multiple downstream consumers.

Which Fabric component should be used?

A. Data Activator
B. Eventstream
C. Pipeline
D. Notebook

Answer: B

Explanation:
Eventstream is specifically designed for real-time event ingestion and routing. Data Activator generates actions, pipelines handle batch orchestration, and notebooks perform transformations rather than ingestion.

Question 2

A solution requires advanced stream processing with custom Python code, joins against reference datasets, and machine learning inference.

Which technology should be selected?

A. Eventhouse
B. Spark Structured Streaming
C. KQL Database
D. Data Activator

Answer: B

Explanation:
Spark Structured Streaming supports complex transformations, enrichment, stateful processing, and machine learning integration through PySpark.

Question 3

A team needs extremely fast analytics over continuously arriving log data and plans to use KQL.

Which storage engine is most appropriate?

A. KQL Database
B. Dataflow Gen2
C. Warehouse
D. Pipeline

Answer: A

Explanation:
KQL databases are optimized for streaming ingestion, time-series analysis, and log analytics.

Question 4

A business user wants automatic notifications whenever inventory levels fall below a threshold.

Which Fabric component is best suited?

A. Eventstream
B. Notebook
C. Data Activator
D. Pipeline

Answer: C

Explanation:
Data Activator monitors data conditions and triggers automated actions such as alerts and notifications.

Question 5

Which Fabric component is primarily responsible for routing real-time events to destinations?

A. Warehouse
B. Eventstream
C. Dataflow Gen2
D. Notebook

Answer: B

Explanation:
Eventstream serves as the ingestion and routing layer for streaming architectures.

Question 6

A company requires an end-to-end platform for ingesting, storing, analyzing, and monitoring streaming events.

Which solution should be recommended?

A. Real-Time Intelligence
B. Dataflow Gen2
C. Warehouse
D. SQL Endpoint

Answer: A

Explanation:
Real-Time Intelligence combines ingestion, analytics, monitoring, alerting, and visualization capabilities into a unified platform.

Question 7

Which technology is best suited for analyzing application logs with time-series queries and low-latency reporting?

A. Notebook
B. Warehouse
C. Eventhouse
D. Pipeline

Answer: C

Explanation:
Eventhouse is optimized for streaming analytics, log analysis, and time-series workloads.

Question 8

A solution requires nightly ingestion of source data into a lakehouse.

Which option is most appropriate?

A. Eventstream
B. Data Activator
C. Eventhouse
D. Pipeline

Answer: D

Explanation:
Nightly ingestion is a batch process and is best handled through scheduled pipeline execution.

Question 9

A data engineer needs to continuously enrich streaming events using lookup data and perform custom business-rule calculations.

Which technology should be selected?

A. Spark Structured Streaming
B. Data Activator
C. Eventstream
D. Dashboard

Answer: A

Explanation:
Spark Structured Streaming provides advanced transformation capabilities including joins, aggregations, and custom code execution.

Question 10

Which statement best describes Eventhouse?

A. A workflow orchestration service for ETL processes
B. A low-code data preparation tool
C. A real-time analytics store optimized for event and telemetry data
D. A machine learning training environment

Answer: C

Explanation:
Eventhouse is designed for high-scale event ingestion, real-time analytics, log analytics, and KQL-based querying of streaming data.

Go to the DP-700 Exam Prep Hub main page.