Introduction to Data Engineering Concepts |5| Streaming Data Fundamentals

Free Resources

• Free Apache Iceberg Course
• Free Copy of “Apache Iceberg: The Definitive Guide”
• Free Copy of “Apache Polaris: The Definitive Guide”
• 2025 Apache Iceberg Architecture Guide
• How to Join the Iceberg Community
• Iceberg Lakehouse Engineering Video Playlist
• Ultimate Apache Iceberg Resource Guide

In contrast to batch processing, where data is collected and processed in chunks, streaming data processing deals with data in motion. Instead of waiting for data to accumulate before running transformations, streaming pipelines ingest and process each piece of data as it arrives. This model enables organizations to respond to events in real time, a capability that’s becoming increasingly essential in domains like finance, security, and customer experience.

In this post, we’ll unpack the core ideas behind streaming, how it works in practice, and the challenges it presents compared to traditional batch systems.

What is Streaming Data?

Streaming data refers to data that is continuously generated by various sources—website clicks, IoT sensors, user interactions, system logs—and transmitted in real time or near-real time. This data typically arrives in small payloads, often as individual events, and needs to be processed with minimal delay.

The goal of a streaming pipeline is to capture this data as it’s generated, perform necessary transformations, and deliver it to its destination with as little latency as possible.

A simple example would be a ride-sharing app that tracks vehicle locations in real time. As each car moves, GPS data is streamed to a backend system that updates the user interface and helps dispatch rides based on current conditions.

How Streaming Systems Work

Unlike batch jobs that execute on a schedule, streaming systems run continuously. They consume data from a source, process it incrementally, and push it to a sink—all without waiting for a dataset to be complete.

At the heart of a streaming system is a message broker or event queue, which acts as a buffer between data producers and consumers. Apache Kafka is a popular choice here. It allows producers to publish events to topics, and consumers to read from those topics independently, often with strong guarantees around ordering and durability.

Once events are ingested, a processing engine takes over. Tools like Apache Flink, Spark Structured Streaming, and Apache Beam allow developers to apply transformations on a per-record basis or over time-based windows. This is where operations like filtering, aggregating, joining, and enriching occur.

These transformations must be designed to handle data that may arrive late, out of order, or in bursts. As such, streaming systems often implement complex logic to manage time—distinguishing between event time (when the event occurred) and processing time (when it was received)—to ensure results are accurate.

Use Cases and Business Impact

The appeal of streaming pipelines lies in their ability to power real-time applications. Fraud detection systems can flag suspicious transactions as they happen. E-commerce platforms can recommend products based on live browsing behavior. Logistics companies can monitor fleet activity and adjust routes on the fly.

In operational analytics, dashboards fed by streaming data provide up-to-the-minute visibility, allowing teams to make informed decisions in response to changing conditions.

Streaming is also a foundational component of event-driven architectures. When services communicate via events, streaming systems act as the glue that ties the application together, enabling asynchronous, decoupled interactions.

Challenges in Streaming Systems

Despite its power, streaming introduces complexity that shouldn’t be underestimated. Handling late or out-of-order data is a major concern. If an event shows up ten minutes after it was supposed to be processed, the system must be smart enough to either incorporate it correctly or account for the gap.

State management is another critical factor. When a pipeline needs to remember information across multiple events—like keeping a running total or maintaining a session—it must manage that state reliably, often across distributed systems.

There’s also the issue of fault tolerance. Streaming systems must be able to recover from crashes or network issues without duplicating results or losing data. This requires sophisticated checkpointing, replay, and exactly-once processing semantics, which tools like Flink and Beam are designed to provide.

Finally, testing and debugging streaming pipelines can be more difficult than batch jobs. Because they run continuously and deal with time-sensitive data, reproducing issues often requires specialized tooling or replay mechanisms.

When to Choose Streaming

Streaming makes sense when low-latency data processing is essential to the business. This could mean operational decision-making, customer experience personalization, or complex event processing in a microservices architecture.

It’s not always the right tool for the job, though. For workloads that don’t require immediate insights—or where simplicity and reliability matter more—batch processing remains the better choice.

As data engineers, the key is to understand the trade-offs and choose the right pattern for each use case.

In the next post, we’ll shift gears and look at how data is modeled for analytics. Understanding the differences between OLTP and OLAP systems, as well as the pros and cons of different schema designs, is critical to building pipelines that serve real business needs.