Announcing General Availability of Real-Time Mode for Spark Declarative Pipelines on Databricks

We introduced Real-Time Mode (RTM) last year that enabled customers to build low-latency, operational use cases. This was a game changer because it allowed customers to process streaming data in milliseconds, up to 100x faster than traditional micro-batch architecture. Furthermore, they no longer needed to maintain separate, specialized engines like Flink alongside Spark, eliminating duplicate codebases, fragmented governance models, and operational overhead.

Today, we are announcing RTM in Spark Declarative Pipelines (SDP) is now Genearally Available. SDP has become the standard way to build streaming and ETL pipelines on Databricks, automatically handling dependency resolution, incremental processing, orchestration, failure recovery, and data quality enforcement. With RTM, customers can build operational, low-latency pipelines while retaining all of those SDP benefits. Whether for fraud scoring, real-time alerting, or pushing fresh features to operational stores, you can now use SDP across all of these use cases.

What Is Real-Time Mode (RTM)?

RTM is an evolution of the Spark Structured Streaming engine that achieves end-to-end latencies in the millisecond range. It does this by streaming data continuously, running all query stages simultaneously, and passing data between stages as soon as it is produced. For the full architecture deep dive, read the blog, “Breaking the microbatch barrier: The architecture of Apache Spark Real-Time Mode.”

RTM can support any workload that benefits from turning data into decisions in milliseconds. Some example use cases include fraud detection, live personalization, real-time recommendation systems, and real-time feature computation. We benchmarked RTM against Apache Flink on actual customer workloads performing feature computation, and RTM is up to 92% faster than specialized engines like Flink. Below are the benchmarks. For more information on the datasets and queries referenced, see this GitHub repository.

Why RTM on Spark Declarative Pipelines

Customers are increasingly defaulting to SDP for their ETL workloads. With RTM, SDP now supports all streaming workloads, including operational low-latency streaming, enabling customers to use a single framework across all pipelines. One framework means one observability stack, one billing model, and no fragmentation across different APIs.

SDP serverless takes it further. There is no compute to choose and no cluster to size. You get sub-second latency, without any complex infra tuning, while paying only for the cores your query uses. RTM flows also get SDP features such as versionless upgrades and low-downtime maintenance, which reduce both the operational work and the disruption of upgrades.

One use case where RTM on SDP has immediate impact is real-time feature computation. ML models for fraud detection or personalization depend on fresh signals, such as a user's total transaction amount over the last 10 minutes, to make accurate predictions. Computing these rolling-window aggregations and landing them in an online store within milliseconds is where RTM on SDP closes the gap. The Databricks Feature Store uses this exact architecture to deliver end-to-end p99 latency of 200ms from Kafka to the online feature store.

Getting started with RTM on SDP

Step 1: Configure the pipeline

Real-Time Mode requires the pipeline mode to be continuous and spark.databricks.streaming.realTimeMode.enabled set to true takes over.

Step 2: Define a real-time update flow

Real-time mode requires an update flow. Use dp.create_sink() to define the output target, then use the @dp.update_flow decorator with pipelines.trigger set to "RealTime" and target pointing to the sink.

For supported sources and sinks, see the documentation.

Writing features to any custom sink

Use cases like fraud detection or real-time personalization require not only that features be computed in real time but also that they land in a queryable, low-latency store. A fraud model needs a customer's latest transaction before it clears, and a personalization engine needs to read what a user just browsed.

Today, you can write to any custom sink like Lakebase, Redis, or any other OLTP store from SDP using a ForEach sink. Define the writer, register it as a foreach_sink, and wire it to an update_flow.

Features land in your operational store within milliseconds of being produced, with no offline-to-online sync in between. Applications and models can query them at ultra-low latency. What’s next for RTM on serverless?

• Running RTM pipelines at lower cost: Bin-pack multiple RTM flows into a single pipeline instead of each query standing up its own resources. The system also auto-tunes source and shuffle partitions based on workload, so users skip manual sizing.
• Expanded sink support: Native Lakebase sink for SDP, with built-in batching, automatic retries, and UC-managed authentication.

Conclusion

RTM extends Spark Declarative Pipelines into a new class of workloads, operational, latency-sensitive applications that demand immediate response to streaming data. By bringing sub-second latency to the SDP APIs your team already uses, it eliminates the need to operate a separate framework or engine for your most time-critical pipelines. Whether you're building fraud detection pipelines, personalization engines, or ML feature computation systems, real-time mode gives you the latency your application demands with the simplicity of SDP.

Real-Time Mode on Spark Declarative Pipelines is now Generally Available.

Technical Resources

Check out the following resources to get started with RTM on SDP:

• Documentation: Use Real-Time Mode on Lakeflow pipelines
• Documentation: Connect to Lakebase streaming sink
• Documentation: Use a ForEach sink to write each row in pipelines
• Code examples: Real-Time Mode on SDP examples