How Writer Keeps Its AI Infrastructure Running with Snowflake Hybrid Tables

Aeril Xie, Data Engineer at Writer

At Writer, data is at the center of everything we do.

As an enterprise AI platform, we route every customer request through LLM inference providers. Every one of those calls generates a trace record: the model called, the prompt, the response, latency, token counts, and outcome. Rather than running a separate observability database alongside our data warehouse, we made Snowflake the destination for raw trace writes too, so our engineers and analysts can query everything in one place.

My job as a data engineer is to make sure that data lands reliably, because it powers our QA processes and gives us the visibility we need into how our models are behaving in production. At the scale we operate, “reliably” is harder than it sounds.

The Problem: Lock Contention at Scale

Every Writer customer interaction that touches an LLM generates a row. At enterprise scale, that’s not a batch job you can schedule. It’s a continuous stream that doesn’t pause.

Our LLM tracing pipeline writes to Snowflake at high velocity: up to 600 rows per minute at peak, and roughly 1.3 million rows per day for our primary traces table. For a while, we were directing this traffic to a standard Snowflake table. And for a while, it mostly worked.

Then it didn’t.

We started seeing write failures from lock contention — too many concurrent writes competing for the same table resource. Standard Snowflake tables aren’t designed for that kind of continuous, high-concurrency write pattern. At our peak traffic levels, INSERT latency at P50 was reaching 2,362 milliseconds. Failure rates climbed to 45% of all insertions.

Every failed write was a lost data point. Initially the data wasn’t business-critical, so the failures were tolerable. But as this tracing data became the foundation of our observability pipeline and QA processes, the gap it left in our data got harder to ignore.

Evaluating the Options

When we opened a support ticket with Snowflake to investigate the failures, our Snowflake partner Joe introduced us to Hybrid Tables. We also briefly considered two alternatives: writing the tracing data directly from our API to a Postgres table (the traditional transactional approach), and staging data in S3 before ingesting it into Snowflake.

Both alternatives would have solved the write problem. But both would have added something we didn’t want: another system to manage, another pipeline to maintain, another failure surface. The whole point of centralizing on Snowflake was to avoid that kind of complexity. We wanted our trace data joinable against the rest of our data without ETL, queryable by the same analysts who work in Snowflake every day.

Hybrid Tables offered a different path. They’re designed for high-concurrency write workloads, use row-oriented storage with row-level locking, and coexist natively with standard Snowflake tables. That means no data movement and no additional infrastructure. The approach was simpler, and simpler architectures are easier to maintain with a lean data engineering team.

Migration: A Table Swap

The migration was straightforward. Our backend already had API integrations writing directly to a Snowflake table as the destination. Switching to Hybrid Tables meant two steps: create the hybrid table with the appropriate schema and primary key constraint, and update the target table reference on the API side. That was essentially all.

We also did a one-time historical data dump from the original standard table into the new hybrid table to preserve continuity. The process was seamless, with no meaningful disruption downstream.

The Architecture: Hybrid for Writes, Standard for Analytics

As the hybrid table grew — past 33 million rows, approaching 40 million — we started to notice that analytic queries running directly against it were getting slower. This makes sense since hybrid tables use a row-store engine optimized for fast single-row access by primary key. Full-table scans, aggregations, and complex JOINs aren’t what they’re designed for. As the table grew, our reconciliation queries started timing out.

Our solution was to separate the write and read workloads architecturally. The hybrid table remains the write destination, so all high-volume API traffic lands there. A Snowflake Task runs on a short cadence, incrementally copying new rows into a standard Snowflake table clustered by timestamp. All analytic and reconciliation queries are re-pointed to that standard table, where micro-partition pruning, columnar aggregation, and the Snowflake result cache can do their jobs properly.

This is the pattern we landed on: Hybrid Tables do what they do best — absorb fast, concurrent writes reliably — while the standard table gives analysts and downstream processes the full power of Snowflake’s columnar engine. The two workloads coexist in the same database, without external infrastructure or complex ETL.

Results

The impact was immediate. After switching to Hybrid Tables:

• INSERT latency dropped ~15x: P50 went from 2,362ms to 154ms.
• Write failures went from ~45% to essentially zero: We went from losing nearly half our writes to running for two to three months with near-zero errors, typically 3 to 10 failures per day at most, well within acceptable bounds.
• Warehouse compute shrank: We were able to downsize the warehouse powering our ingest pipeline from medium to extra small, while handling the same or higher write traffic without contention.

The moment we switched, the errors disappeared. That was the clearest possible signal that we’d made the right call.

Beyond performance, the more meaningful outcome has been data quality. Our tracing data now lands reliably in Snowflake, which means our QA processes are operating on a complete, accurate dataset for the first time. The observability pipeline we’ve built on top of this data is trustworthy in a way it wasn’t before.

What’s Next

Success with this use case has opened up new thinking about how we can use Hybrid Tables beyond tracing. We have high-volume logging data from our API that currently routes to a third-party streaming service. If Hybrid Tables can serve as a stable destination for that traffic, and the evidence so far suggests they can, it would let us consolidate more of our infrastructure inside Snowflake, reduce dependencies, and simplify operations further.

Writer runs on AI, and AI runs on data. When that data doesn’t land reliably, everything downstream suffers, from model observability to QA to the quality of what we ship to customers.

Hybrid Tables solved a problem we had been working around: high-frequency, high-concurrency writes at the scale our LLM infrastructure generates. They did it without adding architectural complexity and without pulling data out of Snowflake. And the migration was simple enough that it didn’t slow us down or require a dedicated effort to execute.

We’re early in exploring the full potential of this capability, but the foundation is solid, and we’re just getting started.

Writer is an enterprise AI platform that helps Fortune 500 companies deploy AI across their business operations. Writer’s full-stack platform — including its foundation models, AI applications, and Palmyra LLMs — is designed for enterprise-grade reliability, security, and performance.

How Writer Keeps Its AI Infrastructure Running with Snowflake Hybrid Tables was originally published in Snowflake Builders Blog: Data Engineers, App Developers, AI, & Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.