Introducing the next generation of serverless time series for Amazon OpenSearch Service

• Autoscaling in seconds – Compute is added in line based on demand within seconds, up to 20 times faster than the previous architecture, so performance stays consistent during traffic surges.
• Scale to zero – When a workload goes idle using the default behavior of zero for minimum OpenSearch Compute Units (OCU), compute scales all the way down to zero after 10 minutes of no activity on your collection, so you stop paying for capacity you aren’t using.
• Minimum and maximum compute settings – Define the user experience on your own terms. If you have an ingest heavy workload, you can adjust the minimum ingest OCUs separately from your search needs for better ingest performance.
• Complete separation of compute from storage – Stateless OpenSearch Compute Units (OCUs) read from and write to a shared storage layer, so compute scales up and down independently of how much data you store.
• Data durability through S3-backed storage fronted by a fast storage layer – Data is persisted to highly durable Amazon Simple Storage Service (Amazon S3) the moment it is ingested, fronted by a fast layer, keeping it available independently of the compute that processes it.
• Shared compute across indices with different encryption schemes – With collection groups, you can share compute across collections while assigning different AWS Key Management Service (AWS KMS) keys, so you get cost efficiency with per-collection encryption isolation.

For search practitioners, these features change the equation for addressing the unpredictability of search-based workloads, whether that’s a large increase in indexing data or searches across the data used in your solutions.

For those who manage the budget and handle capacity planning, you no longer need to pre-plan peak capacity for your search infrastructure. Define the boundaries for maximum spend upfront while still providing a search foundation for your business that draws from a “pay for what you use” offering.

All these benefits provide the foundation for the next generation of time series collections for Amazon OpenSearch Service. If you’re not exploring this feature after reading this post, you might be missing out on a major improvement for your event analytics and observability needs.

Simplifying the way you manage data

Time series workloads are among the most demanding workloads you manage. This often leads to compromise to contain costs typically encountered with search technologies that don’t automatically scale. Historically, time series workloads have static capacity and require a lot of configuration and oversight from your teams. How do you plan for the peak? How long should the solution hold onto logs before running out of storage? How do you deal with accidental debug logs being turned on in production with fixed infrastructure?

With Amazon OpenSearch Serverless next generation of time series collections, you define the boundaries around cost, performance, security, access controls, and data retention with levers provided by both the next generation foundation and the new features introduced with time series collections. Let’s walk through the new controls.

Tiering data

For ingestion, a function critical to providing time series observability with low-latency response, the service provides a fast storage layer that supports ingesting terabytes of data per day while preserving data durability. For observability over time, data might be infrequently accessed. Low-cost storage can hold years of data at a price point acceptable for your business use case. This means you can address compliance needs or long-term retention policies without sacrificing access to older data because of cost.

A new index setting provides an attribute that tells the service when to transition the index data to an infrequent access tier. Keep in mind that, with every index within a serverless collection, the data for that index persists to Amazon S3 at the time of ingestion. Amazon S3 is a highly available and highly durable object store, so you can store and retain vast amounts of data at a low cost. Serverless collections take advantage of Amazon S3 for that durability and availability. Upon ingestion, a copy of the data additionally lives in a fast storage layer until you decide it no longer needs fast query performance as it ages. Regardless of which tier the data lives in, you pay for either low latency or infrequent access tiering. You don’t pay for both at the same time.

For example, if the index is scheduled to transition to warm after 7 days, the backing data transitions to infrequent access using the warm tier. If you query data that is in the warm tier, search latencies might increase. Keep in mind that warm data is considered infrequent access. It has higher search latency because the query interacts with data backed on Amazon S3.

Let’s take a look at the configuration for this feature. Our documentation provides some additional examples for you to consider.

PUT /app-logs
{
  "settings": {
    "index.warm.after": "7d"
  }
}

In the following architecture, you can visualize how data transitions from hot to warm.

Figure 1: Data transitions from the hot tier to the warm tier after 7 days while persisting to Amazon S3

The following flow applies:

• Ingestion stores a hot copy of the data in a low latency hot tier.
• Ingestion also persists the data to Amazon S3.
• When you query your data during searches, the hot data is fetched from the hot tier. If portions of the time series indices span the warm tier, that data is returned in the same response.
• After your index reaches 7 days of age, the hot tier data transitions to a warm state.

Determining time

With classic collections, time series events are automatically assigned an @timestamp field to manage order. With next-generation time series collections, you have complete control over which field is used for time. The @timestamp field remains the default time field. If you want to use a different field, you can override the default. You can either specify the index.timestamp_field attribute as an index setting at index creation, or you can declare it in an index template.

For any document to be successfully indexed, you must populate the desired time field in the document before ingestion. The service rejects documents individually if they have a missing field, a null field, or an invalid date value for a timestamp. If the documents are part of a _bulk API request, the non-conforming documents are marked as errors in the bulk activity. The bulk request still succeeds, with the document errors highlighted in the errors block of the response.

Let’s review the setting you use to override the default timestamp field.

PUT /app-logs
{
  "settings": {
    "index.timestamp_field": "event_time"
  }
}

In the following diagram, the effects of how you provide a time field at ingestion time determine if the document is ingested successfully.

Figure 2: How the time field provided at ingestion determines whether each document is indexed or rejected

There are four possible scenarios:

• Overridden timestamp field or default @timestamp field is invalid.
• Existence of any timestamp field is missing in the document.
• Default @timestamp exists in the document.
• A valid event_time override field is populated with a valid timestamp value in the document.

Managing data retention

Hot and warm tiers provide both performance benefits and cost savings, depending on how you configure the transitions from hot to warm. But keep in mind, your data is always stored on Amazon S3. The benefits of Amazon S3 for long-term retention help you address compliance needs and other directives determined by your business requirements or industry attestations. It also means you can keep your data around indefinitely and have it accessible for query (within reason) as needed. Data tiering settings only control the responsiveness and cost of the data. Retention is solely controlled by the data lifecycle policies for time series serverless collections annotated on each index.

When you have compliance-related workloads with years of data, you historically made tradeoffs on where you placed that data. Most organizations use snapshots or other immutable stores that give them the flexibility to stand up a forensic cluster with a limited scope of data for an audit. This was necessary because of the demands on storage costs. The compute needed to query long-term data while still handling day-to-day ingestion and search presented challenges to many organizations. With the next generation of time series collections, you add more search OCUs for the anticipated traffic, and the collection handles the rest. Operationally, this saves time and the cost of provisioning resources for compliance and auditing needs.

Unlike prior implementations of time series collections on OpenSearch Service where you configure lifecycle policies independently from the index, next generation time series collections places the responsibility of lifecycle on the index with a new setting.

Let’s review that setting so that you understand the shift of responsibility for next generation time series collections. In this example, an index template provides the means by which all indices matching a pattern can consume the same lifecycle policy.

PUT /_index_template/app-logs
{
  "index_patterns": ["app-logs*"],
  "template": {
    "settings": {
      "index.data.retention: "30"
    }
  }
}

When you couple data retention with data tiering for query performance using a seven-day hot tier and data retention for 30 days, you have granular control of your data for all your indexing and search needs.

Figure 3: Data tiering controls performance and cost while a separate retention policy governs removal, with data stored on Amazon S3

Transforming data into actions and insights

In November 2024, Amazon OpenSearch announced a new tier-one solution called the next-generation OpenSearch UI. OpenSearch UI is a comprehensive observability tool that spans multiple data sources, so you can gain insights from OpenSearch and other integrated sources in one place. What’s significant is that you can interface with multiple time series implementations, including next-generation time series collections, as you work through your transition from provisioned offerings to serverless solutions hosted by Amazon OpenSearch.

The Observability workspace is well suited for working with time series workloads. This workspace provides features such as alerting, anomaly detection, visualizations, and other functions such as piped processing language (PPL) features to help you optimize your time series operations.

Providing the foundation for modern discovery

Imagine insights delivered quickly through efficient query processing. Using the autoscaling capabilities of OpenSearch to augment your operations provides you with the foundation to modernize how you manage your infrastructure. OpenSearch Serverless can process vast datasets efficiently. However, that power comes with unpredictable, data-intensive queries that can scan vast amounts of data on the fly. That’s where the next generation of time series collections on Amazon OpenSearch Serverless shines, with autoscaling that flexes on demand to meet the needs of advanced solutions.

Summary

With the next generation of serverless time series collections for Amazon OpenSearch, you can stop compromising. Auto scale in seconds, scale to zero, full separation of compute and storage, Amazon S3-backed durability, and shared compute with per-collection encryption augment the most demanding workloads your teams manage. On top of that foundation, time series adds a small set of levers that simplify how you manage data:

• Data tiering lets each index own its lifecycle, keeping hot data fast while aging data moves to a cost-efficient warm tier, all without ever leaving Amazon S3.
• Time field control determines which field orders your events, with clear, per-document handling of invalid data.
• Data retention policies decouple how long you keep data from how fast you can query it, so compliance and long-term retention no longer force you to stand up forensic clusters or make difficult storage tradeoffs.

Pair those controls with the next-generation OpenSearch UI, and you have an observability platform that takes you from raw events to actionable insights. It provides the elastic, affordable foundation that modern discovery demands. This is logging without compromise: you define the boundaries for cost, performance, security, access, and retention, and the service handles the rest.

Try it out

Stop pre-planning for peak and start paying only for what you use. Create a next-generation time series collection in the Amazon OpenSearch Service console. It’s active in seconds! Point your existing ingestion pipelines and OpenSearch clients at it and set your tiering and retention levers to match your use case. Then launch OpenSearch UI using the Observability workspace and let the tooling help you build your first dashboards and investigations.