Lowering long-term logging bills by 4,800% with Amazon OpenSearch Service

Once you use Amazon OpenSearch Service for time-bound knowledge like server logs, service logs, utility logs, clickstreams, or occasion streams, storage value is among the major drivers for the general value of your answer. Over the past yr, OpenSearch Service has launched options which have opened up new potentialities for storing your log knowledge in numerous tiers, enabling you to commerce off knowledge latency, sturdiness, and availability. In October 2023, OpenSearch Service announced support for im4gn data nodes, with NVMe SSD storage of as much as 30 TB. In November 2023, OpenSearch Service introduced or1, the OpenSearch-optimized occasion household, which delivers as much as 30% price-performance enchancment over current situations in inner benchmarks and makes use of Amazon Simple Storage Service (Amazon S3) to supply 11 nines of sturdiness. Lastly, in Might 2024, OpenSearch Service introduced basic availability for Amazon OpenSearch Service zero-ETL integration with Amazon S3. These new options be part of OpenSearch’s existing UltraWarm instances, which give an as much as 90% discount in storage value per GB, and UltraWarm’s cold storage choice, which helps you to detach UltraWarm indexes and durably retailer not often accessed knowledge in Amazon S3.

This put up works by an instance that can assist you perceive the trade-offs accessible in value, latency, throughput, knowledge sturdiness and availability, retention, and knowledge entry, so as to select the fitting deployment to maximise the worth of your knowledge and reduce the associated fee.

Look at your necessities

When designing your logging answer, you want a transparent definition of your necessities as a prerequisite to creating good trade-offs. Rigorously look at your necessities for latency, sturdiness, availability, and price. Moreover, take into account which knowledge you select to ship to OpenSearch Service, how lengthy you keep knowledge, and the way you intend to entry that knowledge.

For the needs of this dialogue, we divide OpenSearch occasion storage into two courses: ephemeral backed storage and Amazon S3 backed storage. The ephemeral backed storage class contains OpenSearch nodes that use Nonvolatile Reminiscence Categorical SSDs (NVMe SSDs) and Amazon Elastic Block Store (Amazon EBS) volumes. The Amazon S3 backed storage class contains UltraWarm nodes, UltraWarm chilly storage, or1 situations, and Amazon S3 storage you entry with the service’s zero-ETL with Amazon S3. When designing your logging answer, take into account the next:

• Latency – in case you want leads to milliseconds, then you will need to use ephemeral backed storage. If seconds or minutes are acceptable, you’ll be able to decrease your value through the use of Amazon S3 backed storage.
• Throughput – As a basic rule, ephemeral backed storage situations will present greater throughput. Cases which have NVMe SSDs, just like the im4gn, typically present the perfect throughput, with EBS volumes offering good throughput. or1 situations benefit from Amazon EBS storage for major shards whereas utilizing Amazon S3 with segment replication to cut back the compute value of replication, thereby providing indexing throughput that may match and even exceed NVMe-based situations.
• Information sturdiness – Information saved within the sizzling tier (you deploy these as knowledge nodes) has the bottom latency, and likewise the bottom sturdiness. OpenSearch Service supplies automated restoration of information within the sizzling tier by replicas, which give sturdiness with added value. Information that OpenSearch shops in Amazon S3 (UltraWarm, UltraWarm chilly storage, zero-ETL with Amazon S3, and or1 situations) will get the advantage of 11 nines of sturdiness from Amazon S3.
• Information availability – Best practices dictate that you just use replicas for knowledge in ephemeral backed storage. When you have got a minimum of one reproduction, you’ll be able to proceed to entry your entire knowledge, even throughout a node failure. Nevertheless, every reproduction provides a a number of of value. In the event you can tolerate non permanent unavailability, you’ll be able to scale back replicas by or1 situations, with Amazon S3 backed storage.
• Retention – Information in all storage tiers incurs value. The longer you keep knowledge for evaluation, the extra cumulative value you incur for every GB of that knowledge. Determine the utmost period of time you will need to retain knowledge earlier than it loses all worth. In some circumstances, compliance necessities could prohibit your retention window.
• Information entry – Amazon S3 backed storage situations typically have a a lot greater storage to compute ratio, offering value financial savings however with inadequate compute for high-volume workloads. When you have excessive question quantity or your queries span a big quantity of information, ephemeral backed storage is the fitting selection. Direct question (Amazon S3 backed storage) is ideal for big quantity queries for occasionally queried knowledge.

As you take into account your necessities alongside these dimensions, your solutions will information your decisions for implementation. That can assist you make trade-offs, we work by an prolonged instance within the following sections.

OpenSearch Service value mannequin

To grasp how one can value an OpenSearch Service deployment, it is advisable perceive the associated fee dimensions. OpenSearch Service has two completely different deployment choices: managed clusters and serverless. This put up considers managed clusters solely, as a result of Amazon OpenSearch Serverless already tiers knowledge and manages storage for you. Once you use managed clusters, you configure knowledge nodes, UltraWarm nodes, and cluster supervisor nodes, deciding on Amazon Elastic Compute Cloud (Amazon EC2) occasion varieties for every of those features. OpenSearch Service deploys and manages these nodes for you, offering OpenSearch and OpenSearch Dashboards by a REST endpoint. You’ll be able to select Amazon EBS backed situations or situations with NVMe SSD drives. OpenSearch Service prices an hourly value for the situations in your managed cluster. In the event you select Amazon EBS backed situations, the service will cost you for the storage provisioned, and any provisioned IOPs you configure. In the event you select or1 nodes, UltraWarm nodes, or UltraWarm chilly storage, OpenSearch Service prices for the Amazon S3 storage consumed. Lastly, the service charges for data transferred out.

Instance use case

We use an instance use case to look at the trade-offs in value and efficiency. The fee and sizing of this instance are based mostly on greatest practices, and are directional in nature. Though you’ll be able to count on to see related financial savings, all workloads are distinctive and your precise prices could differ considerably from what we current on this put up.

For our use case, Fizzywig, a fictitious firm, is a big mushy drink producer. They’ve many vegetation for producing their drinks, with copious logging from their manufacturing line. They began out small, with an all-hot deployment and producing 10 GB of logs each day. Immediately, that has grown to three TB of log knowledge each day, and administration is mandating a discount in value. Fizzywig makes use of their log knowledge for occasion debugging and evaluation, in addition to historic evaluation over one yr of log knowledge. Let’s compute the price of storing and utilizing that knowledge in OpenSearch Service.

Ephemeral backed storage deployments

Fizzywig’s present deployment is 189 r6g.12xlarge.search knowledge nodes (no UltraWarm tier), with ephemeral backed storage. Once you index knowledge in OpenSearch Service, OpenSearch builds and shops index knowledge buildings which are normally about 10% bigger than the supply knowledge, and it is advisable depart 25% free space for storing for working overhead. Three TB of each day supply knowledge will use 4.125 TB of storage for the primary (major) copy, together with overhead. Fizzywig follows greatest practices, utilizing two reproduction copies for optimum knowledge sturdiness and availability, with the OpenSearch Service Multi-AZ with Standby choice, growing the storage have to 12.375 TB per day. To retailer 1 yr of information, multiply by 12 months to get 4.5 PB of storage wanted.

To provision this a lot storage, they may additionally select im4gn.16xlarge.search situations, or or1.16.xlarge.search situations. The next desk provides the occasion counts for every of those occasion varieties, and with one, two, or three copies of the info.

The previous desk and the next dialogue are strictly based mostly on storage wants. or1 situations and im4gn situations each present greater throughput than r6g situations, which can scale back value additional. The quantity of compute saved varies between 10–40% relying on the workload and the occasion kind. These financial savings don’t move straight by to the underside line; they require scaling and modification of the index and shard technique to completely notice them. The previous desk and subsequent calculations take the final assumption that these deployments are over-provisioned on compute, and are storage-bound. You’ll see extra financial savings for or1 and im4gn, in contrast with r6g, in case you needed to scale greater for compute.

The next desk represents the whole cluster prices for the three completely different occasion varieties throughout the three completely different knowledge storage sizes specified. These are based mostly on on-demand US East (N. Virginia) AWS Region costs and embrace occasion hours, Amazon S3 value for the or1 situations, and Amazon EBS storage prices for the or1 and r6g situations.

This desk provides you the one-copy, two-copy, and three-copy prices (together with Amazon S3 and Amazon EBS prices, the place relevant) for this 4.5 PB workload. For this put up, “one copy” refers back to the first copy of your knowledge, with the replication issue set to zero. “Two copies” features a reproduction copy of all the knowledge, and “three copies” features a major and two replicas. As you’ll be able to see, every reproduction provides a a number of of value to the answer. After all, every reproduction provides availability and sturdiness to the info. With one copy (major solely), you’ll lose knowledge within the case of a single node outage (with an exception for or1 situations). With one reproduction, you would possibly lose some or all knowledge in a two-node outage. With two replicas, you may lose knowledge solely in a three-node outage.

The or1 situations are an exception to this rule. or1 situations can help a one-copy deployment. These situations use Amazon S3 as a backing retailer, writing all index knowledge to Amazon S3, as a way of replication, and for sturdiness. As a result of all acknowledged writes are endured in Amazon S3, you’ll be able to run with a single copy, however with the danger of dropping availability of your knowledge in case of a node outage. If an information node turns into unavailable, any impacted indexes can be unavailable (purple) through the restoration window (normally 10–20 minutes). Rigorously consider whether or not you’ll be able to tolerate this unavailability together with your clients in addition to your system (for instance, your ingestion pipeline buffer). In that case, you’ll be able to drop your value from $14 million to $4.7 million based mostly on the one-copy (major) column illustrated within the previous desk.

Reserved Cases

OpenSearch Service helps Reserved Cases (RIs), with 1-year and 3-year phrases, with no up-front value (NURI), partial up-front value (PURI), or all up-front value (AURI). All reserved occasion commitments decrease value, with 3-year, all up-front RIs offering the deepest low cost. Making use of a 3-year AURI low cost, annual prices for Fizzywig’s workload provides prices as proven within the following desk.

RIs present an easy technique to save value, with no code or structure modifications. Adopting RIs for this workload brings the im4gn value for 3 copies right down to $5.7 million, and the one-copy value for or1 situations right down to $3.2 million.

Amazon S3 backed storage deployments

The previous deployments are helpful as a baseline and for comparability. If truth be told, you’ll select one of many Amazon S3 backed storage choices to maintain prices manageable.

OpenSearch Service UltraWarm situations retailer all knowledge in Amazon S3, utilizing UltraWarm nodes as a sizzling cache on high of this full dataset. UltraWarm works greatest for interactive querying of information in small time-bound slices, equivalent to working a number of queries in opposition to 1 day of information from 6 months in the past. Consider your entry patterns rigorously and take into account whether or not UltraWarm’s cache-like conduct will serve you properly. UltraWarm first-query latency scales with the quantity of information it is advisable question.

When designing an OpenSearch Service area for UltraWarm, it is advisable determine in your sizzling retention window and your heat retention window. Most OpenSearch Service clients use a sizzling retention window that varies between 7–14 days, with heat retention making up the remainder of the total retention interval. For our Fizzywig situation, we use 14 days sizzling retention and 351 days of UltraWarm retention. We additionally use a two-copy (major and one reproduction) deployment within the sizzling tier.

The 14-day, sizzling storage want (based mostly on a each day ingestion charge of 4.125 TB) is 115.5 TB. You’ll be able to deploy six situations of any of the three occasion varieties to help this indexing and storage. UltraWarm shops a single reproduction in Amazon S3, and doesn’t want further storage overhead, making your 351-day storage want 1.158 PiB. You’ll be able to help this with 58 UltraWarm1.massive.search situations. The next desk provides the whole value for this deployment, with 3-year AURIs for the new tier. The or1 situations’ Amazon S3 value is rolled into the S3 column.

You’ll be able to additional scale back the associated fee by transferring knowledge to UltraWarm chilly storage. Chilly storage reduces value by decreasing availability of the info—to question the info, you will need to problem an API name to reattach the goal indexes to the UltraWarm tier. A typical sample for 1 yr of information retains 14 days sizzling, 76 days in UltraWarm, and 275 days in chilly storage. Following this sample, you employ 6 sizzling nodes and 13 UltraWarm1.massive.search nodes. The next desk illustrates the associated fee to run Fizzywig’s 3 TB each day workload. The or1 value for Amazon S3 utilization is rolled into the UltraWarm nodes + S3 column.

By using Amazon S3 backed storage choices, you’re capable of scale back value even additional, with a single-copy or1 deployment at $337,000, and a most of $1 million yearly with or1 situations.

OpenSearch Service zero-ETL for Amazon S3

Once you use OpenSearch Service zero-ETL for Amazon S3, you retain all of your secondary and older knowledge in Amazon S3. Secondary knowledge is the higher-volume knowledge that has decrease worth for direct inspection, equivalent to VPC Movement Logs and WAF logs. For these deployments, you retain nearly all of occasionally queried knowledge in Amazon S3, and solely the latest knowledge in your sizzling tier. In some circumstances, you pattern your secondary knowledge, retaining a proportion within the sizzling tier as properly. Fizzywig decides that they need to have 7 days of all of their knowledge within the sizzling tier. They’ll entry the remaining with direct question (DQ).

Once you use direct question, you’ll be able to retailer your knowledge in JSON, Parquet, and CSV codecs. Parquet format is perfect for direct question and supplies about 75% compression on the info. Fizzywig is utilizing Amazon OpenSearch Ingestion, which might write Parquet format knowledge on to Amazon S3. Their 3 TB of each day supply knowledge compresses to 750 GB of each day Parquet knowledge. OpenSearch Service maintains a pool of compute models for direct question. You’re billed hourly for these OpenSearch Compute Models (OCUs), scaling based mostly on the quantity of information you entry. For this dialog, we assume that Fizzywig could have some debugging classes and run 50 queries each day over at some point price of information (750 GB). The next desk summarizes the annual value to run Fizzywig’s 3 TB each day workload, 7 days sizzling, 358 days in Amazon S3.

That’s fairly a journey! Fizzywig’s value for logging has come down from as excessive as $14 million yearly to as little as $288,000 yearly utilizing direct question with zero-ETL from Amazon S3. That’s a financial savings of 4,800%!

Sampling and compression

On this put up, now we have checked out one knowledge footprint to allow you to deal with knowledge dimension, and the trade-offs you can also make relying on the way you need to entry that knowledge. OpenSearch has further options that may additional change the economics by decreasing the quantity of information you retailer.

For logs workloads, you’ll be able to make use of OpenSearch Ingestion sampling to reduce the size of data you send to OpenSearch Service. Sampling is acceptable when your knowledge as an entire has statistical traits the place an element may be consultant of the entire. For instance, in case you’re working an observability workload, you’ll be able to typically ship as little as 10% of your knowledge to get a consultant sampling of the traces of request dealing with in your system.

You’ll be able to additional make use of a compression algorithm in your workloads. OpenSearch Service lately launched help for Zstandard (zstd) compression that may carry greater compression charges and decrease decompression latencies as in comparison with the default, greatest compression.

Conclusion

With OpenSearch Service, Fizzywig was capable of stability value, latency, throughput, sturdiness and availability, knowledge retention, and most popular entry patterns. They had been capable of save 4,800% for his or her logging answer, and administration was thrilled.

Throughout the board, im4gn comes out with the bottom absolute greenback quantities. Nevertheless, there are a few caveats. First, or1 situations can present greater throughput, particularly for write-intensive workloads. This will imply further financial savings by decreased want for compute. Moreover, with or1’s added sturdiness, you’ll be able to preserve availability and sturdiness with decrease replication, and due to this fact decrease value. One other issue to contemplate is RAM; the r6g situations present further RAM, which accelerates queries for decrease latency. When coupled with UltraWarm, and with completely different sizzling/heat/chilly ratios, r6g situations can be a superb selection.

Do you have got a high-volume, logging workload? Have you ever benefitted from some or all of those strategies? Tell us!

Concerning the Writer

Jon Handler is a Senior Principal Options Architect at Amazon Internet Companies based mostly in Palo Alto, CA. Jon works intently with OpenSearch and Amazon OpenSearch Service, offering assist and steering to a broad vary of shoppers who’ve vector, search, and log analytics workloads that they need to transfer to the AWS Cloud. Previous to becoming a member of AWS, Jon’s profession as a software program developer included 4 years of coding a large-scale, ecommerce search engine. Jon holds a Bachelor’s of the Arts from the College of Pennsylvania, and a Grasp’s of Science and a PhD in Laptop Science and Synthetic Intelligence from Northwestern College.