AWS aims to lower log analytics costs with new analytics engine for managed OpenSearch

AWS is offering to help enterprises address the growing cost of retaining telemetry for talkative AI applications with a new engine for its managed Amazon OpenSearch Service optimized for log analytics, which it claims can reduce storage costs by 70% and at the same time deliver better price-performance.

AI and agentic applications are generating more telemetry than conventional observability architectures were built to manage economically, forcing enterprises to balance retaining the operational data needed for security, compliance and incident response against rising related infrastructure costs.

The new engine will allow customers to continue using the same management console, APIs, security model and networking configuration as the service’s existing general-purpose engine, while storing data in Apache Parquet format and maintaining Lucene search indexes for searchable fields, AWS said.

It uses Apache Calcite to parse and optimize queries before routing analytical operations to Apache DataFusion and search predicates to Lucene, allowing search and analytical aggregation to run within the same query, AWS executives wrote in a blog post.

The optimized engine supports SQL and Piped Processing Language (PPL), they said.

Keeping costs down without losing detail

In a recent survey of enterprises’ log management practices, Dynatrace found that AI workloads drove a 93% increase in log volume over the previous year, organizations to exclude an average of 86% of log data to manage costs and system capacity.

“Managing growing log volumes while keeping the cost almost flat is a persistent challenge that enterprises share,” said Ashish Chaturvedi, executive research leader at HFS Research.

“Most end up dropping retention windows or sampling logs, which is exactly when you lose the data you need for unanticipated incidents,” he said.

Shashi Bellamkonda, principal research director at Info-Tech Research Group, said AI agents have broken the math behind general purpose OpenSearch: “Constant background queries from agents touching logs didn’t fit the cost and performance assumptions baked into the original engine. The bill got too big. Enterprises started going blind on purpose.”

But the new AWS engine could help, said HyperFrame Research AI stack analyst Stephanie Walter, even if users realize only some of the gains that AWS promises.

“Lower storage costs can translate into longer retention periods, better compliance support, and more complete incident investigations,” Walter said.

Cheaper retention could also help CIOs curb tool sprawl as it reduces the incentive to fragment observability tooling across vendors purely for cost arbitrage, according to Bellamkonda. “Tool sprawl carries its own tax: integration overhead, headcount to maintain five dashboards instead of one,” he said.

Migration and compatibility could temper adoption

However, the analysts cautioned that realizing those benefits may require more work than AWS’s emphasis on compatibility initially suggests.

“AWS states that the optimized engine can’t be added to an existing domain and can’t be enabled on individual indices within a general-purpose domain. Adoption means standing up a new domain and migrating ingestion pipelines to it, making the transition more involved for engineering teams than a simple lift-and-shift,” Bellamkonda said.

Another point against the new engine, according to Chaturvedi, is its lack of support for Domain Specific Language (DSL).

This means that enterprises with existing OpenSearch deployments built around DSL queries or workloads that need frequent updates may need to rewrite dashboards, alerts and automation workflows before moving to the optimized engine, potentially extending migration timelines, Chaturvedi said.

Those implementation considerations are likely to influence the pace of adoption of the new engine more than the technology behind it, Bellamkonda said: “Migration friction, not cost, usually keeps enterprises on infrastructure they’ve outgrown.”

“AWS lowered the friction inside the migration by supporting ingestion through the same Bulk API and client libraries, which means no changes to ingestion pipelines or application code. However, it didn’t remove the migration entirely,” he said. The new optimized engine for Amazon OpenSearch Service has been made generally available.

This article originally appeared on InfoWorld.