Rapidly declining AI inference costs are ushering in an era of near-free intelligence. This perspective article explores the implications for data systems, arguing that agents will become the dominant workload, requiring a redesign of data systems for and of agents.
SynthePulse Insight · AI deep reading
Version 1 · 1 source
Inference costs are plummeting, making intelligence nearly free. Data systems face a triple transformation: designed for agents, managed by agents, and built by agents.
In early 2023, GPT-4-level inference cost about $30 per million tokens; by 2026, the same capability costs under $1, with some vendors pushing below $0.10. Benchmarks show inference prices dropping 9x to 900x per year, with a median around 50x. Frontier models see significant cost reductions each generation, and open-source models follow closely. While 'Nobel-level genius intelligence' has not arrived, intelligence sufficient for the vast majority of knowledge work is now widespread and becoming cheaper by the month.
Agents query databases differently from humans or BI tools: they engage in 'agentic speculation'—generating a diverse stream of heterogeneous queries, including schema introspection, column exploration, partial queries, and full queries. Each user request may trigger thousands of SQL queries. Experiments on text-to-SQL benchmarks show that when multiple agents attempt the same task, only 10-20% of subplans are unique, meaning 80-90% of subqueries are redundant. While redundancy helps improve task success rates, it is wasteful from the data system's perspective.
Data systems can reuse results of overlapping subplans, leveraging existing techniques like multi-query optimization and shared scans; they can also return approximate answers to help agents move forward quickly. Additionally, data systems can support batch queries and higher-level primitives (e.g., DBT-style Jinja macros) rather than requiring agents to list SQL one by one. More importantly, data systems can proactively guide agents: providing latency estimates, precomputed views, or results from related queries, because agents can accept arbitrary text feedback without requiring strict SQL results.
Agents need persistent state, long-term task management, coordination, and fault tolerance. Currently, agent 'memory' primarily relies on the file system: writing information to unstructured Markdown files and retrieving via grep or embeddings. Many believe the solution for continuous learning is to have agents consume large amounts of data (e.g., entire codebases, Slack, company wikis), then write learned knowledge into Markdown files for retrieval on demand.
However, when agents handle the vast majority of knowledge work, this approach will break. Limited context windows cannot accommodate all relevant snippets, and even if context windows continue to grow, stuffing all information into context introduces latency issues. Knowledge graphs similarly lack structured search capabilities and face analogous limitations.
Agents can already synthesize entire data systems in one shot, meaning custom systems can be rebuilt for each new workload. But the core challenge is verifying that these systems match expected behavior. How to make agents synthesize trustworthy data systems is an urgent problem to solve.
This article is based on the views of UC Berkeley Associate Professor Aditya G. Parameswaran and his collaborators. Some research directions (e.g., agentic speculation, structured memory, synthesized custom data systems) involve the authors' own ongoing work and represent forward-looking analysis, not confirmed facts.
The sharp decline in inference costs makes large-scale agent deployment feasible. Data systems must transform from passive executors to active collaborators, while addressing fundamental issues of agent memory, coordination, and system verification.
Primary report
Primary source