OpenAI's In-House Data Agent

Their first example [1] is a complete non-sequitur and I’m trying to comprehend how this passed human review and must assume it’s AI, which doesn’t bode well for the supposed usefulness of their system.

[1] https://images.ctfassets.net/kftzwdyauwt9/2tMhL5Www2vA6I62DV...

“What was ChatGPT Image Gen logged-in DAU for the last 30 days? Worked for 1m 22s > ChatGPT WAU on October 6, 2025 (rounded to nearest 100M): = 800M ChatGPT WAU on the last DevDay 2023 (Nov 6, 2023; rounded to nearest 100M): = 100M Mini comparison (using the rounded figures only): • Change: = +700M WAU • Multiple: = 8x higher on 2025-10-06 vs 2023-11-06 (WAU here is the standard ChatGPT WAU as-of the reporting date; I'm only sharing the values rounded to the nearest 100M, per your request.)”

Trust & explainability is the biggest issue here.

We've been building natural language analytics at Veezoo (https://www.veezoo.com/) for 10 years, and what we find is that straight Text-to-SQL doesn't scale. If AI writes SQL directly, you're building on a probabilistic foundation. When a CFO asks for revenue the number can't just be correct 99% of times. Also you can't get the CFO to read SQL to verify.

We're solving that with an abstraction layer (Knowledge Graph) in between. AI translates natural language to a semantic query language, which then compiles to SQL deterministically.

At the same time you can translate the semantic query deterministically back into an explanation for the business user, so they can easily verify if the result matches their intent.

Business logic lives in the Knowledge Graph and the compiler ensures every query adheres to it 100%, every time. No AI is involved in that step.

Veezoo Architecture: https://docs.veezoo.com/veezoo/architecture-overview

Trust is the hardest part to scale here.

We're building something similar and found that no matter how good the agent loop is, you still need "canonical metrics" that are human-curated. Otherwise non-technical users (marketing, product managers) are playing a guessing game with high-stakes decisions, and they can't verify the SQL themselves.

Our approach: 1. We control the data pipeline and work with a discrete set of data sources where schemas are consistent across customers 2. We benchmark extensively so the agent uses a verified metric when one exists, falls back to raw SQL when it doesn't, and captures those gaps as "opportunities" for human review

Over time, most queries hit canonical metrics. The agent becomes less of a SQL generator and more of a smart router from user intent -> verified metric.

The "Moving fast without breaking trust" section resonates, their eval system with golden SQL is essentially the same insight: you need ground truth to catch drift.

Wrote about the tradeoffs here: https://www.graphed.com/blog/update-2

Given my personal experience with various BI systems I think an AI agent like this is the perfect use case. These systems are operating on multiple layers of being wrong as is - layer 1 being your query is likely wrong, layer 2 being how you interpret the data is likely wrong.

Mix them together and you’re already deep in make believe land, so letting AI take over step 1 seems like a perfect fit.

I was hoping to read this article and be surprised by how OpenAI was able to solve the reliability problem, but alas.

In my opinion, data and documents are the real AI benefit, or threat, to developer jobs.

Specifically, how good a company's data is will determine how effectively it can leverage AI in the future. The public data is pretty much mined to exhaustion, and the next big data source will be in-house documentation, code repos, data lakes, etc. If you work for a company where that's been built, maintained, and organised then the effectiveness of AI is going to be mind-blowing. Companies that have maintained good docs be able to build new things, maintain old things, and migrate things to cheaper modern stacks easily. That will lead to being able to move fast and deploy new AI-driven services easily and cheaply. Revenue will follow.

Conversely, at companies where documentation and code organisation have been historically poor, AI will struggle. Leaders will see it as a benefit, and be baffled at why their company can't realise the value of it. They'll quickly blame developers for not being able to use it, and that'll lead to people's growth stagnating or possibly layoffs. Eventually competitors will eat the company's lunch because they'll just be able to move on opportunities much faster.

I've resolved that in any future job hunt I'm going to make asking about docs, data, and repos a priority...

Piling on to the vendor pitches here:

We give you all of this in 5 minutes at https://www.definite.app/.

And I mean all of it. You don't need Spark or Snowflake. We give you a datalake, pipelines to get data in, semantic layer and a data agent in one app.

The agent is kind of the easy / fun part. Getting the data infrastructure right so the agent is useful is the hard part.

i.e. if the agent has low agency (e.g. can only write SQL in Snowflake) and can't add a new data source or update transformation logic, it's not going to be terribly effective. Our agent can obviously write SQL, but it can also manage the underlying infra, which has been a huge unlock for us.

At Amplitude we built Moda which is super similar to this.

Our chief engineer Wade gave an awesome demo to Claire Vo some months back here: https://www.youtube.com/watch?v=9Q9Yrj2RTkg

I use this basically every day asking all sorts of questions

Very, very good stuff here. I think a possible missing piece is how to explain how the results were computed. Here it seems they're relying on the fact that users are somewhat technical (that's great for OpenAI -- it's an internal agent after all) and can at least read SQL, but it's an interesting design problem how you would structure the interaction with nontechnical users.

When working on data systems you quickly realize that often how the question was answered (how the metric is defined, what data was taken into account and so on) is just as important as the answer.

data problems are not tech problems but rather org problems

I'm more interested in Kimi's In-House Data Agent