Your AI Analyst Needs to Name the Denominator

In the synthetic example below, an analyst reports that activation improved from 30% to 37.5%. The arithmetic checks out. Among users who had the full observation window, however, activation fell from 60% to 50%.

Both calculations use the same fixture. Their denominators answer different questions.

That is a useful test for an AI analyst: can it explain who counts before recommending what to do? If a team celebrates the first result without noticing the second, it could miss a reason to investigate onboarding. Neither result, by itself, proves that onboarding caused the change.

Faster analysis makes definitions more consequential

On September 10, OpenAI introduced the Data agent in ChatGPT Work, describing an interface for analyzing connected company data and producing interactive dashboards. Its announcement explicitly includes business terms and metric definitions drawn from semantic layers and trusted sources.

That is the right foundation. It also moves a product decision into the conversation: which definition did this answer use, and does it match what the person meant?

This is not a report of a failure in that product or a hands-on evaluation of it. The fixture is a proposed acceptance test for any product that turns a loose business question into an analysis people may act on.

My thesis: an AI analyst should expose the definition behind a consequential metric, and resolve material ambiguity before presenting a decision-ready conclusion. Showing the query helps, but it cannot settle an unstated business choice.

The same data can support opposite headlines

Suppose activation means completing a core task within seven days of signing up. We compare two acquisition periods at the end of each period. Some users have had seven full days to act; recent arrivals have not.

The following fixture is entirely synthetic. Each row is an aggregate of distinct users, not events. Mature and immature groups are disjoint. An activation is counted at most once, within seven days of signup and before that period's snapshot. The fixture deliberately assigns zero activations to the immature groups. That is an input, not an assumption to make about real users.

The first calculation describes activation observed so far across all signups. The second estimates seven-day activation among users whose windows have closed. They are not interchangeable versions of a single KPI.

The all-signup rate rises by 7.5 percentage points while the mature-cohort rate falls by 10 points. A changing mix of mature and immature users makes the difference possible. Recent users still have time to activate; treating their unfinished windows as final failures changes the question.

For a weekly onboarding review, I would use the mature-cohort definition and show the excluded population. For an operational view of current activity, the observed-so-far rate may be useful with that label. A causal claim about a redesign would need more evidence, including a suitable comparison and checks on cohort composition.

Give the answer a definition the reader can inspect

A team does not need a second metric catalogue inside the chat interface. It needs the answer to point to the existing definition and preserve it through follow-up questions.

For this fixture, a compact answer could read:

Seven-day activation among mature signups fell from 60% to 50%. This excludes 50 users in each period whose observation windows are incomplete. Across all signups, activation observed so far rose from 30% to 37.5%; that is a different measure.

Beside that answer, expose the metric's source and version, numerator, eligible population, observation window, and snapshot. In a real dataset, the definition should also settle identity deduplication, test-account exclusions, timezone, and late-arriving events. The fixture aggregates those details away; a production query cannot.

dbt's metric definitions make this concrete: ratio metrics name numerator and denominator metrics, with filters that can apply to either or both. The conversational layer should resolve to that governed object where one exists, rather than quietly reconstructing a plausible formula.

The product behavior depends on what is missing. Use the approved default when it clearly matches the request, and name it. Compare interpretations when the distinction is useful. Ask a narrow question when an unresolved choice could reverse the conclusion or cross the team's decision threshold: “Do you mean completed seven-day cohorts, or activation observed so far across all signups?”

Evaluate the follow-up, not only the first query

Asking more questions is not automatically safer. A user should not have to rebuild a well-defined metric through a long interview.

The August 30 TIDE-Bench research examines dependent clarification and changes in user intent in conversational text-to-SQL. In its constructed, simulator-based tasks, models that asked more did not reliably identify the dependent ambiguities. Recognizing a revised request also did not guarantee a correct revised query. These findings identify evaluation targets, not failure rates for deployed analytics products.

For a pilot, add this fixture to a small set of representative business questions. Test an explicit mature-cohort request, an ambiguous activation request, and a follow-up that switches to all signups. Review the selected definition, calculation, and answer label together. A changed filter with the old label should fail.

Also include unambiguous questions. Track unnecessary clarification and time to a usable answer alongside definition errors. Otherwise a system that refuses to answer anything can appear unusually reliable.

A semantic layer may already solve most of this

The strongest counterargument is practical: governed metrics and verified queries already exist. Why add another product mechanism?

I agree with the premise. Snowflake's Verified Query Repository pairs questions with reviewed SQL to guide similar requests. Use that work. Duplicating definitions would create another place for them to drift.

The remaining job is to handle an absent definition, a conflict, or a user's explicit change of scope. Make that boundary visible, then test whether the extra interaction earns its cost. I would simplify the proposed clarification policy if a held-out pilot showed no reduction in material definition errors, or if the review burden outweighed the errors avoided.

Start with one metric that drives a recurring decision. Put its approved definition beside an alternative that answers a different question. Make the analyst distinguish them, calculate both correctly, and carry a requested change through to the final label.

That complements the earlier argument in Adoption Is Not an AI Value Metric. Choosing a useful outcome is one decision. Making sure the analyst counted that outcome, for the intended population, is another.