Anthropic’s data science team asked Claude, the company’s AI assistant, real questions about Anthropic’s own business numbers. When the team checked each answer against the correct figure, about 21 percent were right. The team then built one fix, a single official definition for every number the business tracks, stored in one place and checked before every answer. On the same questions, Claude’s share of right answers climbed above 95 percent, and closer to 99 percent in some parts of the business.
A shared definition works like a translator between a company’s raw data and anything that reads it, from dashboards to AI agents. It is known as a semantic layer. A business term goes in, and one official number comes back out, whichever system is asking. Without it, each team and each agent works out its own version of a figure like revenue or active users. One dashboard might count free trial signups as active users while another leaves them out. Neither is wrong by the rules its builder used, so two dashboards show two different numbers for the same question.
Without one official definition to check, a general AI model asked a business question still answers. Faced with a warehouse full of possible tables, it picks a table, a filter and a date range by matching patterns, with no company rule behind the choice. The answer often reads as confident and precise while resting on a definition no one agreed to. A reader has no way to tell a right answer from a merely plausible one.
The team’s fix takes that choice away. Asked about weekly active users, Claude looks up one official number instead of inventing one, the same number every dashboard and team in the company already uses. The team credits that check for the jump in its own testing. The model did not change, and the questions did not get easier.
Even with the definition in place, Claude still misses roughly one question in twenty. Most of those misses, the team says, trace back to missing or outdated documentation of the data, not to Claude’s judgment about which number to trust.
