# Why AI gets your company's numbers wrong | Silicon

URL: https://meetsilicon.com/blog/why-ai-gets-your-company-numbers-wrong/

> AI tools give confident answers about company data, but they guess what your columns mean. What a 2026 benchmark found, and the first step any startup can take.

Guides Why AI gets your company's numbers wrong
Ask an AI tool how many active customers you had last quarter and you will get an answer, usually with a chart and a confident sentence. What it rarely tells you is which definition of "active customer"it used, and whether that matches the one finance put in the board deck.
That gap is the main reason AI answers about company data go wrong. The model usually writes valid SQL but misreads what your columns mean.
## What the benchmark found
In April 2026, dbt Labs updated a benchmark it first ran in 2023 (dbt Labs). The team asked 11 business questions of an insurance dataset, 20 times each, with current AI models, and compared two approaches:
- The AI writes the query itself after reading the table structure (often called text-to-SQL).
- The AI uses metrics someone has already defined, such as "active customer"or "net revenue", and software writes the query. That set of definitions is called a semantic layer.
The test data had been cleaned up and documented first. On it, the AI writing its own queries was right 90.0% of the time with the better of the two leading models. Answering from defined metrics, the same model was right 98.2% of the time, and the other reached 100%.
The failures differed too. When the AI wrote its own query and got it wrong, it returned a plausible wrong number, which can reach a board deck unchallenged. When it worked from defined metrics and could not answer, it returned an error, which prompts someone to ask why.
Two caveats. dbt Labs sells a semantic layer, and the benchmark covers 11 questions on one dataset. The benchmark is open source, though, so anyone can rerun it.
## Why the model guesses
Most company data has no documentation explaining what its columns mean. A status column might hold five values, two of which mean a customer has churned. Revenue appears in billing, in the CRM and in accounting software, and each system records a different event (Signld). Someone who has worked at the company for a year knows which version to use. A model seeing the tables for the first time does not, so it picks the most plausible one.
Newer models help. In the same benchmark, the AI writing its own queries answered 32.7% of all questions correctly in 2023 and 64.5% in 2026. Models still can't supply definitions nobody wrote down. Those live with the people who run each team.
## What fixes it
Write the definitions down once, in a place both people and AI tools can read. In practice, that means:
- Each key metric is defined in one place, and the team that owns it agrees how it is calculated.
- The AI tool answers from those definitions instead of raw tables.
- Every answer shows the definition and data behind it, so anyone can check it.
This is the work most startups postpone, because it rarely feels urgent until two teams bring different numbers to the same meeting. The scope can be small. In dbt's benchmark, adding three cleaned-up tables took the share of questions answerable from defined metrics from none to all of them.
## Where to start
If you are about to roll out an AI tool on company data, pick the five numbers your leadership reviews every week. For each one, write down how it is calculated, which system it comes from and who decides when it changes. That list is the start of a semantic layer, and it gives any AI tool a definition to answer from.
The next step is turning that list into something software can read. A data engineer does this by connecting your systems and encoding each definition. Without one on the team, a data partner can do it for you.
Silicon builds and maintains these definitions for startups as a data partner, and our AI analytics platform answers from them. If you have your list of five, book 30 minutes with the founder and we will go through how each metric would be defined from your current systems.
FAQ
## Frequently asked questions
### Is a semantic layer only for companies with a data team? +
No. It can start as a short written list of definitions and grow from there.
### Does a better AI model remove the need for metric definitions? +
No. A model can only use a definition if someone has written it down.
Keep reading
## More from the blog
Guides Hiring a data engineer vs working with a data partner
### Hiring a data engineer vs working with a data partner
Guides dbt is where data work becomes software work
### dbt is where data work becomes software work
Guides How to monitor dashboard impact before merging a PR
### How to monitor dashboard impact before merging a PR
## Tell us where your data lives today.
Book 30 minutes with the founder. Tell us where your data is today, and see Silicon running on sample data.
Talk to Sujay
30-minute call with the founder