Why a Semantic Layer is Not Enough for AI

If you ask most data teams how they plan to prepare their data for AI, you will usually hear the same answer: "We are building a semantic layer."

As data analysts, we love semantic layers. Tools like dbt, Cube, or LookML solved one of our biggest operational headaches: the classic meeting where the marketing lead and the finance lead show up with two completely different numbers for "Total Revenue."

A semantic layer standardizes metrics, defines relationships between tables, and creates a single source of truth for what we measure. Because semantic layers work so well for BI dashboards, many teams assume they are all an AI agent needs. The logic seems simple. If an analyst can use a semantic layer to understand the data, an LLM should be able to do the same. But a semantic layer is only part of the solution. It's necessary, but it's not enough on its own.

The "What" vs. The "How and Why"

Semantic layers are designed to capture structural knowledge. They are great at defining the what:

• What tables join together?
• What formula defines Gross Margin?
• What columns represent currency?

But an AI agent trying to answer a real business question needs much more than just a formula. It needs behavioral context. It needs to know the how and the why.

A semantic layer can tell the AI how to join the users table to the orders table. But it will not tell the AI that the users table contains thousands of internal test accounts that must be filtered out when looking at growth. It will not tell the AI that because of a system migration in 2023, historical revenue numbers before that date require a specific timezone adjustment to be accurate.

A semantic layer sits strictly on top of the database. But in real life, a massive portion of the context required to understand your data does not live in tables at all.

Where Tribal Knowledge Actually Lives

Think about how a human data analyst works. When I was asked a complicated data question, I didn't just look at a dbt model or how the ETL works.

I searched through Slack to see how the marketing team discussed a recent anomaly in campaign spend. I looked at Git commits to understand why a specific data pipeline was altered last month. I read through Notion or Google Docs to find the onboarding materials that explain the edge cases of a product flow.

This is the "tribal knowledge" of an organization, which makes up the company brain. It's messy, unstructured, and spread across multiple tools.

If you connect an AI agent exclusively to a semantic layer, you completely blindfold it to this unstructured knowledge. You force the AI to analyze your business using only formal definitions, while ignoring the operational reality of how the company actually runs.

Bridging the Gap

A human analyst has intuition (this is something that AI cannot replace, yet). If you see a weird drop in European sales data, you instinctively check Slack for an outage report before sharing that number with the VP.

An LLM does not have intuition. It only knows the context you give it. If you only provide metric formulas, it will follow them blindly and give you answers that are technically correct according to the schema, but completely wrong in reality.

To build AI data assistants that we can actually trust, the modern data stack needs an additional step.

We need an intermediate context layer. This layer sits between your data and the AI, combining the structured definitions from your semantic layer with the unstructured tribal knowledge living in Slack, Git, and company docs. It needs to give the AI the complete story before it ever writes a line of SQL.

To give AI the foundation it actually needs, we have to look beyond database tables and build a context layer that captures how the entire company actually operates. Only by connecting the structured what with the real world how and why can we give AI the complete picture it needs to answer questions accurately.