Extending Data-Centric AI to Context

Two employees ask their company's internal AI the same question on the same day: What's our refund policy for Enterprise customers?

The first gets an answer citing an approved procedure from the internal wiki. The second gets a completely different answer, drawn from a decision made in a Slack thread two weeks ago that never made it into any document.

Both answers are correct. Nothing is broken. Neither is a corrupted version of the other.

This isn't an edge case to be handled. It's a defining property of organizational context, and almost everything hard about building systems on top of it follows from there.

Data was assembled. Context accumulated.

For a growing class of business applications, the model is no longer the only bottleneck. The difference between a good answer and a bad one increasingly depends on what you put into the model, not only which model you picked.

The industry absorbed that conclusion quickly, and absorbed a familiar frame along with it: data-centric AI. That idea, formulated in the context of model training, held that instead of improving the architecture you should improve the data. The model stays fixed; the work moves to the data.

The extension looks straightforward. Data used to mean the training set. Now it means the context. Same principle, wider scope.

The frame is right, useful, and worth keeping. But there's a difference underneath it.

A dataset is assembled. Someone put it together for a need defined in advance - decided what goes in and what stays out, what counts as a correct label, what the system is meant to measure.

Organizational context was never assembled. It accumulated.

A Slack message was written for three people who already knew what it was about. A line in a Jira ticket presumes a conversation that happened out loud. A clause in a policy document was written in response to an incident everyone in the room remembered.

Once we start treating this accumulated material as context for agents, three assumptions become much harder to hold.

Gap one: utility doesn't add up

Information retrieval has always treated relevance as a relationship between query and document. But much of its machinery still judges documents one at a time. Standard metrics sum per-document gain; ranking systems score candidates independently and take the top results.

The field studied the exceptions - personalization, time-dependent relevance, diversity, redundancy, set-level judgment. With agents, the exception increasingly becomes the default: queries are simultaneously person-dependent, time-dependent, and set-dependent.

Chroma's Context Rot report, which tested 18 frontier models on long-context behavior, found that a document can be actively negative rather than neutral - and that semantically similar but irrelevant content does more damage than random text. A close neighbor is more dangerous than a stranger.

The conclusion: the utility of a context is not the sum of the utilities of its items. If that's true, the unit to be judged is the set, not only the item. Scoring items independently and taking the top ten assumes an additivity that isn't always there.

Gap two: the data writes itself

In the classical picture the roles are clean. People produce and label, the model consumes. The dataset is a stable artifact that someone owns.

In agentic systems that picture collapses. The system produces its own context - summaries, extracted insights, memory, findings from earlier runs - consumes it on the next pass, and produces more on top of it.

This is no longer only a data quality problem. It's a closed-loop control problem, with everything that comes with one: drift, re-ingestion of its own errors, and convergence toward states nobody wanted. The same loop is why stale context doesn't throw an error. It just keeps answering.

The sharpest documented example comes from ACE, a context-adaptation framework out of Stanford and SambaNova. When a model repeatedly rewrites its accumulated context, it tends to compress it. In one experiment, 18,282 tokens became 122 and accuracy fell from 66.7 to 57.1 - below what the system scored with no adaptation at all. They call it context collapse.

A dataset doesn't do that. It doesn't erase itself while you're watching.

Gap three: the dataset is a function, not a set

Inside an organization there is no single dataset. What one employee can see differs from what another can see - not because something is misconfigured, but because that's the structure.

This doesn't mean you should build a thousand datasets. It means something more precise: the dataset isn't a set, it's a function from reader to set. You can't materialize it once. You can only sample it for a particular reader.

The consequence is that correctness doesn't live only in the output. The exact same answer can be excellent for one reader and a leak for another.

This also makes ordinary data-cleaning operations hazardous. Deduplication, canonicalization, and merging are precisely what turn raw data into clean data. But merging two artifacts with different readerships can produce an item with no single legitimate reader. Even when permissions survive, merging can erase the circumstances that made the two items different in the first place.

Sometimes what looks like cleaning is information loss.

The missing piece

The three gaps look different:

• Context utility is non-additive.
• Context is recursive.
• Context is observer-dependent.

But underneath them is something more fundamental.

It's sitting in the word context itself.

Context means the circumstances that give a statement its meaning. And those circumstances are precisely what's missing from most pipelines we've built.

Chunking and embedding preserve the part that was written while often discarding the part that gave it meaning. They treat text as though it stood on its own, when organizational artifacts rarely do.

Which means provenance isn't only metadata for establishing trust. Sometimes it's part of the meaning itself.

The fact that something appears in a company document doesn't make it universally true - it tells you someone wrote it, once, for a reason of their own. And a contradiction between two items doesn't necessarily make either wrong. Two conflicting refund policies may both be correct - from two points in time, or for two different lines of business.

What's missing isn't necessarily a correctness label. It's the circumstances that tell you whether the two items are even talking about the same case.

What this means in practice

The unit of storage should be a context item, not just a chunk: a typed body - message, document, record, entity - alongside the circumstances of its creation: who, when, addressed to whom, in response to what, and who is permitted to see it.

Text is one field inside the body, not the unit itself.

The test before indexing: is this item legible to someone who wasn't in the room? If not, it needs enrichment, not indexing.

And when measuring retrieval failure, separate two kinds: not found, and found but illegible outside its context. We measure the first constantly. The second appears in virtually none of the metrics we use.

Back to the two answers

The two different answers that opened this post aren't necessarily a failure of the system. They're part of its specification.

Underneath them sits a deeper fact: every piece of information in an organization was created for someone else, at another moment, out of a mass of shared assumptions nobody bothered to write down.

We've built excellent infrastructure for preserving what was said, and almost nothing for preserving what went without saying.

That's the part that got left out - and the part data-centric AI still needs to account for as we extend it to context.