Why Redaction Isn't Enough: Tokenization vs Destructive Masking

Two approaches exist for keeping sensitive data out of an AI model's hands: destroy it, or disguise it reversibly. Most vendors in this space still default to the first. It's the wrong default, and the reason comes down to what happens to the response you get back, not just the query you send.

What redaction actually does

Redaction — sometimes called destructive masking — removes sensitive data before it reaches an AI model, typically replacing it with a blank, a generic placeholder, or nothing at all. [REDACTED]. ***. Gone.

It solves half the problem. The sensitive value never reaches the external model, which is correct. But redaction has a structural flaw that only shows up once you look at what the AI is supposed to do with the input: it destroys the entity-level structure the model needs to give you a genuinely useful answer.

Ask an AI to summarize a contract, and redaction produces something like: "The NDA between [REDACTED] and [REDACTED], signed [REDACTED], covers IP transfer valued at [REDACTED]." Technically readable, but every placeholder is indistinguishable from every other — the model has no way to know whether the two [REDACTED] tags refer to the same entity mentioned twice, or two different ones.

What tokenization does differently

Tokenization replaces each sensitive value with a structured, reversible token — [ORG_001], [PII_PERSON_001], [DATE_001] — before the query leaves your perimeter. The AI model receives a structurally coherent query: relationships, entity types and surrounding context remain intact, while the sensitive values themselves are replaced with consistent placeholders.

Tokenization solves the consistency problem redaction can't: "[ORG_001] acquired [ORG_002]. [PERSON_001] will retain control of [ORG_002]." The model knows ORG_002 in the second sentence is the same entity as in the first, that PERSON_001 is a person, that a specific relationship exists between them — structure and consistency preserved, real-world identity removed. When the response comes back, the tokens are resolved to the original values before the employee sees the answer.

This is the distinction that matters: redaction removes information permanently. Tokenization removes it temporarily, in a way that can be put back.

Why "no training on your data" doesn't solve this either

A common alternative pitch in this space is contractual: a vendor promises your data won't be used to train their models. It's a real commitment, and it's worth having in a contract. But it answers a different question than the one that matters for compliance.

A no-training commitment is a contractual safeguard. Tokenization is a technical control. The two solve different problems: one asks you to trust what happens to sensitive data after transfer. The other is designed to prevent the sensitive value from being transferred in readable form in the first place.

If your evidence for a regulator is "we have a no-training clause in a vendor agreement," you're relying on a contractual promise, not a demonstrable technical control — and those are treated very differently in an audit.

Tokenization doesn't depend on trusting a vendor's training practices, because the sensitive data never reaches them in readable form to begin with. The original values stay inside your perimeter the entire time. There's nothing external to trust.

The trade-off tokenization doesn't fully solve

Being direct about a limitation matters more than pretending one doesn't exist: tokenization has a real constraint on entity-dependent tasks.

If an employee asks an AI to cross-reference a company name against a sanctions list, or to research a specific organization's public reputation, tokenizing that company name into [ORG_001] removes the exact information the model needs to do the lookup. The model can't check a name it never saw. For these specific use cases — sanctions screening, company or individual research, anything requiring the model to act on the real-world identity of an entity — tokenization creates a genuine limitation that redaction shares, and that no masking approach fully resolves without exposing the underlying identity at some point in the process.

The honest framing: tokenization is the right default for the overwhelming majority of enterprise AI use — summarization, drafting, analysis, code review, internal Q&A — where the model needs to reason about structure and relationships, not verify real-world identity. For the narrower set of tasks that require identity verification against external sources, that step needs a separate, deliberate process — not a blind spot papered over by a vendor claiming their masking handles everything.

What this means when evaluating a vendor

A few direct questions worth asking any AI governance vendor, based on this distinction:

• Is masking reversible or destructive? If the answer is destructive, ask how the vendor handles tasks that need context preserved — summarization, translation, analysis — because redaction and those tasks are in tension by design.
• Where does the original data live during processing? If sensitive values ever leave your controlled perimeter — even briefly, even to the vendor's own infrastructure — that's a transfer that needs its own compliance assessment, tokenization or not.
• What happens with entity-dependent tasks? A vendor who claims their masking has no limitations for tasks requiring real identity resolution is either overselling or hasn't thought through the edge case. The honest answer acknowledges the constraint.

The bottom line

Redaction protects data by destroying information. Tokenization protects data by making it temporarily unreadable to everyone except the system that can put it back. For any AI task that depends on the model actually understanding what it's working with — which is most of them — that difference is the difference between a tool that works and one that technically complies while quietly producing worse output.