Every enterprise that shipped an LLM feature in the last two years now has the same problem sitting in its risk register: the AI sometimes makes things up, and nobody can prove when it doesn't. A chatbot invents a refund policy. A summarizer fabricates a figure in a board deck. A support agent cites a regulation that doesn't exist. Each incident is small; the liability is not.
From embarrassment to obligation
Hallucination used to be a quality issue — embarrassing, but survivable. Three forces are converting it into a compliance issue:
- Regulation. The EU AI Act's transparency and accuracy provisions are now being enforced, and they require providers of general-purpose and high-risk AI systems to assess and mitigate the risk of generating misleading output — with documentation to prove it.
- Litigation. Courts have already held companies responsible for what their chatbots tell customers. "The AI said it, not us" has failed as a defense, which means legal teams now demand controls.
- Procurement. Enterprise buyers have added AI-accuracy questions to vendor security questionnaires. If you sell software with generative features, you are being asked how you monitor hallucination — today.
Every compliance requirement eventually produces a product category. SOC 2 produced compliance automation. GDPR produced privacy platforms. AI-accuracy rules will produce hallucination monitoring.
What the product looks like
The emerging pattern is a monitoring layer that sits between the model and the user. It grounds claims against source documents, scores outputs for unsupported assertions, flags or blocks the risky ones, and — critically — keeps an audit trail. The audit trail is what turns a nice-to-have into a budget line: compliance teams don't buy accuracy, they buy evidence of accuracy.
Vendors are converging on this from three directions: LLM observability companies adding factuality scoring, guardrails startups moving up from safety filters, and fact-checking platforms moving down from media into enterprise. None has claimed the category name yet.
The naming vacuum
And the category has a naming problem worth noticing. "Hallucination detection" describes the mechanism. "AI observability" is too broad. What regulators, lawyers, and buyers actually care about is whether the output is misleading — that's the word in the EU AI Act, in FTC guidance, and in every internal policy document. The company that brands itself around that word won't have to explain what it does. Its name will do the selling.
The takeaway
If the last platform shift produced a compliance industry around data privacy, this one is producing a compliance industry around AI truthfulness. The budgets are being written now, the regulations are in force, and the category-defining brand is still up for grabs.