AI Hallucinations and GRC

Generative AI has moved from experimentation to everyday business use in many companies. Chatbots answer customer questions, copilots support employees, internal knowledge assistants search documents, marketing teams create content with AI, and specialist departments use language models for analysis, summaries and decision drafts. What sounded futuristic only a few years ago has become operational reality.

However, this development also increases a risk that has long been underestimated: AI hallucinations. These are responses that appear convincing, professional and linguistically correct, but are factually wrong, incomplete or misleading. This combination is what makes hallucinations so dangerous. The output looks trustworthy, even though it may not be reliable.

The current regulatory pressure from Italy shows that authorities are taking this risk increasingly seriously. AI providers are expected to inform users more clearly that generated content can contain errors and must be verified. At first glance, this may seem like a consumer protection issue. In reality, it is much more than that: it is a clear signal for Governance, Risk and Compliance. Companies must be able to demonstrate that they do not merely know about AI risks, but actively manage them.

For GRC, this means that AI hallucinations are no longer just a matter for the IT department or technical product discussions. They affect corporate governance, risk management, compliance, data protection, information security, internal controls, third-party risk management, internal audit and reputation management. Any company using or offering generative AI needs a robust control framework.

Key Takeaways

• AI hallucinations are false or misleading AI-generated responses that can appear convincing and professionally accurate.
• Regulatory pressure is increasing because authorities expect more transparency, clearer warnings and stronger safeguards.
• For GRC, new requirements are emerging around governance, risk assessment, compliance, internal controls, documentation and auditability.
• Companies need a central AI inventory to understand which AI systems are being used, where, by whom and with what data.
• Disclaimers alone are not enough. Clear processes, human review, training, monitoring and technical safeguards are required.
• AI applications are particularly critical in customer service, legal, compliance, HR, finance, healthcare, insurance, banking and regulated industries.
• Third-party risk management becomes more important because many companies rely on external AI providers.
• The EU AI Act, AI literacy, data protection, information security and sector-specific requirements must be considered together.
• Internal audit and GRC teams should treat AI hallucinations as a standalone risk scenario.
• Companies that manage AI responsibly build trust, reduce liability risks and strengthen regulatory resilience.

What Are AI Hallucinations?

AI hallucinations occur when a generative AI system produces content that is not reliably based on facts. The system formulates an answer that sounds plausible, but is partly or entirely incorrect. The issue is especially problematic because language models often appear very confident. They do not always qualify their responses, but often present incorrect information in a tone that suggests competence and certainty.

For example, a chatbot may refer to a legal provision that does not exist, cite a fabricated court decision, explain a contract deadline incorrectly or promise a product feature that is not actually available. In an internal corporate context, AI may incorrectly summarise figures from documents, misinterpret policies or assess risks inaccurately in a report.

For users, this is difficult to detect. People tend to perceive well-written answers as more credible. This is exactly where the GRC problem begins. The quality of the language can hide the uncertainty of the information.

AI hallucinations are therefore not just a technical problem. They are a trust problem. They influence decisions, processes and expectations. If companies adopt AI outputs without review, risks arise for customers, employees, business partners and the organisation itself.

Why Italy’s Action Sends an Important Signal

The action taken by the Italian competition and consumer protection authority against several AI providers shows that the regulatory perspective is changing. The key question is no longer only whether an AI system is innovative, powerful or commercially attractive. Increasingly, the decisive question is: are users sufficiently informed about the system’s limitations?

This shift is significant for companies. Authorities are no longer treating hallucinations merely as an unavoidable side effect of modern AI. They are interested in whether providers and operators communicate this risk clearly, whether warnings are visible and whether users are adequately informed before making important decisions.

For GRC, the message is clear. Transparency becomes a control. A notice hidden in general terms and conditions is not enough if users do not understand during the actual AI interaction that the content may be wrong. What matters is whether the warning appears where the risk arises: in the chat window, in the application, before registration, before a purchase or before the use of an AI-generated result.

This is an important distinction. Traditional compliance often relies on policies, contract clauses and documentation. With generative AI, compliance must be embedded more strongly into product design, user journeys and operational processes. The user interface itself becomes part of the control framework.

Why AI Hallucinations Are a GRC Risk

Governance, Risk and Compliance deals with how companies are responsibly managed, how risks are controlled and how legal as well as internal requirements are met. AI hallucinations affect all three areas at the same time.

From a governance perspective, the question is who is responsible for AI systems. Many companies use AI in a decentralised way. Business units test tools, employees use public chatbots, SaaS providers integrate AI functions and IT departments deploy copilots. Without clear responsibilities, a control vacuum emerges.

From a risk perspective, the question is what damage false AI outputs can cause. An inaccurate internal draft may be relatively harmless. A false statement to customers, a misleading financial analysis or a fabricated compliance requirement can have serious consequences.

From a compliance perspective, the question is whether companies are meeting their duty of care. Anyone using AI in regulated areas must be able to explain which risks were assessed, which controls are in place and how problematic developments are identified.

For this reason, hallucinations must not be treated as isolated errors. They are a systemic risk because they can occur in many places at once: customer communication, advisory services, documentation, decision support, reporting, contract review, HR processes and internal analysis.

The New Expectation: Companies Must Make AI Risks Visible

One central problem in many organisations is the lack of transparency around their own AI usage. In practice, there is often more AI in use than is officially known. Employees use freely available tools, departments test software with integrated AI functions, external service providers use generative AI and existing platforms gradually activate new AI capabilities.

For GRC, this lack of transparency is dangerous. What is not known cannot be assessed. What has not been assessed cannot be controlled. And what is not controlled cannot be convincingly defended if something goes wrong.

The first step towards effective AI governance is therefore an AI inventory. Companies must record which AI systems are in use, who owns them, what data is processed, which providers are involved, which users are affected and which decisions or processes are influenced.

This inventory should not be understood as a one-off spreadsheet. It must become part of an ongoing governance process. New AI applications should be reviewed before implementation. Existing applications should be updated regularly. Changes to models, data sources, prompts, interfaces or providers should trigger a new risk assessment.

Third-Party Risk Management: Assessing AI Providers Properly

Many companies do not develop their own AI models, but use external solutions. This may be a major cloud provider, a specialised AI service, a SaaS tool with integrated AI or a platform that combines multiple models. As a result, part of the risk shifts into the supply chain.

Third-party risk management must therefore be expanded. Traditional supplier questions around data protection, information security and availability remain important, but they are no longer sufficient. Companies must also understand how the provider handles hallucinations, model updates, training data, user prompts, output safety and transparency.

Role clarity is also important. Is the provider itself the model provider, only the operator of an application, an integrator, a data processor or an interface provider? Does it use additional sub-providers? Are inputs stored? Is customer data used for training? Where does processing take place? What control and audit options exist?

GRC should not only review the contract when assessing AI providers, but also the operational control environment. This includes technical documentation, security evidence, data protection information, model change processes, incident procedures and clear obligations to notify customers of material changes.

Data Protection and Information Security: Hallucinations Are Not the Only Risk

Although hallucinations are the main focus, companies must not ignore other AI risks. Generative AI can process personal data, disclose confidential information, transfer sensitive data to external systems or be influenced by manipulated inputs.

Data protection and information security must therefore be closely connected to AI governance. Companies should define which types of data may be entered into which AI systems. Personal data, trade secrets, customer data, health data, financial data, legal information and security-relevant information are particularly sensitive.

Prompt injection is also relevant. This refers to attempts by attackers, or unintended inputs, to manipulate the behaviour of an AI system. If an AI system has access to internal documents, emails, databases or actions, this can create a serious security risk.

GRC must therefore not view AI merely as a communication tool. The more AI systems are integrated into processes, data sources and automated actions, the more important access controls, role models, logging and security reviews become.

AI Agents Increase the Pressure on GRC

The next stage of development is AI agents. While traditional chatbots mainly provide answers, agents can perform tasks. They can retrieve information, operate systems, prepare emails, process tickets, trigger orders or initiate workflows.

This significantly increases risk. A hallucinated answer is problematic. A hallucinated action can be even more problematic. If an AI agent changes data, contacts customers or triggers internal processes based on false assumptions, new control requirements emerge.

Companies should therefore be especially cautious when AI is allowed not only to generate text, but also to execute actions. The higher the degree of automation, the more strongly permissions must be limited, approvals embedded and actions logged.

A good GRC principle is: AI should only have the rights it truly needs for a clearly defined purpose. Critical actions should not be fully automated, but should require human confirmation.

How Companies Can Integrate AI Hallucinations into Risk Management

Professional risk management does not treat hallucinations as a vague concern, but as a concrete risk scenario. This scenario should be described, assessed, controlled and monitored.

The assessment should consider who uses the AI, which decisions are influenced, what data is used, how easily users can detect errors and what damage is possible. An internal brainstorming tool is less critical than a customer chatbot for insurance coverage. An AI system that generates marketing ideas has a different risk profile than a system interpreting regulatory requirements.

Companies should classify their AI use cases by risk level. Low-risk applications can operate with lighter controls. Applications with customer impact, legal relevance, personal data or proximity to decision-making require stricter requirements.

It is important that the risk classification does not remain static. A use case can become more critical over time if it reaches more users, receives new data sources or becomes integrated into additional processes.

What a Strong Control Model Should Include

A robust control model for generative AI connects governance, technology, processes and people. It starts with clear rules, but goes much further.

Companies should first define which AI usage is permitted and which is not. They then need to specify how new AI applications are requested, assessed and approved. Critical use cases require additional requirements for testing, user notices, data sources, human review and monitoring.

Technical design also plays a role. Systems should rely on verified knowledge sources where possible, make uncertainty visible and hand over to humans for critical questions. If AI cannot provide a reliable answer, an honest non-answer is better than a convincing hallucination.

In addition, companies need incident management for AI errors. If a false AI output leads to damage, a complaint or regulatory risk, it should be clear who is informed, how the incident is assessed and which corrective measures follow.

Impact on Regulated Industries

Regulated industries are particularly affected. Banks, insurers, healthcare providers, energy companies, telecommunications firms, public authorities and other supervised organisations must manage AI risks especially carefully.

In banking and insurance, false AI outputs can lead to poor advice, incorrect risk assessments or inaccurate information about products and claims. In healthcare, incorrect answers can have especially sensitive consequences. In the public sector, false AI communication can damage trust in administration and public institutions.

Regulated companies should therefore not wait until every specific question has been clarified by supervisory authorities. The direction is clear: transparency, risk management, human control, documentation and evidence are expected.

Common Mistakes Companies Should Avoid

A common mistake is addressing AI risks too late. Many organisations only start building governance once AI is already widely used. By then, tools, processes and habits are often established, making later corrections more difficult.

Another mistake is relying too heavily on provider promises. Even large and well-known providers do not relieve a company of its own responsibility. Anyone integrating AI into their own processes or using it with customers must assess whether the deployment is appropriately controlled.

It is also problematic to believe that good prompts solve the hallucination problem. Better prompts can help, but they do not replace governance. A one-time training session is also insufficient if employees do not have clear rules and escalation paths afterwards.

Perhaps the biggest mistake is a lack of evidence. If something goes wrong, it is not enough to say that AI is used responsibly. Companies must be able to prove which risks were assessed, which controls were implemented and which decisions were made.

Conclusion: AI Hallucinations Are a Test Case for Modern GRC

AI hallucinations show how much the risk profile of modern companies is changing. This is no longer only about traditional IT security, data protection or regulatory documentation. It is about how organisations deal with systems that generate convincing content, but are not automatically reliable.

Regulatory pressure makes this clear: companies must inform users clearly, control AI outputs and manage risks in a demonstrable way. For GRC, this is not a side issue, but a central topic for the future.

Companies that want to use generative AI successfully need more than enthusiasm for efficiency and innovation. They need governance, clear responsibilities, robust controls, trained employees, assessed providers and ongoing monitoring. Only then can they prevent AI from turning from a productivity gain into a compliance and reputational risk.

The good news is that companies do not need to eliminate hallucinations completely in order to use AI responsibly. But they must be able to show that they understand the risk, limit it, make it transparent and control it. That is the new task for GRC.

FAQ

What are AI hallucinations?

AI hallucinations are false, fabricated or misleading outputs generated by an AI system. They often appear convincing even though they are factually incorrect.

Why are AI hallucinations important for GRC?

They can cause poor decisions, incorrect customer information, compliance breaches, liability risks and reputational damage. For this reason, they must be addressed in governance, risk management and compliance.

Is a warning about possible errors enough?

No. A warning is important, but not sufficient. Companies also need risk assessments, human review, technical controls, training, monitoring and clear responsibilities.

Which departments should be involved?

Executive management, compliance, legal, risk management, data protection, information security, IT, business departments, procurement, third-party risk management and internal audit should all be involved.

What is the first step towards better AI governance?

The first step is a complete AI inventory. Companies must know which AI systems are being used, who is responsible, what data is processed and what risks exist.

Which areas are particularly critical?

Customer service, legal, compliance, HR, finance, healthcare, insurance, banking and all processes where AI outputs influence decisions or are used with customers are particularly critical.

How can companies reduce hallucinations?

Helpful measures include verified knowledge sources, clear system boundaries, better prompts, human review, regular testing, monitoring, feedback functions and technical safeguards. However, hallucinations cannot be completely eliminated.

What should internal audit review?

Internal audit should assess whether there is an AI inventory, risk assessments, clear responsibilities, user notices, approval processes, provider reviews, training, logging and incident management for AI errors.

What role does third-party risk management play?

Third-party risk management must assess AI providers, model providers, SaaS solutions and sub-providers. Important questions include data processing, model updates, hallucination controls, security, data protection and auditability.

Why will this topic become more important?

Generative AI is increasingly being integrated into business processes. At the same time, regulatory expectations around transparency, control and evidence are rising. Companies that build AI governance now reduce risk and create trust.