Undetected AI hallucinations pose the bigger business risk

One of the first questions people ask about artificial intelligence is whether it hallucinates. The answer is straightforward: Yes, it does. Large language models occasionally invent references, produce incorrect facts or confidently present information that is simply wrong. Anyone who has spent time using AI has seen it happen.

The problem is that the conversation usually ends there. As engineers, we should be asking a different question. Rather than focusing on whether hallucinations exist, we should be asking what happens when an incorrect answer quietly becomes part of an engineering process or business decision without anyone noticing. As AI becomes integrated into software development, product design and organisational workflows, that risk becomes far more significant than the occasional incorrect response on a chat screen.

The real problem isn't that AI hallucinates but that hallucinations can become embedded in documents, requirements, reports and decisions, where the original error becomes increasingly difficult to trace. That is an engineering problem, not simply an AI problem.

The black-box problem isn't new

Engineers have worked with black boxes for decades. A black box is simply a system whose internal workings are too complex-or impossible-to fully observe. Rather than understanding every internal mechanism, we build confidence by controlling inputs, observing outputs and testing behaviour.

Modern AI fits that description remarkably well. We provide a prompt, the model performs an extraordinary number of calculations and returns an answer. Although the mathematics behind neural networks is understood, predicting exactly why a particular response was generated remains difficult. Even the organisations developing frontier AI models cannot always explain individual outputs.

That uncertainty shouldn't concern engineers-it should feel familiar. Engineering has never depended on perfect knowledge. It depends on testing, validation and understanding where systems become less reliable. AI should be approached in exactly the same way.

AI doesn't reason the way humans do

One misconception about AI is that it thinks like people. It doesn't.

Humans combine deduction, experience, intuition and abductive reasoning-the ability to form the most likely explanation from incomplete information. Much of that happens subconsciously. AI works differently. Large language models predict statistically probable sequences of language based on patterns learned during training. They do not possess intuition, judgement or genuine understanding.

This helps explain why hallucinations occur. When information is incomplete, humans are often comfortable saying, "I don't know." AI is much more likely to generate the most probable answer instead. That answer may sound convincing without actually being correct.

For engineers, confidence should never be confused with evidence. AI outputs deserve the same level of scrutiny we would apply to any other source of information.

Why this matters

Over the past few years I've noticed my own thinking change. I spend far less time trying to write the perfect prompt and far more time understanding how AI behaves. I'm interested in the situations where it performs well, the situations where it becomes unreliable and the controls required to manage that uncertainty.

That shift has convinced me that the future of AI won't be defined by who has access to the most powerful model. It will be defined by who builds the most reliable systems around it. For engineers, that is a much more interesting problem to solve.

Hallucinations are an engineering risk, not an AI flaw

One of the ideas I found most valuable in Ethan Mollick's Co-Intelligence is that AI should be treated as a collaborator, not an oracle. AI contributes speed and scale, while humans remain responsible for judgement, context and accountability. Problems begin when organisations confuse those roles.

As engineers, we shouldn't think of hallucinations as an AI flaw. We should treat them as an engineering risk. Every engineering discipline works with uncertainty. The objective isn't to eliminate failure completely, but to design systems that continue operating safely when failures occur. AI should be approached in exactly the same way.

Apply engineering risk thinking

The same risk frameworks we've always used still apply. The key questions are simple: How likely is an error, and what happens if nobody notices it?

A poor marketing suggestion carries little consequence (except a Starbucks advert in South Korea). An incorrect engineering calculation, financial recommendation or clinical decision carries far greater risk. The higher the consequence, the stronger the controls need to be.

AI governance should therefore begin with the decision being supported, not the technology itself.

Five practical ways to reduce hallucination risk

There is no single solution, but several techniques significantly improve reliability.

First, ground AI in trusted knowledge. Connect it to engineering standards, approved documentation and organisational data rather than relying only on model training.

Second, keep a human in the loop wherever the consequences of error are significant. AI should support decisions, not make final ones.

Third, cross-check important outputs. Use purpose built tools or independent sources to verify critical information.

Fourth, maintain good session hygiene. Incorrect assumptions introduced early in a conversation often influence everything that follows, so keep discussions focused and correct errors as soon as they appear.

Finally, recognise the limits of domain knowledge. Foundation models are trained on public information, but specialist engineering expertise is often underrepresented. Where deterministic calculations or verified formulas already exist, they should remain the source of truth, with AI supporting the workflow rather than replacing established engineering methods.

Engineering AI systems is becoming more important than prompting them

Prompt engineering has attracted enormous attention, but I believe its importance will diminish over time. The organisations creating the most value from AI are not writing the cleverest prompts; they're building reliable systems around AI. How knowledge is retrieved, validated and combined with human judgement will ultimately matter far more than the wording of a prompt.

This isn't a new idea. Good engineering has never relied on one component working perfectly. It relies on designing systems where each component performs the task it is best suited to. AI should be treated no differently.

How this thinking shaped ModernCAD Memory

This philosophy directly influenced the development of ModernCAD Memory.

The challenge wasn't making AI more intelligent but giving AI access to the knowledge that already exists inside engineering organisations. Generic models know engineering concepts, but they don't know your products, standards, design rules or workflows. Without that context, they are forced to make assumptions.

ModernCAD Memory grounds AI in organisational knowledge in the process of generating a response. By connecting AI to engineering standards, CAD data, projects and documented processes, the objective isn't to eliminate hallucinations completely, but to reduce uncertainty by improving the quality of information AI reasons from.

The future belongs to organisations that engineer AI

The competitive advantage won't come from having access to the latest AI model. Those capabilities will increasingly be available to everyone. The advantage will come from building AI into engineering processes with the same discipline we've always applied to other critical systems.

Hallucinations will continue to exist. Our responsibility isn't to expect perfection, but to design systems that detect errors before they become decisions. Ultimately, that's how I think engineers should approach AI-not as something to trust blindly, but as a powerful tool that delivers its greatest value when supported by sound engineering principles.

For more information, visit: https://moderncad.com.au/