By guest blogger, Edmund J Sutfliffe (consultant at Considrd Consulting).
Edmund gave a fascinating talk at our first Future State event in June 2025. This post summarises some of the points raised and discussed at the event.
Insurance was built to price uncertainty – but AI is teaching it to pretend uncertainty doesn’t exist.
After every major disaster, the same pattern emerges: insurance enquiries spike, customers demand better predictions, and the industry responds with increasingly sophisticated AI models promising mathematical certainty about inherently uncertain events. But this cycle – what I call the “certainty trap” – is fundamentally broken, and it’s about to get worse.
The psychology problem
Consider the human psychology at play. We evolved to seek certainty, yet insurance exists precisely to manage uncertainty. This mismatch creates baffling market behaviour that no algorithm can predict.
After Germany’s devastating 2002 Elbe floods – one of the worst flooding events in Central Europe since the Middle Ages – only 19% of households had flood insurance, according to the German Insurance Association (GDV). Despite repeated major floods in 2013 and 2021, coverage reached just 52% by 2022. In Bavaria specifically, coverage remains at only 47% despite the region experiencing multiple catastrophic flood events. Same rivers, increasing flood frequency, but human psychology prevents rational risk pricing.
Meanwhile in the UK, the government had to intervene with Flood Re Insurance in 2016 after a series of major floods between 2007 and 2016 made flood insurance unaffordable or unavailable for high-risk properties. The scheme now subsidises over 265,000 at-risk households, according to Flood Re Insurance ‘s 2023 annual report, with four out of five policyholders who had previously made flood claims seeing their premiums drop by more than 50%. But the scheme expires in 2039, with no clear plan for what happens next. The market has learnt nothing about pricing long-term certainty.
When we layer sophisticated AI onto this foundation, we don’t solve the problem – we amplify it, manufacturing a false precision that is both dangerous and brittle.
Not all AI is created equal
The insurance industry’s rush to adopt AI has created a dangerous conflation. We’re treating all artificial intelligence as if it’s the same technology, when the differences matter enormously:
Expert systems from the 1970s – simple if-then logic – remain completely transparent and auditable. They don’t get smarter over time, but crucially, they don’t get worse or biased either.
Machine learning models for fraud detection and risk pricing can be validated by testing predictions against real-world outcomes. They model phenomena we can measure.
Large language models like ChatGPT represent something fundamentally different. They’re sophisticated autocomplete systems trained on web documents. They predict what sounds convincing, not what’s true.
And here’s the crisis: these are the least validatable AI systems, yet they’re increasingly being used for the most important decisions.
The model collapse problem
As the web fills with AI-generated content, new AI systems train on this polluted data, losing their connection to reality. This ‘model collapse’ affects rare events first – precisely the scenarios insurance must handle. Early degradation is invisible: overall performance looks fine whilst minority case handling silently fails.
Imagine this scenario: An insurer’s new underwriting AI is trained on web data from 2024, which is now polluted with AI-generated property descriptions and accident reports. This synthetic data quietly erases rare but crucial risk factors – like unique wiring types in historic homes or non-standard commercial roof constructions. The model’s overall accuracy on common properties looks fine, but it has silently become blind to the edge cases that lead to catastrophic, multi-million-pound claims. You can’t detect this degradation until it’s too late.
Imagine another scenario: A life insurer’s new underwriting AI is trained on web data from 2024, which is now polluted with AI-generated medical articles and patient summaries. This synthetic data quietly erases rare but crucial risk factors – like specific genetic markers associated with early-onset conditions, unusual drug interactions, or occupational hazards in emerging industries. The model’s overall mortality predictions for healthy 30-year-olds look fine, but it has silently become blind to the rare cases that drive actuarial losses – the unexpected claims that can cost hundreds of thousands of pounds and fundamentally mispriced risk pools. You can’t detect this degradation until the claims arrive, years after the policies were underwritten.
If you can’t detect when your AI is getting worse at handling edge cases, you’re flying blind in the scenarios that matter most.
The wrong tool problem
We’re experiencing a massive tool-task mismatch. LLMs were designed for content generation but are being deployed for high-stakes decisions they cannot reliably make.
Where LLMs excel:Â Customer service chatbots, document drafting, creative writing. Tasks where “good enough” output with natural language interaction adds genuine value.
Where LLMs fail catastrophically: Underwriting decisions, claims assessments, risk pricing. These require consistency, explainability and auditability that LLMs fundamentally cannot provide.
The real cost:Â AI errors compound across thousands of policies. Biased algorithms affect entire customer segments. When economic and legal costs exceed efficiency gains, the business case collapses – along with regulatory trust.
The solution isn’t to abandon AI, it’s to use the right AI for each task. Think of LLMs as sophisticated user interfaces that sit on top of interpretable decision-making systems, never as the engines themselves.
Learning from safety-critical industries
We don’t need to invent solutions from scratch. Nuclear power, aviation and medical devices solved this decades ago through mandatory frameworks requiring reproducibility, traceability, and explainability.
Every nuclear control system decision is traceable. The Boeing 737 MAX crisis demonstrated what happens when automated systems lack adequate transparency. The FDA requires evidence-based explanations for AI diagnostics because doctors need to understand the reasoning.
Insurance decisions affect families’ financial survival. If we require explainable systems for aeroplane autopilots, why not for decisions affecting thousands of policyholders? We already have the blueprints – we just need to adapt the discipline these industries have mastered.
A trustable framework
Building trust requires more than bolting explanations onto black boxes. It requires designing transparency in from the ground up:
Reproducible Builds:Â Bit-for-bit identical deployments that can be recreated exactly. If you can’t rebuild the system that made a decision, you can’t defend that decision.
Traceable Provenance:Â Complete audit trail from code to deployment to decision. Every choice must be documentable for regulators and customers alike.
Explainable by Design:Â Intrinsic transparency, not retrofitted explanations. The system’s logic should be inherently understandable, not reverse-engineered after the fact.
Original Human Data:Â As model collapse spreads through AI-generated content, clean training data becomes a competitive advantage. Preserve it strategically.
Here’s a practical checklist any insurer can apply before deploying AI systems:
- Question the premise:Â Do you actually need AI? Could this be solved with Excel, rules, or statistics?
- Interpretability first:Â Start with explainable models. If you can’t explain it to a sceptical regulator, don’t deploy it.
- Engineering discipline:Â Implement reproducible builds, version control and testing frameworks before considering any AI.
- Layer carefully:Â Use LLMs as sophisticated user interfaces, never as decision-making engines.
The human element
Technology alone won’t solve the certainty trap. We must design for how humans actually think:
Appropriate defaults that guide customers towards good choices rather than exploiting cognitive biases for profit.
Clear language that communicates probability ranges instead of false precision. Honest uncertainty rather than manufactured confidence.
Accounting for bias by building models that understand human decision-making patterns, not just statistical probabilities.
The shift required is cultural: from “what’s the coolest AI?” to “what’s the most appropriate solution for humans?” Balance technical capability with psychological appropriateness and regulatory compliance.
The regulatory reality
This isn’t theoretical. The Cyber Resilience Act, NIS2, and DORA all have implementation deadlines compressing into 2025–2026, just as AI complexity explodes. Organisations that master trustable AI now will have better regulatory relationships, lower compliance costs and more reliable systems as the broader AI landscape becomes less dependable.
Moving forward
Breaking the certainty trap requires acknowledging an uncomfortable truth: customers want certainty, but insurance must communicate uncertainty honestly. The right AI can help – not by manufacturing false precision, but by building systems that are transparent, traceable, and genuinely explainable.
The questions you should be asking:
- Can you explain your AI decisions to the FCA tomorrow?
- How do you detect when your models are silently degrading?
- What original human data are you preserving as model collapse spreads?
- Are your AI systems accidentally amplifying the certainty trap?
Trust in AI won’t come from better models – it will come from better governance. The frameworks exist. The regulations are coming. The only question is whether the industry adapts proactively or reactively.
Given insurance’s track record with certainty, I know which approach serves customers better.
Edmund J, Sutcliffe is a consultant at Considrd Consulting, specialising in AI governance and risk management for regulated industries.
Build IT Now
Want to learn more about what we can do for your business? And how quickly? Go to the Systems iO services page or
Stay updated with our latest insights, industry news, and exclusive content by following us on LinkedIn! Join our growing community of professionals and be part of the conversation. Follow us on LinkedIn and never miss an update!
If you would like to receive our newsletter direct to your inbox, simply sign up at the bottom of this page..
