When the Oracle gets it wrong
By Dipu KV, Sr. President – Operations & Customer Service, Bajaj General Insurance
In Spielberg’s Minority Report, a specialised unit arrests people for crimes they have not yet committed -guided entirely by a predictive system believed to be almost infallible. Crime falls. Society applauds. Efficiency wins.
And then one day, the system gets uncomfortable. Not because it stops working. But because it points at one of its own.
I watch that film very differently today than I did a decade ago. Back then, it felt futuristic. Today, it feels operational. I remain one of AI’s strongest believers in Indian insurance. I believe it will redefine underwriting, claims, servicing, fraud detection, and customer experience over the next decade. But I also believe something equally strongly: Technology worshipped without wisdom eventually becomes risk disguised as progress.
Recent research from Anthropic ’s “Project Mythos” has raised important questions about how advanced AI systems can occasionally develop behaviours that are difficult to predict, explain or control. While these findings emerge from controlled research environments, they serve as a useful reminder that capability and reliability do not always grow at the same pace.
Risk #1 -The Hallucination That Sounds Convincing
Humans usually hesitate when uncertain. AI rarely does. That asymmetry is dangerous.
Large language models powering customer assistants and enterprise copilots are now widely known to generate plausible but incorrect answers -what the industry politely calls “hallucinations.” In creative writing, that can be amusing. In insurance, it can become expensive very quickly.
A wrongly interpreted policy clause. An invented process step. An incorrect claim advisory was delivered with confidence.
Suddenly, efficiency becomes a liability. And unlike human mistakes, AI mistakes compound instantly across thousands of interactions.
Risk #2 -Bias Hidden Inside Data
Every AI model learns from history. And history is rarely neutral.
Insurance datasets carry decades of human decisions, behavioural patterns, geographic concentration, economic inequalities, and access gaps. AI does not magically erase those biases. In many cases, it quietly amplifies them.
Train a pricing model largely on urban claims behaviour, and rural customers get unfairly profiled. Train health-risk models primarily on digitally active populations, and entire customer segments become statistically invisible.
The machine is not being malicious. It is simply inheriting our past. In a country as diverse as India, this becomes more than an ethical concern. It becomes a strategic one.
Risk #3 -The Disappearance of Human Accountability
Perhaps the biggest risk of all is this: the gradual disappearance of ownership. Somewhere between algorithm and outcome, accountability becomes blurry. That is dangerous territory for any trust-led industry.
When a customer’s claim is denied, they deserve an explanation that another human being can stand behind -not a vague statement about risk scores and automated assessments. Because customers do not hold algorithms accountable, they hold organisations accountable.
This is why explainable AI is no longer optional. Regulators across the world are already moving in this direction, and rightly so. But beyond regulation lies something even more important: institutional responsibility.
Risk #4 – When Errors Scale Faster Than Humans Can Correct
Traditional operational risks are usually localised. AI risks are different because they scale instantly. A flawed underwriting rule, an incorrect claims recommendation, or a poorly configured customer service bot can replicate the same error across thousands of interactions before intervention is possible. The challenge is no longer simply preventing mistakes. It is designing systems that can detect, contain, and correct mistakes before they spread.
The Path Forward: Humanising the Digital
None of this means we slow down on AI. It means we mature in how we deploy it. At Bajaj General Insurance, our belief has always been simple: technology is the means. Customer trust is the destination. AI will absolutely reshape insurance. It will improve productivity, sharpen pricing, accelerate servicing, and unlock entirely new operating models.
But the insurers who lead the next decade will not simply be those who adopted AI first. They will be the ones who adopted it responsibly -with governance, humility, transparency, and human judgment built into the core.
“The future will belong neither to organisations that resist AI nor to those that blindly embrace it. It will belong to those that combine artificial intelligence with human accountability. Because in the end, customers do not place their trust in algorithms. They place their trust in us.”