By Dr. Rajendra Pratap Gupta
In 2015, I said during a public address, that the future of healthcare lies in the hands of technology and not doctors with the Union Health Minister (MoS) in the audience along with the most celebrated tech CEO from India besides senior clinicians. In little more than a decade , we are settling down with that fact; for millions of Indians, the first “consultation” no longer happens with a doctor.
It happens on a phone; a symptom checker, a chatbot, an AI triage tool that decides whether a complaint is routine or urgent. In a country with roughly one doctor for every 1,500 people, and far worse ratios in rural districts, this is not dystopia. It is arithmetic. AI is filling a gap human capacity simply cannot, and it is doing so at the very first point of contact, where the cost of a wrong call is highest. The fact that is hard to acknowledge by doctors or HCPs is that the first line of enquiry is on internet or internet based technologies. This raises the question, when the algorithm is wrong, who is accountable?
That debate that machines can debate is effectively over. The harder question is what happens when they diagnose badly, when a screening model misses a malignancy on an Indian scan it was never trained on, when a chatbot reassures someone it should have escalated. In each case, harm has a face. Accountability, too often, does not.
This is the “responsibility gap.” A patient is harmed, and liability dissolves across a chain of actors, the developer who wrote the model, the hospital that deployed it, the clinician who trusted its output, the regulator who cleared it. Each can plausibly point to the next. Worse, automation bias means clinicians increasingly defer to the screen, while the system was designed assuming a human would catch its errors. Everyone is in the loop, but no one is on the top of it !
India, to its credit, has stopped pretending this is someone else’s problem. The CDSCO’s 2025 draft guidance on medical device software finally brought AI tools into a risk-based framework, and AI cancer-detection software is now treated as a Class C device requiring validation, approval and post-market surveillance, including mandatory reporting of misdiagnoses. The ICMR’s ethical guidelines name accountability and liability as a founding principle and demand validation on Indian populations rather than borrowed foreign datasets. This is real progress, and ahead of much of the world.
But we must be clear-eyed. Classification is not accountability. Knowing a tool is “Class C” tells a grieving family nothing about who answers for the error. Regulatory approval and legal liability are different questions, and the second one, who pays, who is named, who is barred, remains thinly drawn. A licence is just a permission slip without responsibility.
Accountability has to be designed in, not bolted on after a tragedy. I would offer four principles.
First, traceable accountability. Every clinical AI deployment must have a named, accountable human owner-an “accountable AI officer” in every hospital running these systems, who answers for outcomes, not a vendor in another jurisdiction.
Second, graded autonomy.The higher the clinical stakes, the firmer the human-in-the-loop. A wellness nudge and a cancer call cannot carry the same governance.
Third, the audit trail.Every AI-influenced decision should be logged, explainable and reconstructable. If we cannot reconstruct why the machine said what it said, we cannot assign responsibility, and we are flying blind.
Fourth, local validation as a precondition, not a courtesy. A model that has never seen an Indian patient has no business making an Indian patient’s first decision.
Underlying all of this is a principle I have argued for years: ‘Data First. AI Later’. Most algorithmic failures are not failures of clever code. They are failures of data; Unrepresentative, unaudited, ungoverned. Accountability, in the end, is a governance problem long before it is a technology problem, and it cannot be outsourced to a model card.
The promise of AI as the first line of care is genuine and, for India’s scale, indispensable. But a first line of care must never become the last line of accountability. If a machine is good enough to make the first decision about a human life, then the humans who built it, deployed it and relied on it must be good enough to answer for it.
AI has a long way to go to win trust through accountability. In my view, AI governance is about ethics in building, rigour & transparency in validation, discretion in deployment, and all this must be strictly governed through a defined regulatory framework.
– The author is the creator of the Functional AI Pyramid – AI Maturity Model Framework and is the former advisor to the Union Health Minister, Government of India