By Anuraag Kochhar, Chief Technology Officer of ShepHertz
A few months back, the IT head of a mid-sized financial services firm had put a handful of AI agents into production for back-office work, mainly document processing and a bit of internal query resolution. It was running well, until one morning it was not. One agent had quietly started reaching into data it could technically access but should never have been touching. No malice involved, no breach to report, just a system doing what it was wired to do, because nobody had drawn the boundary with enough care.
There is a lovely moment in the series finale of HBO’s Silicon Valley where Gilfoyle, watching the team’s self-learning AI quietly tear through every encryption protocol in its path, delivers the developer’s old catchphrase with perfect deadpan: “It’s a feature, not a bug.” The joke now plays out in real boardrooms. What worried that IT head was precisely that the agent had been working exactly to specification. That is the awkward bit.
The challenge today is not getting AI into production. It is governing what happens once it is there.
From pilots to production, faster than policy
Indian enterprises in BFSI, healthcare, and IT services have moved a great deal faster than most board papers acknowledge. NASSCOM’s AI Enterprise Adoption Index 2.0, produced with EY in 2024, looked at 500 enterprises across seven sectors covering roughly three-quarters of India’s GDP, placing the country’s aggregate maturity in the high “Enthusiast” band on a zero to four scale. The “Expert” tier remains out of reach for most. Healthcare in particular is patchy on formal risk and ethics frameworks, precisely where ungoverned AI carries the most serious consequences.
IDC has labelled 2025 the year of the AI Pivot across Asia/Pacific, the moment when enterprises move past experimentation and start operating AI as a production capability. The pivot is real enough. The governance infrastructure to match it, for most organisations, has simply not arrived.
What is actually happening on the ground
IBM’s 2024 CEO Study, conducted with Oxford Economics, found that three out of four chief executives believe trusted AI is impossible without effective governance, yet only 39% say they actually have good generative AI governance in place today. There is no shortage of ambition. There is a serious shortage of instrumentation.
Three gaps keep surfacing. The first is visibility. Most enterprises have no reliable, real-time view of which agents are running, what data they are touching, or how decisions are being reached. Governance that cannot see the system it is meant to govern is not governance at all. It is paperwork.
The second is boundary ambiguity. The moment an agent moves across SaaS platforms, internal APIs, and browser sessions, it inherits the weakest effective control in the chain. Data residency requirements and subprocessor obligations rarely point in the same direction. The compliance surface grows faster than anyone is mapping it.
The third is accountability diffusion. IT assumes legal owns the policy. Legal assumes IT owns the controls. The business unit just wants the agent to work. Nobody, in the end, owns the risk. Deloitte’s State of Ethics and Trust in Technology report from October 2023 found more than half of respondents did not know whether their organisation even had ethical standards for generative AI.
In India, regulators are not waiting for industry to catch up. The Digital Personal Data Protection Act, 2023 sets out clear obligations on lawful processing and cross-border data flows. SEBI codified sole responsibility for AI tool use by regulated entities through Regulation 16C in 2025. The Reserve Bank’s Master Direction on IT Governance took effect on 1 April 2024, followed by the FREE-AI framework with its seven sutras for responsible AI in finance. Supervisors now expect traceability, ownership, and evidence.
Five areas every CTO should start with
Working on AI infrastructure for well over a decade has yielded one lesson that shapes everything: governance that is retrofitted does not hold. It has to be a design condition, present from the first line of architecture, not bolted on the night before an audit.
First, sovereignty by deployment. The platform must run where the customer’s law, retention policy, and audit obligations live. This is not simply about geography. It is about who holds the keys, who can read the logs, and what happens when the vendor is compromised. If the answer involves trust in the vendor’s good behaviour, there is no sovereignty. There is a promise.
Second, guardrails as infrastructure. Policy enforcement belongs in the runtime. Every action an agent takes, every tool it invokes, every piece of data it touches should pass through a layer where the rules are codified and violations are blocked rather than merely logged after the fact.
Third, testing for hallucination as an ongoing discipline. Models drift, prompts mutate, retrieval contexts evolve. A serious testing regime needs adversarial prompts, regression suites tied to specific business outcomes, and the discipline to fail a model that has quietly stopped being correct.
Fourth, version control for models. Every model version touching production data needs a clear provenance trail, an evaluation history, and an explicit rollback path. The day a regulator asks which model produced a particular decision, the answer must come in minutes.
Fifth, the model gateway. Multi-model environments are now the norm. A gateway layer handling routing, policy enforcement, cost tracking, and failover is what stops an AI estate from sprawling into unmanaged territory that nobody can fully account for.
The organisations getting this right
PwC’s research finds that organisations with mature AI governance are 1.7 times more likely to use a documented Responsible AI framework, and 1.5 times more likely to maintain a cross-functional governance board. The interesting outcome is not that they are more cautious. They move faster, because clear rules remove the paralysis of uncertainty.
Enterprises scaling AI meaningfully have done three things: mapped what is actually running in production, put IT, legal, and a business owner in the same room regularly, and invested in runtime telemetry rather than policy documents on SharePoint.
Map the agent footprint. Read the FREE-AI framework and SEBI Regulation 16C carefully. Agents are already on the network. As Gilfoyle would say, that is not a bug. The only question worth asking is who is accountable for what they do there.