The more autonomous AI becomes, the more experience it demands

By Vinay Chhabra, MD and Co-founder, AceCloud (a brand of RTDS)

There is a quiet assumption spreading through boardrooms and engineering teams right now: that as AI systems become more capable of acting on their own, the need for experienced human judgement decreases. The logic seems reasonable on the surface. If the system can decide, why do you need people to make those decisions?

I am not sure that assumption holds. And the more I see how these systems are being deployed, the less confident I am that most organisations are asking the right questions about it.

What is changing is not the need for experience. What is changing is how quickly the consequences of not having it arrive.

For most of the past few years, AI has functioned as a tool. It generated output, surfaced patterns, and supported decisions. Humans remained in the loop at every meaningful step. That model is now shifting faster than most organisations realise. Systems are beginning to take actions inside live environments, interacting with other systems, triggering processes, and influencing outcomes without waiting for human validation at each step.

The burden has moved. It is no longer using AI well. It is operating. That distinction is easy to overlook in theory. It becomes obvious very quickly in practice.

Where autonomy changes the risk equation

What becomes visible first is not capability, but consequence. The moment systems begin to act, the margin for error collapses.

In earlier systems, something could go wrong and still remain contained long enough for someone to step in. There was time to detect, question, and correct. Autonomous systems remove that buffer. They act, and they keep acting. A single incorrect decision does not stay local. It triggers follow-on actions before anyone has the chance to understand what has happened.

This is not a theoretical concern. It is already happening.

I read this interesting report by Euronews (April 2026): PocketOS, a SaaS platform serving car rental businesses, lost its entire production database and all backups within seconds. An AI coding agent, attempting to resolve a routine credential mismatch, accessed a separate API token with broader permissions and deleted the underlying storage volume supporting the database. No confirmation was requested. No human was involved. When questioned, the system was able to list the rules it had violated. It had not malfunctioned. It had acted within the limited logic it was operating under.

The issue here was not that the system acted. It was that it acted without the boundaries, controls, and judgement required for that level of autonomy.

Autonomous systems create leverage, but they also raise the cost of weak operating assumptions. What appears efficient at the surface often carries a much tighter margin for error underneath.

The problem that looks like progress

The more seamlessly an AI system performs, the less organisations tend to question it.

This is not carelessness. It is a natural response to consistent results. But it creates a gap between perceived reliability and actual reliability. This pattern is already visible in how large organisations are approaching AI-led automation.

Klarna, a global fintech company best known for its buy-now-pay-later services and large-scale digital commerce platform, has been among the most aggressive adopters of AI in customer operations. In 2024, the company positioned its AI assistant as capable of handling the workload of hundreds of customer service agents across multiple languages. The system appeared efficient and scalable, reinforcing confidence in large-scale AI-led automation. By 2025, customer satisfaction had declined, and the company began reintroducing human agents. The CEO acknowledged that the organisation had gone too far in replacing human oversight.

One observation that stood out to me while reading through Klarna’s experience, including analysis published by MLQ.ai, was that the challenge was not scale. The system performed at volume. What it lacked was the contextual judgement required for more complex interactions.

Autonomous systems do not signal where they are weak. They perform well enough that organisations stop looking closely. By the time the gap becomes visible, it is rarely small.

What ‘experienced’ means in this context

Experience here is often misunderstood. It is not tenure. It is not familiarity with a specific model or platform. It is the ability to understand how systems behave when they move beyond controlled conditions.

In infrastructure terms, it is the difference between someone who reads a dashboard and someone who knows when a seemingly stable system is already drifting. In operational terms, it is the ability to recognise when a system is producing the right output for the wrong reasons.

What I have observed, both in how we approach building and operating systems and in how enterprises across India are approaching AI adoption, is that capability is being deployed faster than judgement.

Experienced teams are not there to fix the system after it fails. They are there to design the system, assuming it will fail. They define boundaries. They anticipate edge cases. They understand how behaviour changes under load, under variation, and under pressure.

Without that layer, autonomy does not scale safely. It scales risk.

The accountability gaps most organisations haven’t solved

As systems begin to take action, accountability does not disappear. It becomes ambiguous, and ambiguity is where problems compound.

When something goes wrong, the issue is rarely that nobody cares. It is that nobody has a clear line of ownership over how the system behaves. Decisions are distributed across design choices, configurations, data inputs, and system interactions. When those decisions combine to produce an outcome, tracing responsibility becomes difficult.

What I have seen in these situations is not confusion at the moment of failure, but delay. Time is spent understanding what happened, who should intervene, and how to prevent recurrence. By then, the system has already moved on, often repeating the same behaviour. This is not a gap in intelligence. It is a gap in how the system is owned.

The real competitive advantage in the next phase of AI

The organisations that struggle in this transition will not be the ones that failed to adopt AI. They will be the ones who adopted it without building the operational understanding required to manage it.

They will deploy systems that appear stable until they are not. And when they fail, the impact will not be gradual. It will be sudden, compounded, and difficult to unwind.

This becomes particularly relevant in markets like India, where cost sensitivity, regulatory expectations around data, and operational maturity leave less room for error. What I see consistently is that the organisations moving fastest are not always the ones building the depth of judgement required to support what they are deploying.

We have seen this pattern repeat itself in technology. Early momentum creates visibility, but sustained advantage comes from the ability to operate systems reliably under pressure. The organisations that succeed are not the ones that move first. They are the ones that understand what they are running.

The next phase of AI will not reward speed alone. It will reward organisations that understand what they are running, how it behaves under pressure, and where it is likely to fail.