By Dr Kavita Kanabar, Associate Professor, HR and General Management, Vivekanand Education Society’s Institute of Management Studies and Research (VESIMSR)
A loan applicant is turned down by a system, not a person. A résumé is filtered out before a recruiter ever opens it. A delivery app quietly raises its price the longer a customer hesitates. None of these decisions was made, in any real sense, by a human being — yet each one changed a life. That gap between who made the call and who has to answer for it has become the central governance problem of the algorithmic economy.
Artificial intelligence currently lies within the daily machinery of commerce: examining job interviewees, recording credit risk, deciding prices in real time, and triaging consumer criticisms. Executives are inclined to appreciate these AI-enabled systems as tend to praise these systems as propellors of speed and reliability, devoid of exhaustion and inconsistency that crawl into human reasoning. That narrative isn’t wrong, but it is not accurate — since it bypasses the difficult question currently confronting regulators, law courts, and executive boardrooms: when the technique comprehends wrong, discriminates, or merely cannot elucidate itself, then who is accountable?
The Black Box Problem
Majority of the machine learning models have been training on massive datasets and deliberate hundreds, sometimes thousands, of variables at once. Because so many factors interact, no single input fully explains a given output — the model isn’t following one traceable rule so much as a web of statistical correlations. Some of those variables end up carrying far more weight than others, often in ways that aren’t obvious even to the engineers who built the system. The result is a process that can be highly accurate on average and still nearly impossible to explain in any individual case — the essence of the “black box” problem.
Four Industries, One Pattern
During the designing of algorithms for hiring, they are trained to reduce the human mistakes by exposing these algorithms to a company’s current workforce — which indicates that they can wind up replicating the same model already available in the staffing records. Some of the schemes silently screen out the applicants whose résumés display employment interruptions, or sort out résumés with specified language patterns, before a recruiter examines them.
In lending, prejudice can crawl in even when safeguarded characteristics are never entered as inputs: a ZIP code or a shopping pattern can act as a surrogate for race or income, letting bias seep in through approaching from the back.Dynamic pricing raises a related concern — regulators have scrutinized platforms, Priceline among them, over pricing algorithms that adjust fees in ways closer to opaque, black-box decision-making than to a straightforward response to supply and demand. And in customer service, systems built to categorize and triage complaints can end up minimizing the human contact that the most vulnerable customers need most, at exactly the moment a human touch matters.
None of this is necessarily intentional. It is more a failure of oversight — a tendency to treat a deployed model as something that now runs itself, and to treat that autonomy as an excuse.
According to one regional bank’s compliance officer, “The algorithm doesn’t have a bank account, a job, or a conscience. So the responsibility has to sit with the human being who put it to use.”
The Law Is Catching Up
Regulators are only beginning to catch up with the pace at which these systems are being deployed. Until recently, a company could adopt an algorithm and answer to little more than its own board; that grace period is ending fast.
The European Union’s AI Act now classifies hiring, credit-scoring, and similar systems as “high-risk,” requiring documentation, human oversight, and the ability to explain individual decisions. In the United States, the picture is more fragmented: a patchwork of state and city laws — including New York City’s audit requirement for automated hiring tools — has emerged in the absence of a single federal standard, leaving companies to navigate different rules in different jurisdictions.
Even where such rules exist, the harder legal question is what “transparency” actually requires. Is it enough for a company to disclose, in general terms, that an algorithm was involved? Or must it be able to explain the specific factors behind a specific decision? Courts and regulators are increasingly leaning toward the latter — a standard that is far harder, and far more expensive, to meet.
What Accountable Deployment Requires
Systematic and organized corporate entities exhibit few common traits. These firms ensure involvement of human in the— a person with proper authority to overrule or depart from the system’s suggestion, not just a rubber stamp. These professionals provide directives for external audits, even if they are costly and laborious as these audits, instead of settling for a tokenish internal sign-off. They create systems and processes that can generate a restricted, case-precise explanation for a particular decision, in place of generic disclaimer claiming that AI was included in the process. And they allocate well-defined ownership: a named senior manager responsible for how a particular system operates.
Everything is associated with cost. The processes such as controls, regular compliance inspections, and grievance procedures slow down processes that were constructed for giving momentum to the system. However, companies now confronted with lawsuits, governing penalties, and reputational injury due to impenetrable automated decisions, are realizing that sometimes alternative do have their own steep price to be paid.
The Bottom Line
Algorithms typically going to render majority of these significant decisions, not smaller number — that trend isn’t moving backward. What is changing is how far society is inclined to accept decisions that anyone can fully justify. The establishments best prepared to withstand that change will not be the units with maximum powerful algorithms, but the ones prepared to reply a short, unpleasant query, simply put: “If this system hurts somebody, who in this association has to elucidate why — and can they?””