Who Decides When AI Needs Human Help?
As AI systems become capable of completing increasingly complex work, we need to think more carefully about who decides when they're allowed to act, when they need supervision and what happens when they get that decision wrong.
My youngest son is almost two. When he's playing at the park, he makes all sorts of decisions for himself. He'll choose what to play on, attempt things he couldn't manage a few weeks ago and occasionally decide that something is beyond him. Sometimes he asks for help, although he's equally capable of becoming rather determined that he can manage something on his own.
I'm happy to let him make many of those decisions, I mean it's how he learns and I wouldn't want to spend every visit to the playground standing beside him giving instructions. There are, however, decisions I don't expect him to make, particularly around whether something is safe enough to attempt without help. I'll make that judgement based on what he's doing, what might happen if something goes wrong and how well equipped he is to deal with it.
His confidence doesn't necessarily have much bearing on mine.
The comparison only goes so far, and I'm certainly not suggesting that sophisticated AI systems have the reasoning abilities of a toddler. Modern models can recognise uncertainty, identify inconsistencies, use tools to verify information and, in some circumstances, make useful assessments of their own performance.
What interests me is the question of whether the system doing the work should also determine how much supervision that work requires. We're increasingly designing agents that can make decisions, interact with software and progress work without constant human involvement, often with instructions to recognise when they're uncertain and ask for help.
That seems sensible, and in many cases it works, but there's a more difficult question underneath it though, because an agent that misunderstands the situation it is dealing with may also misunderstand why that situation warrants intervention.
Two responsibilities within the same system
Much of the discussion around agentic AI currently centres on capability. Can an agent conduct research, analyse documents, use applications, interact with APIs, recover from errors and continue working towards a goal without someone directing every step?
Increasingly the answer is yes, although reliability varies considerably depending on the task, the environment and the consequences of getting something wrong.
An agent might be asked to review a collection of corporate documents, identify outstanding requirements and prepare a regulatory filing. During that process, it could encounter incomplete information, contradictory instructions or a document that doesn't fit the expected pattern. We might instruct it to flag these issues, ask questions or stop when it cannot establish a reliable answer.
Those are reasonable expectations, particularly as models become better at recognising uncertainty and dealing with unfamiliar situations. The difficulty is that we're asking the agent to perform the substantive work while simultaneously judging whether its understanding of that work is sufficiently reliable to continue.
In practice, this means the system may be responsible for identifying the very circumstances in which its own judgement should no longer be relied upon.
Humans do something similar all the time. Lawyers decide when to seek specialist advice, engineers decide when a change needs additional review, and doctors decide when to refer a patient. The difference here though is that those decisions exist within established professional, organisational and regulatory structures that define responsibility, escalation and authority.
When we build an AI agent, we have to decide how much of that structure we're going to recreate in software and how much we're prepared to leave to the model.
There's nothing inherently wrong with a model assessing its own work, it may well identify an error, recognise a missing authority or discover that it lacks the information required to proceed. These are super valuable capabilities that we should make use of.
The concern is what happens when self assessment becomes the principal mechanism determining whether any other oversight is necessary. If an agent fails to recognise an important exception, it may have no reason, from its own perspective, to escalate it.
The failure to recognise a problem and the failure to request supervision can then become the same failure.
Competence, correctness and authority
I think there are three questions worth separating when we decide how independently an AI system should operate.
The first concerns competence: whether the system can reliably perform the type of work we've given it. We can establish this through testing, experience and continued monitoring, although competence is rarely universal. A system might perform extremely well on routine cases while struggling with particular exceptions.
The second concerns correctness, which is about the individual instance of work. Even a highly competent system can produce an incorrect result, and we need some means of establishing whether the evidence supports its conclusion.
The third is authority, and I think this is where the conversation becomes particularly interesting...
Let's picture an agent that can examine a corporate ownership structure, identify beneficial owners and prepare the relevant regulatory documentation. We might have considerable evidence that it's competent at performing this work, and in a particular case its conclusion might be entirely correct.
Neither of those things necessarily establishes that it should be authorised to update an official register, make a representation to a regulator or submit a filing on behalf of a client. Those permissions depend on the organisation, the client relationship, applicable law and the authority delegated to the person or system performing the action. They don't arise simply because the system has demonstrated that it can do something accurately.
There's also a distinction between an agent having permission to perform an action and that action being appropriate in a particular case. A system might have access to a filing API, for example, without having received approval to submit a particular return.
We sometimes talk about autonomy as though it were a measure of model capability, with more capable systems naturally deserving more independence. Greater capability certainly creates opportunities to delegate more work, although it doesn't remove the need to decide which actions the system should be allowed to take.
An extraordinarily capable agent could still be operating outside its authority, much like a maverick detective.
This matters because it gives us a way to think about supervision that isn't entirely dependent on how confident the model appears to be. Confidence may help us understand when a system needs additional information or verification, but it cannot establish authority on its own.
How good are models at recognising their own limitations?
I don't think the answer is to dismiss self-assessment. There is encouraging research suggesting that models can recognise concerning behaviour in their own outputs, particularly when they're given an opportunity to evaluate their work separately from producing it.
In October 2026, Anthropic published research into using Claude to assess conversations after generating responses. Its self-assessment approach identified 78% of conversations flagged as potentially harmful by a stronger classifier, rising to 94% for high-priority cases, at a reported 1% false-positive rate.
Those are meaningful results, although the research concerns misuse detection rather than legal reasoning or autonomous decision-making. We shouldn't assume that the same performance would carry across to identifying an incorrect regulatory interpretation or recognising that a filing requires additional approval.
What the research does demonstrate is that self-assessment can provide an additional source of information about a model's behaviour. Anthropic's approach uses it as one part of a wider detection system rather than treating it as a replacement for other safeguards.
An agent recognising that it shouldn't proceed is a positive outcome, we want systems to identify uncertainty, question inconsistent evidence and seek clarification where appropriate, rather than blindly continuing towards a goal like some overconfident lawyer.
The harder question for me though is what to do when the agent believes everything is fine.
A model's assessment that it can continue may be well founded, but the absence of a detected problem doesn't establish that there isn't one. This is particularly relevant where the consequences of an undetected error are much greater than the inconvenience of an unnecessary escalation.
For routine document classification, we might accept some mistakes and rely on subsequent checks. For an irreversible external action, we'd probably want stronger assurances, since the underlying model could be identical in both situations, while our expectations about supervision would be very different.
That suggests the supervision decision needs to account for the nature of the action as well as the model's assessment of its own performance.
Would another AI provide better supervision?
One increasingly popular approach is to introduce a second model that reviews the first agent's work. It might examine the evidence, challenge the proposed conclusion, check for missing information or decide whether a particular action should proceed.
There's a lot to like about this approach, particularly where the reviewing system can independently examine the relevant information rather than simply accepting the original agent's account of what happened. It also introduces some rather interesting problems.
Two models relying on the same incomplete collection of documents may reach the same incorrect conclusion. A reviewer might be influenced by the first agent's explanation, particularly if that explanation is coherent and apparently well supported. Even where different models are used, they may share weaknesses arising from similar training data, reasoning patterns or assumptions about the task.
Anthropic's research into automated alignment auditing offers a useful illustration. Its auditing agents have successfully identified concerning model behaviours, and using multiple investigators improved performance in some experiments. The researchers also observed weaknesses, including agents becoming attached to early hypotheses and failing to explore alternatives sufficiently.
That isn't an argument against using AI to supervise AI. In fact, I suspect we'll see considerably more of it, especially as the volume and complexity of agent activity makes comprehensive human review increasingly impractical.
It does suggest that we need to be quite careful about what we mean by independent supervision. If the reviewer has access to the same evidence, uses similar reasoning and relies on the original agent's description of its actions, introducing another model might add less protection than we'd expect.
A genuinely useful review mechanism needs access to information that allows it to challenge the original decision. Depending on the task, that might involve independently retrieving legal authorities, checking source documents, validating calculations or comparing the proposed action against defined permissions and approval requirements.
Sometimes the most effective supervisor may not be another language model at all. It could be an ordinary software control that checks whether a required approval exists, whether the relevant entity has been identified or whether a filing date falls within an acceptable period.
These are established software engineering techniques, and there's no particular reason to abandon them because the component performing the work happens to be intelligent.
The point at which an agent is allowed to act
So let's consider an agent supporting the preparation and submission of a regulatory return.
It retrieves information from internal systems, examines previous filings, reconciles figures, identifies missing information and prepares a draft submission. Much of that work could be completed without a person supervising every individual step, provided the process is appropriately designed and tested.
Eventually, however, the agent reaches the point where the return could be submitted. This changes the nature of the decision, as retrieving a document, extracting a value and submitting information to an external regulator may all be steps within the same workflow, but they carry very different consequences.
If the agent makes a mistake while preparing the draft, there may be opportunities to identify and correct it before anything leaves the organisation. Once information has been submitted, correcting an error may involve further regulatory processes, client communication or other consequences.
I would want the system's architecture to reflect those differences.
An instruction telling the agent not to submit anything without approval is useful, and capable models will often follow it correctly. I would be considerably more comfortable, however, if the system responsible for accepting the submission also required evidence of that approval.
The agent could still do the work, prepare the submission, explain any outstanding issues and request authorisation. What it couldn't do is decide that approval was unnecessary simply because it considered its own output reliable.
There is an important distinction here between a control that asks the model to behave in a particular way and a control that limits what actions are actually available to it. The latter doesn't require us to assume that the model will always recognise when the former applies.
This becomes more interesting when the workflow contains several different approval points. Updating an internal working record might require a lower level of assurance than changing an authoritative client record, while submitting a regulatory return could require approval from someone with a particular role or delegated authority.
Those boundaries need to be informed by how the organisation actually operates, rather than imposed as a generic rule that every consequential looking action must be checked by a human.
There will also be situations where a system can safely perform consequential actions without individual approval, perhaps because the activity is tightly constrained, reversible and supported by strong verification. The point is to make that a deliberate decision, supported by evidence, rather than something the agent determines for itself each time.
What human involvement actually means
I've written previously about human-in-the-loop, human-on-the-loop and human-accountable-for-the-loop and I think the distinction becomes increasingly important here as AI systems become more capable.
Human-in-the-loop is usually understood as someone participating in a particular decision or approval. Human-on-the-loop involves supervising a process, monitoring its operation and intervening when necessary. Human-accountable-for-the-loop is about responsibility for how the overall system has been designed, deployed and operated, including the actions that take place without direct human involvement.
These approaches aren't mutually exclusive, and different parts of a workflow may justify different forms of control but really the challenge is making human involvement meaningful.
If a lawyer has to approve every minor action an agent performs, the process may become slower and more cumbersome than completing the work directly. We also risk creating an environment where approvals become routine, with people accepting recommendations because they've already approved dozens of similar ones.
On the other hand, saying that a lawyer remains responsible for the output doesn't tell us much about how the system operates. Responsibility is difficult to exercise if the person concerned has no visibility into the decisions being made, cannot understand the evidence behind them and has no practical opportunity to intervene.
A meaningful approval should give the reviewer enough information to make an informed decision. That may include the underlying evidence, unresolved questions, relevant legal authority, the proposed action and the consequences of proceeding.
It also requires some thought about what happens after approval. If the underlying information changes, the approval may no longer be valid. If an agent is authorised to perform a particular action, that doesn't necessarily mean it has continuing authority to perform materially different actions later.
These are familiar governance and software engineering problems, although agentic systems make them more prominent because the software itself is now making decisions about how to complete work.
The UK National Cyber Security Centre's 2026 guidance on adopting agentic AI reflects this broader concern, emphasising the need for organisations to retain meaningful oversight and control over systems that can act with increasing independence.
I think the direction is sensible, although I'd like to see the industry become more precise about what meaningful oversight looks like in practice.
Testing the supervision, rather than just the agent
There's another consequence of treating supervision as part of the system's architecture: We need to test whether it works.
Most evaluations of AI systems focus on the quality of the work produced. We might measure whether an agent answers legal research questions correctly, extracts the right information, identifies relevant clauses or completes a workflow successfully.
Those measures are useful, but they tell us relatively little about what happens when the system encounters something it shouldn't handle independently.
Imagine testing a regulatory filing agent against a collection of ordinary scenarios. It performs well, retrieves the right information and prepares accurate submissions. We might reasonably conclude that it's competent at the task.
I'd also want to introduce scenarios in which the appropriate behaviour is to stop, request clarification or refuse to proceed:
- What happens if two source documents disagree about an entity's classification?
- What if an approval was previously granted but has since been withdrawn?
- What if a document retrieved from a trusted repository contains an instruction telling the agent to ignore its usual approval process?
There are also less obvious cases, suppose the agent has all the information required to prepare a filing, but the evidence indicates that no filing is actually required. Can it recognise that the correct outcome is to take no action, or does the goal of completing the workflow encourage it to produce and submit something anyway?
These scenarios test a different a completely capability from ordinary task completion.
They also provide an opportunity to compare different supervision arrangements. We could run the same scenarios with the agent assessing its own work, with an independent reviewing model and with explicit controls governing particular actions.
I'd be interested in how often each arrangement fails to identify a situation requiring intervention, how frequently it interrupts people unnecessarily and whether an agent can proceed when the required authority is absent.
The balance matters. A supervision system that escalates everything may prevent some errors, but it would also undermine the purpose of using agents. One that rarely interrupts anyone might appear efficient while allowing serious mistakes to pass unnoticed.
There isn't a universal threshold that works for every kind of legal work. The acceptable balance depends on the activity, the consequences of failure and the opportunities to detect and correct an error.
I think this is where we should direct more of our evaluation effort as agentic systems move into production.
Where I think this is heading
As models improve, I expect them to become more reliable at recognising their own limitations. They'll also become capable of completing longer and more complicated workflows, which means we'll increasingly want to give them greater freedom to decide how work gets done.
All of which creates a tension... the better an agent becomes at performing work independently, the less practical it may be to supervise every individual action. Yet the more actions it can take, particularly across multiple connected systems, the greater the potential consequences of an undetected mistake.
We need supervision arrangements that can accommodate both of these developments.
Some decisions will remain subject to explicit approval requirements, regardless of how confident or capable an agent becomes. Others might be handled through independent verification, monitoring, sampling or controls that restrict what the system can do. As we gather evidence about performance and failure rates, there may be good reasons to change those boundaries.
I'd expect the most effective approaches to combine model-based judgement with conventional software controls, clear allocation of authority and continuing evaluation. The particular arrangement will vary according to the work, and the boundaries should be open to revision as both the technology and our understanding of its behaviour improve.
What I'd be cautious about is allowing those boundaries to emerge entirely from the agent's own assessment of what it can manage. Ultimately, organisations deploying agents are making decisions about delegated authority. Model capability should inform those decisions, along with evidence about reliability, the availability of safeguards and the consequences of mistakes.
We should absolutely build agents that recognise when they need help. That ability will become increasingly valuable as the work we delegate grows more complex, though we also need to design systems that remain safe and accountable when they don't recognise it.