Book: Human Accountable for The Loop

Somewhere inside an organisation, someone has been told they are responsible for an AI system they cannot stop. They may receive alerts, monitor a dashboard and see their name beside the word owner, yet be unable to inspect the evidence behind an action, change what the system is allowed to do or contain the consequences when something goes wrong. Calling that person the owner may well feel reassuring on paper, but it matters very little if they do not have any meaningful control.

This is the accountability problem at the centre of my new book, Human Accountable for the Loop: How to Keep Ownership, Authority and Control When AI Acts. It asks what human accountability needs to mean once AI moves beyond producing something for us to review and begins taking actions as part of a wider workflow.

The limits of "human in the loop"

Human in the loop has become a standard response to difficult questions about AI. When someone asks who checks the system, who is responsible if it gets something wrong or what prevents it from causing harm, the answer is often that a human will review the output or can intervene if necessary.

Sometimes that does describe a genuine safeguard, perhaps a qualified person may receive the evidence they need, have enough time to examine it properly, possess the authority to disagree and be able to change what happens next. In those circumstances, human review can make a material difference to the outcome.

The problem is that simply placing a person somewhere in the process proves none of this. An approval button may record a click after the consequential choice has effectively been made, while a reviewer facing hundreds of decisions may be able to do little more than confirm what the system has already recommended. A monitor might receive an alert after an action has become irreversible, and someone named as the owner may depend on technical teams or suppliers they have no power to direct.

These arrangements can look convincing in a policy, governance paper or process diagram because the human role is visible. They become much less convincing when you ask what that person can actually see, what decisions they are allowed to make and whether anything changes when they disagree. Human presence only becomes a safeguard when it is supported by the necessary information, time, authority and ability to act.

That distinction matters more as AI moves beyond generating content. Systems can now retrieve information, use tools, coordinate other software, communicate externally and initiate transactions. In legal work, for example, a system might move from drafting a clause to selecting a playbook, updating the matter record, contacting a client or starting the next part of a process.

Reviewing every action becomes less plausible as these workflows become more useful and more autonomous. Adding approval stages to everything can make the process unworkable, but removing those stages while continuing to say that someone remains responsible does not solve the problem either. We need to know whether the organisation can still control what the workflow does, understand why it acted and deal with the consequences it creates.

From reviewing outputs to owning consequences

I developed Human Accountable for the Loop, or HAL, to help organisations examine that wider question. I first published the framework at theloop.legal, which remains its main home and includes an assessment across eight parts of an accountable AI workflow, along with practical tools for identifying where the gaps sit.

The book builds from that framework but has a different job. It looks at why familiar forms of human oversight fail, uses real cases and operating failures to show how accountability breaks down, and works through what applying HAL looks like in practice. Rather than simply describing the framework, it explains the problem behind it and what organisations can do about it.

HAL does not require a person to inspect every automated action. It requires the complete workflow to remain connected to people and organisations with the information, authority and practical ability to explain what happened, control what happens next and correct the position when something goes wrong.

The framework asks eight questions.

1. Who owns the outcome?
   Someone must remain answerable for the workflow from its intended purpose through to its consequences. Owning the model, the software or one stage of the process is not the same as owning what the workflow produces as a whole.
2. What has been delegated?
   Organisations need to be precise about what a system may advise, prepare, decide and execute. These are different forms of authority, even though they are often bundled together under a general statement that the system “assists” a human.
3. What must not happen?
   Some consequences cannot be accepted regardless of how accurate or useful the system may otherwise be. Those limits need to exist in permissions and controls that can be enforced, rather than relying on a prompt or policy to persuade the system not to cross them.
4. Where does the problem go?
   A routine exception, an operational incident and evidence that challenges the original premise of the workflow are different problems. They may require different evidence, different decision-makers and different routes for escalation.
5. What can be proved?
   When something material happens, the organisation needs enough evidence to reconstruct it. That includes what information was available, which rules and permissions applied, what actions were taken, where people intervened and what changed as a result.
6. How will we know the workflow has changed?
   Monitoring cannot stop at measures of model performance. It also needs to cover the actions the system takes, whether its controls continue to work, how people behave around it and what outcomes it produces once used in real work.
7. When must approval reopen?
   Approval should apply to a particular configuration operating under a particular set of conditions, rather than attaching permanently to a product name. A new model, supplier update, permission, data source or use case may change the basis on which the workflow was originally approved.
8. Who carries the consequences?
   Duties, control, evidence and access to a remedy are often divided between providers, deployers, professional advisers and the people affected by the system. Accountability needs to reflect that distribution instead of being assigned to whichever person is easiest to name.

These parts of the framework depend on one another, just giving someone ownership without the necessary decision rights risks turning them into a future scapegoat, while a permission boundary that nobody monitors may stop reflecting what the system can actually do. A hard limit can create another problem if there is no safe fallback, and an escalation route is of limited use if nobody can produce the evidence needed to make a decision.

The same applies to monitoring. A dashboard may present detailed information about the system, but unless those signals reach someone with a reason and the authority to act, it is recording activity rather than helping the organisation control it. Accountability comes from how the controls work together across the workflow, not from pointing to one review stage or one named person.

Accountability without scapegoats

There is a version of human accountability where the system receives the authority and a person is left carrying the blame. The system may make thousands of decisions using permissions chosen elsewhere and services the supposed owner cannot inspect, but when something goes wrong, attention turns to whoever approved the deployment, reviewed one output or happened to be named in the governance document.

That is not meaningful human accountability, a person’s responsibility should reflect the information they received, the time they had, the authority they possessed and whether their intervention could realistically have changed the outcome. The organisation must also remain accountable for the conditions it created through decisions about staffing, procurement, permissions, monitoring and the design of the workflow itself.

Suppliers need to be included in the same analysis, if a provider controls parts of the service, relevant evidence or the ability to make changes, responsibility cannot simply be passed to the organisation using it. The contracts, operating model and technical design need to reflect who can actually do what when a problem occurs.

People remain responsible for decisions they genuinely make and authority they genuinely exercise, but their involvement should not be used to conceal failures created elsewhere in the workflow. Every consequential system needs people capable of explaining, controlling and correcting it, which is different from looking for one person beneath every outcome who can be blamed for it.

Begin with one workflow

AI governance can feel unmanageably large when treated as an organisation-wide programme covering every model, policy and possible future use. I think the more useful place to begin is with one workflow that already acts, or will soon act, in the world.

Start by drawing its boundary from purpose to consequence. Identify who owns the outcome, then list what the system can advise, prepare, decide and execute. It is worth comparing what the policy says about its authority with the permissions that exist in the live environment, because the two are not always the same.

The next step is to test whether the controls work as described. Follow an escalation from its initial trigger through to the person who makes the final decision, then try to cross a supposedly hard boundary and see what actually prevents it. Reconstruct a recent event without asking the people involved to fill gaps from memory, and take something shown on the monitoring dashboard through to the decision it is intended to change.

You can then examine the basis on which the workflow is allowed to continue. That means identifying which changes would invalidate the existing approval, who has the authority to pause the system and what happens to work already in progress if they do. A realistic complaint can also be followed across each supplier, professional and operational boundary to see whether it eventually reaches someone able to explain what happened and provide a remedy.

The gaps usually appear fairly quickly, some of those will be technical, but others will come from contracts, professional duties, organisational structures or the way responsibility has been divided between teams. Many sit at the joins, where each team reasonably assumed that another team was responsible for something nobody had actually taken ownership of.

Why I wrote the book

I wrote Human Accountable for the Loop for leaders, lawyers, risk professionals, product teams, engineers and operational owners deciding when automation should be allowed to act. It takes the framework from theloop.legal and applies it to the messier setting of real organisations, where authority is distributed, suppliers control important parts of the system and the person described as the owner may be several steps removed from what the technology actually does.

This is a practitioner book rather than a prediction about what AI might eventually become. The cases and operating failures help explain the problem, while the framework and accompanying workbook give readers a way to examine a live workflow and decide what needs to change.

The aim is not to sort systems into fixed categories marked safe or unsafe. Most real decisions are more specific: proceed with the workflow, narrow its scope, constrain particular actions, introduce it in stages, increase monitoring, pause it or retire it. Which choice makes sense will depend on what the system can do, what evidence supports its use and whether the organisation can still control the consequences.

No evaluation can recreate every condition a system will meet once it is live. People will use it in unexpected ways, suppliers will release updates, data and policies will change, and controls that worked separately may behave differently once combined. The decision to deploy should therefore never become a permanent commitment to continue, particularly when the evidence supporting the original decision has changed.

An accountable organisation needs to notice when the case for keeping a system live has weakened and be willing to pause it. Sometimes the most important thing an owner can do is acknowledge that the organisation no longer knows enough to keep the workflow running safely.

Human Accountable for the Loop: How to Keep Ownership, Authority and Control When AI Acts is available on Leanpub. Read the book: https://leanpub.com/human-accountable-for-the-loop