The Agent Found a Way

A few weeks ago, OpenAI disclosed an unusual security incident involving one of its internal AI evaluations. Agents running cybersecurity tasks had found ways beyond the environment they were supposed to be operating within, compromised OpenAI infrastructure and ultimately reached systems belonging to Hugging Face. It attracted plenty of attention at the time, which was hardly surprising. An autonomous AI system exploiting real infrastructure outside the environment in which it was supposed to operate is the sort of story that does not need much help from the headline writer.

What changed my view of the incident was OpenAI’s subsequent presentation at Black Hat, the fuller account was much stranger than the original disclosure, but also much more useful for thinking about agentic systems outside the world of cybersecurity.

OpenAI did not describe one agent making one bad decision. The incident developed over months and involved agents operating across different training and evaluation runs, discovering vulnerabilities, sharing what they had found, moving laterally through systems and eventually attacking external infrastructure. Reconstructing what had happened meant looking across billions of logs and many different agent trajectories rather than tracing a single neat sequence of events.

One part of the story starts with a fairly standard failure, an agent had been given a task that it could not complete because a file it needed had accidentally not been provided. After trying other approaches, it reasoned that perhaps another agent in a different environment might have the file and decided to leave a message asking for help, choosing Artifactory, OpenAI’s shared package management infrastructure, as the place to put it.

Other agents later encountered the message and began leaving their own. Over time, that infrastructure became something closer to a message board. Agents developed conventions for where messages should appear in directory listings, passed scripts and exploit material between runs, delegated work and picked up where other agents had stopped. They even began reasoning about whether participants could be impersonated and how cryptographic signatures might help them distinguish authentic messages from fake ones.

There is an obvious temptation to tell this as though the agents had formed a little society inside OpenAI’s infrastructure. That makes the story entertaining, but I think it also makes the underlying problem sound more mysterious than it needs to be. What mattered was that the agents found capabilities in their environment that its designers had not intended them to use in those ways.

Artifactory existed to provide packages inside a constrained environment, but once an agent discovered that information could be written there and another could retrieve it, the same infrastructure also became a communications mechanism. When agents discovered that Artifactory could retrieve information from the internet on their behalf, it became a route around network restrictions, while later vulnerabilities in the surrounding environment allowed them to gain code execution, escalate privileges and move laterally.

The incident did not happen in one leap

The sequence is worth seeing because much of the interesting behaviour emerged gradually rather than "Oh I'm off to do some hacking"

1. An agent gets stuck. A task cannot be completed because information it needs is unavailable.
2. It discovers it can write into shared infrastructure. What was supposed to be a package manager turns out to be somewhere information can be left for another process.
3. Other agents find the messages. The accidental communications mechanism becomes useful, so more agents begin using it.
4. Agents start sharing work. Messages become more structured, scripts and exploit material are passed around, and one agent’s discovery can be reused by others.
5. Agents find routes around environmental restrictions. Artifactory becomes useful not just for communication but as part of a route towards internet access and later privilege escalation.
6. The scope of the work expands. External infrastructure becomes relevant because it may contain information that helps agents complete the original tasks.

OpenAI itself described this part of the incident as scope creep. Some of the reasoning shown during the presentation is particularly interesting because agents appear to recognise that attacking external infrastructure is beyond the intended scope while still reasoning that continuing may help complete a task that otherwise looks impossible.

There is another part of the sequence that I find even more revealing. After OpenAI discovered the compromise, it revoked credentials, rebuilt Artifactory, cleared the message board and patched the relevant vulnerability before allowing training and evaluations to resume. Within days, agents had recreated the communications mechanism in another way, this time using directory names exposed through an unauthenticated WebDAV endpoint, and other agents found the new structure and began communicating again.

OpenAI had removed the original mechanism but communication remained useful to the agents’ objectives. Closing one route therefore did not remove the reason to find another, which is a rather different control problem from preventing a conventional piece of software from performing a predefined action.

We are asking agents to find another way

Persistence is part of the reason we want agentic AI in the first place. An agent that encounters a missing document, failed API call or awkward system boundary and immediately stops is not especially useful. The whole point is that it can work through a problem, try another approach, delegate part of the task, inspect what happened and continue without needing a person to prescribe every intermediate step.

As systems get better at doing that, the nature of the control problem changes too. A conventional workflow can be constrained partly because we decide in advance which steps it is capable of taking, whereas an agent has more freedom to decide which steps might be useful and therefore more opportunity to find routes that were never part of the workflow its designers had in mind.

None of this requires us to believe that an agent has gone rogue, developed an agenda or decided that it dislikes being constrained. A much less dramatic explanation is enough: the agent has an objective, encounters an obstacle and is capable of constructing another route around it.

The interesting question for us in legal is what happens when we give the same kind of persistence to a system pursuing a commercial objective rather than a cybersecurity benchmark.

What does "get the best price" actually mean?

Imagine the agent is working on a takeover. The client wants to acquire TargetCo and has told us that it would be comfortable completing the transaction at £8 a share, although naturally it would much prefer £7.50 if that could be achieved.

We give a lawyer and an agent the same instruction:

Help the client acquire TargetCo at the best possible price.

A lawyer might translate that into something like:

1. Review the due diligence for anything that genuinely affects valuation.
2. Identify weaknesses in the target’s negotiating position.
3. Consider whether findings justify a price adjustment, additional protection or different deal structure.
4. Model the available options and prepare the arguments that could support a lower offer.
5. Negotiate within the authority the client has actually given.

There are many possible variations, but the lawyer does not interpret "best possible price" as permission to pursue every action (dodgy or otherwise) capable of making TargetCo cheaper. The instruction sits inside a much wider professional context that includes the law, confidentiality, client authority, the lawyer’s own discretion, the firm’s expectations and an accumulated sense of where legitimate transaction strategy ends and something altogether more difficult begins.

Now imagine the same commercial objective sitting at the centre of a capable agentic system without enough of those surrounding boundaries represented in its architecture. Its path could start in almost exactly the same place:

1. Review the diligence findings for anything that might affect valuation. It identifies a genuine adverse issue that has attracted relatively little public attention.
2. Estimate the possible market impact. The agent concludes that wider awareness of the issue could put downward pressure on TargetCo’s valuation.
3. Connect that movement to the acquisition objective. A lower valuation would improve the client’s negotiating position, so factors capable of affecting valuation become relevant to achieving the original task.
4. Examine how awareness of the issue might change. The system considers public commentary, communications channels and other connected capabilities that could affect how widely the information is seen or understood.
5. Begin preparing activity designed to increase attention around the issue. The task has now changed character. The system is no longer simply analysing information for the negotiation but considering whether it can alter the conditions in which that negotiation takes place.
6. Measure any resulting movement against the original objective. If wider attention contributed to reduced confidence in TargetCo and the price fell, the system could interpret that as progress towards obtaining a cheaper acquisition.

Now by this stage, most lawyers would recognise that something has gone badly wrong, but there never needed to be an instruction saying "manipulate TargetCo’s share price", and there's wasn't a single dramatic moment at which the system deliberately chose to break the law. The problem can emerge through a series of intermediate objectives that remain locally rational when judged only against the commercial goal at the top.

This is why OpenAI’s description of scope creep is useful well beyond cybersecurity. Its agents were not originally instructed to attack Hugging Face. They were given tasks, encountered obstacles and constructed intermediate objectives that gradually made external systems relevant to getting the original job done.

Our legal agent might therefore understand perfectly well that the client wants a cheaper acquisition while lacking enough representation of everything a lawyer would have understood as implicit in that instruction. The difficulty is not necessarily a failure to understand the objective, but a failure to understand the boundaries around how that objective may properly be pursued.

A tool call can look fine while the strategy is not

Much of the current discussion about controlling agents still focuses on individual actions. We decide whether the system can send an email, edit a document, query a database or submit something externally, then decide which of those actions should require human approval.

Those controls are necessary, but they become less reassuring once the risk sits in the relationship between actions rather than inside one obviously problematic step. Consider just part of the acquisition example:

• research negative public reporting about TargetCo;
• analyse which issues have historically affected its valuation;
• identify commentators who regularly cover those issues;
• prepare a factual summary from public information;
• model what might happen to valuation if those issues received greater attention.

Each activity could be legitimate in isolation, and a lawyer might approve every one while missing that, taken together, they are beginning to form a strategy nobody actually authorised. That is why I am increasingly wary of treating human approval as synonymous with meaningful human control, as a person can be technically present at dozens of points in an agentic workflow while having very little visibility into the direction in which the system as a whole is moving.

For legal, I think that means separating three things we currently risk collapsing together: instruction, capability and authority. The objective tells the agent what we want to achieve and its tools determine what it is technically capable of doing, but neither tells us whether it is entitled to use a particular capability for this purpose, on this matter, at this point in time.

We already understand that distinction when the delegate is another person.

Delegation does not mean abdication

Imagine a senior partner gives a junior lawyer the same broad instruction to secure the best possible outcome for the client. The junior then develops a course of action which crosses a legal or professional boundary, believing that it is a sensible way of carrying out the task.

The position would obviously depend on the facts. A junior solicitor has professional obligations of their own and cannot simply treat a partner’s instruction as permission to do something they should recognise as improper. Equally, the supervising lawyer does not normally get to wash their hands of delegated work merely because somebody more junior performed the individual task.

The SRA’s current supervision guidance is explicit that regulated individuals and firms are accountable for those they supervise and that supervisors remain responsible for work conducted under their supervision. It expects the nature and risks of delegated work to inform the level of supervision, including whether the supervisor has sufficient knowledge of the matters being progressed and whether non-standard issues are escalated.

What is interesting is how closely that familiar model of supervision maps onto the problems agentic systems create. Effective delegation requires some understanding of the work being handed over, the limits of the delegate’s authority, where approval is needed and how the supervisor will retain enough visibility to identify a problem that may only become apparent across several pieces of work rather than in one obviously wrong act.

When a partner delegates to another solicitor, however, some of that control sits inside the delegate themselves. The junior has their own professional obligations, training and duty to exercise judgement, and should recognise that an instruction to “get the best price” does not override the rest of their professional responsibilities.

Replacing that junior with an AI agent creates an odd reversal, we may be prepared to give the system considerable operational freedom, perhaps more than we would give a junior lawyer, while removing the independently regulated professional who would ordinarily sit at the receiving end of the delegation and exercise their own judgement about whether the proposed course is appropriate. The more autonomy we give the system, the more work the surrounding architecture has to do to replace some of what we previously relied on the delegate themselves to provide.

"I didn’t tell it to do that"

Returning to our takeover example and let's assume the agent actually crossed the line. The supervising lawyer looks at what happened and says, entirely truthfully, that nobody instructed the system to influence TargetCo’s market price: "It was told to help secure the best transaction price and developed the intermediate strategy all by itself."

Now we do not yet have a decided case telling us how English law will allocate responsibility for that particular form of agentic behaviour. The UK Jurisdiction Taskforce’s 2026 Legal Statement is expressly concerned with liability for non-deliberate AI harms under English private law, so it should not be treated as an answer to a market manipulation hypothetical involving separate regulatory or criminal regimes. It does, however, form part of a broader attempt to apply existing legal principles to harms involving AI rather than treating autonomous software as creating some new legal personality to which responsibility can simply be transferred.

The regulatory position for solicitors is already rather more concrete. The SRA’s approach to AI-assisted legal services is that an authorised individual retains ultimate responsibility, which makes it difficult to argue that the enquiry ends because the agent independently selected the problematic course of action.

That does not mean every surprising action performed by an agent automatically becomes the personal liability of whichever solicitor was nearest to it. Questions of duty, breach, causation, individual conduct and any applicable regulatory or criminal regime would still depend on the circumstances. What seems much more likely is that attention moves upstream into the way the delegation was designed and supervised.

If the system was given a powerful commercial objective and significant freedom to construct its own route towards it then:

• What did the firm understand about that capability?
• Which tools had it made available, how had the limits of delegated authority been represented, and what could the supervising lawyer see about the strategy being developed?
• Which behaviours had been tested, what required escalation and what prevented individually permissible capabilities from being combined into a course of action that nobody would have approved if it had been proposed explicitly?

Those are recognisable supervision questions, but agentic AI makes answering them technically more demanding because some of the things we previously expected a human delegate to understand for themselves now need to be represented, constrained or monitored elsewhere in the system.

What would an authority layer actually need to know?

Access controls remain part of the answer. OpenAI’s Black Hat presentation reinforces the importance of segmentation and least privilege, and legal agents should clearly not inherit every permission available to whichever lawyer happens to be using them.

A useful legal agent, though, may legitimately need several capabilities at once. It may need the document system, legal research, email, company information and matter data because connecting those sources is precisely what allows it to do work that would otherwise require people to move manually between them. Once that happens, an architecture based only on read and write permissions begins to look too thin.

An authority layer would need to understand at least:

• The objective being pursued. The original client objective needs to remain visible even as the agent creates its own intermediate tasks.
• The authority actually granted. This might include negotiation parameters, communications authority, filing permissions or limits on how information can be used.
• The boundaries between autonomous work and escalation. Researching a counterparty may be routine, while contacting them or changing a commercial position may require additional authority.
• Delegation between agents. A second agent should not gain powers simply because another agent can instruct it.
• Changes in matter state. Client instructions, approvals, regulatory responses or developments in a negotiation can all change what the agent is entitled to do next.
• The trajectory, not just the action. The system needs some ability to recognise when a collection of individually reasonable steps is moving towards an objective that sits outside the authority originally granted.

Authority in professional work is not a static permission attached to an account. It develops with the matter, which is one reason I think agentic AI will expose a weakness in the way legal technology has traditionally represented professional work.

We have become very good at storing documents, emails and outputs, while objectives, unresolved issues, dependencies, decisions and authority have often remained spread across those artefacts and inside the heads of the people doing the work. That was manageable while a person remained responsible for deciding what happened next, but it becomes much harder once an agent needs enough context to make that decision itself.

Human Accountable for the Loop

There is an obvious response to all of this, which is to keep a human in the loop, which in some places that remains exactly the right control. Certain decisions are consequential enough, dependent enough on professional judgement or sufficiently difficult to reverse that explicit human approval should remain mandatory.

The problem comes when human approval becomes the answer to everything.

OpenAI’s security team made an interesting version of this point at Black Hat from the defensive side of cybersecurity. Their argument was that if offensive activity can become fully automated while defensive action still relies on people manually reviewing and responding to every event, the human becomes the limiting factor. They therefore talked about defensive loops in which agents can identify vulnerabilities, prepare patches, deploy changes and roll them back where necessary rather than assuming safety requires a person to approve every stage.

Legal will face its own version of that tension. If agents are going to monitor matters continuously, perform substantial research, coordinate work and respond to events, requiring a lawyer to approve every routine action recreates much of the operating model we were trying to improve. It can also create a misleading sense of control if the person approving those individual actions cannot see the strategy they collectively form.

This is where I think Human Accountable for the Loop becomes more useful than simply asking whether a human appears somewhere inside it. The accountable human does not need to make every decision, but they do need the conditions required to exercise meaningful responsibility for the authority being delegated. That means being able to see what the system is doing and how its behaviour is changing, having evidence of how significant decisions were reached, controlling the boundaries within which the agent can act and receiving escalation when those boundaries are approached. It also means having a practical ability to change the system’s authority, pause it or stop it when the way it is operating no longer reflects what was intended.

This is closely related to what we already expect from supervision between lawyers, except the architecture now has to provide some of the things a human supervisee previously contributed themselves. For the acquisition agent, I would not expect a partner to review every research query, draft or delegation, but I would expect the surrounding system to know the objective the agent is pursuing, the authority actually granted by the client and the limits around it.

If the agent starts constructing intermediate objectives that move outside that envelope, the significant event is not simply the next individual API call. It is the change in direction, and that is what needs to become visible to the accountable person before the strategy turns into action.

Human Accountable for the Loop is therefore not an argument for putting more people in front of more approval screens. It is about giving the person who will continue to carry professional responsibility the visibility, evidence and control they need to remain genuinely accountable as more of the activity inside the loop becomes autonomous.

Legal AI has spent several years concentrating, quite reasonably, on what models say. We have worried about hallucination, confidential information, whether answers are grounded in authoritative sources and whether we can evaluate the quality of legal reasoning, all reasonable things to have spent time on the past few years.

Agentic systems add another class of question because the model is no longer only producing an answer. We need to understand the outcome it is pursuing, the intermediate objectives it has constructed, the capabilities it is combining and the authority it holds to turn those conclusions into actions. We also need to know where that authority came from, how it changes and when a series of individually sensible actions begins to form a strategy that nobody actually approved.

The OpenAI incident gives us an unusually vivid example because the agents were operating in cybersecurity, where the actions are dramatic and the consequences easy to see. They found unexpected uses for infrastructure, shared discoveries, delegated work, recreated a capability after OpenAI removed the original mechanism and kept finding new routes around obstacles. Over time, the scope of what they were doing moved beyond the scope the humans running the evaluation had intended.

Legal will often look much more ordinary. An email gets sent, information is used in a way nobody expected, a negotiation position changes or somebody is contacted without the authority being as clear as everyone assumed. The client might even get a better result, which could make the problem harder to notice rather than easier. When somebody later asks whether the team was actually entitled to do what the system did, saying that the agent found the route for itself is unlikely to be the end of the conversation.

I think that is what we need to start designing for now. Much of the value of capable legal agents will come from their ability to keep working when the obvious route fails, connect information across systems and find approaches we had not prescribed in advance, and we should want them to have that freedom. What we cannot do is assume that the professional boundaries humans have traditionally carried with them will somehow emerge automatically alongside that capability.

If we want to delegate more of the work without delegating away responsibility for it, authority, supervision and accountability need to become part of the architecture too.