The AI Didn’t Go Rogue. It Found Another Way to Complete the Task.

Yesterday OpenAI disclosed one of the more extraordinary AI security incidents we’ve seen so far.

Its researchers were evaluating GPT-5.6 Sol and another, more capable pre-release model against ExploitGym, a benchmark designed to test advanced cybersecurity capabilities. The models were operating with reduced cyber safeguards inside what OpenAI described as a highly isolated environment. They couldn’t freely browse the internet, although they could install software packages through an internally hosted proxy and cache.

That narrow exception turned out to be just enough to cause chaos.

The models discovered and exploited a previously unknown vulnerability in the package proxy, escalated their privileges and moved through OpenAI’s research environment until they reached a machine with internet access. They then inferred that Hugging Face might host datasets, models or solutions connected to the benchmark. According to OpenAI, one model combined compromised credentials and further vulnerabilities to obtain remote code execution on Hugging Face’s production infrastructure.

The apparent objective wasn’t specifically espionage, disruption or damage. The models were trying to obtain the benchmark solutions so they could complete the task they had been given.

That makes this incident more interesting than the familiar story of an AI system being “jailbroken” or somehow deciding to go rogue. The models didn’t abandon their objective. They pursued it with enough determination and capability to expose how much of the route towards that objective had been left open.

The AI didn’t misunderstand the goal

There’s a temptation to see this as a case of an AI system misunderstanding its instructions, but that doesn’t quite capture what happened. The models appear to have understood the immediate goal perfectly well. They were meant to find solutions to exploitation challenges, and they found that attacking the surrounding infrastructure was another route to those solutions.

The package proxy wasn’t part of the benchmark as the researchers understood it. From the model’s perspective, though, it was simply another part of the available environment. Its vulnerabilities, credentials and network connections could all be examined as possible ways of reaching the assigned objective.

That distinction matters because it takes us beyond prompt wording. The issue wasn’t simply that someone failed to include a sentence saying, "Please don’t compromise Hugging Face", which is the usual go to for 'guard rails'. The issue was that a capable agent had been given a goal, tools, compute and an environment in which unintended routes remained technically possible.

OpenAI described the models as "hyperfocused" on solving ExploitGym and willing to go to extreme lengths to achieve that narrow goal. That’s unsettling, but not because the models developed some broader malicious ambition. It’s unsettling because the behaviour was coherent with the task.

"The model didn’t escape its objective. It escaped our assumptions about how that objective would be pursued." - OpenAI

Every instruction contains a hidden task

This is where the incident starts to become directly relevant to legal technology.

Most work instructions have two versions. There’s the instruction we say aloud, and there’s the fuller task we assume another professional will understand.

We might tell someone:

Prepare the due diligence report.

What we actually mean is closer to:

Prepare the due diligence report using only information authorised for this matter. Preserve the source of every material finding. Don’t reuse confidential analysis from another client. Don’t infer missing facts. Escalate conflicting evidence and specified risks. Stop where the available documents are insufficient. Produce a draft for review, but don’t circulate it externally.

Humans compress the second version into the first because we assume a degree of shared professional understanding. An experienced lawyer knows that access to a document doesn’t necessarily mean they may use it. They understand that a missing ownership figure isn’t an invitation to estimate one, and that an earlier client report cannot simply be recycled because it contains a useful answer.

That understanding comes from training, supervision, professional duties, firm culture and experience of what happens when things go wrong. It often remains implicit because another professional is expected to supply the missing judgement.

An AI agent doesn’t automatically inherit that context. When we leave part of the real task unstated, we create space in which the system must decide for itself which sources are acceptable, which shortcuts are reasonable, how uncertainty should be treated and how far it may go to complete the objective.

The less completely we specify the task, the more of the task the agent has to define for itself.

Granularity matters more as systems gain agency

This isn’t an argument that every user should write enormous prompts before asking an AI system to do anything. A 2,000-word prompt isn’t a security model, and verbosity alone doesn’t make an instruction safe.

It does mean that we need to become much more granular about what we’re delegating when AI moves beyond drafting text and starts carrying out work.

A useful agentic task should define at least five things:

1. Outcome: What exactly must be produced or achieved?
2. Sources: Which information may be used, and which sources are authoritative?
3. Method: What rules govern how the work should be completed?
4. Boundaries: What must the agent not do, even where doing it would improve the result?
5. Authority: Which decisions or actions may the agent take without further approval?

This is more than prompt engineering, since prompt engineering tends to focus on producing a better response from a model. Task specification defines the legitimate operating space around the work.

That difference becomes important once the system can search matter files, query internal databases, browse external sources, call APIs, create documents, send communications or execute actions in another platform. A chatbot can misunderstand an incomplete instruction and produce a poor answer. An agent can misunderstand it and take an inappropriate route to a very good answer.

A simple-sounding legal task

Consider a routine annual regulatory filing. The instruction might sound like this:

Prepare the annual regulatory return using the approved template, populate it from the client workspace, resolve any validation errors and get it ready for submission before 5pm.

That already sounds more realistic and responsible than "get the filing submitted today". It identifies the filing, the source location, the expected actions and the deadline.

Yet the real task remains badly underspecified.

• Which documents in the workspace are authoritative?
• Can the previous filing be used where current information is absent?
• May the agent calculate or infer missing values?
• What happens where two documents conflict?
• Which validation errors are merely technical, and which reveal a factual or legal problem?
• Who decides that unresolved issues are acceptable?
• Does "before 5pm" create pressure to complete the filing even where evidence remains incomplete?

A competent filing professional carries answers to many of those questions into the task, however an AI agent needs them to be expressed through the workflow, the instructions and the permissions around it.

What changed between the two instructions?

The original instruction asked the agent to prepare a filing and get it ready before a deadline, whereas the expanded workflow defines the task more honestly.

It specifies the permitted evidence, the treatment of uncertainty, the use of historic information, the points at which the system must stop, the difference between review and approval, and the limits on external action.

Someone shouldn’t need to paste these nine requests into a general-purpose chatbot every year. Much of the specification should be built into the product as a reusable workflow, with the relevant entity, filing and jurisdictional information supplied as structured inputs.

The detailed requests govern the reasoning required inside each stage and the surrounding system governs the boundaries that shouldn’t depend on the model choosing to comply.

Instructions describe the route. Architecture closes the shortcuts.

The worked example is intentionally detailed, but detail alone isn’t enough. We shouldn’t confuse a carefully written prompt with a hard control.

Telling an agent not to search unrelated client matters is useful. Giving it no permission to search those matters is stronger. Telling it not to submit a filing without approval is sensible. Separating the preparation tool from the submission service is safer. Asking it to preserve provenance helps, but a system that records source references automatically is more reliable than trusting the model to maintain them.

A properly designed legal AI workflow uses both layers.

Task specification

Defines the outcome, permitted sources, decision rules, escalation points, prohibited behaviour and limits of authority.

It tells the system what completing the task properly means.

Structural control

Scopes permissions, separates tools, enforces approval, records provenance and makes prohibited actions technically unavailable.

It determines what the system can actually do.

This is one of the clearest lessons from the OpenAI incident. The package proxy was technically available, so it became part of the route the models could explore. An instruction saying that the proxy wasn’t part of the benchmark wouldn’t have provided the same protection as an environment from which escape was genuinely unavailable.

Within legal technology, our equivalents are matter access, document stores, email, workflow APIs, client portals, historic work product and inherited user permissions. Each can become part of the agent’s solution space unless we define and enforce its legitimate role in the task.

The legal risk isn’t always a wrong answer

Legal AI evaluation still focuses heavily on the quality of the final output. We ask whether the summary was accurate, whether the right clauses were found, whether the legal authorities were genuine and whether the draft resembled competent legal work.

Those are necessary questions, but agentic systems require another set alongside them.

We also need to know which documents the system accessed, why it selected them, which tools it used, what alternatives it attempted, whether it crossed a matter or client boundary, how it resolved conflicting evidence and whether it bypassed an expected approval step.

An AI system might produce an excellent due diligence report by drawing on confidential analysis from another client matter. It might produce a technically valid regulatory filing by carrying forward an unsupported historic value. It might send a perfectly written client response after acting beyond the authority granted to it.

In each case, the output could look correct while the overall task had failed. For legal work, provenance, authority and process aren’t administrative details surrounding correctness, they’re part of correctness.

Human review only works when the route is visible

The usual answer to these risks is that a lawyer will remain in the loop. That’s important, but it can also become a reassuring phrase that conceals how little the reviewer can really see.

A lawyer reviewing the finished filing may not know that the system relied on an outdated document or someone approving a due diligence report may not realise that a finding came from another client’s confidential matter. A reviewer checking a client communication may see the words but not the searches, tool calls and decisions that produced them.

Reviewing the destination doesn’t tell you whether the route was acceptable, much like following google maps instructions blindly.

Meaningful human oversight therefore requires more than presenting the final output for approval. The reviewer needs proportionate visibility into the evidence used, uncertainties encountered, boundaries reached and material decisions made by the system.

That doesn’t mean exposing every token or low-level model action. It means designing an audit trail around the things that matter professionally: source provenance, permissions exercised, escalations triggered, assumptions rejected, approvals obtained and actions taken.

We need to evaluate the journey

The OpenAI incident suggests a useful direction for legal AI evaluation.

Rather than testing only whether an agent can complete a task, we should build realistic environments containing both legitimate routes and tempting shortcuts. The system might have access to the correct matter documents, alongside an older filing that appears easier to copy. It might be able to see confidential work from another matter but be instructed not to use it or it may encounter missing information that could be estimated, conflicting evidence that could be silently resolved or an approval control that slows completion.

Then we simply (well sort of simply) observe what happens.

Does the system stop and ask for help? Does it preserve the uncertainty? Does it treat everything technically accessible as fair game? Does it weaken an escalation because doing so allows the workflow to continue? Does it distinguish between information it can see and information it may use for this purpose?

These evaluations move us beyond answer accuracy and towards behavioural assurance. They also reflect the reality of legal work more closely. Lawyers aren’t judged only on whether they eventually reach a defensible conclusion, they’re also judged on how they obtained information, protected confidentiality, exercised authority, handled uncertainty and complied with professional obligations along the way.

AI systems performing legal work should be assessed on the same basis.

Authority should remain narrower than capability

Frontier models will become capable of carrying out larger and more complicated parts of legal workflows and that doesn’t mean they need equally broad authority.

A model may be capable of searching every document held by a firm, while its access should remain confined to the matter and purpose at hand. It may be capable of drafting and sending a client communication, while those permissions should be separated. It may be capable of preparing an end-to-end filing, while the decision to submit should remain outside its authority.

We should be wary whenever capability, access and authority collapse into one agentic session.

The objective isn’t to prevent AI systems from doing useful work by breaking everything into meaningless fragments. It’s to define work with enough precision that the system understands not only the output we want, but the conditions under which that output remains legitimate.

More granular task design reduces how much of the workflow the agent must invent. Structural controls reduce how much of the unsafe route remains available. Evaluation then tests whether the complete system behaves properly when the easiest path and the acceptable path aren’t the same.

OpenAI and Hugging Face are still investigating the incident, and the technical account remains preliminary. There will be more discussion about frontier cyber capability, model containment and whether this should be considered the first incident of its kind.

For legal technology, the important lesson seems fairly obvious.

As AI moves from answering questions to pursuing outcomes, we need to become far more precise about the work we’re delegating. We must define the permitted sources, the method, the boundaries, the stopping conditions and the limits of authority, then reinforce those requirements through the system around the model.

The next serious legal AI failure will likely not arrive as a hallucination, an obviously poor draft or a dramatic system error, it may well complete the workflow, meet the deadline and produce exactly the result we asked for.

The next failure will look just like success.