What happens when AI becomes part of the computer?
I was listening to a conversation between Ben Lorica and Pete Warden, the founder of Useful Sensors and the person behind Moonshine AI, about local models, speech recognition and the changing economics of AI. Pete made an observation that I thought was really interesting: Imagine if every time someone tapped an icon on their phone, the application developer had to pay a few pence to another company, we'd have ended up with a very different generation of software, and plenty of applications probably wouldn't exist.
That's effectively the position we're in with much of today's AI software. We build applications that make requests to models, pay for those requests, wait for responses and manage the dependencies that come with using an external service. Those costs have been falling considerably, but they still influence which features developers build and how frequently they use AI. Legal technology has largely followed this approach, with assistants that review contracts, summarise documents, extract obligations, answer questions and draft correspondence. The lawyer identifies something they want to do, sends it to an AI system and receives an output.
Some of these products are rather good, and I don't think the underlying approach is going away. What interests me is what happens when we no longer need to treat every interaction with AI as a separate transaction, particularly when smaller models are becoming capable enough to perform useful work directly on the hardware we already own.
We've designed AI around the prompt, rather than the work
Think about how a lawyer works through a matter. They might read an agreement, notice that a defined term looks inconsistent, check an earlier instruction, speak to a colleague, update their notes and return to a provision they weren't entirely comfortable with. Much of that activity involves small acts of interpretation, comparison and organisation, contributing to the lawyer's understanding of the matter without necessarily being difficult work in isolation.
Our AI applications tend to concentrate on larger, more identifiable activities. Review this agreement, summarise these documents or generate a report. That makes sense when each interaction involves calling an external model, because we want to get enough value from the request to justify making it. It also gives developers something relatively straightforward to package and sell, with an obvious starting point and a measurable output.
Now imagine the application already has access to a capable local model. As the lawyer reads an agreement, the software could identify defined terms, recognise references to legislation, compare terminology across provisions and highlight something that appears inconsistent with an earlier document. The lawyer wouldn't necessarily need to ask for any of this, because the application could perform some of those checks as part of its ordinary behaviour, much as our writing software already checks spelling while we type.
I'm not suggesting legal interpretation is equivalent to spellchecking. A model can misunderstand a provision, and an incorrect suggestion could be considerably more consequential than a spelling mistake. The checks would need to be designed around those limitations, with clear supporting evidence and appropriate review. Nevertheless, the interesting change is that we could build intelligence into the ordinary experience of using software, rather than requiring someone to stop what they're doing and explain a task to an assistant.
I suspect there's an enormous amount of value in that distinction, particularly when you consider how much legal work involves maintaining an understanding of information that's constantly changing.
We probably don't need one enormous model doing everything
There was also a discussion on the advantages of compound AI systems, where applications combine several specialised models and conventional software components rather than relying on a single model to handle everything. This makes particular sense to me in legal services, where a single workflow might involve document processing, factual extraction, deterministic validation, legal interpretation and formal approval, each requiring something quite different from the technology supporting it.
Consider a regulatory filing workflow. We might need to identify entities in source documents, extract dates, check mandatory fields, compare information against published requirements and prepare a submission. Some of those activities could be handled by relatively small models running on a laptop, others by conventional validation rules, while the more complicated legal questions might warrant a much more capable model. There's no particular reason why recognising a company registration number should require the same model that we use to reason through an unusual contractual provision.
We've spent a lot of time comparing models and asking which is best for legal work, though perhaps we should be spending more time understanding the different types of intelligence legal work actually requires, then designing systems that use the appropriate capability for each activity. That also makes it easier to test individual components, understand their limitations and replace them as better alternatives become available.
Apple is already moving in this direction with its Foundation Models framework, exposing on-device language capabilities to application developers. Google has been developing similar capabilities within Chrome, while tools such as Ollama have made running open-weight models considerably more accessible. The quality of smaller models has improved significantly, although I wouldn't go as far as saying model quality is no longer an issue. Legal work presents some particularly awkward problems around ambiguous instructions, conflicting authorities, jurisdictional differences and the consequences of apparently minor mistakes.
Still, there are plenty of useful activities that don't require the most capable model available. Once those capabilities are running locally, we can start thinking about applications that would have been difficult to justify when every interaction incurred a separate inference cost.
Local doesn't have to mean one computer
There's another possibility here that I think deserves more attention, particularly for law firms operating across multiple jurisdictions. When we talk about local AI, we usually imagine someone running a model on their laptop, perhaps because they want privacy, lower costs or the ability to work without an internet connection. But a modern law firm might have hundreds or thousands of computers, many with increasingly capable processors, large amounts of memory and dedicated AI accelerators.
What if we treated some of those devices as a shared computing resource, rather than thinking of local inference as something that only happens for the person sitting in front of the machine?
Imagine we have 1,000 documents to review for a client in Jersey. The review involves extracting particular provisions, identifying relevant obligations, checking terminology and producing structured findings for a lawyer to assess. We've already established that a smaller model can perform the relevant extraction and classification tasks to an acceptable standard, using a tested model, a defined review methodology and a way of measuring the quality of its outputs.
We could send the documents to a conventional cloud inference service, but what if the client requires the processing to remain within Jersey, or the firm has decided that particular information shouldn't leave that jurisdiction? One option would be to provision dedicated infrastructure in Jersey, which might be sensible for a sustained workload, but it isn't necessarily the only option.
What if we could distribute the review across a collection of approved devices already operating there?
A coordinating service could divide the documents into manageable batches, identify machines with sufficient available capacity and allocate work to them. Each device would run the same approved model and return structured results, including references to the passages supporting its findings. A laptop might process a small batch while it's connected to power and otherwise idle, while a more capable desktop could take a larger allocation. The lawyer would receive a consolidated review, with the ability to inspect individual findings and trace them back to their source documents.
We wouldn't need 1,000 devices to review 1,000 documents. The available machines could process batches in parallel, with the scheduler balancing capacity, processing time and the importance of completing the work. If a device disconnected or became busy, unfinished work could be reassigned to another authorised device within the same permitted processing boundary.
I'm thinking of this as a jurisdiction-aware inference mesh, where the organisation can distribute suitable AI workloads across approved hardware while retaining control over where processing takes place.
The idea isn't that every laptop should become an uncontrolled server, or that employees should have client documents arriving on their personal machines. It would need to operate entirely within a managed environment, using approved devices, encrypted communications, isolated processing and carefully controlled access. We'd also need to establish that the data really does remain within the required boundary, considering the routing of documents, temporary storage, model telemetry, results, logs and backups alongside the legal requirements governing the particular matter.
There are practical questions too:
- Would the available devices have enough memory?
- How much power would they consume?
- Could we reliably process the documents within the required time?
- Would maintaining the distributed system cost more than using a dedicated server?
- Will the office suddenly sound like a jet taking off when a big review lands
Those questions need testing rather than assumptions and there will undoubtedly be workloads where conventional infrastructure is more efficient.
What is interesting here is the possibility of making better use of computing capacity that organisations already have. As local models improve, some of that hardware could potentially perform useful inference work when it isn't otherwise occupied, without requiring every workload to be sent to a centralised service.
For firms operating across Jersey, Guernsey, the Cayman Islands, the BVI, Hong Kong and other jurisdictions, there could be opportunities to place processing close to the people and information it serves, while retaining control over where particular workloads are allowed to run. It also creates interesting possibilities for resilience, capacity planning and how firms manage their dependence on external AI providers.
I wouldn't expect this to replace centralised infrastructure, and I certainly wouldn't suggest distributing work across laptops simply because we can. The wider point is that local AI doesn't necessarily have to be personal AI, it could become a distributed capability of the organisation, with jurisdiction, security, capacity and model suitability determining where individual tasks are processed.
Understanding what's happening might be more valuable than acting on it
There's a separate implication for how we design legal applications, and I think it might ultimately have a greater impact on the day-to-day experience of lawyers than the infrastructure itself.
Imagine a lawyer attending a meeting about a fund formation. Someone raises a question about an investor's eligibility, another person agrees to obtain an updated document, and the team decides to revisit a provision in the subscription agreement. A conventional AI meeting assistant might produce a summary afterwards, which is useful enough, although we're already seeing plenty of products offering that capability.
What I'd find more interesting is an application that recognises the different types of information being discussed and helps connect them to the wider matter. When the lawyer subsequently opens the subscription agreement, the application could surface the unresolved question from the meeting alongside the relevant provision. It could also recognise that someone was supposed to obtain an updated document, without assuming that the task has been completed.
None of that necessarily requires the AI to take an action on the lawyer's behalf, and that's something I think we're overlooking in the current enthusiasm for autonomous agents. We've become rather focused on how much work an AI system can complete independently, giving it more tools, access to more systems and the ability to execute longer sequences of tasks. There's value in that, particularly for well-defined processes where the actions can be independently validated, but I'm less convinced that greater autonomy is always the most useful direction for legal work.
Consider how much time lawyers spend reconstructing context. Why was a particular drafting decision made? Where did an instruction originate? Has someone already considered this issue? What changed between two versions of a document, and did we actually agree to do something about it?
Software that could reliably help answer those questions might deliver substantial value without ever sending an email, changing a document or submitting a filing. It would need to recognise relationships, retain supporting evidence and understand the difference between something that was suggested and something that was actually decided. That's a different engineering problem from building an agent that completes tasks, and potentially a very good fit for smaller, specialised models working alongside a structured matter record.
Understanding something doesn't give the AI permission to act
This brings me back to something I've written about previously in Human Accountable for the Loop, as applications become better at interpreting what's happening, there's a temptation to let them act on that interpretation, particularly when the technology makes doing so increasingly straightforward.
Suppose someone says during a meeting, "We should probably ask the client for an updated register." An AI model might correctly identify that as a potential action, and it might even know who would normally be responsible for requesting it... but should that automatically become a task, should the system send the request, or should it update the official matter record? Those are different decisions, requiring different levels of authority, and treating them as interchangeable would create problems regardless of how accurate the underlying model becomes.
We need to distinguish between what an AI system has observed, what it has inferred and what someone has actually authorised it to do. Making inference cheaper doesn't change that responsibility. In some ways, it makes the distinction more important because applications could generate far more observations and suggestions than they do today, and someone still needs to decide which are meaningful.
I'd rather have a system that identifies a potential inconsistency and shows me the evidence than one that confidently fixes something it has misunderstood. For legal applications, I think there's a strong argument for keeping the underlying evidence, the AI's working interpretations and the authorised matter record separate. A model might identify a potential obligation in an agreement, but that doesn't automatically make it an accepted legal conclusion. It might recognise a possible task in a conversation, but that doesn't mean someone has agreed to undertake it.
The lawyer should be able to move useful information into the formal record without having to reconstruct everything manually, while retaining control over which observations become decisions or actions. That gives us a way to benefit from more continuous intelligence without assuming that more intelligence should automatically mean more autonomy.
The economics could change what we bother to build
One of the reasons I'm interested in all this is that changing the cost of intelligence can change the kinds of software that make commercial sense. Today, we tend to justify AI features by pointing to recognisable activities. How much time does a contract review save? How quickly can we produce a summary? How many documents can we classify? Those are all sensible questions, particularly when inference is a variable cost and the application needs to demonstrate a return on that expenditure.
However, what about the value of software that notices a small inconsistency while someone is reading, or reminds them that a point was left unresolved during an earlier discussion? Individually, those interactions might not justify calling an expensive model, yet across an entire matter they could be genuinely useful. If the intelligence is already available on the device, the calculation changes, and developers have more freedom to experiment with features whose value emerges through repeated use rather than one obvious output.
Local inference isn't free, of course. Someone still pays for the hardware, electricity, model deployment, security and maintenance, and a large distributed workload might be less efficient than running the same model on dedicated infrastructure. There's nevertheless a meaningful difference between a capability with a recurring external charge for every interaction and one that can be used repeatedly on hardware the organisation already owns.
I suspect we'll eventually stop describing some of these capabilities as AI at all. We don't think of spelling suggestions, autocomplete or document indexing as separate products requiring us to visit an assistant. They're simply things we expect our software to do. Perhaps we'll eventually have similar expectations around identifying inconsistent terminology, recognising unresolved questions, connecting related documents or organising information from meetings.
The challenge will be ensuring those capabilities are sufficiently reliable and don't overwhelm users with suggestions they neither need nor trust. Making intelligence more accessible doesn't automatically make every application better, particularly if the result is a constant stream of observations competing for someone's attention.
Perhaps we're designing around the wrong unit of work
I don't think cloud AI is going away (our pensions would collapse if it did), there will continue to be good reasons to use powerful centrally hosted models, particularly for complicated reasoning, large-scale processing and activities where local hardware isn't sufficient. Nor do I think every organisation should immediately start turning its laptops into a distributed inference cluster. That would be an interesting way to create a rather expensive IT support problem if we approached it without proper engineering.
What I do think is that we're approaching a point where the location of inference becomes a design choice rather than an assumption. Some intelligence might run on the lawyer's device, some on an approved machine elsewhere in the office, some within a jurisdiction-specific infrastructure environment and some through a frontier cloud model. The application could decide where particular work is allowed to run, based on the information involved, the capabilities required and the organisation's policies, without the user needing to understand the underlying infrastructure.
That could also give firms greater control over the models they depend on. A tested extraction model could remain in use until a replacement has been properly evaluated, rather than having an entire workflow depend on the changing behaviour of one external provider. For legal services, where consistency, confidentiality and defensibility matter, I think that control could become increasingly valuable.
The most interesting consequence, though, is what this means for the experience of legal work. We've spent the last few years building applications around things people explicitly ask AI to do. Summarise this, review that, draft something or answer a question. Those are useful capabilities, but they're also a reflection of how we've been accessing intelligence.
A lawyer's work isn't really a collection of prompts. It's a developing understanding of a problem, shaped by documents, conversations, instructions, research and professional judgement. The applications we've built are often good at processing individual pieces of that information, but considerably less effective at helping people maintain the relationships between them as a matter progresses.
I wonder whether the next generation of legal software will be less concerned with how many tasks it can complete autonomously and more concerned with helping people maintain that understanding as they work. If we can make intelligence available locally, distribute appropriate workloads across trusted infrastructure and use more powerful models only when they're actually needed, we have considerably more freedom in how we design those applications.
The opportunity isn't simply to run today's AI assistants on smaller computers, it's to build legal software that actually works differently because intelligence no longer has to be somewhere else.