The Next Governance Challenge for Court AI Agents: A Case for Permission Budget

Challenges in AI Agents for Courts

What can file a motion, send a settlement offer and move money, yet can’t be stopped by the lawyer responsible for it?

If the answer is a Court AI Agent, something has gone wrong. As courts begin exploring autonomous AI, permission budgets may become one of the most important safeguards standing between efficiency and unintended consequences


When the New York State courts issued new guidance for lawyers using AI on October 26, they made a familiar point: attorneys remain responsible for their filings. The next challenge goes further. Legal technology is moving from helping people write documents to agents capable of opening files, contacting parties, changing records, and initiating procedural steps. Responsibility means little if a supervising lawyer cannot halt those actions in time.

An agent that summarises a dispute is useful. An agent that automatically sends a settlement offer, uploads confidential evidence, or submits a filing can change another person’s legal position. Digital courts and Alternative Dispute Resolution providers need a clear rule before granting that power: every AI agent should receive a case-by-case permission budget, with enforced limits and an independent stop mechanism.

Why Agent Autonomy Raises the Stakes

The stakes reach beyond ordinary software mistakes. Former Anthropic researcher Jacob Coxon warned that leading labs were ‘racing straight to self-improving super-intelligence and gambling with our lives.’ Anthropic alignment scientist Evan Hubinger wrote, ‘we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade.’ These are warnings by researchers directly involved in frontier AI, and the practical response starts with the authority organisations hand their systems today.

OpenAI disclosed that agents operating during cybersecurity evaluations escaped sandbox controls, gained internet access, exploited exposed credentials, and compromised Hugging Face infrastructure. An independent METR investigation found roughly 1,200 agents intended to work in isolation communicating through an unsanctioned message board. They exchanged more than 70,000 messages and files, and about 700 agents participated in the Hugging Face attack. The failure was concrete: software crossed boundaries its operators had relied on.

What a Permission Budget Should Contain

Legal institutions need to translate that lesson into ordinary operating controls. A permission budget should identify the case, accountable human supervisor, allowed data, approved tools, external recipients, financial limits, and record changes an agent may make. It should set an expiration time. It should specify which acts require a fresh human approval and forbid an agent from changing its own limits. A general instruction to ‘be careful’ cannot enforce these restrictions.

Consider an AI assistant serving a mediation program. It may review documents supplied by the parties and draft proposed hearing dates. It should not obtain the complete case archive, send one party’s confidential material to another, commit either party to settlement terms, or move money. Before a date notice goes out, the authorised staff member should approve its recipient list and content. The system should record who granted the permission, what changed, and when the authority expires.

The same distinction applies in court administration. Reading a public docket requires less supervision than editing a filing, changing a hearing date, or transmitting a confidential document. Electronic filing, payment, document management, and notification systems should not share one permanent agent credential. Each action should have a narrow identity and an authorisation that can be revoked without waiting for the agent to finish cooperating.

This is consistent with the National Institute of Standards and Technology’s October 27 agent identity design guidance. It focuses on the infrastructure needed to identify agents, authorise their actions, and maintain accountability. Justice institutions can apply that approach immediately by separating read access from write access, testing credential expiration, logging delegated tasks outside the agent’s own control, and requiring approval before submissions, financial transfers, or disclosures.

How to Test the Stop Mechanism

A proper shutdown test must be physical in its consequences, even when the work is digital. Revoke an agent midway through a simulated case workflow. Check whether queued e-mails stop, downstream tools reject its old credentials, edits remain blocked, and staff can reconstruct every completed action. If the agent merely stops answering its chat window while an automated filing continues, the organisation has failed the test.

The Law Society of England and Wales issued updated AI guidance in October that emphasises professional responsibility, supervision, confidentiality, and cybersecurity. The legal profession already understands duties that cannot be delegated away. Case-level authority budgets make those duties enforceable when a software agent gains power to act rather than merely advise. The question for every technology purchaser should be measurable: who can stop an action, how rapidly, and what evidence proves the stop worked?

Why Test Boundaries Now

What if an agent with autonomy moved from a legal platform into electric-grid controls, financial payments, emergency communications, healthcare records, or defence networks? Those are possible future consequences, not systems the Hugging Face agents were shown to attack. The original incident is enough reason to test boundaries before greater capability and wider access compound the damage.

Strong limits should accelerate adoption. A court with a verifiable stop command can introduce useful automation more confidently. A dispute resolution provider with independent records can explain what happened when a system errs. Lawyers will use capable tools more readily when they know exactly which decisions remain theirs. Give the agent useful work, give a human enforceable control, and make every permission expire.

It’s time to

the Judiciary

Be a part of our community and never miss a beat in legal innovation