Building Responsible AI Systems for a More Equitable Future

For AI used in legal services and other high-impact decisions, responsible development requires measurable safeguards, accountable human review, and a clear way to challenge errors.

Imagine a legal aid chatbot that tells a user they are ineligible for assistance because it misinterprets a form or relies on outdated information. The answer, however wrong, may sound authoritative, and the user could abandon a valid claim. The problem is not simply that the system made a mistake. It is that the mistake occurred in a process where the affected person may lack the knowledge or resources to identify it.

Organisations deploying Artificial Intelligence need to examine such risks before a system reaches users. That means defining what the tool is allowed to do, testing how it performs across different groups, checking the evidence behind its outputs, and establishing who is responsible when something goes wrong.

For legal technology providers, courts, law firms, and public institutions, these controls should form part of system design and daily operations – not remain as broad commitments in an ethics policy.

The first safeguard is to set clear boundaries around the system’s purpose. A tool that searches legal documents or classifies case files creates different risks from one that recommends whether a person qualifies for legal aid or suggests a course of action in a dispute.

Organisations should document the intended use, the users, the decisions the system may support, and the decisions that must remain with an authorised professional. They should also identify foreseeable misuse, such as staff treating a generated legal explanation as verified advice.

This assessment should determine the level of testing and oversight required. An internal document-search tool may need accuracy checks and access controls. A system influencing eligibility, case prioritisation, or legal recommendations requires stronger evidence of reliability, documented review procedures, and a way to challenge consequential outcomes.

The UNESCO recommendation on AI ethics emphasises human rights, inclusion, and human oversight. For an organisation, those principles should translate into specific requirements: what decisions the system can support, what evidence it must provide, and when it must refer a matter to a human.

A model can perform well overall while failing more often for particular users. Aggregate accuracy alone may conceal problems involving language, disability, age, regional terminology, or the quality of source documents.

Testing should therefore include relevant subgroups and realistic operating conditions. For a legal document classifier, this might mean examining whether scanned documents, unusual formatting, or files containing regional names are more likely to be misclassified. For a legal information assistant, tests should cover incomplete questions, ambiguous facts, outdated legal material, and questions outside its permitted scope.

Teams should measure more than one performance indicator. Depending on the task, useful measures include:

• Precision: Of the cases the system identifies as belonging to a category, how many are actually in that category?
• Recall: Of all cases that belong to the category, how many does the system identify?
• False-positive and false-negative rates: How often does it incorrectly flag a case or fail to identify one?
• Subgroup performance: Do error rates differ materially across relevant groups or document types?
• Abstention and escalation rates: How often does the system decline to answer or refer a case for human review?

The appropriate measures depend on the consequences of an error. In legal aid screening, failing to identify a person who may qualify can be materially different from flagging someone who ultimately does not qualify. Neither error should be assessed without considering how the result is reviewed and corrected.

Testing should use a held-out evaluation set that was not used to tune the system. High-impact applications also benefit from independent review, documented test cases, and adversarial testing designed to expose weaknesses. When a material disparity appears, the deployment team should investigate its cause rather than relying on a single overall score.

The NIST framework provides a useful structure for this work through four functions: Govern, Map, Measure, and Manage. Its lifecycle approach makes risk assessment a continuing process rather than a one-time approval exercise.

Generative AI can produce fluent explanations, summaries, and citations without establishing that the information is correct. In legal settings, this is a serious operational risk because fabricated authorities or incorrect procedural guidance can damage a case and undermine professional obligations.

The U.S. case Mata v. Avianca illustrates the danger. In 2023, a federal court sanctioned lawyers after fictitious judicial opinions generated using ChatGPT were included in a legal submission. The Mata sanctions order records how the unsupported authorities entered the filing.

Legal technology systems need controls that address the source of an answer, not just how convincing it sounds. For document retrieval and legal research tools, practical controls include:

• Restricting retrieval to approved legal sources where appropriate.
• Requiring citations that can be opened and checked against the original text.
• Verifying that cited decisions, provisions, and quotations actually support the answer.
• Showing the source date and jurisdiction, particularly where law changes over time.
• Requiring the system to state when the available material is insufficient rather than fill gaps with an unsupported answer.

Retrieval-augmented generation can help ground responses in a defined collection of documents, but it does not eliminate errors. A system may retrieve the wrong passage, misread an exception, or draw a conclusion that the cited source does not support. Source verification and substantive review remain necessary for consequential legal work.

A human approval step provides little protection if the reviewer lacks time, relevant expertise, or authority to reject the system’s recommendation. Effective oversight needs to be designed into the workflow.

Organisations should specify which outputs require review, what evidence the reviewer must examine, and who can stop or reverse an automated process. Reviewers need access to the original records and relevant source material, not only a model-generated summary.

Audit logs should record the model version, relevant input or reference identifiers, output, reviewer action, and any correction or override, subject to applicable privacy and retention requirements. These records help teams investigate disputed decisions and identify recurring errors. They should not be used to collect unnecessary sensitive information.

Staff training also matters. Users should understand that confidence expressed in natural language is not proof of correctness. Organisations should monitor whether reviewers routinely accept recommendations without checking them, especially under time pressure.

Human oversight is not a substitute for model testing. It is a separate control that must be supported by adequate information, training, authority, and time.

Legal and justice applications can involve confidential client communications, personal records, financial information, and sensitive case details. Organisations should identify what data a system needs and prevent it from accessing or retaining information without a justified purpose.

Before using an external provider, teams should review data-processing terms, retention settings, model-training policies, access permissions, incident notification obligations, and arrangements for deleting information. Role-based access, encryption, secure logging, and controlled testing environments can reduce exposure. Sensitive files should not be entered into unapproved public AI tools merely because they offer convenient features.

Accessibility should also be tested. A system may be technically accurate yet unusable for someone relying on a screen reader, using a mobile device, or communicating in a language that the system handles poorly. The W3C’s accessibility guidance offers a foundation for reviewing whether digital services work for people with different abilities and access needs.

Finally, teams need a failure plan. They should be able to disable a problematic feature, return to a previous model version, notify affected users when appropriate, and route urgent cases to a qualified person. A system should not remain in use simply because its risks are inconvenient to address.

Approval before launch does not establish that a system will remain reliable. Legal sources change, user behaviour shifts, software is updated, and tools are sometimes applied to tasks beyond their original scope.

Ongoing monitoring should track error rates, complaints, corrections, subgroup disparities, failed source checks, security incidents, and the frequency of human overrides. Teams should define thresholds that trigger investigation and assign responsibility for acting on the results.

A material rise in unsupported citations, for example, may require restricting a feature while the cause is investigated. A widening performance gap between language groups may call for additional evaluation data, workflow changes, or suspension of the affected use case. Every material model or data update should trigger a risk-based assessment before the changed system is relied upon.

The commercial emphasis on AI governance is also increasing. Markntel Advisor estimates that the AI model risk management segment was valued at USD 6.41 billion in 2025 and projects it to reach USD 14.55 billion by 2032, representing a compound annual growth rate of 12.42% during 2026–2032. Its broader AI market forecast projects the overall AI market to grow from USD 802 billion in 2026 to USD 1.729 trillion by 2032, at a CAGR of 13.66%.

These are commercial forecasts, not measures of fairness or proof that organisations have effective safeguards in place. They do, however, show the scale of the AI sector and the growing relevance of dedicated model-risk controls as organisations deploy systems in consequential settings.

Responsible AI requires defined ownership across the system’s lifecycle. Business leaders should approve the intended use and risk tolerance; technical teams should validate performance and maintain controls; legal and compliance teams should assess applicable obligations; and operational users should report failures and exercise review procedures.

For legal technology, the standard must be especially clear: an automated answer should not prevent a person from receiving appropriate assistance, checking a source, or challenging an outcome. Organisations need evidence that their safeguards work in practice, not just documentation that they exist.

The test of a responsible AI system is not whether it can produce a plausible answer quickly. It is whether the organisation can explain its role, measure its errors, protect the people affected by it, and correct failures before they cause avoidable harm.