Legal AI training data and evaluation

Define the task and legal context

Specify the jurisdiction, relevant date and source material. A task about a contract clause is different from a request to research current law.

• Contract review: identify provisions and explain departures from a supplied playbook.
• Document analysis: extract relevant facts with references to the source.
• Answer evaluation: check whether a conclusion follows from the provided authorities.
• Issue spotting: identify missing facts and questions requiring further review.

Evaluate support, not just fluency

A plausible answer may still misstate a source or omit a material qualification. Make those failures visible in the rubric.

• Accuracy of quotations, references and source attribution.
• Application of the stated jurisdiction and time frame.
• Distinction between facts, assumptions and conclusions.
• Recognition of conflicting sources and unanswered questions.

Document expert judgments

Agree on the reviewer qualifications, task instructions and process for resolving disagreement. Preserve the explanation for each judgment rather than reducing every decision to a score.

• Reference answers or acceptable answer criteria.
• Evidence supporting each judgment.
• Documented disagreement and resolution.
• Held-out evaluation tasks kept separate from training examples.

Bring a focused project brief

Describe the intended users, legal workflow and current model failures. Confirm rights and access arrangements before providing documents.

• Specify the jurisdiction, specialty and required reviewer experience.
• State whether tasks use supplied sources or require additional research.
• Identify confidentiality and other restrictions on the material.
• Agree on output format, acceptance criteria and delivery scope.

Common questions