← Legal AI Failure Database

#005

Anonymized Sandbox Run — Federal Brief RAG Failure

High Jurisdictional Hallucination · Internal RAG Eval Stack (anonymized)
Internal Eval Federal Brief RAG Sandbox

logged 2026-06-08 · incident 2026-05-30

1. The Crime Scene (The Symptom)

During an internal evaluation run, a standard RAG pipeline drafting a § 1983 qualified-immunity section cited a Ninth Circuit opinion for a binding rule in the Second Circuit. The case exists; the jurisdiction assignment is wrong. The failure was caught pre-filing — but would have passed a citation-existence-only linter.

Sandbox capture of a common production failure: correct case, wrong circuit, presented as controlling authority.

2. The Artifact (Raw LLM Output)

"Qualified immunity analysis in the Second Circuit follows the framework set forth
in Lange v. California, 141 S. Ct. 2011 (2021), as applied in Johnson v. City of
Seattle, 994 F.3d 874 (9th Cir. 2021), which the Second Circuit adopted as
binding precedent for excessive-force claims at the motion-to-dismiss stage."

"See also District of Columbia v. Wesby, 583 U.S. 48 (2018) (extending Ninth Circuit
notice pleading standards to Second Circuit § 1983 actions)."

3. The Evidence Breakdown (The Core Contradiction)

4. The Dali Verification Check

Dali jurisdictional audit: circuit_mismatch — cited Ninth Circuit authority for Second Circuit binding proposition. Internal eval flagged before export; logged here as ground-truth training data for the failure database.

View Dali audit trail →

Infrastructure Note: Every failure logged in this database is reproduced, parsed, and verified utilizing the open evidentiary infrastructure layer at Dali.

Dataset: huggingface.co/datasets/yenklabs/legal-ai-failure-database