Courts across the U.S. are grappling with a wave of legal filings that contain fabricated precedent generated by AI tools like ChatGPT.
Judges and regulators are responding with fines, sanctions, and formal guidelines, as AI hallucinations in court filings pressure attorneys to verify AI-generated content in an increasingly automated legal landscape.
AI hallucinations occur when an artificial intelligence system, like ChatGPT, produces information that is false, misleading, or entirely fabricated, but is presented in a way that appears confident and credible. This practice often leads to legal disciplinary actions.
In May, the Utah Court of Appeals sanctioned Utah-based lawyer Richard Bednar after it was found that he used ChatGPT to produce AI hallucinations, filing false citations to a nonexistent court case.
Going back to June 2023, a Manhattan federal judge fined attorneys $5,000 for citing non-existent cases generated by ChatGPT in a New York case.
Since then, such instances have multiplied. In June, Damien Charlotin, a legal researcher, found 95 AI-involved fraudulent cases in the United States since June 2023, with 58 occurring this year.
Despite the risks, the growing use of AI in legal proceedings is driven by pressures to save time, cut costs for firms, and meet rising client expectations.
According to the American Bar Association, in 2024, 30.2% of surveyed attorneys reported that their offices were using AI tools. Among them, 52.1% identified ChatGPT as the most commonly used.
The ABA further notes that even with concerns over inaccuracies and limited reliability, AI adoption within the legal profession appears promising due to its potential to save time.
With over half of AI-using attorneys relying on ChatGPT, the potential for mistakes or deliberate reliance on fabricated content is likely to grow, raising ethical and legal concerns across the profession.
The rise of AI in legal practice underscores the need for systemic safeguards, including disclosure requirements and careful oversight, to ensure that efficiency does not come at the expense of proper legal proceedings.
