As Supreme Court Justice Felix Frankfurter put it: "The history of American freedom is, in no small measure, the history of procedure." The scale of threats to due process, however, has greatly expanded. And so have the opportunities to correct uses of wholly error-prone systems by government, in ways that harm our liberty and property rights.
In Minnesota, a recent federal lawsuit alleges that a private insurer illegally denied "elderly patients care owed to them under Medicare Advantage Plans" by deploying an AI model known to have a 90% error rate, to override determinations by physicians. The tool is proprietary, and the company denied requests from patients or physicians for more information; I am aware of no independent research evaluating its performance. If the allegations are correct, then the model, used in many states, could have resulted in errors in millions of cases. (Federal regulations, made effective in 2024, now forbid exclusive reliance on AI to make such determinations.)
Another significant source of error involves not an automated system operating alone, but the way human decision-makers rely on something automated. As NIST has explained, in the context of facial recognition technology, "what matters then is the human response," because the human police officer will review the candidate photos that the system pulls to decide whether any of them is a suspect. The system may score the images, or call the level of match "strong," using unclear criteria, and, as the National Academy of Sciences concluded in a landmark 2024 report, such interpretations "can prejudice or bias human review of images." The evidence on the interaction of these systems and the people that use them is limited, but what research has been done suggests that people can be very bad at recognizing a face from a series of images. Even in a setting when the task was to examine high-quality passport photos, participants made errors 50 percent of the time. Eyewitness misidentifications are a leading cause of wrongful convictions, and it is well understood that suggestive police lineups can alter the memory of an eyewitness. And we know that false arrests can and have resulted from the use of untested facial recognition systems. Except in a handful of states that have passed some regulations, the roll-out of facial recognition has occurred freely.
