Legal Judgment Prediction: If You Are Going to Do It, Do It Right

The field of Legal Judgment Prediction (LJP) has witnessed significant growth in the past decade, with over 100 papers published in the past three years alone. Many claim accuracies of over 90%. Now GPT-4 passed the bar. The promises (and the alarms) of a robojudge have been around for years, yet nothing seems to actually happen. In this talk we will discuss why that is. We will address how these systems are built, who they are for, who is developing them, and at what stage of the process things seem to go wrong. We will look at the distinctions between predicting court decisions and the practices of legal professionals in their daily work and explore how a lack of attention to the identity and needs of end-users has fostered the misconception that LJP is a near-solved challenge suitable for practical application. But is there a solution?