Analyzing European Court of Human Rights Judgments using Natural Language Processing: Lessons learned from Discriminative and Generative Modeling

This talk presents a synthesis of our research into analyzing judgment texts from the European Court of Human Rights using natural language processing methods. Over the course of several studies on case outcome classification, we have explored challenges such as measuring and supporting alignment of models with legal expertise, identifying and mitigating spurious correlations, surveying disagreements in expert markup, and testing model calibration against judge vote ratios. In the area of generative models, we benchmarked a group of language models on a question answering dataset we constructed from the court’s caselaw guides, where responses are attributed to specific judgments. We also developed an LLM-pipeline designed to create structured reports of ECHR caselaw for a given topic. Central discussion themes of the talk include the integration of deep neural models and legal expertise, challenges introduced by composite systems, and experimental evaluation in ill-defined tasks.