Exploring embeddings and document structure recognition for extractive summarization of Portuguese Supreme Court judgments

Abstract

Legal text summarization is increasingly relevant due to the growing availability of digital legal documents. In this study, we address the challenges of summarizing judgments from the Portuguese Supreme Court of Justice (STJ) by leveraging domain-adapted embedding models tailored to the Portuguese legal domain, which lacks a robust ecosystem of specialized embeddings. This paper presents the development of domain-specific embeddings for Portuguese legal language, designed to capture the semantics of jurisprudential statements with higher precision. Beyond proposing this resource, the study includes an empirical validation through expert (judge) assessment, demonstrating the embeddings’ applicability to real judicial reasoning. Our findings also demonstrate that domain-specific embeddings significantly enhance the quality of extractive summaries, even when employing simple algorithms. Additionally, we propose a segmentation model that identifies the inherent structure and roles of different sections within judgments, a critical step given their extensive length and varying purposes. This model further improves the quality and relevance of generated summaries. Our approach achieved a ROUGE-1 score of 57.23 and a ROUGE-2 score of 25.44. We evaluated summaries for coherence, completeness, and accuracy using state-of-the-art large language models (Llama and GPT-4o). The LLM-based ratings show limited agreement with ROUGE and with each other, and a cross-evaluation analysis combining these signals with feedback from STJ judges highlights the limitations of any single metric for evaluating legal summaries. Finally, a human evaluation conducted by STJ judges provided valuable insights into the strengths and limitations of our approach.