FIRE 2024

Large language models (LLMs) have made significant progress in natural language processing tasks and demonstrate considerable potential in the legal domain. However, legal applications demand high standards of accuracy, reliability, and fairness. Applying existing LLMs to legal systems without careful evaluation of their potential and limitations could pose significant risks in legal practice. In this talk, I will introduce our recent efforts in developing a novel Chinese legal benchmark LexEval. This benchmark is notable in the following three aspects: (1) Ability Modeling: We propose a new taxonomy of legal cognitive abilities to organize different tasks. (2) Scale: To our knowledge, LexEval is one of the largest legal evaluation datasets, comprising 23 tasks and 14,150 questions. (3) Data: we utilize formatted existing datasets, exam datasets and newly annotated datasets by legal experts to comprehensively evaluate the various capabilities of LLMs. LexEval not only focuses on the ability of LLMs to apply fundamental legal knowledge but also dedicates efforts to examining the ethical issues involved in their application. Based on the dataset, we organize a legal LLM benchmark workshop named CAIL (China AI and Law Challenge). We hope these efforts will offer more valuable insights into the challenges and potential solutions for developing legal AI systems and LLM evaluation pipelines.

A nice property of large language models is that they use natural language for both their input and their output, allowing them to easily be applied to different tasks and combined together in arbitrary pipelines. In a similar spirit, learned sparse retrieval (LSR) provides a way to use natural language to align embedding spaces and modalities for search. In this talk, I will expand on this vision and describe some of my recent work using LSR across text and image modalities. Given that a bag-of-terms representation is not without downsides, I will also describe LSR's challenges and my recent work addressing them by improving the expressiveness of learned sparse representations for retrieval.