FIRE 2024

Forum for Information Retrieval Evaluation

DAIICT , Gandhinagar

12th - 15th December

SoftMax converts unconstrained logits into a multinomial distribution. Analogously, Sinkhorn-Knopp iterations convert an unconstrained square matrix into a doubly stochastic approximation to a permutation matrix. A differentiable network that proposes a soft permutation in response to input features is a powerful tool for optimal transport, applicable to diverse text and graph matching tasks. It enables us to featurize complex data artifacts into sets of embeddings, rather than single, fixed-length vectors, and learn to compare a `query' set against a `document' set, suitable for scoring tasks in retrieval and textual entailment. Warming up from set-of-vector retrieval as a modified form of ColBERT, we discuss vector subset retrieval and efficient indexing, then proceed to matching problems in graphs with rich node and edge features, even if Sinkhorn-Knopp iterations may not directly apply to all of them. Knowledge graphs (KGs) have entities (Einstein, relativity) for nodes and relations (discovered) for edges. Nodes and edges have canonical IDs, but are also known by numerous textual aliases in various languages. Aligning KGs from various languages, i.e., inferring that two entities or relations are one and the same, can help maintain a "super-KG" like WikiData. We will describe a formulation where KG alignment is in synergy with inference of missing edges, using vector set similarity at its heart. While hallucination by large language models (LLMs) is generally regarded as a nuisance, we have found that allowing an LLM to hallucinate a relational schema (tables, columns, foreign keys) from a natural language question, then aligning the hallucinated schema graph to the actual schema graph of the target database can improve schema retrieval, and thereby, text to SQL generation. Moving on to classical combinatorial graph problems, such as subgraph isomorphism, graph edit distance, and maximal clique, we build new network gadgets around graph neural networks (GNNs) and Sinkhorn-Knopps networks, leading to a series of related solutions to these problems. GNNs can help bypass intractable quadratic assignment problems and replace them with transportation as approximations, induced by contextual node or edge embeddings. We conclude with a few remarks on the ongoing evolution of architectures where text and graph encoders and decoders must interact to solve retrieval and generation tasks.
Relevant Paper
Large language models (LLMs) have made significant progress in natural language processing tasks and demonstrate considerable potential in the legal domain. However, legal applications demand high standards of accuracy, reliability, and fairness. Applying existing LLMs to legal systems without careful evaluation of their potential and limitations could pose significant risks in legal practice. In this talk, I will introduce our recent efforts in developing a novel Chinese legal benchmark LexEval. This benchmark is notable in the following three aspects: (1) Ability Modeling: We propose a new taxonomy of legal cognitive abilities to organize different tasks. (2) Scale: To our knowledge, LexEval is one of the largest legal evaluation datasets, comprising 23 tasks and 14,150 questions. (3) Data: we utilize formatted existing datasets, exam datasets and newly annotated datasets by legal experts to comprehensively evaluate the various capabilities of LLMs. LexEval not only focuses on the ability of LLMs to apply fundamental legal knowledge but also dedicates efforts to examining the ethical issues involved in their application. Based on the dataset, we organize a legal LLM benchmark workshop named CAIL (China AI and Law Challenge). We hope these efforts will offer more valuable insights into the challenges and potential solutions for developing legal AI systems and LLM evaluation pipelines.
Hate speech and other offensive and objectionable online content pose a huge challenge to societies. Offensive content undermines objective discussions and can lead to the radicalization of debates. Content Moderation is necessary to suppress offensive content and due to the massive amount of posts, AI needs to support the identification of problematic content. This decision introduces AI as an actor into everyday life of millions of social media users. Content moderation is based on text classification but the core of the technology needs to be embedded in a social context during development and evaluation as well as deployment. Challenges for the evaluation and typical results need to be considered when discussing current research on content moderation. The regula- tion of Hate speech and content moderation also needs to be ad- dressed. How can AI strike the right balance between censoring and overblocking on the one hand and freedom of speech on the other? The Digital Service Act within the EU poses an influential model which regulates the factors for removing content and which aims at an transparent implementation.
A nice property of large language models is that they use natural language for both their input and their output, allowing them to easily be applied to different tasks and combined together in arbitrary pipelines. In a similar spirit, learned sparse retrieval (LSR) provides a way to use natural language to align embedding spaces and modalities for search. In this talk, I will expand on this vision and describe some of my recent work using LSR across text and image modalities. Given that a bag-of-terms representation is not without downsides, I will also describe LSR's challenges and my recent work addressing them by improving the expressiveness of learned sparse representations for retrieval.


ACM SIGIR


To be announced soon.