FIRE 2022

Explainable AI (XAI)

Procheta Sen, University of Liverpool

The concept of `Explainability' has become widely popular in recent Artificial Intelligence (AI) literature. The motivation of explainability is to increase the trust of humans in an AI system. The tutorial will give a brief introduction of explainable AI. Then it will specifically focus on methodologies used to explain natural language processing (NLP) and information retrieval (IR) models. With recent advancements in deep neural networks, NLP models are not always transparent to the end user. Explainability can help a user understand the reason behind the output of a NLP model. Similar argument also holds for neural IR models. In IR, the output of an explainable model can vary depending on the target audience (i.e to whom we are showing the explanations). In the end the tutorial will also cover the applications of explainability in NLP and IR. The aim of the tutorial is to gather researchers interested in explainability from NLP and IR and discuss the emergent research directions on this topic.

Data Centric AI for Information retrieval and Text processing

Debjyoti Paul, Amazon

One of the major challenges that have been observed across academia and industry - too much focus on state of the art model (SOTA) performance. While evaluating a model based on SOTA is a standard practice and achieving better SOTA performance does prove "research merit" of a proposed modeling technique, any model can be as good as the data. This follows from the well known principle of "garbage in garbage out ". Research suggests today we spend around 1% effort in evaluating and improving our "data" quality and 99% effort on improving our models. Seldom we are spending time on evaluating data. Due to unavailability of abundant labelled data most of the research problems, machine learning models are benchmarked against evaluation done by human experts or data labeled explicitly by humans. The challenge in such scenarios is that there might be a lot of subjectivity or variability involved in labels provided by human experts. Even if labels are reliable, there is no standard way to measure data quality. Such labeling inconsistencies and data quality can be hardly addressed by modeling improvement. Research suggests that improving data quality can lead to achieving better SOTA keeping the modeling infrastructure the same. Hence as scientists and researchers, the question stands at how we should work on a practice to improve our data and model hand in hand. Since models are as good as data, evaluating data quality needs paradigm research effort. The whole approach of Data Centric AI is to focus on evaluating data continuously just like it is done for the model. This cannot be solved with traditional exploratory data analysis, especially in information retrieval/text mining. In this tutorial we will learn:

1. Measuring labeling inconsistencies for Information retrieval and NLP
2. Measuring Bias in data.
3. Data Lineage
4. Perils of synthetic data generation.
5. Are these outliers or just underrepresented data?
6. How to deal with noisy data? How do I know my data is noisy to start with?