FIRE 2026

Forum for Information Retrieval Evaluation

Indian Statistical Institute, Kolkata

17th-20th December

Most research on search agents is driven by benchmark evaluation. Yet practitioners increasingly face a different question: how to choose a model for a task that no benchmark covers yet. This talk proposes that the search behaviour of a model offers grounds for such decisions. Using controlled experimentation on model search behaviour in product search, I show that models tend to differ in how they click, judge relevance, and reformulate, and that these differences are broadly stable across collections, forming behavioural signatures. Furthermore, models of indistinguishable effectiveness deliver largely different results to the user due to their signatures. I then discuss how behavioural signatures can inform which tasks a model suits when no benchmark is available.
For decades, IR evaluation has rested on a simple and effective idea: fix a set of queries, collect relevance judgments, and compare ranked lists. This test-collection approach gave us TREC, CLEF, NTCIR, and FIRE, and it made IR one of the most empirically rigorous areas of computing. Agentic IR systems strain nearly every assumption behind it. An agent does not simply return a ranked list. It plans, issues its own queries, calls tools, reads and synthesizes sources, and acts on a person's behalf, often over many steps. So what exactly do we judge, and who does the judging?
In this talk, I will draw on recent work from my lab and collaborators to lay out five challenges: (1) static benchmarks that cannot keep up with agents that adapt, personalize, and may have memorized the test; (2) simulated users that are far more cooperative than real people; (3) the gap between a correct answer and a useful outcome for the person's task; (4) the circularity of using LLMs to judge LLM-based agents; and (5) the "delegation paradox," where the more we hand off to agents, the more auditing, trust, and oversight we need. For each, I will share what we have tried, what worked, and what did not. I will argue that these questions get harder as agents serve people across languages and cultures, which is where FIRE's community has much to contribute, including through this year's Multilingual Agentic Search Task.
Children naturally turn to standard Information Access Systems to seek information; yet the extent to which their information seeking behaviors differs from adults and what they actually need from these systems has yet to be fully explored.
In this talk, I will share what we have learned from studying how children search in the classroom, positioning these findings within existing literature. I will discuss children’s perception of how different search tools work, from search engines to LLMs and hybrid systems, in term of utility and trust. I will also describe how we combined their views with those of their teachers and parents to create child anti-personas, which can guide the design of scaffolding for children when searching online.
As we explore how established information-seeking models need to be adapted to reflect children’s perspectives, we will also learn how to design effective search experiences for other non-mainstream searchers.


ACM SIGIR


TCS Research


To be announced soon.