[Back]

A. Lipani, J. Palotti, M. Lupu, F. Piroi, G. Zuccon, A. Hanbury:
"Fixed-cost Pooling Strategies based on IR Evaluation Measures";
in: "ECIR 2017: Advances in Information Retrieval", issued by: Springer; Springer Nature Switzerland AG 2021, Cham, 2017, ISBN: 978-3-319-56608-5, 357 - 368.

Recent studies have reconsidered the way we operationalise the pooling method, by considering the practical limitations often encountered by test collection builders. The biggest constraint is often the budget available for relevance assessments and the question is how best - in terms of the lowest pool bias - to select the documents to be assessed given a fixed budget. Here, we explore a series of 3 new pooling strategies introduced in this paper against 3 existing ones and a baseline. We show that there are significant differences depending on the evaluation measure ultimately used to assess the runs. We conclude that adaptive strategies are always best, but in their absence, for top-heavy evaluation measures we can continue to use the baseline, while for P@100 we should use any of the other non-adaptive strategies.

http://dx.doi.org/10.1007/978-3-319-56608-5_28

http://publik.tuwien.ac.at/files/publik_264549.pdf