by Tang and Sanderson (ECIR 2010)
This article presents a user study which shows that users prefer search results which are not only (i) relevant but also (ii) cover many different places (=spatial diversity). It summarizes different approaches mentioned in the literature for assessing spatial diversity and performs a user study based on the following three metrics: (i) Geometric Mean (GM), SC-1, and SC-2.
- Recall considering diversity at rank K: S-recall at K = total number of subtopics found in the K documents / total number of subtopics in the given topic.
- Crowdsourcing: identify cheaters by inserting "gold standard data" into the task
- The evaluation is based on a three level preference schema (prefer results, neutral, prefer alternative).