Judging the spatial relevance of documents for GIR

by Clough and Joho (Advances in Information Retrieval 2006)

This articles describes a pilot study which assesses both thematic and geographic relevance based on a three point scale. An evaluation shows that the inter-annotator agreement was much worse for geographic relevance than for the thematic one (k=0.1886, p<0.05 for thematic versus k=0.1388, p<0.05 for spatial relevance).

Evaluation metrics: multi-rater kappa test

Comments: The results show that there are much higher differences in the users' perception of spatial IR results than the ones observed for thematic IR which suggests that the requirements for spatial IR tend to be more heterogeneous.