SIGIR2019検索実験ヒント集

Fernando Diaz @841io

There are many retrieval tasks, including ad hoc retrieval, topic tracking, distributed IR, query reformulation. Understanding how your problem relates to these will give you access to a lot of prior work and data to use as a foundation. 2/11 trec.nist.gov/data.html

Fernando Diaz @841io

If you are running ad hoc retrieval experiments (i.e. rank text documents based on user keywords or a question), TREC provides a great set of data and metrics for this. 3/11

Fernando Diaz @841io

In order to reduce some of the overhead and standardize experiments with these collections, I have a repository which will convert all queries and judgments into a single format broken down by common query/document subsets. 4/11 github.com/diazf/trec-data

Fernando Diaz @841io

It also includes a stopword list if people want to share one. 5/11

Fernando Diaz @841io

For algorithms, there are many open source implementations beyond Lucene that provide modern document ranking and can consume TREC data out of box. 6/11 cs.cmu.edu/~callan/Papers…

Fernando Diaz @841io

I still use Indri and have updated the code to make it easy to test common strong baselines (e.g. proximity, pseudo-relevance feedback, external expansion). 7/11 github.com/diazf/indri

Fernando Diaz @841io

You should pay particular attention to picking the right baseline based on whether your new algorithm has access to proximity information, corpus analysis, or non-target data. 8/11 github.com/diazf/indri#ba…

Fernando Diaz @841io

All of these baselines rank based purely on text content. If you have side information/document metadata or want a learned ranking model, you should use the Terrier, RankLib, or Solr implementations of learning-to-rank. 9/11

Fernando Diaz @841io

Finally, remember to justify your retrieval metrics. There are very strong assumptions behind metrics like NDCG, P@k, and MAP. The correct metric depends on how you expect your algorithm to be used. 10/11 @ian_soboroff

Fernando Diaz @841io

Thanks for your time and, for #sigir2019 authors specifically, I encourage you to read @peter_r_bailey's excellent "SIGIR Paper Writing Tips". 11/11 microsoft.com/en-us/research…