Simulating Information Retrieval Test Collections
- Simulated test
- Categories:Computers & Internet
- Language:English(Translation Services Available)
- Publication date:September,2020
- Pages:184
- Retail Price:(Unknown)
- Size:190mm×234mm
- Page Views:167
- Words:(Unknown)
- Star Ratings:
- Text Color:Black and white
Request for Review Sample
Through our website, you are submitting the application for you to evaluate the book. If it is approved, you may read the electronic edition of this book online.
Special Note:
The submission of this request means you agree to inquire the books through RIGHTOL,
and undertakes, within 18 months, not to inquire the books through any other third party,
including but not limited to authors, publishers and other rights agencies.
Otherwise we have right to terminate your use of Rights Online and our cooperation,
as well as require a penalty of no less than 1000 US Dollars.
Description
We present three different simple types of text generator which work at a micro level: Markov models, neural net models, and substitution ciphers. We also describe macro level methods where we can engineer macro properties of a corpus, giving a range of models for each of the salient properties: document length distribution, word frequency distribution (for independent and non-independent cases), word length and textual representation, and corpus growth.
We present results of emulating existing corpora and for scaling up corpora by two orders of magnitude. We show that simulated collections generated with relatively simple methods are suitable for some purposes and can be generated very quickly. Indeed it may sometimes be feasible to embed a simple lightweight corpus generator into an indexer for the purpose of efficiency studies.
Naturally, a corpus of artificial text cannot support IR experimentation in the absence of a set of compatible queries. We discuss and experiment with published methods for query generation and query log emulation.
We present a proof-of-the-pudding study in which we observe the predictive accuracy of efficiency and effectiveness results obtained on emulated versions of TREC corpora. The study includes three open-source retrieval systems and several TREC datasets. There is a trade-off between confidentiality and prediction accuracy and there are interesting interactions between retrieval systems and datasets. Our tentative conclusion is that there are emulation methods which achieve useful prediction accuracy while providing a level of confidentiality adequate for many applications.
Author
Bodo Billerbeck is an applied data scientist at Microsoft Bing, working mostly on core search problems. His interests lie in finding often data-driven solutions to multiple problems in the search stack, including indexing, query reformulation, matching, answer selection and insertion, as well as evaluation. Since completing his Ph.D. at RMIT University in 2005 he briefly worked at Sensis.com.au, but soon moved on to Microsoft. After spending some time embedded at Microsoft Research in Cambridge, UK, he returned to Australia, and recently has come full circle and is enjoying an honorary fellow position at RMIT.
Paul Thomas is an applied scientist at Microsoft. His research is in information retrieval: particularly in how people use web search systems and how we should evaluate these systems, as well as interfaces for search including search with different types of results, search on mobile devices, and search as conversation. He has previously worked at Australia's CSIRO and the Australian National University.
Nick Craswell is a research manager at Microsoft Bing. Since obtaining his Ph.D. from the Australian National University in 2000 on the topic of distributed information retrieval, Nick has worked on enterprise search, expert search, anchor text, click graphs, image search, offline and online evaluation of Web search, query-independent evidence in ranking, evaluation metrics and web ranking and neural ranking models. He has been a driving force in the TREC web tracks and enterprise tracks and was instrumental in creating various widely used test collections.