Skip to content

Latest commit

 

History

History
90 lines (66 loc) · 3.7 KB

README.md

File metadata and controls

90 lines (66 loc) · 3.7 KB
Raw

OSIRRC Docker Image for Galago

Build Status

Michael Zarozinski

This is the docker image for Galago v3.15 conforming to the OSIRRC jig for the Open-Source IR Replicability Challenge (OSIRRC) at SIGIR 2019. This image is available on Docker Hub.

This image is based on the Indri image by Claudia Hauff (https://github.com/osirrc/indri-docker)

  • Supported test collections: robust04
  • Supported hooks: index, search

Quick Start

First, clone the jig and follow its setup instructions - in particular, do not forget to also clone and install trec_eval inside the jig directory.

The following jig command can be used to index TREC disks 4/5 for robust04:

python run.py prepare \
  --repo osirrc2019/galago \
  --tag v0.0.2 \
  --collections robust04=/path/to/disk45=trectext

The following jig command can be used to perform a retrieval run on the collection with the robust04 test collection.

python run.py search \
  --repo osirrc2019/galago \
  --tag v0.0.2 \
  --output ./output \
  --top_k 1000 \
  --qrels qrels/qrels.robust04.txt \
  --topic topics/topics.robust04.txt \
  --collection robust04 \
  --opts out_file_name="output_robust04.txt"

Passing Parameters to Galago

Since both the jig and Galago specify parameters using a double dash, we can't use the double dash when specifying a Galago parameter because the jig will think it's a jig parameter. We also can't use an equal sign when specifying a Galago parameter value due to the way the jig parses key/value pairs. So, to specify a Galago parameter we use the search_options field of the --opts parameter. For example, if you wanted to change the default value of mu when running Galago outside of the jig you would specify --mu=1000. Using the jig, you would specify --opts search_options="mu:1000".

Retrieval Methods

The Galago image supports the following retrieval method:

  • QL : Query likelihood with Dirichlet smoothing.

    • Parameters :
      • mu : default 1500.

  • BM25 :

    • Parameters :
      • b : Controls degree of length normalization (values 0..1; default 0.75).
      • k : Normalization parameter for document length (default 1.2).
      • w : Custom weight for term (default 1.0).

  • SDM : Sequential Dependence Model with Dirichlet smoothing.

    • Parameters :
      • uniw : Unigram query component weight (default 0.8).
      • odw : Ordered window query component weight (default 0.15).
      • uww : Unordered window query component weight (default 0.05).
      • windowLimit : Window proximity limit (default 2).
      • fast : Faster operators (true/false) (bigram/ubigram operators).
      • sdm.od.op : Ordered window operator (default "ordered").
      • sdm.uw.op : Unordered window operator (default "unordered").
      • sdm.od.width : Window width (default 1).
      • sdm.uw.width : Window width (default 4).

Expected Results

robust04

Galago Parameter(s) MAP P@10 P@30 NDCG@10 NDCG@30
QL 0.1948 0.3888 0.2659 0.4038 0.3573
QL (--opts search_options="mu:1000") 0.1962 0.3916 0.2663 0.4077 0.3582
BM25 (--opts search_options="scorer:bm25") 0.1825 0.3799 0.2581 0.3889 0.3439
SDM (--opts search_options="operatorWrap:sdm") 0.1984 0.3888 0.2251 0.3997 0.3610