The premise behind the TREC Replicable Runs Initiative (RRI) is that submitted TREC runs should be replicable. That is, one team should be able to replicate another team's runs, as the prelude to further analysis, improvements, etc. While conceptually simple, this ideal has been frustratingly hard to achieve: just ask Voorhees et al. (EVIA 2016)! Their "Open Runs" initiative in TREC 2015 received the submission of 79 participating run ids associated with 19 distinct code repositories that yielded... zero runs that successfully replicated!
The TREC 2019 Replicable Runs Initiative (RRI) represents another try at this effort in the context of the TREC 2019 Deep Learning (DL) Track, building on the infrastructure from the Open-Source IR Replicability Challenge (OSIRRC), a workshop at SIGIR 2019.
All participants in the TREC DL Track are invited to participate! In the online submission form of each run, you'll get a chance to indicate if you'd like to deliver a Docker image, conforming to the OSSIRC jig specifications, that will replicate your run. Teams can designate as few or as many submissions for this condition as they wish. The deadline for the TREC DL Track is August 7, 2019. The deadline for the delivery of the Docker images is August 21, 2019.
After receiving the Docker images, we (the team at the University of Waterloo) will run the Docker images to see if, indeed, the runs are replicable! If we encounter errors with an image, we'll debug and consult with the teams to the extent time allows. And yes, Docker images that require (modest amounts of) GPUs are okay (after all, this is deep learning...).
If you are interested in participating in this initiative, please email Jimmy Lin as soon as possible so we can get a sense of the scope of these efforts and hardware resources required.
- August 7, 2019: Submission of runs (indication of participation)
- August 21, 2019: Docker images due
- Ellen M. Voorhees, Shahzad Rajput, Ian Soboroff. Promoting Repeatability Through Open Runs. EVIA 2016.