Document Retrieval(08/11/2020-Present)

Based the questions in the Question Answering Dataset and the documents which answered the questions a document ranking task was formulated. There are 3.2 million documents and the goal is to rank based on their relevance.

Relevance labels are derived from what passages was marked as having the answer in the QnA dataset making this one of the largest relevance datasets ever.

This dataset is the focus of the 2020 and 2019 TREC Deep Learning Track and has been used as a teaching aid for ACM SIGIR/SIGKDD AFIRM Summer School on Machine Learning for Data Mining and Search.

In 2020 we release a set of cleaned and formated clicks for all documents in the collection. This collection of 20 million clicks is called ORCAS.

Tasks

1. Document Re-Ranking:Given a candidate top 100 document as retrieved by BM25, re-rank documents by relevance.
2. Document Full Ranking:Given a corpus of 3.2m documents generate a candidate top 100 documents sorted by relevance.

Relevant Links

MSMARCO Document Ranking Github NIST Judgments for TREC 2019 Deep Learning Track Overview of the TREC 2019 deep learning track Paper ORCAS Dataset TREC 2020 Deep Learning TREC 2019 Deep Learning
Link to full leaderboard

Passage Retrieval(10/26/2018-Present)

Based on the passages and questions in the Question Answering Dataset, a passage ranking task was formulated. There are 8.8 million passages and the goal is to rank based on their relevance.

Relevance labels are derived from what passages was marked as having the answer in the QnA dataset making this one of the largest relevance datasets ever.

This dataset is the focus of the 2020 and 2019 TREC Deep Learning Track and has been used as a teaching aid for ACM SIGIR/SIGKDD AFIRM Summer School on Machine Learning for Data Mining and Search.

In 2020 we release a set of cleaned and formated clicks for all documents in the collection. This collection of 20 million clicks is called ORCAS.

Tasks

1. Passage Re-Ranking:Given a candidate top 1000 passages as retrieved by BM25, re-rank passage by relevance.
2. Passage Full Ranking:Given a corpus of 8.8m passages generate a candidate top 1000 passages sorted by relevance.

Relevant Links

NIST Judgments for TREC 2019 Deep Learning Track Overview of the TREC 2019 deep learning track Paper ORCAS Dataset MSMARCO Passage Ranking Github TREC 2020 Deep Learning TREC 2019 Deep Learning

Dataset Download links

Passage Ranking Leaderboard(10/26/2018-Present) ranked by MRR on Eval

KeyPhrase Extraction:RETIRED(10/18/2019-10/30/2020)

Keyphrase extraction on open domain document is an up and coming area that can be used for many NLP tasks like document ranking, Topic Clusetring, etc. To enable the research community to build performant KeyPhrase Extraction systems we have build OpenKP a human annotated extraction of Keyphrases on a wide variety of documents.

The dataset features 148,124 real world web documents along with a human annotation indicating the 1-3 most relevant keyphrases. More information about the dataset and our initial experiments can be found in the paper Open Domain Web Keyphrase Extraction Beyond Language Modeling which was an oral presentation at EMNLP-IJCNLP 2019. It is part of the MSMARCO dataset family and research projects like this power the core document understanding pipeline that Bing uses.

Tasks

1. Given a document produce the top 3 most salient keyphrases

Relevant Links

KeyPhrase Extraction Github Repo Paper

Dataset Download links

KeyPhrase Extraction(10/18/2019) ranked by F1 @3 on Eval

Question Answering and Natural Langauge Generation: RETIRED(12/01/2016-10/30/2020)

The original focus of MSMARCO was to provide a corpus for training and testing systems which given a real domain user query systems would then provide the most likley candidate answer and do so in language which was natural and conversational.

This data comes in three tasks/forms: Original QnA dataset(v1.1), Question Answering(v2.1), Natural Language Generation(v2.1). The original question answering datset featured 100,000 examples and was released in 2016. Leaderboard is now closed but data is availible below.

The current competitive tasks are Question Answering and Natural Language Generation. Question Answering features over 1,000,000 queries and is much like the original QnA dataset but bigger and with higher quality. The Natural Language Generation dataset features 180,000 examples and builds upon the QnA dataset to deliver answers that could be spoken by a smart speaker.

Tasks

1. QnA(v1.1 now closed):Given a query and 10 candidate passages select the most relvant one and use it to answer the question.
2. QnA(v2.1):Given a query and 10 candidate passages select the most relvant one and use it to answer the question.
3. NLGEN(v2.1):Given a query and 10 candidate passages select the most relvant one and use it to answer the question. Provide your answer in a way in which it could be read from a smart speaker and make sense without any additional context.

Relevant Links

Paper Question Answering Github Repo

Dataset Download links

Question Answering Task: RETIRED(03/01/2018-10/30/2020) Leaderboard

Natural Language Generation Task:RETIRED(03/01/2018-10/30/2020)

MS MARCO V1:RETIRED(12/01/2016-03/31/2018)

Usefulness Data(Released 02/02/2020)

Data associated with the WebConf 2020's paper Leading Conversational Search by Suggesting Useful Questions

Relevant Links

Usefulness Github Repo

Dataset Download links

Conversational Search(Released 04/23/2019)

Truly Conversational Search is the next logic step in the journey to generate intelligent and useful AI. To understand what this may mean, researchers have voiced a continuous desire to study how people currently converse with search engines. As a result we have released a large corpus of anonymized user search sessions.

We hope the community can use this corpus to explore what conversations with search engines look like.

Relevant Links

Conversational Search Github Repo

Dataset Download links

Optimal Crawling(Released 04/23/2019)

The dataset used for Optimal Freshness Crawl Under Politeness Constraints and Staying up to Date with Online Content Changes Using Reinforcement Learning for Scheduling which are both focused on providing an optimal crawling schedule for a search engine based ont he changing nature of the internet.

There is currently no public task associated with this dataset

Relevant Links

Paper Paper 2 Optimal Crawl Github Repo

Dataset Download links

Message to NLP community and MSMARCO Community

TLDR: We are closing the MSMARCO QnA and NLGEN Leaderboard. Last Submissions 10/23.

Dear NLP community and Question Answering enthusiasts,
When we released MSMARCO v2 back in March of 2018 we did not expect how much love this dataset would receive from the community. Needless to say we have been humbled by not only the number of submissions to the leaderboard but also all the remarkable research that incorporated this dataset as part of their benchmarking efforts. While we originally envisioned that this dataset will be useful to the NLP and QnA communities, we were again humbled by how the dataset was adopted and evolved by the IR community for document and passage retrieval tasks. However, as you may have guessed, maintaining a public resource like the MS MARCO leaderboard takes significant time and effort and we are grateful to our small but dedicated team of volunteers that maintain this website. As we look forward to the future, we believe that given the small size of this team and the limited resources, it is time to refocus our energy and time on the scenarios where MS MARCO can provide the most value to the research community moving forward. Towards that goal, we have made the hard (but what we believe is the right) decision to retire the Question answering and Language Generation leaderboard. Both tasks have not made large leaps in quality in the last year and we want to refocus our efforts on the document and passage retrieval tasks where the engagement with the research communities are actively growing in the present. As a result, the last day for any submissions to the MSMARCO Question Answering, Natural Language Generation, and KeyPhrase Extraction leaderboards is October 23, 2020. Submissions to both the document and passage retrieval leaderboards will continue as usual. We will continue to host all the datasets (including those specifically for the tasks being retired), as we believe they can still serve as valuable resources for future research. We want to again thank all the participants for their submissions and support for MS MARCO and we hope to see the community around the IR tasks continue to grow more in the future. We are always listening for feedback, so please continue to send us your suggestions and requests.

Sincerely
The MS MARCO Team

Changes To Dataset

10.23.2020Task Retirement

1. Retire QnA V2 Task 2. Retire NLGEN V2 Task 3. Retire OpenKP Task

08.11.2020New Task

1. Released Document Ranking task and 3 baselines.

07.30.2020New Data

1. Released ORCAS Click data

02.11.2020:New Data

1. Released Usefulness Data

10.22.2019:New Datasets

1. Released OpenKP Keyphrase Extraction dataset! 2. Released Optimal Crawling Dataset!

05.06.2019:Fixed Encoding issues with Ranking Dataset

1. Updated various encoding issues in ranking dataset.

04.23.2019:We have released a conversational search dataset

1. Brand new conversational search Dataset

10.26.2018:We have released a new ranking dataset based on the v2.1 dataset

1. Brand new Ranking Dataset
2. Basic Baseline and evaluation function

04.23.2018:We have released an updated to the dataset. V2.1 Includes the following:
1. Over 1 million queries
2. ~182k Well Formed Answers
3. Query type is now included for every query.
4. Bias in Evaluation set fixed(a small portion of answers for the V2.0 Evaluation set were able to be found in the v1.1 set and the v2.0 well formed sets, these have been removed from eval and added to train).
5. Utilities and Readme now availible.

03.01.2018:We have released an updated to the dataset. V2.0 Includes the following:
1. ~900,000 unique queries
2. ~160k Well Formed Answers

01.30.2017:We have released an update to the dataset! V1.1 contains the follwing:
1. Improvments to dataset and evaluation scripts

12.01.2016:We have released our dataset! V1.0 contains the follwing:
1. 100,000 unique query answer pairs

MS MARCO Submission Instructions

Once you have built a model that meets your expectations on evaluation with the dev set, you can submit your test results to get official evaluation on the test set. To ensure the integrity of the official test results, we do not release the correct answers for test set to the public.

To submit your model for official evaluation on the test set for the document ranking task, follow the instructions here.

To submit your model for official evaluation on the test set for other tasks, follow the below steps:

1. Generate your candidate output for the dev set.
2. Run the official evaluation methodologies found in the task specific git repo and verify your systems are running as expected
3. Generate your candidate output for the eval/test set and submit the following information by emailing us

Your email should include

1. Candidate evaluation file
2. Candidate dev file
3. Individual/Team Name: Name of the individual or the team to appear in the leaderboard [Required]
4. Individual/Team Institution: Name of the institution of the individual or the team to appear in the leaderboard [Optional]
5. Model code: Training code for the model [Recommended]
6. Model information: Name of the model/technique to appear in the leaderboard [Required]
7. Paper Information: Name, Citation, URL of the paper if model is from a published work to appear in the leaderboard [Optional]

To avoid "P-hacking" we discourage too many submissions from the same group in a short period of time.

About MS MARCO

Email us

Current Team