Dataset and Leaderboard

What is SMCalFlow?

One central challenge in conversational AI is the design of a dialogue state representation that agents can use to reason about the information and actions available to them. We’ve developed a new representational framework for dialogue that enables efficient machine learning of complex conversations.

SMCalFlow is a large English-language dialogue dataset, featuring natural conversations about tasks involving calendars, weather, places, and people. Each turn is annotated with an executable dataflow program featuring API calls, function composition, and complex constraints built from strings, numbers, dates and times. For more information about dataflow-based dialogue agents and details about the dataset, check out our blog post and paper:

Blog postTask-Oriented Dialogue as Dataflow Synthesis (TACL '20)

The code and links to trained models are available on github:


Getting Started

To participate in the leaderboard, download a copy of the SMCalFlow dataset:

The dataset is distributed under theCC BY-SA 4.0 license.

Once you have a built a model that works to your expectations on the dev set, you submit it to get official scores on the dev and a hidden test set. To preserve the integrity of test results, we do not release the test set to the public. Instead, we require you to submit your model so that we can run it on the test set for you. We will not access your code or model for any purpose other than to evaluate your model on SMCalFlow. Here's a tutorial walking you through official evaluation of your model:

Submission Tutorial

Have Questions?

Email us at and



RankModelTest AccuracyTest Accuracy Ignoring Refer


May 31, 2021
Value-Agnostic Conversational Semantic Parsing (single model)

Semantic Machines



Aug 16, 2020
Seq2Seq (single model)

Semantic Machines