Yardl

Appearance

Introduction ​

Yardl is a simple schema language and command-line tool that generates domain types and serialization code.

Simple Example

Given a Yardl definition like this:

yaml
# This is an example protocol, which is defined as a Header value
# followed by a stream of zero or more Sample values
MyProtocol: !protocol
  sequence:

    # A Header value
    header: Header

    # A stream of Samples
    samples: !stream
      items: Sample

# Header is a record with a single string field
Header: !record
  fields:
    subject: string

# Sample is a record made up of a datetime and
# a vector of integers
Sample: !record
  fields:

    # The time the sample was taken
    timestamp: datetime

    # A vector of integers
    data: int*
# This is an example protocol, which is defined as a Header value
# followed by a stream of zero or more Sample values
MyProtocol: !protocol
  sequence:

    # A Header value
    header: Header

    # A stream of Samples
    samples: !stream
      items: Sample

# Header is a record with a single string field
Header: !record
  fields:
    subject: string

# Sample is a record made up of a datetime and
# a vector of integers
Sample: !record
  fields:

    # The time the sample was taken
    timestamp: datetime

    # A vector of integers
    data: int*

After running yardl generate, you can write code like the following to write data to standard out in a compact binary format:

cpp
#include <iostream>

#include "generated/binary/protocols.h"

int main() {
  playground::binary::MyProtocolWriter writer(std::cout);

  writer.WriteHeader({"123"});

  writer.WriteSamples({std::chrono::system_clock::now(), {1, 2, 3}});
  writer.WriteSamples({std::chrono::system_clock::now(), {4, 5, 6, 7}});

  // signal the end of the samples stream
  writer.EndSamples();
}
#include <iostream>

#include "generated/binary/protocols.h"

int main() {
  playground::binary::MyProtocolWriter writer(std::cout);

  writer.WriteHeader({"123"});

  writer.WriteSamples({std::chrono::system_clock::now(), {1, 2, 3}});
  writer.WriteSamples({std::chrono::system_clock::now(), {4, 5, 6, 7}});

  // signal the end of the samples stream
  writer.EndSamples();
}

And then another executable can read it in from standard in:

cpp
#include <iostream>

#include "generated/binary/protocols.h"

int main() {
  playground::binary::MyProtocolReader reader(filename);

  playground::Header header;
  reader.ReadHeader(header);

  std::cout << "Read Header.subject: " << header.subject << std::endl;

  playground::Sample sample;
  while (reader.ReadSamples(sample)) {
    std::cout << "Read Sample.data.size(): " << sample.data.size() << std::endl;
  }

  return 0;
}
#include <iostream>

#include "generated/binary/protocols.h"

int main() {
  playground::binary::MyProtocolReader reader(filename);

  playground::Header header;
  reader.ReadHeader(header);

  std::cout << "Read Header.subject: " << header.subject << std::endl;

  playground::Sample sample;
  while (reader.ReadSamples(sample)) {
    std::cout << "Read Sample.data.size(): " << sample.data.size() << std::endl;
  }

  return 0;
}

Motivation ​

Yardl is conceptually similar to, and inspired by, Avro, Protocol Buffers, Bond, and others, but it was designed primarily with raw medical instrument signal data in mind. Some of its features are:

  • Persistence to HDF5 files as well as a compact binary format suitable for streaming over a network. There is also a much less efficient NDJSON format that is easier to manually inspect or use with other tools.
  • Built-in support for multidimensional arrays and complex numbers.
  • The schema is always embedded in the serialized data
  • "Clean" generated code with types that are easy to program against.
  • Generics
  • Computed fields

Modeling a data domain in Yardl brings a number of benefits over writing the code by hand:

  • Writing correct and efficient serialization code can be tricky
  • Schema versioning, compatibility, and conversions are handled for you
  • You do not need to worry about consistency across different programming languages
  • Comments could be used to generate documentation

Project Status ​

We are releasing this project order to get community feedback and contributions. It is not complete and is not ready for production use at this time. We expect to introduce breaking changes until the project reaches V1.

We currently support C++ and Python codegen. Other planned features include:

  • Python and MATLAB support for reading data with a different schema version
  • Constraints
  • Improvements to the language and editing experience