Yardl

Appearance

Introduction ​

Yardl is a simple schema language and command-line tool that generates domain types and serialization code.

Simple Example

Given a Yardl definition like this:

yaml
# This is an example protocol, which is defined as a Header value
# followed by a stream of zero or more Sample values
MyProtocol: !protocol
  sequence:
    header: Header
    samples: !stream
      items: Sample

# Header is a record with a single string field
Header: !record
  fields:
    subject: string

# Sample is a record made up of a datetime and
# a vector of integers
Sample: !record
  fields:
    timestamp: datetime
    data: int*
# This is an example protocol, which is defined as a Header value
# followed by a stream of zero or more Sample values
MyProtocol: !protocol
  sequence:
    header: Header
    samples: !stream
      items: Sample

# Header is a record with a single string field
Header: !record
  fields:
    subject: string

# Sample is a record made up of a datetime and
# a vector of integers
Sample: !record
  fields:
    timestamp: datetime
    data: int*

After running yardl generate, you can write code like the following to write data to a file in a compact binary format:

matlab
addpath("./matlab/");

Sample = @sandbox.Sample;
samples = [Sample(yardl.DateTime.now(), [1, 2, 3]), Sample(yardl.DateTime.now(), [4, 5, 6])];

outfile = "sandbox.bin";

w = sandbox.binary.MyProtocolWriter(outfile);
w.write_header(sandbox.Header("Me"));
w.write_samples(samples);
w.end_samples();
w.close();
addpath("./matlab/");

Sample = @sandbox.Sample;
samples = [Sample(yardl.DateTime.now(), [1, 2, 3]), Sample(yardl.DateTime.now(), [4, 5, 6])];

outfile = "sandbox.bin";

w = sandbox.binary.MyProtocolWriter(outfile);
w.write_header(sandbox.Header("Me"));
w.write_samples(samples);
w.end_samples();
w.close();

And then another script can read it in from the file:

matlab
addpath("./matlab/");

infile = "sandbox.bin";

r = sandbox.binary.MyProtocolReader(infile);
disp(r.read_header());
while r.has_samples()
    sample = r.read_samples();
    disp(sample.timestamp.to_datetime());
    disp(sample.data);
end
r.close();
addpath("./matlab/");

infile = "sandbox.bin";

r = sandbox.binary.MyProtocolReader(infile);
disp(r.read_header());
while r.has_samples()
    sample = r.read_samples();
    disp(sample.timestamp.to_datetime());
    disp(sample.data);
end
r.close();

Motivation ​

Yardl is conceptually similar to, and inspired by, Avro, Protocol Buffers, Bond, and others, but it was designed primarily with raw medical instrument signal data in mind. Some of its features are:

  • Persistence to HDF5 files as well as a compact binary format suitable for streaming over a network. There is also a much less efficient NDJSON format that is easier to manually inspect or use with other tools.
  • Built-in support for multidimensional arrays and complex numbers.
  • The schema is always embedded in the serialized data
  • "Clean" generated code with types that are easy to program against.
  • Generics
  • Computed fields

Modeling a data domain in Yardl brings a number of benefits over writing the code by hand:

  • Writing correct and efficient serialization code can be tricky
  • Schema versioning, compatibility, and conversions are handled for you
  • You do not need to worry about consistency across different programming languages
  • Comments could be used to generate documentation

Project Status ​

We are releasing this project order to get community feedback and contributions. It is not complete and is not ready for production use at this time. We expect to introduce breaking changes until the project reaches V1.

We currently support C++ and Python codegen. Other planned features include:

  • Python and MATLAB support for reading data with a different schema version
  • Constraints
  • Improvements to the language and editing experience