Performance Tips ​

Batched Reads and Writes ​

Generated protocol reader and writer classes have read and write methods for each step. When a step is a stream, there will also be overloads that read or write a batch of values as an std::vector in one go. For small data types, using batched reads and writes can make a dramatic difference in throughput, especially for HDF5 files.

Use Fixed Data Types When Possible ​

For HDF5, using variable-length collections (like !vector without a length or !array without fixed dimension sizes) has lower throughput than their fixed-sized counterparts.

Avoid NDJSON Encoding ​

The NDJSON serialization format is great for debugging and interoperability with other tools (like jq) but is it orders of magnitude less efficient than the binary or HDF5 formats.