Key-Value Serialisation

Every transaction executed by the primary on its Key-Value store is serialised before being replicated to all backups of the CCF network and written to the ledger. The serialisation format is defined per ccf::kv::Map and distinctly for the key and value types.

Tip

Selecting the right serialision format for a KV map depends on the application logic but is generally a trade-off between performance and auditability of the ledger. For example, the default serialisation format for ccf::kv::Map is JSON and allows for easy parsing of transactions in the public ledger. For more performance sensitive use cases, apps may define or use their own serialisers.

Custom key and value types

User-defined types can be used for both the key and value types of a ccf::kv::Map. It must be possible to use the key type as the key of an std::map (so it must be copyable, assignable, and less-comparable), and both types must be serialisable. By default, when using a ccf::kv::Map, serialisation converts to JSON. To add support to your custom types, it should usually be possible to use the DECLARE_JSON_... macros:

struct CustomClass
{
  std::string s;
  size_t n;
};

// These macros allow the default nlohmann JSON serialiser to be used
DECLARE_JSON_TYPE(CustomClass);
DECLARE_JSON_REQUIRED_FIELDS(CustomClass, s, n);

Custom serialisers can also be defined. The serialiser itself must be a type implementing to_serialised and from_serialised functions for the target type:

struct CustomSerialiser
{
  /**
   * Format:
   * [ 8 bytes=n | 8 bytes=size_of_s | size_of_s bytes=s... ]
   */

  static constexpr auto size_of_n = 8;
  static constexpr auto size_of_size_of_s = 8;
  static ccf::kv::serialisers::SerialisedEntry to_serialised(
    const CustomClass& cc)
  {
    const auto s_size = cc.s.size();
    const auto total_size = size_of_n + size_of_size_of_s + s_size;
    ccf::kv::serialisers::SerialisedEntry serialised(total_size);
    uint8_t* data = serialised.data();
    memcpy(data, (const uint8_t*)&cc.n, size_of_n);
    data += size_of_n;
    memcpy(data, (const uint8_t*)&s_size, size_of_size_of_s);
    data += size_of_size_of_s;
    memcpy(data, (const uint8_t*)cc.s.data(), s_size);
    return serialised;
  }

  static CustomClass from_serialised(
    const ccf::kv::serialisers::SerialisedEntry& ser)
  {
    CustomClass cc;
    const uint8_t* data = ser.data();
    cc.n = *(const uint64_t*)data;
    data += size_of_n;
    const auto s_size = *(const uint64_t*)data;
    data += size_of_size_of_s;
    cc.s.resize(s_size);
    std::memcpy(cc.s.data(), data, s_size);
    return cc;
  }
};

To use these serialised for a specific map declare the map as a ccf::kv::TypedMap, adding the appropriate serialiser types for the key and value types:

using CustomSerialisedMap = ccf::kv::
  TypedMap<CustomClass, CustomClass, CustomSerialiser, CustomSerialiser>;

Note

Any external tools which wish to parse the ledger will need to know the serialisation format of the tables they care about. It is recommended, though not enforced, that you size-prefix each entry so it can be skipped by tools which do not understand the serialised format.