Roadmap

This is a living document containing the FASTER team’s priorities as well as release notes for previous releases. Items refer to FASTER C# unless indicated otherwise. For C++ info, scroll to C++ Porting Notes.

(Scroll to Release Notes)

Past and Future Work

The following is a summary of the FASTER team’s past work and backlog for the next 6 months. Completed items are included to provide the context and progress of the work.

Past Work

General

• Full Read, Upsert, RMW functionality
• Bulk delete via key expiry: user-initiated truncation from head of log (ShiftBeginAddress)
• Persistence support for larger-than-memory data
• Scan operation support for scanning a specified range of log
• Ability to resize (grow) the hash table in powers of two (user-initiated)
• Separately configurable optional read cache for read-heavy workloads
• Support for separate user-defined object hash/equality comparers and object serializers
• Remove C# dynamic code-gen for quicker instantiation, stability, debuggability
• Full support for .NET core and Linux/Mac
• Individual key delete support
• Support iteration over all and only live key-value pairs (different from log scan)
• Full support for async interface to FasterKV
• Full support for async/await threading model in C#: PR
• IFunctions specified via sessions
• Remove generic type constraints, add default serializers and comparers for common types

Log, Cache, and Storage

• Variable sized keys and values using separate object log, see [here]
• Two allocators (specializing for Blittable and Generic key-value types), with common extensible base class
• Generic IDevice abstraction with out-of-the-box implementations for local storage
• Segmented log on storage (log file broken into segments of configurable size)
• Allocator support for copying reads to tail, useful for update-intensive workloads
• Highly configurable allocator parameters (log and read cache) with respect to memory size, page size, mutable fraction of memory, etc.
• Support for runtime shifting of address markers (e.g., begin, head, read-only) for dynamic tuning of memory contents of allocators (log and read cache).
• Log compaction by rolling forward older active keys
• Support for subscribing to the hybrid log (push-based, as record batches become read-only): PR
• Support for callback when records in hybrid log become read-only: PR
• Support for cloud storage, starting with Azure Page Blobs: PR
• Support for tiered storage: PR
• Support for sharded storage: PR
• Support for FasterLog as an independent first-class abstraction: PR
• Reduced log memory footprint significantly (min pages in memory reduced to 1)
• Improved performance of fine-grained epoch protection
• Support per-entry checksums and persistent iterators in FasterLog
• Full support for async interface to FasterLog
• Improved IDevice v2 interface without Overlapped, improved local storage scalability

Checkpoint and Recovery

• CPR-based checkpointing and recovery (snapshot and fold-over modes), see [here] for details
• Optional separate checkpointing of index and log (so that index may be checkpointed less frequently)
• Auto-recover to most recent checkpoint
• Make checkpointing use a pluggable user-specified interface for providing devices and performing metadata commit: PR
• Generic state machine for checkpointing (internals)
• Choose checkpoint type (Snapshot and FoldOver) on a per-checkpoint basis
• Process all checkpoints through a IDevice-based checkpoint and log commit manager
• Limit number of preloaded pages during recovery: PR

Ongoing and Future Work

• Look for key via chain starting at given logical addresses: PR
• Read record directly via logical address: PR
• Client-server interface to FASTER
• Scale-out and elasticity support
• RDMA IDevice implementation
• Support for Incremental Snapshot checkpoint type
• Expose incremental data structures over FasterKV hash chains
• Handle log logical addresses greater than 48 bit (up to 64 bit)
• Checksums for storage pages

Release Notes

Find all recent release notes via our releases page here. Older release notes are below:

FASTER v2019.10.31.1

• Support for FasterLog as an independent first-class abstraction: PR
• Reduced log memory footprint significantly (min pages in memory reduced to 1)
• Improved performance of fine-grained epoch protection
• Support per-entry checksums and persistent iterators in FasterLog
• Full support for async interface to FasterLog

FASTER v2019.8.27.1

• Improved support for varlen blittable allocator (iteration, compaction): PR
• BREAKING CHANGE: change return type for InPlaceUpdater and ConcurrentWriter functions to bool. User has to return true for usual in-place-update (IPU) behavior, and return false to force retry as read-copy-update (RCU).
• Support for cloud storage, starting with Azure Page Blobs: PR
• Support for tiered storage: PR
• Make checkpointing use a pluggable user-specified interface for providing devices and performing metadata commit: PR
• Support for sharded storage: PR

FASTER v2019.7.23.1

• Object log recovery bug fix: PR
• Option to enable file buffering for local storage device
• Optimizing what is loaded to hybrid log memory during recovery (prior head address onwards only)
• Removing direct call of callback when IO completes synchronously: PR
• Fixed checkpoint recovery bug: PR
• Adding FILE_SHARE_DELETE when deleteOnClose is used: PR
• Support for callback when records in hybrid log become read-only: PR

FASTER v2019.4.24.4

• Added support for variable sized (inline) structs without object log: PR
• Removed statics from codebase to better support multiple instances: PR
• Fixes related to scheduling pending operations: PR

FASTER v2019.4.1.1

• Log compaction by rolling forward older active keys: PR
• Individual key delete support: PR

FASTER v2019.3.16.1 (cumulative feature list)

• Full Read, Upsert, RMW functionality
• Bulk delete via key expiry: user-initiated truncation from head of log (ShiftBeginAddress)
• Persistence support for larger-than-memory data
• Scan operation support for scanning a specified range of log
• Ability to resize (grow) the hash table in powers of two (user-initiated)
• Separately configurable optional read cache for read-heavy workloads
• Support for separate user-defined object hash/equality comparers and object serializers
• Remove C# dynamic code-gen for quicker instantiation, stability, debuggability
• Full support for .NET core and Linux/Mac
• Experimental feature: DeleteFromMemory to delete recently added keys that are still in memory
• Variable sized keys and values using separate object log, see [here]
• Two allocators (specializing for Blittable and Generic key-value types), with common extensible base class
• Generic IDevice abstraction with out-of-the-box implementations for local storage
• Segmented log on storage (log file broken into segments of configurable size)
• Allocator support for copying reads to tail, useful for update-intensive workloads
• Highly configurable allocator parameters (log and read cache) with respect to memory size, page size, mutable fraction of memory, etc.
• Support for runtime shifting of address markers (e.g., begin, head, read-only) for dynamic tuning of memory contents of allocators (log and read cache).
• CPR-based checkpointing and recovery (snapshot and fold-over modes), see [here] for details
• Optional separate checkpointing of index and log (so that index may be checkpointed less frequently)
• Auto-recover to most recent checkpoint

C++ Porting Notes

FASTER C++ is a fairly direct port of FASTER C# using C++ based coding and style guidelines. It supports the following features as of now:

• Full Read, Upsert, RMW functionality
• Persistence support for larger-than-memory data
• Variable sized payloads; no separate object log, see [here]
• Log segments on storage, with truncation from head of log
• CPR-based checkpointing and recovery (both snapshot and fold-over modes), see [here]
• Ability to resize the hash table
• C++: Added a new value_size() method to RmwContext for RCU operations: PR
• Azure storage device: PR