A Brief History of Garnet
Hi everyone! I just wanted to start off this blog with a short history of Garnet and how it came to exist. At Microsoft Research, we have been working on storage technology for a while. In 2016, we started working on a new key-value store design based on epoch protection and a powerful storage API. This project, called FASTER, was open-sourced in 2018 and gained a lot of traction within Microsoft and in the larger community. FASTER has over 6k stars and over half a million NuGet downloads. Over the next several years, we built follow-on capabilities such as recoverability (CPR) and serverless support (Netherite), and the project was widely adopted.
Around 2021, we noticed a huge interest in remote cache-stores, particularly using APIs such as the RESP API of Redis. Developers loved the flexibility of both the API and the deployment model as a separate process/service. The cost savings compared to accessing cloud databases directly drove the adoption of caching layers, and these soon grew to take up a significant portion of the operating cost of large services. When the pandemic hit and online service usage spiked, there was a strong need for lowering costs and improving performance (throughput and latency) for such caching layers.
We took on the challenge of building a new system, called Garnet, which could provide extremely high performance end-to-end in a client-server setup while allowing clients to remain largely unmodified by adopting RESP, the most popular wire protocols out there. After a lot of design effort, we came up with a server threading model that could indeed make a huge end-to-end difference, often by orders-of-magnitude, in performance for basic get and set operations, with unmodified client code. This gave us the confidence to build out Garnet's feature set towards use in real scenarios.
The next question was API coverage. The Redis API is vast, and we were just a small research team. Thankfully, our stack was built on .NET, which made tremendous advances in both performance and richness of libraries. We designed a generic yet powerful way to define and use custom data structures, and were able to quickly implement complex datatypes such as Sorted Set, List, and Hash by reusing data structures in C#. We then built complex structures such as HyperLogLog and Bitmap as well, and added transactions and extensibility features.
The next requirement was scale-out and recovery, for which we designed write-ahead operation logging, sharding capability, replication, and key migration for dynamic scale-out. By keeping basic compatibility with the Redis API, we were able to add these features in a way that existing client code could be largely unchanged.
After thousands of unit tests and a couple of years working with first-party teams at Microsoft deploying Garnet in production (more on this in future blog posts!), we felt it was time to release it publicly, just like we did with FASTER five years back. We wanted developers across the planet to benefit from this powerful technology and contribute back to the codebase as well. We felt that the modern C# codebase would be particularly attractive here, in terms of ease of expansion, maintenance, and contribution. Coming from MSR, we also wanted people in academia to conduct research, collaborate with us, and expand various aspects of the system.
So, explore Garnet, see if you can find use for it in your applications, consider helping us out with expanding its capabilities, and as always, let us know what you think!