Bringing TiKV to Rust Devroom at FOSDEM 2018

At the crack of dawn on February 1, I landed in Brussels, Belgium, for the first time in my life. The goal of my trip wasn’t to taste the local cuisine, tour world-famous museums, or grab a pint of the local brew (though I ended up doing all those things anyway). It was to deliver a talk three days later at “FOSDEM 2018 Rust Devroom” about our experience at PingCAP using Rust to build TiKV, a distributed transactional Key-Value storage engine.

FOSDEM 2018 Rust Devroom is the 1st edition of a dedicated Rust venue co-located with FOSDEM 2018. It was, by all accounts, very popular; at least a few hundred people visited the Rust Devroom, where my talk was supposed to be held. Because the room could only hold about 120 people, people had to line up to get into talks. When I showed up for my event, there was already a long line snaking through the hallway. I lined up with everyone else and almost didn’t get into my own talk!

People lining up to listen to their favorite Rust talk. Photo credit: Andrew Hobden

You can find the slides of my presentation here. But we are all busy people, so just in case you don’t have time to go through it all, here’s a summary of the highlights and where you can go learn more!

Building TiKV, a distributed key-value store with Rust

Building a distributed key-value store with Rust

Presenting in front of a group of Rustaceans at FOSDEM 2018. Photo credit: Andrew Hobden

In my talk, I shared our experiences on the following topics from our process of building TiKV:

1. Why another database and the challenges of building a modern distributed transactional Key-Value store: consistency, horizontal scalability, scalability, stability, performance, ACID compliance, High Availability, etc.

2. How we build the TiKV core system using Rust, including the backend store, the gRPC framework, and the consensus replication mechanism.

• Backend store: TiKV adopts RocksDB as the backend storage engine for its high-performance and fast storage, and usesRaft and Multi-raft to ensure data safety and horizontal scalability. We implemented the Raft Consensus algorithm in Rust.

• Transaction: As a distributed Key-Value database, TiKV also supports transaction and is ACID compliant. To maintain consistency among the Multi-raft groups, TiKV adopts an optimized two-phase commit (2PC) protocol and supports Multiversion Concurrency Control (MVCC), Snapshot Isolation, and Optimistic Transaction.

• For different machines to communicate to each other, TiKV uses gRPC, a high-performance universal RPC framework. We develop agRPC library for Rust built on C Core library and futures.

3. How we monitor the cluster and diagnose problems: to monitor the metrics of the cluster and gain insights, TiKV adopts Prometheus as the monitoring solution. We build a Prometheus instrumentation library for Rustapplications, which is listed as a third-party client on the Official Prometheus Client Libraries.

4. How we use the failure injection test to guarantee data safety. Inspired by FreeBSD’s failpoints, TiKV implemented a fail point to inject errors by users at runtime in Rust.

The following diagram shows the TiKV architecture:

TiKV Architecture

Our team has big plans beyond just building a full-featured distributed transactional Key-Value database like TiKV. We have already built a stateless SQL layer, TiDB, mainly for Online transaction processing (OLTP) that works directly with TiKV, and TiSpark, an Apache Spark driver which sits on top of TiKV for handling heavy Online analytical processing(OLAP) workloads. Our ultimate vision is to build a Hybrid Transactional/Analytical Processing database that empowers businesses to meet both workloads with a single database and enables real-time business analysis based on live transactional data.

Architecture of TiDB, a Hybrid Transactional/Analytical Processing (HTAP) database

TiKV, TiDB, and TiSpark have been widely adopted in production environments by companies, ranging from e-commerce and gaming to financial services and bike-sharing. We are working to publish specific use case stories from our customers soon, and the best way to receive them is to subscribe to our blog. Stay tuned for more!

Last but not least, we welcome everyone to fork, star, use, and contribute to the following projects that we’ve written in Rust for TiKV:

• Rustwrapper for RocksDB

• Raft Consensus algorithm implemented inRust

• gRPC library for Rust built on C Core library and futures

• Prometheus instrumentation library for Rust applications

• Fail points for Rust

We look forward to building a strong and vibrant Rust community together!