Announcing TiDB 6.1

Author: Shawn Ma (Product Marketing Manager at PingCAP)

Editors: Calvin Weng, Tom Dewan

We are happy to announce that TiDB 6.1 was released on June 13. This release marks the official addition of the Long Term Support (LTS) release to our existing Development Milestone Release (DMR) mode. The combination of these two different release modes of TiDB can take into account both the rapidly changing market demands and the stability requirements from enterprise users.

In this post, I will formally introduce the LTS release and its corresponding stability updates. In addition, I will share some enhancements to TiDB’s HTAP capabilities and integrations with the ecosystem. 

For the full list of enhancements and fixes, see the TiDB 6.1 Release Notes.

Introducing Long Term Support 

Our DMR releases are every two months, so we can quickly respond to the ever-changing requirements or trends in the market. Users don’t need to wait long for their new feature requests to be answered. However, there won’t be long term maintenance for all DMR releases. This is where the LTS release comes in. Every six months, a DMR version will be selected for an LTS release. Within the support term, the LTS version will receive bug or issue fixes continuously. Therefore, If you don’t have any specific feature requests, it’s always safe to deploy the latest LTS version and use it in production. On the other hand, if you have a specific feature requirement from a DMR release, you can choose that DMR version for a Proof of Concept (PoC) trial and wait until the corresponding LTS version is released to upgrade TiDB to a stable production state. 

As the first LTS release, TiDB 6.1 includes the following stability updates:

• Improved memory stability in high stress scenarios of TiKV
• Resolved an out of memory (OOM) issue caused by excessive Raft log replication traffic
• Resolved the issue of repeated long lasting OOM errors while restarting TiDB after crash in high stress scenarios
• Improved TiDB Server memory usage tracking statistics and strengthened the query memory quota limit
• Optimized statistics memory usage

There are also 42 fixes for known issues. For full details, see the TiDB 6.1 Release Notes.

Enhanced HTAP capabilities 

A top priority in all TiDB releases is enhancing its performance as an HTAP database. TiDB 6.1 is packed with the following HTAP enhancements:

• Partitioning general availability (GA) for analytics engine
• Window function support under Massively Parallel Processing (MPP)
• Non-transactional data manipulation language (DML)
• TiFlash optimizations for write performance

Partitioning GA for analytics engine

List partitioning and dynamic pruning were introduced as experimental features in TiDB 5.1. 

However, due to performance issues with static pruning under the MPP mode, TiFlash did not fully support the partitioning feature. After months of development, we are glad to share that list partitioning, dynamic partition pruning, and partitioning support under MPP are now generally available. 

Unlike row-based storage, a columnar storage engine only supports coarse-grained indexing. This cannot satisfy the data filtering requirement, especially in analytics scenarios. Partitioning, the most efficient data filtering method for column storage, has a very high priority in analytical scenarios. 

For example, take the order management scenario. A common way to partition order data is by creation date, which divides a month’s data into 30 partitions. However, the analytical queries conducted by the user often focus more on the data within the past three to seven days. If partitioning is not supported for the analytics engine, TiDB could only perform a full table scan. Filtering the date field wastes storage bandwidth and CPU usage. With TiDB 6.1, queries in this scenario can be a couple of times faster according to the benchmark test. For more information on partitioning in TiDB, see the partitioning documentation.

More efficient querying with window functions

Another enhancement in TiDB 6.1 that benefits analytical scenarios is the support of window functions. In this version, the framework support for window functions is implemented for the MPP execution, and three most commonly used functions, rank, dense_rank, and row_number are supported. Thanks to the distributed architecture, the memory consumption of window functions is offloaded, and its performance greatly improved. Based on our sample test with a 100 GB data set, window function computing is 282% faster when querying with a single node in the MPP mode. Using multiple nodes will boost the computing efficiency even more. We will gradually support more window functions in the future releases.

Faster batch processing with non-transactional DML

In addition, TiDB 6.1 introduces non-transactional DML statements to process data changes in large batches. Non-transactional DML statements split a common DML into multiple SQL executions to enhance the performance and ease of use in batch processing scenarios. Note that this enhancement works at the expense of transaction atomicity and isolation. In this release, TiDB only supports non-transactional DELETE operations. For example, you can use the following statement to eliminate the expired data without worrying about the transaction size limit:

BATCH ON id LIMIT 2 DELETE FROM orders where create_date < '2022-05-01';

This statement executes the DELETE operation in two batches to control the memory usage of a single operation.

Better integration and connection with the ecosystem 

A database is never meant to be used alone, and TiDB is no exception. As an open source distributed NewSQL database, TiDB has continuously improved its connection with the broader ecosystem. In this new version, TiDB introduces user-level locks and support for synchronizing data to Kafka in Avro format under PingCAP’s change data capture tool, TiCDC.

User-level locks 

User-level locks are a user-named lock management system provided by MySQL built-in functions. They can be used at places where SQL statements accept SQL functions. In SQL statements like SELECT or UPDATE, you can block clauses like GROUP or WHERE at different code points. . User-level locks are widely used in object-relational mapping (ORM) frameworks such as Rails, Elixir, and Ecto. TiDB 6.1 introduces MySQL-compatible lock management functions such as GET_LOCK, RELEASE_LOCK, and RELEASE_ALL_LOCKS. More will be added in future releases. 

Embracing Avro

For TiDB, TiCDC is the export hub of real-time changes of cluster data. This makes it vital to support commonly used data formats for more efficient consumer parsing and simpler development. Avro is a data serialization system supported by many data analytics and data integration products. It features a compressed binary format, high transmission rates, and rich data formats. TiCDC can convert TiDB’s incremental data into Avro format before sending it to Kafka. This integrates TiDB with many other platforms and tools in the ecosystem, such as Kafka, Snowflake, and SQL Server. In TiDB 6.1, real-time synchronization of TiDB data changes to other systems is made easier with the help of the Avro format—be it real-time data integration or change of subscription-triggered operations. 

Get started now with TiDB 6.1 

As the first official LTS version with the enhancements mentioned above, TiDB 6.1 is an ideal choice for all users who had feature requests in TiDB 6.0. Whether you are already using TiDB 6.0 or are just investigating, it is recommended that you start directly with TiDB 6.1 or upgrade to 6.1 via TiUP. 

We will work with the TiDB community to continuously deliver the powerful capabilities and stable experiences that you would expect from an LTS version.

TiDB 6.1 will be available on TiDB Cloud soon. Stay tuned.

You are welcome to join our community on Slack and TiDB Internals to share your thoughts and feedback with us.