Transforming Web3 Data Infrastructure with TiDB Serverless

Chaintool is a pioneering risk management platform for Web3 businesses. With its goal to revolutionize the way metadata is shared and consumed in the Web3 industry, Chaintool, along with multiple other leading companies in the Web3 domain, has developed Microscope, an open-source metadata management platform.

Microscope is designed with the mission to advance the Web3 industry through efficient, reliable, and accessible metadata management. To be specially, their goal is to streamline metadata contribution, sharing, and consumption through a sophisticated sharing mechanism, collaborative tools, and incentivization schemes. As the platform expands its product categories, it’s expected to become standard data infrastructure for the Web3 industry.

Chaintool’s role as a platform operator is dual-pronged. While it incentivizes and encourages all participants to contribute and share data through various operational means, it also aims to optimize collaborative tools to ensure a smooth and user-friendly experience in data upload, management, and consumption.

To achieve this mission, Chaintool has chosen TiDB Serverless, a fully-managed, serverless DBaaS deployment of TiDB as their underlying technology. TiDB is an advanced, open source, distributed SQL database for modern applications. 

In this post, we dive deeper into why Chaintool selected TiDB’s serverless DBaaS and how it aligns with their business and operational needs. 

Understanding the data landscape 

Microscope’s data processing characteristics encompass both transactional and analytical requirements. Key aspects include:

• OLTP focus at early stage: Initially, the data sharing platform relies heavily on Online Transaction Processing (OLTP), including standard database operations such as insertions, deletions, modifications, and retrieval. The stored data relates to transactional tags rather than the raw transactional data. This keeps the overall data volume relatively compact, with an expected current capacity of hundreds of GBs and a potential scale-up to tens of TBs in the long run.
• Increasing storage requirements: As more institutional participants join the platform, the need for data storage is set to grow linearly. This growth requires batch data storage capabilities.
• Rising API calls:  As the platform evolves, Chaintool predicts a rise in API data calls and requests from both its enterprise and individual users. 
• Emerging OLAP needs: The platform has identified an emerging need for Online Analytical Processing (OLAP). This is mainly for operational analysis from a user perspective, with a lower real-time query requirement. The platform’s queries per second (QPS) are expected to reach into the hundreds. 

Challenges and infra selection criteria

The data processing scenarios facing Chaintool are not complicated and demanding, with matured solutions on the market. However, for a young innovative project, the challenges mostly come from the project rather than the underlying infrastructure and technology. 

• User trust and data contribution: As an innovative data-sharing platform, Chaintool’s first challenge was to establish trust among its users and encourage them to contribute data.
• Involvement of diverse stakeholders: The project involves participation from multiple organizations with varied technical expertise. This challenge is to ensure all engineers could seamlessly integrate into the project with minimal learning curves.
• Adapting to rapid business innovation: As an early-stage venture with fast-paced iterations, the platform demanded an adaptable infrastructure that could accommodate agile development and reduce costs, especially during the critical early phase with immense uncertainties.

These challenges translated into specific requirements for the underlying data infrastructure:

• Scalable and performant database: A requirement for a database that can scale and perform effectively, with integrated OLTP and OLAP capabilities.
• Ease of use: Easy onboarding with a low learning curve for engineers.
• Cost-efficient pricing model: Considering the dynamic nature of the project, a flexible, consumption-based pricing model that would financially favor the business was vital to success.

The solution: TiDB Serverless

Upon careful evaluation, Chaintool selected TiDB for its unique capabilities and advantages that effectively addressed their challenges. Here are the reasons for their selection:

• Scalability and growth: TiDB Serverless automatically scales hundreds of nodes up and down to meet critical workloads in real-time. This level of scalability perfectly accommodated the expected growth of the data-sharing platform.
• Ease of use: TiDB’s serverless DBaaS is a fully-managed service with a straightforward interface design and efficient documentation. This allows for easy adoption, even for those with minimal database experience.
• Pay-as-you-go pricing: TiDB Serverless is charged only for actual usage, allowing Chaintool to flexibly configure resources according to the dynamic business needs and progress. This pricing model freed the technical users from complex cost planning and was especially useful during the project’s early stages.
• HTAP capabilities: The solution offered a simple yet powerful HTAP capability that eliminated the need for ETL processes, providing real-time analysis with a single click.
• Support for agile business: TiDB Serverless came equipped with features like online DDL, non-blocking index additions, and Adhoc query support. These are instrumental in accommodating the agile and evolving nature of Chaintool’s operations.
• Seamless data migration: TiDB Serverless offered a seamless data migration experience using a MySQL standard library Python API and product-based batch import. This ensured smooth and efficient data transition for the platform.

TiDB Serverless, in essence, provided an easy-to-adopt, cost-effective, and scalable solution, ensuring Chaintool could effectively meet its technical needs while optimizing for future growth.

Benefits of TiDB Serverless

Over the past six months, TiDB Serverless has provided Microscope with numerous advantages that significantly improved its business operations: 

• Easier onboarding and collaboration: The ease of onboarding offered by the platform facilitated the involvement of more partners in the project, which lowered collaboration and communication expenses and guaranteed agile project iteration.
• Optimized costs and adaptability to business evolution: As Chaintool’s project expanded and the business requirements evolved, TiDB Serverless’s consumption-based pricing significantly reduced costs while accommodating fluctuating demands and project progress. 
• Future-ready database solution: TiDB’s serverless DBaaS offers both elastic scalability and HTAP capabilities, enabling it to support Chaintool’s growing data storage requirements, and ensure a future-ready database solution as business evolves. 

As the platform evolves, Chaintool plans to explore more TiDB Serverless use cases, including the potential use of TiFlash.