Memory usage of Restoring KVStore is much larger than expected

[Lloyd-Pottiger]

Bug Report

Summary

In v6.5.9, after a restart, TiFlash recovered with 2 GiB of uncommitted data, occupying 7.4 GiB of memory.

Test Procedure

1. Start a v6.5.9 Cluster

   tiup cluster deploy -y test v6.5.9 topology.yaml 
   tiup cluster start test

2. BR Import chbenchmark-1500

   tiup br:v6.5.9 restore full --pd http://172.31.8.1:2379 --send-credentials-to-tikv=false --check-requirements=false --storage=s3://qa-workload-datasets/benchmark/ch-1k-v5 --s3.region=us-west-2

3. Create TiFlash Replicas

   mysql --host 172.31.7.1 --port 4000 -uroot
   mysql> use tpcc;
   mysql> alter database tpcc set tiflash replica 1;

   mysql> select * from information_schema.tiflash_replica;

4. Create a New Table and Start a Transaction Without Committing

   mysql> create table tpcc.orders2 like tpcc.orders;
   mysql> alter table tpcc.orders2 set tiflash replica 1;

   mysql> set autocommit = false;
   mysql> set tidb_mem_quota_query = 12 * 1024 * 1024 * 1024;

   mysql> begin;
   mysql> insert into tpcc.orders2 select * from tpcc.orders limit 15000000;

5. Observe Data Volume in kvstore

   cd /tidb-deploy/tiflash-9000/data/page/kvstore
   du ./* -h -d 1

   218M        ./blobfile_10
   251M        ./blobfile_11
   207M        ./blobfile_12
   235M        ./blobfile_13
   207M        ./blobfile_14
   207M        ./blobfile_15
   243M        ./blobfile_4
   241M        ./blobfile_5
   207M        ./blobfile_6
   207M        ./blobfile_7
   202M        ./blobfile_8
   248M        ./blobfile_9
   364K        ./wal
   

   htop -p <tiflash_pid>

   Memory Usage: 9.6 GiB

6. Restart TiFlash Process

   sudo kill -9 <tiflash_pid>
   sleep 5

   htop -p <new_tiflash_pid>

   Peak Memory Usage During Startup: ~10 GiB
   Memory Usage After Startup:

7. Commit the Transaction

   mysql> commit

   Peak Memory Usage During Write: 12 GiB
   Memory Usage After Write: 2.6 GiB