Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge TiKV near 7.4 #351

Merged
merged 42 commits into from
Aug 28, 2023
Merged

Conversation

CalvinNeo
Copy link
Member

What problem does this PR solve?

Issue Number: close #xxx

Problem Summary:

What is changed and how it works?

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

None

MyonKeminta and others added 30 commits July 28, 2023 09:28
)

close tikv#15109

This PR updates the way to calculate the last_change_info, which is not very proper before. This also fixes a assertion failure 
in `next_last_change_info` that might happen when it executes in parallel with GC in compaction filter.

Signed-off-by: MyonKeminta <[email protected]>
close tikv#15122

Signed-off-by: hillium <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
…5185)

ref tikv#15108

* raftstore-v2: support unsafe recovery create peer

  Reuse split mechanism to create peer. Because in v2 the only way to
  create and initialize to peer is via a snapshot message.

* raftstore-v2: support unsafe recovery destroy peer

Signed-off-by: Neil Shen <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ref tikv#15108

* raftstore-v2: support unsafe recovery demote
* raftstore-v2: do not step force leader on confchange

Signed-off-by: Neil Shen <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Co-authored-by: Connor <[email protected]>
…ikv#15284)

close tikv#15282

trigger compaction when no valid split key can be found

Signed-off-by: SpadeA-Tang <[email protected]>
…kv#15249)

ref tikv#15192

Signed-off-by: 3pointer <[email protected]>

Co-authored-by: tonyxuqqi <[email protected]>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
…it. (tikv#15247)

close tikv#15246

Modify the return value of the scan function under the RawStorage trait.

* fix the scan function only return Value.

Signed-off-by: joker53-1 <[email protected]>

Co-authored-by: Andy Lok <[email protected]>
ref tikv#12842

- Restore merge status when there's an error between trim check and propose.
- Fix a case where snapshot just before prepare merge can cause rollback.
- Mute an assertion that fails in v1 bulk_insert testing.
- Use SkipUntil in compaction filter, together with the newly introduced
  UnsafeFilter API, this should remove any tombstones regardless of the context.

Signed-off-by: tabokie <[email protected]>
ref tikv#15258

Signed-off-by: Neil Shen <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
close tikv#15288

Add error type for "mismatch peer id".

Signed-off-by: you06 <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
close tikv#15310

Rust uses CamelCase for struct names.
And add a test for #[doc(hidden)] field.

Signed-off-by: Neil Shen <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
close tikv#15200

reject read index if the applied index + raft_log_gc_count_limit <= read_index

Signed-off-by: bufferflies <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ref tikv#15276

Signed-off-by: hillium <[email protected]>

Co-authored-by: qupeng <[email protected]>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ref tikv#15056

Add logs for split region so that we know the time a split takes end-to-end: from the split proposed to the last splitt region finishes the initialization.

Signed-off-by: tonyxuqqi <[email protected]>
close tikv#15324

Ribbon filter may cause OOM  when compacting a sst with many entries.

See tikv#15324

Signed-off-by: Neil Shen <[email protected]>
ref tikv#15279

Also, the timeout for etcd implementation was reduced to 5s.

Signed-off-by: hillium <[email protected]>
Signed-off-by: 山岚 <[email protected]>
close tikv#15328

Signed-off-by: Yang Keao <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
…io. (tikv#15291)

ref tikv#15070, close tikv#15268

Make a bugfix on the detection strategy on slowness when there exists io-hang exceptions.

Signed-off-by: lucasliang <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
close tikv#15346

Make online unsafe recovery abort on timeout

Signed-off-by: Connor1996 <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
…v#15307)

close tikv#15306

add deadline check in get and batch_get request handler.

Signed-off-by: crazycs520 <[email protected]>
close tikv#15344

Using space in log kv can cause slog to emit "", which inflates
the log file. Also, we prefer underscores instead of hyphens as
it aligns with Rust variable naming conventions.
By unifying the style of kv names, makes it easier to search
information during debugging.

Signed-off-by: Neil Shen <[email protected]>
close tikv#15355

Disable `periodic-compaction` and `ttl` features as default.

Signed-off-by: lucasliang <[email protected]>
close tikv#15316

Signed-off-by: glorv <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ref tikv#15082

Record and print `duration to last update_safe_ts` and `duration to last consume leader`.

Signed-off-by: ekexium <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
overvenus and others added 11 commits August 21, 2023 09:29
close tikv#15304

resource_metering: free leaked thread CPU stats

Signed-off-by: Neil Shen <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
…#15342)

ref tikv#15282

consider duplicated mvcc versions when check compact
close tikv#15388

Signed-off-by: wshwsh12 <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
…ket's version is stale (tikv#15224)

close tikv#15123

reject cop request if the given buckets verison is stale, 
 and return the new error called `bucket_version_not_match` which will carry with the latest bucket keys, 
so the client should update it cache depend on this errror. And then split this 
cop request again.

Signed-off-by: bufferflies <[email protected]>
ref tikv#15401

Add some metrics about async snapshot

Signed-off-by: SpadeA-Tang <[email protected]>
close tikv#15397

Signed-off-by: Yang Keao <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
close tikv#15371

Fix incorrect raftstore applys memory trace. When performing dropping apply fsm, it should reset the 
memory trace to zero instead of calling `update_memory_trace`, otherwise it would record extra footage
of the interior vector capacity.

Signed-off-by: Connor1996 <[email protected]>

Co-authored-by: tonyxuqqi <[email protected]>
ref tikv#13311

Adding more log information for diagnosing the issue where inconsistencies between `region` and `region range` information in the meta may occur during region split and self-destroy scenarios.

Signed-off-by: cfzjywxk <[email protected]>
ref tikv#15269

Make flow control of compaction pending bytes global in
Partitioned-raft-kv as well.
The major reason is because compaction pending bytes is a trend signal.
If overall the TiKV's pending bytes is increasing, then it means 
the write speed is too fast and it needs to slow down. Essentially it
means TiKV's capability is not enough to handle the traffic. 
And the side benefit is that it could greatly save the memory because
we only need one VecDeq object for compaction pending bytes instead of
one per region.
The last reason to not make it regional is that the threshold is hard
to set because the proper threshold depends on the region size 
(The idea is that if there're fewer regions, then we have higher
tolerance in the extreme case if we only have one region, then it
should be same as SingleRocksDB.), and that's the indicator that we
should make it global.

Signed-off-by: tonyxuqqi <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Co-authored-by: Neil Shen <[email protected]>
…s panel (tikv#15415)

close tikv#15413

update procinfo to hanlde no data in context switch metrics panel

Signed-off-by: SpadeA-Tang <[email protected]>
@CLAassistant
Copy link

CLAassistant commented Aug 24, 2023

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
9 out of 22 committers have signed the CLA.

✅ HuSharp
✅ SpadeA-Tang
✅ lijie0123
✅ YangKeao
✅ glorv
✅ ekexium
✅ wshwsh12
✅ Leavrth
✅ CalvinNeo
❌ MyonKeminta
❌ overvenus
❌ YuJuncen
❌ 3pointer
❌ joker53-1
❌ tabokie
❌ you06
❌ tonyxuqqi
❌ LykxSassinator
❌ Connor1996
❌ bufferflies
❌ crazycs520
❌ cfzjywxk
You have signed the CLA already but the status is still pending? Let us recheck it.

@CalvinNeo CalvinNeo changed the title Merge tikv near 7.4 Merge TiKV near 7.4 Aug 24, 2023
@CalvinNeo CalvinNeo merged commit 8872f4d into pingcap:raftstore-proxy Aug 28, 2023
2 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.