Skip to content

Commit

Permalink
Merge #105368 #105624 #106515 #106939
Browse files Browse the repository at this point in the history 
105368: backupccl: add unit tests for FileSSTSink r=rhu713 a=rhu713

Backfill unit tests for the basic functionality of FileSSTSink with additional test cases involving inputs of keys with many entries in its revision history.

Epic: CRDB-27758

Release note: None

105624: jobsprofiler: dump trace recording on job completion r=dt a=adityamaru

This change teaches the job resumer to fetch and write its trace recording before finishing its tracing span. These traces will be consumed by the job profiler bundle that is being introduced in #105384. These traces will be valuable in understanding a job's execution characteristics during each resumption, even if the job has reached a terminal state.

Currently, this behaviour is opt-in and has been enabled for backups, restore, import and physical replication jobs.

Informs: #102794
Release note: None

106515: DEPS: bump across etcd-io/raft#81 and disable conf change validation r=erikgrinaker a=tbg

We don't want raft to validate conf changes, since that causes issues due to false positives (the check is above raft, but needs to be below raft to always work correctly). We are
taking responsibility for carrying out only valid conf changes, as we always
have.

See also etcd-io/raft#80.

Fixes #105797.
Epic: CRDB-25287
Release note (bug fix): under rare circumstances, a replication change could get
stuck when proposed near lease/leadership changes (and likely under overload),
and the replica circuit breakers could trip. This problem has been addressed.
Note to editors: this time it's really addressed (fingers crossed); a previous
attempt with an identical release note had to be reverted.


106939: changefeedccl: fix flake in TestParquetRows r=miretskiy a=jayshrivastava

changefeedccl: fix flake in TestParquetRows
Previously, this test would flake when rows were not emitted
in the exact order they were inserted/modified. This change
makes the test resilient to different ordering.

Epic: None
Fixes: #106911
Release note: None

---

util/parquet: make metadata transparent in tests
Previously, users of the library would need to explicitly
call `NewWriterWithReaderMetadata()` to configure the parquet
writer to add metadata required to use reader utils in
`pkg/util/parquet/testutils.go`. This led to a lot of code
uncessary code duplication. This moves the logic to decide if
metadata should be written to `NewWriter()` so callers do not
need to do the extra work.

Epic: None
Release note: None

Co-authored-by: Rui Hu <[email protected]>
Co-authored-by: adityamaru <[email protected]>
Co-authored-by: Tobias Grieger <[email protected]>
Co-authored-by: Jayant Shrivastava <[email protected]>
  • Loading branch information
@adityamaru @tbg @jayshrivastava
5 people committed Jul 17, 2023
5 parents 8cf0ba4 + 7165325 + 912983c + 4585dde + a1c5b0e commit 212d1e9
Show file tree
Hide file tree
Showing 44 changed files with 1,396 additions and 479 deletions.
6 changes: 3 additions & 3 deletions DEPS.bzl
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10577,10 +10577,10 @@ def go_deps():
],
build_file_proto_mode = "default",
importpath = "go.etcd.io/raft/v3",
sha256 = "7540ce70b9c79987eb5a09f693302931ac88e34757397451c5d44c202649adb3",
strip_prefix = "go.etcd.io/raft/[email protected]20230626154957-a10cd4571633",
sha256 = "292954320a69953ac8366cd7a83e4a50b1cfd858643a4cdf9947ab9607f1a142",
strip_prefix = "go.etcd.io/raft/[email protected]20230717153924-72a6e6c9f3ee",
urls = [
"https://storage.googleapis.com/cockroach-godeps/gomod/go.etcd.io/raft/v3/io_etcd_go_raft_v3-v3.0.0-20230626154957-a10cd4571633.zip",
"https://storage.googleapis.com/cockroach-godeps/gomod/go.etcd.io/raft/v3/io_etcd_go_raft_v3-v3.0.0-20230717153924-72a6e6c9f3ee.zip",
],
)
go_repository(
Expand Down
2 changes: 1 addition & 1 deletion build/bazelutil/distdir_files.bzl
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1023,7 +1023,7 @@ DISTDIR_FILES = {
"https://storage.googleapis.com/cockroach-godeps/gomod/go.etcd.io/etcd/client/pkg/v3/io_etcd_go_etcd_client_pkg_v3-v3.5.0.zip": "c0ca209767c5734c6ed023888ba5be02aab5bd3c4d018999467f2bfa8bf65ee3",
"https://storage.googleapis.com/cockroach-godeps/gomod/go.etcd.io/etcd/client/v2/io_etcd_go_etcd_client_v2-v2.305.0.zip": "91fcb507fe8c193844b56bfb6c8741aaeb6ffa11ee9043de2af0f141173679f3",
"https://storage.googleapis.com/cockroach-godeps/gomod/go.etcd.io/etcd/io_etcd_go_etcd-v0.5.0-alpha.5.0.20200910180754-dd1b699fc489.zip": "d982ee501979b41b68625693bad77d15e4ae79ab9d0eae5f6028205f96a74e49",
"https://storage.googleapis.com/cockroach-godeps/gomod/go.etcd.io/raft/v3/io_etcd_go_raft_v3-v3.0.0-20230626154957-a10cd4571633.zip": "7540ce70b9c79987eb5a09f693302931ac88e34757397451c5d44c202649adb3",
"https://storage.googleapis.com/cockroach-godeps/gomod/go.etcd.io/raft/v3/io_etcd_go_raft_v3-v3.0.0-20230717153924-72a6e6c9f3ee.zip": "292954320a69953ac8366cd7a83e4a50b1cfd858643a4cdf9947ab9607f1a142",
"https://storage.googleapis.com/cockroach-godeps/gomod/go.mongodb.org/mongo-driver/org_mongodb_go_mongo_driver-v1.5.1.zip": "446cff132e82c64af7ffcf48e268eb16ec81f694914aa6baecb06cbbae1be0d7",
"https://storage.googleapis.com/cockroach-godeps/gomod/go.mozilla.org/pkcs7/org_mozilla_go_pkcs7-v0.0.0-20200128120323-432b2356ecb1.zip": "3c4c1667907ff3127e371d44696326bad9e965216d4257917ae28e8b82a9e08d",
"https://storage.googleapis.com/cockroach-godeps/gomod/go.opencensus.io/io_opencensus_go-v0.24.0.zip": "203a767d7f8e7c1ebe5588220ad168d1e15b14ae70a636de7ca9a4a88a7e0d0c",
Expand Down
2 changes: 1 addition & 1 deletion go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -219,7 +219,7 @@ require (
github.com/xdg-go/scram v1.1.2
github.com/xdg-go/stringprep v1.0.4
github.com/zabawaba99/go-gitignore v0.0.0-20200117185801-39e6bddfb292
go.etcd.io/raft/v3 v3.0.0-20230626154957-a10cd4571633
go.etcd.io/raft/v3 v3.0.0-20230717153924-72a6e6c9f3ee
go.opentelemetry.io/otel v1.0.0-RC3
go.opentelemetry.io/otel/exporters/jaeger v1.0.0-RC3
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.0.0-RC3
Expand Down
4 changes: 2 additions & 2 deletions go.sum
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2320,8 +2320,8 @@ go.etcd.io/etcd v0.5.0-alpha.5.0.20200910180754-dd1b699fc489/go.mod h1:yVHk9ub3C
go.etcd.io/etcd/api/v3 v3.5.0/go.mod h1:cbVKeC6lCfl7j/8jBhAK6aIYO9XOjdptoxU/nLQcPvs=
go.etcd.io/etcd/client/pkg/v3 v3.5.0/go.mod h1:IJHfcCEKxYu1Os13ZdwCwIUTUVGYTSAM3YSwc9/Ac1g=
go.etcd.io/etcd/client/v2 v2.305.0/go.mod h1:h9puh54ZTgAKtEbut2oe9P4L/oqKCVB6xsXlzd7alYQ=
go.etcd.io/raft/v3 v3.0.0-20230626154957-a10cd4571633 h1:C6cmDqdkeGrJ6fT7OFCbnPauTEJvFY+HZSWk2N0PUvI=
go.etcd.io/raft/v3 v3.0.0-20230626154957-a10cd4571633/go.mod h1:tP6U+sRzrl75ltgmFcdZg9reZVEyM3vKTxAWmwpHtB8=
go.etcd.io/raft/v3 v3.0.0-20230717153924-72a6e6c9f3ee h1:7KfWqaO+jirXdEFWpwK1fa2NIm1nmBf3fP0AH6NydCk=
go.etcd.io/raft/v3 v3.0.0-20230717153924-72a6e6c9f3ee/go.mod h1:tP6U+sRzrl75ltgmFcdZg9reZVEyM3vKTxAWmwpHtB8=
go.mongodb.org/mongo-driver v1.0.3/go.mod h1:u7ryQJ+DOzQmeO7zB6MHyr8jkEQvC8vH7qLUO4lqsUM=
go.mongodb.org/mongo-driver v1.1.1/go.mod h1:u7ryQJ+DOzQmeO7zB6MHyr8jkEQvC8vH7qLUO4lqsUM=
go.mongodb.org/mongo-driver v1.1.2/go.mod h1:u7ryQJ+DOzQmeO7zB6MHyr8jkEQvC8vH7qLUO4lqsUM=
Expand Down
9 changes: 9 additions & 0 deletions pkg/ccl/backupccl/backup_job.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ import (
"github.com/cockroachdb/cockroach/pkg/ccl/utilccl"
"github.com/cockroachdb/cockroach/pkg/cloud"
"github.com/cockroachdb/cockroach/pkg/cloud/cloudpb"
"github.com/cockroachdb/cockroach/pkg/clusterversion"
"github.com/cockroachdb/cockroach/pkg/jobs"
"github.com/cockroachdb/cockroach/pkg/jobs/joberror"
"github.com/cockroachdb/cockroach/pkg/jobs/jobspb"
Expand Down Expand Up @@ -221,6 +222,9 @@ func backup(
// Create a channel that is large enough that it does not block.
perNodeProgressCh := make(chan map[execinfrapb.ComponentID]float32, numTotalSpans)
storePerNodeProgressLoop := func(ctx context.Context) error {
if !execCtx.ExecCfg().Settings.Version.IsActive(ctx, clusterversion.V23_1) {
return nil
}
for {
select {
case prog, ok := <-perNodeProgressCh:
Expand Down Expand Up @@ -477,6 +481,11 @@ func (b *backupResumer) ForceRealSpan() bool {
return true
}

// DumpTraceAfterRun implements the TraceableJob interface.
func (b *backupResumer) DumpTraceAfterRun() bool {
return true
}

// Resume is part of the jobs.Resumer interface.
func (b *backupResumer) Resume(ctx context.Context, execCtx interface{}) error {
// The span is finished by the registry executing the job.
Expand Down
24 changes: 16 additions & 8 deletions pkg/ccl/backupccl/file_sst_sink.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -47,17 +47,25 @@ type fileSSTSink struct {
out io.WriteCloser
outName string

flushedFiles []backuppb.BackupManifest_File
flushedSize int64
flushedFiles []backuppb.BackupManifest_File
flushedSize int64

// flushedRevStart is the earliest start time of the export responses
// written to this sink since the last flush. Resets on each flush.
flushedRevStart hlc.Timestamp
completedSpans int32

// completedSpans contain the number of completed spans since the last
// flush. This counter resets on each flush.
completedSpans int32

// stats contain statistics about the actions of the fileSSTSink over its
// entire lifespan.
stats struct {
files int
flushes int
oooFlushes int
sizeFlushes int
spanGrows int
files int // number of files created.
flushes int // number of flushes.
oooFlushes int // number of out of order flushes.
sizeFlushes int // number of flushes due to file exceeding targetFileSize.
spanGrows int // number of times a span was extended.
}
}

Expand Down
Loading

0 comments on commit 212d1e9

Please sign in to comment.