Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sentry: collection.go:258: adding uncommitted × × (151) version 1083: memory usage exceeds limit for uncommittedDescriptors: root: memory budget exceeded: 10240 bytes requested, 1610606678 currently... #136073

Open
cockroach-sentry opened this issue Nov 23, 2024 · 1 comment
Open

Sentry: collection.go:258: adding uncommitted × × (151) version 1083: memory usage exceeds limit for uncommittedDescriptors: root: memory budget exceeded: 10240 bytes requested, 1610606678 currently... #136073

cockroach-sentry opened this issue Nov 23, 2024 · 1 comment
Labels
branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) X-blathers-triaged blathers was able to find an owner

Comments

@cockroach-sentry
Copy link
Collaborator

cockroach-sentry commented Nov 23, 2024 •
edited by cockroach-jira-scripts
Loading

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/6085258209/?referrer=webhooks_plugin

Panic Message:

collection.go:258: adding uncommitted × × (151) version 1083: memory usage exceeds limit for uncommittedDescriptors: root: memory budget exceeded: 10240 bytes requested, 1610606678 currently allocated, 1610612736 bytes in budget
(1)
Wraps: (2) assertion failure
Wraps: (3) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).AddUncommittedDescriptor.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/collection.go:258
  | github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).AddUncommittedDescriptor
  | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/collection.go:273
  | github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).WriteDescToBatch
  | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/collection.go:304
  | github.com/cockroachdb/cockroach/pkg/sql.(*planner).writeTableDescToBatch
  | 	github.com/cockroachdb/cockroach/pkg/sql/table.go:340
  | github.com/cockroachdb/cockroach/pkg/sql.(*planner).writeSchemaChangeToBatch
  | 	github.com/cockroachdb/cockroach/pkg/sql/table.go:297
  | github.com/cockroachdb/cockroach/pkg/sql.(*changeDescriptorBackedPrivilegesNode).startExec
  | 	github.com/cockroachdb/cockroach/pkg/sql/grant_revoke.go:372
  | github.com/cockroachdb/cockroach/pkg/sql.startExec.func2
  | 	github.com/cockroachdb/cockroach/pkg/sql/plan.go:520
  | github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/walk.go:112
  | github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal
  | 	github.com/cockroachdb/cockroach/pkg/sql/walk.go:299
  | github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visit
  | 	github.com/cockroachdb/cockroach/pkg/sql/walk.go:79
  | github.com/cockroachdb/cockroach/pkg/sql.walkPlan
  | 	github.com/cockroachdb/cockroach/pkg/sql/walk.go:43
  | github.com/cockroachdb/cockroach/pkg/sql.startExec
  | 	github.com/cockroachdb/cockroach/pkg/sql/plan.go:523
  | github.com/cockroachdb/cockroach/pkg/sql.(*planNodeToRowSource).Start
  | 	github.com/cockroachdb/cockroach/pkg/sql/plan_node_to_row_source.go:175
  | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Columnarizer).Init
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/columnarizer.go:183
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).init
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:94
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:92
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Init
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:103
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).Run.(*BatchFlowCoordinator).init.func2
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:245
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:92
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).init
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:244
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).Run
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:278
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Run
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:320
  | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run
  | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:905
  | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun
  | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1945
  | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll.func3
  | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1660
  | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll
  | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1663
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2401
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1940
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1125
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:145
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithProfiling
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:3399
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:144
Wraps: (4) adding uncommitted × × (151) version 1083
Wraps: (5) memory usage exceeds limit for uncommittedDescriptors: root: memory budget exceeded: 10240 bytes requested, 1610606678 currently allocated, 1610612736 bytes in budget
  | -- cause hidden behind barrier
  | memory usage exceeds limit for uncommittedDescriptors: root: memory budget exceeded: 10240 bytes requested, 1610606678 currently allocated, 1610612736 bytes in budget
  | (1) attached stack trace
  |   -- stack trace:
  |   | github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*uncommittedDescriptors).upsert
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/uncommitted_descriptors.go:200
  |   | github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).AddUncommittedDescriptor
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/collection.go:273
  |   | github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).WriteDescToBatch
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/collection.go:304
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*planner).writeTableDescToBatch
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/table.go:340
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*planner).writeSchemaChangeToBatch
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/table.go:297
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*changeDescriptorBackedPrivilegesNode).startExec
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/grant_revoke.go:372
  |   | github.com/cockroachdb/cockroach/pkg/sql.startExec.func2
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/plan.go:520
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal.func1
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/walk.go:112
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/walk.go:299
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visit
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/walk.go:79
  |   | github.com/cockroachdb/cockroach/pkg/sql.walkPlan
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/walk.go:43
  |   | github.com/cockroachdb/cockroach/pkg/sql.startExec
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/plan.go:523
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*planNodeToRowSource).Start
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/plan_node_to_row_source.go:175
  |   | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Columnarizer).Init
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/columnarizer.go:183
  |   | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).init
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:94
  |   | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:92
  |   | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Init
  |   | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:103
  |   | github.com/cockroachdb/cockroach...
Stacktrace (expand for inline code snippets):

cockroach/pkg/sql/conn_executor_exec.go

Lines 143 to 145 in 03498e5

}
err = ex.execWithProfiling(ctx, ast, preparedStmt, func(ctx context.Context) error {
ev, payload, err = ex.execStmtInOpenState(ctx, parserStmt, portal, pinfo, res, canAutoCommit)

cockroach/pkg/sql/conn_executor_exec.go

Lines 3398 to 3400 in 03498e5

} else {
err = op(ctx)
}

cockroach/pkg/sql/conn_executor_exec.go

Lines 144 to 146 in 03498e5

err = ex.execWithProfiling(ctx, ast, preparedStmt, func(ctx context.Context) error {
ev, payload, err = ex.execStmtInOpenState(ctx, parserStmt, portal, pinfo, res, canAutoCommit)
return err

cockroach/pkg/sql/conn_executor_exec.go

Lines 1124 to 1126 in 03498e5

} else {
if err := ex.dispatchToExecutionEngine(stmtCtx, p, res); err != nil {
stmtThresholdSpan.Finish()

cockroach/pkg/sql/conn_executor_exec.go

Lines 1939 to 1941 in 03498e5

ex.sessionTracing.TraceExecStart(ctx, "distributed")
stats, err = ex.execWithDistSQLEngine(
ctx, planner, stmt.AST.StatementReturnType(), res, distribute, progAtomic,

cockroach/pkg/sql/conn_executor_exec.go

Lines 2400 to 2402 in 03498e5

}
err = ex.server.cfg.DistSQLPlanner.PlanAndRunAll(ctx, evalCtx, planCtx, planner, recv, evalCtxFactory)
}

cockroach/pkg/sql/distsql_running.go

Lines 1662 to 1664 in 03498e5

)
}()

cockroach/pkg/sql/distsql_running.go

Lines 1659 to 1661 in 03498e5

defer cleanup()
dsp.PlanAndRun(
ctx, evalCtx, planCtx, planner.txn, planner.curPlan.main, recv, finishedSetupFn,

cockroach/pkg/sql/distsql_running.go

Lines 1944 to 1946 in 03498e5

recv.expectedRowsRead = int64(physPlan.TotalEstimatedScannedRows)
dsp.Run(ctx, planCtx, txn, physPlan, recv, evalCtx, finishedSetupFn)
}

cockroach/pkg/sql/distsql_running.go

Lines 904 to 906 in 03498e5

noWait := planCtx.getPortalPauseInfo() != nil
flow.Run(ctx, noWait)
}

cockroach/pkg/sql/colflow/vectorized_flow.go

Lines 319 to 321 in 03498e5

log.VEvent(ctx, 1, "running the batch flow coordinator in the flow's goroutine")
f.batchFlowCoordinator.Run(ctx)
}

cockroach/pkg/sql/colflow/flow_coordinator.go

Lines 277 to 279 in 03498e5

if err := f.init(ctx); err != nil {
f.pushError(err)

cockroach/pkg/sql/colflow/flow_coordinator.go

Lines 243 to 245 in 03498e5

func (f *BatchFlowCoordinator) init(ctx context.Context) error {
return colexecerror.CatchVectorizedRuntimeError(func() {
f.input.Root.Init(ctx)

cockroach/pkg/sql/colexecerror/error.go

Lines 91 to 93 in 03498e5

}()
operation()
return retErr

cockroach/pkg/sql/colflow/flow_coordinator.go

Lines 244 to 246 in 03498e5

return colexecerror.CatchVectorizedRuntimeError(func() {
f.input.Root.Init(ctx)
})

cockroach/pkg/sql/colflow/stats.go

Lines 102 to 104 in 03498e5

// execution time (e.g. in the statement bundle).
err := colexecerror.CatchVectorizedRuntimeError(bic.init)
bic.stopwatch.Stop()

cockroach/pkg/sql/colexecerror/error.go

Lines 91 to 93 in 03498e5

}()
operation()
return retErr

cockroach/pkg/sql/colflow/stats.go

Lines 93 to 95 in 03498e5

func (bic *batchInfoCollector) init() {
bic.Input.Init(bic.ctx)
}

cockroach/pkg/sql/colexec/columnarizer.go

Lines 182 to 184 in 03498e5

ctx = c.StartInternal(ctx, "columnarizer" /* name */)
c.input.Start(ctx)
if execStatsHijacker, ok := c.input.(execinfra.ExecStatsForTraceHijacker); ok {

cockroach/pkg/sql/plan_node_to_row_source.go

Lines 174 to 176 in 03498e5

// This starts all of the nodes below this node.
if err := startExec(p.params, p.node); err != nil {
p.MoveToDraining(err)

cockroach/pkg/sql/plan.go

Lines 522 to 524 in 03498e5

}
return walkPlan(params.ctx, plan, o)
}

cockroach/pkg/sql/walk.go

Lines 42 to 44 in 03498e5

v := makePlanVisitor(ctx, observer)
v.visit(plan)
return v.err

cockroach/pkg/sql/walk.go

Lines 78 to 80 in 03498e5

}
v.visitInternal(plan, name)
return plan

cockroach/pkg/sql/walk.go

Lines 298 to 300 in 03498e5

}
}

cockroach/pkg/sql/walk.go

Lines 111 to 113 in 03498e5

}
v.err = v.observer.leaveNode(name, plan)
}()

cockroach/pkg/sql/plan.go

Lines 519 to 521 in 03498e5

}
return n.startExec(params)
},

cockroach/pkg/sql/grant_revoke.go

Lines 371 to 373 in 03498e5

if !d.Dropped() {
if err := p.writeSchemaChangeToBatch(ctx, d, b); err != nil {
return err

cockroach/pkg/sql/table.go

Lines 296 to 298 in 03498e5

}
return p.writeTableDescToBatch(ctx, tableDesc, b)
}

cockroach/pkg/sql/table.go

Lines 339 to 341 in 03498e5

return p.Descriptors().WriteDescToBatch(
ctx, p.extendedEvalCtx.Tracing.KVTracingEnabled(), tableDesc, b,

cockroach/pkg/sql/catalog/descs/collection.go

Lines 303 to 305 in 03498e5

if err := tc.AddUncommittedDescriptor(ctx, desc); err != nil {
return err

cockroach/pkg/sql/catalog/descs/collection.go

Lines 272 to 274 in 03498e5

tc.markAsShadowedName(desc.GetID())
return tc.uncommitted.upsert(ctx, desc)
}

cockroach/pkg/sql/catalog/descs/collection.go

Lines 257 to 259 in 03498e5

if err != nil {
err = errors.NewAssertionErrorWithWrappedErrf(err, "adding uncommitted %s %q (%d) version %d",
desc.DescriptorType(), desc.GetName(), desc.GetID(), desc.GetVersion()) 

pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt at line 144
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execWithProfiling at line 3399
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt.func1 at line 145
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmtInOpenState at line 1125
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).dispatchToExecutionEngine at line 1940
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execWithDistSQLEngine at line 2401
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRunAll at line 1663
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRunAll.func3 at line 1660
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRun at line 1945
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).Run at line 905
pkg/sql/colflow/vectorized_flow.go in pkg/sql/colflow.(*vectorizedFlow).Run at line 320
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).Run at line 278
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).init at line 244
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 92
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).Run.(*BatchFlowCoordinator).init.func2 at line 245
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).Init at line 103
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 92
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).init at line 94
pkg/sql/colexec/columnarizer.go in pkg/sql/colexec.(*Columnarizer).Init at line 183
pkg/sql/plan_node_to_row_source.go in pkg/sql.(*planNodeToRowSource).Start at line 175
pkg/sql/plan.go in pkg/sql.startExec at line 523
pkg/sql/walk.go in pkg/sql.walkPlan at line 43
pkg/sql/walk.go in pkg/sql.(*planVisitor).visit at line 79
pkg/sql/walk.go in pkg/sql.(*planVisitor).visitInternal at line 299
pkg/sql/walk.go in pkg/sql.(*planVisitor).visitInternal.func1 at line 112
pkg/sql/plan.go in pkg/sql.startExec.func2 at line 520
pkg/sql/grant_revoke.go in pkg/sql.(*changeDescriptorBackedPrivilegesNode).startExec at line 372
pkg/sql/table.go in pkg/sql.(*planner).writeSchemaChangeToBatch at line 297
pkg/sql/table.go in pkg/sql.(*planner).writeTableDescToBatch at line 340
pkg/sql/catalog/descs/collection.go in pkg/sql/catalog/descs.(*Collection).WriteDescToBatch at line 304
pkg/sql/catalog/descs/collection.go in pkg/sql/catalog/descs.(*Collection).AddUncommittedDescriptor at line 273
pkg/sql/catalog/descs/collection.go in pkg/sql/catalog/descs.(*Collection).AddUncommittedDescriptor.func1 at line 258

Tags

Tag Value
Command server
Environment v23.2.3
Go Version go1.21.5 X:nocoverageredesign
Platform linux amd64
Distribution CCL
Cockroach Release v23.2.3
Cockroach SHA 03498e5
# of CPUs 32
# of Goroutines 651

Jira issue: CRDB-44856
@cockroach-sentry cockroach-sentry added branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. labels Nov 23, 2024
@blathers-crl blathers[crl]
Copy link

blathers-crl bot commented Nov 23, 2024

CC'ing via the CODEOWNERS-based sentry heuristic:

  • @cockroachdb/sql-foundations

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@blathers-crl blathers-crl bot added T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) X-blathers-triaged blathers was able to find an owner labels Nov 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) X-blathers-triaged blathers was able to find an owner
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@cockroach-sentry