Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Incorrect Error Message when Volume Mount Fails #4562

Open
cbears opened this issue Apr 16, 2024 · 1 comment · May be fixed by #4564
Open

Incorrect Error Message when Volume Mount Fails #4562

cbears opened this issue Apr 16, 2024 · 1 comment · May be fixed by #4564
Assignees
@nixpanic
Labels
bug Something isn't working component/rbd Issues related to RBD

Comments

@cbears
Copy link

cbears commented Apr 16, 2024

Describe the bug

Null variable backtrace when attempting to mount a volume as opposed to an error describing the issue.

Environment details

  • Image/version of Ceph CSI driver : origin/release-v3.11

  • Mounter used for mounting PVC (for cephFS its fuse or kernel. for rbd its
    krbd or rbd-nbd) : rbd

Steps to reproduce

Steps to reproduce the behavior:

  1. Cause a reason for the mount to fail, for instance, not being able to connect to ceph
  2. Observe bad error message.

Actual results

Expected behavior

The snippet below is from internal/rbd/nodeserver.go. When an error occurs variable rv is NULL, and thus the rv.Destroy() call can't possibly work. The code should check if rv is Null first and then do an rv.Destroy().

    rv, err = GenVolFromVolID(ctx, volID, cr, req.GetSecrets())
    if err != nil {
      rv.Destroy()
      log.ErrorLog(ctx, "error generating volume %s: %v", volID, err)

      return nil, status.Errorf(codes.Internal, "error generating volume %s: %v", volID, err)
    }

Logs

4f89650cde07543a9529ff93415f5eb006fce9f211fd63251e2ced5dcf5b9323/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":6}},"volume_id":"nginx-pv"}
2024-04-16T20:50:39.579157107Z stderr F E0416 20:50:39.579119 3190592 utils.go:220] panic occurred: runtime error: invalid memory address or nil pointer dereference
2024-04-16T20:50:39.579345709Z stderr F goroutine 15477 [running]:
2024-04-16T20:50:39.579350155Z stderr F runtime/debug.Stack()
2024-04-16T20:50:39.579352493Z stderr F 	/usr/local/go/src/runtime/debug/stack.go:24 +0x5e
2024-04-16T20:50:39.579354524Z stderr F runtime/debug.PrintStack()
2024-04-16T20:50:39.579359937Z stderr F 	/usr/local/go/src/runtime/debug/stack.go:16 +0x13
2024-04-16T20:50:39.579362102Z stderr F github.com/ceph/ceph-csi/internal/csi-common.panicHandler.func1()
2024-04-16T20:50:39.57936392Z stderr F 	/go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:221 +0xca
2024-04-16T20:50:39.579365844Z stderr F panic({0x1ea8620?, 0x3a72420?})
2024-04-16T20:50:39.579367951Z stderr F 	/usr/local/go/src/runtime/panic.go:914 +0x21f
2024-04-16T20:50:39.579370465Z stderr F github.com/ceph/ceph-csi/internal/rbd.(*NodeServer).populateRbdVol(0x27c0370?, {0x27c0370, 0xc00070f1d0}, 0xc0006b0300, 0xc?)
2024-04-16T20:50:39.579372381Z stderr F 	/go/src/github.com/ceph/ceph-csi/internal/rbd/nodeserver.go:201 +0xe10
2024-04-16T20:50:39.579374428Z stderr F github.com/ceph/ceph-csi/internal/rbd.(*NodeServer).NodeStageVolume(0xc00005e9d0, {0x27c0370?, 0xc00070f1d0}, 0xc0006b0300)
2024-04-16T20:50:39.579376247Z stderr F 	/go/src/github.com/ceph/ceph-csi/internal/rbd/nodeserver.go:344 +0x42d
2024-04-16T20:50:39.579378354Z stderr F github.com/container-storage-interface/spec/lib/go/csi._Node_NodeStageVolume_Handler.func1({0x27c0370, 0xc00070f1d0}, {0x20c3220?, 0xc0006b0300})
2024-04-16T20:50:39.579380296Z stderr F 	/go/src/github.com/ceph/ceph-csi/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:7223 +0x72
2024-04-16T20:50:39.579382162Z stderr F github.com/ceph/ceph-csi/internal/csi-common.panicHandler({0x27c0370?, 0xc00070f1d0?}, {0x20c3220?, 0xc0006b0300?}, 0xc000148c60?, 0xc000a8a140?)
2024-04-16T20:50:39.579388353Z stderr F 	/go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:226 +0x77
2024-04-16T20:50:39.579390295Z stderr F github.com/ceph/ceph-csi/internal/csi-common.NewMiddlewareServerOption.ChainUnaryServer.func2.1({0x27c0370?, 0xc00070f1d0?}, {0x20c3220?, 0xc0006b0300?})
2024-04-16T20:50:39.579392562Z stderr F 	/go/src/github.com/ceph/ceph-csi/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:48 +0x45
2024-04-16T20:50:39.579394677Z stderr F github.com/ceph/ceph-csi/internal/csi-common.logGRPC({0x27c0370, 0xc00070f1d0}, {0x20c3220?, 0xc0006b0300?}, 0x1d76520?, 0xc0000bcc00)
2024-04-16T20:50:39.57939655Z stderr F 	/go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:201 +0x183
2024-04-16T20:50:39.579398543Z stderr F github.com/ceph/ceph-csi/internal/csi-common.NewMiddlewareServerOption.ChainUnaryServer.func2.1({0x27c0370?, 0xc00070f1d0?}, {0x20c3220?, 0xc0006b0300?})
2024-04-16T20:50:39.579400417Z stderr F 	/go/src/github.com/ceph/ceph-csi/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:48 +0x45
2024-04-16T20:50:39.579407469Z stderr F github.com/ceph/ceph-csi/internal/csi-common.contextIDInjector({0x27c0370, 0xc00070f110}, {0x20c3220, 0xc0006b0300}, 0xc000a065e8?, 0xc0000bcc40)
2024-04-16T20:50:39.579409518Z stderr F 	/go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:189 +0x132
2024-04-16T20:50:39.579419808Z stderr F github.com/ceph/ceph-csi/internal/csi-common.NewMiddlewareServerOption.ChainUnaryServer.func2({0x27c0370, 0xc00070f110}, {0x20c3220, 0xc0006b0300}, 0xc0004666c0, 0x1ea9400?)
2024-04-16T20:50:39.579421812Z stderr F 	/go/src/github.com/ceph/ceph-csi/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:53 +0x135
2024-04-16T20:50:39.579423803Z stderr F github.com/container-storage-interface/spec/lib/go/csi._Node_NodeStageVolume_Handler({0x2125660?, 0xc00005e9d0}, {0x27c0370, 0xc00070f110}, 0xc00068e380, 0xc000293b90)
2024-04-16T20:50:39.579427137Z stderr F 	/go/src/github.com/ceph/ceph-csi/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:7225 +0x135
2024-04-16T20:50:39.57942903Z stderr F google.golang.org/grpc.(*Server).processUnaryRPC(0xc00043d000, {0x27c0370, 0xc00070f080}, {0x27c8c60, 0xc0002b96c0}, 0xc00088d200, 0xc000293e00, 0x3a88c60, 0x0)
2024-04-16T20:50:39.579431282Z stderr F 	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1386 +0xe23
2024-04-16T20:50:39.579433132Z stderr F google.golang.org/grpc.(*Server).handleStream(0xc00043d000, {0x27c8c60, 0xc0002b96c0}, 0xc00088d200)
2024-04-16T20:50:39.579434887Z stderr F 	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1797 +0x100c
2024-04-16T20:50:39.579436944Z stderr F google.golang.org/grpc.(*Server).serveStreams.func2.1()
2024-04-16T20:50:39.57943887Z stderr F 	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1027 +0x8b
2024-04-16T20:50:39.579440744Z stderr F created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 15476
2024-04-16T20:50:39.579442583Z stderr F 	/go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1038 +0x135
2024-04-16T20:50:39.579489141Z stderr F E0416 20:50:39.579456 3190592 utils.go:203] ID: 4176 Req-ID: nginx-pv GRPC error: rpc error: code = Internal desc = panic runtime error: invalid memory address or nil pointer dereference

Additional context

Add any other context about the problem here.

For example:

Any existing bug report which describe about the similar issue/behavior
@cbears
Copy link
Author

cbears commented Apr 16, 2024

Error msg:

MountVolume.MountDevice failed for volume "nginx-pv" : rpc error: code = Internal desc = panic runtime error: invalid memory address or nil pointer dereference

@nixpanic nixpanic added bug Something isn't working component/rbd Issues related to RBD labels Apr 17, 2024
@nixpanic nixpanic self-assigned this Apr 17, 2024
@nixpanic nixpanic linked a pull request Apr 17, 2024 that will close this issue
Open
nixpanic added a commit to nixpanic/ceph-csi that referenced this issue Apr 17, 2024
@nixpanic
rbd: prevent calling rbdVolume.Destroy() after an error
a4b6ef4 
It seems possible that the .Destroy() function is called on a nil
pointer. This would cause a panic in the node-plugin.

Depending on how far GenVolFromVolID() comes, the returned rbdVolume can
be connected. If an error occurs, the rbdVolume should not be connected
at all, so call the .Destroy() function in those cases too.

Fixes: ceph#4562
Signed-off-by: Niels de Vos <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nixpanic nixpanic

Labels
bug Something isn't working component/rbd Issues related to RBD
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

rbd: prevent calling rbdVolume.Destroy() after an error nixpanic/ceph-csi
2 participants
@nixpanic @cbears