-
Notifications
You must be signed in to change notification settings - Fork 548
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Prevent dataloss due to the concurrent RPC calls (occurrence is very low) #4970
Conversation
/test ci/centos/mini-e2e/k8s-1.31 |
cafad3a
to
8c455bd
Compare
/test ci/centos/mini-e2e/k8s-1.31 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wow, that was quick, thanks. 🤩🚀 This is also pretty much the fix that we had in mind for this. 👍
|
||
return nil, status.Errorf(codes.Aborted, util.TargetPathOperationAlreadyExistsFmt, targetPath) | ||
} | ||
defer ns.VolumeLocks.Release(targetPath) | ||
|
||
// Check if that target path exists properly | ||
notMnt, err := ns.createTargetMountPath(ctx, targetPath, isBlock) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If I see it correctly, createTargetMountPath
still uses the (possibly unreliable?) IsLikelyNotMountPoint
under the hood; should this be adjusted to the same IsMountPoint
that the cephfs implementation uses? Or can this remain as-is, as discussed in #4966 (comment)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes thats correct, it checks the sources, The main problem is with lock and the os.RemoveAll, if os.Remove fails we will get another RPC for unpublish which will trigger umount
if its actually a mount. For now lets keep it simple with what we have today.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM.
It would be good to backport this to active releases, I think?
@Mergifyio queue |
✅ The pull request has been merged automaticallyThe pull request has been merged automatically at 00d252e |
We should not be dependent on the CO to ensure that it will serialize the request instead of that we need to have own internal locks to ensure that we dont do concurrent operations for same request. Signed-off-by: Madhu Rajanna <[email protected]>
We should not be dependent on the CO to ensure that it will serialize the request instead of that we need to have own internal locks to ensure that we dont do concurrent operations for same request. Signed-off-by: Madhu Rajanna <[email protected]>
using os.RemoveAll will remove everything in the director after the Umount we should be using os.Remove only to remove the empty directory Signed-off-by: Madhu Rajanna <[email protected]>
using os.RemoveAll will remove everything in the director after the Umount we should be using os.Remove only to remove the empty directory Signed-off-by: Madhu Rajanna <[email protected]>
/test ci/centos/k8s-e2e-external-storage/1.30 |
/test ci/centos/mini-e2e-helm/k8s-1.30 |
/test ci/centos/k8s-e2e-external-storage/1.31 |
/test ci/centos/mini-e2e/k8s-1.30 |
/test ci/centos/upgrade-tests-cephfs |
/test ci/centos/mini-e2e-helm/k8s-1.31 |
/test ci/centos/k8s-e2e-external-storage/1.29 |
/test ci/centos/mini-e2e/k8s-1.31 |
/test ci/centos/upgrade-tests-rbd |
/test ci/centos/mini-e2e-helm/k8s-1.29 |
/test ci/centos/mini-e2e/k8s-1.29 |
This PR includes series for commits for the following actions
Thanks to @martin-helmich @hensur and @Lucaber for the detailed analysis
fixes: #4966