Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LargeObjectFile should support cleanup in case of upload error #99

Open
majewsky opened this issue Oct 13, 2017 · 1 comment
Open

LargeObjectFile should support cleanup in case of upload error #99

majewsky opened this issue Oct 13, 2017 · 1 comment

Comments

@majewsky
Copy link
Contributor

majewsky commented Oct 13, 2017

The internal struct implementing the LargeObjectFile interface keeps track of which segments it has uploaded since {Dynamic,Static}LargeObjectCreate. When the upload fails (either during PUT of another segment or PUT of the manifest), it is useful to know which segments were already uploaded in order to perform cleanup.

Since LargeObjectFile does not have a method to expose its segments, one can only perform this cleanup by listing the uploaded segments, i.e.

names, err := conn.ObjectNameAll(segmentContainer,
  &swift.ObjectsOpts{Prefix: segmentPrefix},
)
... handle err ...
result, err := conn.BulkDelete(segmentContainer, names)

However, this is prone to eventual consistency issues: The segments might not yet be visible in the container listing. I see two possible options:

  1. Add a method like SegmentNames() []string to the LargeObjectFile interface.
  2. Add a method Cleanup() error or Reset() error to the LargeObjectFile interface that deletes all segments which were uploaded since {Dynamic,Static}LargeObjectCreate.
@ncw ncw added the enhancement label Nov 9, 2017
@ncw
Copy link
Owner

ncw commented Nov 9, 2017

I like your Cleanup() error proposal. Fancy sending a PR?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants