Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dataset Upload with Force Replace #9072

Open
wants to merge 43 commits into
base: dev
Choose a base branch
from
Open

Dataset Upload with Force Replace #9072

wants to merge 43 commits into from

Commits on Jul 22, 2024

  1. [syft/store] fix KeyValueStorePartition._remove_keys bug not removi… 

    …ng a unique key

[tests/unit] adding a test to delete and reupload a dataset with the same name
    @khoaguin
    khoaguin committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    d785b0e View commit details
    Browse the repository at this point in the history

  2. Merge branch 'dev' into upload-dataset-with-force-replace

    @khoaguin
    khoaguin committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    569cd93 View commit details
    Browse the repository at this point in the history

  3. [syft/chore] add message=... for some SyftError

    @khoaguin
    khoaguin committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    d2c6e7f View commit details
    Browse the repository at this point in the history

  4. Merge branch 'dev' into upload-dataset-with-force-replace

    @khoaguin
    khoaguin committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    a8d0618 View commit details
    Browse the repository at this point in the history

Commits on Jul 23, 2024

  1. [tests/unit] add a test for request service to check if its `stash.up… 

    …date` and `stash.delete` works
    @khoaguin
    khoaguin committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    e830aad View commit details
    Browse the repository at this point in the history

  2. [syft/dataset] implement upload dataset with force_replace 

    - fix bug `dataset.get_all` shows deleted dataset (now only show if specify with a flag)
- improve repr_html of deleted datasets
- test force_replace functionality in `test_reupload_dataset`
    @khoaguin
    khoaguin committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    61d5bf1 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'dev' into upload-dataset-with-force-replace

    @khoaguin
    khoaguin committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    bfe674a View commit details
    Browse the repository at this point in the history

Commits on Jul 24, 2024

  1. [syft/dataset] update DatasetUpdate

    @khoaguin
    khoaguin committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    6f10586 View commit details
    Browse the repository at this point in the history

Commits on Jul 25, 2024

  1. Merge branch 'dev' into upload-dataset-with-force-replace

    @khoaguin
    khoaguin committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    e40ce31 View commit details
    Browse the repository at this point in the history

  2. [syft/dataset] merge with dev and move logic to include deleted dat… 

    …asets to DatasetStash.get_all

- continue implementing uploading with force replace
[tests/unit] modify test_reupload_dataset to reflect new changes
    @khoaguin
    khoaguin committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    1e9861c View commit details
    Browse the repository at this point in the history

  3. [syft/datasite_client] refactoring upload_dataset: separating logic… 

    … that uploads a dataset's assets into a separate function
    @khoaguin
    khoaguin committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    50b2e0d View commit details
    Browse the repository at this point in the history

  4. [syft/datasite_client] separating logic to upload a new dataset and r… 

    …eplace an existed dataset into 2 functions
    @khoaguin
    khoaguin committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    1d907ee View commit details
    Browse the repository at this point in the history

  5. [syft/dataset_service] done DatasetService.replace endpoint (withou… 

    …t deleting the old action objects)
    @khoaguin
    khoaguin committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    d7912e1 View commit details
    Browse the repository at this point in the history

Commits on Jul 26, 2024

  1. [tests/unit] add a test for uploading with force replace a small dataset

    @khoaguin
    khoaguin committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    8c1b034 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'dev' into upload-dataset-with-force-replace

    @khoaguin
    khoaguin committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    fa89985 View commit details
    Browse the repository at this point in the history

Commits on Jul 30, 2024

  1. [syft/dataset] add blob storage entry id attributes for Asset and Cre… 

    …ateAsset
    @khoaguin
    khoaguin committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    4ee5e39 View commit details
    Browse the repository at this point in the history

  2. [tests/unit] implement unit test for uploading a big dataset with for… 

    …ce replace
    @khoaguin
    khoaguin committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    689ee61 View commit details
    Browse the repository at this point in the history

  3. [tests/unit] add a check to test access permissions for blob entries … 

    …after upload a big dataset
    @khoaguin
    khoaguin committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    9b63d2a View commit details
    Browse the repository at this point in the history

  4. Merge branch 'dev' into upload-dataset-with-force-replace

    @khoaguin
    khoaguin committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    df04ad2 View commit details
    Browse the repository at this point in the history

Commits on Jul 31, 2024

  1. Merge branch 'dev' into upload-dataset-with-force-replace

    @khoaguin
    khoaguin committed Jul 31, 2024
    Configuration menu
    Copy the full SHA
    85a9915 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'dev' into upload-dataset-with-force-replace

    @khoaguin
    khoaguin committed Jul 31, 2024
    Configuration menu
    Copy the full SHA
    4d6b30a View commit details
    Browse the repository at this point in the history

  3. [syft/dataset] add migrations for Asset and CreateAsset

    @khoaguin
    khoaguin committed Jul 31, 2024
    Configuration menu
    Copy the full SHA
    04bb965 View commit details
    Browse the repository at this point in the history

  4. [tests/unit] get the ds_client directly from conftest instead of an r… 

    …emoved function
    @khoaguin
    khoaguin committed Jul 31, 2024
    Configuration menu
    Copy the full SHA
    68b8b1c View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2024

  1. Merge branch 'dev' into upload-dataset-with-force-replace

    @khoaguin
    khoaguin committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    87caff9 View commit details
    Browse the repository at this point in the history

Commits on Aug 5, 2024

  1. Merge branch 'dev' into upload-dataset-with-force-replace

    @khoaguin
    khoaguin committed Aug 5, 2024
    Configuration menu
    Copy the full SHA
    859a166 View commit details
    Browse the repository at this point in the history

Commits on Aug 6, 2024

  1. Merge branch 'dev' into upload-dataset-with-force-replace

    @khoaguin
    khoaguin committed Aug 6, 2024
    Configuration menu
    Copy the full SHA
    c62e115 View commit details
    Browse the repository at this point in the history

Commits on Aug 7, 2024

  1. [syft/dataset] Define DatasetV1 class that contains a list of `Asse… 

    …tV1`

Co-authored-by: Eelco van der Wel <[email protected]>
    @khoaguin @eelcovdw
    khoaguin and eelcovdw committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    6c0487c View commit details
    Browse the repository at this point in the history

  2. [syft/dataset] migrating functions to transform list of assets from V… 

    …1 to V2 for Dataset
    @khoaguin
    khoaguin committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    59994b1 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'dev' into upload-dataset-with-force-replace

    @khoaguin
    khoaguin committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    5b7a12b View commit details
    Browse the repository at this point in the history

  4. fix lint

    @khoaguin
    khoaguin committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    b8788ad View commit details
    Browse the repository at this point in the history

Commits on Aug 8, 2024

  1. [syft/dataset] migrating functions to transform list of assets from … 

    …V2 to V1 for Dataset
    @khoaguin
    khoaguin committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    adcff8b View commit details
    Browse the repository at this point in the history

  2. [syft/dataset] handling exception when user does not have permission … 

    …to upload / replace a dataset

- Merge branch 'dev' into upload-dataset-with-force-replace
    @khoaguin
    khoaguin committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    c1b9c79 View commit details
    Browse the repository at this point in the history

  3. [tests/unit] modify `test_datasite_client_cannot_upload_dataset_with_… 

    …non_mock` since now we return a SyftError instead of rasing an exception
    @khoaguin
    khoaguin committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    21b5e6b View commit details
    Browse the repository at this point in the history

  4. protocol: fix handling of syft object subclass in annotation repr 

    dataset: add unique and search attr fields to DatasetV1 class
    @shubham3121
    shubham3121 committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    01ae483 View commit details
    Browse the repository at this point in the history

Commits on Aug 12, 2024

  1. Merge branch 'dev' into upload-dataset-with-force-replace

    @shubham3121
    shubham3121 committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    97dc55a View commit details
    Browse the repository at this point in the history

Commits on Aug 13, 2024

  1. [syft/dataset] remove mock and data blob entries on the Asset level a… 

    …nd corresponding migrations
    @khoaguin
    khoaguin committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    8d627ac View commit details
    Browse the repository at this point in the history

  2. Merge branch 'dev' into upload-dataset-with-force-replace

    @khoaguin
    khoaguin committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    a925f63 View commit details
    Browse the repository at this point in the history

Commits on Aug 16, 2024

  1. Merge branch 'dev' into upload-dataset-with-force-replace

    @khoaguin
    khoaguin committed Aug 16, 2024
    Configuration menu
    Copy the full SHA
    435754c View commit details
    Browse the repository at this point in the history

Commits on Aug 23, 2024

  1. Merge branch 'dev' into upload-dataset-with-force-replace

    @khoaguin
    khoaguin committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    adca0ee View commit details
    Browse the repository at this point in the history

Commits on Sep 10, 2024

  1. Merging datasite_client.py with dev

    @khoaguin
    khoaguin committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    a69b3de View commit details
    Browse the repository at this point in the history

  2. Merge branch 'dev' into upload-dataset-with-force-replace

    @khoaguin
    khoaguin committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    fff17e9 View commit details
    Browse the repository at this point in the history

  3. [dataset_service] stop unwrapping replace if success

    @khoaguin
    khoaguin committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    f053035 View commit details
    Browse the repository at this point in the history

  4. [protocol_version] add CreateAsset to protocol_version.json

    @khoaguin
    khoaguin committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    37a052c View commit details
    Browse the repository at this point in the history