Bug: Can't update inside of a metadata context when iterating over a different query

[kbolashev]

Reproduction test (assuming a datasource with 1.txt..5.txt datapoints):

def test_add_in_a_different_context(datasource: "Datasource"):
    path = "1.txt"
    field = "field"

    with datasource.metadata_context() as ctx:
        ctx.update_metadata(path, {field: "old"})

    with datasource.metadata_context() as ctx:
        for dp in datasource[field].is_null().all():
            dp[field] = "new"

    res = [dp.metadata.get(field, None) for dp in datasource.all()]
    assert res == ["old", "new", "new", "new", "new"]

Right now this is failing, because res == ["old", None, None, None, None]

Happens because datasource[field].is_null().all() ends up creating a new datasource object, and so closing the original datasource's context doesn't end up uploading the implicit datapoints.

To fix this I extracted all implicit contexts in a global dictionary keyed by the datasource ID, that way no matter if you're doing subqueries, you're always getting the metadata context of this datasource that ends up being uploaded at some point.

Made sure that all the backend E2E tests are passing after this.

[dagshub]

Join the discussion on DagsHub!

[sdafni]

too bad we need a global thing, but i can't think of any other solution

[sdafni]

where is this test than?
separate PR?