Tracking 3.0 Release Blockers

[jhamman]

We are now in the last mile of the 3.0 refactor. This issue is tracking what we are considering release blockers: There is also a milestone for After 3.0.0 where we are putting tickets that can come later.

Docs

• [v3] Documentation: Update developer docs #1765
• [v3] Documentation: Update API docs #1766
• [v3] Documentation: update tutorial docs #1767
• [v3] Documentation: How to extend Zarr #1770
• [V3] v2 -> v3 migration #1798
• Add documentation for zarr.array #2245
• Add documentation for zarr.group #2246
• Add documentation for zarr.api.asynchronous #2247
• Add documentation for zarr.api.synchronous #2248
• Add documentation for zarr.metadata #2251
• Add documentation for zarr.asyncgroup #2252
• Add documentation for AsyncArray #2253
• Links in function signatures not being resolved #2289

Array

• [v3] Support numcodecs via entrypoints  #1750
• [v3] Support zero-sized arrays #1977
• [v3] array written by tensorstore returns all nulls #2029
• [v3] default compressor / codec pipeline #2267
• [v3] V2 Codec pipeline is not consistent with legacy usage of filters #2325
• [v3] Fixed-width unicode string support in zarr v3 #2347
• [v3] AttributeError: 'Array' object has no attribute 'append' #2404
• [v3] support write_empty_chunks #2409

Group

• [v3] reimplement / develop Group.info and Array.info properties #2135
• [v3] Group.{zeros, ones, etc.} should accept positional name arguments #2402
• [v3] Implement Group.tree #2406

Top level API

• [v3] array creation: codecs #1943
• [v3] zarr.save - AttributeError: 'str' object has no attribute 'shape' #2403
• [v3] Implement zarr.copy and zarr.copy_all #2407
• [v3] zarr.array silently ignores order kwarg #2408
• [v3] It is possible to create v3 data with features not currently written into the spec (dtypes, codecs, etc.) #2411

Store API

• [v3] Finalize store API #1755
• [v3] Implement store methods for erase and rename operations #2108
• [v3] deleting a group does not delete subgroups #2191
• RemoteStore doesn't strip protocol correctly, breaking list operations #2342
• [v3] Re-opening an existing RemoteStore V3 group causes the first child to be deleted #2359

Testing / CI

• [v3] CI ENH: Add testing with the development version of dependencies #1496
• [v3] Test environments and CI #1648
• [v3] port zarr-python 2 test suite to v3 branch #1759
• [v3] Develop integration test suite that checks compatibility with downstream libraries  #1762
• [v3] Windows support #2438

This is non exhaustive but these are the major known issues. Feel free to post additional tickets and we can evaluate if they are really blockers.

[DimitriPapadopoulos]

I would suggest:

• Inconsistent overridden method #2377
• Cyclic imports #2380
• Review TODOs before v3? #2419

[will-moore]

Hi,
Thanks for all the work towards v3.
I wanted to check if this is a blocker / existing issue etc or if I should create one...?
This is using main branch 680142f :

import zarr
from zarr.storage import LocalStore

path = "root.zarr"
fmt = 2
store = LocalStore(path, mode="w")
root = zarr.group(store=store, zarr_format=fmt)

root_group = zarr.open(f"{path}", zarr_format=fmt)

Gives me:

$ python test_open_v2_group.py 
Traceback (most recent call last):
  File "/Users/wmoore/Desktop/ZARR/ome-zarr-py/test_open_v2_group.py", line 48, in <module>
    root_group = zarr.open(f"{path}", zarr_format=fmt)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/_compat.py", line 43, in inner_f
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/api/synchronous.py", line 77, in open
    obj = sync(
          ^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/sync.py", line 141, in sync
    raise return_result
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/sync.py", line 100, in _runner
    return await coro
           ^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/api/asynchronous.py", line 308, in open
    return await open_array(store=store_path, zarr_format=zarr_format, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/api/asynchronous.py", line 1077, in open_array
    return await AsyncArray.open(store_path, zarr_format=zarr_format)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/array.py", line 680, in open
    metadata_dict = await get_array_metadata(store_path, zarr_format=zarr_format)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/array.py", line 135, in get_array_metadata
    raise FileNotFoundError(store_path)
FileNotFoundError: file://root.zarr

[d-v-b]

Hi, Thanks for all the work towards v3. I wanted to check if this is a blocker / existing issue etc or if I should create one...? This is using main branch 680142f :

import zarr
from zarr.storage import LocalStore

path = "root.zarr"
fmt = 2
store = LocalStore(path, mode="w")
root = zarr.group(store=store, zarr_format=fmt)

root_group = zarr.open(f"{path}", zarr_format=fmt)

Gives me:

$ python test_open_v2_group.py 
Traceback (most recent call last):
  File "/Users/wmoore/Desktop/ZARR/ome-zarr-py/test_open_v2_group.py", line 48, in <module>
    root_group = zarr.open(f"{path}", zarr_format=fmt)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/_compat.py", line 43, in inner_f
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/api/synchronous.py", line 77, in open
    obj = sync(
          ^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/sync.py", line 141, in sync
    raise return_result
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/sync.py", line 100, in _runner
    return await coro
           ^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/api/asynchronous.py", line 308, in open
    return await open_array(store=store_path, zarr_format=zarr_format, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/api/asynchronous.py", line 1077, in open_array
    return await AsyncArray.open(store_path, zarr_format=zarr_format)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/array.py", line 680, in open
    metadata_dict = await get_array_metadata(store_path, zarr_format=zarr_format)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/array.py", line 135, in get_array_metadata
    raise FileNotFoundError(store_path)
FileNotFoundError: file://root.zarr

It looks like you are not passing the store instance to zarr.open. Does
root_group = zarr.open(store, path=path, zarr_format=fmt) work?

[will-moore]

If I pass in both the store and the path, it fails with FileNotFoundError: file://root.zarr/root.zarr, so it looks like I don't need both? It also seems I want to create a store with read mode. But this is still failing (but works with fmt = 3)...

read_store = LocalStore(path)
root_group = zarr.open(read_store, zarr_format=fmt)

FileNotFoundError: file://root.zarr

$ python test_open_v2_group.py 
Traceback (most recent call last):
  File "/Users/wmoore/Desktop/ZARR/ome-zarr-py/test_open_v2_group.py", line 49, in <module>
    root_group = zarr.open(read_store, zarr_format=fmt)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/_compat.py", line 43, in inner_f
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/api/synchronous.py", line 77, in open
    obj = sync(
          ^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/sync.py", line 141, in sync
    raise return_result
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/sync.py", line 100, in _runner
    return await coro
           ^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/api/asynchronous.py", line 308, in open
    return await open_array(store=store_path, zarr_format=zarr_format, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/api/asynchronous.py", line 1077, in open_array
    return await AsyncArray.open(store_path, zarr_format=zarr_format)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/array.py", line 680, in open
    metadata_dict = await get_array_metadata(store_path, zarr_format=zarr_format)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/array.py", line 135, in get_array_metadata
    raise FileNotFoundError(store_path)
FileNotFoundError: file://root.zarr

[d-v-b]

ugh, this flow should definitely be super easy, sorry that it's not working. If I have some time tonight I will dig into this.

[will-moore]

@d-v-b with this change, that test passes for me and fixes about 100 failing tests at ome/ome-zarr-py#404

diff --git a/src/zarr/api/asynchronous.py b/src/zarr/api/asynchronous.py
index 40f9b8d5..8dc69250 100644
--- a/src/zarr/api/asynchronous.py
+++ b/src/zarr/api/asynchronous.py
@@ -306,7 +306,7 @@ async def open(
 
     try:
         return await open_array(store=store_path, zarr_format=zarr_format, **kwargs)
-    except (KeyError, NodeTypeValidationError):
+    except (KeyError, NodeTypeValidationError, FileNotFoundError):
         # KeyError for a missing key
         # NodeTypeValidationError for failing to parse node metadata as an array when it's
         # actually a group