From d7c0556029ab514fadedda40086f4d1de64efe42 Mon Sep 17 00:00:00 2001 From: jmoore Date: Thu, 7 Apr 2022 16:00:26 +0200 Subject: [PATCH 001/102] First draft of bioformats2raw.layout --- latest/index.bs | 29 ++++++++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-) diff --git a/latest/index.bs b/latest/index.bs index 53723745..f52ca74a 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -100,6 +100,14 @@ The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL “RECOMMENDED”, “MAY”, and “OPTIONAL” are to be interpreted as described in [RFC 2119](https://tools.ietf.org/html/rfc2119). +

+Transitional metadata is added to the specification with the +intention of removing it in the future. Implementations may be expected (MUST) or +encouraged (SHOULD) support the reading of the data, but writing will usually +be optional (MAY). Examples of transitional metadata include custom additions by +implementations that are later submitted as a formal specification. (See [[#bf2raw]]) +

+ Some of the JSON examples in this document include commments. However, these are only for clarity purposes and comments MUST NOT be included in JSON objects. @@ -241,9 +249,28 @@ keys as specified below for discovering certain types of data, especially images If part of [[#multiscale-md]], the length of "axes" MUST be equal to the number of dimensions of the arrays that contain the image data. +"bioformats2raw.layout" (transitional) {#bf2raw} +------------------------------------------------ + +[=Transitional=] "bioformats2raw.layout" metadata identifies a group which implicitly describes a series of images. +The need for the collection stems from the common "multi-image file" scenario in microscopy. Parsers like Bio-Formats +define a strict, stable ordering of the images in a single container that can be used to refer to them by other tools. +In order to capture that information within an OME-NGFF dataset, `bioformats2raw` internally introduced a wrapping +layer. An upcoming NGFF specification will replace this layout with explicit metadata. + +Conforming groups: +- MUST have the value "3" for the "bioformats2raw.layout" key at the top of the hierarchy. +- MUST store separate "multiscales" images in consecutively numbered groups starting from 0 (i.e. "0/", "1/", "2/", "3/", ...). +- MAY have OME metadata representing the entire collection of images in a file named "OME/METADATA.ome.xml". + +Conforming readers: +- SHOULD parse all images. +- MAY choose to show all images within the collection or offer the user a choice of images, as with HCS plates. +- MAY ignore other groups or arrays under the root of the hierarchy. + "coordinateTransformations" metadata {#trafo-md} -------------------------------------- +------------------------------------------------ "coordinateTransformations" describe a series of transformations that map between two coordinate spaces (defined by "axes"). For example, to map a discrete data space of an array to the corresponding physical space. From 8ee02d1e60e216496ccd0ffb30f80892057269e7 Mon Sep 17 00:00:00 2001 From: jmoore Date: Fri, 8 Apr 2022 12:11:20 +0200 Subject: [PATCH 002/102] Add layout example --- .../examples/valid_strict/bioformats2raw_layout.json | 3 +++ latest/index.bs | 10 ++++++++-- 2 files changed, 11 insertions(+), 2 deletions(-) create mode 100644 latest/examples/valid_strict/bioformats2raw_layout.json diff --git a/latest/examples/valid_strict/bioformats2raw_layout.json b/latest/examples/valid_strict/bioformats2raw_layout.json new file mode 100644 index 00000000..c5eadb8a --- /dev/null +++ b/latest/examples/valid_strict/bioformats2raw_layout.json @@ -0,0 +1,3 @@ +{ + "bioformats2raw.layout" : 3 +} \ No newline at end of file diff --git a/latest/index.bs b/latest/index.bs index f52ca74a..d48c9ea1 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -258,8 +258,14 @@ define a strict, stable ordering of the images in a single container that can be In order to capture that information within an OME-NGFF dataset, `bioformats2raw` internally introduced a wrapping layer. An upcoming NGFF specification will replace this layout with explicit metadata. -Conforming groups: -- MUST have the value "3" for the "bioformats2raw.layout" key at the top of the hierarchy. +Conforming groups MUST have the value "3" for the "bioformats2raw.layout" key in their `.zattrs` metadata at the top of the hierarchy: + +
+path: examples/valid_strict/bioformats2raw_layout.json
+highlight: json
+
+ +Additionally: - MUST store separate "multiscales" images in consecutively numbered groups starting from 0 (i.e. "0/", "1/", "2/", "3/", ...). - MAY have OME metadata representing the entire collection of images in a file named "OME/METADATA.ome.xml". From ee1545552bac2758e55574f6c55cedbda2d15624 Mon Sep 17 00:00:00 2001 From: jmoore Date: Wed, 20 Apr 2022 08:24:41 +0200 Subject: [PATCH 003/102] Improve wording of "SHOULD parse multiple images", thanks to Ilan see: https://github.com/ome/ngff/issues/104#issuecomment-1092706651 --- latest/index.bs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/latest/index.bs b/latest/index.bs index d48c9ea1..d7031960 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -270,8 +270,8 @@ Additionally: - MAY have OME metadata representing the entire collection of images in a file named "OME/METADATA.ome.xml". Conforming readers: -- SHOULD parse all images. -- MAY choose to show all images within the collection or offer the user a choice of images, as with HCS plates. +- SHOULD detect the presence of more than one image and make users aware of the fact; +- MAY choose to show all images within the collection or offer the user a choice of images, as with HCS plates; - MAY ignore other groups or arrays under the root of the hierarchy. From c8e98e4c390d1a2d12f33377e2a21e37a4567247 Mon Sep 17 00:00:00 2001 From: jmoore Date: Wed, 20 Apr 2022 08:26:06 +0200 Subject: [PATCH 004/102] Clarify contents of METADATA.ome.xml, thanks to Melissa --- latest/index.bs | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/latest/index.bs b/latest/index.bs index d7031960..b38b3c86 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -266,8 +266,9 @@ highlight: json Additionally: -- MUST store separate "multiscales" images in consecutively numbered groups starting from 0 (i.e. "0/", "1/", "2/", "3/", ...). -- MAY have OME metadata representing the entire collection of images in a file named "OME/METADATA.ome.xml". +- MUST store separate "multiscales" images in consecutively numbered groups starting from 0 (i.e. "0/", "1/", "2/", "3/", ...); +- MAY have OME metadata representing the entire collection of images in a file named "OME/METADATA.ome.xml"; +- if so, MUST represent every "multiscales" group as exactly one OME-XML "Image" in the same order as the group numbers. Conforming readers: - SHOULD detect the presence of more than one image and make users aware of the fact; From 6403d883aa1ca559cc0b37368b5f78a84245e22f Mon Sep 17 00:00:00 2001 From: jmoore Date: Thu, 21 Apr 2022 12:12:11 +0200 Subject: [PATCH 005/102] Add schema & test for bf2raw metadata --- .../bioformats2raw_layout.json | 0 latest/index.bs | 2 +- latest/tests/test_validation.py | 15 +++++++++++++++ 3 files changed, 16 insertions(+), 1 deletion(-) rename latest/examples/{valid_strict => bf2raw}/bioformats2raw_layout.json (100%) diff --git a/latest/examples/valid_strict/bioformats2raw_layout.json b/latest/examples/bf2raw/bioformats2raw_layout.json similarity index 100% rename from latest/examples/valid_strict/bioformats2raw_layout.json rename to latest/examples/bf2raw/bioformats2raw_layout.json diff --git a/latest/index.bs b/latest/index.bs index b38b3c86..b9fd8765 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -261,7 +261,7 @@ layer. An upcoming NGFF specification will replace this layout with explicit met Conforming groups MUST have the value "3" for the "bioformats2raw.layout" key in their `.zattrs` metadata at the top of the hierarchy:
-path: examples/valid_strict/bioformats2raw_layout.json
+path: examples/bf2raw/bioformats2raw_layout.json
 highlight: json
 
diff --git a/latest/tests/test_validation.py b/latest/tests/test_validation.py index 3a6a5761..51ff7b05 100644 --- a/latest/tests/test_validation.py +++ b/latest/tests/test_validation.py @@ -7,11 +7,15 @@ from jsonschema.exceptions import ValidationError +with open('schemas/bf2raw.schema') as f: + bf2raw_schema = json.load(f) with open('schemas/image.schema') as f: image_schema = json.load(f) with open('schemas/strict_image.schema') as f: strict_image_schema = json.load(f) + schema_store = { + bf2raw_schema['$id']: bf2raw_schema, image_schema['$id']: image_schema, strict_image_schema['$id']: strict_image_schema, } @@ -19,7 +23,9 @@ resolver = RefResolver.from_schema(image_schema, store=schema_store) validator = Draft202012Validator(image_schema, resolver=resolver) strict_validator = Draft202012Validator(strict_image_schema, resolver=resolver) +bf2raw_validator = Draft202012Validator(bf2raw_schema, resolver=resolver) +bf2raw_files = list(glob.glob("examples/bf2raw/*.json")) valid_strict_files = list(glob.glob("examples/valid_strict/*.json")) valid_files = list(glob.glob("examples/valid/*.json")) invalid_files = list(glob.glob("examples/invalid/*.json")) @@ -31,6 +37,15 @@ def ids(files): return [str(x).split("/")[-1][0:-5] for x in files] +@pytest.mark.parametrize( + "testfile", bf2raw_files, ids=ids(bf2raw_files)) +def test_bf2raw(testfile): + with open(testfile) as f: + data = ''.join(line for line in f if not line.lstrip().startswith('//')) + jsondata = json.loads(data) + bf2raw_validator.validate(jsondata) + + @pytest.mark.parametrize( "testfile", valid_strict_files, ids=ids(valid_strict_files)) def test_valid_strict(testfile): From 890e8a281daac0bf00ed3e32a864f24d1fc72168 Mon Sep 17 00:00:00 2001 From: jmoore Date: Thu, 21 Apr 2022 15:45:31 +0200 Subject: [PATCH 006/102] Add missing schema file --- latest/schemas/bf2raw.schema | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 latest/schemas/bf2raw.schema diff --git a/latest/schemas/bf2raw.schema b/latest/schemas/bf2raw.schema new file mode 100644 index 00000000..834aee24 --- /dev/null +++ b/latest/schemas/bf2raw.schema @@ -0,0 +1,14 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://ngff.openmicroscopy.org/latest/schemas/bf2raw.schema", + "title": "NGFF container produced by bioformats2raw", + "description": "JSON from OME-NGFF .zattrs", + "type": "object", + "properties": { + "bioformats2raw.layout": { + "description": "The top-level identifier metadata added by bioformats2raw", + "type": "number", + "enum": [3] + } + } +} From 2b67a36515b8eaeab643f217167e6cb8ad4d9bcc Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Wed, 11 May 2022 12:41:08 +0200 Subject: [PATCH 007/102] Add applicable versions statement --- latest/index.bs | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/latest/index.bs b/latest/index.bs index ca2104f7..41128c55 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -256,17 +256,18 @@ If part of [[#multiscale-md]], the length of "axes" MUST be equal to the number [=Transitional=] "bioformats2raw.layout" metadata identifies a group which implicitly describes a series of images. The need for the collection stems from the common "multi-image file" scenario in microscopy. Parsers like Bio-Formats define a strict, stable ordering of the images in a single container that can be used to refer to them by other tools. -In order to capture that information within an OME-NGFF dataset, `bioformats2raw` internally introduced a wrapping -layer. An upcoming NGFF specification will replace this layout with explicit metadata. -Conforming groups MUST have the value "3" for the "bioformats2raw.layout" key in their `.zattrs` metadata at the top of the hierarchy: +In order to capture that information within an OME-NGFF dataset, `bioformats2raw` internally introduced a wrapping layer. +The bioformats2raw layout can be found in NGFF versions v0.2, v0.3, and v0.4. An upcoming NGFF specification will replace +this layout with explicit metadata.
 path: examples/bf2raw/bioformats2raw_layout.json
 highlight: json
 
-Additionally: +Conforming groups: +- MUST have the value "3" for the "bioformats2raw.layout" key in their `.zattrs` metadata at the top of the hierarchy: - MUST store separate "multiscales" images in consecutively numbered groups starting from 0 (i.e. "0/", "1/", "2/", "3/", ...); - MAY have OME metadata representing the entire collection of images in a file named "OME/METADATA.ome.xml"; - if so, MUST represent every "multiscales" group as exactly one OME-XML "Image" in the same order as the group numbers. From 31d34c3faf9a520d7fa46e32a6973906a42ac3f9 Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Mon, 30 May 2022 12:32:05 +0200 Subject: [PATCH 008/102] Update text with suggestions --- latest/index.bs | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/latest/index.bs b/latest/index.bs index 41128c55..d21d9586 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -267,11 +267,15 @@ highlight: json Conforming groups: -- MUST have the value "3" for the "bioformats2raw.layout" key in their `.zattrs` metadata at the top of the hierarchy: +- MUST have the value "3" for the "bioformats2raw.layout" key in their `.zattrs` metadata at the top of the hierarchy; - MUST store separate "multiscales" images in consecutively numbered groups starting from 0 (i.e. "0/", "1/", "2/", "3/", ...); -- MAY have OME metadata representing the entire collection of images in a file named "OME/METADATA.ome.xml"; +- SHOULD have OME metadata representing the entire collection of images in a file named "OME/METADATA.ome.xml" which + MUST adhere to the specification available at https://docs.openmicroscopy.org/ome-model/6.2.2/specifications/minimum.html; - if so, MUST represent every "multiscales" group as exactly one OME-XML "Image" in the same order as the group numbers. +Please note that the "plate" key takes precedence and parsing of such datasets should follow [[#plate-md]]. It is not +possible to mix collections of images with plates at present. + Conforming readers: - SHOULD detect the presence of more than one image and make users aware of the fact; - MAY choose to show all images within the collection or offer the user a choice of images, as with HCS plates; From d44a0668bd530ffedf612f8c8a90d7d9d143a4a2 Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Mon, 30 May 2022 12:35:03 +0200 Subject: [PATCH 009/102] Add bf2raw examples config --- latest/examples/bf2raw/.config.json | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 latest/examples/bf2raw/.config.json diff --git a/latest/examples/bf2raw/.config.json b/latest/examples/bf2raw/.config.json new file mode 100644 index 00000000..2525328c --- /dev/null +++ b/latest/examples/bf2raw/.config.json @@ -0,0 +1,3 @@ +{ + "schema": "schemas/bf2raw.schema" +} From 7574d228377b2e554421704a8d8a4aa56a7ac17d Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Mon, 30 May 2022 12:46:04 +0200 Subject: [PATCH 010/102] Add test to find missing configs --- latest/tests/test_validation.py | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/latest/tests/test_validation.py b/latest/tests/test_validation.py index 9e441fad..8c13b113 100644 --- a/latest/tests/test_validation.py +++ b/latest/tests/test_validation.py @@ -89,3 +89,17 @@ def test_run(suite): resolver = RefResolver.from_schema(suite.schema, store=schema_store) validator = Validator(suite.schema, resolver=resolver) suite.validate(validator) + + +def test_example_configs(): + """ + Test that all example folders have a config file + """ + missing = [] + for subdir in os.walk("examples"): + has_examples = glob.glob(f"{subdir[0]}/*.json") + has_config = glob.glob(f"{subdir[0]}/.config.json") + if has_examples and not has_config: + missing.append(subdir[0]) + if missing: + raise Exception(f"Directories missing configs: {missing}") From ff5d7ec16f84933678a31829dfd26930694e4c75 Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Mon, 30 May 2022 12:57:09 +0200 Subject: [PATCH 011/102] Split subitems to pass linting --- latest/index.bs | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/latest/index.bs b/latest/index.bs index 4cbc2662..d5dc002c 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -269,8 +269,9 @@ highlight: json Conforming groups: - MUST have the value "3" for the "bioformats2raw.layout" key in their `.zattrs` metadata at the top of the hierarchy; - MUST store separate "multiscales" images in consecutively numbered groups starting from 0 (i.e. "0/", "1/", "2/", "3/", ...); -- SHOULD have OME metadata representing the entire collection of images in a file named "OME/METADATA.ome.xml" which - MUST adhere to the specification available at https://docs.openmicroscopy.org/ome-model/6.2.2/specifications/minimum.html; +- SHOULD have OME metadata representing the entire collection of images in a file named "OME/METADATA.ome.xml" which: + - MUST adhere to the OME-XML specification but + - MAY make use of the minimum specification available at https://docs.openmicroscopy.org/ome-model/6.2.2/specifications/minimum.html; - if so, MUST represent every "multiscales" group as exactly one OME-XML "Image" in the same order as the group numbers. Please note that the "plate" key takes precedence and parsing of such datasets should follow [[#plate-md]]. It is not From e2119e5ccd9c97da8717c53b889f377c39d121a9 Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Mon, 30 May 2022 12:57:16 +0200 Subject: [PATCH 012/102] Add graphical layout representation --- latest/index.bs | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/latest/index.bs b/latest/index.bs index d5dc002c..d7500d49 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -274,6 +274,17 @@ Conforming groups: - MAY make use of the minimum specification available at https://docs.openmicroscopy.org/ome-model/6.2.2/specifications/minimum.html; - if so, MUST represent every "multiscales" group as exactly one OME-XML "Image" in the same order as the group numbers. +
+series.ome.zarr           # One converted fileset from bioformats2raw
+    ├── .zgroup
+    ├── .zattrs               # Contains "bioformats2raw.layout" metadata
+    ├── OME                   # Special group for containing OME metadata
+    │   └── METADATA.ome.xml  # OME-XML file stored within the Zarr fileset
+    ├── 0                     # First image in the collection
+    ├── 1                     # Second image in the collection
+    └── ...
+
+ Please note that the "plate" key takes precedence and parsing of such datasets should follow [[#plate-md]]. It is not possible to mix collections of images with plates at present. From 4fcc045e12caa18f1cd51aa27b4aa6da8c168550 Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Mon, 30 May 2022 12:59:12 +0200 Subject: [PATCH 013/102] Fix missing whitespace --- latest/index.bs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/latest/index.bs b/latest/index.bs index d7500d49..5a2727e9 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -275,7 +275,7 @@ Conforming groups: - if so, MUST represent every "multiscales" group as exactly one OME-XML "Image" in the same order as the group numbers.
-series.ome.zarr           # One converted fileset from bioformats2raw
+series.ome.zarr               # One converted fileset from bioformats2raw
     ├── .zgroup
     ├── .zattrs               # Contains "bioformats2raw.layout" metadata
     ├── OME                   # Special group for containing OME metadata

From 57acc23fc67f367ea6d0d3bb65a3cb7769f400ba Mon Sep 17 00:00:00 2001
From: Melissa Linkert 
Date: Tue, 5 Jul 2022 15:54:25 -0500
Subject: [PATCH 014/102] Add "series" attribute in "OME" group under
 bioformats2raw.layout

Documents the current state of https://github.com/glencoesoftware/bioformats2raw/pull/157
---
 latest/examples/bf2raw/ome.json |  3 +++
 latest/index.bs                 | 13 ++++++++++++-
 2 files changed, 15 insertions(+), 1 deletion(-)
 create mode 100644 latest/examples/bf2raw/ome.json

diff --git a/latest/examples/bf2raw/ome.json b/latest/examples/bf2raw/ome.json
new file mode 100644
index 00000000..be7e9ed6
--- /dev/null
+++ b/latest/examples/bf2raw/ome.json
@@ -0,0 +1,3 @@
+{
+  "series" : [ "0", "1" ]
+}
diff --git a/latest/index.bs b/latest/index.bs
index 5a2727e9..ba0fc50d 100644
--- a/latest/index.bs
+++ b/latest/index.bs
@@ -272,13 +272,23 @@ Conforming groups:
 - SHOULD have OME metadata representing the entire collection of images in a file named "OME/METADATA.ome.xml" which:
   - MUST adhere to the OME-XML specification but
   - MAY make use of the minimum specification available at https://docs.openmicroscopy.org/ome-model/6.2.2/specifications/minimum.html;
-- if so, MUST represent every "multiscales" group as exactly one OME-XML "Image" in the same order as the group numbers.
+- if so, MUST represent every "multiscales" group as exactly one OME-XML "Image" in the same order as the group numbers;
+- if "OME/METADATA.ome.xml" is present, "OME" MUST be a Zarr group which:
+  - MUST contain a "series" attribute. "series" MUST be a list of string objects, each of which is a path to an image group.
+    The order of the paths MUST match the order of both the "multiscales" metadata and the "Image" elements in "OME/METADATA.ome.xml".
+
+
+path: examples/bf2raw/ome.json
+highlight: json
+
 series.ome.zarr               # One converted fileset from bioformats2raw
     ├── .zgroup
     ├── .zattrs               # Contains "bioformats2raw.layout" metadata
     ├── OME                   # Special group for containing OME metadata
+    │   ├── .zgroup
+    │   ├── .zattrs           # Contains "series" metadata
     │   └── METADATA.ome.xml  # OME-XML file stored within the Zarr fileset
     ├── 0                     # First image in the collection
     ├── 1                     # Second image in the collection
@@ -290,6 +300,7 @@ possible to mix collections of images with plates at present.
 
 Conforming readers:
 - SHOULD detect the presence of more than one image and make users aware of the fact;
+- MAY use the "series" attribute in the "OME" group to determine a list of valid groups;
 - MAY choose to show all images within the collection or offer the user a choice of images, as with HCS plates;
 - MAY ignore other groups or arrays under the root of the hierarchy.
 

From 5eca16930bdb8f8fff34ffd52df796f4906db5bc Mon Sep 17 00:00:00 2001
From: Josh Moore 
Date: Mon, 18 Jul 2022 13:17:16 -0500
Subject: [PATCH 015/102] Update the MUST/SHOULD semantics

---
 latest/index.bs | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/latest/index.bs b/latest/index.bs
index ba0fc50d..d5d6b2b9 100644
--- a/latest/index.bs
+++ b/latest/index.bs
@@ -268,14 +268,18 @@ highlight: json
 
 Conforming groups:
 - MUST have the value "3" for the "bioformats2raw.layout" key in their `.zattrs` metadata at the top of the hierarchy;
-- MUST store separate "multiscales" images in consecutively numbered groups starting from 0 (i.e. "0/", "1/", "2/", "3/", ...);
 - SHOULD have OME metadata representing the entire collection of images in a file named "OME/METADATA.ome.xml" which:
   - MUST adhere to the OME-XML specification but
+  - MUST use `` elements and
   - MAY make use of the minimum specification available at https://docs.openmicroscopy.org/ome-model/6.2.2/specifications/minimum.html;
-- if so, MUST represent every "multiscales" group as exactly one OME-XML "Image" in the same order as the group numbers;
-- if "OME/METADATA.ome.xml" is present, "OME" MUST be a Zarr group which:
-  - MUST contain a "series" attribute. "series" MUST be a list of string objects, each of which is a path to an image group.
+
+Additionally:
+- If "OME/METADATA.ome.xml" is present, "OME" MUST be a Zarr group which:
+  - MAY contain a "series" attribute. "series" MUST be a list of string objects, each of which is a path to an image group.
     The order of the paths MUST match the order of both the "multiscales" metadata and the "Image" elements in "OME/METADATA.ome.xml".
+- If if "OME/METADATA.ome.xml" or the "series" attribute do not exist:
+  - separate "multiscales" images MUST be stored in consecutively numbered groups starting from 0 (i.e. "0/", "1/", "2/", "3/", ...).
+- Every "multiscales" group MUST represent exactly one OME-XML "Image" in the same order as either the series index or the group numbers.
 
 
 path: examples/bf2raw/ome.json

From f3e5064a51d9b63649f8133fae0203b694f071a7 Mon Sep 17 00:00:00 2001
From: Sebastien Besson 
Date: Mon, 29 Aug 2022 09:28:09 +0100
Subject: [PATCH 016/102] Add section about JSON schemas

Described schemas location, testing & deployment as well as the
management and the testing of official JSON examples
---
 README.md | 22 +++++++++++++++++++++-
 1 file changed, 21 insertions(+), 1 deletion(-)

diff --git a/README.md b/README.md
index 1393bf16..26c7f21e 100644
--- a/README.md
+++ b/README.md
@@ -29,6 +29,26 @@ Commits on GitHub can be viewed using web services from the W3C:
 * Update changelog at the bottom of `latest/index.bs`
 * Find references to previous version and _in most cases_, bump to the current version.
 
+## JSON schemas
+
+For each top-level metadata key of the OME-NGFF specification, JSON schemas are maintained
+for each version of the specification and stored under `$VERSION/schemas/` or `latest/schemas/`.
+Tests validating these schemas must be implementing to follow principles of the
+[JSON schema test suite](https://github.com/json-schema-org/JSON-Schema-Test-Suite)
+and stored under `$VERSION/tests/` or `latest/tests/` to allow their execution on each CI build.
+
+All official example snippets must also be extracted and managed as separate JSON files under
+`$VERSION/examples/` or `latest/examples/`, validated by the appropriate schema by adding a
+`.config.json` file specifying the JSON schema to use and included in the
+specification document using the
+[include-code](https://tabatkins.github.io/bikeshed/#including-code) directive.
+
+The official OME-NGFF JSON schemas are published under
+https://ngff.openmicroscopy.org//schemas/.schema using the
+[Spec prod](https://github.com/ome/spec-prod) GitHub action. When a new JSON schema is introduced,
+this action needs to be reviewed updated to update the deployment and allow the publication
+of the schema
+
 # Release process
 
 * copy `latest/index.bs` to `$VERSION/index.bs`
@@ -43,7 +63,7 @@ Commits on GitHub can be viewed using web services from the W3C:
   * Version in the citation block including release date
 * Update https://github.com/ome/spec-prod for the new version
 
-## Citing
+# Citing
 
 Please see https://ngff.openmicroscopy.org/latest#citing for the latest
 citation.

From 439c86c721c79cd18abf0e5da5837c8ad5512de9 Mon Sep 17 00:00:00 2001
From: Sebastien Besson 
Date: Mon, 29 Aug 2022 09:58:10 +0100
Subject: [PATCH 017/102] Address Will's comments

---
 README.md | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/README.md b/README.md
index 26c7f21e..6b123172 100644
--- a/README.md
+++ b/README.md
@@ -33,7 +33,7 @@ Commits on GitHub can be viewed using web services from the W3C:
 
 For each top-level metadata key of the OME-NGFF specification, JSON schemas are maintained
 for each version of the specification and stored under `$VERSION/schemas/` or `latest/schemas/`.
-Tests validating these schemas must be implementing to follow principles of the
+Tests validating these schemas must be implemented to follow principles of the
 [JSON schema test suite](https://github.com/json-schema-org/JSON-Schema-Test-Suite)
 and stored under `$VERSION/tests/` or `latest/tests/` to allow their execution on each CI build.
 
@@ -46,8 +46,8 @@ specification document using the
 The official OME-NGFF JSON schemas are published under
 https://ngff.openmicroscopy.org//schemas/.schema using the
 [Spec prod](https://github.com/ome/spec-prod) GitHub action. When a new JSON schema is introduced,
-this action needs to be reviewed updated to update the deployment and allow the publication
-of the schema
+this action needs to be reviewed to update the deployment script and allow the publication
+of the schema.
 
 # Release process
 

From cf4e04c16cc7dda695ce511cacf4f895e20e4d50 Mon Sep 17 00:00:00 2001
From: Josh Moore 
Date: Wed, 14 Sep 2022 11:42:03 +0200
Subject: [PATCH 018/102] Fix doubly indented bullets

---
 latest/index.bs | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)

diff --git a/latest/index.bs b/latest/index.bs
index d5d6b2b9..00eb87ba 100644
--- a/latest/index.bs
+++ b/latest/index.bs
@@ -267,18 +267,21 @@ highlight: json
 
Conforming groups: + - MUST have the value "3" for the "bioformats2raw.layout" key in their `.zattrs` metadata at the top of the hierarchy; - SHOULD have OME metadata representing the entire collection of images in a file named "OME/METADATA.ome.xml" which: - - MUST adhere to the OME-XML specification but - - MUST use `` elements and - - MAY make use of the minimum specification available at https://docs.openmicroscopy.org/ome-model/6.2.2/specifications/minimum.html; + - MUST adhere to the OME-XML specification but + - MUST use `` elements and + - MAY make use of the minimum specification available at https://docs.openmicroscopy.org/ome-model/6.2.2/specifications/minimum.html; Additionally: + - If "OME/METADATA.ome.xml" is present, "OME" MUST be a Zarr group which: - - MAY contain a "series" attribute. "series" MUST be a list of string objects, each of which is a path to an image group. - The order of the paths MUST match the order of both the "multiscales" metadata and the "Image" elements in "OME/METADATA.ome.xml". + - MAY contain a "series" attribute. + - "series" MUST be a list of string objects, each of which is a path to an image group. + - The order of the paths MUST match the order of both the "multiscales" metadata and the "Image" elements in "OME/METADATA.ome.xml". - If if "OME/METADATA.ome.xml" or the "series" attribute do not exist: - - separate "multiscales" images MUST be stored in consecutively numbered groups starting from 0 (i.e. "0/", "1/", "2/", "3/", ...). + - separate "multiscales" images MUST be stored in consecutively numbered groups starting from 0 (i.e. "0/", "1/", "2/", "3/", ...). - Every "multiscales" group MUST represent exactly one OME-XML "Image" in the same order as either the series index or the group numbers.

From 89c322d5fef37a84d557d6539e69eb7a0988836b Mon Sep 17 00:00:00 2001
From: Josh Moore 
Date: Wed, 14 Sep 2022 11:46:40 +0200
Subject: [PATCH 019/102] Make minimum spec a link

---
 latest/index.bs | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/latest/index.bs b/latest/index.bs
index 00eb87ba..0c7949be 100644
--- a/latest/index.bs
+++ b/latest/index.bs
@@ -271,8 +271,8 @@ Conforming groups:
 - MUST have the value "3" for the "bioformats2raw.layout" key in their `.zattrs` metadata at the top of the hierarchy;
 - SHOULD have OME metadata representing the entire collection of images in a file named "OME/METADATA.ome.xml" which:
     - MUST adhere to the OME-XML specification but
-    - MUST use `` elements and
-    - MAY make use of the minimum specification available at https://docs.openmicroscopy.org/ome-model/6.2.2/specifications/minimum.html;
+    - MUST use `` elements as opposed to ``, `` or ``;
+    - MAY make use of the [minimum specification](https://docs.openmicroscopy.org/ome-model/6.2.2/specifications/minimum.html).
 
 Additionally:
 

From 369143776f8dbd6451f25b96d17b8ee2de2c0dc0 Mon Sep 17 00:00:00 2001
From: Josh Moore 
Date: Wed, 14 Sep 2022 11:59:23 +0200
Subject: [PATCH 020/102] Add no-toc sections

---
 latest/index.bs | 43 +++++++++++++++++++++++++------------------
 1 file changed, 25 insertions(+), 18 deletions(-)

diff --git a/latest/index.bs b/latest/index.bs
index 0c7949be..51d4b529 100644
--- a/latest/index.bs
+++ b/latest/index.bs
@@ -261,34 +261,21 @@ In order to capture that information within an OME-NGFF dataset, `bioformats2raw
 The bioformats2raw layout can be found in NGFF versions v0.2, v0.3, and v0.4. An upcoming NGFF specification will replace
 this layout with explicit metadata.
 
+
+

Attributes

+
 path: examples/bf2raw/bioformats2raw_layout.json
 highlight: json
 
-Conforming groups: - -- MUST have the value "3" for the "bioformats2raw.layout" key in their `.zattrs` metadata at the top of the hierarchy; -- SHOULD have OME metadata representing the entire collection of images in a file named "OME/METADATA.ome.xml" which: - - MUST adhere to the OME-XML specification but - - MUST use `` elements as opposed to ``, `` or ``; - - MAY make use of the [minimum specification](https://docs.openmicroscopy.org/ome-model/6.2.2/specifications/minimum.html). - -Additionally: - -- If "OME/METADATA.ome.xml" is present, "OME" MUST be a Zarr group which: - - MAY contain a "series" attribute. - - "series" MUST be a list of string objects, each of which is a path to an image group. - - The order of the paths MUST match the order of both the "multiscales" metadata and the "Image" elements in "OME/METADATA.ome.xml". -- If if "OME/METADATA.ome.xml" or the "series" attribute do not exist: - - separate "multiscales" images MUST be stored in consecutively numbered groups starting from 0 (i.e. "0/", "1/", "2/", "3/", ...). -- Every "multiscales" group MUST represent exactly one OME-XML "Image" in the same order as either the series index or the group numbers. -
 path: examples/bf2raw/ome.json
 highlight: json
 
+

Layout

+
 series.ome.zarr               # One converted fileset from bioformats2raw
     ├── .zgroup
@@ -302,6 +289,26 @@ series.ome.zarr               # One converted fileset from bioformats2raw
     └── ...
 
+

Details

+ +Conforming groups: + +- MUST have the value "3" for the "bioformats2raw.layout" key in their `.zattrs` metadata at the top of the hierarchy; +- SHOULD have OME metadata representing the entire collection of images in a file named "OME/METADATA.ome.xml" which: + - MUST adhere to the OME-XML specification but + - MUST use `` elements as opposed to ``, `` or ``; + - MAY make use of the [minimum specification](https://docs.openmicroscopy.org/ome-model/6.2.2/specifications/minimum.html). + +Additionally: + +- If "OME/METADATA.ome.xml" is present, "OME" MUST be a Zarr group which: + - MAY contain a "series" attribute. + - "series" MUST be a list of string objects, each of which is a path to an image group. + - The order of the paths MUST match the order of both the "multiscales" metadata and the "Image" elements in "OME/METADATA.ome.xml". +- If if "OME/METADATA.ome.xml" or the "series" attribute do not exist: + - separate "multiscales" images MUST be stored in consecutively numbered groups starting from 0 (i.e. "0/", "1/", "2/", "3/", ...). +- Every "multiscales" group MUST represent exactly one OME-XML "Image" in the same order as either the series index or the group numbers. + Please note that the "plate" key takes precedence and parsing of such datasets should follow [[#plate-md]]. It is not possible to mix collections of images with plates at present. From 499caeeebc3f645dccb30851a160722ac8c5784e Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Wed, 14 Sep 2022 12:24:47 +0200 Subject: [PATCH 021/102] Re-arrange and add more text --- latest/index.bs | 30 +++++++++++++++++------------- 1 file changed, 17 insertions(+), 13 deletions(-) diff --git a/latest/index.bs b/latest/index.bs index 51d4b529..732150cf 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -261,21 +261,10 @@ In order to capture that information within an OME-NGFF dataset, `bioformats2raw The bioformats2raw layout can be found in NGFF versions v0.2, v0.3, and v0.4. An upcoming NGFF specification will replace this layout with explicit metadata. - -

Attributes

- -
-path: examples/bf2raw/bioformats2raw_layout.json
-highlight: json
-
- -
-path: examples/bf2raw/ome.json
-highlight: json
-
-

Layout

+Typical Zarr layout produced by running `bioformats2raw` on a fileset that contains more than image (series > 1): +
 series.ome.zarr               # One converted fileset from bioformats2raw
     ├── .zgroup
@@ -289,6 +278,21 @@ series.ome.zarr               # One converted fileset from bioformats2raw
     └── ...
 
+

Attributes

+ +The top-level `.zattrs` file must contain the `bioformats2raw.layout` key: +
+path: examples/bf2raw/bioformats2raw_layout.json
+highlight: json
+
+ +The `.zattrs` file within the OME group may contain the "series" key: + +
+path: examples/bf2raw/ome.json
+highlight: json
+
+

Details

Conforming groups: From c7582e558ef775ad22cde4864fbe2f50fe151537 Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Thu, 15 Sep 2022 11:24:54 +0200 Subject: [PATCH 022/102] Make changes based on feedback from Will --- latest/index.bs | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/latest/index.bs b/latest/index.bs index 732150cf..335dc6e3 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -308,8 +308,8 @@ Additionally: - If "OME/METADATA.ome.xml" is present, "OME" MUST be a Zarr group which: - MAY contain a "series" attribute. - "series" MUST be a list of string objects, each of which is a path to an image group. - - The order of the paths MUST match the order of both the "multiscales" metadata and the "Image" elements in "OME/METADATA.ome.xml". -- If if "OME/METADATA.ome.xml" or the "series" attribute do not exist: + - The order of the paths MUST match the order of the "Image" elements in "OME/METADATA.ome.xml". +- If "OME/METADATA.ome.xml" or the "series" attribute do not exist: - separate "multiscales" images MUST be stored in consecutively numbered groups starting from 0 (i.e. "0/", "1/", "2/", "3/", ...). - Every "multiscales" group MUST represent exactly one OME-XML "Image" in the same order as either the series index or the group numbers. @@ -317,8 +317,8 @@ Please note that the "plate" key takes precedence and parsing of such datasets s possible to mix collections of images with plates at present. Conforming readers: -- SHOULD detect the presence of more than one image and make users aware of the fact; -- MAY use the "series" attribute in the "OME" group to determine a list of valid groups; +- SHOULD make users aware of the presence of more than one image (i.e. SHOULD NOT default to only opening the first image); +- MAY use the "series" attribute in the "OME" group to determine a list of valid groups to display; - MAY choose to show all images within the collection or offer the user a choice of images, as with HCS plates; - MAY ignore other groups or arrays under the root of the hierarchy. From 399b70b4e0709e16be6fc79e365b0cbb74d9d670 Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Thu, 15 Sep 2022 11:33:35 +0200 Subject: [PATCH 023/102] Add bf2raw plate example --- ...{bioformats2raw_layout.json => image.json} | 0 latest/examples/bf2raw/plate.json | 21 +++++++++++++++++++ latest/index.bs | 14 +++++++++---- 3 files changed, 31 insertions(+), 4 deletions(-) rename latest/examples/bf2raw/{bioformats2raw_layout.json => image.json} (100%) create mode 100644 latest/examples/bf2raw/plate.json diff --git a/latest/examples/bf2raw/bioformats2raw_layout.json b/latest/examples/bf2raw/image.json similarity index 100% rename from latest/examples/bf2raw/bioformats2raw_layout.json rename to latest/examples/bf2raw/image.json diff --git a/latest/examples/bf2raw/plate.json b/latest/examples/bf2raw/plate.json new file mode 100644 index 00000000..af80bbad --- /dev/null +++ b/latest/examples/bf2raw/plate.json @@ -0,0 +1,21 @@ +{ + "bioformats2raw.layout" : 3, + "plate" : { + "columns" : [ { + "name" : "1" + } ], + "name" : "Plate Name 0", + "wells" : [ { + "path" : "A/1", + "rowIndex" : 0, + "columnIndex" : 0 + } ], + "field_count" : 1, + "rows" : [ { + "name" : "A" + } ], + "acquisitions" : [ { + "id" : 0 + } ], + "version" : "0.4" + } \ No newline at end of file diff --git a/latest/index.bs b/latest/index.bs index 335dc6e3..e3940d3a 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -282,7 +282,16 @@ series.ome.zarr # One converted fileset from bioformats2raw The top-level `.zattrs` file must contain the `bioformats2raw.layout` key:
-path: examples/bf2raw/bioformats2raw_layout.json
+path: examples/bf2raw/image.json
+highlight: json
+
+ +If the top-level group represents a plate, the `bioformats2raw.layout` metadata will be present but +the "plate" key key takes precedence and parsing of such datasets should follow [[#plate-md]]. It is not +possible to mix collections of images with plates at present. + +
+path: examples/bf2raw/plate.json
 highlight: json
 
@@ -313,9 +322,6 @@ Additionally: - separate "multiscales" images MUST be stored in consecutively numbered groups starting from 0 (i.e. "0/", "1/", "2/", "3/", ...). - Every "multiscales" group MUST represent exactly one OME-XML "Image" in the same order as either the series index or the group numbers. -Please note that the "plate" key takes precedence and parsing of such datasets should follow [[#plate-md]]. It is not -possible to mix collections of images with plates at present. - Conforming readers: - SHOULD make users aware of the presence of more than one image (i.e. SHOULD NOT default to only opening the first image); - MAY use the "series" attribute in the "OME" group to determine a list of valid groups to display; From 740a11b814f55574772995a85373b8ab9802c62d Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Thu, 15 Sep 2022 14:18:48 +0200 Subject: [PATCH 024/102] Add schema for ome series --- latest/examples/bf2raw/plate.json | 3 ++- latest/examples/ome/.config.json | 3 +++ .../{bf2raw/ome.json => ome/series-2.json} | 0 latest/index.bs | 2 +- latest/schemas/ome.schema | 17 +++++++++++++++++ 5 files changed, 23 insertions(+), 2 deletions(-) create mode 100644 latest/examples/ome/.config.json rename latest/examples/{bf2raw/ome.json => ome/series-2.json} (100%) create mode 100644 latest/schemas/ome.schema diff --git a/latest/examples/bf2raw/plate.json b/latest/examples/bf2raw/plate.json index af80bbad..bd98a16e 100644 --- a/latest/examples/bf2raw/plate.json +++ b/latest/examples/bf2raw/plate.json @@ -18,4 +18,5 @@ "id" : 0 } ], "version" : "0.4" - } \ No newline at end of file + } +} diff --git a/latest/examples/ome/.config.json b/latest/examples/ome/.config.json new file mode 100644 index 00000000..8a611ccf --- /dev/null +++ b/latest/examples/ome/.config.json @@ -0,0 +1,3 @@ +{ + "schema": "schemas/ome.schema" +} diff --git a/latest/examples/bf2raw/ome.json b/latest/examples/ome/series-2.json similarity index 100% rename from latest/examples/bf2raw/ome.json rename to latest/examples/ome/series-2.json diff --git a/latest/index.bs b/latest/index.bs index e3940d3a..c1e0275b 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -298,7 +298,7 @@ highlight: json The `.zattrs` file within the OME group may contain the "series" key:
-path: examples/bf2raw/ome.json
+path: examples/ome/series-2.json
 highlight: json
 
diff --git a/latest/schemas/ome.schema b/latest/schemas/ome.schema new file mode 100644 index 00000000..bd600a2a --- /dev/null +++ b/latest/schemas/ome.schema @@ -0,0 +1,17 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://ngff.openmicroscopy.org/latest/schemas/ome.schema", + "title": "NGFF group produced by bioformats2raw to contain OME metadata", + "description": "JSON from OME-NGFF OME/.zattrs linked to an OME-XML file", + "type": "object", + "properties": { + "series": { + "description": "An array of the same length and the same order as the images defined in the OME-XML", + "type": "array", + "items": { + "type": "string" + }, + "minContains": 1 + } + } +} From 1f91482de0b9571be641292e73931e55ed0b1f19 Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Mon, 19 Sep 2022 19:03:32 +0200 Subject: [PATCH 025/102] Apply suggestions from code review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Melissa Linkert Co-authored-by: Sébastien Besson --- latest/index.bs | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/latest/index.bs b/latest/index.bs index c1e0275b..b991e82e 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -104,7 +104,7 @@ The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL

Transitional metadata is added to the specification with the intention of removing it in the future. Implementations may be expected (MUST) or -encouraged (SHOULD) support the reading of the data, but writing will usually +encouraged (SHOULD) to support the reading of the data, but writing will usually be optional (MAY). Examples of transitional metadata include custom additions by implementations that are later submitted as a formal specification. (See [[#bf2raw]])

@@ -263,7 +263,7 @@ this layout with explicit metadata.

Layout

-Typical Zarr layout produced by running `bioformats2raw` on a fileset that contains more than image (series > 1): +Typical Zarr layout produced by running `bioformats2raw` on a fileset that contains more than one image (series > 1):
 series.ome.zarr               # One converted fileset from bioformats2raw
@@ -278,7 +278,7 @@ series.ome.zarr               # One converted fileset from bioformats2raw
     └── ...
 
-

Attributes

+

Attributes

The top-level `.zattrs` file must contain the `bioformats2raw.layout` key:
@@ -287,7 +287,7 @@ highlight: json
 
If the top-level group represents a plate, the `bioformats2raw.layout` metadata will be present but -the "plate" key key takes precedence and parsing of such datasets should follow [[#plate-md]]. It is not +the "plate" key MUST also be present, takes precedence and parsing of such datasets should follow [[#plate-md]]. It is not possible to mix collections of images with plates at present.

From 861227b38b02183f1d9a438ee5e84746d6930ec9 Mon Sep 17 00:00:00 2001
From: Josh Moore 
Date: Thu, 22 Sep 2022 08:24:10 +0200
Subject: [PATCH 026/102] Apply to v0.4 only

---
 latest/index.bs | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/latest/index.bs b/latest/index.bs
index b991e82e..78052cd1 100644
--- a/latest/index.bs
+++ b/latest/index.bs
@@ -258,8 +258,8 @@ The need for the collection stems from the common "multi-image file" scenario in
 define a strict, stable ordering of the images in a single container that can be used to refer to them by other tools.
 
 In order to capture that information within an OME-NGFF dataset, `bioformats2raw` internally introduced a wrapping layer.
-The bioformats2raw layout can be found in NGFF versions v0.2, v0.3, and v0.4. An upcoming NGFF specification will replace
-this layout with explicit metadata.
+The bioformats2raw layout has been added to v0.4 as a transitional specification to specify filesets that already exist
+in the wild. An upcoming NGFF specification will replace this layout with explicit metadata.
 
 

Layout

From 8e612ef97449e1572a9389f3e38bfa81589876d3 Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Thu, 22 Sep 2022 08:28:58 +0200 Subject: [PATCH 027/102] Re-iterate plate precedence without OME/METADATA.ome.xml --- latest/index.bs | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/latest/index.bs b/latest/index.bs index 78052cd1..4fe84c52 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -315,10 +315,11 @@ Conforming groups: Additionally: - If "OME/METADATA.ome.xml" is present, "OME" MUST be a Zarr group which: - - MAY contain a "series" attribute. - - "series" MUST be a list of string objects, each of which is a path to an image group. - - The order of the paths MUST match the order of the "Image" elements in "OME/METADATA.ome.xml". + - MAY contain a "series" attribute. If so: + - "series" MUST be a list of string objects, each of which is a path to an image group. + - The order of the paths MUST match the order of the "Image" elements in "OME/METADATA.ome.xml". - If "OME/METADATA.ome.xml" or the "series" attribute do not exist: + - existing "plate" metadata will take precedence if it exists, or - separate "multiscales" images MUST be stored in consecutively numbered groups starting from 0 (i.e. "0/", "1/", "2/", "3/", ...). - Every "multiscales" group MUST represent exactly one OME-XML "Image" in the same order as either the series index or the group numbers. From a0919beb2840ca3f7cbee369d2cd38a008c978a2 Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Thu, 22 Sep 2022 08:35:48 +0200 Subject: [PATCH 028/102] Backport latest/bf2raw to 0.4 --- 0.4/examples/bf2raw/.config.json | 3 ++ 0.4/examples/bf2raw/image.json | 3 ++ 0.4/examples/bf2raw/plate.json | 22 ++++++++ 0.4/examples/ome/.config.json | 3 ++ 0.4/examples/ome/series-2.json | 3 ++ 0.4/index.bs | 86 ++++++++++++++++++++++++++++++++ 0.4/schemas/bf2raw.schema | 14 ++++++ 0.4/schemas/ome.schema | 17 +++++++ 8 files changed, 151 insertions(+) create mode 100644 0.4/examples/bf2raw/.config.json create mode 100644 0.4/examples/bf2raw/image.json create mode 100644 0.4/examples/bf2raw/plate.json create mode 100644 0.4/examples/ome/.config.json create mode 100644 0.4/examples/ome/series-2.json create mode 100644 0.4/schemas/bf2raw.schema create mode 100644 0.4/schemas/ome.schema diff --git a/0.4/examples/bf2raw/.config.json b/0.4/examples/bf2raw/.config.json new file mode 100644 index 00000000..2525328c --- /dev/null +++ b/0.4/examples/bf2raw/.config.json @@ -0,0 +1,3 @@ +{ + "schema": "schemas/bf2raw.schema" +} diff --git a/0.4/examples/bf2raw/image.json b/0.4/examples/bf2raw/image.json new file mode 100644 index 00000000..c5eadb8a --- /dev/null +++ b/0.4/examples/bf2raw/image.json @@ -0,0 +1,3 @@ +{ + "bioformats2raw.layout" : 3 +} \ No newline at end of file diff --git a/0.4/examples/bf2raw/plate.json b/0.4/examples/bf2raw/plate.json new file mode 100644 index 00000000..bd98a16e --- /dev/null +++ b/0.4/examples/bf2raw/plate.json @@ -0,0 +1,22 @@ +{ + "bioformats2raw.layout" : 3, + "plate" : { + "columns" : [ { + "name" : "1" + } ], + "name" : "Plate Name 0", + "wells" : [ { + "path" : "A/1", + "rowIndex" : 0, + "columnIndex" : 0 + } ], + "field_count" : 1, + "rows" : [ { + "name" : "A" + } ], + "acquisitions" : [ { + "id" : 0 + } ], + "version" : "0.4" + } +} diff --git a/0.4/examples/ome/.config.json b/0.4/examples/ome/.config.json new file mode 100644 index 00000000..8a611ccf --- /dev/null +++ b/0.4/examples/ome/.config.json @@ -0,0 +1,3 @@ +{ + "schema": "schemas/ome.schema" +} diff --git a/0.4/examples/ome/series-2.json b/0.4/examples/ome/series-2.json new file mode 100644 index 00000000..be7e9ed6 --- /dev/null +++ b/0.4/examples/ome/series-2.json @@ -0,0 +1,3 @@ +{ + "series" : [ "0", "1" ] +} diff --git a/0.4/index.bs b/0.4/index.bs index a0c3f83d..34de5cc3 100644 --- a/0.4/index.bs +++ b/0.4/index.bs @@ -99,6 +99,14 @@ The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL “RECOMMENDED”, “MAY”, and “OPTIONAL” are to be interpreted as described in [RFC 2119](https://tools.ietf.org/html/rfc2119). +

+Transitional metadata is added to the specification with the +intention of removing it in the future. Implementations may be expected (MUST) or +encouraged (SHOULD) to support the reading of the data, but writing will usually +be optional (MAY). Examples of transitional metadata include custom additions by +implementations that are later submitted as a formal specification. (See [[#bf2raw]]) +

+ Some of the JSON examples in this document include commments. However, these are only for clarity purposes and comments MUST NOT be included in JSON objects. @@ -240,6 +248,84 @@ keys as specified below for discovering certain types of data, especially images If part of [[#multiscale-md]], the length of "axes" MUST be equal to the number of dimensions of the arrays that contain the image data. +"bioformats2raw.layout" (transitional) {#bf2raw} +------------------------------------------------ + +[=Transitional=] "bioformats2raw.layout" metadata identifies a group which implicitly describes a series of images. +The need for the collection stems from the common "multi-image file" scenario in microscopy. Parsers like Bio-Formats +define a strict, stable ordering of the images in a single container that can be used to refer to them by other tools. + +In order to capture that information within an OME-NGFF dataset, `bioformats2raw` internally introduced a wrapping layer. +The bioformats2raw layout has been added to v0.4 as a transitional specification to specify filesets that already exist +in the wild. An upcoming NGFF specification will replace this layout with explicit metadata. + +

Layout

+ +Typical Zarr layout produced by running `bioformats2raw` on a fileset that contains more than one image (series > 1): + +
+series.ome.zarr               # One converted fileset from bioformats2raw
+    ├── .zgroup
+    ├── .zattrs               # Contains "bioformats2raw.layout" metadata
+    ├── OME                   # Special group for containing OME metadata
+    │   ├── .zgroup
+    │   ├── .zattrs           # Contains "series" metadata
+    │   └── METADATA.ome.xml  # OME-XML file stored within the Zarr fileset
+    ├── 0                     # First image in the collection
+    ├── 1                     # Second image in the collection
+    └── ...
+
+ +

Attributes

+ +The top-level `.zattrs` file must contain the `bioformats2raw.layout` key: +
+path: examples/bf2raw/image.json
+highlight: json
+
+ +If the top-level group represents a plate, the `bioformats2raw.layout` metadata will be present but +the "plate" key MUST also be present, takes precedence and parsing of such datasets should follow [[#plate-md]]. It is not +possible to mix collections of images with plates at present. + +
+path: examples/bf2raw/plate.json
+highlight: json
+
+ +The `.zattrs` file within the OME group may contain the "series" key: + +
+path: examples/ome/series-2.json
+highlight: json
+
+ +

Details

+ +Conforming groups: + +- MUST have the value "3" for the "bioformats2raw.layout" key in their `.zattrs` metadata at the top of the hierarchy; +- SHOULD have OME metadata representing the entire collection of images in a file named "OME/METADATA.ome.xml" which: + - MUST adhere to the OME-XML specification but + - MUST use `` elements as opposed to ``, `` or ``; + - MAY make use of the [minimum specification](https://docs.openmicroscopy.org/ome-model/6.2.2/specifications/minimum.html). + +Additionally: + +- If "OME/METADATA.ome.xml" is present, "OME" MUST be a Zarr group which: + - MAY contain a "series" attribute. If so: + - "series" MUST be a list of string objects, each of which is a path to an image group. + - The order of the paths MUST match the order of the "Image" elements in "OME/METADATA.ome.xml". +- If "OME/METADATA.ome.xml" or the "series" attribute do not exist: + - existing "plate" metadata will take precedence if it exists, or + - separate "multiscales" images MUST be stored in consecutively numbered groups starting from 0 (i.e. "0/", "1/", "2/", "3/", ...). +- Every "multiscales" group MUST represent exactly one OME-XML "Image" in the same order as either the series index or the group numbers. + +Conforming readers: +- SHOULD make users aware of the presence of more than one image (i.e. SHOULD NOT default to only opening the first image); +- MAY use the "series" attribute in the "OME" group to determine a list of valid groups to display; +- MAY choose to show all images within the collection or offer the user a choice of images, as with HCS plates; +- MAY ignore other groups or arrays under the root of the hierarchy. "coordinateTransformations" metadata {#trafo-md} ------------------------------------- diff --git a/0.4/schemas/bf2raw.schema b/0.4/schemas/bf2raw.schema new file mode 100644 index 00000000..834aee24 --- /dev/null +++ b/0.4/schemas/bf2raw.schema @@ -0,0 +1,14 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://ngff.openmicroscopy.org/latest/schemas/bf2raw.schema", + "title": "NGFF container produced by bioformats2raw", + "description": "JSON from OME-NGFF .zattrs", + "type": "object", + "properties": { + "bioformats2raw.layout": { + "description": "The top-level identifier metadata added by bioformats2raw", + "type": "number", + "enum": [3] + } + } +} diff --git a/0.4/schemas/ome.schema b/0.4/schemas/ome.schema new file mode 100644 index 00000000..bd600a2a --- /dev/null +++ b/0.4/schemas/ome.schema @@ -0,0 +1,17 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://ngff.openmicroscopy.org/latest/schemas/ome.schema", + "title": "NGFF group produced by bioformats2raw to contain OME metadata", + "description": "JSON from OME-NGFF OME/.zattrs linked to an OME-XML file", + "type": "object", + "properties": { + "series": { + "description": "An array of the same length and the same order as the images defined in the OME-XML", + "type": "array", + "items": { + "type": "string" + }, + "minContains": 1 + } + } +} From 59719273573446facb1a639a4242a501186bbd32 Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Thu, 22 Sep 2022 19:41:15 +0200 Subject: [PATCH 029/102] Re-word the 'series' section --- 0.4/index.bs | 11 ++++++----- latest/index.bs | 11 ++++++----- 2 files changed, 12 insertions(+), 10 deletions(-) diff --git a/0.4/index.bs b/0.4/index.bs index 34de5cc3..2df0d380 100644 --- a/0.4/index.bs +++ b/0.4/index.bs @@ -310,14 +310,15 @@ Conforming groups: - MUST use `` elements as opposed to ``, `` or ``; - MAY make use of the [minimum specification](https://docs.openmicroscopy.org/ome-model/6.2.2/specifications/minimum.html). -Additionally: +Additionally, the logic for finding the Zarr group for each image follows the following logic: -- If "OME/METADATA.ome.xml" is present, "OME" MUST be a Zarr group which: +- If "plate" metadata is present, images MUST be located at the defined location. + - Matching "series" metadata as described next MAY be provided for tools that are unaware of the "plate" specification. +- If the "OME" Zarr group exists, it: - MAY contain a "series" attribute. If so: - "series" MUST be a list of string objects, each of which is a path to an image group. - - The order of the paths MUST match the order of the "Image" elements in "OME/METADATA.ome.xml". -- If "OME/METADATA.ome.xml" or the "series" attribute do not exist: - - existing "plate" metadata will take precedence if it exists, or + - The order of the paths MUST match the order of the "Image" elements in "OME/METADATA.ome.xml" if provided. +- If the "series" attribute does not exist and no "plate" is present: - separate "multiscales" images MUST be stored in consecutively numbered groups starting from 0 (i.e. "0/", "1/", "2/", "3/", ...). - Every "multiscales" group MUST represent exactly one OME-XML "Image" in the same order as either the series index or the group numbers. diff --git a/latest/index.bs b/latest/index.bs index 4fe84c52..631f6d88 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -312,14 +312,15 @@ Conforming groups: - MUST use `` elements as opposed to ``, `` or ``; - MAY make use of the [minimum specification](https://docs.openmicroscopy.org/ome-model/6.2.2/specifications/minimum.html). -Additionally: +Additionally, the logic for finding the Zarr group for each image follows the following logic: -- If "OME/METADATA.ome.xml" is present, "OME" MUST be a Zarr group which: +- If "plate" metadata is present, images MUST be located at the defined location. + - Matching "series" metadata as described next MAY be provided for tools that are unaware of the "plate" specification. +- If the "OME" Zarr group exists, it: - MAY contain a "series" attribute. If so: - "series" MUST be a list of string objects, each of which is a path to an image group. - - The order of the paths MUST match the order of the "Image" elements in "OME/METADATA.ome.xml". -- If "OME/METADATA.ome.xml" or the "series" attribute do not exist: - - existing "plate" metadata will take precedence if it exists, or + - The order of the paths MUST match the order of the "Image" elements in "OME/METADATA.ome.xml" if provided. +- If the "series" attribute does not exist and no "plate" is present: - separate "multiscales" images MUST be stored in consecutively numbered groups starting from 0 (i.e. "0/", "1/", "2/", "3/", ...). - Every "multiscales" group MUST represent exactly one OME-XML "Image" in the same order as either the series index or the group numbers. From 88fd0420e62c71223d1b306e3e81e170a8318d90 Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Thu, 22 Sep 2022 21:21:08 +0200 Subject: [PATCH 030/102] Make plate/series link a "SHOULD" --- 0.4/index.bs | 2 +- latest/index.bs | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/0.4/index.bs b/0.4/index.bs index 2df0d380..5a7b196f 100644 --- a/0.4/index.bs +++ b/0.4/index.bs @@ -313,7 +313,7 @@ Conforming groups: Additionally, the logic for finding the Zarr group for each image follows the following logic: - If "plate" metadata is present, images MUST be located at the defined location. - - Matching "series" metadata as described next MAY be provided for tools that are unaware of the "plate" specification. + - Matching "series" metadata (as described next) SHOULD be provided for tools that are unaware of the "plate" specification. - If the "OME" Zarr group exists, it: - MAY contain a "series" attribute. If so: - "series" MUST be a list of string objects, each of which is a path to an image group. diff --git a/latest/index.bs b/latest/index.bs index 631f6d88..c57809a6 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -315,7 +315,7 @@ Conforming groups: Additionally, the logic for finding the Zarr group for each image follows the following logic: - If "plate" metadata is present, images MUST be located at the defined location. - - Matching "series" metadata as described next MAY be provided for tools that are unaware of the "plate" specification. + - Matching "series" metadata (as described next) SHOULD be provided for tools that are unaware of the "plate" specification. - If the "OME" Zarr group exists, it: - MAY contain a "series" attribute. If so: - "series" MUST be a list of string objects, each of which is a path to an image group. From 7b7c43f3016e9731246be8f549ce6b49e17a4c69 Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Mon, 26 Sep 2022 15:16:41 +0200 Subject: [PATCH 031/102] Add 'transitional' to 'omero' spec --- 0.4/index.bs | 6 +++--- latest/index.bs | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/0.4/index.bs b/0.4/index.bs index 5a7b196f..69e418ff 100644 --- a/0.4/index.bs +++ b/0.4/index.bs @@ -402,10 +402,10 @@ if not datasets: datasets = [x["path"] for x in multiscales[0]["datasets"]] ``` -"omero" metadata {#omero-md} ----------------------------- +"omero" metadata (transitional) {#omero-md} +------------------------------------------- -Information specific to the channels of an image and how to render it +[=Transitional=] information specific to the channels of an image and how to render it can be found under the "omero" key in the group-level metadata: ```json diff --git a/latest/index.bs b/latest/index.bs index c57809a6..f19c0d94 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -405,10 +405,10 @@ if not datasets: datasets = [x["path"] for x in multiscales[0]["datasets"]] ``` -"omero" metadata {#omero-md} ----------------------------- +"omero" metadata (transitional) {#omero-md} +------------------------------------------- -Information specific to the channels of an image and how to render it +[=Transitional=] information specific to the channels of an image and how to render it can be found under the "omero" key in the group-level metadata: ```json From 106c3010dcb4079eb0a69868a11cc5943f942fdf Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Mon, 26 Sep 2022 15:16:48 +0200 Subject: [PATCH 032/102] Add 0.4.1 changelog --- 0.4/index.bs | 5 +++++ latest/index.bs | 5 +++++ 2 files changed, 10 insertions(+) diff --git a/0.4/index.bs b/0.4/index.bs index 69e418ff..e441720b 100644 --- a/0.4/index.bs +++ b/0.4/index.bs @@ -694,6 +694,11 @@ Version History {#history} Description + + 0.4.1 + 2022-09-26 + transitional metadata for image collections ("bioformats2raw.layout") + 0.4.0 2022-02-08 diff --git a/latest/index.bs b/latest/index.bs index f19c0d94..2dff9985 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -697,6 +697,11 @@ Version History {#history} Description + + 0.4.1 + 2022-09-26 + transitional metadata for image collections ("bioformats2raw.layout") + 0.4.0 2022-02-08 From a4fa896b09a2968afd055ab40765a0629de4f532 Mon Sep 17 00:00:00 2001 From: Isaac Virshup Date: Wed, 12 Oct 2022 14:42:31 +0200 Subject: [PATCH 033/102] Fix variable --- .github/workflows/review.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/review.yml b/.github/workflows/review.yml index 4b0fe271..4b478cd2 100644 --- a/.github/workflows/review.yml +++ b/.github/workflows/review.yml @@ -21,6 +21,6 @@ jobs: issue-number: ${{ github.event.pull_request.number }} body: | #### Automated Review URLs - * [render latest/index.bs](http://api.csswg.org/bikeshed/?url=https://raw.githubusercontent.com/ome/ngff/${{ github.head.sha }}/latest/index.bs) - * [diff latest modified](https://services.w3.org/htmldiff?doc1=https%3A%2F%2Fngff.openmicroscopy.org%2Flatest%2F&doc2=http%3A%2F%2Fapi.csswg.org%2Fbikeshed%2F%3Furl%3Dhttps%3A%2F%2Fraw.githubusercontent.com%2Fome%2Fngff%2F${{ github.head.sha }}%2Flatest%2Findex.bs) + * [render latest/index.bs](http://api.csswg.org/bikeshed/?url=https://raw.githubusercontent.com/ome/ngff/${{ github.event.pull_request.head.sha }}/latest/index.bs) + * [diff latest modified](https://services.w3.org/htmldiff?doc1=https%3A%2F%2Fngff.openmicroscopy.org%2Flatest%2F&doc2=http%3A%2F%2Fapi.csswg.org%2Fbikeshed%2F%3Furl%3Dhttps%3A%2F%2Fraw.githubusercontent.com%2Fome%2Fngff%2F${{ github.event.pull_request.head.sha }}%2Flatest%2Findex.bs) edit-mode: replace From 40034d58d1d1e963a447c09cc4fe1685cea08ce9 Mon Sep 17 00:00:00 2001 From: William Moore Date: Wed, 2 Nov 2022 16:56:11 +0000 Subject: [PATCH 034/102] Fix 'units' -> 'unit' in 0.4 and latest schemas, examples and test files --- .../multiscales_transformations.json | 4 +- 0.4/schemas/image.schema | 2 +- 0.4/tests/image_suite.json | 92 +++++++++---------- 0.4/tests/strict_image_suite.json | 18 ++-- .../multiscales_transformations.json | 4 +- latest/schemas/image.schema | 2 +- latest/tests/image_suite.json | 92 +++++++++---------- latest/tests/strict_image_suite.json | 18 ++-- 8 files changed, 116 insertions(+), 116 deletions(-) diff --git a/0.4/examples/multiscales_strict/multiscales_transformations.json b/0.4/examples/multiscales_strict/multiscales_transformations.json index 82836d0f..0097c47c 100644 --- a/0.4/examples/multiscales_strict/multiscales_transformations.json +++ b/0.4/examples/multiscales_strict/multiscales_transformations.json @@ -5,12 +5,12 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ diff --git a/0.4/schemas/image.schema b/0.4/schemas/image.schema index 59e8e28e..aec5db5f 100644 --- a/0.4/schemas/image.schema +++ b/0.4/schemas/image.schema @@ -124,7 +124,7 @@ "type": "string", "enum": ["space"] }, - "units": { + "unit": { "type": "string" } } diff --git a/0.4/tests/image_suite.json b/0.4/tests/image_suite.json index f22fa67f..42906985 100644 --- a/0.4/tests/image_suite.json +++ b/0.4/tests/image_suite.json @@ -14,17 +14,17 @@ { "name": "t", "type": "time", - "units": "micrometer" + "unit": "micrometer" }, { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ @@ -60,12 +60,12 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ @@ -115,12 +115,12 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ] } @@ -138,12 +138,12 @@ { "name": "y", "type": "space", - "units": "micron" + "unit": "micron" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ @@ -204,12 +204,12 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ] } @@ -231,12 +231,12 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ @@ -270,12 +270,12 @@ { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ @@ -344,12 +344,12 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ @@ -382,12 +382,12 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ @@ -476,12 +476,12 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ @@ -530,11 +530,11 @@ "axes": [ { "type": "space", - "units": "micron" + "unit": "micron" }, { "type": "space", - "units": "micron" + "unit": "micron" } ], "datasets": [ @@ -567,7 +567,7 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ @@ -641,12 +641,12 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ @@ -679,12 +679,12 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ @@ -725,12 +725,12 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ @@ -754,12 +754,12 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [], @@ -779,12 +779,12 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "version": "0.4" @@ -829,12 +829,12 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ @@ -867,12 +867,12 @@ { "name": "y", "type": "invalid", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ @@ -905,12 +905,12 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ @@ -1032,12 +1032,12 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ @@ -1087,12 +1087,12 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ @@ -1117,12 +1117,12 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ diff --git a/0.4/tests/strict_image_suite.json b/0.4/tests/strict_image_suite.json index 22b34c5e..ec6295d3 100644 --- a/0.4/tests/strict_image_suite.json +++ b/0.4/tests/strict_image_suite.json @@ -122,12 +122,12 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ @@ -205,12 +205,12 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ] } @@ -228,12 +228,12 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ @@ -279,17 +279,17 @@ { "name": "z", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ diff --git a/latest/examples/multiscales_strict/multiscales_transformations.json b/latest/examples/multiscales_strict/multiscales_transformations.json index c99040c2..80dbedab 100644 --- a/latest/examples/multiscales_strict/multiscales_transformations.json +++ b/latest/examples/multiscales_strict/multiscales_transformations.json @@ -5,12 +5,12 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ diff --git a/latest/schemas/image.schema b/latest/schemas/image.schema index 3926c4f4..055b058a 100644 --- a/latest/schemas/image.schema +++ b/latest/schemas/image.schema @@ -124,7 +124,7 @@ "type": "string", "enum": ["space"] }, - "units": { + "unit": { "type": "string" } } diff --git a/latest/tests/image_suite.json b/latest/tests/image_suite.json index 39cb73d5..7c3300b5 100644 --- a/latest/tests/image_suite.json +++ b/latest/tests/image_suite.json @@ -14,17 +14,17 @@ { "name": "t", "type": "time", - "units": "micrometer" + "unit": "micrometer" }, { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ @@ -60,12 +60,12 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ @@ -115,12 +115,12 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ] } @@ -138,12 +138,12 @@ { "name": "y", "type": "space", - "units": "micron" + "unit": "micron" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ @@ -204,12 +204,12 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ] } @@ -231,12 +231,12 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ @@ -270,12 +270,12 @@ { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ @@ -344,12 +344,12 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ @@ -382,12 +382,12 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ @@ -476,12 +476,12 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ @@ -530,11 +530,11 @@ "axes": [ { "type": "space", - "units": "micron" + "unit": "micron" }, { "type": "space", - "units": "micron" + "unit": "micron" } ], "datasets": [ @@ -567,7 +567,7 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ @@ -641,12 +641,12 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ @@ -679,12 +679,12 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ @@ -725,12 +725,12 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ @@ -754,12 +754,12 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [], @@ -779,12 +779,12 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "version": "0.5-dev" @@ -829,12 +829,12 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ @@ -867,12 +867,12 @@ { "name": "y", "type": "invalid", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ @@ -905,12 +905,12 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ @@ -1032,12 +1032,12 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ @@ -1087,12 +1087,12 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ @@ -1117,12 +1117,12 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ diff --git a/latest/tests/strict_image_suite.json b/latest/tests/strict_image_suite.json index b8aeebe1..7bff90ca 100644 --- a/latest/tests/strict_image_suite.json +++ b/latest/tests/strict_image_suite.json @@ -122,12 +122,12 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ @@ -205,12 +205,12 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ] } @@ -228,12 +228,12 @@ { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ @@ -279,17 +279,17 @@ { "name": "z", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "y", "type": "space", - "units": "micrometer" + "unit": "micrometer" }, { "name": "x", "type": "space", - "units": "micrometer" + "unit": "micrometer" } ], "datasets": [ From 548573ef35c608ffe878382ae0fdb68d7bc8dbf4 Mon Sep 17 00:00:00 2001 From: JulianHn <53080416+JulianHn@users.noreply.github.com> Date: Thu, 10 Nov 2022 10:23:02 +0100 Subject: [PATCH 035/102] Replace link to OME logo with an https link Link to the OME logo was provided as non SSL http link. Replace it with an https:// link to fix mixed-content warnings on https://ngff.openmicroscopy.org/latest/ --- latest/header.include | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/latest/header.include b/latest/header.include index 28305fe4..e98bf6fe 100644 --- a/latest/header.include +++ b/latest/header.include @@ -11,7 +11,7 @@
- OME logo (6 circles in a hexagon) From ee4d5dab677636a28f1f65c248a751e279a0d1fe Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Tue, 31 Jan 2023 16:49:33 +0100 Subject: [PATCH 036/102] Add redirects for tools and data for the OME-Zarr paper --- data/index.html | 6 ++++++ tools/index.html | 6 ++++++ 2 files changed, 12 insertions(+) create mode 100644 data/index.html create mode 100644 tools/index.html diff --git a/data/index.html b/data/index.html new file mode 100644 index 00000000..a04bf2af --- /dev/null +++ b/data/index.html @@ -0,0 +1,6 @@ + + + + +

If you are not redirected, click here.

+ diff --git a/tools/index.html b/tools/index.html new file mode 100644 index 00000000..ec7101c1 --- /dev/null +++ b/tools/index.html @@ -0,0 +1,6 @@ + + + + +

If you are not redirected, click here.

+ From 6161d54931260132d09af1fab703edd02c65474d Mon Sep 17 00:00:00 2001 From: Virginia Scarlett Date: Thu, 9 Feb 2023 15:35:44 -0500 Subject: [PATCH 037/102] rewrote labels section --- .../label_strict/colors_properties.json | 20 ++-- latest/index.bs | 91 ++++++++++--------- 2 files changed, 58 insertions(+), 53 deletions(-) diff --git a/latest/examples/label_strict/colors_properties.json b/latest/examples/label_strict/colors_properties.json index 7f1d6f20..2b96b648 100644 --- a/latest/examples/label_strict/colors_properties.json +++ b/latest/examples/label_strict/colors_properties.json @@ -1,25 +1,27 @@ { "image-label": { - "version": "0.5-dev", + "version": "0.4", "colors": [ { - "label-value": 1, - "rgba": [255, 255, 255, 255] + "label-value": 0, + "rgba": [0, 0, 128, 128] }, { - "label-value": 4, - "rgba": [0, 255, 255, 128] + "label-value": 1, + "rgba": [0, 128, 0, 128] } ], "properties": [ { - "label-value": 1, + "label-value": 0, "area (pixels)": 1200, - "class": "foo" + "class": "intercellular space" }, { - "label-value": 4, - "area (pixels)": 1650 + "label-value": 1, + "area (pixels)": 1650, + "class": "cell", + "cell type": "neuron" } ], "source": { diff --git a/latest/index.bs b/latest/index.bs index bb935c48..8b96bef9 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -3,9 +3,9 @@ Title: Next-generation file formats (NGFF) Shortname: ome-ngff Level: 1 Status: LS-COMMIT -Status: w3c/ED +Status: w3c/CG-FINAL Group: ome -URL: https://ngff.openmicroscopy.org/latest/ +URL: https://ngff.openmicroscopy.org/0.4/ Repository: https://github.com/ome/ngff Issue Tracking: Forums https://forum.image.sc/tag/ome-ngff Logo: http://www.openmicroscopy.org/img/logos/ome-logomark.svg @@ -16,13 +16,11 @@ Markup Shorthands: markdown yes Editor: Josh Moore, University of Dundee (UoD) https://www.dundee.ac.uk, https://orcid.org/0000-0003-4028-811X Editor: Sébastien Besson, University of Dundee (UoD) https://www.dundee.ac.uk, https://orcid.org/0000-0001-8783-1429 Editor: Constantin Pape, European Molecular Biology Laboratory (EMBL) https://www.embl.org/sites/heidelberg/, https://orcid.org/0000-0001-6562-7187 -Text Macro: NGFFVERSION 0.5-dev Abstract: This document contains next-generation file format (NGFF) Abstract: specifications for storing bioimaging data in the cloud. Abstract: All specifications are submitted to the https://image.sc community for review. -Status Text: The current released version of this specification is -Status Text: 0.4. Migration scripts -Status Text: will be provided between numbered versions. Data written with these latest changes +Status Text: This is the 0.4 release of this specification. Migration scripts +Status Text: will be provided between numbered versions. Data written with the latest version Status Text: (an "editor's draft") will not necessarily be supported.
@@ -330,9 +328,8 @@ Conforming readers: - MAY choose to show all images within the collection or offer the user a choice of images, as with HCS plates; - MAY ignore other groups or arrays under the root of the hierarchy. - "coordinateTransformations" metadata {#trafo-md} ------------------------------------------------- +------------------------------------- "coordinateTransformations" describe a series of transformations that map between two coordinate spaces (defined by "axes"). For example, to map a discrete data space of an array to the corresponding physical space. @@ -380,7 +377,7 @@ Each "multiscales" dictionary MAY contain the field "coordinateTransformations", The transformations MUST follow the same rules about allowed types, order, etc. as in "datasets:coordinateTransformations" and are applied after them. They can for example be used to specify the `scale` for a dimension that is the same for all resolutions. -Each "multiscales" dictionary SHOULD contain the field "name". It SHOULD contain the field "version", which indicates the version of the multiscale metadata of this image (current version is [NGFFVERSION]). +Each "multiscales" dictionary SHOULD contain the field "name". It SHOULD contain the field "version", which indicates the version of the multiscale metadata of this image (current version is 0.4). Each "multiscales" dictionary SHOULD contain the field "type", which gives the type of downscaling method used to generate the multiscale image pyramid. It SHOULD contain the field "metadata", which contains a dictionary with additional information about the downscaling method. @@ -414,7 +411,7 @@ can be found under the "omero" key in the group-level metadata: ```json "id": 1, # ID in OMERO "name": "example.tif", # Name as shown in the UI -"version": "0.5-dev", # Current version +"version": "0.4", # Current version "channels": [ # Array matching the c dimension size { "active": true, @@ -444,59 +441,65 @@ for more information. "labels" metadata {#labels-md} ------------------------------ -The special group "labels" found under an image Zarr contains the key `labels` containing -the paths to label objects which can be found underneath the group: +In OME-Zarr, Zarr arrays representing pixel-annotation data are stored in a group called "labels". Some applications--notably image segmentation--produce +a new image that is in the same coordinate system as a corresponding multiscale image (usually having the same dimensions and coordinate transformations). +This new image is composed of integer values corresponding to certain labels with custom meanings. For example, pixels take the value 1 or 0 +if the corresponding pixel in the original image represents cellular space or intercellular space, respectively. +Such an image is referred to in this specification as a 'label image'. + +The "labels" group is nested within an image group, at the same level of the Zarr hierarchy as the resolution levels for the original image. +The "labels" group is not itself an image; it contains images. The label images MUST be of a discrete (integer) type, i.e. one of +[`uint8`, `int8`, `uint16`, `int16`, `uint32`, `int32`, `uint64`, `int64`]. Intermediate groups between "labels" and the images within it are allowed, +but these MUST NOT contain metadata. Names of the images in the "labels" group are arbitrary. + +The `.zattrs` file associated with the "labels" group MUST contain a JSON object with the key `labels`, whose value is a JSON array of paths to the +labeled multiscale image(s). All images in the "labels" group SHOULD be listed here. For example: ```json { "labels": [ - "orphaned/0" + "cell_space_segmentation/" ] } ``` -Unlisted groups MAY be labels. - -"image-label" metadata {#label-md} ----------------------------------- +The `.zattrs` file within an image within a "labels" group MUST contain a `multiscales` JSON object. Within the `multiscales` object, the JSON array +associated with the `datasets` key MUST have the same number of entries (scale levels) as the original unlabeled image. -Groups containing the `image-label` dictionary represent an image segmentation -in which each unique pixel value represents a separate segmented object. -`image-label` groups MUST also contain `multiscales` metadata and the two -"datasets" series MUST have the same number of entries. +In addition to the `multiscales` key, the JSON object in this image-level `.zattrs` file SHOULD contain another key, `image-label`, +whose value is also a JSON object. The `image-label` object stores information about the display colors, source image, and optionally, +further arbitrary properties of the label image. That `image-label` object SHOULD contain the following keys: first, a `colors` key, +whose value MUST be a JSON array describing color information for the unique label values. Second, a `version` key, whose value MUST be a +string specifying the version of the OME-NGFF `image-label` schema. -The `image-label` dictionary SHOULD contain a `colors` key whose value MUST be a -list of JSON objects describing the unique label values. Each color object MUST -contain the `label-value` key whose value MUST be an integer specifying the -pixel value for that label. It MAY contain an `rgba` key whose value MUST be an array -of four integers between 0 and 255 `[uint8, uint8, uint8, uint8]` specifying the label -color as RGBA. All the values under the `label-value` key MUST be unique. Clients -who choose to not throw an error SHOULD ignore all except the _last_ entry. +Conforming readers SHOULD display labels using the colors specified by the `colors` JSON array, as follows. This array contains one +JSON object for each unique custom label. Each of these objects MUST contain the `label-value` key, whose value MUST be the integer +corresponding to a particular label. In addition to the `label-value` key, the objects in this array MAY contain an `rgba` key whose +value MUST be an array of four integers between 0 and 255, inclusive. These integers represent the `uint8` values of red, green, and +blue that comprise the final color to be displayed at the pixels with this label. The fourth integer in the `rgba` array represents alpha, +or the opacity of the color. Additional keys under `colors` are allowed. -Some implementations MAY represent overlapping labels by using a specially assigned -value, for example the highest integer available in the pixel range. +Next, the `image-label` object MAY contain the following keys: a `properties` key, and a `source` key. -The `image-label` dictionary MAY contain a `properties` key whose value MUST be a -list of JSON objects which also describes the unique label values. Each property object -MUST contain the `label-value` key whose value MUST be an integer specifying the pixel -value for that label. Additionally, an arbitrary number of key-value pairs -MAY be present for each label value denoting associated metadata. Not all label -values must share the same key-value pairs within the properties list. +Like the `colors` key, the value of the `properties` key MUST be an array of JSON objects describing the set of unique possible pixel values. +Each object in the `properties` array MUST contain the `label-value` key, whose value again MUST be an integer specifying the pixel value for that label. +Additionally, an arbitrary number of key-value pairs MAY be present for each label value, denoting arbitrary metadata associated with that label. +Label-value objects within the `properties` array do not need to have the same keys. -The `image-label` dictionary MAY contain a `source` key whose value MUST be a JSON -object containing information on the image the label is associated with. If included, -it MAY include a key `image` whose value MUST be a string specifying the relative -path to a Zarr image group. The default value is "../../" since most labels are stored -under a subgroup named "labels/" (see above). +The value of the `source` key MUST be a JSON object containing information about the original image from which the label image derives. +This object MAY include a key `image`, whose value MUST be a string specifying the relative path to a Zarr image group. +The default value is `../../` since most labeled images are stored in a "labels" group that is nested within the original image group. -The `image-label` dictionary SHOULD contain a `version` key whose value MUST be a string -specifying the version of the image-label specification. +Here is an example of a simple `image-label` object for a label image in which 0s and 1s represent intercellular and cellular space, respectively:
 path: examples/label_strict/colors_properties.json
 highlight: json
 
+In this case, the pixels consisting of a 0 in the Zarr array will be displayed as 50% blue and 50% opacity. Pixels with a 1 in the Zarr array, +which correspond to cellular space, will be displayed as 50% green and 50% opacity. + "plate" metadata {#plate-md} ---------------------------- @@ -541,7 +544,7 @@ other `name` in the `rows` list. Care SHOULD be taken to avoid collisions on case-insensitive filesystems (e.g. avoid using both `Aa` and `aA`). The `plate` dictionary SHOULD contain a `version` key whose value MUST be a string specifying the -version of the plate specification. +version of the plate specificaton. The `plate` dictionary MUST contain a `wells` key whose value MUST be a list of JSON objects defining the wells of the plate. Each well object MUST contain a `path` key whose value MUST From 5b430a97b93ced1e87375f000927fae0a1a55ed1 Mon Sep 17 00:00:00 2001 From: Virginia Scarlett Date: Thu, 9 Feb 2023 15:43:09 -0500 Subject: [PATCH 038/102] updated changelog on latest/index.bs --- latest/index.bs | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/latest/index.bs b/latest/index.bs index 8b96bef9..b01f7499 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -674,6 +674,11 @@ Version History {#history} Description + + 0.4.1 + 2023-02-09 + expand on "labels" description + 0.4.1 2022-09-26 From 80316ca87f37d42db15e4179bf04b223a13e2c00 Mon Sep 17 00:00:00 2001 From: Virginia Scarlett Date: Fri, 10 Feb 2023 15:01:49 -0500 Subject: [PATCH 039/102] cleaned up latest/index.bs in response to PR comment --- latest/index.bs | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/latest/index.bs b/latest/index.bs index b01f7499..73fb5024 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -3,9 +3,9 @@ Title: Next-generation file formats (NGFF) Shortname: ome-ngff Level: 1 Status: LS-COMMIT -Status: w3c/CG-FINAL +Status: w3c/ED Group: ome -URL: https://ngff.openmicroscopy.org/0.4/ +URL: https://ngff.openmicroscopy.org/latest/ Repository: https://github.com/ome/ngff Issue Tracking: Forums https://forum.image.sc/tag/ome-ngff Logo: http://www.openmicroscopy.org/img/logos/ome-logomark.svg @@ -16,11 +16,13 @@ Markup Shorthands: markdown yes Editor: Josh Moore, University of Dundee (UoD) https://www.dundee.ac.uk, https://orcid.org/0000-0003-4028-811X Editor: Sébastien Besson, University of Dundee (UoD) https://www.dundee.ac.uk, https://orcid.org/0000-0001-8783-1429 Editor: Constantin Pape, European Molecular Biology Laboratory (EMBL) https://www.embl.org/sites/heidelberg/, https://orcid.org/0000-0001-6562-7187 +Text Macro: NGFFVERSION 0.5-dev Abstract: This document contains next-generation file format (NGFF) Abstract: specifications for storing bioimaging data in the cloud. Abstract: All specifications are submitted to the https://image.sc community for review. -Status Text: This is the 0.4 release of this specification. Migration scripts -Status Text: will be provided between numbered versions. Data written with the latest version +Status Text: The current released version of this specification is +Status Text: 0.4. Migration scripts +Status Text: will be provided between numbered versions. Data written with these latest changes Status Text: (an "editor's draft") will not necessarily be supported.
@@ -328,8 +330,9 @@ Conforming readers: - MAY choose to show all images within the collection or offer the user a choice of images, as with HCS plates; - MAY ignore other groups or arrays under the root of the hierarchy. + "coordinateTransformations" metadata {#trafo-md} -------------------------------------- +------------------------------------------------ "coordinateTransformations" describe a series of transformations that map between two coordinate spaces (defined by "axes"). For example, to map a discrete data space of an array to the corresponding physical space. @@ -377,7 +380,7 @@ Each "multiscales" dictionary MAY contain the field "coordinateTransformations", The transformations MUST follow the same rules about allowed types, order, etc. as in "datasets:coordinateTransformations" and are applied after them. They can for example be used to specify the `scale` for a dimension that is the same for all resolutions. -Each "multiscales" dictionary SHOULD contain the field "name". It SHOULD contain the field "version", which indicates the version of the multiscale metadata of this image (current version is 0.4). +Each "multiscales" dictionary SHOULD contain the field "name". It SHOULD contain the field "version", which indicates the version of the multiscale metadata of this image (current version is [NGFFVERSION]). Each "multiscales" dictionary SHOULD contain the field "type", which gives the type of downscaling method used to generate the multiscale image pyramid. It SHOULD contain the field "metadata", which contains a dictionary with additional information about the downscaling method. @@ -411,7 +414,7 @@ can be found under the "omero" key in the group-level metadata: ```json "id": 1, # ID in OMERO "name": "example.tif", # Name as shown in the UI -"version": "0.4", # Current version +"version": "0.5-dev", # Current version "channels": [ # Array matching the c dimension size { "active": true, @@ -544,7 +547,7 @@ other `name` in the `rows` list. Care SHOULD be taken to avoid collisions on case-insensitive filesystems (e.g. avoid using both `Aa` and `aA`). The `plate` dictionary SHOULD contain a `version` key whose value MUST be a string specifying the -version of the plate specificaton. +version of the plate specification. The `plate` dictionary MUST contain a `wells` key whose value MUST be a list of JSON objects defining the wells of the plate. Each well object MUST contain a `path` key whose value MUST From f90e0e4a43e60fffd4c8f6f74e2f68792cb254e9 Mon Sep 17 00:00:00 2001 From: Virginia Scarlett Date: Fri, 17 Feb 2023 11:26:26 -0500 Subject: [PATCH 040/102] fixed version and clarified some points in labels --- latest/examples/label_strict/colors_properties.json | 2 +- latest/index.bs | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/latest/examples/label_strict/colors_properties.json b/latest/examples/label_strict/colors_properties.json index 2b96b648..c96820e1 100644 --- a/latest/examples/label_strict/colors_properties.json +++ b/latest/examples/label_strict/colors_properties.json @@ -1,6 +1,6 @@ { "image-label": { - "version": "0.4", + "version": "0.5-dev", "colors": [ { "label-value": 0, diff --git a/latest/index.bs b/latest/index.bs index 73fb5024..7c329b72 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -451,12 +451,12 @@ if the corresponding pixel in the original image represents cellular space or in Such an image is referred to in this specification as a 'label image'. The "labels" group is nested within an image group, at the same level of the Zarr hierarchy as the resolution levels for the original image. -The "labels" group is not itself an image; it contains images. The label images MUST be of a discrete (integer) type, i.e. one of +The "labels" group is not itself an image; it contains images. The pixels of the label images MUST be integer data types, i.e. one of [`uint8`, `int8`, `uint16`, `int16`, `uint32`, `int32`, `uint64`, `int64`]. Intermediate groups between "labels" and the images within it are allowed, but these MUST NOT contain metadata. Names of the images in the "labels" group are arbitrary. The `.zattrs` file associated with the "labels" group MUST contain a JSON object with the key `labels`, whose value is a JSON array of paths to the -labeled multiscale image(s). All images in the "labels" group SHOULD be listed here. For example: +labeled multiscale image(s). All label images SHOULD be listed within this metadata file. For example: ```json { @@ -466,7 +466,7 @@ labeled multiscale image(s). All images in the "labels" group SHOULD be listed h } ``` -The `.zattrs` file within an image within a "labels" group MUST contain a `multiscales` JSON object. Within the `multiscales` object, the JSON array +The `.zattrs` file for the label image MUST implement the multiscales specification. Within the `multiscales` object, the JSON array associated with the `datasets` key MUST have the same number of entries (scale levels) as the original unlabeled image. In addition to the `multiscales` key, the JSON object in this image-level `.zattrs` file SHOULD contain another key, `image-label`, From 96a656adf47bf01326ad4e5915f6796c851a0091 Mon Sep 17 00:00:00 2001 From: Virginia Scarlett Date: Mon, 27 Feb 2023 13:11:52 -0500 Subject: [PATCH 041/102] removed pesky trailing slash --- latest/index.bs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/latest/index.bs b/latest/index.bs index 7c329b72..aa5feb95 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -461,7 +461,7 @@ labeled multiscale image(s). All label images SHOULD be listed within this metad ```json { "labels": [ - "cell_space_segmentation/" + "cell_space_segmentation" ] } ``` From f0705d9f75a3791127cda1fab1e2a9e8ca1ac18b Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Tue, 14 Mar 2023 09:02:31 +0100 Subject: [PATCH 042/102] add sphinx infra --- Makefile | 20 ++++++++++++++++++++ conf.py | 27 +++++++++++++++++++++++++++ index.rst | 20 ++++++++++++++++++++ make.bat | 35 +++++++++++++++++++++++++++++++++++ 4 files changed, 102 insertions(+) create mode 100644 Makefile create mode 100644 conf.py create mode 100644 index.rst create mode 100644 make.bat diff --git a/Makefile b/Makefile new file mode 100644 index 00000000..d4bb2cbb --- /dev/null +++ b/Makefile @@ -0,0 +1,20 @@ +# Minimal makefile for Sphinx documentation +# + +# You can set these variables from the command line, and also +# from the environment for the first two. +SPHINXOPTS ?= +SPHINXBUILD ?= sphinx-build +SOURCEDIR = . +BUILDDIR = _build + +# Put it first so that "make" without argument is like "make help". +help: + @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) + +.PHONY: help Makefile + +# Catch-all target: route all unknown targets to Sphinx using the new +# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). +%: Makefile + @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) diff --git a/conf.py b/conf.py new file mode 100644 index 00000000..d142710a --- /dev/null +++ b/conf.py @@ -0,0 +1,27 @@ +# Configuration file for the Sphinx documentation builder. +# +# For the full list of built-in configuration values, see the documentation: +# https://www.sphinx-doc.org/en/master/usage/configuration.html + +# -- Project information ----------------------------------------------------- +# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information + +project = 'NGFF' +copyright = '2023, NGFF Community' +author = 'NGFF Community' + +# -- General configuration --------------------------------------------------- +# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration + +extensions = [] + +templates_path = ['_templates'] +exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store'] + + + +# -- Options for HTML output ------------------------------------------------- +# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output + +html_theme = 'alabaster' +html_static_path = ['_static'] diff --git a/index.rst b/index.rst new file mode 100644 index 00000000..2bafe936 --- /dev/null +++ b/index.rst @@ -0,0 +1,20 @@ +.. NGFF documentation master file, created by + sphinx-quickstart on Tue Mar 14 08:54:12 2023. + You can adapt this file completely to your liking, but it should at least + contain the root `toctree` directive. + +Welcome to NGFF's documentation! +================================ + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + + +Indices and tables +================== + +* :ref:`genindex` +* :ref:`modindex` +* :ref:`search` diff --git a/make.bat b/make.bat new file mode 100644 index 00000000..954237b9 --- /dev/null +++ b/make.bat @@ -0,0 +1,35 @@ +@ECHO OFF + +pushd %~dp0 + +REM Command file for Sphinx documentation + +if "%SPHINXBUILD%" == "" ( + set SPHINXBUILD=sphinx-build +) +set SOURCEDIR=. +set BUILDDIR=_build + +%SPHINXBUILD% >NUL 2>NUL +if errorlevel 9009 ( + echo. + echo.The 'sphinx-build' command was not found. Make sure you have Sphinx + echo.installed, then set the SPHINXBUILD environment variable to point + echo.to the full path of the 'sphinx-build' executable. Alternatively you + echo.may add the Sphinx directory to PATH. + echo. + echo.If you don't have Sphinx installed, grab it from + echo.https://www.sphinx-doc.org/ + exit /b 1 +) + +if "%1" == "" goto help + +%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% +goto end + +:help +%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% + +:end +popd From 3270186087a304986ff65ebd84c2fe41e85d4662 Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Tue, 14 Mar 2023 09:04:45 +0100 Subject: [PATCH 043/102] Move to markdown --- .readthedocs.yaml | 8 ++++++++ conf.py | 3 ++- index.rst => index.md | 0 requirements.txt | 1 + 4 files changed, 11 insertions(+), 1 deletion(-) create mode 100644 .readthedocs.yaml rename index.rst => index.md (100%) create mode 100644 requirements.txt diff --git a/.readthedocs.yaml b/.readthedocs.yaml new file mode 100644 index 00000000..21e4796c --- /dev/null +++ b/.readthedocs.yaml @@ -0,0 +1,8 @@ +version: 2 + +sphinx: + configuration: conf.py + +python: + install: + - requirements: requirements.txt diff --git a/conf.py b/conf.py index d142710a..68f61cd4 100644 --- a/conf.py +++ b/conf.py @@ -13,7 +13,8 @@ # -- General configuration --------------------------------------------------- # https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration -extensions = [] +extensions = ["myst_parser"] +source_suffix = [".rst", ".md"] templates_path = ['_templates'] exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store'] diff --git a/index.rst b/index.md similarity index 100% rename from index.rst rename to index.md diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 00000000..f0694bdc --- /dev/null +++ b/requirements.txt @@ -0,0 +1 @@ +myst-parser From 0c52d38a11a5380a4d2137caab403fb6848a26f9 Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Tue, 14 Mar 2023 09:38:03 +0100 Subject: [PATCH 044/102] Add bikeshed build --- 0.1/index.bs | 3 +-- 0.2/index.bs | 3 +-- 0.3/index.bs | 3 +-- 0.4/index.bs | 3 +-- conf.py | 19 ++++++++++++++++++- requirements.txt | 1 + 6 files changed, 23 insertions(+), 9 deletions(-) diff --git a/0.1/index.bs b/0.1/index.bs index b60204f1..52a33115 100644 --- a/0.1/index.bs +++ b/0.1/index.bs @@ -501,8 +501,7 @@ Projects which support reading and/or writing OME-NGFF data include: -Diagram of related projects +Diagram of related projects All implementations prevent an equivalent representation of a dataset which can be downloaded or uploaded freely. An interactive version of this diagram is available from the [OME2020 Workshop](https://downloads.openmicroscopy.org/presentations/2020/Dundee/Workshops/NGFF/zarr_diagram/). diff --git a/0.2/index.bs b/0.2/index.bs index 4e25ad2c..487f91a3 100644 --- a/0.2/index.bs +++ b/0.2/index.bs @@ -559,8 +559,7 @@ Projects which support reading and/or writing OME-NGFF data include: -Diagram of related projects +Diagram of related projects All implementations prevent an equivalent representation of a dataset which can be downloaded or uploaded freely. An interactive version of this diagram is available from the [OME2020 Workshop](https://downloads.openmicroscopy.org/presentations/2020/Dundee/Workshops/NGFF/zarr_diagram/). diff --git a/0.3/index.bs b/0.3/index.bs index 84192b65..47bc2dcf 100644 --- a/0.3/index.bs +++ b/0.3/index.bs @@ -569,8 +569,7 @@ Projects which support reading and/or writing OME-NGFF data include: -Diagram of related projects +Diagram of related projects All implementations prevent an equivalent representation of a dataset which can be downloaded or uploaded freely. An interactive version of this diagram is available from the [OME2020 Workshop](https://downloads.openmicroscopy.org/presentations/2020/Dundee/Workshops/NGFF/zarr_diagram/). diff --git a/0.4/index.bs b/0.4/index.bs index d39b2c5f..e2a41769 100644 --- a/0.4/index.bs +++ b/0.4/index.bs @@ -639,8 +639,7 @@ Projects which support reading and/or writing OME-NGFF data include: -Diagram of related projects +Diagram of related projects All implementations prevent an equivalent representation of a dataset which can be downloaded or uploaded freely. An interactive version of this diagram is available from the [OME2020 Workshop](https://downloads.openmicroscopy.org/presentations/2020/Dundee/Workshops/NGFF/zarr_diagram/). diff --git a/conf.py b/conf.py index 68f61cd4..cecd49b2 100644 --- a/conf.py +++ b/conf.py @@ -17,7 +17,7 @@ source_suffix = [".rst", ".md"] templates_path = ['_templates'] -exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store'] +exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store', '.git', '.pytest_cache', '**/.pytest_cache', '**/.tox', 'README.md', 'LICENSE.md', 'CONTRIBUTING.md'] @@ -26,3 +26,20 @@ html_theme = 'alabaster' html_static_path = ['_static'] + + +# Run bikeshed build +import glob +import os +import subprocess +import shutil + + +for index_file in glob.glob("[0-9]*/index.bs"): + + output_file = index_file.replace("bs", "html") + subprocess.check_call(f"bikeshed spec {index_file} {output_file}", shell=True) + + dir_name = index_file.split("/")[0] + os.makedirs(f"_build/{dir_name}", exist_ok=True) + shutil.copyfile(output_file, f"_build/{output_file}") diff --git a/requirements.txt b/requirements.txt index f0694bdc..73e17e38 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1 +1,2 @@ +bikeshed myst-parser From 8c4a40dd3d1651567a1ba10b1a82c107948486fd Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Tue, 14 Mar 2023 09:45:25 +0100 Subject: [PATCH 045/102] add simple tools/ and data/ hooks --- data/index.html | 6 ------ data/index.md | 5 +++++ index.md => index.rst | 14 ++++---------- tools/index.html | 6 ------ tools/index.md | 5 +++++ 5 files changed, 14 insertions(+), 22 deletions(-) delete mode 100644 data/index.html create mode 100644 data/index.md rename index.md => index.rst (62%) delete mode 100644 tools/index.html create mode 100644 tools/index.md diff --git a/data/index.html b/data/index.html deleted file mode 100644 index a04bf2af..00000000 --- a/data/index.html +++ /dev/null @@ -1,6 +0,0 @@ - - - - -

If you are not redirected, click here.

- diff --git a/data/index.md b/data/index.md new file mode 100644 index 00000000..74187369 --- /dev/null +++ b/data/index.md @@ -0,0 +1,5 @@ +NGFF Data Resources +=================== + +* A +* B diff --git a/index.md b/index.rst similarity index 62% rename from index.md rename to index.rst index 2bafe936..d281cb56 100644 --- a/index.md +++ b/index.rst @@ -3,18 +3,12 @@ You can adapt this file completely to your liking, but it should at least contain the root `toctree` directive. -Welcome to NGFF's documentation! -================================ +Next-generation file formats (NGFF) +=================================== .. toctree:: :maxdepth: 2 :caption: Contents: - - -Indices and tables -================== - -* :ref:`genindex` -* :ref:`modindex` -* :ref:`search` + data/index + tools/index diff --git a/tools/index.html b/tools/index.html deleted file mode 100644 index ec7101c1..00000000 --- a/tools/index.html +++ /dev/null @@ -1,6 +0,0 @@ - - - - -

If you are not redirected, click here.

- diff --git a/tools/index.md b/tools/index.md new file mode 100644 index 00000000..77c10657 --- /dev/null +++ b/tools/index.md @@ -0,0 +1,5 @@ +NGFF Tools +========== + +* A +* B From d574f2e2e8fc87db9e9f696f701d289c3372e879 Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Tue, 14 Mar 2023 09:49:21 +0100 Subject: [PATCH 046/102] move to sphinx book theme --- conf.py | 2 +- requirements.txt | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/conf.py b/conf.py index cecd49b2..7a53693d 100644 --- a/conf.py +++ b/conf.py @@ -24,7 +24,7 @@ # -- Options for HTML output ------------------------------------------------- # https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output -html_theme = 'alabaster' +html_theme = 'sphinx_book_theme' html_static_path = ['_static'] diff --git a/requirements.txt b/requirements.txt index 73e17e38..e43d0c22 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,2 +1,3 @@ bikeshed myst-parser +sphinx-book-theme From f9d931934e605b6a81460e024f93e3031cb3cb45 Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Tue, 14 Mar 2023 09:55:52 +0100 Subject: [PATCH 047/102] Add other top-levels and redirect --- .gitignore | 5 ++++- about/index.md | 5 +++++ index.html | 6 ------ index.rst | 9 +++++++++ publications/index.md | 5 +++++ specifications/index.md | 5 +++++ 6 files changed, 28 insertions(+), 7 deletions(-) create mode 100644 about/index.md delete mode 100644 index.html create mode 100644 publications/index.md create mode 100644 specifications/index.md diff --git a/.gitignore b/.gitignore index 53de2396..77cf68a2 100644 --- a/.gitignore +++ b/.gitignore @@ -1,2 +1,5 @@ */.DS_Store -**/__pycache__/* \ No newline at end of file +**/__pycache__/* +*/index.html +_build +.tox diff --git a/about/index.md b/about/index.md new file mode 100644 index 00000000..77c10657 --- /dev/null +++ b/about/index.md @@ -0,0 +1,5 @@ +NGFF Tools +========== + +* A +* B diff --git a/index.html b/index.html deleted file mode 100644 index 74f803c4..00000000 --- a/index.html +++ /dev/null @@ -1,6 +0,0 @@ - - - - -

If you are not redirected in five seconds, click here.

- diff --git a/index.rst b/index.rst index d281cb56..d82817cd 100644 --- a/index.rst +++ b/index.rst @@ -10,5 +10,14 @@ Next-generation file formats (NGFF) :maxdepth: 2 :caption: Contents: + about/index data/index + publications/index + specifications/index tools/index + +.. raw:: html + + diff --git a/publications/index.md b/publications/index.md new file mode 100644 index 00000000..77c10657 --- /dev/null +++ b/publications/index.md @@ -0,0 +1,5 @@ +NGFF Tools +========== + +* A +* B diff --git a/specifications/index.md b/specifications/index.md new file mode 100644 index 00000000..77c10657 --- /dev/null +++ b/specifications/index.md @@ -0,0 +1,5 @@ +NGFF Tools +========== + +* A +* B From 8453ce0b4b3805f212407058f2bb4fc95d1cc663 Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Tue, 14 Mar 2023 09:59:55 +0100 Subject: [PATCH 048/102] fix latest landing page --- conf.py | 2 +- index.rst | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/conf.py b/conf.py index 7a53693d..81405d9c 100644 --- a/conf.py +++ b/conf.py @@ -35,7 +35,7 @@ import shutil -for index_file in glob.glob("[0-9]*/index.bs"): +for index_file in ["latest/index.bs"] + glob.glob("[0-9]*/index.bs"): output_file = index_file.replace("bs", "html") subprocess.check_call(f"bikeshed spec {index_file} {output_file}", shell=True) diff --git a/index.rst b/index.rst index d82817cd..b82d8a2d 100644 --- a/index.rst +++ b/index.rst @@ -19,5 +19,5 @@ Next-generation file formats (NGFF) .. raw:: html From d3d780c8aee8dbeb00b61fb452be1558efa85390 Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Tue, 14 Mar 2023 10:01:05 +0100 Subject: [PATCH 049/102] also fix latest img tag --- latest/index.bs | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/latest/index.bs b/latest/index.bs index aa5feb95..d0996a9d 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -648,8 +648,7 @@ Projects which support reading and/or writing OME-NGFF data include: -Diagram of related projects +Diagram of related projects All implementations prevent an equivalent representation of a dataset which can be downloaded or uploaded freely. An interactive version of this diagram is available from the [OME2020 Workshop](https://downloads.openmicroscopy.org/presentations/2020/Dundee/Workshops/NGFF/zarr_diagram/). From d16ba80a5c90ce564a562d31dd677fef46022ca7 Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Tue, 14 Mar 2023 10:35:14 +0100 Subject: [PATCH 050/102] Fix headers --- about/index.md | 10 ++++++---- publications/index.md | 10 ++++++---- specifications/index.md | 12 ++++++++---- 3 files changed, 20 insertions(+), 12 deletions(-) diff --git a/about/index.md b/about/index.md index 77c10657..63e535f3 100644 --- a/about/index.md +++ b/about/index.md @@ -1,5 +1,7 @@ -NGFF Tools -========== +About +===== -* A -* B +* Specifications +* Data Resources +* Tools +* Publications diff --git a/publications/index.md b/publications/index.md index 77c10657..083f6e54 100644 --- a/publications/index.md +++ b/publications/index.md @@ -1,5 +1,7 @@ -NGFF Tools -========== +Publications +============ -* A -* B +* OME-NGFF +* OME-Zarr +* vEM +* Talks? Videos diff --git a/specifications/index.md b/specifications/index.md index 77c10657..089e527a 100644 --- a/specifications/index.md +++ b/specifications/index.md @@ -1,5 +1,9 @@ -NGFF Tools -========== +NGFF Specifications +=================== -* A -* B +* [0.1](0.1/) +* [0.2](0.2/) +* [0.3](0.3/) +* [0.4](0.4/) +* [latest](latest/) +* By default https://ngff.openmicroscopy.org will redirect to latest From b0f14680f7365c4e3227a377097995812cefc286 Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Tue, 14 Mar 2023 11:32:25 +0100 Subject: [PATCH 051/102] skip unchanged bikeshed files to speed up build --- conf.py | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/conf.py b/conf.py index 81405d9c..de07654a 100644 --- a/conf.py +++ b/conf.py @@ -38,6 +38,14 @@ for index_file in ["latest/index.bs"] + glob.glob("[0-9]*/index.bs"): output_file = index_file.replace("bs", "html") + + src_time = os.path.getmtime(index_file) + if os.path.exists(output_file): + out_time = os.path.getmtime(output_file) + if src_time < out_time: + print(f"Skipping unchanged {index_file}") + continue + subprocess.check_call(f"bikeshed spec {index_file} {output_file}", shell=True) dir_name = index_file.split("/")[0] From 1af53ec7a263bea36e09049eb8fd14f2c918e155 Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Tue, 14 Mar 2023 12:11:45 +0100 Subject: [PATCH 052/102] Add datatables for tools --- conf.py | 8 ++++++++ tools/index.md | 5 ----- tools/index.rst | 7 +++++++ tools/list.csv | 19 +++++++++++++++++++ 4 files changed, 34 insertions(+), 5 deletions(-) delete mode 100644 tools/index.md create mode 100644 tools/index.rst create mode 100644 tools/list.csv diff --git a/conf.py b/conf.py index de07654a..b71a8334 100644 --- a/conf.py +++ b/conf.py @@ -27,6 +27,14 @@ html_theme = 'sphinx_book_theme' html_static_path = ['_static'] +html_css_files = [ + 'https://cdn.datatables.net/v/dt/dt-1.11.5/datatables.min.css', +] + +html_js_files = [ + 'https://cdn.datatables.net/v/dt/dt-1.11.5/datatables.min.js', + 'main.js', +] # Run bikeshed build import glob diff --git a/tools/index.md b/tools/index.md deleted file mode 100644 index 77c10657..00000000 --- a/tools/index.md +++ /dev/null @@ -1,5 +0,0 @@ -NGFF Tools -========== - -* A -* B diff --git a/tools/index.rst b/tools/index.rst new file mode 100644 index 00000000..a967ae43 --- /dev/null +++ b/tools/index.rst @@ -0,0 +1,7 @@ +Tools +===== + +.. csv-table:: NGFF Tools + :file: list.csv + :header-rows: 1 + :class: datatable diff --git a/tools/list.csv b/tools/list.csv new file mode 100644 index 00000000..7a28259a --- /dev/null +++ b/tools/list.csv @@ -0,0 +1,19 @@ +Tool,Type,First release,OME/Zarr support,GitHub stars,Organization,Language,Other Languages,Usage,Distribution,Image.sc,Additional plugins,Publication +`AGAVE `_,visualization,11-2020,01-2023,23,Allen Cell,C++,Python,GUI,github release,yes,, +`ITKWidgets `_,visualization,07-2022,05-2022,487,Insight Software Consortium,Python,,Jupyter,PyPI,yes,ImJoy, +`MoBIE/BDV `_,visualization,05-2020,01-2021,19,MoBIE,Java,,GUI,Fiji,yes,,https://www.biorxiv.org/content/10.1101/2022.05.27.493763v1 +`napari `_,visualization,09-2018,08-2021,1600,napari,Python,,GUI,"PyPI, conda",yes,napari-ome-zarr, +`neuroglancer `_,visualization,06-2016,06-2020,781,Google,C++,Python,GUI,Web,,, +`Viv `_,visualization,1-2020,11-2020,191,Harvard Medical School,JavaScript,WebGL,GUI,Web,yes,vizarr,https://doi.org/10.1038/s41592-022-01482-7 +`webKnossos `_,visualization,10-2011,05-2022,95,scalable minds,JavaScript,Scala,GUI,Web,yes,,https://doi.org/10.1038/nmeth.4331 +`website-3d-cell-viewer `_,visualization,08-2020,01-2023,1,Allen Cell,JavaScript,WebGL,GUI,Web,,, +`bioformats2raw `_,converter,04-2020,04-2020,47,Glencoe Software,Java,Shell,CLI,"zip, conda, docker",yes,, +`ome-ngff-validator `_,utility,07-2022,07-2022,2,OME,JavaScript,,GUI,Web,yes,, +`NGFF-Converter `_,converter,04-2022,04-2022,9,Glencoe Software,Java,,GUI,"dmg, exe",yes,, +`aicsimageio `_,library,11-2018,02-2023,139,Allen Cell,Python,,Python,PyPi,yes,, +`tensorstore `_,library,04-2020,06-2020,1100,Google,C++,,"CMake, Python",,,, +`Fractal `_,library,9-2022,9-2022,27,Fractal,Python,,"CLI, Python",PyPI,not yet,, +`Bio-Formats `_,library,02-2007,04-2021,315,OME,Java,,Java,Jar,yes,ZarrReader, +`ome-zarr-py `_,library,10-2020,10-2020,77,OME,Python,,Python,PyPI,yes,, +`OMERO `_,server,12-2006,06-2020,170,OME,Java,Python,Server,Zip,yes,omero-web-zarr and ZarReader, +`OMERO.web `_,,,,,,,,,,,, From ff7649f1fe54048b26a691c194f4341702fe10ae Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Tue, 14 Mar 2023 12:43:08 +0100 Subject: [PATCH 053/102] Minimal content for all subpages --- about/index.md | 14 ++++++++++---- data/index.md | 16 ++++++++++++---- index.rst | 1 - specifications/index.md | 19 +++++++++++-------- 4 files changed, 33 insertions(+), 17 deletions(-) diff --git a/about/index.md b/about/index.md index 63e535f3..24e9865b 100644 --- a/about/index.md +++ b/about/index.md @@ -1,7 +1,13 @@ About ===== -* Specifications -* Data Resources -* Tools -* Publications +In addition to the next-generation file format (NGFF) [specifications](../specifications/index.md), +the pages listed below are intended to provide an overview of external resources available +for working with NGFF data. + +The following pages are intended to provide an overview of the available resources in the NGFF space: + +* [Data Resources](../data/index.md): List of data repositories where OME-Zarr data can be found. +* [Tools](../tools/index.rst): GUIs and libraries that can be used for accessing OME-Zarr formatted data. +* [Publications](../publications/index.md): List of publications referencing OME-NGFF or publishing + datasets in OME-Zarr. diff --git a/data/index.md b/data/index.md index 74187369..875fcd50 100644 --- a/data/index.md +++ b/data/index.md @@ -1,5 +1,13 @@ -NGFF Data Resources -=================== +Data Resources +============== -* A -* B +| Catalog | Dashboards / Datasets | Zarr Files | Size | Storage | +|-----------------|-------------------------------------------------------------------------------------------------------------------|------------|-------|---------| +| BIA Samples | | 90 | 200GB | S3-like | +| Glencoe | | TBD | TBD | TBD | +| DANDI | | 3914 | 355TB | AWS | +| EMBL-HD | | 21 | TBD | Minio | +| IDR Samples | | 88 | 3TB | S3-like | +| Neural Dynamics | | 90 | 200TB | AWS | +| Sanger | | 10 | 1TB | S3-like | +| webKnossos | | 69 | 70TB | Mix | diff --git a/index.rst b/index.rst index b82d8a2d..1f7cabab 100644 --- a/index.rst +++ b/index.rst @@ -8,7 +8,6 @@ Next-generation file formats (NGFF) .. toctree:: :maxdepth: 2 - :caption: Contents: about/index data/index diff --git a/specifications/index.md b/specifications/index.md index 089e527a..6fed3084 100644 --- a/specifications/index.md +++ b/specifications/index.md @@ -1,9 +1,12 @@ -NGFF Specifications -=================== +Specifications +============== -* [0.1](0.1/) -* [0.2](0.2/) -* [0.3](0.3/) -* [0.4](0.4/) -* [latest](latest/) -* By default https://ngff.openmicroscopy.org will redirect to latest + + +By default will redirect to latest From 1227dc87eeeee24c79152ca775ac131e9bfb2669 Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Tue, 14 Mar 2023 12:46:12 +0100 Subject: [PATCH 054/102] Add html extra path --- conf.py | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/conf.py b/conf.py index b71a8334..3030ee78 100644 --- a/conf.py +++ b/conf.py @@ -36,6 +36,14 @@ 'main.js', ] +html_extra_path = [ + '0.1', + '0.2', + '0.3', + '0.4', + 'latest', +] + # Run bikeshed build import glob import os From aab0d57fdb37425c5ef93d7df6c1dea95146b501 Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Tue, 14 Mar 2023 13:37:33 +0100 Subject: [PATCH 055/102] Generate bikeshed into a subdirectory --- .gitignore | 1 + conf.py | 44 +++++++++++++++++++++++--------------------- 2 files changed, 24 insertions(+), 21 deletions(-) diff --git a/.gitignore b/.gitignore index 77cf68a2..445e2bf8 100644 --- a/.gitignore +++ b/.gitignore @@ -2,4 +2,5 @@ **/__pycache__/* */index.html _build +_bikeshed .tox diff --git a/conf.py b/conf.py index 3030ee78..9230d272 100644 --- a/conf.py +++ b/conf.py @@ -25,6 +25,7 @@ # https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output html_theme = 'sphinx_book_theme' + html_static_path = ['_static'] html_css_files = [ @@ -37,33 +38,34 @@ ] html_extra_path = [ - '0.1', - '0.2', - '0.3', - '0.4', - 'latest', + '_bikeshed', ] +# #################################### # Run bikeshed build -import glob -import os -import subprocess -import shutil +# #################################### + +def bikeshed(): + import glob + import os + import subprocess -for index_file in ["latest/index.bs"] + glob.glob("[0-9]*/index.bs"): + for index_file in ["latest/index.bs"] + glob.glob("[0-9]*/index.bs"): - output_file = index_file.replace("bs", "html") + output_file = "_bikeshed/" + index_file.replace("bs", "html") + output_dir = os.path.dirname(output_file) + os.makedirs(output_dir, exist_ok=False) - src_time = os.path.getmtime(index_file) - if os.path.exists(output_file): - out_time = os.path.getmtime(output_file) - if src_time < out_time: - print(f"Skipping unchanged {index_file}") - continue + # Give the loop a chance to skip files if no build is needed/requested + if "BIKESHED" not in os.environ and os.path.exists(output_file): + src_time = os.path.getmtime(index_file) + out_time = os.path.getmtime(output_file) + if src_time < out_time: + print(f"Skipping unchanged {index_file}") + continue - subprocess.check_call(f"bikeshed spec {index_file} {output_file}", shell=True) + subprocess.check_call(f"bikeshed spec {index_file} {output_file}", shell=True) - dir_name = index_file.split("/")[0] - os.makedirs(f"_build/{dir_name}", exist_ok=True) - shutil.copyfile(output_file, f"_build/{output_file}") +bikeshed() +del bikeshed From 4f5049ece023d44684381ec2ac9ec47bfb5f841b Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Tue, 14 Mar 2023 13:42:01 +0100 Subject: [PATCH 056/102] Fix `` element for newer bikeshed versions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Apparently a new release of bikeshed is now unhappy with the use of `` and `` is required: ``` $ bikeshed spec "latest/index.bs" "latest/index.out.html" LINE 651:1: Tag wasn't closed at end of file. ✘ Did not generate, due to fatal errors Failed ``` --- 0.1/index.bs | 3 +-- 0.2/index.bs | 3 +-- 0.3/index.bs | 3 +-- 0.4/index.bs | 3 +-- latest/index.bs | 3 +-- 5 files changed, 5 insertions(+), 10 deletions(-) diff --git a/0.1/index.bs b/0.1/index.bs index b60204f1..52a33115 100644 --- a/0.1/index.bs +++ b/0.1/index.bs @@ -501,8 +501,7 @@ Projects which support reading and/or writing OME-NGFF data include: -Diagram of related projects +Diagram of related projects All implementations prevent an equivalent representation of a dataset which can be downloaded or uploaded freely. An interactive version of this diagram is available from the [OME2020 Workshop](https://downloads.openmicroscopy.org/presentations/2020/Dundee/Workshops/NGFF/zarr_diagram/). diff --git a/0.2/index.bs b/0.2/index.bs index 4e25ad2c..487f91a3 100644 --- a/0.2/index.bs +++ b/0.2/index.bs @@ -559,8 +559,7 @@ Projects which support reading and/or writing OME-NGFF data include: -Diagram of related projects +Diagram of related projects All implementations prevent an equivalent representation of a dataset which can be downloaded or uploaded freely. An interactive version of this diagram is available from the [OME2020 Workshop](https://downloads.openmicroscopy.org/presentations/2020/Dundee/Workshops/NGFF/zarr_diagram/). diff --git a/0.3/index.bs b/0.3/index.bs index 84192b65..47bc2dcf 100644 --- a/0.3/index.bs +++ b/0.3/index.bs @@ -569,8 +569,7 @@ Projects which support reading and/or writing OME-NGFF data include: -Diagram of related projects +Diagram of related projects All implementations prevent an equivalent representation of a dataset which can be downloaded or uploaded freely. An interactive version of this diagram is available from the [OME2020 Workshop](https://downloads.openmicroscopy.org/presentations/2020/Dundee/Workshops/NGFF/zarr_diagram/). diff --git a/0.4/index.bs b/0.4/index.bs index d39b2c5f..e2a41769 100644 --- a/0.4/index.bs +++ b/0.4/index.bs @@ -639,8 +639,7 @@ Projects which support reading and/or writing OME-NGFF data include: -Diagram of related projects +Diagram of related projects All implementations prevent an equivalent representation of a dataset which can be downloaded or uploaded freely. An interactive version of this diagram is available from the [OME2020 Workshop](https://downloads.openmicroscopy.org/presentations/2020/Dundee/Workshops/NGFF/zarr_diagram/). diff --git a/latest/index.bs b/latest/index.bs index aa5feb95..d0996a9d 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -648,8 +648,7 @@ Projects which support reading and/or writing OME-NGFF data include: -Diagram of related projects +Diagram of related projects All implementations prevent an equivalent representation of a dataset which can be downloaded or uploaded freely. An interactive version of this diagram is available from the [OME2020 Workshop](https://downloads.openmicroscopy.org/presentations/2020/Dundee/Workshops/NGFF/zarr_diagram/). From f98512fa4a767eb00894ddb3de2772009ad46484 Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Wed, 15 Mar 2023 08:17:28 +0100 Subject: [PATCH 057/102] Add meeting history --- about/index.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/about/index.md b/about/index.md index 24e9865b..5e4fecc3 100644 --- a/about/index.md +++ b/about/index.md @@ -11,3 +11,16 @@ The following pages are intended to provide an overview of the available resourc * [Tools](../tools/index.rst): GUIs and libraries that can be used for accessing OME-Zarr formatted data. * [Publications](../publications/index.md): List of publications referencing OME-NGFF or publishing datasets in OME-Zarr. + +Additionally, notes and recordings of the passt NGFF community calls are available: + +| Call | Date | Presenters | Forum thread | Notes | +|------|------|------------|--------------|-------| +|1| 2020-10-20 | | [image.sc](https://forum.image.sc/t/upcoming-calls-on-next-gen-bioimaging-data-tools-starting-oct-29/43489) | [hackmd](https://hackmd.io/_sftykiGR9mSyUan3l1WmA) | +|2| 2020-12-02 (with I2K)| | [image.sc](https://forum.image.sc/t/next-call-on-next-gen-bioimaging-data-tools-dec-2/45816) | [hackmd](https://hackmd.io/X348vzCaTRSmIpsa3dK-Sg)| +|3| 2021-02-23| |[image.sc](https://forum.image.sc/t/next-call-on-next-gen-bioimaging-data-tools-feb-23/48386)| [hackmd](https://hackmd.io/Ndb5IHRmQn2PCCNBLkG-fQ)| +|4| 2021-06 with Fiji Hackathon| |[image.sc](https://forum.image.sc/t/fiji-hackathon-2021-big-data-days-a/53926)| | | +|5 (0.4/axes)| 2021-09-02| Constantin | [image.sc](https://forum.image.sc/t/next-call-on-next-gen-bioimaging-data-tools-feb-23/48386) | [hackmd](https://hackmd.io/GZ1euZUSRZeqPTJj9WJEtg)| +|6 (0.4/axes)| 2022-01-27| Constantin, Will, Seb| [image.sc](https://forum.image.sc/t/next-call-on-next-gen-bioimaging-data-tools-2022-01-27/60885)| [hackmd](https://hackmd.io/QfiBKHIoTZ-CJSp3q0Wykg)| +|7 (0.5/tables & transforms)| 2022-10-05| Kevin & John| [image.sc](https://forum.image.sc/t/ome-ngff-community-call-transforms-and-tables/71792)| [hackmd](https://hackmd.io/TyfrLiCqRteL0Xfc8HRiOA) | +|8 (Metadata) | 2023-03-15 | Wouter-Michiel | [image.sc](https://forum.image.sc/t/community-call-metadata-in-ome-ngff/77570/10) | [hackmd](https://hackmd.io/BqnK9Wm4QpGYAhYOoaFBQQ) | From ef04a77941991d7f62849d01b9c1970ad38c3c13 Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Fri, 17 Mar 2023 17:43:27 +0100 Subject: [PATCH 058/102] Disable publications --- index.rst | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/index.rst b/index.rst index 1f7cabab..b5cbc035 100644 --- a/index.rst +++ b/index.rst @@ -11,10 +11,13 @@ Next-generation file formats (NGFF) about/index data/index - publications/index specifications/index tools/index +.. + # Temporarily disabled + publications/index + .. raw:: html From 5b837cad252eb448c71094a361e1052cea8eaa16 Mon Sep 17 00:00:00 2001 From: William Moore Date: Fri, 21 Apr 2023 17:02:33 +0100 Subject: [PATCH 070/102] Move Intro text from specification index.bs to about page --- about/index.md | 51 ++++++++++++++++++++++++++++++++++++++++++++++ latest/index.bs | 54 ------------------------------------------------- 2 files changed, 51 insertions(+), 54 deletions(-) diff --git a/about/index.md b/about/index.md index 5e4fecc3..16c25e46 100644 --- a/about/index.md +++ b/about/index.md @@ -1,6 +1,57 @@ About ===== +Bioimaging science is at a crossroads. Currently, the drive to acquire more, +larger, preciser spatial measurements is unfortunately at odds with our ability +to structure and share those measurements with others. During a global pandemic +more than ever, we believe fervently that global, collaborative discovery as +opposed to the post-publication, "data-on-request" mode of operation is the +path forward. Bioimaging data should be shareable via open and commercial cloud +resources without the need to download entire datasets. + +At the moment, that is not the norm. The plethora of data formats produced by +imaging systems are ill-suited to remote sharing. Individual scientists +typically lack the infrastructure they need to host these data themselves. When +they acquire images from elsewhere, time-consuming translations and data +cleaning are needed to interpret findings. Those same costs are multiplied when +gathering data into online repositories where curator time can be the limiting +factor before publication is possible. Without a common effort, each lab or +resource is left building the tools they need and maintaining that +infrastructure often without dedicated funding. + +This document defines a specification for bioimaging data to make it possible +to enable the conversion of proprietary formats into a common, cloud-ready one. +Such next-generation file formats layout data so that individual portions, or +"chunks", of large data are reference-able eliminating the need to download +entire datasets. + + +Why "NGFF"? +----------- + +A short description of what is needed for an imaging format is "a hierarchy +of n-dimensional (dense) arrays with metadata". This combination of features +is certainly provided by `HDF5` +from the [HDF Group](https://www.hdfgroup.org), which a number of +bioimaging formats do use. HDF5 and other larger binary structures, however, +are ill-suited for storage in the cloud where accessing individual chunks +of data by name rather than seeking through a large file is at the heart of +parallelization. + +As a result, a number of formats have been developed more recently which provide +the basic data structure of an HDF5 file, but do so in a more cloud-friendly way. +In the [PyData](https://pydata.org/) community, the Zarr [[zarr]] format was developed +for easily storing collections of [NumPy](https://numpy.org/) arrays. In the +[ImageJ](https://imagej.net/) community, N5 [[n5]] was developed to work around +the limitations of HDF5 ("N5" was originally short for "Not-HDF5"). +Both of these formats permit storing individual chunks of data either locally in +separate files or in cloud-based object stores as separate keys. + +A [current effort](https://zarr-specs.readthedocs.io/en/core-protocol-v3.0-dev/protocol/core/v3.0.html) +is underway to unify the two similar specifications to provide a single binary +specification. The editor's draft will soon be entering a [request for comments (RFC)](https://github.com/zarr-developers/zarr-specs/issues/101) phase with the goal of having a first version early in 2021. As that +process comes to an end, this document will be updated. + In addition to the next-generation file format (NGFF) [specifications](../specifications/index.md), the pages listed below are intended to provide an overview of external resources available for working with NGFF data. diff --git a/latest/index.bs b/latest/index.bs index d0996a9d..23248b5b 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -26,60 +26,6 @@ Status Text: will be provided between numbered versions. Data written with these Status Text: (an "editor's draft") will not necessarily be supported.
-Introduction {#intro} -===================== - -Bioimaging science is at a crossroads. Currently, the drive to acquire more, -larger, preciser spatial measurements is unfortunately at odds with our ability -to structure and share those measurements with others. During a global pandemic -more than ever, we believe fervently that global, collaborative discovery as -opposed to the post-publication, "data-on-request" mode of operation is the -path forward. Bioimaging data should be shareable via open and commercial cloud -resources without the need to download entire datasets. - -At the moment, that is not the norm. The plethora of data formats produced by -imaging systems are ill-suited to remote sharing. Individual scientists -typically lack the infrastructure they need to host these data themselves. When -they acquire images from elsewhere, time-consuming translations and data -cleaning are needed to interpret findings. Those same costs are multiplied when -gathering data into online repositories where curator time can be the limiting -factor before publication is possible. Without a common effort, each lab or -resource is left building the tools they need and maintaining that -infrastructure often without dedicated funding. - -This document defines a specification for bioimaging data to make it possible -to enable the conversion of proprietary formats into a common, cloud-ready one. -Such next-generation file formats layout data so that individual portions, or -"chunks", of large data are reference-able eliminating the need to download -entire datasets. - - -Why "NGFF"? {#why-ngff} -------------------------------------------------------------------------------------------------- - -A short description of what is needed for an imaging format is "a hierarchy -of n-dimensional (dense) arrays with metadata". This combination of features -is certainly provided by HDF5 -from the HDF Group, which a number of -bioimaging formats do use. HDF5 and other larger binary structures, however, -are ill-suited for storage in the cloud where accessing individual chunks -of data by name rather than seeking through a large file is at the heart of -parallelization. - -As a result, a number of formats have been developed more recently which provide -the basic data structure of an HDF5 file, but do so in a more cloud-friendly way. -In the [PyData](https://pydata.org/) community, the Zarr [[zarr]] format was developed -for easily storing collections of [NumPy](https://numpy.org/) arrays. In the -[ImageJ](https://imagej.net/) community, N5 [[n5]] was developed to work around -the limitations of HDF5 ("N5" was originally short for "Not-HDF5"). -Both of these formats permit storing individual chunks of data either locally in -separate files or in cloud-based object stores as separate keys. - -A [current effort](https://zarr-specs.readthedocs.io/en/core-protocol-v3.0-dev/protocol/core/v3.0.html) -is underway to unify the two similar specifications to provide a single binary -specification. The editor's draft will soon be entering a [request for comments (RFC)](https://github.com/zarr-developers/zarr-specs/issues/101) phase with the goal of having a first version early in 2021. As that -process comes to an end, this document will be updated. - OME-NGFF {#ome-ngff} -------------------- From df07964f7dfec8344422b7bacd46e942f48ac37d Mon Sep 17 00:00:00 2001 From: William Moore Date: Fri, 21 Apr 2023 17:03:05 +0100 Subject: [PATCH 071/102] Add intro text to home page --- index.rst | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/index.rst b/index.rst index bb9e0b19..ba720124 100644 --- a/index.rst +++ b/index.rst @@ -6,6 +6,14 @@ Next-generation file formats (NGFF) =================================== +OME-NGFF is an imaging format being developed by the bioimaging community to +address issues of scalability and interoperability. +Please see the :doc:`about` section for an introduction. +The OME-NGFF specification is detailed under :doc:`specifications`. +Various Image viewers and other software for working with NGFF data +are listed on the :doc:`tools` page. +Sample NGFF datasets provided by the community can be found under :doc:`data`. + .. toctree:: :maxdepth: 2 From 2d100a4a5a942eeae0301a605fb7d720fd223672 Mon Sep 17 00:00:00 2001 From: William Moore Date: Wed, 26 Apr 2023 16:10:56 +0100 Subject: [PATCH 072/102] Remove 'will redirect...' sentence --- specifications/index.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/specifications/index.md b/specifications/index.md index 6fed3084..dc7dad15 100644 --- a/specifications/index.md +++ b/specifications/index.md @@ -8,5 +8,3 @@ Specifications
  • 0.4
  • latest
  • - -By default will redirect to latest From 42908b5dafa87714821aa0e5ff3eb0ec42d255e0 Mon Sep 17 00:00:00 2001 From: William Moore Date: Wed, 26 Apr 2023 16:13:32 +0100 Subject: [PATCH 073/102] Try to fix links from home page --- index.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/index.rst b/index.rst index ba720124..60056d44 100644 --- a/index.rst +++ b/index.rst @@ -8,11 +8,11 @@ Next-generation file formats (NGFF) OME-NGFF is an imaging format being developed by the bioimaging community to address issues of scalability and interoperability. -Please see the :doc:`about` section for an introduction. -The OME-NGFF specification is detailed under :doc:`specifications`. +Please see the :doc:`about/index` section for an introduction. +The OME-NGFF specification is detailed under :doc:`specifications/index`. Various Image viewers and other software for working with NGFF data -are listed on the :doc:`tools` page. -Sample NGFF datasets provided by the community can be found under :doc:`data`. +are listed on the :doc:`tools/index` page. +Sample NGFF datasets provided by the community can be found under :doc:`data/index`. .. toctree:: :maxdepth: 2 From 6c7a74d4a63e3604e91eb3189264639be81e85a5 Mon Sep 17 00:00:00 2001 From: William Moore Date: Wed, 26 Apr 2023 16:21:39 +0100 Subject: [PATCH 074/102] Typo fix and links updates --- about/index.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/about/index.md b/about/index.md index 16c25e46..36d20e0c 100644 --- a/about/index.md +++ b/about/index.md @@ -47,10 +47,9 @@ the limitations of HDF5 ("N5" was originally short for "Not-HDF5"). Both of these formats permit storing individual chunks of data either locally in separate files or in cloud-based object stores as separate keys. -A [current effort](https://zarr-specs.readthedocs.io/en/core-protocol-v3.0-dev/protocol/core/v3.0.html) +A [current effort](https://zarr.readthedocs.io/en/stable/spec/v3.html) is underway to unify the two similar specifications to provide a single binary -specification. The editor's draft will soon be entering a [request for comments (RFC)](https://github.com/zarr-developers/zarr-specs/issues/101) phase with the goal of having a first version early in 2021. As that -process comes to an end, this document will be updated. +specification. See this [blog post](https://zarr.dev/blog/zep1-update/) for more information. In addition to the next-generation file format (NGFF) [specifications](../specifications/index.md), the pages listed below are intended to provide an overview of external resources available @@ -63,7 +62,7 @@ The following pages are intended to provide an overview of the available resourc * [Publications](../publications/index.md): List of publications referencing OME-NGFF or publishing datasets in OME-Zarr. -Additionally, notes and recordings of the passt NGFF community calls are available: +Additionally, notes and recordings of the past NGFF community calls are available: | Call | Date | Presenters | Forum thread | Notes | |------|------|------------|--------------|-------| From 973b5916c522c331da0613b1fa14ff1814c751db Mon Sep 17 00:00:00 2001 From: William Moore Date: Wed, 26 Apr 2023 16:30:56 +0100 Subject: [PATCH 075/102] Mention napari-ome-zarr in Tools. Longer intro senetence to improve page layout --- tools/index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tools/index.md b/tools/index.md index d821ee8d..b9709411 100644 --- a/tools/index.md +++ b/tools/index.md @@ -1,6 +1,6 @@ # Tools -A list of tools which support OME-Zarr +A list of tools which support OME-Zarr. These are developed by various members of the NGFF community. ## Visualization Tools @@ -17,7 +17,7 @@ A list of tools which support OME-Zarr ### napari - +The [napari viewer](https://github.com/napari/napari) supports OME-Zarr with the [napari-ome-zarr plugin](https://github.com/ome/napari-ome-zarr) ### Neuroglancer From b48d85fa5ec809084ebca5684f2a3a857f142847 Mon Sep 17 00:00:00 2001 From: William Moore Date: Wed, 26 Apr 2023 16:46:33 +0100 Subject: [PATCH 076/102] Add into sentence and dates to Specifications page --- specifications/index.md | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/specifications/index.md b/specifications/index.md index dc7dad15..a40e0708 100644 --- a/specifications/index.md +++ b/specifications/index.md @@ -1,10 +1,16 @@ Specifications ============== +The current released version of the OME-NGFF specification is 0.4. + +The latest version is still undergoing changes and is not yet released. + +Version history: + From 291d5dad91a682c2858e6c5ef3f5f800891cfce0 Mon Sep 17 00:00:00 2001 From: giovp Date: Thu, 27 Apr 2023 18:05:26 +0200 Subject: [PATCH 077/102] add better description of omero --- latest/index.bs | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/latest/index.bs b/latest/index.bs index d0996a9d..d70b50ca 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -441,6 +441,11 @@ can be found under the "omero" key in the group-level metadata: See https://docs.openmicroscopy.org/omero/5.6.1/developers/Web/WebGateway.html#imgdata for more information. +The "omero" metadata is optional, but if present it MUST contain the field "channels", which is an array of dictionaries describing the channels of the image. +Each dictionary in "channels" MUST contain the field "color", which is a string of 6 hexadecimal digits specifying the color of the channel in RGB format. +Each dictionary in "channels" MUST contain the field "window", which is a dictionary describing the windowing of the channel. +The field "window" MUST contain the fields "min" and "max", which are the minimum and maximum values of the window, respectively. It MUST also contain the fields "start" and "end", which are the start and end values of the window, respectively. + "labels" metadata {#labels-md} ------------------------------ From d5130910195a140f4f9c20d127b484ca2cdab14c Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Fri, 28 Apr 2023 15:32:54 +0200 Subject: [PATCH 078/102] Update about/index.md --- about/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/about/index.md b/about/index.md index 36d20e0c..99713ee8 100644 --- a/about/index.md +++ b/about/index.md @@ -47,7 +47,7 @@ the limitations of HDF5 ("N5" was originally short for "Not-HDF5"). Both of these formats permit storing individual chunks of data either locally in separate files or in cloud-based object stores as separate keys. -A [current effort](https://zarr.readthedocs.io/en/stable/spec/v3.html) +An [updated Zarr version (v3)](https://zarr-specs.readthedocs.io/) is underway to unify the two similar specifications to provide a single binary specification. See this [blog post](https://zarr.dev/blog/zep1-update/) for more information. From 9d7d203bffd5763a2e5fa62fb50299e031e5f188 Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Fri, 5 May 2023 14:45:05 +0200 Subject: [PATCH 079/102] Add zenodo badge --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 6b123172..4684d196 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,5 @@ +[![DOI](https://zenodo.org/badge/313652456.svg)](https://zenodo.org/badge/latestdoi/313652456) + # ome-ngff [Next-generation file format (NGFF) specifications](https://ngff.openmicroscopy.org/latest/) for storing bioimaging data in the cloud. From 5e96b9bfa889101e769b93ecb9baeffb42796f56 Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Fri, 5 May 2023 15:00:51 +0200 Subject: [PATCH 080/102] Add review URL for RTD --- .github/workflows/review.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/review.yml b/.github/workflows/review.yml index 4b478cd2..f3f3883d 100644 --- a/.github/workflows/review.yml +++ b/.github/workflows/review.yml @@ -21,6 +21,7 @@ jobs: issue-number: ${{ github.event.pull_request.number }} body: | #### Automated Review URLs + * [Readthedocs](https://ngff--{{ github.event.pull_request.number }}.org.readthedocs.build/) * [render latest/index.bs](http://api.csswg.org/bikeshed/?url=https://raw.githubusercontent.com/ome/ngff/${{ github.event.pull_request.head.sha }}/latest/index.bs) * [diff latest modified](https://services.w3.org/htmldiff?doc1=https%3A%2F%2Fngff.openmicroscopy.org%2Flatest%2F&doc2=http%3A%2F%2Fapi.csswg.org%2Fbikeshed%2F%3Furl%3Dhttps%3A%2F%2Fraw.githubusercontent.com%2Fome%2Fngff%2F${{ github.event.pull_request.head.sha }}%2Flatest%2Findex.bs) edit-mode: replace From d225f97a79c9bbbb4ab774f2d3795ae8d02b7581 Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Mon, 8 May 2023 11:21:01 +0200 Subject: [PATCH 081/102] Update data table Now matches v4 from biorxiv: https://www.biorxiv.org/content/10.1101/2023.02.17.528834v4 --- data/index.md | 27 ++++++++++++++------------- 1 file changed, 14 insertions(+), 13 deletions(-) diff --git a/data/index.md b/data/index.md index d49a8d39..f4cdcec0 100644 --- a/data/index.md +++ b/data/index.md @@ -1,19 +1,20 @@ Data Resources ============== -| Catalog | Descriptions | Zarr Files | Size | -| ------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------- | ------------ | ------- | -| [BIA Samples](https://bit.ly/bia-ome-ngff-samples) | Hosting provided by EBI | 90 | 200GB | -| [CZB-Zebrahub](https://zebrahub.ds.czbiohub.org/imaging) | Hosting provided by czbiohub | 5 | 1.2TB | -| [Glencoe](https://glencoesoftware.com/ngff) | Hosting provided by Glencoe Software, Inc. | TBD | TBD | -| [DANDI](https://dandiarchive.org/dandiset/000108) ([identifiers.org][dandi2],[github][dandi3]) | Hosting provided by AWS Open Data Program | 3914 | 355TB | -| [EMBL-HD](https://mobie.github.io/specs/ngff.html) | Hosting provided by EMBL | 21 | TBD | -| [IDR Samples](https://idr.github.io/ome-ngff-samples/) | Hosting provided by EBI | 88 | 3TB | -| [Neural Dynamics](https://registry.opendata.aws/allen-nd-open-data/) | Hosting provided by AWS Open Data Program | 90 | 200TB | -| [Sanger](https://www.sanger.ac.uk/project/ome-zarr/) | Hosting provided by Sanger, UK | 10 | 1TB | -| [SpatialData](https://github.com/scverse/spatialdata-notebooks/tree/main/datasets) | Hosting provided by EMBL | 10 | 25GB | -| [webKnossos](https://zarr.webknossos.org) | Hosting provided by scalableminds GmbH | 69 | 70TB | -| [SSBD](https://ssbd.riken.jp/ssbd-ome-ngff-samples) | Hosting provided by SSBD | 12 | 196GB | +| Catalog | Hosting | Zarr Files | Size | +| ------------------------------------------------------------------------ | -----------------------------------------------------| ------------ | -------- | +| [BIA Samples](https://bit.ly/bia-ome-ngff-samples) | EBI | 90 | 200 GB | +| [Cell Painting Gallery](https://github.com/broadinstitute/cellpainting-gallery) | AWS Open Data Program | 136 | 20 TB | +| [CZB-Zebrahub](https://zebrahub.ds.czbiohub.org/imaging) | czbiohub | 5 | 1.2 TB | +| [DANDI](https://dandiarchive.org/dandiset/000108) ([identifiers.org][dandi2],[github][dandi3]) | AWS Open Data Program | 3914 | 355 TB | +| [Glencoe](https://glencoesoftware.com/ngff) | Glencoe Software, Inc. | 8 | 165 GB | +| [IDR Samples](https://idr.github.io/ome-ngff-samples/) | EBI | 88 | 3 TB | +| [MoBIE](https://mobie.github.io/specs/ngff.html) | EMBL-HD | 21 | 2 TB | +| [Neural Dynamics](https://registry.opendata.aws/allen-nd-open-data/) | AWS Open Data Program | 90 | 200 TB | +| [Sanger](https://www.sanger.ac.uk/project/ome-zarr/) | Sanger, UK | 10 | 1 TB | +| [SpatialData](https://github.com/scverse/spatialdata-notebooks/tree/main/datasets) | EMBL-HD | 10 | 25 GB | +| [SSBD](https://ssbd.riken.jp/ssbd-ome-ngff-samples) | SSBD | 12 | 196 GB | +| [webKnossos](https://zarr.webknossos.org) | scalableminds GmbH | 69 | 70 TB | [dandi2]: https://identifiers.org/DANDI:000108 [dandi3]: https://github.com/dandisets/000108 From 06a66bbdc6d2ada3ecd75f46236ef1ad84601a9e Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Mon, 8 May 2023 11:23:44 +0200 Subject: [PATCH 082/102] Include fix for workflow in next PR --- .github/workflows/review.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/review.yml b/.github/workflows/review.yml index f3f3883d..0679ea76 100644 --- a/.github/workflows/review.yml +++ b/.github/workflows/review.yml @@ -21,7 +21,7 @@ jobs: issue-number: ${{ github.event.pull_request.number }} body: | #### Automated Review URLs - * [Readthedocs](https://ngff--{{ github.event.pull_request.number }}.org.readthedocs.build/) + * [Readthedocs](https://ngff--${{ github.event.pull_request.number }}.org.readthedocs.build/) * [render latest/index.bs](http://api.csswg.org/bikeshed/?url=https://raw.githubusercontent.com/ome/ngff/${{ github.event.pull_request.head.sha }}/latest/index.bs) * [diff latest modified](https://services.w3.org/htmldiff?doc1=https%3A%2F%2Fngff.openmicroscopy.org%2Flatest%2F&doc2=http%3A%2F%2Fapi.csswg.org%2Fbikeshed%2F%3Furl%3Dhttps%3A%2F%2Fraw.githubusercontent.com%2Fome%2Fngff%2F${{ github.event.pull_request.head.sha }}%2Flatest%2Findex.bs) edit-mode: replace From f7c605cc4c3e6e6f7072863a7fed4b9f2cc9a22e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?D=C5=BEenan=20Zuki=C4=87?= Date: Wed, 8 Mar 2023 09:47:04 -0500 Subject: [PATCH 083/102] Add ITKIOOMEZarrNGFF to lists of implementations and tools --- latest/index.bs | 3 +++ tools/index.md | 4 ++++ 2 files changed, 7 insertions(+) diff --git a/latest/index.bs b/latest/index.bs index 23248b5b..df53c7fe 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -592,6 +592,9 @@ Projects which support reading and/or writing OME-NGFF data include:
    [vizarr](https://github.com/hms-dbmi/vizarr/)
    A minimal, purely client-side program for viewing Zarr-based images with Viv & ImJoy.
    +
    [ITKIOOMEZarrNGFF](https://github.com/InsightSoftwareConsortium/ITKIOOMEZarrNGFF/)
    +
    ITK IO for images stored in OME-NGFF format.
    + Diagram of related projects diff --git a/tools/index.md b/tools/index.md index 78cfe91d..2fbb92c9 100644 --- a/tools/index.md +++ b/tools/index.md @@ -85,3 +85,7 @@ The [napari viewer](https://github.com/napari/napari) supports OME-Zarr with the ### Nyxus + + +### ITKIOOMEZarrNGFF + From ef4e60549e1250abfc163ec5aeb89cd5f80f45f2 Mon Sep 17 00:00:00 2001 From: giovp Date: Thu, 6 Jul 2023 03:03:33 +0200 Subject: [PATCH 084/102] add omero description to 0.4 and remove from latest --- 0.4/index.bs | 6 ++++++ latest/index.bs | 5 ----- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/0.4/index.bs b/0.4/index.bs index e2a41769..b92f2ddb 100644 --- a/0.4/index.bs +++ b/0.4/index.bs @@ -438,6 +438,12 @@ can be found under the "omero" key in the group-level metadata: See https://docs.openmicroscopy.org/omero/5.6.1/developers/Web/WebGateway.html#imgdata for more information. +The "omero" metadata is optional, but if present it MUST contain the field "channels", which is an array of dictionaries describing the channels of the image. +Each dictionary in "channels" MUST contain the field "color", which is a string of 6 hexadecimal digits specifying the color of the channel in RGB format. +Each dictionary in "channels" MUST contain the field "window", which is a dictionary describing the windowing of the channel. +The field "window" MUST contain the fields "min" and "max", which are the minimum and maximum values of the window, respectively. +It MUST also contain the fields "start" and "end", which are the start and end values of the window, respectively. + "labels" metadata {#labels-md} ------------------------------ diff --git a/latest/index.bs b/latest/index.bs index d70b50ca..d0996a9d 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -441,11 +441,6 @@ can be found under the "omero" key in the group-level metadata: See https://docs.openmicroscopy.org/omero/5.6.1/developers/Web/WebGateway.html#imgdata for more information. -The "omero" metadata is optional, but if present it MUST contain the field "channels", which is an array of dictionaries describing the channels of the image. -Each dictionary in "channels" MUST contain the field "color", which is a string of 6 hexadecimal digits specifying the color of the channel in RGB format. -Each dictionary in "channels" MUST contain the field "window", which is a dictionary describing the windowing of the channel. -The field "window" MUST contain the fields "min" and "max", which are the minimum and maximum values of the window, respectively. It MUST also contain the fields "start" and "end", which are the start and end values of the window, respectively. - "labels" metadata {#labels-md} ------------------------------ From 7e97a30157e942bd839e8e41c1c3d5fd2a737dc8 Mon Sep 17 00:00:00 2001 From: giovp Date: Thu, 6 Jul 2023 03:09:08 +0200 Subject: [PATCH 085/102] remove window and color as required --- latest/schemas/image.schema | 4 ---- 1 file changed, 4 deletions(-) diff --git a/latest/schemas/image.schema b/latest/schemas/image.schema index 0c4f9de9..7e7309f1 100644 --- a/latest/schemas/image.schema +++ b/latest/schemas/image.schema @@ -94,10 +94,6 @@ "type": "boolean" } }, - "required": [ - "window", - "color" - ] } } }, From b17f0e31143dba646cc273bade3bc082234d6759 Mon Sep 17 00:00:00 2001 From: giovp Date: Mon, 10 Jul 2023 15:27:20 +0200 Subject: [PATCH 086/102] fix comme --- latest/schemas/image.schema | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/latest/schemas/image.schema b/latest/schemas/image.schema index 7e7309f1..b181697a 100644 --- a/latest/schemas/image.schema +++ b/latest/schemas/image.schema @@ -78,7 +78,7 @@ "start", "min", "end", - "max" + "max", ] }, "label": { @@ -93,7 +93,7 @@ "active": { "type": "boolean" } - }, + } } } }, From 8f74346d37eec149effd8f5297b7a60951e65797 Mon Sep 17 00:00:00 2001 From: giovp Date: Mon, 10 Jul 2023 15:28:51 +0200 Subject: [PATCH 087/102] remove comme --- latest/schemas/image.schema | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/latest/schemas/image.schema b/latest/schemas/image.schema index b181697a..cf2b477f 100644 --- a/latest/schemas/image.schema +++ b/latest/schemas/image.schema @@ -78,7 +78,7 @@ "start", "min", "end", - "max", + "max" ] }, "label": { From de9a147c424a3ade859edd1d53fe844e1cdb26d9 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Tue, 1 Aug 2023 16:02:19 -0400 Subject: [PATCH 088/102] chore: add .vscode to gitignore --- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index 445e2bf8..0b5844ed 100644 --- a/.gitignore +++ b/.gitignore @@ -4,3 +4,4 @@ _build _bikeshed .tox +.vscode \ No newline at end of file From 99b1bfcd904c2b488951363dc362a6db7035736d Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Tue, 1 Aug 2023 16:04:25 -0400 Subject: [PATCH 089/102] chore: add trailing whitespace --- .gitignore | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.gitignore b/.gitignore index 0b5844ed..89502188 100644 --- a/.gitignore +++ b/.gitignore @@ -4,4 +4,4 @@ _build _bikeshed .tox -.vscode \ No newline at end of file +.vscode From 6c01252a29c19f06e758d2460e15830691d4955a Mon Sep 17 00:00:00 2001 From: William Moore Date: Tue, 15 Aug 2023 20:18:59 +0100 Subject: [PATCH 090/102] Add publications --- publications/index.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/publications/index.md b/publications/index.md index 083f6e54..f3e872fd 100644 --- a/publications/index.md +++ b/publications/index.md @@ -1,7 +1,5 @@ Publications ============ -* OME-NGFF -* OME-Zarr -* vEM -* Talks? Videos +* [OME-NGFF: a next-generation file format for expanding bioimaging data-access strategies](https://www.nature.com/articles/s41592-021-01326-w) 29th November 2021 +* [OME-Zarr: a cloud-optimized bioimaging file format with international community support](https://link.springer.com/article/10.1007/s00418-023-02209-1) 10th July 2023 From d0a71d07bececa647d4f88ceb77284ebf56e3b8c Mon Sep 17 00:00:00 2001 From: Alan Watson Date: Wed, 23 Aug 2023 14:36:26 -0400 Subject: [PATCH 091/102] Update index.md to add stack_to_multiscale_ngff --- tools/index.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/tools/index.md b/tools/index.md index 2fbb92c9..41c280a2 100644 --- a/tools/index.md +++ b/tools/index.md @@ -52,6 +52,9 @@ The [napari viewer](https://github.com/napari/napari) supports OME-Zarr with the ### NGFF-Converter +### stack_to_multiscale_ngff + + ## Libraries From f1af2c2757727f06270e8b59f793116a010a6fd6 Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Fri, 15 Sep 2023 08:51:45 +0200 Subject: [PATCH 092/102] Add publications to the menu Follow on to #212 to enable the publication sub-page in the menu. --- index.rst | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/index.rst b/index.rst index 60056d44..1b099ccb 100644 --- a/index.rst +++ b/index.rst @@ -6,7 +6,7 @@ Next-generation file formats (NGFF) =================================== -OME-NGFF is an imaging format being developed by the bioimaging community to +OME-NGFF is an imaging format specification being developed by the bioimaging community to address issues of scalability and interoperability. Please see the :doc:`about/index` section for an introduction. The OME-NGFF specification is detailed under :doc:`specifications/index`. @@ -19,12 +19,9 @@ Sample NGFF datasets provided by the community can be found under :doc:`data/ind about/index data/index + publications/index specifications/index tools/index -.. - # Temporarily disabled - publications/index - .. raw:: html From a6cb3ca33a58be945168e2379598a7d894a270df Mon Sep 17 00:00:00 2001 From: Tommaso Comparin <3862206+tcompa@users.noreply.github.com> Date: Tue, 17 Oct 2023 13:48:18 +0200 Subject: [PATCH 093/102] Use "may" for `_ARRAY_DIMENSIONS` image attribute in v0.4 (ref #217) --- 0.4/index.bs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/0.4/index.bs b/0.4/index.bs index b92f2ddb..dbbc9f13 100644 --- a/0.4/index.bs +++ b/0.4/index.bs @@ -145,7 +145,7 @@ For this example we assume an image with 5 dimensions and axes called `t,c,z,y,x ├── .zgroup # Each image is a Zarr group, or a folder, of other groups and arrays. ├── .zattrs # Group level attributes are stored in the .zattrs file and include │ # "multiscales" and "omero" (see below). In addition, the group level attributes - │ # must also contain "_ARRAY_DIMENSIONS" if this group directly contains multi-scale arrays. + │ # may also contain "_ARRAY_DIMENSIONS" if this group directly contains multi-scale arrays. │ ├── 0 # Each multiscale level is stored as a separate Zarr array, │ ... # which is a folder containing chunk files which compose the array. From ac4355841c89ab1438dcb84992e59e3a5127b41e Mon Sep 17 00:00:00 2001 From: Tommaso Comparin <3862206+tcompa@users.noreply.github.com> Date: Tue, 17 Oct 2023 13:48:25 +0200 Subject: [PATCH 094/102] Use "may" for `_ARRAY_DIMENSIONS` image attribute in latest (ref #217) --- latest/index.bs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/latest/index.bs b/latest/index.bs index bb42932e..b97a0911 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -93,7 +93,7 @@ For this example we assume an image with 5 dimensions and axes called `t,c,z,y,x ├── .zgroup # Each image is a Zarr group, or a folder, of other groups and arrays. ├── .zattrs # Group level attributes are stored in the .zattrs file and include │ # "multiscales" and "omero" (see below). In addition, the group level attributes - │ # must also contain "_ARRAY_DIMENSIONS" if this group directly contains multi-scale arrays. + │ # may also contain "_ARRAY_DIMENSIONS" if this group directly contains multi-scale arrays. │ ├── 0 # Each multiscale level is stored as a separate Zarr array, │ ... # which is a folder containing chunk files which compose the array. From 3fca4de96356213d9538dcc8bef688385c2bc703 Mon Sep 17 00:00:00 2001 From: Tommaso Comparin <3862206+tcompa@users.noreply.github.com> Date: Tue, 17 Oct 2023 13:51:12 +0200 Subject: [PATCH 095/102] Add reference to xarray compatibility (re: `_ARRAY_DIMENSIONS`, ref #217) --- 0.4/index.bs | 2 +- latest/index.bs | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/0.4/index.bs b/0.4/index.bs index dbbc9f13..a2df9c05 100644 --- a/0.4/index.bs +++ b/0.4/index.bs @@ -145,7 +145,7 @@ For this example we assume an image with 5 dimensions and axes called `t,c,z,y,x ├── .zgroup # Each image is a Zarr group, or a folder, of other groups and arrays. ├── .zattrs # Group level attributes are stored in the .zattrs file and include │ # "multiscales" and "omero" (see below). In addition, the group level attributes - │ # may also contain "_ARRAY_DIMENSIONS" if this group directly contains multi-scale arrays. + │ # may also contain "_ARRAY_DIMENSIONS" for compatibility with xarray if this group directly contains multi-scale arrays. │ ├── 0 # Each multiscale level is stored as a separate Zarr array, │ ... # which is a folder containing chunk files which compose the array. diff --git a/latest/index.bs b/latest/index.bs index b97a0911..8d73c405 100644 --- a/latest/index.bs +++ b/latest/index.bs @@ -93,7 +93,7 @@ For this example we assume an image with 5 dimensions and axes called `t,c,z,y,x ├── .zgroup # Each image is a Zarr group, or a folder, of other groups and arrays. ├── .zattrs # Group level attributes are stored in the .zattrs file and include │ # "multiscales" and "omero" (see below). In addition, the group level attributes - │ # may also contain "_ARRAY_DIMENSIONS" if this group directly contains multi-scale arrays. + │ # may also contain "_ARRAY_DIMENSIONS" for compatibility with xarray if this group directly contains multi-scale arrays. │ ├── 0 # Each multiscale level is stored as a separate Zarr array, │ ... # which is a folder containing chunk files which compose the array. From f9b9bb9581ffc9da44bf7b8da87cd7c130c80a01 Mon Sep 17 00:00:00 2001 From: William Moore Date: Wed, 18 Oct 2023 11:58:38 +0100 Subject: [PATCH 096/102] Add build.os to readthedocs.yml --- .readthedocs.yaml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.readthedocs.yaml b/.readthedocs.yaml index 21e4796c..ee26e22e 100644 --- a/.readthedocs.yaml +++ b/.readthedocs.yaml @@ -1,5 +1,8 @@ version: 2 +build: + os: ubuntu-22.04 + sphinx: configuration: conf.py From b01dcac9ae3826e65772f413691a25f4feddcb49 Mon Sep 17 00:00:00 2001 From: William Moore Date: Wed, 18 Oct 2023 15:52:17 +0100 Subject: [PATCH 097/102] Add build.tools to .readthedocs.yaml --- .readthedocs.yaml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.readthedocs.yaml b/.readthedocs.yaml index ee26e22e..11ce7bdd 100644 --- a/.readthedocs.yaml +++ b/.readthedocs.yaml @@ -2,6 +2,8 @@ version: 2 build: os: ubuntu-22.04 + tools: + python: "3.10" sphinx: configuration: conf.py From b1e7c383fb1cdb60a15cc2c4a32f755df150b8e4 Mon Sep 17 00:00:00 2001 From: Jonas Windhager Date: Fri, 17 Nov 2023 14:58:46 +0100 Subject: [PATCH 098/102] add tools/libraries + descriptions & evaluations --- tools/index.md | 112 ++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 88 insertions(+), 24 deletions(-) diff --git a/tools/index.md b/tools/index.md index 41c280a2..b3b0d5d0 100644 --- a/tools/index.md +++ b/tools/index.md @@ -1,94 +1,158 @@ # Tools -A list of tools which support OME-Zarr. These are developed by various members of the NGFF community. +A list of tools and libraries with OME-Zarr support. These are developed by various members of the OME-NGFF community. If you think your tool/library should be listed here, please [open a pull request](https://github.com/ome/ngff). -## Visualization Tools +In addition to this collection, an evaluation of selected tools/libraries is available on . + + +## Image viewers with OME-Zarr support ### AGAVE +A desktop application for viewing multichannel volume data. + ### ITKWidgets +A Python interface for visualization on the web platform to interactively generate insights into multidimensional images, point sets, and geometry. + ### MoBIE/BDV +A Fiji plugin for exploring and sharing big multi-modal image and associated tabular data. + ### napari -The [napari viewer](https://github.com/napari/napari) supports OME-Zarr with the [napari-ome-zarr plugin](https://github.com/ome/napari-ome-zarr) + + +A fast, interactive, multi-dimensional image viewer for Python that supports OME-Zarr with the [napari-ome-zarr](https://github.com/ome/napari-ome-zarr) plugin. ### Neuroglancer +A WebGL-based viewer for volumetric data. -### Viv + +### Viv (Avivator, Vizarr, Vitessce, ...) - +A WebGL-powered toolkit for interactive visualization of high-resolution, multiplexed bioimaging datasets. + +The viv toolkit is used by the [Avivator](https://avivator.gehlenborglab.org), [Vizarr](https://github.com/hms-dbmi/vizarr) and [Vitessce](http://vitessce.io) image viewers, among others. + -### webKnossos + +### WEBKNOSSOS +An open-source tool for annotating and exploring large 3D image datasets. + -### website-3d-cell-viewer +### AICS Image Viewer +A browser-based volume viewer built with React and WebGL (Three.js). -## Utilities + +## Tools for converting image files to OME-Zarr ### bioformats2raw - -### ome-ngff-validator - +Java application to convert image file formats, including .mrxs, to an intermediate Zarr structure compatible with the OME-NGFF specification. ### NGFF-Converter +A GUI application for conversion of bioimage formats into OME-NGFF (Next-Generation File Format) or OME-TIFF. + + ### stack_to_multiscale_ngff +A tool for converting multi-terabyte stacks of images into a multiscale OME-NGFF Zarr. + + +## Libraries for reading/writing OME-Zarr files + +Various libraries for reading/writing OME-Zarr files in Python were evaluated as part of the BioVisionCenter "Next generation bioimage analysis workflows hackathon" 2023 in Zurich, Switzerland: https://github.com/jwindhager/ome-ngff-readers-writers/. -## Libraries -### aicsimageio +### AICSImageIO +Image Reading, Metadata Conversion, and Image Writing for Microscopy Images in pure Python. -### tensorstore - +### bfio + -### Fractal - +A Python interface to Bioformats using jpype for direct access to the library. ### Bio-Formats +A standalone Java library for reading and writing life sciences image file formats. + + +### ngff-zarr + + +A lean and kind Open Microscopy Environment (OME) Next Generation File Format (NGFF) Zarr implementation. + + +### multiscale-spatial-image + + +Generate a multiscale, chunked, multi-dimensional spatial image data structure that can serialized to OME-NGFF. + ### ome-zarr-py +A Python library for reading and writing multi-resolution images stored as Zarr filesets, according to the OME NGFF spec. -### bfio - +### ITKIOOMEZarrNGFF + -### SpatialData - +An ITK external module for IO of images stored in Zarr-backed OME-NGFF file format. + + +## Other tools and libraries + +### Fractal + + +A framework to process high content screening data at scale and to prepare it for interactive visualization. ### Nyxus +A scalable library for calculating features from intensity-label image data. -### ITKIOOMEZarrNGFF - + +### ome-ngff-validator + + +Web page for validating OME-NGFF files. + + +### SpatialData + + +An open and universal framework for processing spatial omics data. + + +### tensorstore + + +Library for reading and writing large multi-dimensional arrays (e.g., Zarr). From 7e157e02e7634a3719b88034362cd5116e93d92e Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Mon, 15 Jan 2024 12:57:18 +0100 Subject: [PATCH 099/102] Add community calendar --- community/index.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 community/index.md diff --git a/community/index.md b/community/index.md new file mode 100644 index 00000000..d62c92e9 --- /dev/null +++ b/community/index.md @@ -0,0 +1,21 @@ +# Community + +## Office Hours + + + +

    We'll be hosting office hours on Wednesdays, rotating between +APAC/AU/EU- and AMER/AU/EU-friendly times. + +Please join us if you have questions about NGFF and want to learn more about +the OME-Zarr storage format. We'll discuss the specification, the implementations, +what's new in the ecosystem, how you can get involved in the community and +much more.

    + +

    Please see the NGFF community calendar for exact +timings:

    + + + + +Download the .ics file and add it to your calendar so won't miss any of our meetings! From 24df4626e08d9d84c39b1abe4690f2209040d756 Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Mon, 15 Jan 2024 13:11:53 +0100 Subject: [PATCH 100/102] Minor fixes --- community/index.md | 5 +++++ index.rst | 1 + 2 files changed, 6 insertions(+) diff --git a/community/index.md b/community/index.md index d62c92e9..5212f53b 100644 --- a/community/index.md +++ b/community/index.md @@ -2,7 +2,12 @@ ## Office Hours + + +[**Join here!**](https://openmicroscopy-org.zoom.us/j/82526689042?pwd=ZIde7mcvZugQGmJ8Bm9piFo5Tzwdy1.1)

    We'll be hosting office hours on Wednesdays, rotating between APAC/AU/EU- and AMER/AU/EU-friendly times. diff --git a/index.rst b/index.rst index 1b099ccb..ded27a68 100644 --- a/index.rst +++ b/index.rst @@ -18,6 +18,7 @@ Sample NGFF datasets provided by the community can be found under :doc:`data/ind :maxdepth: 2 about/index + community/index data/index publications/index specifications/index From ef5568263918ba7ac504405fb3b05d00f3fed17e Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Mon, 15 Jan 2024 15:19:11 +0100 Subject: [PATCH 101/102] Attempt to fix calendar timezone Apparently the ` + Download the .ics file and add it to your calendar so won't miss any of our meetings! From b09f0cc13a1295f3a82ecf192ff1492f5f2fa5dd Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Mon, 15 Jan 2024 15:26:26 +0100 Subject: [PATCH 102/102] Use ctz=local for replacement --- community/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/community/index.md b/community/index.md index 081695b2..895609d4 100644 --- a/community/index.md +++ b/community/index.md @@ -20,6 +20,6 @@ much more.

    Please see the NGFF community calendar for exact timings:

    - + Download the .ics file and add it to your calendar so won't miss any of our meetings!