Native Released to System Repositories/Stores

[cmhulbert]

Native releases should be available in platform specific ways:L

• Windows Store?
• Apple Store?
• Debian PPA

Needs dependency management:

• libblosc1

[mkitti]

I understand that @hubbardp has an active Apple developer key

We may qualify for a fee waiver for an Apple membership fee waiver since HHMI is a non-profit. I have not been able to clarify if applying for the waiver would violate a HHMi policy or not.
https://developer.apple.com/support/membership-fee-waiver/

[mkitti]

Blosc1 availability summary

The latest release of c-blosc is 1.21.4. Importantly, this upgraded Zstd from 1.5.2 to 1.5.5 due to a known rare compression bug in Zstandard pre-1.5.5.

Ubuntu LTS and Debian stable contain out dated versions of libblosc and zstandard. The latest release of libblosc1 in Debian unstable is 1.21.3+ds-1.

Homebrew and MacPorts have c-blosc 1.21.4. The dependency graph is not clear to me at the moment. Homebrew and MacPorts also ship an up-to-date zstandard 1.5.5

I have not found a "native" Windows based distribution of a modular built blosc on either Chocholatey, win-get, or the Microsoft Store. conda-forge contains an update to date recipe of blosc 1.21.4.

conda-forge contains blosc 1.21.4 for all platforms.
https://github.com/conda-forge/blosc-feedstock/blob/main/recipe/meta.yaml
https://github.com/conda-forge/blosc-feedstock/blob/main/recipe/bld.bat
https://github.com/conda-forge/blosc-feedstock/blob/main/recipe/build.sh

Ubuntu libblosc1 version 1.21.1 in Ubuntu 22.04 LTS

https://packages.ubuntu.com/jammy/libblosc1
liblz4-1 (>= 0.0~r130)
libsnappy1v5 (>= 1.1.7) [not riscv64]
libzstd1 (>= 1.3.2)
zlib1g (>= 1:1.1.4)

Debian libblosc1 version 1.21.3 in Debian bookworm stable

https://packages.debian.org/bookworm/libblosc1
liblz4-1 (>= 0.0~r130)
libsnappy1v5 (>= 1.1.9)
libzstd1 (>= 1.5.2)
dep: zlib1g (>= 1:1.1.4)

Homebrew c-blosc 1.21.4

https://formulae.brew.sh/formula/c-blosc

MacPorts 1.21.4

https://ports.macports.org/port/blosc/

Chocholatey

https://community.chocolatey.org/packages?q=blosc
(none exists)

win-get

https://winget.run/search?query=blosc
(none exists)

conda-forge

https://anaconda.org/conda-forge/blosc
osx-arm64 1.21.4
osx-arm64 blosc 1.21.4
linux-64 1.21.4
linux-aarch64 1.21.4
osx-64 1.21.4 (Intel)
win-64 1.21.4

[mkitti]

For Windows, I recommend obtaining the DLLs in this fashion via a conda enabled PowerShell:

conda create -n blosc_env -c conda-forge blosc
conda activate blosc_env
cd $ENV:CONDA_PREFIX\Library\bin
dir

Copy and paste all the DLLs into the Paintera bin directory. This is the Windows way.

    Directory: C:\conda\3\envs\blosc_env\Library\bin


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----          5/6/2022     11:21          21984 api-ms-win-core-console-l1-1-0.dll
-a----          5/6/2022     11:29          21976 api-ms-win-core-console-l1-2-0.dll
-a----          5/6/2022     11:21          21984 api-ms-win-core-datetime-l1-1-0.dll
-a----          5/6/2022     11:21          21984 api-ms-win-core-debug-l1-1-0.dll
-a----          5/6/2022     11:21          21976 api-ms-win-core-errorhandling-l1-1-0.dll
-a----          5/6/2022     11:21          21992 api-ms-win-core-fibers-l1-1-0.dll
-a----          5/6/2022     11:21          26088 api-ms-win-core-file-l1-1-0.dll
-a----          5/6/2022     11:21          21976 api-ms-win-core-file-l1-2-0.dll
-a----          5/6/2022     11:22          21976 api-ms-win-core-file-l2-1-0.dll
-a----          5/6/2022     11:21          21984 api-ms-win-core-handle-l1-1-0.dll
-a----          5/6/2022     11:21          21984 api-ms-win-core-heap-l1-1-0.dll
-a----          5/6/2022     11:21          21984 api-ms-win-core-interlocked-l1-1-0.dll
-a----          5/6/2022     11:22          22008 api-ms-win-core-libraryloader-l1-1-0.dll
-a----          5/6/2022     11:21          21984 api-ms-win-core-localization-l1-2-0.dll
-a----          5/6/2022     11:21          21976 api-ms-win-core-memory-l1-1-0.dll
-a----          5/6/2022     11:21          21984 api-ms-win-core-namedpipe-l1-1-0.dll
-a----          5/6/2022     11:21          22008 api-ms-win-core-processenvironment-l1-1-0.dll
-a----          5/6/2022     11:21          21976 api-ms-win-core-processthreads-l1-1-0.dll
-a----          5/6/2022     11:21          21976 api-ms-win-core-processthreads-l1-1-1.dll
-a----          5/6/2022     11:21          21992 api-ms-win-core-profile-l1-1-0.dll
-a----          5/6/2022     11:21          21984 api-ms-win-core-rtlsupport-l1-1-0.dll
-a----          5/6/2022     11:21          21976 api-ms-win-core-string-l1-1-0.dll
-a----          5/6/2022     11:21          21976 api-ms-win-core-synch-l1-1-0.dll
-a----          5/6/2022     11:22          21976 api-ms-win-core-synch-l1-2-0.dll
-a----          5/6/2022     11:21          21984 api-ms-win-core-sysinfo-l1-1-0.dll
-a----          5/6/2022     11:21          21976 api-ms-win-core-timezone-l1-1-0.dll
-a----          5/6/2022     11:21          21992 api-ms-win-core-util-l1-1-0.dll
-a----          5/6/2022     11:21          21984 api-ms-win-crt-conio-l1-1-0.dll
-a----          5/6/2022     11:21          26080 api-ms-win-crt-convert-l1-1-0.dll
-a----          5/6/2022     11:21          21984 api-ms-win-crt-environment-l1-1-0.dll
-a----          5/6/2022     11:21          21984 api-ms-win-crt-filesystem-l1-1-0.dll
-a----          5/6/2022     11:21          21984 api-ms-win-crt-heap-l1-1-0.dll
-a----          5/6/2022     11:21          21984 api-ms-win-crt-locale-l1-1-0.dll
-a----          5/6/2022     11:21          30184 api-ms-win-crt-math-l1-1-0.dll
-a----          5/6/2022     11:21          30176 api-ms-win-crt-multibyte-l1-1-0.dll
-a----          5/6/2022     11:21          75232 api-ms-win-crt-private-l1-1-0.dll
-a----          5/6/2022     11:21          21984 api-ms-win-crt-process-l1-1-0.dll
-a----          5/6/2022     11:21          26104 api-ms-win-crt-runtime-l1-1-0.dll
-a----          5/6/2022     11:21          26080 api-ms-win-crt-stdio-l1-1-0.dll
-a----          5/6/2022     11:22          26072 api-ms-win-crt-string-l1-1-0.dll
-a----          5/6/2022     11:21          21984 api-ms-win-crt-time-l1-1-0.dll
-a----          5/6/2022     11:21          22000 api-ms-win-crt-utility-l1-1-0.dll
-a----         5/16/2023     07:09          69632 blosc.dll
-a----         5/10/2023     03:01         327576 concrt140.dll
-a----         1/26/2023     05:00         118784 liblz4.dll
-a----          7/7/2023     05:26         525312 libzstd.dll
-a----         1/26/2023     05:01         161792 lz4.exe
-a----         5/10/2023     03:01         578384 msvcp140.dll
-a----         5/10/2023     03:01          35704 msvcp140_1.dll
-a----         5/10/2023     03:01         267160 msvcp140_2.dll
-a----         5/10/2023     03:01          50072 msvcp140_atomic_wait.dll
-a----         5/10/2023     03:01          31640 msvcp140_codecvt_ids.dll
-a----         3/11/2023     06:39          75264 snappy.dll
-a----          5/6/2022     11:22        1123808 ucrtbase.dll
-a----         5/10/2023     03:01         414104 vcamp140.dll
-a----         5/10/2023     03:01         346008 vccorlib140.dll
-a----         5/10/2023     03:01         191864 vcomp140.dll
-a----         5/10/2023     03:01         109440 vcruntime140.dll
-a----         5/10/2023     03:01          49560 vcruntime140_1.dll
-a----         6/12/2023     09:09          89088 zlib.dll
-a----          7/7/2023     05:26         525312 zstd.dll
-a----          7/7/2023     05:26         129024 zstd.exe

[mkitti]

Wider Linux Distribution Scan for blosc packages:
https://pkgs.org/search/?q=blosc
https://pkgs.org/search/?q=libblosc1
https://pkgs.org/download/blosc
https://pkgs.org/download/c-blosc

[mkitti]

I was also finally note that we really need to move to Blosc2 for better upstream support from the authors as well as the second generation frame format. Blosc1 is becoming increasingly difficult to support.

[cmhulbert]

Thanks @mkitti for the info. I think what we ideally will use is the platform specific standard auto-updating sites, to make using Paintera as straight-forward from the end user. For example, if we package Paintera on windows, in a way that requires we get the dlls from conda first, It would still require that the user has conda installed (unless we can have it as a dependency?). Perhaps we can do this at package time on the windows runner, and just provide the dlls to the installer?

As for blosc2, I understand it's backwards compatible; Is it easier to package and move to blosc2 say, than maintain blosc1?

[mkitti]

No, the user would not need conda. We would just use conda to obtain the binaries and include them in the Windows package. The expectation on Windows is for us to vendor such dependencies into the package.

[cmhulbert]

Thanks @mkitti! We had a productive chat to discuss this more, a brief summary as follows:

• Since we have an ubuntu PPA, and ubuntu provides blosc dependency at a platform level, we should prioritize releasing on Ubuntu first
  • Should also consider Debian, since it likely will not be difficult after this
• Apple Store is probably the best target for Macs
  • Needs signing keys at least. Not clear if this is best to sign by individual, lab, or as HHMI. HHMI may already have signing keys, and this may be broadly more useful to try and get, even if there currently are not.
  • Blosc is available through homebrew and macports, but if we are publishing to Apple Store, it may be best to package the libraries with Paintera
• Windows target is not clear at the moment
  • It's likely that windows store is ideal, but I'm not familiar enough at the moment
  • Also we will likely need to package the blosc library with Paintera

So next steps as far as I have is to generate the deb package with the blosc dependency, and release to our ppa. That should be fairly straight forward. After, can discuss more specifics around Apple/Windows stores. In the meantime, I can also work on publishing the SNAPSHOT releases on github like we currently are, but with the bundled bloscl library. Likely the best way to do this is as @mkitti mentioned above, by pulling them from the conda recipes in the github action runner, and package them.

Some other notes, we will want to setup Paintera to publish SNAPSHOT releases on an M1 mac runner as welll.

[cmhulbert]

Just an update for now, but the deb package built and released here ubuntu-latest should now depend on libblosc1