How Should NetCDF Support LZ4 Compression?

[edwardhartnett]

Introduction

From the LZ4 GitHub page (https://github.com/lz4/lz4):

LZ4 is lossless compression algorithm, providing compression speed > 500 MB/s per core, scalable with multi-cores CPU. It features an extremely fast decoder, with speed in multiple GB/s per core, typically reaching RAM speed limits on multi-core systems.

LZ4 is one of the codecs used by BLOSC. It is also highly favored in the Zarr community.

Community Codec Repository

I tried to support LZ4 in the CCR project (https://github.com/ccr/ccr). But while it appeared to work fine, it actually made the data file bigger, not smaller. I am assured that the LZ4 library works well, so this was either a problem with the HDF5 filter, or with how I used it.

Next Steps

LZ4 needs to be working in CCR, to demonstrate that it works and so that it can be benchmarked.

[DennisHeimbigner]

I see LZ4 has a dual license (only worse). be careful.

[edwardhartnett]

Worth noting, but as with the other compression libraries, we will never be providing the library - just the glue code and the HDF5 plugin (which has an open source license). So license considerations of lz4 are entirely up to the user to sort out. (And the same is true of szip, which also has a dual license.)

[dopplershift]

@DennisHeimbigner Careful, the license says:

This repository uses 2 different licenses :
- all files in the `lib` directory use a BSD 2-Clause license
- all other files use a GPLv2 license, unless explicitly stated otherwise

So it should be BSD-2-clause for the relevant library parts.

@edwardhartnett I promise I'm not trying to be antagonistic 😉 Having said that, I vehemently disagree:

1. Unidata ships pre-compiled binaries for Windows
2. It would be irresponsible to promote a feature which has problematic licensing. That would only serve to fracture the netCDF user community between those who can use it and those who can't. It would literally be repeating the problems of szip in the past.

[edwardhartnett]

These licence considerations are not going to stop users or the CCR project from using LZ4. These are lawyers' quibbles.

The software is licensed for free use. That is all we need it to be. It is already freely available. It is used extensively by Zarr, which is used by a lot of FOSS projects and science institutions. Szip has a much more restrictive license and it has been supported since netcdf-c-4.0.

I fail to see the real problem here.

[WardF]

I don't see the problem either, insofar as we are obliged to worry about these considerations so that our downstream users don't need to.

I cannot for a moment imagine simply saying 'to hell with the license, we're not even going to worry about it', from a mechanical, ethical, or practical standpoint :). It would paint us in a pretty sloppy light, and I can only imagine what the impact would be if we were to somehow lose the goodwill or trust of projects for whom this is, in fact, a dealbreaker. Our commercial users might not be our primary community, but it is no small thing that netCDF has become so ubiquitous throughout the open source and commercial scientific software strata. It would be a shame to waste all the effort people have put in (including your own, @edwardhartnett!) through a moment of laxity.

I understand it's frustrating to have to worry about these concerns; I would like to dispense with a lot of the "project management"-adjacent tasks I spend my time on instead of writing code. But it comes part-and-parcel with the job, I suppose, so we may as well make a good job of it.

In a nutshell, I can see the destination I think we are most likely to end up at; netCDF will not distribute binaries that will fundamentally change the license or encumber our community with dual/changing licenses. So the Windows distribution will not include bundled binaries that do so. We will distribute the bundles we can, and provide instructions for compiling the others. That's the logical solution here, I think, and will not preclude supporting these compressors in the code itself, when the appropriate binaries are available.

Despite this seeming like the obvious solution to me, however, diligence and professional courtesy demand the conversation :).

[edwardhartnett]

No one suggested we say "to hell with licenses."

However the package is already licenced for free use, and is widely used. So our requirements are met. We do not have to be more strict than the license allows - that's the point of the license.

[dopplershift]

I agree that the license is lz4 is fine, so we're good there--at least for the lz4 licensing.

For future cases, it's not as simple as you'd like to make it. GPL is "free to use" but has ramifications far beyond that.

[JohnLCaron]

Apparently theres a pure java port at

https://github.com/airlift/aircompressor

maybe we can use. Apache 2 license,

This https://stackoverflow.com/questions/60974402/lz4-and-zstd-for-java says "though with less features". Not sure what that means. what features does your implementation use?

[DennisHeimbigner]

Ed - you are aware of the viral nature of GPL2 and 3, right?
LGPL presents fewer problems.