Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support unsafe casting in Delta filter #657

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

ehgus
Copy link
Contributor

@ehgus ehgus commented Nov 24, 2024

[Description of PR]

This is a fix of #653.
The change enable unsafe casting in Delta filter.
In addition, unit tests are added to catch such bug next.

TODO:

  • Unit tests and/or doctests in docstrings
  • Tests pass locally
  • Docstrings and API docs for any new/modified user-facing classes and functions
  • Changes documented in docs/release.rst
  • Docs build locally
  • GitHub Actions CI passes
  • Test coverage to 100% (Codecov passes)

Copy link

codecov bot commented Nov 24, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.92%. Comparing base (86f5ce4) to head (81a744d).
Report is 5 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #657   +/-   ##
=======================================
  Coverage   99.92%   99.92%           
=======================================
  Files          62       62           
  Lines        2721     2736   +15     
=======================================
+ Hits         2719     2734   +15     
  Misses          2        2           
Files with missing lines Coverage Δ
numcodecs/delta.py 100.00% <100.00%> (ø)
numcodecs/tests/test_delta.py 100.00% <100.00%> (ø)

Copy link
Member

@jakirkham jakirkham left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Dohyeon! 🙏

Appreciate you looking into this issue

While this is certainly a way forward, think we need to look more carefully at the casting implications for different type combinations and make sure we are handling them sensibly

Should add there can remain overflow issues, which we need to be cognizant of and handle appropriately. Ideally by erroring as opposed to silently writing incorrect data

@@ -67,7 +67,7 @@ def encode(self, buf):
if arr.dtype == bool:
np.not_equal(arr[1:], arr[:-1], out=enc[1:])
else:
np.subtract(arr[1:], arr[:-1], out=enc[1:])
np.subtract(arr[1:], arr[:-1], out=enc[1:], casting='unsafe')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure we want to do this generally. Instead would recommend checking whether casting is reasonable for the given types using np.can_cast or similar. Then either allow the appropriate casting or accept there will be an intermediate with a copy

Should add we need to look at both cases in this branch (bool and other) and handle each of them appropriately

We should double check decode as well

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added overflow tests for floating points.
I left comments on why the integer overflow tests are not included on the test_delta.py.
More specifically, silent integer overflow is the limitation of Numpy (see numpy/numpy#8987).
I don't have to change Delta's document because the possible silent failure has already been described.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you elaborate on what need to check about decode?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants