-
Notifications
You must be signed in to change notification settings - Fork 88
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support unsafe casting in Delta filter #657
base: main
Are you sure you want to change the base?
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #657 +/- ##
=======================================
Coverage 99.92% 99.92%
=======================================
Files 62 62
Lines 2721 2736 +15
=======================================
+ Hits 2719 2734 +15
Misses 2 2
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks Dohyeon! 🙏
Appreciate you looking into this issue
While this is certainly a way forward, think we need to look more carefully at the casting implications for different type combinations and make sure we are handling them sensibly
Should add there can remain overflow issues, which we need to be cognizant of and handle appropriately. Ideally by erroring as opposed to silently writing incorrect data
@@ -67,7 +67,7 @@ def encode(self, buf): | |||
if arr.dtype == bool: | |||
np.not_equal(arr[1:], arr[:-1], out=enc[1:]) | |||
else: | |||
np.subtract(arr[1:], arr[:-1], out=enc[1:]) | |||
np.subtract(arr[1:], arr[:-1], out=enc[1:], casting='unsafe') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure we want to do this generally. Instead would recommend checking whether casting is reasonable for the given types using np.can_cast
or similar. Then either allow the appropriate casting or accept there will be an intermediate with a copy
Should add we need to look at both cases in this branch (bool
and other) and handle each of them appropriately
We should double check decode
as well
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I added overflow tests for floating points.
I left comments on why the integer overflow tests are not included on the test_delta.py
.
More specifically, silent integer overflow is the limitation of Numpy (see numpy/numpy#8987).
I don't have to change Delta's document because the possible silent failure has already been described.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you elaborate on what need to check about decode
?
cd0b0f7
to
0780934
Compare
[Description of PR]
This is a fix of #653.
The change enable unsafe casting in
Delta
filter.In addition, unit tests are added to catch such bug next.
TODO: