Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement expression transform that handles CDF columns #525

Open
OussamaSaoudi-db opened this issue Nov 22, 2024 · 0 comments
Open

Implement expression transform that handles CDF columns #525

OussamaSaoudi-db opened this issue Nov 22, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@OussamaSaoudi-db
Copy link
Collaborator

OussamaSaoudi-db commented Nov 22, 2024

Please describe why this is necessary.

Predicates are used to filter rows in Snapshot and in TableChanges for change data feed. Currently, predicates are simply passed to the engine if one is specified. However, predicates may reference columns that do not exist in the data the engine is reading. For example, Generated Columns, and Change Data Feed Generated Columns.

While CDF predicates may be used to filter actions during log replay as an optimization, these columns must be dropped in any predicate applied to physical data.

Describe the functionality you are proposing.

Implement a way to transform logical predicates to physical ones. The physical predicate used by the engine that reference columns in the data. The transformed predicate must not be more selective than the original one.

Additional context

No response

@OussamaSaoudi-db OussamaSaoudi-db added the enhancement New feature or request label Nov 22, 2024
@OussamaSaoudi-db OussamaSaoudi-db changed the title Implement expression transform that drops generated and partition columns Implement expression transform that handles generated columns Nov 22, 2024
@OussamaSaoudi-db OussamaSaoudi-db changed the title Implement expression transform that handles generated columns Implement expression transform that handles CDF columns Nov 22, 2024
@OussamaSaoudi-db OussamaSaoudi-db changed the title Implement expression transform that handles CDF columns Implement Data Skipping Filter that handles CDF columns Nov 22, 2024
@OussamaSaoudi-db OussamaSaoudi-db changed the title Implement Data Skipping Filter that handles CDF columns Implement expression transform that handles CDF columns Nov 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant