You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Predicates are used to filter rows in Snapshot and in TableChanges for change data feed. Currently, predicates are simply passed to the engine if one is specified. However, predicates may reference columns that do not exist in the data the engine is reading. For example, Generated Columns, and Change Data Feed Generated Columns.
While CDF predicates may be used to filter actions during log replay as an optimization, these columns must be dropped in any predicate applied to physical data.
Describe the functionality you are proposing.
Implement a way to transform logical predicates to physical ones. The physical predicate used by the engine that reference columns in the data. The transformed predicate must not be more selective than the original one.
Additional context
No response
The text was updated successfully, but these errors were encountered:
OussamaSaoudi-db
changed the title
Implement expression transform that drops generated and partition columns
Implement expression transform that handles generated columns
Nov 22, 2024
OussamaSaoudi-db
changed the title
Implement expression transform that handles generated columns
Implement expression transform that handles CDF columns
Nov 22, 2024
OussamaSaoudi-db
changed the title
Implement expression transform that handles CDF columns
Implement Data Skipping Filter that handles CDF columns
Nov 22, 2024
OussamaSaoudi-db
changed the title
Implement Data Skipping Filter that handles CDF columns
Implement expression transform that handles CDF columns
Nov 22, 2024
Please describe why this is necessary.
Predicates are used to filter rows in
Snapshot
and inTableChanges
for change data feed. Currently, predicates are simply passed to the engine if one is specified. However, predicates may reference columns that do not exist in the data the engine is reading. For example, Generated Columns, and Change Data Feed Generated Columns.While CDF predicates may be used to filter actions during log replay as an optimization, these columns must be dropped in any predicate applied to physical data.
Describe the functionality you are proposing.
Implement a way to transform logical predicates to physical ones. The physical predicate used by the engine that reference columns in the data. The transformed predicate must not be more selective than the original one.
Additional context
No response
The text was updated successfully, but these errors were encountered: