Spark date handling settings can mess up dates in Standardization-Conformance combined job #2189
Labels
bug
Something isn't working
Conformance
Conformance Job affected
priority: high
Critical to the health of the project
Standardization
Standardization Job affected
Milestone
Describe the bug
While #2175/#2184 solves for the ability to process dates prior to 1900 in Enceladus 3 (without code change), the way how it is done actually introduced a possible bug of messing up the dates.
In case a pair of different settings (LEGACY-CORRECTED) is used for read and write a combined job of Standardization&Conformance will mess up the dates.
To Reproduce
Steps to reproduce the behavior OR commands run:
Expected behavior
Data should remain correct
Additional context
Consider adding the information of used date reading standard into the _INFO file.
The text was updated successfully, but these errors were encountered: