Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Spark date handling settings can mess up dates in Standardization-Conformance combined job #2189

Open
benedeki opened this issue Apr 16, 2023 · 0 comments
Labels
bug Something isn't working Conformance Conformance Job affected priority: high Critical to the health of the project Standardization Standardization Job affected
Milestone

Comments

@benedeki
Copy link
Collaborator

Describe the bug

While #2175/#2184 solves for the ability to process dates prior to 1900 in Enceladus 3 (without code change), the way how it is done actually introduced a possible bug of messing up the dates.
In case a pair of different settings (LEGACY-CORRECTED) is used for read and write a combined job of Standardization&Conformance will mess up the dates.

To Reproduce

Steps to reproduce the behavior OR commands run:

  1. Have a dataset with timestamps pre 1900 from ambiguous interval
  2. Run Standardization & Conformance job with settings read LEGACY, write CORRECTED
  3. Dates will be messed up

Expected behavior

Data should remain correct

Additional context

Consider adding the information of used date reading standard into the _INFO file.

@benedeki benedeki added bug Something isn't working Conformance Conformance Job affected Standardization Standardization Job affected priority: high Critical to the health of the project labels Apr 16, 2023
@benedeki benedeki added this to the Release 3.1 milestone Apr 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working Conformance Conformance Job affected priority: high Critical to the health of the project Standardization Standardization Job affected
Projects
None yet
Development

No branches or pull requests

1 participant