Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add test to check crc files are ignored in list_log_files #502

Open
OussamaSaoudi-db opened this issue Nov 15, 2024 · 0 comments
Open

Add test to check crc files are ignored in list_log_files #502

OussamaSaoudi-db opened this issue Nov 15, 2024 · 0 comments
Labels
enhancement New feature or request good first issue Good for newcomers

Comments

@OussamaSaoudi-db
Copy link
Collaborator

Please describe why this is necessary.

The function list_log_files generates ParsedLogPath by parsing file paths in _delta_log. Parsing may fail if there is an unexpected file path. Crc files are commonly written to _delta_log by delta implementations. We must ensure that crc files do not cause log file listing to fail. They must instead be ignored.

Describe the functionality you are proposing.

Add a test to ensure that crc files are ignored. This test must ensure that the following example _delta_log directory does not cause list_log_files to fail:

_delta_log/.00000000000000000000.json.crc
_delta_log/.00000000000000000001.checkpoint.parquet.crc
_delta_log/.00000000000000000001.json.crc
_delta_log/._SUCCESS.crc
_delta_log/._last_checkpoint.crc
_delta_log/00000000000000000000.json
_delta_log/00000000000000000001.checkpoint.parquet
_delta_log/00000000000000000001.json
_delta_log/_SUCCESS
_delta_log/_last_checkpoint

Additional context

You can find the function in question at kernel/src/log_segment.rs. The test file for log segment is in kernel/src/log_segmnt/tests.rs.

@OussamaSaoudi-db OussamaSaoudi-db added enhancement New feature or request good first issue Good for newcomers labels Nov 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good first issue Good for newcomers
Projects
None yet
Development

No branches or pull requests

1 participant