Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase batch size when streaming client audit logs #948

Open
matthew-white opened this issue Aug 4, 2023 · 1 comment
Open

Increase batch size when streaming client audit logs #948

matthew-white opened this issue Aug 4, 2023 · 1 comment
Labels
blocked Blocked by other work performance Performance, benchmarking

Comments

@matthew-white
Copy link
Member

The client_audits table can get pretty big. One form that I've encountered had 16K submissions, but 1.2 million client_audits rows. It took 2 minutes to download the submissions.csv.zip for this form, most of which was spent streaming from client_audits (via ClientAudits.streamForExport()). The resulting submissions.csv.zip was 15 MB, and the combined client audit log CSV file was 105 MB unzipped.

It'd be nice to improve the performance of this streaming. It seems possible that a form could have even more client_audits rows. One idea I have for improving the performance is increasing the batch size when streaming from Postgres. Configuring the batch size isn't documented in the Slonik readme, but there was a PR for it here: gajus/slonik#322. However, the PR was released with 27.1.0, which is ahead of the version of Slonik that we're on.

@matthew-white matthew-white added the blocked Blocked by other work label Aug 4, 2023
@matthew-white
Copy link
Member Author

matthew-white commented Mar 15, 2024

Increasing the batch size is blocked by #1119.

@matthew-white matthew-white added the performance Performance, benchmarking label Nov 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
blocked Blocked by other work performance Performance, benchmarking
Projects
None yet
Development

No branches or pull requests

1 participant