Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Supports streams of events instead of large JSON documents to make select/generate more efficient #112

Open
flosell opened this issue Jul 12, 2020 · 0 comments
Labels
enhancement generate-feature Issues regarding the generation of IAM Policies from Events performance select-feature Issues regarding the selection of relevant events

Comments

@flosell
Copy link
Owner

flosell commented Jul 12, 2020

At the moment, trailscraper select produces a full JSON document with the same format as it would appear in a normal CloudTrail JSON file. trailscraper generate reads this format.

This is very flexible as it composes well with other tools that might be used to deal with this kind of data. However, it takes a while longer (and possibly generates more overhead) since each step needs to completely finish their operation before handing it off the the next step.
If we'd output the selected events one by one as we found them, generate could already get to work on them.

Next step:

  1. Prototype if this would actually make a significant improvement to decide if it's worth complicating the usage over this
@flosell flosell added enhancement generate-feature Issues regarding the generation of IAM Policies from Events performance select-feature Issues regarding the selection of relevant events labels Jul 12, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement generate-feature Issues regarding the generation of IAM Policies from Events performance select-feature Issues regarding the selection of relevant events
Projects
None yet
Development

No branches or pull requests

1 participant