Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Map ItemBatcher/ItemSelector #294

Open
wants to merge 10 commits into
base: master
Choose a base branch
from

Conversation

agrare
Copy link
Member

@agrare agrare commented Nov 5, 2024

The ItemBatcher provides a way to group large array inputs into larger batches for better performance

You can provide a MaxItemsPerBatch or MaxItemsPerBatchPath which limits the number of items in a batch, as well as a MaxInputBytesPerBatch and MaxInputBytesPerBatchPath which limits the size of the input in bytes.

The output will be a set of {"Items": []} payloads, one for each batch.

Batching Items

A Map State MAY have an "ItemBatcher" field, whose value MUST be a JSON object and is called the ItemBatcher Configuration. The ItemBatcher Configuration causes the interpreter to batch selected items into sub-arrays before passing them to each invocation. The interpreter will limit each sub-array to the maximum number of items specified by the "MaxItemsPerBatch" field, and to the maximum size in bytes specified by the "MaxInputBytesPerBatch" field.

The ItemBatcher Configuration MAY have a "BatchInput" field, whose value MUST be a Payload Template. An ItemBatcher Configuration MAY have a "MaxItemsPerBatch" field, whose value MUST be a positive integer. An ItemBatcher Configuration MAY have a "MaxInputBytesPerBatch" field, whose value MUST be a positive integer.

The default of "ItemBatcher" is the selected item. Put another way, the interpreter will not batch items if no "ItemBatcher" field is provided.

Both the "MaxItemsPerBatch" and "MaxInputBytesPerBatch" can be provided indirectly. A Map State may have "MaxItemsPerBatchPath" and "MaxInputBytesPerBatchPath" fields which MUST be Reference Paths which, when resolved, MUST select fields whose values are positive integers. A Map State MUST NOT include both "MaxItemsPerBatch" and "MaxItemsPerBatchPath" or both "MaxInputBytesPerBatch" and "MaxInputBytesPerBatchPath".

An ItemBatcher Configuration MUST contain at least one of "MaxItemsPerBatch", "MaxItemsPerBatchPath", "MaxInputBytesPerBatch", or "MaxInputBytesPerBatchPath".

https://docs.aws.amazon.com/step-functions/latest/dg/input-output-itembatcher.html
#241

@agrare agrare requested a review from Fryguy as a code owner November 5, 2024 20:06
@agrare agrare force-pushed the map_state_item_batcher branch 4 times, most recently from 43adee0 to a521ed3 Compare November 6, 2024 15:51
@agrare agrare changed the title [WIP] Add Map ItemBatcher Add Map ItemBatcher Nov 6, 2024
@agrare agrare changed the title Add Map ItemBatcher Add Map ItemBatcher/ItemSelector Nov 8, 2024
@agrare agrare mentioned this pull request Nov 8, 2024
7 tasks
@agrare agrare added the enhancement New feature or request label Nov 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants