Skip to content

Commit

Permalink
tweak the in-mem buffer for more stable throughput
Browse files Browse the repository at this point in the history
  • Loading branch information
msf committed Jul 25, 2024
1 parent edc19bf commit febd30f
Showing 1 changed file with 4 additions and 2 deletions.
6 changes: 4 additions & 2 deletions ingester/mainloop.go
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,10 @@ func (i *ingester) Run(ctx context.Context, startBlockNumber int64, maxCount int
defer close(blockNumbers)

// We buffer the block channel so that RPC requests can be made concurrently with sending blocks to Dune.
// We limit the buffer size to the same number of concurrent requests, so we exert some backpressure.
blocks := make(chan models.RPCBlock, maxBatchSize)
// We limit the buffer size to the k * maxBatchSize,
// so we exert some backpressure.
// but having enough to feed full batches to DuneAPI
blocks := make(chan models.RPCBlock, 2*maxBatchSize)
defer close(blocks)

// Start MaxBatchSize goroutines to consume blocks concurrently
Expand Down

0 comments on commit febd30f

Please sign in to comment.