kafka source topic/partition discovery in vector

[canopenerda]

Hi folks,

I'm aware that kafka source support regex for topics and I would like to know how new topics matching the regex are discovered, in what interval, and how vector react to rebalance when partition number is changed for topics.

Thanks!

[pront]

Hi @canopenerda,

The Kafka consumer subscribes to the specified topics via the consumer.subscribe(&topics) call. This is done on startup. We basically delegate to rdkafka::StreamConsumer.

Re-subscription happens as part of the rebalancing process, so the consumer can subscribe to topics that are different from those originally specified in the configuration if the regex matches new or updated topics in the Kafka cluster.

[canopenerda]

Thanks, @pront ! As per your comments, the subscription to topics is done on startup, will vector pick up new topics that match regex and emerge after vector started automatically without a restart?

[pront]

Yes, I believe it's possible when using regexes.