Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat request: forwarding records unchanged to a producer client topic #844

Open
Nesze opened this issue Oct 18, 2024 · 1 comment
Open
Labels
enhancement New feature or request

Comments

@Nesze
Copy link
Contributor

Nesze commented Oct 18, 2024

We have many use cases where we consume a record from topic A and forward it unchanged to topic B (in some cases even to separate clusters), essentially relaying it.

This means in our handlers we need to set the consumed record topic to the actual target topic first, can't just blindly forward, e.g.

func(ctx context.Context, r kgo.Record) error {
    r.Topic = produceTopic
    return producer.Produce(ctx, r)
}

It would be handy to be able to configure the producer client to always produce to a given topic. Making r.Topic = produceTopic redundant.

Also, while the docs are clear about this, for newcomers like us, preferring the record topic over the value provided by kgo.DefaultProduceTopic resulted in forwarding the records back to the original topic producing lots of duplicates : ) when switching to this pkg. Therefore in-house we're considering adding an option like AlwaysUseDefaultProduceTopic for our kgo.Client wrappers to guard us against this in the future.

Has anyone had similar issues? And if yes, how did you handle it? Did you build anything specific?
Is there any appetite for adding a similar option to this pkg to tweak the final target topic?

@twmb twmb added the enhancement New feature or request label Oct 28, 2024
@mikeywuu
Copy link

I would love to use this library in a POC for a new service which relays records from the consumer assigned topic to a destination topic which's name is build dynamically.
Therefore, this is unfortunately also blocking me and I'll closely monitor this issue :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

4 participants
@twmb @Nesze @mikeywuu and others