Support for clustering Kafka via new Docker swarm mode?

[bluepuma77]

Are there plans to support clustering Kafka via Docker swarm mode, which is new in Docker v1.12.0? With the new swarm you can define services and automatically scale them up or down. Plus docker handles load balancing automatically between instances of the service (default round robin).

docker service create --replicas 1 --name kafka spotify/kafka
docker service scale kafka=3

Intro: https://docs.docker.com/engine/swarm/
Concepts: https://docs.docker.com/engine/swarm/key-concepts/

[baseman]

You might be looking for something like this...

• https://github.com/wurstmeister/kafka-docker

You can use this projects against your swarm using docker compose. Note: I'm about to try this myself.

[brandonlamb]

Got here from google on this.

Docker Swarm != Docker Engine Swarm.

Docker Engine Swarm is not currently compatible with docker compose (it will just start all containers on the single host).

My team is trying to figure out how to handle the new Docker Engine Swarm and kafka

[ygizhitsa]

with the new engine 1.12 kafka might not need to use zookeeper . topics and partitions also might fit beautifully with stacks and bundles (experimental mode)

[mostolog]

@ygizhitsa Could you provide a link or more info about that?

[shay1bz]

Hi, any 2019 update on this issue?