Inconsistent data about inbound connectors because of multiple replicas

[markfarkas-camunda]

Describe the Bug

Since we have 2 replicas, fetching data for inbound connectors become inconsistent. Sometimes we fetch data from one replica, sometimes from the other, causing strange changes in the Webhook/Subscription tab in web-modeler.

Live example: https://modeler.ultrawombat.com/diagrams/b972d768-7208-4b48-a9e1-69bc819ba99a--kafka-incorrect-activation-condition?v=760,386,1

Modeler._.Kafka.incorrect.activation.condition.-.Google.Chrome.2024-08-02.11-37-58.mp4

Steps to Reproduce

1. Create any inbound connector
2. Do something to have some logs
3. Open the Subscription/Webhook tab in web-modeler and observe that sometimes we have duplicated data

Expected Behavior

We should not have the duplicated data

[sbuettner]

I think we saw this coming: https://github.com/camunda/product-hub/issues/1827#issuecomment-1836240235

[sbuettner]

@crobbins215 to sync @theburi regarding this and hot it fits into https://github.com/camunda/product-hub/issues/1063

[markfarkas-camunda]

While investigating I identified an other issue/improvement opportunity: we do not distinguish between logs coming from cluster1 and cluster2. We always merge all the incoming logs and show them together. It would be nice to show the logs separated by clusters. Separation by nodes on the other hand might not make much sense if we somehow identify the log entries (e.g. based on timestamp) and only add new ones but never delete any. This would ofc also require a guarantee that we always have info from all nodes. Regardless, I've reached out to console and sre team about this to have the whole picture.

[markfarkas-camunda]

This issue was also referred to here: https://github.com/camunda/product-hub/issues/2250#issuecomment-2134580579