Vertical scaling support

[AntonAleksandrov13]

Is your feature request related to a problem? Please describe.
We are managing a large amount of objects with messaging-topology-operator(75k+), when operator restarts due to update/eviction/others reason, reconciliation of all objects takes about 30+ minutes. During this time, it is highly likely that we would want to add a new object. This means that the worst case scenario a reconciliation of a new object could take 30+ minutes. This is critical for us as this creates a bottleneck in the system where other applications have to wait.

It seems like MaxConcurrentReconciles is not set in the operator which means that every controller can perform 1 operation at the time.

Describe the solution you'd like
I would like to see configurable MaxConcurrentReconciles value using env variables.

Describe alternatives you've considered
I have tried horizontally scale the operator, but the second replica does nothing. As a workaround, I have considered using ControllerMesh(https://blog.kusionstack.io/controllermesh-empowering-effortless-operator-management-2f57b6203719), but this seems too hacky and might not be an option to the most users.

[Zerpet]

This sounds reasonable. I don't have time to work on this immediately, but I'll be happy to review a contribution.

[AntonAleksandrov13]

No worries. I can work on it