MQTT alternative topic structure proposal

[torbennehmer]

Hi there,

while thinking about autodiscovery and related topics (#333), I mused about an alternative approache to structuring the MQTT topic landscape. Right now, the world we have is simple, which has its own advantages, but it makes more dynamic integration scenarios more demanding on the other hand. The idea would be to introduce another MQTT mode, (keeping backwards compatibility), that does something like this:

• Each heater function gets its own topic
• BSB-LAN would generally at least listen to every one simultaneously (wildcard subscription to the rescue)
• Subtopics will give additional control (like setter topics and dynamic subscriptions)
• On this MQTT autodiscovery for external tools like Homeassistant could be built.

To illustrate this with an example, using /bsb as a root topic (less typing)

/bsb/status => online/offline state as is
/bsb/700 => root topic for heater function 700, will deliver the actual raw value without any unit for easy consumption
/bsb/700/set => topic to set a new value for function 700 ("command topic" if you will)
/bsb/700/update-interval => gives the update interval in whatever unit (TBD) if set
/bsb/700/set-update-interval => command topic for setting an auto-update interval
/bsb/700/last-update => ISO Timestamp of the last update of the sensor value (if persistent/retained f.x.)
/bsb/700/... => maybe more if we'd need them for managing the topic (needs a POC here)

If a topic is not configured (in whatever way), there will be no MQTT messages to /bsb/700 or /bsb/700/update-interval, the topics will be dead.

As soon as the topic is activated either by static configuration or by a dynamic message to set-update-interval, things will go live. BSB-LAN would have to keep track of all active topics of course, so there will be some limits to this, but that should be manageable.

Additional idea: This would also allow us to opportunistically dump any value with sniff out of the bus traffic to the MQTT layer. In addition, any time we read a value of the bus, we could dump it into the broker.

More generally, if we take advantage of retained messages, we could even make the MQTT topic structure sort of an external state where we protocol whatever we see. We can also use idle time on the main loop to cycle heater functions and update them f.x. daily (enough for most parameters) one at a time.

It would have to take a bit of experimentation how to balance this idea against the rather vast number of heater functions available, but assuming that we have a long running system, this shouldn't be that bad. Also, with a almost continuously running broker, we don't have to keep track of too many things in an Arduinos limited system memory, most of the heavy lifting would be done by the broker.

What do you think?

Best regards, Torben

[fredlcore]

Actually, I don't really see the advantages here, but that may be because I don't use MQTT ;). Maybe you could make them a bit more clear. As for sniffing the bus, I'm not sure whether that makes sense because it's easily possible that we "miss" a broadcast telegram because the Arduino is doing something else at that moment. That could mean that a cached value is no longer accurate which is why I would always recommend to read a specific parameter when it's required.

[torbennehmer]

Actually, I don't really see the advantages here, but that may be because I don't use MQTT ;). Maybe you could make them a bit more clear.

I was coming from several angles:

First, the current implementation uses MQTT with a command like syntax. You take a single topic, then parse commands out of it. This works, but the idea is closer to an RPC or WebService Approach, then a messaging telegram system. Messaging Systems normally deploy a number of topics, each message type being an individual topic.

For example, take the MQTT structure of the shelly switches.

https://shelly-api-docs.shelly.cloud/_review/mqtt/#shelly-switch-mqtt

They have a separate topic for each command you can send (here, one per relay). You don't send "1=on" to your command topic, you send "on" to the command topic for the specific relay in the switch. Most MQTT integrations also expect a command topic and a payload, with appropriate defaults. Most integrations simply ask you for the topic and push the actual parameter right into it. A combination like setting the command topic to "710=23" would need explicit scripting in Homeassistant, while setting a command topic simply to "23" is more easily done.

Of course, you'll still have to parse everything out, but, in general terms, this works by stripping the MQTT prefix of the topic in question, then separating the topic by its hierarchy separator and parsing each level at a time.

The idea outlined above would have a base topic for every heater function and append its structure below it. This should make more complex MQTT implementations possible as well. Based on a structure like that, you can implement almost arbitrary command handler APIs on top of it.

As for sniffing the bus, I'm not sure whether that makes sense because it's easily possible that we "miss" a broadcast telegram because the Arduino is doing something else at that moment. That could mean that a cached value is no longer accurate which is why I would always recommend to read a specific parameter when it's required.

Understood and agreed as a rule of thumb - I've observed that as well. I was thinking of a more opportunistic approach. If we can read off a bus telegram for any topic subscribed, it would be better then ignoring it.

Actually, I was thinking about combining this with update intervals. So if we see a value we're interested in, why not collect it and push it to MQTT. If we do not retain the value, it is as transient as it gets, but why shouldn't we push it, at the time of the capture it is correct, after all.

However, this will only make sense if you have a way to spread out the MQTT topic tree, so that a subscriber can be flexible about this.

Now combine this with the subscription/tracking suggestions above, you can have some best-of-both-worlds:

a) You tell BSB-LAN to watch a value at a certain interval (minutes, days etc.). This works like we do it now, so when captured, we put it on the bus and get on with it. Combining this with more flexible tracking values, this could even reduce the load on the bus. E.g. my own system regularly broadcasts the temperature of the heater it self ("Kesseltemperatur") to everybody who cares. I also track this value manually, but BSB wouldn't actually have to track it, as the additional polls are effectively uneccessary.

b) If you're really interested in a value, go tell the system to poll it anyway, so you're sure of it.

c) for setup values, this could also allow us to poll them only rarely (like once a day) but having a decent chance nevertheless to capture a change done by any other part of the system.

I was also thinking to rate limit any query the MQTT polling does, so that we can get back to the bus as often as possible to keep the system responsive. Right now, when listening to 15 or 20 values, BSB is blocked for several seconds, which is something I'd actually like to avoid.

All this of course needs a bit of a more complex structure, but I think the benefit could be huge, as I think we could easily track much more values with this approach, especially when taking advantage of flexible polling. Many values could be polled only every 10 minutes, some of them only hourly, others every 30 seconds etc.

Again, I was thinking about an incremental approach, starting with a PoC to get a feel on it. This is also, why I suggested just using another MQTT mode to be able to separate the handling code easily.

Best regards
Torben

[fredlcore]

Thank you for this detailed explanation, now it makes more sense to me. As for me, I'd be fine with it, but as I said, since I'm not a MQTT user, programming would have to be done by someone else :).

[dukess]

They have a separate topic for each command you can send (here, one per relay). You don't send "1=on" to your command topic, you send "on" to the command topic for the specific relay in the switch. ...

Looks reasonable!

If we can read off a bus telegram for any topic subscribed, it would be better then ignoring it.

👍

I was also thinking to rate limit any query the MQTT polling does, so that we can get back to the bus as often as possible to keep the system responsive.

For these reasons, it makes sense not to recommend making requests to multiple functions at once for WEB/JSON interfaces.

[torbennehmer]

I was also thinking to rate limit any query the MQTT polling does, so that we can get back to the bus as often as possible to keep the system responsive.

For these reasons, it makes sense not to recommend making requests to multiple functions at once for WEB/JSON interfaces.

I think this could be a general area of improvement, there are quite a few things that block the system, not surprisingly given that we don't have a modern OS below the code :) I don't completely understand, how especially the bus interface works, but at least for MQTT I am aiming to yield back to loop as often as possible, so that the adapter stays responsive whatever we have to do.

I have to toy around with what is actually possible with the limited resources of an Arudino - honestly, this is a new (and exciting) world to me.

MQTT could give us a real good advantage here, as it is an asynchrous system. With some intelligent queueing inside the MQTT code, I'm fairly sure that we can scale up to even large number of options being queried given enough time and a good priorization.

If this works as an example, we could also look into where we could actually take advantage of something like that on other parts of the code as well.

What I am wondering: Querying the bus takes several hundret milliseconds. Does the code need to activly work for an answer during that time on the bus? Or could the telegram messanging be somehow done more asynchronously, e.g. so that we can get an interrupt on incoming bus telegrams so that we read them into a queue and yield back to the code outside the handler?

[fredlcore]

As for not blocking BSB-LAN due to longer requests: This should generally not be a problem because once the "I've got a new toy!" phase is over ;), people won't be accessing the webinterface that often anymore. Once you have set up your home automation system of whatever kind, it will poll/send the pre-defined parameters and unless you have unnecessary short intervals, the last request will already be processed before the next one comes in.

The problem would only become relevant if you have two home automation systems accessing BSB-LAN independently. I had this when I was testing the HomeKit implementation while also maintaining acces from my standard FHEM system. HomeKit's timeout was way too short, so it never really got a response. But that may not be a standard use case.

But yes, the send function would not have to wait for the reply if some kind of queueing system would be in place. Since we switched to HardwareSerial, the UART will also buffer incoming telegrams, so we may not even have to work with interrupts. Two problems may arise: A matching mechanism would have to be put into place to safely match sent and received telegrams. The high nibble of the telegram type byte could be used for that (e.g. 06/16/26/36/46 instead of just 06 for querying a parameter). This is how LPB originally also did it. For LPB (and IIRC also BSB), the heater would reply with the same high nibble, despite the different telegram type (so 07/17/27/37 etc. instead of just 07). Matching could also/additionally be done via the command ID transmitted in almost all telegrams.

The other thing to keep in mind is the PPS handling. Other than BSB/LPB, telegrams are sent here constantly between the heater and BSB-LAN, and any delay longer than half a second or so will appear as a lost connection to the heater. So the current Send() function must still be able to bypass the proposed queue.

All in all, this is not a trivial change and has far-reaching effects. But yes, if done properly it would certainly make accessing BSB-LAN more stable in the above-mentioned settings.

[torbennehmer]

@Luposoft63 @dukess What do you think about this idea?

[Luposoft63]

@torbennehmer
in my custom-code I use MQTT more individually.
e.g. the value-parameters has different intervals and a value for hysteresis to decrease sending messages.
status-parameters will only send if changed. (the interval of query the heater is very short for it)

Actually I try to find out the best values to get I nice diagrams with as less values as possible.
But this need a lot of updating the arduino. :-(

I had also the idea to changed this values on the fly. Actually without solution.
So I can say, I'm a great fan of your idea.
It's absolutely more easy to handle this with different topics.

[fredlcore]

Just for bringing closure to this discussion: Most of the suggestions in the first post have now been implemented, including the topic hierarchy with sub-topics etc. Individual update intervals are still not possible since BSB-LAN does not support that "under the hood", but a new sub-topic /poll has been introduced for each parameter which forces an update of that parameter. So with a corresponding automation that publishes to the /poll topic, individual update intervals can be achieved for each parameter.