HLK-LD2410 plugin causes mqtt data flooding

[starfish]

I noticed considerable data flooding from this plugin to the mqtt broker - how can I reduce it?
It looks like for each serial data packet from the sensor a mqtt message is sent.
sorry I placed the above question at the wrong place - despite that TD-er kindly replied:

You can act on the events sent by the task in the rules.
From there you can try to decimate the traffic, for example only to send if there was no presence for N amount of time.
Sending to MQTT can then be done via the publish command, or you copy the required values from the event to a dummy task and then call taskRun for that dummy task.
For this you should no longer send from the LD2410 task, but only from this dummy task to the MQTT controller.

basically I understand it, but having no experience with these tasks and rules, I would appreciate some help or examples.

[Ath]

Can you show your Device settings page?
Normally the Interval setting can be left at 0, as the values will be sent if any data is changed. This may cause many consecutive messages to be sent, as some values are quite volatile. A solution can be to use different values, or less values, to only send out the more stable values, like Presence of Output pin state (in Engineering mode).

Edit: Oh, and not unimportant, enable the "Generate Events only when changed" checkbox, to only send events (and data to Controllers) if something has changed...

[starfish]

@Ath thanks for your reply - I know the function of "interval" and "Generate Events only when changed" from other sensors, but here they have no influence - obviously changing values on other outputs cause these presence outputs to send consecutive mqtt messages of the same value in 1 sec. intervals (or shorter).

mqtt-explorer.PNG (9.5 KiB) Viewed 3901 times

[starfish]

here the same output with only 2 outputs activated (here too the other output triggers consecutive messages of the same value)

2-outputs.PNG (22.63 KiB) Viewed 3900 times

[Ath]

You may have noticed that this (type of) sensor is quite 'sensitive', especially the 'Energy' values fluctuate a lot. So leaving these out of the list of output values will help in reducing the number of messages.

I notice that you have the sensor connected to an ESP8266, though I've initially thought about not enabling this sensor for ESP8266, as the constant stream of data is quite a load for such low-resources MCU. When running on ESP32 it may run more smooth, but that of course won't change the constant stream of data from the sensor, so the number of triggers won't reduce much.

When enabling the Engineering mode setting to Yes, you also have the added feature of calibration of the sensor, reducing the sensitivity f.e. at greater distances, as the sensor can sort of 'see' through walls, that might give undesired interference/triggers.

[starfish]

I agree that the sensor is very sensitive, but the good thing is, that its sensitivity can be tailored to your needs. Anyhow, I dont understand is, why an event on 1 channel triggers an output on all other channels too. this makes the "Generate Events only when changed" and interval settings not giving the expected result.

[Ath]

Anyhow, I dont understand is, why an event on 1 channel triggers an output on all other channels too.

That trigger is "if anything is changed", then all configured values are sent out to the Controller (and as events, either all or separate, depending on the 'Single event with all values' setting). Some controllers (servers) expect all values to be available to send as a single message, others get up to 4 separate messages.

[starfish]

ok I understand, then its by design. my server (iobroker) treats every topic/channel as a single datapoint and stores it in a database which is cluttered with unnecessary data by this way. Until now espeasy was a very good choice filing everything at the right place.

Firefox_Screenshot_2024-06-11T11-24-24.233Z.png (54.7 KiB) Viewed 3857 times

[Ath]

ok I understand, then its by design. my server (iobroker) treats every topic/channel as a single datapoint

Well, ESPEasy also sends all values as separate datapoints to the server, when using an MQTT Controller, as that's the regular behavior of MQTT. The change-trigger detection is done at the LD2410 Sensor level (plugin), where each used value (max 4) is compared to the last value read. We don't support any throttling at this point, nor change-detection at a Values level, so if any of the values is changed, all (4) values are sent.
I see that you haven't enabled de-duplication in the Controller settings, but even when enabled, current Controller-side de-duplication of data is on the complete message set, so that won't make much of a difference in the number of messages sent.

Why is it an issue that all 4 values are sent to iobroker? AFAICS, that's kind of the nature of MQTT, and data collection in general, that there may be rapidly updating data, changed or not.

Re-thinking all this, it gives me some ideas on possible value-level de-duplication, but that will probably only be made available in ESP32 builds, as it will use some additional memory (nr of tasks * 4 * Value storage size + some overhead), and memory is already quite limited on ESP8266

[starfish]

you are right, de-duplication helped a little bit but not substantially. I am not specialist, but from what I have seen, data packages in json format are sent every time at least one of the datapoints changed - guess this corresponds to single event with all values. I think the problem is that with this kind of sensor we have datapoints which change very often. Maybe you find a solution - until then I only select the output I need for further processing in iobroker. Thank you for looking into it.

[Ath]

All values of this sensor are available in the [<taskname>#<valuename>] format, like any of the regular task values, so you can do some rules with deduplication like this:

Code: Select all

on System#Boot do
  LoopTimerSet,1,10 // Every 10 seconds, loop forever
endon

on Rules#Timer=1 do // Only send if value changed, initial %vN% values are 0
  if %v1% <> [Presence#MovEnergy]
    let,1,[Presence#MovEnergy]
    publish,move-2/presence/MovEnergy,%v1%
  endif
  if %v2% <> [Presence#StatEnergy]
    let,2,[Presence#StatEnergy]
    publish,move-2/presence/StatEnergy,%v2%
  endif
  // etc.
endon

The exact list of available valuenames for this plugin is here

Edit: Fixed a typo in the code

[starfish]

its just a matter of creating a rule using the above script and reboot right? it doesnt work here - something else to do?
well, I corrected the taskname to "presence" and used other valuenames

Code: Select all

on System#Start do
  LoopTimerSet,1,10 // Every 10 seconds, loop forever
endon

on Rules#Timer=1 do // Only send if value changed, initial %vN% values are 0
  if %v1% <> [presence#MovPres]
    let,1,[presence#MovPres]
    publish,move-4/presence/MovPres,%v1%
  endif
  if %v2% <> [presence#StatPres]
    let,2,[presence#StatPres]
    publish,move-4/presence/StatPres,%v2%
  endif
  // etc.
endon

is publish using the active mqtt-broker?

[Ath]

its just a matter of creating a rule using the above script and reboot right? it doesnt work here - something else to do?
well, I corrected the taskname to "presence" and used other valuenames

Ah, I messed up with the System event (typed it from memory without possibility to test)

Code: Select all

on System#Boot do
  LoopTimerSet,1,10 // Every 10 seconds, loop forever
endon

on Rules#Timer=1 do // Only send if value changed, initial %vN% values are 0
  if %v1% <> [presence#MovPres]
    let,1,[presence#MovPres]
    publish,move-4/presence/MovPres,%v1%
  endif
  if %v2% <> [presence#StatPres]
    let,2,[presence#StatPres]
    publish,move-4/presence/StatPres,%v2%
  endif
  // etc.
endon

Corrected the code.
You could have used the LoopTimerSet command from the Tools page to activate it, but I didn't explain that

is publish using the active mqtt-broker?

The Publish command uses the first enabled MQTT Controller

[starfish]

thanks a lot. now it works - and with the example I can now try to optimize.

[starfish]

when editing the ld2410 device you can choose 1 to 4 output values. I reduced that to 1 in order to have a clean still presence signal. To my big surprise I found the other values also transmitted on mqtt. But this time not influencing the transmission of the other values (as mentioned above). Is this intentionally?
howsoever, its exactly what I need. I dont even need the above rules. But why the number of output values configuration in that case?

[Ath]

Can you show the relevant part of your configuration?

[starfish]

espeasy select.png (292.69 KiB) Viewed 3649 times

[Ath]

I expect that the MovPres and StatPres values are left-over from previous configuration settings? Can you delete them in some way? Or at least have an idea on how recent that data is?

[TD-er]

In MQTT Explorer you can simply click on the topic and you'll see the timestamp of the last received message in that topic (if seen while MQTT explorer was running in the current session)

[starfish]

mistery solved. embarassing enough, by mistake, Ath´s last rule was not really deleted (probably not saving deletion). This was generating these "ghost" topics despite I removed them from the output-list. please excuse.

[Ath]

Ah, thanks for the update. I was already expecting a cause like this, NP

[starfish]

right, its more plausible but to be honest I am still not very happy with this solution, as move-events often are considerably delayed.

[Ath]

right, its more plausible but to be honest I am still not very happy with this solution, as move-events often are considerably delayed.

The plugin is sending out the data nearly as soon as it is detected by the sensor, but is limited to send at most once per 100 msec (fixed interval, independent from the plugin Interval value). You can even leave the Interval setting at 0 as this is not required to be set to get the data sent to Controllers or the events generated.
Any unexpected delays can be caused by the ESP8266 in use. The original development (and library) has been based with an ESP32 in mind, as that has more memory available, and can better handle the high-speed communication (256000 baud) from the sensor. Only after some successful testing we chose to also support this for ESP8266, though maybe we should have added a warning in the documentation. Having other plugins active also is a possible cause for delays.

You can try to install an ESP32 instead (any of the supported ESP32 models (Classic, S2, S3, C3 are supported and C2 and C6 preliminary supported for ESPEasy) should suffice), that should give better performance/response.

[starfish]

The plugin is sending out the data nearly as soon as it is detected by the sensor, ...

is that true also for the above rule?

on System#Boot do
LoopTimerSet,1,10 // Every 10 seconds, loop forever
endon ...

[TD-er]

Not sure what you're asking...

When a task has a new value, it will send out one or more events. (depending on whether the "send all in single event" checkbox is checked)

Those events are processed by the rules as soon as possible.

But if you have a timer set, then the events for that timer will be dealt with when that timer expires.
So that's not in sync with when a task performs a reading.

You could however call "taskrun" from the rules based on a timer.
But that's probably not what you're asking, right?

[starfish]

sorry, no more time left. come back to it after vacation (in Skåne btw.)