HLK-LD2410 plugin causes mqtt data flooding

Moderators: grovkillen, Stuntteam, TD-er

Post Reply
Message
Author
starfish
Normal user
Posts: 46
Joined: 16 Feb 2017, 23:46

HLK-LD2410 plugin causes mqtt data flooding

#1 Post by starfish » 10 Jun 2024, 16:56

I noticed considerable data flooding from this plugin to the mqtt broker - how can I reduce it?
It looks like for each serial data packet from the sensor a mqtt message is sent.
sorry I placed the above question at the wrong place - despite that TD-er kindly replied:
You can act on the events sent by the task in the rules.
From there you can try to decimate the traffic, for example only to send if there was no presence for N amount of time.
Sending to MQTT can then be done via the publish command, or you copy the required values from the event to a dummy task and then call taskRun for that dummy task.
For this you should no longer send from the LD2410 task, but only from this dummy task to the MQTT controller.
basically I understand it, but having no experience with these tasks and rules, I would appreciate some help or examples.

User avatar
Ath
Normal user
Posts: 3655
Joined: 10 Jun 2018, 12:06
Location: NL

Re: HLK-LD2410 plugin causes mqtt data flooding

#2 Post by Ath » 10 Jun 2024, 20:38

Can you show your Device settings page?
Normally the Interval setting can be left at 0, as the values will be sent if any data is changed. This may cause many consecutive messages to be sent, as some values are quite volatile. A solution can be to use different values, or less values, to only send out the more stable values, like Presence of Output pin state (in Engineering mode).

Edit: Oh, and not unimportant, enable the "Generate Events only when changed" checkbox, to only send events (and data to Controllers) if something has changed... ;)
/Ton (PayPal.me)

starfish
Normal user
Posts: 46
Joined: 16 Feb 2017, 23:46

Re: HLK-LD2410 plugin causes mqtt data flooding

#3 Post by starfish » 11 Jun 2024, 06:17

@Ath thanks for your reply - I know the function of "interval" and "Generate Events only when changed" from other sensors, but here they have no influence - obviously changing values on other outputs cause these presence outputs to send consecutive mqtt messages of the same value in 1 sec. intervals (or shorter).
mqtt-explorer.PNG
mqtt-explorer.PNG (9.5 KiB) Viewed 211 times
Attachments
Firefox_Screenshot_2024-06-11T03-35-26.024Z.png
Firefox_Screenshot_2024-06-11T03-35-26.024Z.png (142.37 KiB) Viewed 211 times
Firefox_Screenshot_2024-06-11T03-38-05.212Z.png
Firefox_Screenshot_2024-06-11T03-38-05.212Z.png (136.92 KiB) Viewed 211 times

starfish
Normal user
Posts: 46
Joined: 16 Feb 2017, 23:46

Re: HLK-LD2410 plugin causes mqtt data flooding

#4 Post by starfish » 11 Jun 2024, 06:28

here the same output with only 2 outputs activated (here too the other output triggers consecutive messages of the same value)
2-outputs.PNG
2-outputs.PNG (22.63 KiB) Viewed 210 times

User avatar
Ath
Normal user
Posts: 3655
Joined: 10 Jun 2018, 12:06
Location: NL

Re: HLK-LD2410 plugin causes mqtt data flooding

#5 Post by Ath » 11 Jun 2024, 08:24

You may have noticed that this (type of) sensor is quite 'sensitive', especially the 'Energy' values fluctuate a lot. So leaving these out of the list of output values will help in reducing the number of messages.

I notice that you have the sensor connected to an ESP8266, though I've initially thought about not enabling this sensor for ESP8266, as the constant stream of data is quite a load for such low-resources MCU. When running on ESP32 it may run more smooth, but that of course won't change the constant stream of data from the sensor, so the number of triggers won't reduce much.

When enabling the Engineering mode setting to Yes, you also have the added feature of calibration of the sensor, reducing the sensitivity f.e. at greater distances, as the sensor can sort of 'see' through walls, that might give undesired interference/triggers.
/Ton (PayPal.me)

starfish
Normal user
Posts: 46
Joined: 16 Feb 2017, 23:46

Re: HLK-LD2410 plugin causes mqtt data flooding

#6 Post by starfish » 11 Jun 2024, 08:52

I agree that the sensor is very sensitive, but the good thing is, that its sensitivity can be tailored to your needs. Anyhow, I dont understand is, why an event on 1 channel triggers an output on all other channels too. this makes the "Generate Events only when changed" and interval settings not giving the expected result.

User avatar
Ath
Normal user
Posts: 3655
Joined: 10 Jun 2018, 12:06
Location: NL

Re: HLK-LD2410 plugin causes mqtt data flooding

#7 Post by Ath » 11 Jun 2024, 11:04

starfish wrote: 11 Jun 2024, 08:52 Anyhow, I dont understand is, why an event on 1 channel triggers an output on all other channels too.
That trigger is "if anything is changed", then all configured values are sent out to the Controller (and as events, either all or separate, depending on the 'Single event with all values' setting). Some controllers (servers) expect all values to be available to send as a single message, others get up to 4 separate messages.
/Ton (PayPal.me)

starfish
Normal user
Posts: 46
Joined: 16 Feb 2017, 23:46

Re: HLK-LD2410 plugin causes mqtt data flooding

#8 Post by starfish » 11 Jun 2024, 13:34

ok I understand, then its by design. my server (iobroker) treats every topic/channel as a single datapoint and stores it in a database which is cluttered with unnecessary data by this way. Until now espeasy was a very good choice filing everything at the right place.
Firefox_Screenshot_2024-06-11T11-24-24.233Z.png
Firefox_Screenshot_2024-06-11T11-24-24.233Z.png (54.7 KiB) Viewed 167 times

User avatar
Ath
Normal user
Posts: 3655
Joined: 10 Jun 2018, 12:06
Location: NL

Re: HLK-LD2410 plugin causes mqtt data flooding

#9 Post by Ath » 11 Jun 2024, 15:00

starfish wrote: 11 Jun 2024, 13:34 ok I understand, then its by design. my server (iobroker) treats every topic/channel as a single datapoint
Well, ESPEasy also sends all values as separate datapoints to the server, when using an MQTT Controller, as that's the regular behavior of MQTT. The change-trigger detection is done at the LD2410 Sensor level (plugin), where each used value (max 4) is compared to the last value read. We don't support any throttling at this point, nor change-detection at a Values level, so if any of the values is changed, all (4) values are sent.
I see that you haven't enabled de-duplication in the Controller settings, but even when enabled, current Controller-side de-duplication of data is on the complete message set, so that won't make much of a difference in the number of messages sent.

Why is it an issue that all 4 values are sent to iobroker? AFAICS, that's kind of the nature of MQTT, and data collection in general, that there may be rapidly updating data, changed or not.

Re-thinking all this, it gives me some ideas on possible value-level de-duplication, but that will probably only be made available in ESP32 builds, as it will use some additional memory (nr of tasks * 4 * Value storage size + some overhead), and memory is already quite limited on ESP8266 :?
/Ton (PayPal.me)

starfish
Normal user
Posts: 46
Joined: 16 Feb 2017, 23:46

Re: HLK-LD2410 plugin causes mqtt data flooding

#10 Post by starfish » 11 Jun 2024, 15:23

you are right, de-duplication helped a little bit but not substantially. I am not specialist, but from what I have seen, data packages in json format are sent every time at least one of the datapoints changed - guess this corresponds to single event with all values. I think the problem is that with this kind of sensor we have datapoints which change very often. Maybe you find a solution - until then I only select the output I need for further processing in iobroker. Thank you for looking into it.

User avatar
Ath
Normal user
Posts: 3655
Joined: 10 Jun 2018, 12:06
Location: NL

Re: HLK-LD2410 plugin causes mqtt data flooding

#11 Post by Ath » 11 Jun 2024, 15:38

All values of this sensor are available in the [<taskname>#<valuename>] format, like any of the regular task values, so you can do some rules with deduplication like this:

Code: Select all

on System#Boot do
  LoopTimerSet,1,10 // Every 10 seconds, loop forever
endon

on Rules#Timer=1 do // Only send if value changed, initial %vN% values are 0
  if %v1% <> [Presence#MovEnergy]
    let,1,[Presence#MovEnergy]
    publish,move-2/presence/MovEnergy,%v1%
  endif
  if %v2% <> [Presence#StatEnergy]
    let,2,[Presence#StatEnergy]
    publish,move-2/presence/StatEnergy,%v2%
  endif
  // etc.
endon
The exact list of available valuenames for this plugin is here

Edit: Fixed a typo in the code :oops:
Last edited by Ath on 12 Jun 2024, 08:47, edited 2 times in total.
/Ton (PayPal.me)

starfish
Normal user
Posts: 46
Joined: 16 Feb 2017, 23:46

Re: HLK-LD2410 plugin causes mqtt data flooding

#12 Post by starfish » 12 Jun 2024, 06:36

its just a matter of creating a rule using the above script and reboot right? it doesnt work here - something else to do?
well, I corrected the taskname to "presence" and used other valuenames

Code: Select all

on System#Start do
  LoopTimerSet,1,10 // Every 10 seconds, loop forever
endon

on Rules#Timer=1 do // Only send if value changed, initial %vN% values are 0
  if %v1% <> [presence#MovPres]
    let,1,[presence#MovPres]
    publish,move-4/presence/MovPres,%v1%
  endif
  if %v2% <> [presence#StatPres]
    let,2,[presence#StatPres]
    publish,move-4/presence/StatPres,%v2%
  endif
  // etc.
endon
is publish using the active mqtt-broker?

User avatar
Ath
Normal user
Posts: 3655
Joined: 10 Jun 2018, 12:06
Location: NL

Re: HLK-LD2410 plugin causes mqtt data flooding

#13 Post by Ath » 12 Jun 2024, 08:47

starfish wrote: 12 Jun 2024, 06:36 its just a matter of creating a rule using the above script and reboot right? it doesnt work here - something else to do?
well, I corrected the taskname to "presence" and used other valuenames
Ah, I messed up with the System event :oops: (typed it from memory without possibility to test)

Code: Select all

on System#Boot do
  LoopTimerSet,1,10 // Every 10 seconds, loop forever
endon

on Rules#Timer=1 do // Only send if value changed, initial %vN% values are 0
  if %v1% <> [presence#MovPres]
    let,1,[presence#MovPres]
    publish,move-4/presence/MovPres,%v1%
  endif
  if %v2% <> [presence#StatPres]
    let,2,[presence#StatPres]
    publish,move-4/presence/StatPres,%v2%
  endif
  // etc.
endon
Corrected the code.
You could have used the LoopTimerSet command from the Tools page to activate it, but I didn't explain that :roll:
starfish wrote: 12 Jun 2024, 06:36 is publish using the active mqtt-broker?
The Publish command uses the first enabled MQTT Controller
/Ton (PayPal.me)

starfish
Normal user
Posts: 46
Joined: 16 Feb 2017, 23:46

Re: HLK-LD2410 plugin causes mqtt data flooding

#14 Post by starfish » 12 Jun 2024, 12:14

thanks a lot. now it works - and with the example I can now try to optimize.

Post Reply

Who is online

Users browsing this forum: No registered users and 0 guests