Limiting data transfer

[AndrewJ]

Hi everyone,
I use a number of ESPeasy nodes (mostly NodeMCUs) which publish data to OpenHAB2 (openhabianpi) using Mosquitto, then the system saves values in an InfluxDB database and graphs them in Grafana. It works pretty well, but I find I'm storing a lot of data. Often the system is storing a new value for only a small change.

I'd like to have a way to check (ideally in the ESPeasy) whether the value has changed more than a preset amount or percentage, before publishing a new value. Is this already possible in the software, and I just haven't found it yet? Or would Rules be a good way to do this? Does anyone have experience of doing this?

AndrewJ

[krikk]

have a look at retention rules in the influxdb documentation...

[AndrewJ]

Hello krikk,
Thanks for the quick reply and the tip about InfluxDB. Yes, I'll be able to make some improvements in the InfluxDB persistence, and that will help with the storage end of things. All the same, I would still like to reduce the amount of "unnecessary" data-values at the start of the chain (in the ESPeasy nodes) if I can.

My reasons for this are that it sometimes appears that my wifi router (or my Raspberry Pi?) gets choked with data and hangs for a while. Sometimes the successive values differ by not very much, but each one still produces a message in MQTT, and a state change in OpenHAB.

I should also point out that the measurements I'm storing in InfluxDB are only about one third of the total being sent from sensors - there are more values being published by nodes and received in OpenHAB than the ones going into InfluxDB. So if I could "filter" the data published by the ESPeasy nodes at source, it would make a significant difference to the data flow.

I'm thinking along the lines of (in pseudo code) ...
Measure new value
Does it differ from last published value by more than x ?
IF it does, publish new value and update last published value
ELSE
do nothing, wait for next new value and start again.

Do you think this could work? Would it be best done in a Rule (perhaps needs a Dummy task too?) or is there a better way?
TIA for any suggestions,
Regards
AndrewJ

[budman1758]

Not sure if it will help with the differences in data but methinks one way to slow it down so to speak is increase the sensor delay time? If I'm not mistaken this might help with the excess.

[AndrewJ]

Hi @budman1758,
Thanks for that. Yes, I'm starting to think that the various "delays" (there are three in different places) may provide much of the control I need.

Yesterday I did some more searching in the forum and came across a very good summary by @Shardan (here: https://letscontrolit.com/forum/viewtop ... lay#p14668) about what each "delay" does. I'm still experimenting with them, to understand them and how to get the "best" result. I guess like so many things it's a compromise, I think when I first set things up I went in with fairly rapid updates, maybe more than I really need!
Thanks again for your advice.
AndrewJ