Rule for measurement verification -> skip value?

[Oetsch]

Hello,
I´m using an ultrasonic sensor RCW-0001 to measure a water level in a tank. Sadly round about every 5th measuremt brings up a value that is not true and ~20% higher than the other other values. Tried to fix this without success so far. Now I thought of creating a rule in ESPEasy which checks the newest value in comparison to the last value and in case this step is higher than x% the value is skipped.

Can you please tell me if this is possible and how it can be done by a rule? So far I´m only using a rule to show the value on display and it is transferred via mqtt.
The measured value is transformed to show it in % and not in cm by the formula: (116-%value%)/(116-15)*100

Code: Select all

On HCSR04#distance do
  7dn,[HCSR04#distance]
EndOn

The Log Information says:

Code: Select all

154955987: ULTRASONIC : TaskNr: 1 Distance: 94.79 cm
154955997: EVENT: HCSR04#Distance=21.00
154956004: ACT : 7dn,21
154956006: Command: 7dn
154956007: 7DGT : Show Number=21

Thank you very much!

[TD-er]

You could keep track of the last value by storing it in a variable. (see command "let"
In the rules block you then compare the difference to this last value.
If it is within the range, you can update the last value and do whatever you need to do.

There is also the level plugin: https://github.com/letscontrolit/ESPEas ... _Level.ino
Sadly it isn't documented very well.
But as far as I remember, it can be used to keep track of values and some hysteresis.
Not sure if it also allows to filter on values to allow only the ones within this hysteresis.

[Oetsch]

Thx.
Do you think this can work? Don´t want to crash my esp from external connection.

Code: Select all

Let 1,[HCSR04#distance]

On HCSR04#distance do
 	if [HCSR04#distance]/[var#1]>0.9 and [HCSR04#distance]/[var#1]<1.1 do
  	7dn,[HCSR04#distance]
  	Let 1,[HCSR04#distance]
  	endif 	
EndOn

Will have a lock for the plug_in but this is totaly new topic for me.

[Ath]

Code: Select all

Let 1,[HCSR04#distance]

On HCSR04#distance do
 	if [HCSR04#distance]/[var#1]>0.9 and [HCSR04#distance]/[var#1]<1.1 do
  	7dn,[HCSR04#distance]
  	Let 1,[HCSR04#distance]
  	endif 	
EndOn

I'd do a few modifications:

Code: Select all

On HCSR04#distance do
  if [var#1]=0
    Let 1,[HCSR04#distance]
  endif
  if [HCSR04#distance]/[var#1]>0.9 and [HCSR04#distance]/[var#1]<1.1
    7dn,[HCSR04#distance]
    Let 1,[HCSR04#distance]
  endif 	
EndOn

Rules code has to be within an 'on ... do/endon' eventhandler, so the initialization wasn't going to work, and the 'if' doesn't have a 'do' (or 'then').

[Oetsch]

Thank you very much!
So for initialization process the variable is always =0 after boot up of the esp. Great to know and Thx!

[Oetsch]

Hi
tested this version but there was no output on the display. So modified the code with some publish outputs to check what is working and where is the problem.

Initial Publish (Debug1) is working after reboot but publish of the Level within the defined range is not working. Measurement is in the + or - 10% range so should work but I think the if condition <> does not work with the / calculation before. Is there anything wrong? Do I have to use brackets?

Code: Select all

 if ([HCSR04#distance]/[var#1])>0.5 and ([HCSR04#distance]/[var#1])<1.5

Code: Select all

On System#Boot do 
 Let 1,0
Endon

On HCSR04#distance do
  if [var#1]=0
    Let 1,[HCSR04#distance]
  Publish %sysname%/HCSR04/Debug1,[var#1]
  endif
  if [HCSR04#distance]/[var#1]>0.9 and [HCSR04#distance]/[var#1]<1.1
    7dn,[HCSR04#distance]
    Let 1,[HCSR04#distance]
    Publish %sysname%/HCSR04/Level,[var#1]
  endif 	
EndOn

[Ath]

Code: Select all

On System#Boot do 
 Let 1,0
Endon

On HCSR04#distance do
  if [var#1]=0
    Let 1,[HCSR04#distance]
  Publish %sysname%/HCSR04/Debug1,[var#1]
  endif
  if [HCSR04#distance]/[var#1]>0.9 and [HCSR04#distance]/[var#1]<1.1
    7dn,[HCSR04#distance]
    Let 1,[HCSR04#distance]
    Publish %sysname%/HCSR04/Level,[var#1]
  endif 	
EndOn

Ah, we fell in that trap before The if processing doesn't do any calculation, so the values to compare have to be calculated before being compared.
This should work:

Code: Select all

On System#Boot do 
 Let 1,0
Endon

On HCSR04#distance do
  if [var#1]=0
    Let 1,[HCSR04#distance]
    Publish %sysname%/HCSR04/Debug1,[var#1]
  endif
  Let 2,[HCSR04#distance]/[var#1]
  if %v2%>0.9 and %v2%<1.1  // %v2% is 'shorthand' for [var#2]
    7dn,[HCSR04#distance]
    Let 1,[HCSR04#distance]
    Publish %sysname%/HCSR04/Level,[var#1]
  endif 	
EndOn

[Oetsch]

Thank you very much Ath! It´s working very good.

final version:

Code: Select all

On HCSR04#distance do
  if [var#1]=0
    Let 1,[HCSR04#distance]
  endif
  Let 2,[HCSR04#distance]/[var#1]
  if %v2%>0.9 and %v2%<1.1  // %v2% is 'shorthand' for [var#2]
    7dn,[HCSR04#distance]
    Let 1,[HCSR04#distance]
    Publish %sysname%/HCSR04/Level,[var#1]
  endif 	
EndOn

I think i will include additional if clauses for low tank levels with a wider range of acceptable changes otherwise it will not work during filling/emtpy process in that low level status but this should be easier for me to implement based your given help. Thx!

[TD-er]

You can also make it more self-adjusting.
Now you only update var#1 if it is within the +/- 10% range.
You can also update it always, but only update it with a fraction of the difference.

Code: Select all

Let 1,%v1% * (%v2% / 8)

This factor of 8 will act as a filter, with higher values causing a longer response delay.
This way it will self correct after a while if the level has suddenly changed.

[Oetsch]

Thanks TD-er!
Think I understand but have to think a little longer about it.

Isn´t that a preferable solution when there is always some bigger error within the measurment result?

At the moment I belive ~5 data values are ok having an error of >1%, only 1 value has a big error of ~50%. Continuosly correcting with this 50% error brings up a much more worse result than just skipping this value, right?

At the moment I skipped the decimals because this really is like showing too much wrong output.

During steady status/operation I do not realy like changing values on the display because it clearly shows that the measurement isn´t good. But on the other hand it clearly shows that the measurement is still working with the changing values in a fix cycle time.

But otherwise I cannot slow down the value change to much because during filling/emtying procedure i like to control the level and stop the pump without overfilling or running the pump without water.

[TD-er]

Well that's why I suggest to only correct the value with a fraction of its difference.

Usually you will use the difference, but I use the ratio here as it was already present anyway.
Using the ratio is maybe a bit dangerous if the noise is constant, not depending on the measured value.
So that's something you need to look into for this application anyway.

By only moving slowly to the erroneous value, you still accept values which are in the +/- 10% range as in your example 80% of the samples do have the correct value.
Assume you have a single sample that's 50% off, by using 1/8th of that, the 'average' value will not move outside the 10% range and the other samples will correct it anyway.

But if it is still causing issues, you can increase this "8" value to like 16 or more.
It will cause the corrections to move more slowly, but noise will be filtered out.

[Oetsch]

Thx, think I understand your procedure. I would say you are calculating a new value to be shown for visualisation. This value will not change dramatically in one shot because of the filter. My thoughts were about using the measured values for display. Prio to the visualisation there is filter/skipping process. This are two different solution ways.

In case of your way I can slow down the change of value visualisation but in the background I still have access to the last measured value ([HCSR04#distance]) including error but this can be used as worst case value e.g. to turn of the pump.

So for the first test i will modify to:

Code: Select all

On HCSR04#distance do
  if [var#1]=0
    Let 1,[HCSR04#distance]
  endif
  Let 2,[HCSR04#distance]/[var#1]
  Let 1,%v1% * (%v2% / 8)
    7dn,[var#1]
    Publish %sysname%/HCSR04/Level,[var#1]
EndOn

[TD-er]

See the XLS file I linked with this PR: https://github.com/letscontrolit/ESPEasy/pull/3194

This allows you to play a bit with the same filtering principe to get a feeling for it.
N.B. it is using the "add" instead of the ratio and it is using larger (inverted) factors, as it needs to average over a plugin sampling 10 per sec.
But the general idea is the same.

[Oetsch]

Thank you very much!
Heavy topic for me, not sure if i ever had this in maths or signal theorie but when then it is gone for years.So takes some time to get on track...
Think I got a feeling what is done in the XLS sheet. Think the single stage is best for me to understand and the difference or what is done in stage 1 /2 not realy.
If I´m playing with a random factor of 0.5 the single stage calculation is going fastest and also nearest to the real value of 1 and so my understanding is that this is working best in my example.

Can you give me a name for the filtering using this factor mechanism? I like to look up and maybe I can also find in german language so it´s easier to understand and rember for me if i can see the theoretical formulas behind incl. explenation.

Please forget my status of understanding regarding measured values incl. noise and calculated values for visualization. This understanding is wrong, got it. When you are using a filter right, it is very good idea to use the result for everything

What I also understand out of this sheet etc. Having more samples will help improving the result. So it´s a good idea to increase the cycle time of measurement to minimum for a maximum count of samples, right?

There seems to be a mistake in my code above because the log info shows some warnings. Guess the formal typing of code is not right because of the reported to many arguments:

Code: Select all

19517276: EVENT: HCSR04#Distance=21.64
19517291: ACT  : Let 2,22/21.00
19517292: Command: Let
19517298: ACT  : Let 1,21.00 * (1.05 / 8)
19517300: Command: Let
19517302: Too many arguments: cmd=Let Arg1=1 Arg2=21.00 ExtraArg3=* ExtraArg4=(1.05 ExtraArg5=/ ExtraArg6=8) lineLength=24
19517303: Line: _Let 1,21.00 * (1.05 / 8)_
19517304: Command not executed! See: https://github.com/letscontrolit/ESPEasy/issues/2724
19517311: Command unknown: 'Let 1,21.00 * (1.05 / 8)'
19517314: ACT  : 7dn,21.00
19517316: Command: 7dn
19517318: 7DGT : Show Number=21
19517324: ACT  : Publish HeizKeller/HCSR04/Level,21.00
19517326: Command: Publish

In case of general wrong formula, guess i can find it if i know the name of the "factor-filter". google gave me not a good result so far

[TD-er]

I used spaces in the let command...
Just remove those and it should be fine I guess (or you assign it to another variable first)

Code: Select all

On HCSR04#distance do
  if [var#1]=0
    Let 1,[HCSR04#distance]
  endif
  Let 2,[HCSR04#distance]/[var#1]
  Let 1,%v1%*%v2%
  Let 1,%v1%/8     // determines filter speed
  7dn,[var#1]
  Publish %sysname%/HCSR04/Level,[var#1]
EndOn

The filter I described is an IIR filter.
See for some introduction and the difference between IIR and FIR: https://community.sw.siemens.com/s/arti ... versus-iir
Not sure if that introduction matches your way of learning about this kind of theory. I always like to see charts to understand it.

You just have to test a bit with the fractional value.
And also keep in mind this may not be the best approach when the samples with the clearly wrong value always have the same error, regardless whether you're at either end of the measurement range.
If that's what is happening, you may need to do something like this:

Code: Select all

On HCSR04#distance do
  if [var#1]=0
    Let 1,[HCSR04#distance]
  endif
  Let 2,[HCSR04#distance]/[var#1]
  Let 3,%v1%-%v2%     // store difference
  Let 3,%v3%/8           // determines filter speed
  Let 1,%v1%-%v3%     // subtract fraction of difference
  7dn,[var#1]
  Publish %sysname%/HCSR04/Level,[var#1]
EndOn

[Oetsch]

Thx, read some information and got an better idea (i think) but it´s hard to understand the theoretical background of IIR filter and the practical programming what is done here.

First let me ask one important question for me. What is the expectation to get as result out of the programming?

Code: Select all

On HCSR04#distance do
  if [var#1]=0
    Let 1,[HCSR04#distance]
  endif
  Let 2,[HCSR04#distance]/[var#1]
  Let 1,%v1%*%v2%
  Let 1,%v1%/8     // determines filter speed
  7dn,[var#1]
  Publish %sysname%/HCSR04/Level,[var#1]
EndOn

I´m assuming related to the XLS file that the result should go step by step into the direction of the real value. For the XLS example there have been used 1500 steps to achieve this. After some sample steps yesterday the result was quiet far away but even today it is not really a good value as shown by the log output:

Code: Select all

71909930: EVENT: HCSR04#Distance=21.11
71909944: ACT  : Let 2,21.11/4.60
71909945: Command: Let
71909951: ACT  : Let 3,4.60-4.59
71909953: Command: Let
71909958: ACT  : Let 3,0.01/8
71909960: Command: Let
71909966: ACT  : Let 1,4.60-0.00
71909969: Command: Let
71909972: ACT  : 7dn,4.60
71909973: Command: 7dn
71909975: 7DGT : Show Number=4
71909981: ACT  : Publish HeizKeller/HCSR04/Level,4.60
71909982: Command: Publish

Measured value is 21.11% and the %v1% output is 4.60. My stomage feeling says that with regard to the theoretical formulas the result of %v1% has to be added to the current measured value or summed up to a never ending sum? But this also does not give reasonable results.

So, sorry but my understanding is not completed so I´m also struggeling also with your second code. Tested it a little bit but the outcome is similar to the code above. I´m also wondering because the results do not change so much when I´m playing with the filter speed. A problem because it is only calculated with up to 2 decimals? 0,04/8= 0,005 -> = 0,00 So no changes for the output %v1% ?

Code: Select all

72614964: EVENT: HCSR04#Distance=21.53
72614981: ACT  : Let 2,21.53/4.62
72614982: Command: Let
72614988: ACT  : Let 3,4.62-4.66
72614990: Command: Let
72614996: ACT  : Let 3,-0.04/8
72614997: Command: Let
72615003: ACT  : Let 1,4.62--0.00
72615004: Command: Let
72615007: ACT  : 7dn,4.62
72615008: Command: 7dn
72615010: 7DGT : Show Number=4
72615016: ACT  : Publish HeizKeller/HCSR04/Level,4.62
72615017: Command: Publish

[TD-er]

The idea of an IIR is quite simple.

What you try to do is this:

diff = new - old
value = old + (diff / N)

So for this, you need to look at the last code I posted, as the one you show is the one working on a ratio, not a difference.
Working with a ratio makes it a bit more counter-intuitive, so please stick to the one where you use the difference as I describe here.

The fastest response (and no filtering at all) is when N = 1.
Then you just add the 'diff' to the 'old' value, which directly results in the 'new' value.

With N = 2, you get a more slower approach to the 'new' value.
For example:
old = 0, new = 10
If all next values are 10, it will theoretically take forever to reach 10, but you're approaching it quite fast with low values for N.
That's where the "infinite" stands for as you will never exactly reach "10".

What happens if you get a regular pattern like this:
5, 5, 5, 10, 5, 5, 5, 10, etc.

Then for N > 1, you will see a value which is above 5 and is wiggling a bit between 2 values as it will increase when you hit "10" and slowly decrease for every "5".
For large N, the amplitude of this wiggling will be low and for low values of N (e.g. N = 2) it will have a high amplitude. (for N = 1, the amplitude of this wiggling will be 10 - 5 = 5)

I created a test sheet, to illustrate it all:

https://www.dropbox.com/s/li619oac3wz3s ... .xlsx?dl=0

You can set the N and a start value.
This will help you understand what I mean with the response time and also what the bounds are in which the values oscillate.
The min and max out are the min and max values of the last 4 filter out values.

[Oetsch]

The idea of an IIR is quite simple.

What you try to do is this:

diff = new - old
value = old + (diff / N)

Big Thx. This is easy and right for me to understand Joking and also the rest is realy helpful.

But now I was able to modify the code and the result is working with more or less good results.
I changed the subtract fraction of difference to an addition.

Code: Select all

On HCSR04#distance do
  if [var#1]=0
    Let 1,[HCSR04#distance]
  endif
  Let 2,[HCSR04#distance]-[var#1] // store difference
  Let 3,%v2%/16           // determines filter speed
  Let 1,%v1%+%v3%     // add fraction of difference
  7dn,[var#1]
  Publish %sysname%/HCSR04/Level,[var#1]
EndOn

Result:

Code: Select all

689183: ULTRASONIC : TaskNr: 1 Distance: 93.74 cm 
689252: EVENT: HCSR04#Distance=22.04
689266: ACT  : Let 2,22.04-21.35
689267: Command: Let
689273: ACT  : Let 3,0.69/16
689274: Command: Let
689280: ACT  : Let 1,21.35+0.04
689281: Command: Let
689284: ACT  : 7dn,21.39
689285: Command: 7dn
689287: 7DGT : Show Number=21
689293: ACT  : Publish HeizKeller/HCSR04/Level,21.39
689295: Command: Publish

Guess now i can play and have a better feeling for the outcome.

[Oetsch]

Thx @TD-er & @Ath

I´m really happy with the solution. Currently the fraction of the difference is set to 14 and therefore the local display shows more or less no changes during stable tank level.
But for filling and emptying it is sensetive enough to show changes and even stop the pump early before the tank is emty or overfilled. So great solution for me thx!