pulsecounter acting strange after latest changes

Moderators: grovkillen, Stuntteam, TD-er

Post Reply
Message
Author
GravityRZ
Normal user
Posts: 206
Joined: 23 Dec 2019, 21:24

pulsecounter acting strange after latest changes

#1 Post by GravityRZ » 29 Apr 2020, 09:40

with build 20200410 i had a completely stable solution for at least 3 weeks
no extra or missing pulses.

now with build 20200426 i am experiencing strange things
i changed my rules script so i could use the setpulsecountertotal command instead of a dummy -device

first day i experienced that the pulsecounter total was missing 1 pulse
no rules involved because this happens directly in the plugin

to be save i restored the old rules which use count=1 as a trigger and increment a dummy device total
in this setup the dummy device total had an extra pulse.
strange thing in this situation is that the pulsecounter total was right

no reboots happened at that time so i suspect the changes made regarding restoring RTC values did not create this, nor did using the 4th var as a value.
at the moment i am going back to 20200410 to see if i still have a stable solution again

@TD-er,

can you see if changes between 20200410 and 20200426 has something to do with this

TD-er
Core team member
Posts: 8642
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: pulsecounter acting strange after latest changes

#2 Post by TD-er » 29 Apr 2020, 11:15

OK, can you itemize what parts you think may have different behavior?
Reading these changes in story-like format are a bit hard to interpret and itemizing may help to categorize what things have been checked and ruled out and what not.

Just by reading your post (for a few times), it looks like:
- the plugin itself does record an extra pulse
- You made sure it was never a count>1 between intervals. (or at least not in the run with the old rules)

Do you still use the setpulsecountertotal command?
If so, it could be you set it to some value and almost immediately there is a new pulse registered.
Any offset should then be visible right after setting the counter total.

GravityRZ
Normal user
Posts: 206
Joined: 23 Dec 2019, 21:24

Re: pulsecounter acting strange after latest changes

#3 Post by GravityRZ » 29 Apr 2020, 12:21

TD-er wrote: 29 Apr 2020, 11:15 OK, can you itemize what parts you think may have different behavior?
Reading these changes in story-like format are a bit hard to interpret and itemizing may help to categorize what things have been checked and ruled out and what not.

Just by reading your post (for a few times), it looks like:
- the plugin itself does record an extra pulse
CORRECT: it is adding an extra pulse in case of the 20200426 firmware and with the old rules it is sometimes missing a pulse
i check on Count=1 but this does not change the total because the plugin does that

- You made sure it was never a count>1 between intervals. (or at least not in the run with the old rules)

Do you still use the setpulsecountertotal command?
Only on a cold boot, i retrieve the value from domoticz and set it to the pulsecounter so this does not account for extra/missing pulses
If so, it could be you set it to some value and almost immediately there is a new pulse registered.
Any offset should then be visible right after setting the counter total.
i added the complete setup here with the 3 scenarios
in a nutshell
20200410 build using count=1 as trigger to increment dummy counter is stable

20200426 using build in pulsecounter total is adding an extra pulse
20200426 build using count=1 as trigger to increment dummy counter is missing a pulse
Attachments
esp-devices-rules.rar
(153.73 KiB) Downloaded 235 times

TD-er
Core team member
Posts: 8642
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: pulsecounter acting strange after latest changes

#4 Post by TD-er » 29 Apr 2020, 12:28

GravityRZ wrote: 29 Apr 2020, 12:21 [...]
20200426 using build in pulsecounter total is adding an extra pulse
20200426 build using count=1 as trigger to increment dummy counter is missing a pulse
This does look like it is still possible the pulse counter sometimes has "count=2".
Can you extend the rules slightly to store the counts where count > 1? You can use the %eventvalue% for that.

GravityRZ
Normal user
Posts: 206
Joined: 23 Dec 2019, 21:24

Re: pulsecounter acting strange after latest changes

#5 Post by GravityRZ » 29 Apr 2020, 13:00

do you mean like a test to see if there are pulses with values higher then 1

can you explain how to do this with %eventvalue% because i do not understand

TD-er
Core team member
Posts: 8642
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: pulsecounter acting strange after latest changes

#6 Post by TD-er » 29 Apr 2020, 13:15

Yep, that's what I mean.

Just taken from your Word document:

Code: Select all

On Watermeter#Count do						// When Pulse is detected
	if [Watermeter#Count] = 1 				// if pulse is received
		TaskValueSet 3,4,[Water#CounterTotal]+1  	// increase Water#CounterTotal with 1 liter
		SendToHTTP 192.168.1.50,8084,/json.htm?type=command&param=udevice&idx=344&nvalue=0&svalue=[Water#CounterTotal]
		TaskValueSet 3,1,60000/[Watermeter#Time]	// set Water#Flow
		SendToHTTP 192.168.1.50,8084,/json.htm?type=command&param=udevice&idx=338&nvalue=0&svalue=[Water#Flow]
	endif
		if [Water#Flow] > [Water#MaxFlow]
		TaskValueSet 3,3,[Water#Flow]			// copy Water#Flow to Water#MaxFlow
EndOn
Not going to make it all, but just as illustration:

Code: Select all

On Watermeter#Count do						// When Pulse is detected
	if [Watermeter#Count] = 1 				// if pulse is received
	  // Normal operations, do your stuff.
	else
	  // Not normal so it is either 0 or > 1
	  // Just assuming you have a dummy task and task variable called [debug#total] at task position 12, variable position 1
	  TaskValueSet 12,1,[debug#total]+[Watermeter#Count]
	endif
EndOn
Just based on your description, it is incorrect behavior if the count > 1, but if you ignore it you will loose counts.
So maybe a simple fix in your rules can be to count any count > 1 as just a single count.

Code: Select all

	if [Watermeter#Count] >= 1 				// if pulse is received

GravityRZ
Normal user
Posts: 206
Joined: 23 Dec 2019, 21:24

Re: pulsecounter acting strange after latest changes

#7 Post by GravityRZ » 29 Apr 2020, 14:15

ok, that i can do(the %eventvalue% was not clear)

that would proof why pulses are missing but that will not proof where the extra pulse came from because that one is added directly to the total of the pulsecounter.

in this thread we tried to solve a problem regarding the RTC values which turned out to be a different problem becaue Bulthaup was using a formula
https://www.letscontrolit.com/forum/vie ... f=6&t=6926
i remember that you changed a sequence just to be sure this was not a problem.
i think it was this issue
https://github.com/letscontrolit/ESPEasy/pull/2996

could this have caused it or is this change already in build 20200410

TD-er
Core team member
Posts: 8642
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: pulsecounter acting strange after latest changes

#8 Post by TD-er » 29 Apr 2020, 15:40

About the %eventvalue%

Just have a look at this part of your own rules:

Code: Select all

On Watermeter#Count do
The event starts with: "Watermeter#Count"
But the actual event is: "Watermeter#Count=1" (or =2..., you get the idea)
So the event has a value, which exists after the "=" sign.
In the rules you can refer to this value using %eventvalue%
So just like referring to [Watermeter#Count], which could have changed, you can also use %eventvalue% in the rules.
The content of %eventvalue% is the same as it was at the moment the event was fired.
Since the pulse counter acts on interrupts, it is possible the actual value of [Watermeter#Count] has changed by the time you refer to it from rules.


That pull request you refer to was already merged in build 20200410.

GravityRZ
Normal user
Posts: 206
Joined: 23 Dec 2019, 21:24

Re: pulsecounter acting strange after latest changes

#9 Post by GravityRZ » 29 Apr 2020, 15:55

ah,
finally clear to me

i let it run for a couple of days to see if it stays stable with current build 20200410(it did for 3 weeks)
after that i only update the firmware and let it run again for a couple of days.
finally i will change the rules to see if that affects things and will start the debugging

GravityRZ
Normal user
Posts: 206
Joined: 23 Dec 2019, 21:24

Re: pulsecounter acting strange after latest changes

#10 Post by GravityRZ » 15 May 2020, 17:32

well i did a lot of testing but sofar

after a week instantly 4 extra pulses
after another week of testing 1 missing pulse

i tested if the pulsecounter was triggered on values higher then 1 but this is not the case
i also did set the pulsecounter to the total amount and that value always matched to the one on the dummy device

so it looks like the extra pulse is cominf from the pulsecounter itself

because it ran completely stable for 3 weeks eith the previous build i am going to downgrade to version 20200410 or even 20200328

i will let it run again for another 3 weeks
if in those 3 weeks i do not get any extra or missing pulses then we might need to revert some changes we made in the p003 pulsecounter

not the setpulsecountertotal but the things before that.

will get back after 3 weeks

#update with 20200328 2 missing pulses so going back to 20200328

GravityRZ
Normal user
Posts: 206
Joined: 23 Dec 2019, 21:24

Re: pulsecounter acting strange after latest changes

#11 Post by GravityRZ » 02 Jun 2020, 13:17

ok.

it seems that after running stable for so long i tried to looosten up the settings
i apparently changed the wifi no sleep

after setting wifi to no sleep again the system seems stable also with the 20200426 build.

it seems that this one setting can cause extra pulses

could it be that some way when the wifi is sleeping and a pulse is counted the system comes back to life and again registers a pulse?

it has nothing to do with pulses missing because the total is calculated in the esp instead of sending over 1 pulse at a time

TD-er
Core team member
Posts: 8642
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: pulsecounter acting strange after latest changes

#12 Post by TD-er » 02 Jun 2020, 22:06

If sleeping wifi was responsible for less pulses, then it would make sense.
I can't explain what you're experiencing.

GravityRZ
Normal user
Posts: 206
Joined: 23 Dec 2019, 21:24

Re: pulsecounter acting strange after latest changes

#13 Post by GravityRZ » 11 Jan 2021, 16:03

@TD-er
i let this one run for a very long time but i still get some extra pulses some time.

to make things easy i use the direct numbers from the pulsecounter itself
eg. the Watermeter#total is the exact watermeternumber which i set with SetPulseCounterTotal

so if here is an extra pulse we know for sure it is coming from the pulsecounter itself(do to waterpressure jitter or a bug in the debouncetime functionality)

now rather than trying to find the cause i want to correct it.
i will try to measure the Watermeter#Time and if i record a low value(below 2000) i know it is caused by a faulty reading

now my question is

what is the best way to intercept this: i do not know if all the variables of the pulsecounter(Count, Total, Time do update at the same time)

On Watermeter#Count do (i use a 1 second interval on the pulsecounter so it probes every second)
If Watermeter#count=1

or

On Watermeter#Time do
if Watermeter#Time <2000


and should i test the Watermeter# variables or is it better to use %eventvalue%
if %eventvalue% <2000

if i can figur out how to intercept faulty reading i can then autocorrect is by substracting a pulse from the total

pulsecounter.jpg
pulsecounter.jpg (32.27 KiB) Viewed 7560 times
pulsecounter2.jpg
pulsecounter2.jpg (79.83 KiB) Viewed 7557 times

TD-er
Core team member
Posts: 8642
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: pulsecounter acting strange after latest changes

#14 Post by TD-er » 11 Jan 2021, 16:41

You should always use the %eventvalue% notation in handling events.
The reason is very simple;
Events may be queued and processed one at a time.
So if you handle an event, the event values were stored at the time the event was created and added to the queue, but the task values they reflect may have been changed at the moment you handle the event.

This is an issue if two of the same events end up in the event queue before they are handled.
When an event from the queue is being handled, only one event is processed in a loop.
Also the event queue is only handled if the ESP doesn't have some urgent scheduled things to do, like the 50/sec call etc.
Just take a look at the timing stats page to see if there are pieces of blocking code taking a long time to execute. Some may take several seconds, which make it very plausible the event queue may contain the same event twice (with different event values).
If you then look at the task values instead of the event values, you may see the values reflecting the later event in the queue. Especially for counters this may result in extra registered counts.

GravityRZ
Normal user
Posts: 206
Joined: 23 Dec 2019, 21:24

Re: pulsecounter acting strange after latest changes

#15 Post by GravityRZ » 11 Jan 2021, 17:58

looked at the timimgstats but do not know what to look for. i attached it


regarding the queue

i only had 1 On --Do command which was active on a 1 second base. i now added a second one to record possible low timings and store it in Maxflow for the time being
if i record a reading lower than 2000 and find out i have an extra pulse i know how to solve it.

Code: Select all

On System#Boot do     						// When the ESP boots, do
	TaskValueSet 3,2,0					// TaskValueSet TASKnr,VARnr,Value, Reset the Flow counter to 0
EndOn

On MQTTimport#Connected do
	if [Watermeter#Total] = 0
	SendToHTTP 192.168.1.50,8084,json.htm?type=command&param=switchlight&idx=334&switchcmd=On //initiate mqtt broadcast
	endif	
Endon

On DomoticzWatermeter#CounterTotal do
	if [Watermeter#Total] = 0
	SetPulseCounterTotal,[DomoticzWatermeter#CounterTotal],1
	endif
Endon

On Clock#Time=All,00:00 do					
	TaskValueSet 3,1,[Watermeter#Total]			
EndOn


On Watermeter#Count do						// When Pulse is detected
	if [Watermeter#Count] = 1 				// if pulse is received
		Publish domoticz/in,'{"idx":344,"nvalue":0,"svalue":"[Watermeter#Total]"}'
		TaskValueSet 3,2,60000/[Watermeter#Time]	// set Water#Flow
		Publish domoticz/in,'{"idx":338,"nvalue":0,"svalue":"[DomoticzWatermeter#Flow]"}'
		TaskValueSet 3,4,[Watermeter#Total]-[DomoticzWatermeter#CounterTotal]
	endif
Endon	
On Watermeter#Time do	
	if %eventvalue% < [DomoticzWatermeter#MaxFlow]
	TaskValueSet 3,3,%eventvalue%	// copy time to maxflow
	endif
EndOn
timingstats.jpg
timingstats.jpg (504.28 KiB) Viewed 7545 times

TD-er
Core team member
Posts: 8642
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: pulsecounter acting strange after latest changes

#16 Post by TD-er » 11 Jan 2021, 21:11

When looking at your rules, I notice you have 2 entries for the "Watermeter"
When handling the Watermeter#Count event, you also look at the [Watermeter#Time] value.
However, the order of values of the pulse plugin is:
- Count
- Total
- Time

That's also the order in which the events are sent.
So if you process the #Count event, the #Time event is already placed in the queue.
It probably doesn't matter a lot, as long as those events are processed before the new events of that plugin are added to the queue.

If a new event is added to the event queue before the others are processed, you're in trouble.

So what you may want to do, is store the event value of #Count in a variable and use that one in the handling of #Time. (as that's the last one in a row)
This way you don't have to check the task values but can rely on the event values.

TD-er
Core team member
Posts: 8642
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: pulsecounter acting strange after latest changes

#17 Post by TD-er » 11 Jan 2021, 23:11

Based on this discussion I added a new feature and will now make a test build for you.

See the last commit on this PR: https://github.com/letscontrolit/ESPEasy/pull/3439

TD-er
Core team member
Posts: 8642
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: pulsecounter acting strange after latest changes

#18 Post by TD-er » 12 Jan 2021, 00:03

See for test build:
https://github.com/letscontrolit/ESPEas ... -758278897

This allows you to set the checkbox to combine all values of a task in a single event.
Image

GravityRZ
Normal user
Posts: 206
Joined: 23 Dec 2019, 21:24

Re: pulsecounter acting strange after latest changes

#19 Post by GravityRZ » 12 Jan 2021, 10:04

thanks.

the second on Watermeter#Time was only added to see if i can spot an anommaly (very low time)
normally i only have 1 on Watermeter event and with that one i still get extra pulses.

regarding your extra functionality.

1 event only has one %eventvalue%, this is the reason i masde the second event(to capture the %eventvalue% which represends the Time)

is it possible that when an event is launched we get all the different eventvalues so we can check on those

EG
On Watermeter#Count do
%eventvalue1% (represents Count)
%eventvalue2% (represents Total)
%eventvalue3% (represents Time)

UPDATE:
looking at your code this is exactly what you mean, very nice.
will test the new build.

FYI:
the Time represents the amount of liters
i checked and in my situation the max amount of liters per minute is 28
if i use 30 liters this means 2000ms (60000/2000)

so if i spot a time lower than 2000ms this means it is an anomally which could be a sign of an exptra pulse
i suspect that an extra pulse time will be much lower than 2000ms but below 2000ms is save.

GravityRZ
Normal user
Posts: 206
Joined: 23 Dec 2019, 21:24

Re: pulsecounter acting strange after latest changes

#20 Post by GravityRZ » 12 Jan 2021, 11:10

UPDATE:

eveything seems to be working after all

important to mention is that if we use the values combined then we MUST USE #All
aparently the #Count, #Total and #Time do not work to trigger events

So in my code i needed to change Watermeter#Count into Watermeter#All

Code: Select all

On Watermeter#Count do						// When Pulse is detected
	if [Watermeter#Count] = 1 				// if pulse is received
		Publish domoticz/in,'{"idx":344,"nvalue":0,"svalue":"[Watermeter#Total]"}'
		TaskValueSet 3,2,60000/[Watermeter#Time]	// set Water#Flow
		Publish domoticz/in,'{"idx":338,"nvalue":0,"svalue":"[DomoticzWatermeter#Flow]"}'
		TaskValueSet 3,4,[Watermeter#Total]-[DomoticzWatermeter#CounterTotal]
	endif
	if %eventvalue3% < [DomoticzWatermeter#MinTime]
	TaskValueSet 3,3,%eventvalue3%	// copy time to MinTime
	endif
EndOn

mqttimport device.jpg
mqttimport device.jpg (134.28 KiB) Viewed 7504 times

TD-er
Core team member
Posts: 8642
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: pulsecounter acting strange after latest changes

#21 Post by TD-er » 12 Jan 2021, 12:58

Yep, indeed you need to change the event to use "#All" in the rules block handling the event.

I'm glad we had this discussion, as it gave me an idea for a very simple optimization which also makes the rules a lot less complex in the end.
Especially on plugins like the pulse counter which can be updated very irregularly.

I guess further optimizations on the pulse counter are mainly on the accuracy of the pulse counting and thus may probably only be possible on ESP32 as that one has proper interrupts on GPIO pins and can provide a lot more insight in the actual pulse received.

Post Reply

Who is online

Users browsing this forum: No registered users and 20 guests