Pulse Counter P003 enhancement. How to give back?

Moderators: grovkillen, Stuntteam, TD-er

Message
Author
Kim_Panda
Normal user
Posts: 40
Joined: 25 Jan 2021, 17:03

Re: Pulse Counter P003 enhancement. How to give back?

#51 Post by Kim_Panda » 14 Mar 2021, 13:36

TD-er wrote: 14 Mar 2021, 00:37 RTC memory is not flash.
RTC memory is just memory that doesn't get erased at a reboot, as long as the power remains.
Thanks TD-er for that good hint! I was not aware of that fact and I did a quick research which now ties pieces together for me.
What is called UserVar (UserVarStruct) in P003 is actually utilizing the RTC-Memory :idea: (the 512 bytes of free usable memory of the RealTimeClock which survives a deep sleep or reset, as long as power is on). Cf. ESPEasyRTC.cpp.
This explains, why the P003 counters survives a warm-reset and thus also a deep sleep.
I wrongly thought, that this is used in other plugins, too, but it seems not.
The current implementation limits the use to 4 of these 4 byte variables per plugin task.
:idea: I learned a lot. Thanks.

TD-er
Core team member
Posts: 4507
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Pulse Counter P003 enhancement. How to give back?

#52 Post by TD-er » 14 Mar 2021, 14:32

You're welcome :)

Kim_Panda
Normal user
Posts: 40
Joined: 25 Jan 2021, 17:03

Re: Pulse Counter P003 enhancement. How to give back?

#53 Post by Kim_Panda » 15 Mar 2021, 23:39

Hi TD-er,
I have finally completed my P003 Pulse Counter enhancement and issue a Pull Request in a minute.
I followed your "cook book" for pull requests, from above.
I hope I followed the procedure as intended. It's not easy doing it for the first time.

You will see, that there are a lot of changes in the code. The first was adding
comments and introducing #defines with meaningful names so that I and hopefully
others can more easily understand the code. And secondly the enhanced PULSE functionality
as earlier promised in this dialogue.

In addition I prepared an update for https://espeasy.readthedocs.io/en/lates ... /P003.html
in a Word file and a copy as .pdf, so you can easily take it over into the ESPEasy documentation.

Please have a look, what I produced and check if code and docu fits into ESPEasy.
I tested a lot, but, for sure, I did not test all possible use cases and combinations.
As soon as you released a new build, @GravityRZ may test his use case, as others may also do
and my attached documentation should help to successfully use the new functionality.
Attachments
ESPEasy_P003_2.2_Docu.zip
Docu for new Mode types in P003
(88.65 KiB) Downloaded 16 times

User avatar
Ath
Normal user
Posts: 691
Joined: 10 Jun 2018, 12:06
Location: NL

Re: Pulse Counter P003 enhancement. How to give back?

#54 Post by Ath » 16 Mar 2021, 07:53

Kim_Panda wrote: 15 Mar 2021, 23:39 In addition I prepared an update for https://espeasy.readthedocs.io/en/lates ... /P003.html
in a Word file and a copy as .pdf, so you can easily take it over into the ESPEasy documentation.
That documentation is in fact part of the github repository, please check out the docs folder. It is in Sphinx format. Will take a little time getting used to (at least it was for me ;)), but this way you're able to create documentation as part of a PR. It's published once the nightly build, containing the merged PR, is created.

When coding using VSCode, you can create a local build by opening a terminal, 'cd docs' then 'make.bat html' and after some grundling and a couple of (mostly ignorable) (error)messages you can open index.html from docs\build\html to see your result. (You will need to have Python 3.8 or newer installed locally, and initially run 'pip3 install -r requirements.txt')
/Ton

Kim_Panda
Normal user
Posts: 40
Joined: 25 Jan 2021, 17:03

Re: Pulse Counter P003 enhancement. How to give back?

#55 Post by Kim_Panda » 17 Mar 2021, 17:14

Kim_Panda wrote: 13 Mar 2021, 19:47
GravityRZ wrote: 13 Mar 2021, 15:24 EDIT: or am i misinterpreting things that the above descibed total time(pulsehigh+pulselow) is not yet in the 20210223 build
You are right, the new version is not yet available. Coming soon. I will let you know here. :ugeek:
[/quote]


@GravityRZ:
Here we are! I completed my work on the P003 enhancement now.
My source is available in
https://github.com/KimPanda4/ESPEasy/tr ... nhancement
and I hope that it will now be pulled into the official ESPEay version and included in the next build.
You can follow the progress here:
https://github.com/letscontrolit/ESPEasy/pull/3545

When next ESPEasy build is released, I hope you find time to test the new PULSE modes for your application. Have fun!

GravityRZ
Normal user
Posts: 194
Joined: 23 Dec 2019, 21:24

Re: Pulse Counter P003 enhancement. How to give back?

#56 Post by GravityRZ » 17 Mar 2021, 18:16

thanks,
will test it as soon as it get's released

TD-er
Core team member
Posts: 4507
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Pulse Counter P003 enhancement. How to give back?

#57 Post by TD-er » 17 Mar 2021, 18:46

GravityRZ wrote: 17 Mar 2021, 18:16 thanks,
will test it as soon as it get's released
Maybe you can test the GitHub Actions test build ?

GravityRZ
Normal user
Posts: 194
Joined: 23 Dec 2019, 21:24

Re: Pulse Counter P003 enhancement. How to give back?

#58 Post by GravityRZ » 17 Mar 2021, 20:03

that page points to a 404 page(not found)

TD-er
Core team member
Posts: 4507
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Pulse Counter P003 enhancement. How to give back?

#59 Post by TD-er » 17 Mar 2021, 20:10

Can you see this page: https://github.com/letscontrolit/ESPEas ... /661555319
?
At the bottom of the right panel it links to "binaries"
Maybe the URL changes per session.

GravityRZ
Normal user
Posts: 194
Joined: 23 Dec 2019, 21:24

Re: Pulse Counter P003 enhancement. How to give back?

#60 Post by GravityRZ » 17 Mar 2021, 20:34

did a quick test with the normal build. 4M1M

when set to pulselow no pulse is registered. nothing is counted
when set to falling it works
when set back to pulse low still no pulse registered

no time anymore tonight.

can test further tomorrow but for now i switched back to the 0223 build

TD-er
Core team member
Posts: 4507
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Pulse Counter P003 enhancement. How to give back?

#61 Post by TD-er » 17 Mar 2021, 21:25

Thanks for testing for now.

Kim_Panda
Normal user
Posts: 40
Joined: 25 Jan 2021, 17:03

Re: Pulse Counter P003 enhancement. How to give back?

#62 Post by Kim_Panda » 18 Mar 2021, 02:23

I also did a quick test with ESP_Easy_mega_20210317_normal_ESP8266_4M1M.bin
For me its working in both modes.
@GravityRZ: Please see what Debounce time you use. You may need to adjust it.
When its too long, nothing is counted, if the pulse is not stable after Debounce time for another period of the same length.
The rising/falling mode, however, does count any edge after end of debounce, regardless if stable or not.
In my tests I had the best results with Debounce Time of 1/10 to 1/4 of the shortest expected pulse length.
So when using PULSE low and your shortest low period is 500ms, I would start with 80 ms for bouncing.
When signal is then after 80ms stable for another 80ms it is counted. The remaining time up to 500 is then a safety buffer for retries.

If it does not work with shorter Debounce time, you may set the log level to "debug" and inspect the P003 statistical log entries

Code: Select all

# good example
1608232 : Debug  : P003:Stats[step0|1|2|3|tot(ok/nok/ign)] [lo|hi]: [41|33|33/0|32/0/1|32] [609|1311]
1608590 : Debug  : P003:Stats[step0|1|2|3|tot(ok/nok/ign)] [lo|hi]: [43|34|34/0|33/0/1|33] [358|1311]
1609229 : Debug  : P003:Stats[step0|1|2|3|tot(ok/nok/ign)] [lo|hi]: [44|35|35/0|34/0/1|34] [358|638]
# bad signal or too long debounce time:
1716342 : Debug  : P003:Stats[step0|1|2|3|tot(ok/nok/ign)] [lo|hi]: [118|45|53/2|35/8/10|35] [141|106972]
1716524 : Debug  : P003:Stats[step0|1|2|3|tot(ok/nok/ign)] [lo|hi]: [122|46|54/3|35/8/11|35] [141|107154]
1716656 : Debug  : P003:Stats[step0|1|2|3|tot(ok/nok/ign)] [lo|hi]: [124|47|55/3|35/8/12|35] [141|107285]
1716882 : Debug  : P003:Stats[step0|1|2|3|tot(ok/nok/ign)] [lo|hi]: [130|48|56/4|35/8/13|35] [141|107511]
When the number after the slashes ("/") increase while the Total (here "|35]" not changes, no stable signal is detected.
More details are in the docu.
Thanks for testing.

TD-er
Core team member
Posts: 4507
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Pulse Counter P003 enhancement. How to give back?

#63 Post by TD-er » 18 Mar 2021, 08:52

@Kim_Panda
Does this mean existing settings may no longer work after update?
If it is required to change the settings, we do have a mechanism to add a transition to the settings.

See the function BuildFixes() which is called if BUILD (stored in the settings) is not the same anymore.
So we may also need to update that value.

Kim_Panda
Normal user
Posts: 40
Joined: 25 Jan 2021, 17:03

Re: Pulse Counter P003 enhancement. How to give back?

#64 Post by Kim_Panda » 18 Mar 2021, 10:46

TD-er wrote: 18 Mar 2021, 08:52 Does this mean existing settings may no longer work after update?
I think no, because the existing settings should work unchanged,
but the new PULSE mode does now the additional checks in order to be more accurate.
The old modes were in this sense more tolerant, but not as accurate.

Let's see, what GravityRZ finds.
@GravityRZ: as an initial start, you can start with Debounce time of 20ms or even 0 (then you have the implicit indefinite delay between (0-20ms) of the ESPEasy's function PLUGIN_FIFTY_PER_SECOND) and then increase the DB time when too many pulses are counted.
What type of signal are you counting? How long is the signal typically low/high?

GravityRZ
Normal user
Posts: 194
Joined: 23 Dec 2019, 21:24

Re: Pulse Counter P003 enhancement. How to give back?

#65 Post by GravityRZ » 18 Mar 2021, 12:55

Hi Kim_Panda.

will test again

i had the debounce set to 280ms

the pulse is low approx 500ms and high 5500ms


normally i added a magnet to the pulsesensor to make it more sensitive so the low time increases.
this also avoided extra pluses when the sensor is on the edge of detecting a pulse (eg very close so it might jitter, on/off)
with the normal falling pulse this resulted in a way more accurate reading(but not 100%)

so at the moment i am trying worst case scenario without the magnet.
if i get stable results i will perhaps use the magnet again to incease accuracy but i was hoping your false detection will solve it.

i would expect that when i select a really low debounce like 20ms it will block jitters and hopefully also will block extra close proximity jitters because of your algoritm used

TD-er
Core team member
Posts: 4507
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Pulse Counter P003 enhancement. How to give back?

#66 Post by TD-er » 18 Mar 2021, 13:32

How is your reed switch electrically connected?
Does it have a clear state when "open"? (e.g. using a pull-up resistor)
If not, then it may also act as an antenna.

GravityRZ
Normal user
Posts: 194
Joined: 23 Dec 2019, 21:24

Re: Pulse Counter P003 enhancement. How to give back?

#67 Post by GravityRZ » 18 Mar 2021, 17:04

it is not a reed switch but proximity sensor (senses the proximity of metal)

the sensor itself is connected to 5V
the output is connected to an optocoupler and uses a 10K pull up resistor
the only problem area is when the metal is really close but the sensor is on the point of sensing it
this is when you could expect on/off situations

then there is offcourse the backfeed from sudden interuption of waterflow when this happens when it is also near the metal.

all and all it is getting pretty accurate but i am still aiming for that 100%

GravityRZ
Normal user
Posts: 194
Joined: 23 Dec 2019, 21:24

Re: Pulse Counter P003 enhancement. How to give back?

#68 Post by GravityRZ » 18 Mar 2021, 17:19

reflashed with 20210317 normal
selected debounce time to 20ms

pulses are coming in
total time seems to be the total time between pulses, nice work

i will leave it as is for now and will see if i get extra/missing pulses

if i read things right the plugin is doing a triple check on whether the pulse is legit (eg in 3 different times it reads the pulse the same)

if this works it would be a good solution for both jitter when the sensor is near the metal as well as backfeed from sudden pressure loss/increase

Kim_Panda
Normal user
Posts: 40
Joined: 25 Jan 2021, 17:03

Re: Pulse Counter P003 enhancement. How to give back?

#69 Post by Kim_Panda » 20 Mar 2021, 13:50

I published a small correction of a reference in the P003 Commands docu, because
I saw, that is was not appropriate in the context of the "Command Reference" page.

NB: The new PULSE counter is meanwhile since 3 weeks counting my gas meter 100% correctly (no false count at all 8-) ) with around 1000 pulses per day (low varying frequency) with a simple reed contact between GPIO2 and GND of an ESP-01 (4MB).
The shortest high or low signals are around 5-10 seconds but can also last for an hour or so.
I am using "PULSE change" with 200ms Debounce time and a typical send "Interval" of 900 sec (sometimes 10 sec. while testing). I am very satisfied with this result ;) .

GravityRZ
Normal user
Posts: 194
Joined: 23 Dec 2019, 21:24

Re: Pulse Counter P003 enhancement. How to give back?

#70 Post by GravityRZ » 21 Mar 2021, 10:48

HaHa,

while you are counting rather slow pulses i am counting faster pulses and i am also using a 1 seconds refresh which puts a lot of load on the system

here are the tests sofar
debounce time
20ms=+1 pulse
100ms= -1 pulse
40ms= -3 pulse
30ms=+1 pulse
35ms= -1 pulse

things are also dependent on the max flow at that moment. when there is a hifgh flow, the pulsecounter has less time to calculate if the pulse is valid so a high debounce might create missing pulses

i do a second check in rules that whenever the gpio changes it also counts with a different variable
eg i have 2 counters (1 pulsecounter and 1 normal counter) and these need to be the same, always.
using GPIO i noticed that this affects the pulsecounter counting(i had to lower to 20ms to get it to count correct)

i changed all that and now my second counter uses On Pulsecounter#Count

sofar looking good

regarding the load, it now is at about 38% percent (pulsecounter on 1 second interval, taskcounter on 60 second interval) so no problems there either

the extra/missing pulses are probably caused by the counter being on the verge of changing.
it looks like the Low functionality is taking care of that but i can live with that if it is not 100% perfect because this is a really difficult situation which changes a lot's

nevertheless, before i used a magnet to dampen these pulses but now i am doing it without, thanks to the new pulsecounter functionality.

Did you know that it actually counts a pulse AFTER it changes from low to high.
i found this out because the sensor was on and stopped exactly above the metal(so it stayed on and low)

it registered that pulse after i advanced the watermeter and the signal went from low to high

Kim_Panda
Normal user
Posts: 40
Joined: 25 Jan 2021, 17:03

Re: Pulse Counter P003 enhancement. How to give back?

#71 Post by Kim_Panda » 22 Mar 2021, 02:13

Hey, that sounds good! Thanks for testing.

Without knowing the details, I would say, that the situations where you count too much pulses (+1), the debounce time is too short and you are sometimes counting a bounce. Otherwise too few pulses (-1/-3) can mean, that the debounce time is too long and you miss a pulse. In your case both meets at about 35ms which could mean, that it is a very extreme use case where the counter is at its limits. You have then after the debounce time (e.g. after 35ms) not always a stable pulse for another 35ms.

In my "rather slow case" I could use the statistical log records to analyse the counting situation, but setting the log level down to "debug" in your case could slow the system too much down or mess it up completely.
Nevertheless I am interested to see, what this statistic counters are in your case. Its lucky that the statistical counts are always running independent of the log level. What you could do (provided you are also interested) is, in a "quiet period", to open two ESPEasy web pages in parallel, one with the Log output open, the other with the Task Settings open and then set there the log level to "debug" for short period and capture some of the "P003:Stats..." entries. This gives in inside view what the counter was doing.
GravityRZ wrote: 21 Mar 2021, 10:48 Did you know that it actually counts a pulse AFTER it changes from low to high.
i found this out because the sensor was on and stopped exactly above the metal(so it stayed on and low)
Yes, you are right. the counting takes place, when the pulse has ended, because only then we know, how long the pulse was and provide the right "time" value.

Kim_Panda
Normal user
Posts: 40
Joined: 25 Jan 2021, 17:03

Re: Pulse Counter P003 enhancement. How to give back?

#72 Post by Kim_Panda » 22 Mar 2021, 02:24

@TD-er:
I am not quite sure, if I formally correctly completed the Pull request.
https://github.com/letscontrolit/ESPEas ... -612946234
Can you please have a look. I think I resolved all issues and the change is ready.

GravityRZ
Normal user
Posts: 194
Joined: 23 Dec 2019, 21:24

Re: Pulse Counter P003 enhancement. How to give back?

#73 Post by GravityRZ » 22 Mar 2021, 19:08

could you elaborate on this

1 screen open log got it (do i set this to debug)

2 Tasksettings open end set log level to debug(where in tasksettings do i do this)


Kim_Panda wrote: 22 Mar 2021, 02:13 Hey, that sounds good! Thanks for testing.

Without knowing the details, I would say, that the situations where you count too much pulses (+1), the debounce time is too short and you are sometimes counting a bounce. Otherwise too few pulses (-1/-3) can mean, that the debounce time is too long and you miss a pulse. In your case both meets at about 35ms which could mean, that it is a very extreme use case where the counter is at its limits. You have then after the debounce time (e.g. after 35ms) not always a stable pulse for another 35ms.

In my "rather slow case" I could use the statistical log records to analyse the counting situation, but setting the log level down to "debug" in your case could slow the system too much down or mess it up completely.
Nevertheless I am interested to see, what this statistic counters are in your case. Its lucky that the statistical counts are always running independent of the log level. What you could do (provided you are also interested) is, in a "quiet period", to open two ESPEasy web pages in parallel, one with the Log output open, the other with the Task Settings open and then set there the log level to "debug" for short period and capture some of the "P003:Stats..." entries. This gives in inside view what the counter was doing.
GravityRZ wrote: 21 Mar 2021, 10:48 Did you know that it actually counts a pulse AFTER it changes from low to high.
i found this out because the sensor was on and stopped exactly above the metal(so it stayed on and low)
Yes, you are right. the counting takes place, when the pulse has ended, because only then we know, how long the pulse was and provide the right "time" value.

TD-er
Core team member
Posts: 4507
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Pulse Counter P003 enhancement. How to give back?

#74 Post by TD-er » 22 Mar 2021, 22:35

Log level can be set in Tools -> Advanced.
You can set the log level for serial, a syslog service, web log and if included in the build log to SD card.

GravityRZ
Normal user
Posts: 194
Joined: 23 Dec 2019, 21:24

Re: Pulse Counter P003 enhancement. How to give back?

#75 Post by GravityRZ » 24 Mar 2021, 20:00

that was clear but i did not understood what to do in the second window

TD-er wrote: 22 Mar 2021, 22:35 Log level can be set in Tools -> Advanced.
You can set the log level for serial, a syslog service, web log and if included in the build log to SD card.

TD-er
Core team member
Posts: 4507
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Pulse Counter P003 enhancement. How to give back?

#76 Post by TD-er » 24 Mar 2021, 20:05

What is unclear?

Kim_Panda
Normal user
Posts: 40
Joined: 25 Jan 2021, 17:03

Re: Pulse Counter P003 enhancement. How to give back?

#77 Post by Kim_Panda » 25 Mar 2021, 13:39

GravityRZ wrote: 24 Mar 2021, 20:00 that was clear but i did not understood what to do in the second window

TD-er wrote: 22 Mar 2021, 22:35 Log level can be set in Tools -> Advanced.
You can set the log level for serial, a syslog service, web log and if included in the build log to SD card.
Here is an example screenshot of the two parallel windows
Log-Example.jpg
Log-Example.jpg (246.05 KiB) Viewed 929 times

GravityRZ
Normal user
Posts: 194
Joined: 23 Dec 2019, 21:24

Re: Pulse Counter P003 enhancement. How to give back?

#78 Post by GravityRZ » 25 Mar 2021, 13:47

ok clear now.

at the moment i have added the magnet again to extend the pulseLOW time.

this because i can not get a 100% correct reading all the time.

normally if i would extend the debouncetime it blocks the pulses for an extended time.
with the pulseLOW functionality it does the same but because it does triple checks it needs time for that as well

eg, when i select 30ms it has enough time to calculate but less time to filter so i get +1 pulse
when i select 35ms or higher it has more time to block pulses but it runs out of time probably because the pulse is to short.
now that i have extended the pulselenght i should be able to select much higher debounce times and hopefully filter out the wrong ones.

GravityRZ
Normal user
Posts: 194
Joined: 23 Dec 2019, 21:24

Re: Pulse Counter P003 enhancement. How to give back?

#79 Post by GravityRZ » 25 Mar 2021, 18:24

i will make a dump so we can see what is going on.
this is with the magnet attached so long pulse
i assume that these numbers show when the pulse is low, high, what time it is taking etc

Code: Select all

96407434: EVENT: Pulsecounter#Count=0
96407476: EVENT: Pulsecounter#Count=0 Processing time:42 milliSeconds
96407527: EVENT: Pulsecounter#Total=2134210
96407568: EVENT: Pulsecounter#Total=2134210 Processing time:41 milliSeconds
96407627: EVENT: Pulsecounter#Time=10325
96407666: EVENT: Pulsecounter#Time=10325 Processing time:40 milliSeconds
96407673: P003:Stats[step0|1|2|3|tot(ok/nok/ign)] [lo|hi]: [2135078|2134937|2134937/0|2134937/0/0|2134211] [3603|6423]
96414197: P003:Stats[step0|1|2|3|tot(ok/nok/ign)] [lo|hi]: [2135079|2134938|2134938/0|2134938/0/0|2134211] [3603|6524]
96417728: P003:Stats[step0|1|2|3|tot(ok/nok/ign)] [lo|hi]: [2135080|2134939|2134939/0|2134939/0/0|2134212] [3531|6524]
96424339: P003:Stats[step0|1|2|3|tot(ok/nok/ign)] [lo|hi]: [2135081|2134940|2134940/0|2134940/0/0|2134212] [3531|6610]
and this is without the magnet attached. pulse should be shorter

Code: Select all

101308468: EVENT: Pulsecounter#Count=0
101308507: EVENT: Pulsecounter#Count=0 Processing time:39 milliSeconds
101308568: EVENT: Pulsecounter#Total=2134319
101308607: EVENT: Pulsecounter#Total=2134319 Processing time:40 milliSeconds
101308668: EVENT: Pulsecounter#Time=6421
101308707: EVENT: Pulsecounter#Time=6421 Processing time:39 milliSeconds
101286931: P003:Stats[step0|1|2|3|tot(ok/nok/ign)] [lo|hi]: [2135302|2135146|2135146/0|2135146/0/0|2134315] [8412|133136]
101287549: P003:Stats[step0|1|2|3|tot(ok/nok/ign)] [lo|hi]: [2135303|2135147|2135147/0|2135147/0/0|2134316] [618|133136]
101293316: P003:Stats[step0|1|2|3|tot(ok/nok/ign)] [lo|hi]: [2135304|2135148|2135148/0|2135148/0/0|2134316] [618|5766]
101293876: P003:Stats[step0|1|2|3|tot(ok/nok/ign)] [lo|hi]: [2135305|2135149|2135149/0|2135149/0/0|2134317] [559|5766]
101299609: P003:Stats[step0|1|2|3|tot(ok/nok/ign)] [lo|hi]: [2135306|2135150|2135150/0|2135150/0/0|2134317] [559|5733]
101305950: P003:Stats[step0|1|2|3|tot(ok/nok/ign)] [lo|hi]: [2135308|2135152|2135152/0|2135152/0/0|2134318] [524|5815]

both with 40ms debounce time
flow is not to high(9 pulses per minute) so in this situation this would proably work

can you see anything which would cause it to mallfunction when the flow is around 25 pulses a minute

Kim_Panda
Normal user
Posts: 40
Joined: 25 Jan 2021, 17:03

Re: Pulse Counter P003 enhancement. How to give back?

#80 Post by Kim_Panda » 26 Mar 2021, 22:38

GravityRZ wrote: 25 Mar 2021, 18:24 can you see anything which would cause it to mallfunction when the flow is around 25 pulses a minute
The short answer to your question is "no". But here is the long answer:

Interpretation of the logs:
In both cases only clean pulses were detected, i.e. that all three read steps always did read the same level (low/high).
This applies for all the runtime since the last reset. This is indicated by the "/0" and "/0/0".

The "[3603|6523]" in the first case indicate, that (at logging time) the low signal was 3,603 seconds and the high 6,523 seconds.
In the second case the shortest low 524 ms ([524|5815]) and the longest high 133,136 seconds ([8412|133136]).

The difference of the first two numbers ([2135302|2135146|) show, that 156 signal edges were ignored in the debounce time.
The first "/0" shows, that no single debounce effect happened, so you could probably shorten the debounce time a little bit.

You see that the last number in the fist block (e.g. "|2134211]"), which is the TotalCount, is only incremented every second log entry, so you are not using "Pulse change".
I think you are using "Pulse low" because the TotcalCount is incremented whenever the length of the low signal was updated.

And the sequence of long numbers indicate e.g. for "[2135302|2135146|2135146/0|2135146/0/0|2134315]",
- that 2135302 times the signal changed from high to low (interrupt)
- 2135146 reads were done in step 1
- 2135146 reads in step 2 (no single wrong level ("/0|")
- 2135146 reads in step 3 (no single wrong level ("/0/") and no wrong spikes within a stable signal("/0]).
- And, when I calculated correctly, you manually set or increased the pule counter to/at 2133484,
- because with "PULSE low" and if you had started with TotalCount=0 the last large number (2134315) should be half of the the previous (2135146).
A lot of information, isn't it?

But it does not show, why you sometimes count +/- 1 pulse. :shock:

:!: What you could do is, to keep the counter running over a rush-hour period, and after that capture some log entries.
Then the numbers after the "/" still indicate, where possible problems may have been.

BTW: In the version, where I am currently working on (TD-er asked me to enhance code readability),
I have added some further "overdue indicators", which show, if the Pulse Counter plugin was blocked/delayed by other ESPEasy processing.
I did that, because I found in my tests of my new version with one of my older ESP8266s, that the Pulse Counter is blocked sometimes for several seconds. This ESP8266 has a bad antenna and frequent reconnects.
This would explain occasional wrong counting. This normally occurred after the message "Error : MQTT : Intentional reconnect". In this period only interrupts were processed, but not the normal stepped check processing in P003.
I intend to investigate this a little more by use of these "overdue indicators". But this is not yet in your current version and I could only observe the problem yet with one specific ESP8266.

TD-er
Core team member
Posts: 4507
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Pulse Counter P003 enhancement. How to give back?

#81 Post by TD-er » 27 Mar 2021, 08:53

Network activity, like "MQTT reconnect" can be so called "blocking" code.
Meaning no other code will be executed, except for interrupt code, maybe.
The reason for this "maybe" is that I am not sure if GPIO-triggered interrupts are at the same level as WiFi interrupts.

So what can happen here is that the ESP is busy processing the login (or simply an attempt to resolve a DNS request) and thus will not execute code that may process the recorded pulses.

What can be done to overcome this:

1) We can simply count the pulses and if > 1 we know we missed some and then assume those were valid triggers.
This has a relative high risk of counting invalid triggers.

2) We can make the interrupt driven code slightly more complex by looking at the current state of the debounce (e.g. rising edge seen, this is a falling edge, compute time inbetween) instead of doing this when processing the collected values from the interrupts.
This may need disabling/enabling interrupts, and take longer to process thus limiting the max. possible frequency that can be counted.

3) We can add more data to be collected so we can record upto 2 (or N) samples between 2 processing steps.
This does take more memory, but maybe we can store it in a dynamically allocated std::vector, so it only takes resources when actually used.
I don't think our std::vector implementation does use bound checking, so it hardly is an overhead in computation time. Not sure what to use as an index though. Taskindex?


Problem remains that we don't know up front how long code can be blocking.
For example, a DNS resolve may block for as long as the set timeout in a controller (default 1000 msec) Sometimes it is tried twice => 2 sec blocking.
One way to reduce the chance of DNS lookup blocking calls is to address your controller via IP instead of DNS.
The same for NTP, but that may require you to set a local NTP server (e.g. some routers support it)

Kim_Panda
Normal user
Posts: 40
Joined: 25 Jan 2021, 17:03

Re: Pulse Counter P003 enhancement. How to give back?

#82 Post by Kim_Panda » 27 Mar 2021, 14:46

TD-er wrote: 27 Mar 2021, 08:53 Network activity, like "MQTT reconnect" can be so called "blocking" code.

:( hmmm, bad news.
Had somebody thought about making these IP-drivers non blocking (asynchronous) ?
From an idealistic point of view, this would be the obvious thing to do, rather than trying to sanitize this problem in various unrelated places? ... I know, that is not easy and not quickly done.
I saw that the "ESP_WiFiManager" project
https://github.com/khoih-prog/ESP_WiFiM ... er-library is working on that with their ESPAsync_WiFiManager
https://github.com/khoih-prog/ESPAsync_WiFiManager. I don't know if they got it stable now. Some months ago, I did a try, but couldn't get the async version running. May be they are ready by now.
What libraries is ESPEasy based on?

Kim_Panda
Normal user
Posts: 40
Joined: 25 Jan 2021, 17:03

Re: Pulse Counter P003 enhancement. How to give back?

#83 Post by Kim_Panda » 27 Mar 2021, 14:52

TD-er wrote: 27 Mar 2021, 08:53 Meaning no other code will be executed, except for interrupt code, maybe.
The reason for this "maybe" is that I am not sure if GPIO-triggered interrupts are at the same level as WiFi interrupts.
I think you are right. I think I saw in my test, that interrupts were still counted in the blocking situation.

TD-er
Core team member
Posts: 4507
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Pulse Counter P003 enhancement. How to give back?

#84 Post by TD-er » 27 Mar 2021, 14:59

Kim_Panda wrote: 27 Mar 2021, 14:46
TD-er wrote: 27 Mar 2021, 08:53 Network activity, like "MQTT reconnect" can be so called "blocking" code.

:( hmmm, bad news.
Had somebody thought about making these IP-drivers non blocking (asynchronous) ?
From an idealistic point of view, this would be the obvious thing to do, rather than trying to sanitize this problem in various unrelated places? ... I know, that is not easy and not quickly done.
[...]
What libraries is ESPEasy based on?
ESPEasy has its own WiFi code.
The main reason is that when I tried to fix WiFi issues all existing libraries also had those issues and there was none for ESP32.

By the way the network connection methods used in ESPEasy have been asynchronous for at least 3 years now.
Still some callbacks are not, like WiFiClient.connect and performing a DNS resolve.
Also we don't use the async webserver since that one was somewhat "pre-alpha" stage for quite some time when most of the development for ESPEasy's way of serving web pages was done.

Kim_Panda
Normal user
Posts: 40
Joined: 25 Jan 2021, 17:03

Re: Pulse Counter P003 enhancement. How to give back?

#85 Post by Kim_Panda » 28 Mar 2021, 00:34

TD-er wrote: 27 Mar 2021, 14:59 By the way the network connection methods used in ESPEasy have been asynchronous for at least 3 years now.
Still some callbacks are not, like WiFiClient.connect and performing a DNS resolve.
Ahh, this sounds much better.
Thanks for explanation.

Kim_Panda
Normal user
Posts: 40
Joined: 25 Jan 2021, 17:03

Re: Pulse Counter P003 enhancement. How to give back?

#86 Post by Kim_Panda » 28 Mar 2021, 01:33

TD-er wrote: 27 Mar 2021, 08:53 What can be done to overcome this:
Your three approaches are valid, each with its disadvantages.
And when the only place with a certain guaranty for timely processing is in the trigger,
I agree, that the best approach is, to move some more functionality in the trigger and use the time difference between the edges with the principle:
a) we trigger always on "change"
b) when the time difference between rising and next falling is long enough, it is regarded as valid high pulse. And for low pulse respectively.
Then my current triple check is not needed any more, but still the validity checks in step 3.
But it is basically a new code with the disadvantage, that the runtime of the ISR increases and might though that delay other tasks.

From my point of view, I think I can live with the current enhanced pulse counter, as with my normal ESP8266, the counter still woks in production 100% perfect for my low frequency use case.
And with the new "overdue" statistics which I a have added my new version, we can probably see, what causes the +/- 1 counts in GravityRZ's use case.

I just published my updated version, so its now ready for your takeover.
When you made it available for GravityRZ, he could test with the new OverDueStats and see if that helps to clear his counter losses.
Here is the graphical instruction.
logging.png
logging.png (145.33 KiB) Viewed 803 times

TD-er
Core team member
Posts: 4507
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Pulse Counter P003 enhancement. How to give back?

#87 Post by TD-er » 28 Mar 2021, 10:11

I added some remarks to your PR.

Kim_Panda
Normal user
Posts: 40
Joined: 25 Jan 2021, 17:03

Re: Pulse Counter P003 enhancement. How to give back?

#88 Post by Kim_Panda » 28 Mar 2021, 17:18

TD-er wrote: 28 Mar 2021, 10:11 I added some remarks to your PR.
All changes are for your takeover now in
https://github.com/KimPanda4/ESPEasy/tr ... nhancement

I did a successful quick test and also a successful test build for custom_ESP8266_4M1M and custom_ESP32_4M316k.

Do I need to press the "Resolve conversation" button in the PR? What is the purpose for that button?

User avatar
Ath
Normal user
Posts: 691
Joined: 10 Jun 2018, 12:06
Location: NL

Re: Pulse Counter P003 enhancement. How to give back?

#89 Post by Ath » 28 Mar 2021, 18:08

Kim_Panda wrote: 28 Mar 2021, 17:18 Do I need to press the "Resolve conversation" button in the PR? What is the purpose for that button?
It does indeed resolve & close the conversation, making it easier to track what's not resolved yet (at least, that's how I use it ;))
The repo isn't setup to block a merge while reviews/conversations are open, but it is good practice to have everything resolved before the merge is applied.
/Ton

Kim_Panda
Normal user
Posts: 40
Joined: 25 Jan 2021, 17:03

Re: Pulse Counter P003 enhancement. How to give back?

#90 Post by Kim_Panda » 02 Apr 2021, 20:16

Ath wrote: 28 Mar 2021, 18:08 It does indeed resolve & close the conversation, making it easier to track what's not resolved yet (at least, that's how I use it ;))
Ok, thanks.

Kim_Panda
Normal user
Posts: 40
Joined: 25 Jan 2021, 17:03

Re: Pulse Counter P003 enhancement. How to give back?

#91 Post by Kim_Panda » 03 Apr 2021, 01:29

TD-er wrote: 28 Mar 2021, 10:11 I added some remarks to your PR.
All requested changes are done. And also command handling is revised. Docu is updated. See PR.
I closed the conversations that I regard as completed.
New Version is pushed to
https://github.com/KimPanda4/ESPEasy/tr ... nhancement
puhhh :roll:

GravityRZ
Normal user
Posts: 194
Joined: 23 Dec 2019, 21:24

Re: Pulse Counter P003 enhancement. How to give back?

#92 Post by GravityRZ » 08 Apr 2021, 20:12

looking forward testing it

Kim_Panda
Normal user
Posts: 40
Joined: 25 Jan 2021, 17:03

Re: Pulse Counter P003 enhancement. How to give back?

#93 Post by Kim_Panda » 11 Apr 2021, 14:40

GravityRZ wrote: 08 Apr 2021, 20:12 looking forward testing it
Today I did a minor enhancement to the documentation.
I completed all my work. I am not familiar with the formal release process in ESPEasy,
but I hope that @TD-er will provide you soon with a build, so you could do your test.

User avatar
Ath
Normal user
Posts: 691
Joined: 10 Jun 2018, 12:06
Location: NL

Re: Pulse Counter P003 enhancement. How to give back?

#94 Post by Ath » 11 Apr 2021, 15:11

Kim_Panda wrote: 11 Apr 2021, 14:40 I completed all my work. I am not familiar with the formal release process in ESPEasy,
but I hope that @TD-er will provide you soon with a build, so you could do your test.
Pre-release builds can be downloaded from the PR page, from the 'build-list' at the end of the page, select one of the Details links, then from that page select the Artifacts (1) link on the right, and download the Binaries.zip file, where you can extract the desired .bin file. (IMHO: Super useful feature of Github!)

@Kim_Panda: You could use the button below that list to merge the latest 'mega' changes into your branch. (Only available for the author of the branch and Project maintainers, not for 'ordinary' users like me :))
/Ton

GravityRZ
Normal user
Posts: 194
Joined: 23 Dec 2019, 21:24

Re: Pulse Counter P003 enhancement. How to give back?

#95 Post by GravityRZ » 11 Apr 2021, 15:59

thanks
testing the 0411 build now.

the 0317 build was already very stable.
when playing with the debounce time set to 100ms it only registered 1 extra pulse in a week
when set to 80ms it did not create any false(+ or - pulses) for a week now

with my setup (watermeter with proximity sensor reacting to metal) there is always a change of extra or missing pulses when the sensor is on the verge of changing.

give me some time to see how stable it is.

Kim_Panda
Normal user
Posts: 40
Joined: 25 Jan 2021, 17:03

Re: Pulse Counter P003 enhancement. How to give back?

#96 Post by Kim_Panda » 11 Apr 2021, 23:06

Ath wrote: 11 Apr 2021, 15:11 Pre-release builds can be downloaded from the PR page, from the 'build-list' at the end of the page, select one of the Details links, then from that page select the Artifacts (1) link on the right, and download the Binaries.zip file, where you can extract the desired .bin file. (IMHO: Super useful feature of Github!)

@Kim_Panda: You could use the button below that list to merge the latest 'mega' changes into your branch. (Only available for the author of the branch and Project maintainers, not for 'ordinary' users like me :))
Thank's Ton. Indeed, a good feature.
I just started the merge with the latest mega changes and it succeeded :) .

Since you know Github quite well, allow me a question: What is the best way to also merge my https://github.com/KimPanda4/ESPEasy/tr ... nhancement (which I branched from mega a while ago) with the latest mega changes (somewhere in the Github web page, with GitHub Desktop or in my VS-Code/PlatformIO and what is the appropriate command(s))?

Kim_Panda
Normal user
Posts: 40
Joined: 25 Jan 2021, 17:03

Re: Pulse Counter P003 enhancement. How to give back?

#97 Post by Kim_Panda » 11 Apr 2021, 23:20

GravityRZ wrote: 11 Apr 2021, 15:59 thanks
testing the 0411 build now.
...
give me some time to see how stable it is.
@GravityRZ: You may want the documentation to interpret the statistical log messages.
I attach here the relevant page as .pdf.

Post Reply

Who is online

Users browsing this forum: No registered users and 31 guests