Change Total value of a pulse counter with command

[bulthaup]

Hi,
Sometimes I need to do maintenance in my electrical/network setup and I have to reboot the ESPEasy device during the day. I'm running pulse counters and after a reboot the total of the day so far is lost. Luckily I can gather the collected amount of pulses from my mqtt broker logs.

I'm looking for a way to push a Total value to the pulse counter. I tried it with taskvalueset but it doesn't work?

http://192.168.1.173/control?cmd=taskvalueset,1,2,163 (where 163 is the amount of Liters water we used so far during the day).

[TD-er]

Taskvalueset is only supposed to work on Dummy variables.

You could use rules to add 2 values and output those to your controller of choice.
Then you can set the offset again in a dummy variable and just let the rules output the sum of both.

[bulthaup]

Then you can set the offset again in a dummy variable and just let the rules output the sum of both.

Would you mind explaining this further in an example? Does the value of such dummy variable survive a reboot? Because I'm issuing a reboot every night at 0:00 with a rule to start collecting the Total count of my watermeter again for a new day.

I understand your approach but it's kinda hard for me to put it in writing (I'm not an experienced coder.. ) But I think what you mean that on a normal day, just the Total value + 0 is saved in another variable and that variable is published to a controller? So If I have to reboot my device I can push a value to a dummy to that variable with the control cmd and espeasy will continue to publish the sum of both the new and old count until a reboot?

Alvast bedankt! Thank in advance!

[ThomasB]

TD-er is probably in deep sleep at this hour. So I'll try to lend a hand while he counts sheep.

Does the value of such dummy variable survive a reboot?

On recent Mega releases the Dummy vars are retained in a warm boot. So no worries if you save the current count to a dummy var then use a rule to reboot the device. Keep in mind that dummy vars will be initialized to zero on a power reset (cold boot).

So If I have to reboot my device I can push a value to a dummy to that variable with the control cmd and espeasy will continue to publish the sum of both the new and old count until a reboot?

Yes, before the rule does the reboot just save the latest calculated Total count to a dummy var. During use, publish the calculated Total value (saved dummy value + new measured total) to your controller.

Edit: He's awake, so pardon my intrusion.

- Thomas

[TD-er]

Yes, the Dummy plugin should get the last values restored after a reboot.
There is however a very big BUT here.
These values can only be kept if the ESP is not loosing its power.
If you remove power, the RTC memory is lost in which these values are being stored.

If you need that, then you have to wait until we have added support for I2C EEPROM/FRAM for this purpose.

[bulthaup]

Yes, the Dummy plugin should get the last values restored after a reboot.
There is however a very big BUT here.
These values can only be kept if the ESP is not loosing its power.
If you remove power, the RTC memory is lost in which these values are being stored.

If you need that, then you have to wait until we have added support for I2C EEPROM/FRAM for this purpose.

Hi TD-er,

What is the best way to approach this? So far I have come up with a small script that copies existing value of the pulsemeter to a dummy variable.

on watermeter#Total do
let,1,[watermeter#Total]
TaskValueSet 2,1,[VAR#1]
endon


It takes some time before the dummy variable is updated though. And after a reboot, the value of the dummy plugin is gone. I think I'm overwriting it again with 0 on init.

Can someone help me out please?

[TD-er]

In the last source (not sure if it was already included in a build) the values from the pulse counter are restored from RTC.
There was a logic error in that plugin which prevented the restore of those values for that plugin.

[bulthaup]

Hi TD-er,

I'll keep an eye on the commits in the next couple of weeks then! thanks!

[TD-er]

Hi TD-er,

I'll keep an eye on the commits in the next couple of weeks then! thanks!

Well you can try last night's build.
I had to fix a recent change in PlatformIO to make the build work, but eventually I was able to get it running and I noticed there was a zip file attached.
See: https://github.com/letscontrolit/ESPEasy/releases

So please let me know if it has been fixed now.

[bulthaup]

Hi TD-er,

I'll keep an eye on the commits in the next couple of weeks then! thanks!

Well you can try last night's build.
I had to fix a recent change in PlatformIO to make the build work, but eventually I was able to get it running and I noticed there was a zip file attached.
See: https://github.com/letscontrolit/ESPEasy/releases

So please let me know if it has been fixed now.

Hi TD-er,

How long does it take for the values to be written to the RTC memory? My tests show that when I up the total pulses and reboot immediately afterwards, sometimes an older value is restored again. When I do a cold boot from no power, rebooting after a few pulses always resets to 0 again.

[TD-er]

It stores the values in the PLUGIN_READ call.
So if that's never performed then it will never store them in the RTC.
PLUGIN_READ is called every <interval> seconds or when you run taskRun from the rules.

[GravityRZ]

i think i am using the functionality you mean.

i do not care if the pulsecounter looses its values after a reboot because i copy all the values i need to a dummy device which survives a reboot
i also attach my script
i am using mqtt to send out stuff to domoticz but also added the http variant commented out with //
please note i hasve set my pulsecounter interval to 1 second.(i would love to have functionality which only triggers when sensing a pulse instead of polling every second )

this way i get instant values for both pulses and flow calculation

question: do the values of your pulsecounters update realtime on screen.
in my setup they do not, only the copied values from the dummy device do update realtime
eg, i do not see Count, Total or Time change( i need to press F5 which refreshes my screen)

devices.JPG (72.17 KiB) Viewed 25517 times

Code: Select all

On System#Boot do     						// When the ESP boots, do
	TaskValueSet 3,1,0					// TaskValueSet TASKnr,VARnr,Value, Reset the Flow counter to 0
	TaskValueSet 3,2,0					// TaskValueSet TASKnr,VARnr,Value, Reset the PreviousFlow counter to 0
	TimerSet,1,30      					// Set Timer 1 for the next event in 30 seconds
EndOn

On Clock#Time=All,0:00 do					// Reset Pulsecounter and LitersToday
	if [Water#Flow] = 0
		resetpulsecounter,1
		TaskValueSet 3,4,0
	endif
EndOn
 
On Watermeter#Count do						// When Pulse is detected
	if [Watermeter#Count] = 1 and [Watermeter#Time] > 2000	// send pulse only when Time is not to low to avoid jitter
		Publish domoticz/in,'{"idx":337,"nvalue":0,"svalue":"1"}'
		//SendToHTTP 192.168.1.50,8084,/json.htm?type=command&param=udevice&idx=337&nvalue=0&svalue=1
		TaskValueSet 3,1,60000/[Watermeter#Time]	// set Water#Flow
		TaskValueSet 3,4,[Watermeter#Total]		// copy Watermeter#Total to Water#LitersToday
		Publish domoticz/in,'{"idx":338,"nvalue":0,"svalue":"[Water#Flow]"}'
		//SendToHTTP 192.168.1.50,8084,/json.htm?type=command&param=udevice&idx=338&nvalue=0&svalue=[Water#Flow]
	endif
	if [Water#Flow] > [Water#MaxFlow]
		TaskValueSet 3,3,[Water#Flow]			// copy Water#Flow to Water#MaxFlow
	endif
EndOn

On Rules#Timer=1 do  						// When Timer 1 expires, do
	if [Water#Flow] > 0 or [Water#PreviousFlow] > 0	// Only send value if flow or previousflow > 0
		Publish domoticz/in,'{"idx":338,"nvalue":0,"svalue":"[Water#Flow]"}'
		//SendToHTTP 192.168.1.50,8084,/json.htm?type=command&param=udevice&idx=338&nvalue=0&svalue=[Water#Flow]
		TaskValueSet 3,2,[Water#Flow]			// set Flow to PreviousFlow
		TaskValueSet 3,1,0
	endif
	TimerSet,1,30      					// Set Timer 1 for the next event in 30 seconds
Endon

[TD-er]

The update is not realtime for several reasons:
- The JavaScript code performing the update does only check for updates every N seconds, with N being the minimum of all enabled tasks
- Values in the UserVar array are only updated in the PLUGIN_READ call

This last one could be changed into re-scheduling the PLUGIN_READ when a value is updated (e.g. when interval is set to 0)
Problem may be that it will also trigger a send to controller action and with multiple pulses per second that would quickly lead to reboots or WiFI disconnects due to running out of resources.

The UserVar array is the one containing the task variables and that one is also stored in the RTC memory to survive a reboot.

[GravityRZ]

i understand

for multiple pulses per second it is not a good feature but for measuring the watermeter (max 1 pulse every 2 seconds).
maybe making it an option and when that option is enabled force the minimum debounce time to 1-2 seconds to avoid running out of resources

[TD-er]

I was thinking about the same idea, to make it an option, with a "minimum time between events"
So if there is an update, it will schedule a read at last time + minimum interval and set an internal bit to make sure no new read is scheduled until that bit is cleared again (e.g. when a read is performed)

Can you make an issue for that on GitHub, or else I will forget it for sure.

[GravityRZ]

ok i will mention it on github as a wanted feature.
no hurry

[bulthaup]

Nice feature, thanks!

[bulthaup]

I was thinking about the same idea, to make it an option, with a "minimum time between events"

Hi TD-er,

I have a nodemcu v3 running mega-20200328 and it collected pulses from my NPN sensor and reed sensor the whole day for water and gas. I have been using the resetpulsecounter command succesfully every day at 00u00 to reset the counters.
The only thing I didn't try is to reset the device during the day, hoping that it would have saved my already collected pulses.

I just tried it and only task 1 seemed to be stored in memory. Task 2 data was gone after the reboot while it should have already collected 41 pulses from the reed sensor.

Any idea what is going on?

11.PNG (15.36 KiB) Viewed 25198 times

[TD-er]

The pulse counter does not save the data on every count, so if the task has not been "run" (see taskrun command) and/or the interval is set to 0 or a very long interval, then you will miss all pulses since the last time the task was "run".

[bulthaup]

The pulse counter does not save the data on every count, so if the task has not been "run" (see taskrun command) and/or the interval is set to 0 or a very long interval, then you will miss all pulses since the last time the task was "run".

Hi TD-er,

The interval is set to 10 seconds for both pulse counter so in the 19hrs of uptime after resetting the counter last midnight, I would assume it should have run at least a couple of times already no?

water.PNG (19.45 KiB) Viewed 25186 times

gas.PNG (19.33 KiB) Viewed 25186 times

[TD-er]

Hmm that sounds like an undocumented feature.
Since documenting things is not my favorite, I guess I will try to change the code

[bulthaup]

Well, I did some tests on another nodemcu before upgrading to the release where you added the pulsereset command. My tests were fine. I could add 2 pulse counters and both tasks survived a reboot, but I only counted a few pulses on each task and didn't let the nodemcu run for a few hours. Nevertheless, still doesn't explain why there is nothing in the RTC memory for 1 specific task.

For some reason one of my tasks never wrote their value to the RTC then?

btw: could this issue be related?

https://github.com/letscontrolit/ESPEasy/issues/2883

[TD-er]

It could be related.
I think not updating on every pulse, or at least frequently is cause for confusion.
So I have to change this to make sure it is saved a lot more often, or at least more predictable.

I think (and will be proven wrong in no-time I guess) it is less of an issue if a few pulses are missing on relative high frequency pulse counts.
After all, a reboot itself is cause for missing pulses as a reboot takes at least 500 msec + some additional time if the reboot is caused by the watchdog timer.

So I guess it should update on every pulse, but no faster than every 2 seconds or so.
And right now I do write the entire UserVar to RTC, but maybe it should only be written the changed values. (have to look into checksum calculation then)

[bulthaup]

It could be related.

Hi TD-er,

Did another test today. I had collected 180 pulses for water and 43 pulses for gas today after 15 hours runtime since 00u00. (resetpulse command). I deliberately executed the "taskrun, 1" and "taskrun, 2" command and after a reboot, all pulses were not restored from RTC.

So I don't know what's going on for me but sometimes it works, sometimes it doesn't. Yesterday only task 1 was restored after reboot, today none of the 2 tasks with a pulse counter.


Can someone confirm this issue?

[GravityRZ]

i tested this but i can not replicate it.

i do not have rules enabled

i tried Taskrun,1 Taskrun,2 reboot
all the counters still there

tried it multiple times, also when generating new pulses, doing a Taskrun,1 Taskrun,2 and immediate reboot

[bulthaup]

On my system, the only rule I have active and running is this one:

On Clock#Time=All,23:59 do
timerSet 1,55
Endon

on Rules#Timer=1 do
resetpulsecounter,1
resetpulsecounter,2
EndOn


It basically resets all pulse counters 5 seconds before midnight.

[GravityRZ]

why not do.

Code: Select all

On Clock#Time=All,23:59 do
resetpulsecounter,1
resetpulsecounter,2
Endon

the esp is not the fastest device so the more clockcycles you can save might be helpfull

this is the way i do it.and yes it is not on the second

[bulthaup]

I think not updating on every pulse, or at least frequently is cause for confusion.

Just did some tests with your change in the rule but after a reset, only the Total counts of task 1 is restored. Task 2 stays 0 again.

Is there a way of debugging this?

[GravityRZ]

@bulthaup.

are you interested in testing a version which can initialize the pulsecounter.

tested it and it seems to work

initpulsecounter,2,55555 will initialize counter of pulsecounter 2 (pulsecounter 1 also works)

i just copied some code from the resetpulsecounter and modified it.
please note i am not a programmer

if this works then people with real programming skills need to look at it.

[GravityRZ]

I think not updating on every pulse, or at least frequently is cause for confusion.

Just did some tests with your change in the rule but after a reset, only the Total counts of task 1 is restored. Task 2 stays 0 again.

Is there a way of debugging this?

hmm, i just tested this but after a reboot both counters are restored.

are we right that we are talking about 3 different things

1 resetpulsecounter (resets the pulsecounter to 000, time excluded)
2 pulsecounter values are restored after a reboot(warm)
3 initialize the pulsecounter on a certain value (which survives a warm reboot) and the name of this thread

1 seems to work
2 also seems to work for multiple counters but i must say i also noticed strange things sometines which i could not replicate
3 modied the code and added an initpulsecounter command which works for multiple counters, also modifed the reset so time is also set to 0 on a reset

[TD-er]

I looked at the code and one thing jumps to mind.
The values are restored after the callback functions are set, so maybe the unit receives a pulse almost immediately and that may overwrite the values in the UserVar array.

I will swap it so you can test it.

Edit: See https://github.com/letscontrolit/ESPEasy/pull/2996

[bulthaup]

I looked at the code and one thing jumps to mind.
The values are restored after the callback functions are set, so maybe the unit receives a pulse almost immediately and that may overwrite the values in the UserVar array.

I will swap it so you can test it.

Edit: See https://github.com/letscontrolit/ESPEasy/pull/2996

It might be possible. Task 1 is a NPN sensor which uses 5V from power source of the nodemcu. Task 2 is a simple reed switch.

Is it possible to provide me a bin file like this one? normal_ESP8266_4M1M.bin

I don't have the tools and knowledge (yet..) to make my own builds.


@GravityRZ, yes 1) is correct and working for me also! 2) Well, that's the issue at the moment 3) That was at first the goal to tackle the problem of a reboot during the day. I wanted to have a way to pick up where I left off before the device reboots.

[GravityRZ]

i made the changes TD-er did to avoid wiping out the values on a reset.
i also added the initpulsecounter command.


maybe not smart to do several changes but i tried it myself and it seems to work.
in case you get strange results i would ask TD-er to make a build for you since i make mine with Arduino software

the format to use initpulsecounter is
initpulsecounter,TASK, VALUE

eg: initpulsecounter,1,12345 will set the total counter of pulsecounter 1 to 12345

i did an OTA update and that works.
sometimes when it does not work you need to flash the binary with espflaher

ESP_Easy_mega-20200328_normal_ESP8266_4M1M-modifiedP003.rar

(523.29 KiB) Downloaded 168 times

[TD-er]

I made a test build of that pull request I linked: https://www.dropbox.com/s/r4bnqwgp6ctav ... 6.zip?dl=0

[bulthaup]

Hi, thanks for the build. It still doesn't work for me

As you can see, the "Time" value for the gasmeter (task 2) got restored somehow??


I'm going to make a new system with a D1 mini to rule out that the nodemcu is bugging me. I'll provide some feedback asap

Before:

before.jpg (92.22 KiB) Viewed 24837 times

After reboot:

after.jpg (92.29 KiB) Viewed 24837 times

[TD-er]

What mode do you use for these pulse count tasks?

[bulthaup]

Well, for both counters I use mode FALLING. Debounce time of 500ms

[bulthaup]

I can reproduce the issue with a D1 Mini.

A simple reed sensor (Aleph PS-3150 Proximity Sensor) is connected to GPIO-14 (D5) and the other leg to GND.

Adding it to task1 or task2 doesn't make any difference. Executing the "taskrun, 1" command also does not trigger the values to be stored to survive a warm reboot. I also didn't execute the command and just waited 5 minutes.

I tried mega-20200328 and your PR build TD-er.

Capture.PNG (24.74 KiB) Viewed 24815 times

[GravityRZ]

could gpio-2 be a problem.

[bulthaup]

I don't think so. I switched the reed switch to D7 (GPIO-13) and the issue remains. The sensor is adding pulses so I don't know why they are not kept in RTC memory
I hooked up a NPN sensor as well on task2. It's same scenario as my "production" system. In this test setup on a D1 mini, I'm not using any rules btw.


You can check the video I made here:

https://www.dropbox.com/s/t3gz0gn4gd8am ... 9.mp4?dl=0

[GravityRZ]

ok, clear.
will do some testing to se if i can somehow trigger it.
if we know what triggers it then TD-er probably knows where to look

will flash the original build to test

[GravityRZ]

whatever i do i can not replicate it.

what i however found was that when i was playing with adding/deleting devices i noticed sometimes that when deleting a device and adding it again i got back some of the old data for count, total and time.

maybe you can try to wipe the esp completely by flashing a 4MB blank image first and then later flash the latest mega build.
also to avoid getting the problem i would advice to use the dummy device for real data.

so copy stuff from pulsecounter to dummy device and use those.

i only use the pulsecounter as a trigger.
i do the counter total and increment of that counter with rules.

sofar 6 days stable, not a single missing of extra pulse.

[TD-er]

Hmm that's an interesting point you just made.
If we delete the task, the task value data is not removed from the RTC stored data.
So it may cause other issues (maybe) if you change tasks.

I will add a clear of the data stored in the UserVar if a task is removed.

I doubt it will be a real issue with this problem, but it is good to have predictable behavior

[bulthaup]

whatever i do i can not replicate it.

What pulse meter do you use? Do you have a simple reed switch to test also?

[bulthaup]

Hmm that's an interesting point you just made.
If we delete the task, the task value data is not removed from the RTC stored data.
So it may cause other issues (maybe) if you change tasks.

I will add a clear of the data stored in the UserVar if a task is removed.

I doubt it will be a real issue with this problem, but it is good to have predictable behavior

I can confirm this behavior. Deleting a task and adding it under same number and name, restores old data from RTC to it. Funny thing is, that when you perform the resetpulsecounter command, the value goes to 0. If you then delete the task and re-add it, the total count is again restored from RTC.

[GravityRZ]

i use a manual pulse switch and sometimes just 2 wires connecting together

i have D5 connected with a 10k pullup resistor and D6 without a pullup resistor

i would advice always to use a pullup resistor

what powersupply are you using?

whatever i do i can not replicate it.

What pulse meter do you use? Do you have a simple reed switch to test also?

[TD-er]

I just added a pull request to clear the stored task values when adding a new task.
Just to make sure no strange values may remain when adding a new task.
See: https://github.com/letscontrolit/ESPEasy/pull/3009

[GravityRZ]

thanks.

any chance if the initpulsecounter command also will be implemented.
might come in handy

i tested it and it seems to work like expected but i am not a coder and do not know how to do this stuff in github

here is the code

Code: Select all

     case PLUGIN_WRITE:
      {
        String command = parseString(string, 1);
        String param1 = parseString(string, 3);
        if (command == F("resetpulsecounter"))
        {
          // Allow for an optional taskIndex parameter. When not given it will take the first task with this plugin.
          const taskIndex_t taskIndex = parseCommandArgumentTaskIndex(string, 1);
          if (validTaskIndex(taskIndex)) {
            if (event->TaskIndex != taskIndex) {
              break;
            }
          }
          Plugin_003_pulseCounter[event->TaskIndex] = 0;
          Plugin_003_pulseTotalCounter[event->TaskIndex] = 0;
          Plugin_003_pulseTime[event->TaskIndex] = 0;
          
          success = true; // Command is handled.
        }
        if (command == F("initpulsecounter"))
        {
          int par1;
          if (validIntFromString(param1, par1))
          {
             const taskIndex_t taskIndex = parseCommandArgumentTaskIndex(string, 1);
          if (validTaskIndex(taskIndex)) {
            if (event->TaskIndex != taskIndex) {
              break;
            }
          }
        Plugin_003_pulseTotalCounter[event->TaskIndex] = par1;
        success = true; // Command is handled.
          }
        }
        break;
      }
  }
  return success;
}

[TD-er]

thanks.

any chance if the initpulsecounter command also will be implemented.
might come in handy

i tested it and it seems to work like expected but i am not a coder and do not know how to do this stuff in github
[...]

I would suggest using "SetPulseCounterTotal", as init suggests more than just setting a value.

I added it to the pull request and also made the taskIndex parameter the last one, as it is an optional parameter (and thus must be the last one)

[GravityRZ]

thanks