Page 1 of 1

Local storage in ESP when offline ??

Posted: 14 Aug 2017, 14:06
by vincen
Hi

I'd like to use one of my ESP module for a weather sensor collection but Wifi to each it'll connect won't be avalaible from time to time. I use MQTT to publish datas of my ESP and send them to my home automation system. Is there a way to get the ESP to store values in local when wifi/mqtt broker is unavaible and push them to broker when it's avalaible again.

Thanks

Vincèn

Re: Local storage in ESP when offline ??

Posted: 14 Aug 2017, 15:01
by grovkillen
Not yet official but later you will be able to do this if you have a SD card activated. See this merge: https://github.com/letscontrolit/ESPEasy/pull/429

As I understand it (I have not yet tried it myself) the file will be ever growing until you halt it. A good way that maybe is possible to add in the future that it is only written to during downtime of broker and send in a dump on reconnect? (THIS IS ONLY MY WISHFUL THINKING, not a official statement of the developer).

Re: Local storage in ESP when offline ??

Posted: 14 Aug 2017, 15:41
by Shardan
grovkillen wrote: 14 Aug 2017, 15:01 Not yet official but later you will be able to do this if you have a SD card activated. See this merge: https://github.com/letscontrolit/ESPEasy/pull/429

As I understand it (I have not yet tried it myself) the file will be ever growing until you halt it. A good way that maybe is possible to add in the future that it is only written to during downtime of broker and send in a dump on reconnect? (THIS IS ONLY MY WISHFUL THINKING, not a official statement of the developer).
Another thng that should be implementet together with that would be data reduction in my opinion.
A possibility to set a parameter that says "store only if changed".
The reduction of lag size is dramatically, I tried such a parameter with the logs in my homecontrol - fhem has it built in.
It is possible to reduce data just by setting a long delay, sure.
But a "save on change" does not tie you to a time schedule. If changes are there, they are logged. No log if no changes.
So if a fast change occurs, you get it as detailes as the "Delay" setting allows. If there is no change, there is no data logged.

That would be my whish with that.

Re: Local storage in ESP when offline ??

Posted: 14 Aug 2017, 15:42
by vincen
grovkillen wrote: 14 Aug 2017, 15:01 Not yet official but later you will be able to do this if you have a SD card activated. See this merge: https://github.com/letscontrolit/ESPEasy/pull/429
As I understand it (I have not yet tried it myself) the file will be ever growing until you halt it. A good way that maybe is possible to add in the future that it is only written to during downtime of broker and send in a dump on reconnect? (THIS IS ONLY MY WISHFUL THINKING, not a official statement of the developer).
Thanks a lot for link and yep it's a first step to do what I wanted that is very similar to you ;) I posted my suggestion at github (to use that feature as a temporary spool when broker/nettwork/wifi is unavailable and then push all data to broker when it's back up and delete the spool and so on ;)

Re: Local storage in ESP when offline ??

Posted: 14 Aug 2017, 16:19
by vincen
Shardan wrote: 14 Aug 2017, 15:41Another thng that should be implementet together with that would be data reduction in my opinion.
A possibility to set a parameter that says "store only if changed".
The reduction of lag size is dramatically, I tried such a parameter with the logs in my homecontrol - fhem has it built in.
It is possible to reduce data just by setting a long delay, sure.
But a "save on change" does not tie you to a time schedule. If changes are there, they are logged. No log if no changes.
So if a fast change occurs, you get it as detailes as the "Delay" setting allows. If there is no change, there is no data logged.
Great idea too ;) an option to tell to send to broker only if value(s) have changed since previous transmission ;) You should suggest it on github for developpers no ?

Re: Local storage in ESP when offline ??

Posted: 14 Aug 2017, 19:50
by Shardan
I'll add it to github then...

Re: Local storage in ESP when offline ??

Posted: 15 Aug 2017, 09:38
by vincen
Shardan wrote: 14 Aug 2017, 19:50 I'll add it to github then...
;) Hope they can make it in firmware soon, would be so cool :P

Re: Local storage in ESP when offline ??

Posted: 15 Aug 2017, 10:04
by AndrewJ
+1 for me.

I've experimented with a rule to achieve the same, it works reasonably well, but it's cumbersome. Would be much better if integrated into the firmware.

Re: Local storage in ESP when offline ??

Posted: 20 Aug 2017, 16:41
by vincen
AndrewJ wrote: 15 Aug 2017, 10:04 +1 for me.
I've experimented with a rule to achieve the same, it works reasonably well, but it's cumbersome. Would be much better if integrated into the firmware.
Do you mind to share your rule(s) ? Thanks ;)

Re: Local storage in ESP when offline ??

Posted: 20 Aug 2017, 20:50
by AndrewJ
Hello, Vincen,
My rule works with a Dummy task, see config below
Dummy_task.PNG
Dummy_task.PNG (24.78 KiB) Viewed 23478 times
The rule is as follows:-

// Filter temps to reduce data load...
On DS18B201#Temperature do
If [DS18B201#Temperature]<[MQTT_filter#T1_fall_trigger]
Event Send_T1
endif

If [DS18B201#Temperature]>[MQTT_filter#T1_rise_trigger]
Event Send_T1
endif
endon

On Send_T1 do
Publish %sysname%/DS18B201/Temperature [DS18B201#Temperature]
TaskValueSet 6,1, [DS18B201#Temperature] // last value sent
TaskValueSet 6,2, [DS18B201#Temperature]+0.2 // new trigger at which to send on rise
TaskValueSet 6,3, [DS18B201#Temperature]-0.2 // new trigger at which to send on fall
endon

It does work reasonably well, but quite a lot of config for just one measurement! (It would be much better if the ESPEasy code could contain an option to send a new value only when a change of more than xxx ).
Hope this is of interest. if you see any ways to improve it, please let me know. :)
Regards,
Andrew

Re: Local storage in ESP when offline ??

Posted: 21 Aug 2017, 08:36
by vincen
AndrewJ wrote: 20 Aug 2017, 20:50My rule works with a Dummy task, see config below
Thanks a lot Andrew ;) Will let you know if I find some ways to enhance it ;)

Re: Local storage in ESP when offline ??

Posted: 24 Sep 2017, 15:02
by Drum
This sounds like it writes to the Memory chip? If this is true, I believe this could cause issues unless implemented very carefully. As I understand it the memory chip is somewhat limited in the number of writes, around 100,000. See viewtopic.php?f=5&t=287

SD or micro SD would be better as they are much easier to replace and is also bigger.

Re: Local storage in ESP when offline ??

Posted: 30 Apr 2018, 13:20
by digger
Just wondering if anything progressed with the storage of data on the local device when the WIFI is offline? I have some temperature process' I need to log and we need to maintain a constant data trail, so a WIFI dropout would break it.

I have had a hunt around the forum and saw a mention of native SD Card support for system logs etc in the new mega release, but could really see anything about logging data to a card and later retrieval.

Any pointers appreciated.

Cheers

Re: Local storage in ESP when offline ??

Posted: 30 Apr 2018, 15:42
by grovkillen
Yes it's supported. Haven't tried it myself but you should be able to dump it to a CSV file but you need to fetch the file and prices it yourself later when WiFi is back.

Re: Local storage in ESP when offline ??

Posted: 01 May 2018, 14:00
by digger
Thanks for the reply.

Sorry for Being a NOOB, but where can I find the info to set this up. I checked under devices etc and could not see anything relating to an SD card there (I believe I am running the latest Mega Release).

Once setup, I would probably look at some sort of bulk upload option to thinkspeak to process the missing data.

Cheers

Re: Local storage in ESP when offline ??

Posted: 01 May 2018, 17:51
by grovkillen
It's not documented, so for now you're a bit on your own. We are focusing on getting the v2.1 released :?

Re: Local storage in ESP when offline ??

Posted: 02 May 2018, 07:07
by digger
Thanks for that, I will stop hunting for the documentation. I am sure with the little knowledge I have I can brick something :-P

One quick question though, should the card show up in the device list at this point or is that not the case?

Re: Local storage in ESP when offline ??

Posted: 25 Sep 2018, 08:09
by digger
Just thought I would try my luck and see if there had been any update on being able to log data in the absence of a WIFI connection and if there was any helpful information yet on how to set this up as I cannot work out where to even start with this.

Many thanks.

Re: Local storage in ESP when offline ??

Posted: 25 Jun 2020, 08:04
by tparvais
Any news on this idea ?

Re: Local storage in ESP when offline ??

Posted: 25 Jun 2020, 12:00
by TD-er
We have the "Cache controller", to which you can send data as any other controller.

See this project where it is used and described: https://www.letscontrolit.com/forum/vie ... ome#p45191

Re: Local storage in ESP when offline ??

Posted: 07 Jan 2022, 21:11
by nibiru
Hello to all! I managed to implement data storage using a dummy "Regulator - Level Control" device.
This device has one parameter stored in non-volatile memory.
https://www.letscontrolit.com/wiki/inde ... velControl
Parameter read and write commands are supported:
write

Code: Select all

config,task,<taskname>,setlevel,<value|calculation> 
and read

Code: Select all

[<taskname>#getLevel]
My thermostat project uses 2 temperature settings - day and night, so i used two dummy "Regulator - Level Control".
Its names - TempDayNV and TempNightNV.
In settings in device must be:
GPIO → Level low: - Left NONE, its true dummy regulator!!
Check Task: - select any task, no matter what.
Check Value: - also any.
Set Level: - this is valye of stored parameter, you can predefine it.

This is how the parameters are read at the start of the board:

Code: Select all

on System#Boot do
TaskValueSet,DefaultTemperature,Day,[TempDayNV#getLevel] 
TaskValueSet,DefaultTemperature,Night,[TempNightNV#getLevel]
TaskValueSet,Variables,ManualTemperature,[TempDayNV#getLevel]
endon
This is how the parameters are written to non-volatile memory:

Code: Select all

on Clock#Time=All,**:** do 
if [DefaultTemperature#Day] <> [TempDayNV#getLevel] 
config,task,TempDayNV,setlevel,[DefaultTemperature#Day]
endif
if [DefaultTemperature#Night] <> [TempNightNV#getLevel] 
config,task,TempNightNV,setlevel,[DefaultTemperature#Night]
endif
endon
So every minute check, if parameter changed, and save if need. Not immediately!

This is how the dashboard.esp controls work:

Code: Select all

<table>
<caption>Default Temperature</caption>
<tr>
<th>Day=[DefaultTemperature#Day]C</th>
<th>Night=[DefaultTemperature#Night]C</th>
</tr>
<tr>
<td><a class='button link' href="dashboard.esp?cmd=event,DayTemperatureUp">+0.1C</a></td>
<td><a class='button link' href="dashboard.esp?cmd=event,NightTemperatureUp">+0.1C</a></td>
</tr>
<tr>
<td><a class='button link' href="dashboard.esp?cmd=event,DayTemperatureDown">-0.1C</a></td>
<td><a class='button link' href="dashboard.esp?cmd=event,NightTemperatureDown">-0.1C</a></td>
</tr>
</table> 
These are the dashboard control handlers:

Code: Select all

on DayTemperatureUp do
TaskValueSet 5,1,[DefaultTemperature#Day]+0.1
endon

on DayTemperatureDown do
TaskValueSet 5,1,[DefaultTemperature#Day]-0.1
endon

on NightTemperatureUp do
TaskValueSet 5,2,[DefaultTemperature#Night]+0.1
endon

on NightTemperatureDown do
TaskValueSet 5,2,[DefaultTemperature#Night]-0.1
endon
TaskValueSet 5,2 and TaskValueSet 5,1 (TaskValueSet,DefaultTemperature,Day, TaskValueSet,DefaultTemperature,Night) - this is simple "Generic - Dummy Device", used for variables.
The above method has been tried and tested and works well.

I also tried to save the configuration directly from the dashboard, but apparently this operation takes a lot of processor time, the processor freezes, and the loading of the dashboard page is interrupted, although the parameters are saved.

Re: Local storage in ESP when offline ??

Posted: 07 Jan 2022, 21:22
by Ath
Please be aware that, after the merge of PR #3891, the changed level won't be saved immediately (for newly added tasks) to avoid flash-wear-out. An extra save command would be needed to actually store the setting. (Or an added checkbox checked)