Local storage in ESP when offline ??

Moderators: grovkillen, Stuntteam, TD-er

Post Reply
Message
Author
vincen
Normal user
Posts: 92
Joined: 26 Jun 2017, 07:15

Local storage in ESP when offline ??

#1 Post by vincen » 14 Aug 2017, 14:06

Hi

I'd like to use one of my ESP module for a weather sensor collection but Wifi to each it'll connect won't be avalaible from time to time. I use MQTT to publish datas of my ESP and send them to my home automation system. Is there a way to get the ESP to store values in local when wifi/mqtt broker is unavaible and push them to broker when it's avalaible again.

Thanks

Vincèn

User avatar
grovkillen
Core team member
Posts: 3621
Joined: 19 Jan 2017, 12:56
Location: Hudiksvall, Sweden
Contact:

Re: Local storage in ESP when offline ??

#2 Post by grovkillen » 14 Aug 2017, 15:01

Not yet official but later you will be able to do this if you have a SD card activated. See this merge: https://github.com/letscontrolit/ESPEasy/pull/429

As I understand it (I have not yet tried it myself) the file will be ever growing until you halt it. A good way that maybe is possible to add in the future that it is only written to during downtime of broker and send in a dump on reconnect? (THIS IS ONLY MY WISHFUL THINKING, not a official statement of the developer).
ESP Easy Flasher [flash tool and wifi setup at flash time]
ESP Easy Webdumper [easy screendumping of your units]
ESP Easy Netscan [find units]
Official shop: https://firstbyte.shop/
Sponsor ESP Easy, we need you :idea: :idea: :idea:

Shardan
Normal user
Posts: 1156
Joined: 03 Sep 2016, 23:27
Location: Bielefeld / Germany

Re: Local storage in ESP when offline ??

#3 Post by Shardan » 14 Aug 2017, 15:41

grovkillen wrote: 14 Aug 2017, 15:01 Not yet official but later you will be able to do this if you have a SD card activated. See this merge: https://github.com/letscontrolit/ESPEasy/pull/429

As I understand it (I have not yet tried it myself) the file will be ever growing until you halt it. A good way that maybe is possible to add in the future that it is only written to during downtime of broker and send in a dump on reconnect? (THIS IS ONLY MY WISHFUL THINKING, not a official statement of the developer).
Another thng that should be implementet together with that would be data reduction in my opinion.
A possibility to set a parameter that says "store only if changed".
The reduction of lag size is dramatically, I tried such a parameter with the logs in my homecontrol - fhem has it built in.
It is possible to reduce data just by setting a long delay, sure.
But a "save on change" does not tie you to a time schedule. If changes are there, they are logged. No log if no changes.
So if a fast change occurs, you get it as detailes as the "Delay" setting allows. If there is no change, there is no data logged.

That would be my whish with that.
Regards
Shardan

vincen
Normal user
Posts: 92
Joined: 26 Jun 2017, 07:15

Re: Local storage in ESP when offline ??

#4 Post by vincen » 14 Aug 2017, 15:42

grovkillen wrote: 14 Aug 2017, 15:01 Not yet official but later you will be able to do this if you have a SD card activated. See this merge: https://github.com/letscontrolit/ESPEasy/pull/429
As I understand it (I have not yet tried it myself) the file will be ever growing until you halt it. A good way that maybe is possible to add in the future that it is only written to during downtime of broker and send in a dump on reconnect? (THIS IS ONLY MY WISHFUL THINKING, not a official statement of the developer).
Thanks a lot for link and yep it's a first step to do what I wanted that is very similar to you ;) I posted my suggestion at github (to use that feature as a temporary spool when broker/nettwork/wifi is unavailable and then push all data to broker when it's back up and delete the spool and so on ;)

vincen
Normal user
Posts: 92
Joined: 26 Jun 2017, 07:15

Re: Local storage in ESP when offline ??

#5 Post by vincen » 14 Aug 2017, 16:19

Shardan wrote: 14 Aug 2017, 15:41Another thng that should be implementet together with that would be data reduction in my opinion.
A possibility to set a parameter that says "store only if changed".
The reduction of lag size is dramatically, I tried such a parameter with the logs in my homecontrol - fhem has it built in.
It is possible to reduce data just by setting a long delay, sure.
But a "save on change" does not tie you to a time schedule. If changes are there, they are logged. No log if no changes.
So if a fast change occurs, you get it as detailes as the "Delay" setting allows. If there is no change, there is no data logged.
Great idea too ;) an option to tell to send to broker only if value(s) have changed since previous transmission ;) You should suggest it on github for developpers no ?

Shardan
Normal user
Posts: 1156
Joined: 03 Sep 2016, 23:27
Location: Bielefeld / Germany

Re: Local storage in ESP when offline ??

#6 Post by Shardan » 14 Aug 2017, 19:50

I'll add it to github then...
Regards
Shardan

vincen
Normal user
Posts: 92
Joined: 26 Jun 2017, 07:15

Re: Local storage in ESP when offline ??

#7 Post by vincen » 15 Aug 2017, 09:38

Shardan wrote: 14 Aug 2017, 19:50 I'll add it to github then...
;) Hope they can make it in firmware soon, would be so cool :P

AndrewJ
Normal user
Posts: 229
Joined: 14 Feb 2017, 12:38

Re: Local storage in ESP when offline ??

#8 Post by AndrewJ » 15 Aug 2017, 10:04

+1 for me.

I've experimented with a rule to achieve the same, it works reasonably well, but it's cumbersome. Would be much better if integrated into the firmware.

vincen
Normal user
Posts: 92
Joined: 26 Jun 2017, 07:15

Re: Local storage in ESP when offline ??

#9 Post by vincen » 20 Aug 2017, 16:41

AndrewJ wrote: 15 Aug 2017, 10:04 +1 for me.
I've experimented with a rule to achieve the same, it works reasonably well, but it's cumbersome. Would be much better if integrated into the firmware.
Do you mind to share your rule(s) ? Thanks ;)

AndrewJ
Normal user
Posts: 229
Joined: 14 Feb 2017, 12:38

Re: Local storage in ESP when offline ??

#10 Post by AndrewJ » 20 Aug 2017, 20:50

Hello, Vincen,
My rule works with a Dummy task, see config below
Dummy_task.PNG
Dummy_task.PNG (24.78 KiB) Viewed 23323 times
The rule is as follows:-

// Filter temps to reduce data load...
On DS18B201#Temperature do
If [DS18B201#Temperature]<[MQTT_filter#T1_fall_trigger]
Event Send_T1
endif

If [DS18B201#Temperature]>[MQTT_filter#T1_rise_trigger]
Event Send_T1
endif
endon

On Send_T1 do
Publish %sysname%/DS18B201/Temperature [DS18B201#Temperature]
TaskValueSet 6,1, [DS18B201#Temperature] // last value sent
TaskValueSet 6,2, [DS18B201#Temperature]+0.2 // new trigger at which to send on rise
TaskValueSet 6,3, [DS18B201#Temperature]-0.2 // new trigger at which to send on fall
endon

It does work reasonably well, but quite a lot of config for just one measurement! (It would be much better if the ESPEasy code could contain an option to send a new value only when a change of more than xxx ).
Hope this is of interest. if you see any ways to improve it, please let me know. :)
Regards,
Andrew

vincen
Normal user
Posts: 92
Joined: 26 Jun 2017, 07:15

Re: Local storage in ESP when offline ??

#11 Post by vincen » 21 Aug 2017, 08:36

AndrewJ wrote: 20 Aug 2017, 20:50My rule works with a Dummy task, see config below
Thanks a lot Andrew ;) Will let you know if I find some ways to enhance it ;)

Drum
Normal user
Posts: 300
Joined: 07 Feb 2016, 11:56

Re: Local storage in ESP when offline ??

#12 Post by Drum » 24 Sep 2017, 15:02

This sounds like it writes to the Memory chip? If this is true, I believe this could cause issues unless implemented very carefully. As I understand it the memory chip is somewhat limited in the number of writes, around 100,000. See viewtopic.php?f=5&t=287

SD or micro SD would be better as they are much easier to replace and is also bigger.

digger
Normal user
Posts: 10
Joined: 21 Aug 2016, 12:11

Re: Local storage in ESP when offline ??

#13 Post by digger » 30 Apr 2018, 13:20

Just wondering if anything progressed with the storage of data on the local device when the WIFI is offline? I have some temperature process' I need to log and we need to maintain a constant data trail, so a WIFI dropout would break it.

I have had a hunt around the forum and saw a mention of native SD Card support for system logs etc in the new mega release, but could really see anything about logging data to a card and later retrieval.

Any pointers appreciated.

Cheers

User avatar
grovkillen
Core team member
Posts: 3621
Joined: 19 Jan 2017, 12:56
Location: Hudiksvall, Sweden
Contact:

Re: Local storage in ESP when offline ??

#14 Post by grovkillen » 30 Apr 2018, 15:42

Yes it's supported. Haven't tried it myself but you should be able to dump it to a CSV file but you need to fetch the file and prices it yourself later when WiFi is back.
ESP Easy Flasher [flash tool and wifi setup at flash time]
ESP Easy Webdumper [easy screendumping of your units]
ESP Easy Netscan [find units]
Official shop: https://firstbyte.shop/
Sponsor ESP Easy, we need you :idea: :idea: :idea:

digger
Normal user
Posts: 10
Joined: 21 Aug 2016, 12:11

Re: Local storage in ESP when offline ??

#15 Post by digger » 01 May 2018, 14:00

Thanks for the reply.

Sorry for Being a NOOB, but where can I find the info to set this up. I checked under devices etc and could not see anything relating to an SD card there (I believe I am running the latest Mega Release).

Once setup, I would probably look at some sort of bulk upload option to thinkspeak to process the missing data.

Cheers

User avatar
grovkillen
Core team member
Posts: 3621
Joined: 19 Jan 2017, 12:56
Location: Hudiksvall, Sweden
Contact:

Re: Local storage in ESP when offline ??

#16 Post by grovkillen » 01 May 2018, 17:51

It's not documented, so for now you're a bit on your own. We are focusing on getting the v2.1 released :?
ESP Easy Flasher [flash tool and wifi setup at flash time]
ESP Easy Webdumper [easy screendumping of your units]
ESP Easy Netscan [find units]
Official shop: https://firstbyte.shop/
Sponsor ESP Easy, we need you :idea: :idea: :idea:

digger
Normal user
Posts: 10
Joined: 21 Aug 2016, 12:11

Re: Local storage in ESP when offline ??

#17 Post by digger » 02 May 2018, 07:07

Thanks for that, I will stop hunting for the documentation. I am sure with the little knowledge I have I can brick something :-P

One quick question though, should the card show up in the device list at this point or is that not the case?

digger
Normal user
Posts: 10
Joined: 21 Aug 2016, 12:11

Re: Local storage in ESP when offline ??

#18 Post by digger » 25 Sep 2018, 08:09

Just thought I would try my luck and see if there had been any update on being able to log data in the absence of a WIFI connection and if there was any helpful information yet on how to set this up as I cannot work out where to even start with this.

Many thanks.

tparvais
Normal user
Posts: 99
Joined: 28 Oct 2015, 23:13

Re: Local storage in ESP when offline ??

#19 Post by tparvais » 25 Jun 2020, 08:04

Any news on this idea ?

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Local storage in ESP when offline ??

#20 Post by TD-er » 25 Jun 2020, 12:00

We have the "Cache controller", to which you can send data as any other controller.

See this project where it is used and described: https://www.letscontrolit.com/forum/vie ... ome#p45191

nibiru
New user
Posts: 6
Joined: 17 Oct 2021, 22:16
Location: Ukraine

Re: Local storage in ESP when offline ??

#21 Post by nibiru » 07 Jan 2022, 21:11

Hello to all! I managed to implement data storage using a dummy "Regulator - Level Control" device.
This device has one parameter stored in non-volatile memory.
https://www.letscontrolit.com/wiki/inde ... velControl
Parameter read and write commands are supported:
write

Code: Select all

config,task,<taskname>,setlevel,<value|calculation> 
and read

Code: Select all

[<taskname>#getLevel]
My thermostat project uses 2 temperature settings - day and night, so i used two dummy "Regulator - Level Control".
Its names - TempDayNV and TempNightNV.
In settings in device must be:
GPIO → Level low: - Left NONE, its true dummy regulator!!
Check Task: - select any task, no matter what.
Check Value: - also any.
Set Level: - this is valye of stored parameter, you can predefine it.

This is how the parameters are read at the start of the board:

Code: Select all

on System#Boot do
TaskValueSet,DefaultTemperature,Day,[TempDayNV#getLevel] 
TaskValueSet,DefaultTemperature,Night,[TempNightNV#getLevel]
TaskValueSet,Variables,ManualTemperature,[TempDayNV#getLevel]
endon
This is how the parameters are written to non-volatile memory:

Code: Select all

on Clock#Time=All,**:** do 
if [DefaultTemperature#Day] <> [TempDayNV#getLevel] 
config,task,TempDayNV,setlevel,[DefaultTemperature#Day]
endif
if [DefaultTemperature#Night] <> [TempNightNV#getLevel] 
config,task,TempNightNV,setlevel,[DefaultTemperature#Night]
endif
endon
So every minute check, if parameter changed, and save if need. Not immediately!

This is how the dashboard.esp controls work:

Code: Select all

<table>
<caption>Default Temperature</caption>
<tr>
<th>Day=[DefaultTemperature#Day]C</th>
<th>Night=[DefaultTemperature#Night]C</th>
</tr>
<tr>
<td><a class='button link' href="dashboard.esp?cmd=event,DayTemperatureUp">+0.1C</a></td>
<td><a class='button link' href="dashboard.esp?cmd=event,NightTemperatureUp">+0.1C</a></td>
</tr>
<tr>
<td><a class='button link' href="dashboard.esp?cmd=event,DayTemperatureDown">-0.1C</a></td>
<td><a class='button link' href="dashboard.esp?cmd=event,NightTemperatureDown">-0.1C</a></td>
</tr>
</table> 
These are the dashboard control handlers:

Code: Select all

on DayTemperatureUp do
TaskValueSet 5,1,[DefaultTemperature#Day]+0.1
endon

on DayTemperatureDown do
TaskValueSet 5,1,[DefaultTemperature#Day]-0.1
endon

on NightTemperatureUp do
TaskValueSet 5,2,[DefaultTemperature#Night]+0.1
endon

on NightTemperatureDown do
TaskValueSet 5,2,[DefaultTemperature#Night]-0.1
endon
TaskValueSet 5,2 and TaskValueSet 5,1 (TaskValueSet,DefaultTemperature,Day, TaskValueSet,DefaultTemperature,Night) - this is simple "Generic - Dummy Device", used for variables.
The above method has been tried and tested and works well.

I also tried to save the configuration directly from the dashboard, but apparently this operation takes a lot of processor time, the processor freezes, and the loading of the dashboard page is interrupted, although the parameters are saved.

User avatar
Ath
Normal user
Posts: 3415
Joined: 10 Jun 2018, 12:06
Location: NL

Re: Local storage in ESP when offline ??

#22 Post by Ath » 07 Jan 2022, 21:22

Please be aware that, after the merge of PR #3891, the changed level won't be saved immediately (for newly added tasks) to avoid flash-wear-out. An extra save command would be needed to actually store the setting. (Or an added checkbox checked)
/Ton (PayPal.me)

Post Reply

Who is online

Users browsing this forum: No registered users and 38 guests