Local storage in ESP when offline ??
Moderators: grovkillen, Stuntteam, TD-er
Local storage in ESP when offline ??
Hi
I'd like to use one of my ESP module for a weather sensor collection but Wifi to each it'll connect won't be avalaible from time to time. I use MQTT to publish datas of my ESP and send them to my home automation system. Is there a way to get the ESP to store values in local when wifi/mqtt broker is unavaible and push them to broker when it's avalaible again.
Thanks
Vincèn
I'd like to use one of my ESP module for a weather sensor collection but Wifi to each it'll connect won't be avalaible from time to time. I use MQTT to publish datas of my ESP and send them to my home automation system. Is there a way to get the ESP to store values in local when wifi/mqtt broker is unavaible and push them to broker when it's avalaible again.
Thanks
Vincèn
- grovkillen
- Core team member
- Posts: 3621
- Joined: 19 Jan 2017, 12:56
- Location: Hudiksvall, Sweden
- Contact:
Re: Local storage in ESP when offline ??
Not yet official but later you will be able to do this if you have a SD card activated. See this merge: https://github.com/letscontrolit/ESPEasy/pull/429
As I understand it (I have not yet tried it myself) the file will be ever growing until you halt it. A good way that maybe is possible to add in the future that it is only written to during downtime of broker and send in a dump on reconnect? (THIS IS ONLY MY WISHFUL THINKING, not a official statement of the developer).
As I understand it (I have not yet tried it myself) the file will be ever growing until you halt it. A good way that maybe is possible to add in the future that it is only written to during downtime of broker and send in a dump on reconnect? (THIS IS ONLY MY WISHFUL THINKING, not a official statement of the developer).
ESP Easy Flasher [flash tool and wifi setup at flash time]
ESP Easy Webdumper [easy screendumping of your units]
ESP Easy Netscan [find units]
Official shop: https://firstbyte.shop/
Sponsor ESP Easy, we need you
ESP Easy Webdumper [easy screendumping of your units]
ESP Easy Netscan [find units]
Official shop: https://firstbyte.shop/
Sponsor ESP Easy, we need you
Re: Local storage in ESP when offline ??
Another thng that should be implementet together with that would be data reduction in my opinion.grovkillen wrote: ↑14 Aug 2017, 15:01 Not yet official but later you will be able to do this if you have a SD card activated. See this merge: https://github.com/letscontrolit/ESPEasy/pull/429
As I understand it (I have not yet tried it myself) the file will be ever growing until you halt it. A good way that maybe is possible to add in the future that it is only written to during downtime of broker and send in a dump on reconnect? (THIS IS ONLY MY WISHFUL THINKING, not a official statement of the developer).
A possibility to set a parameter that says "store only if changed".
The reduction of lag size is dramatically, I tried such a parameter with the logs in my homecontrol - fhem has it built in.
It is possible to reduce data just by setting a long delay, sure.
But a "save on change" does not tie you to a time schedule. If changes are there, they are logged. No log if no changes.
So if a fast change occurs, you get it as detailes as the "Delay" setting allows. If there is no change, there is no data logged.
That would be my whish with that.
Regards
Shardan
Shardan
Re: Local storage in ESP when offline ??
Thanks a lot for link and yep it's a first step to do what I wanted that is very similar to you I posted my suggestion at github (to use that feature as a temporary spool when broker/nettwork/wifi is unavailable and then push all data to broker when it's back up and delete the spool and so ongrovkillen wrote: ↑14 Aug 2017, 15:01 Not yet official but later you will be able to do this if you have a SD card activated. See this merge: https://github.com/letscontrolit/ESPEasy/pull/429
As I understand it (I have not yet tried it myself) the file will be ever growing until you halt it. A good way that maybe is possible to add in the future that it is only written to during downtime of broker and send in a dump on reconnect? (THIS IS ONLY MY WISHFUL THINKING, not a official statement of the developer).
Re: Local storage in ESP when offline ??
Great idea too an option to tell to send to broker only if value(s) have changed since previous transmission You should suggest it on github for developpers no ?Shardan wrote: ↑14 Aug 2017, 15:41Another thng that should be implementet together with that would be data reduction in my opinion.
A possibility to set a parameter that says "store only if changed".
The reduction of lag size is dramatically, I tried such a parameter with the logs in my homecontrol - fhem has it built in.
It is possible to reduce data just by setting a long delay, sure.
But a "save on change" does not tie you to a time schedule. If changes are there, they are logged. No log if no changes.
So if a fast change occurs, you get it as detailes as the "Delay" setting allows. If there is no change, there is no data logged.
Re: Local storage in ESP when offline ??
+1 for me.
I've experimented with a rule to achieve the same, it works reasonably well, but it's cumbersome. Would be much better if integrated into the firmware.
I've experimented with a rule to achieve the same, it works reasonably well, but it's cumbersome. Would be much better if integrated into the firmware.
Re: Local storage in ESP when offline ??
Hello, Vincen,
My rule works with a Dummy task, see config below The rule is as follows:-
// Filter temps to reduce data load...
On DS18B201#Temperature do
If [DS18B201#Temperature]<[MQTT_filter#T1_fall_trigger]
Event Send_T1
endif
If [DS18B201#Temperature]>[MQTT_filter#T1_rise_trigger]
Event Send_T1
endif
endon
On Send_T1 do
Publish %sysname%/DS18B201/Temperature [DS18B201#Temperature]
TaskValueSet 6,1, [DS18B201#Temperature] // last value sent
TaskValueSet 6,2, [DS18B201#Temperature]+0.2 // new trigger at which to send on rise
TaskValueSet 6,3, [DS18B201#Temperature]-0.2 // new trigger at which to send on fall
endon
It does work reasonably well, but quite a lot of config for just one measurement! (It would be much better if the ESPEasy code could contain an option to send a new value only when a change of more than xxx ).
Hope this is of interest. if you see any ways to improve it, please let me know.
Regards,
Andrew
My rule works with a Dummy task, see config below The rule is as follows:-
// Filter temps to reduce data load...
On DS18B201#Temperature do
If [DS18B201#Temperature]<[MQTT_filter#T1_fall_trigger]
Event Send_T1
endif
If [DS18B201#Temperature]>[MQTT_filter#T1_rise_trigger]
Event Send_T1
endif
endon
On Send_T1 do
Publish %sysname%/DS18B201/Temperature [DS18B201#Temperature]
TaskValueSet 6,1, [DS18B201#Temperature] // last value sent
TaskValueSet 6,2, [DS18B201#Temperature]+0.2 // new trigger at which to send on rise
TaskValueSet 6,3, [DS18B201#Temperature]-0.2 // new trigger at which to send on fall
endon
It does work reasonably well, but quite a lot of config for just one measurement! (It would be much better if the ESPEasy code could contain an option to send a new value only when a change of more than xxx ).
Hope this is of interest. if you see any ways to improve it, please let me know.
Regards,
Andrew
Re: Local storage in ESP when offline ??
This sounds like it writes to the Memory chip? If this is true, I believe this could cause issues unless implemented very carefully. As I understand it the memory chip is somewhat limited in the number of writes, around 100,000. See viewtopic.php?f=5&t=287
SD or micro SD would be better as they are much easier to replace and is also bigger.
SD or micro SD would be better as they are much easier to replace and is also bigger.
Re: Local storage in ESP when offline ??
Just wondering if anything progressed with the storage of data on the local device when the WIFI is offline? I have some temperature process' I need to log and we need to maintain a constant data trail, so a WIFI dropout would break it.
I have had a hunt around the forum and saw a mention of native SD Card support for system logs etc in the new mega release, but could really see anything about logging data to a card and later retrieval.
Any pointers appreciated.
Cheers
I have had a hunt around the forum and saw a mention of native SD Card support for system logs etc in the new mega release, but could really see anything about logging data to a card and later retrieval.
Any pointers appreciated.
Cheers
- grovkillen
- Core team member
- Posts: 3621
- Joined: 19 Jan 2017, 12:56
- Location: Hudiksvall, Sweden
- Contact:
Re: Local storage in ESP when offline ??
Yes it's supported. Haven't tried it myself but you should be able to dump it to a CSV file but you need to fetch the file and prices it yourself later when WiFi is back.
ESP Easy Flasher [flash tool and wifi setup at flash time]
ESP Easy Webdumper [easy screendumping of your units]
ESP Easy Netscan [find units]
Official shop: https://firstbyte.shop/
Sponsor ESP Easy, we need you
ESP Easy Webdumper [easy screendumping of your units]
ESP Easy Netscan [find units]
Official shop: https://firstbyte.shop/
Sponsor ESP Easy, we need you
Re: Local storage in ESP when offline ??
Thanks for the reply.
Sorry for Being a NOOB, but where can I find the info to set this up. I checked under devices etc and could not see anything relating to an SD card there (I believe I am running the latest Mega Release).
Once setup, I would probably look at some sort of bulk upload option to thinkspeak to process the missing data.
Cheers
Sorry for Being a NOOB, but where can I find the info to set this up. I checked under devices etc and could not see anything relating to an SD card there (I believe I am running the latest Mega Release).
Once setup, I would probably look at some sort of bulk upload option to thinkspeak to process the missing data.
Cheers
- grovkillen
- Core team member
- Posts: 3621
- Joined: 19 Jan 2017, 12:56
- Location: Hudiksvall, Sweden
- Contact:
Re: Local storage in ESP when offline ??
It's not documented, so for now you're a bit on your own. We are focusing on getting the v2.1 released
ESP Easy Flasher [flash tool and wifi setup at flash time]
ESP Easy Webdumper [easy screendumping of your units]
ESP Easy Netscan [find units]
Official shop: https://firstbyte.shop/
Sponsor ESP Easy, we need you
ESP Easy Webdumper [easy screendumping of your units]
ESP Easy Netscan [find units]
Official shop: https://firstbyte.shop/
Sponsor ESP Easy, we need you
Re: Local storage in ESP when offline ??
Thanks for that, I will stop hunting for the documentation. I am sure with the little knowledge I have I can brick something
One quick question though, should the card show up in the device list at this point or is that not the case?
One quick question though, should the card show up in the device list at this point or is that not the case?
Re: Local storage in ESP when offline ??
Just thought I would try my luck and see if there had been any update on being able to log data in the absence of a WIFI connection and if there was any helpful information yet on how to set this up as I cannot work out where to even start with this.
Many thanks.
Many thanks.
Re: Local storage in ESP when offline ??
Any news on this idea ?
Re: Local storage in ESP when offline ??
We have the "Cache controller", to which you can send data as any other controller.
See this project where it is used and described: https://www.letscontrolit.com/forum/vie ... ome#p45191
See this project where it is used and described: https://www.letscontrolit.com/forum/vie ... ome#p45191
Re: Local storage in ESP when offline ??
Hello to all! I managed to implement data storage using a dummy "Regulator - Level Control" device.
This device has one parameter stored in non-volatile memory.
https://www.letscontrolit.com/wiki/inde ... velControl
Parameter read and write commands are supported:
write
and read
My thermostat project uses 2 temperature settings - day and night, so i used two dummy "Regulator - Level Control".
Its names - TempDayNV and TempNightNV.
In settings in device must be:
GPIO → Level low: - Left NONE, its true dummy regulator!!
Check Task: - select any task, no matter what.
Check Value: - also any.
Set Level: - this is valye of stored parameter, you can predefine it.
This is how the parameters are read at the start of the board:
This is how the parameters are written to non-volatile memory:
So every minute check, if parameter changed, and save if need. Not immediately!
This is how the dashboard.esp controls work:
These are the dashboard control handlers:
TaskValueSet 5,2 and TaskValueSet 5,1 (TaskValueSet,DefaultTemperature,Day, TaskValueSet,DefaultTemperature,Night) - this is simple "Generic - Dummy Device", used for variables.
The above method has been tried and tested and works well.
I also tried to save the configuration directly from the dashboard, but apparently this operation takes a lot of processor time, the processor freezes, and the loading of the dashboard page is interrupted, although the parameters are saved.
This device has one parameter stored in non-volatile memory.
https://www.letscontrolit.com/wiki/inde ... velControl
Parameter read and write commands are supported:
write
Code: Select all
config,task,<taskname>,setlevel,<value|calculation>
Code: Select all
[<taskname>#getLevel]
Its names - TempDayNV and TempNightNV.
In settings in device must be:
GPIO → Level low: - Left NONE, its true dummy regulator!!
Check Task: - select any task, no matter what.
Check Value: - also any.
Set Level: - this is valye of stored parameter, you can predefine it.
This is how the parameters are read at the start of the board:
Code: Select all
on System#Boot do
TaskValueSet,DefaultTemperature,Day,[TempDayNV#getLevel]
TaskValueSet,DefaultTemperature,Night,[TempNightNV#getLevel]
TaskValueSet,Variables,ManualTemperature,[TempDayNV#getLevel]
endon
Code: Select all
on Clock#Time=All,**:** do
if [DefaultTemperature#Day] <> [TempDayNV#getLevel]
config,task,TempDayNV,setlevel,[DefaultTemperature#Day]
endif
if [DefaultTemperature#Night] <> [TempNightNV#getLevel]
config,task,TempNightNV,setlevel,[DefaultTemperature#Night]
endif
endon
This is how the dashboard.esp controls work:
Code: Select all
<table>
<caption>Default Temperature</caption>
<tr>
<th>Day=[DefaultTemperature#Day]C</th>
<th>Night=[DefaultTemperature#Night]C</th>
</tr>
<tr>
<td><a class='button link' href="dashboard.esp?cmd=event,DayTemperatureUp">+0.1C</a></td>
<td><a class='button link' href="dashboard.esp?cmd=event,NightTemperatureUp">+0.1C</a></td>
</tr>
<tr>
<td><a class='button link' href="dashboard.esp?cmd=event,DayTemperatureDown">-0.1C</a></td>
<td><a class='button link' href="dashboard.esp?cmd=event,NightTemperatureDown">-0.1C</a></td>
</tr>
</table>
Code: Select all
on DayTemperatureUp do
TaskValueSet 5,1,[DefaultTemperature#Day]+0.1
endon
on DayTemperatureDown do
TaskValueSet 5,1,[DefaultTemperature#Day]-0.1
endon
on NightTemperatureUp do
TaskValueSet 5,2,[DefaultTemperature#Night]+0.1
endon
on NightTemperatureDown do
TaskValueSet 5,2,[DefaultTemperature#Night]-0.1
endon
The above method has been tried and tested and works well.
I also tried to save the configuration directly from the dashboard, but apparently this operation takes a lot of processor time, the processor freezes, and the loading of the dashboard page is interrupted, although the parameters are saved.
Re: Local storage in ESP when offline ??
Please be aware that, after the merge of PR #3891, the changed level won't be saved immediately (for newly added tasks) to avoid flash-wear-out. An extra save command would be needed to actually store the setting. (Or an added checkbox checked)
/Ton (PayPal.me)
Who is online
Users browsing this forum: Ahrefs [Bot] and 36 guests