Any way to restore config.dat automatically?

Moderators: grovkillen, Stuntteam, TD-er

Post Reply
Message
Author
dampa
Normal user
Posts: 87
Joined: 19 Jul 2018, 01:48

Any way to restore config.dat automatically?

#1 Post by dampa » 29 Jul 2019, 22:15

I'd like to build an backup/restore for all the settings in ESPEasy. Currently I've created a backup of all the parts (config.dat, security.dat, rules1-4.txt) and restore for everythiong but config.txt. It seems the only way is to open ESPEasy on the device, go to tools, select upload, pick the file and then upload it.

What I've done is write a node-red flow that when you give it the IP address of the ESPEasy device and it dumps everything into
ESPBRR
--uuuuunn-backup (where uuuu is the unit name and nnn is the unit number)
----date@time (date at sigh time)
------config.dat
------rules1.txt
------rules2.txt
------rules3.txt
------rules4.txt
------security.dat

Here is an real life example
ESPBRR
--Node29_backups
----2019-07-28@104509
------config.dat
------rules4.txt
------rules3.txt
------rules2.txt
------rules1.txt
------security.dat
------index.md
--coffee30_backups
----2019-07-28@105547

The restore process works by displaying the contents of the ESPBRR directory:
Screen Shot 2019-07-29 at 4.04.47 PM.png
Screen Shot 2019-07-29 at 4.04.47 PM.png (12.52 KiB) Viewed 9973 times
when you select a device you see it's backups:
Screen Shot 2019-07-29 at 4.08.04 PM.png
Screen Shot 2019-07-29 at 4.08.04 PM.png (20.73 KiB) Viewed 9973 times
and when you pick a date, you can select all the files or any combination:
Screen Shot 2019-07-29 at 4.09.08 PM.png
Screen Shot 2019-07-29 at 4.09.08 PM.png (47.16 KiB) Viewed 9973 times
And it all works except for the config.dat file....

Is there any chance we will ever be able to restore the config.dat other than thru the menu's?

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Any way to restore config.dat automatically?

#2 Post by TD-er » 30 Jul 2019, 12:28

What is the difference between these restore calls?
I can imagine there might be a check in the code not to accept uploads of all filetypes from an external call (security design choice)

Did you try to do a POST request to save the config.dat file?

By the way, this backup/restore looks really nice :)

dampa
Normal user
Posts: 87
Joined: 19 Jul 2018, 01:48

Re: Any way to restore config.dat automatically?

#3 Post by dampa » 30 Jul 2019, 12:56

I've been trying to figure out what a post would look like.

For all the other files I use curl in this form "curl -v -F upload='@path-to-file/rules1.txt' http://192.168.48.29/upload" and it works fine but the config.dat doesn't. Even though it looks like it works, the settings don't change.

Any guidance?

p.s. if I get this working I'll be posting it. I have another Node-Red flow that lists all devices using ESPEasy and my goal would be to add that code so it could automatically backup all your settings for every device. Actually I could get the backup to run now, I just wanted to have the restore work for everything.

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Any way to restore config.dat automatically?

#4 Post by TD-er » 30 Jul 2019, 16:17

The settings will only change when you reboot the node.
Have you tried that, upload the settings file, wait a few seconds and then call for a reboot of the unit?

dampa
Normal user
Posts: 87
Joined: 19 Jul 2018, 01:48

Re: Any way to restore config.dat automatically?

#5 Post by dampa » 31 Jul 2019, 10:29

That doesn't help. Here is how I tested
1) from tools did a backup of the configuration
2) deleted the two controlers I had - rebooted
3) used curl to load the config.dat file - rebooted
4) controlers missing
5) loaded config.dat via tools - controlers showed up with out a reboot

Any other ideas?

User avatar
ThomasB
Normal user
Posts: 1064
Joined: 17 Jun 2018, 20:41
Location: USA

Re: Any way to restore config.dat automatically?

#6 Post by ThomasB » 31 Jul 2019, 22:07

Any other ideas?
Here's my ideas:

Serial Log: Use ESPEz's serial log and set it to debug level. Check the log messages during a config.dat upload to see if anything useful is posted. I also noticed that you are using curl's verbose (-v) flag, so check its messages too.

File Format: The config.dat is a big binary file and the rule.txt files are small ascii files. With that in mind, re-confirm that you can successfully upload both security.dat AND notification.dat binary files. And as a reminder, don't add a Admin Password to the config page (field must be empty) since doing that would block the curl uploads.

File Size: To check for a file size related problem, try uploading (using curl) a ~10KB size binary file, such as jpeg image. Keep in mind that there is limited SPIFFS space. But I expect a test file that is close to 10KB will fit without issue. After you upload it, verify it is valid (view uploaded image using the Tools=>flash filelist page). After the test be sure to delete the file to free up space.

Final thoughts: Maybe curl's config.dat upload is too fast and breaking the transfer. Curl has a --limit-rate parameter that can be used to slow things down.

Keep in mind that these ideas are just guesses. But might be useful as you troubleshoot the problem.

- Thomas

dampa
Normal user
Posts: 87
Joined: 19 Jul 2018, 01:48

Re: Any way to restore config.dat automatically?

#7 Post by dampa » 01 Aug 2019, 01:03

Thomas, thanks for the suggestions! I'll try them over the next few days. I thought I was going to have the whole week free but turns out my daughters family is back tonight from camping and that means I'll be invaded by grandkids tomorrow (grin-smile)

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Any way to restore config.dat automatically?

#8 Post by TD-er » 03 Aug 2019, 01:41

Also consider the amount of free space on the SPIFFS.
The amount reported to be free/available is not the amount of data you can store on SPIFFS.

Also I can imagine the config.dat file cannot be overwritten like that.
ESPeasy internally is using that file in a random access manner to read parts of it very often.
I guess we may need to upload it under another name and then watch for the specific name at boot and then rename it (and maybe throw away the old one or rename that one as last known fall back)

If you cannot find another way of uploading this file, can you make an issue for it on GitHub?

dampa
Normal user
Posts: 87
Joined: 19 Jul 2018, 01:48

Re: Any way to restore config.dat automatically?

#9 Post by dampa » 09 Aug 2019, 13:04

If you cannot find another way of uploading this file, can you make an issue for it on GitHub?
Will do.

Had to put this on a back burner for a bit due to family issues (argue, the grandkids have invaded :lol: ) and an issue with MQTT (another thread).

Post Reply

Who is online

Users browsing this forum: Ahrefs [Bot] and 34 guests