Automatic backup of settings and rules?

[Andrew Mamohin]

I have several ESP32 devices with EspEasy firmware. Sometimes I change the settings and rules. And I forget it. I would like to make an automatic copy of settings and rules from all such devices on my server. Maybe someone did this? Is it possible to take settings and rules from EspEasy with a command from the server?

[Ath]

There are several scripts floating around for doing that, f.e.:
This GH Issue thread suggests a script for updating, but also backing up, a series of ESPEasy units. I'm sure you can use that as an inspiration/starting point.

And next to that I've created PR #4777 to backup all configuration, or all files, of an ESPEasy unit in a .tar file.

[Andrew Mamohin]

The script does not backup settings and rules. I'm not interested in updating the firmware.
Has PR #4777 not yet been implemented?
Currently there is no URL to download settings and rules?
I would make a script with searching through all addresses myself.

[Ath]

The script does not backup settings and rules. I'm not interested in updating the firmware.

A later version of the script, in one of the comments, does include that backup feature AFAICS, you can extend that to include rules1.txt .. rules4.txt.

[TD-er]

You can just right-click on any file on the file browser page and use that URL via some tools like wget.

[Ath]

Has PR #4777 not yet been implemented?

The latest available build with that feature is available from this GH Actions run, so you could help with testing that feature

[Andrew Mamohin]

You can just right-click

Wow! I didn't know about this possibility. But the config that I saved in this way is different from the config that I uploaded through the button in the web interface.

Code: Select all

09.10.2023  11:10           131 275 config.dat
09.10.2023  11:11           131 072 config_ZAL_U45_Build20230508_20231009111120.dat

[Andrew Mamohin]

Sorry! This is strange behavior of wget for windows. The correct file is downloaded on the debian server.
Thank you, now I will make an automatic backup, just like I wanted!

[TD-er]

If a file gets saved by ESPEasy, it will update the settings if the settings file was from a previous version.

So this will actually alter the file contents.

I have no idea why the file should be larger than a 128k, which is 131072 bytes.

[Andrew Mamohin]

I made a script to backup my modules (it turns out there are already 12 of them). Now backups are created every night and stored for a month. Thanks for the advice!

[TD-er]

Maybe you can also consider to have the flow of data in the opposite direction.
There is already experimental support in ESPEasy for provisioning.
This means you can let ESPEasy fetch files from a HTTP server.

Just to give you some ideas, this is how I do it for some nodes.
Each node does have some specific settings like calibration data on ADC and the unit nr.
So Rules Set 1 is generic for all nodes and rules set 2 contains node specific info.
Rules set 3 contains specific info on how data can be provisioned.

At system boot I call this event: Event,LoadCalibration

Code: Select all

On LoadCalibration Do
  // Test units @Gijs
  if %cpu_id% = 0x1280E0
    config,task,bat,SetCalib,TwoPoint,392,13.9846,571,19.9924
    config,task,batBackup,SetCalib,TwoPoint,1785,10.000
    unit,2
  // Test units @Gijs
  elseif %cpu_id% = 0x0C17E4
    config,task,bat,SetCalib,TwoPoint,370.8615,13,520.857,18
    config,task,batBackup,SetCalib,TwoPoint,1785,10.000
    unit,3
    
  endif
  Save
EndOn

Rules3.txt:

Code: Select all

// Last edit: 2023-04-07 14:05

On updateSettings Do
  provisionprovision
  provisionconfig
Endon

On updateCredentials Do
  provisionsecurity  
Endon

On updateRules Do
  provisionrules,1
  provisionrules,2
  provisionrules,3
Endon

On updateRulesSettings Do
  AsyncEvent,updateSettings
  AsyncEvent,updateRules
  Reboot
Endon

// e.g.
// event,PerformFirmwareUpdate=firmware_max_ESP32_16M8M_LittleFS.bin
On PerformFirmwareUpdate=* Do
  pwm,2,100,0,8
  provisionfirmware,%eventvalue1%
Endon

On provisionfirmware#success=* Do
  gpio,2,0
  Reboot
Endon

On provisionfirmware#failure Do
  gpio,2,0
  Reboot  
Endon

You need to have provisioning support enabled in your build (it might be already present in the max builds, not 100% sure)
On the Settings Archive page you then get some extra checkboxes where you configure what is allowed to be fetched and what not and where it can be fetched from.

As you can see, you can even trigger commands via for example MQTT to update the firmware of a node.

[Andrew Mamohin]

As you can see, you can even trigger commands via for example MQTT to update the firmware of a node.

This is all very interesting, thank you!
But I will never update the firmware automatically.

[TD-er]

Not automatically, but via a command.
So it isn't a periodical pull, but more like a command triggered pull.
For ESP32 (and variants) there are quite a few safety checks built in as I have had some issues while remote updating nodes and those finally rebooted back into the old firmware. (no guarantees)
Those were some nerve wrecking minutes waiting till they came back online.

And it is also quite practical for initial deployment.
Someone I know is using an initial image of the entire flash (with some basic settings) to flash newly manufactured units and these then pull the full configuration and latest build from a local webserver.
This way he has pre-programmed all ESPs with a single image and then only needs to power them up and they do automatically fetch their own config and he also has some extra rules included to perform a HTTP call with the MAC address to "register" the device and let his deployment script generate a new rules file with some new parameters set based on that MAC address.