maybe easier to use Dashboard replacement

Message

ThomasB · #301 Post by **ThomasB** » 02 May 2025, 18:14

Would you mind sending me the config.dat of the attic device.

Before I post it, I just want to confirm that sensitive info isn't in the config.dat file.

- Thomas

ThomasB · #302 Post by **ThomasB** » 02 May 2025, 18:33

@chromo23, Good news! The task deletion issue isn't your problem.
@TD-er, Bad news. Seems ESPEasy has a task deletion bug that shows up when a device with a lot of tasks is rebooted.

- Thomas

chromo23 · #303 Post by **chromo23** » 02 May 2025, 21:11

ThomasB wrote: ↑02 May 2025, 18:14 Before I post it, I just want to confirm that sensitive info isn't in the config.dat file.

All the sensitive data should be in the security.dat as far as I know.

ThomasB wrote: ↑02 May 2025, 18:33 @chromo23, Good news! The task deletion issue isn't your problem.

Good news indeed, so the config.dat is not necessary anymore.

ThomasB wrote: ↑02 May 2025, 18:33 @TD-er, Bad news. Seems ESPEasy has a task deletion bug that shows up when a device with a lot of tasks is rebooted.

Are the 36h already over?

I had a similar issue two day ago where my rules disappeared after saving. Not sure if this is related....

Ath · #304 Post by **Ath** » 02 May 2025, 22:07

I've been using the latest (mega) code a lot lately, while working on a new plugin, using an ESP32-C6 that has all Devices/Tasks slots occupied, but haven't seen any loss of tasks after a reboot (not adding/changing tasks a lot, but rebooting a lot after updating the firmware...).

The issue mentioned doesn't make sense to me anyway, why would a recently added task being removed during/after a reboot

(and not any other task?), feels more like unstable flash storage or power supply to me

But I'm quite sure that's properly checked/validated.

ThomasB · #305 Post by **ThomasB** » 02 May 2025, 23:09

I'm using the external antenna version of this ESP32 device:
https://www.aliexpress.us/item/3256806150187598.html

I did more debugging and things are much weirder than expected. I can modify some entries such as task names, value names, and value decimals, as well as the rules. Those survive a reboot. But adding new tasks or changing most task/hardware/config entries are lost after a reboot. That is to say, they happily exist until Tools->Reboot.

If the consensus is that the ESP32 has failed then I will do a device swap. But I'll wait for more comments before climbing the ladder.

By chance is there a config.dat viewer? Would be interesting to see what is being saved.

- Thomas

TD-er · #306 Post by **TD-er** » 02 May 2025, 23:35

How are these added?
Can you try to call the "save" command before reboot?
Also if the "save" returns almost immediately, it didn't save anything as it was deemed to be the same file content as when loaded.

And to be sure... You're not trying to edit stuff after restoring a tar upload and before a reboot?

TD-er · #307 Post by **TD-er** » 02 May 2025, 23:37

Saving rules does use the upload call and I guess EasyFetch also tries to save something via the same upload path.
Maybe there is something seriously wrong with the upload and then flushing to the file system?

ThomasB · #308 Post by **ThomasB** » 03 May 2025, 00:01

I am adding them from the ESPEasy setup pages.

I tried using the Tools-Command->save this morning. No joy. Tried again just now, command executes immediately (no delay).

I'm not trying to edit stuff after restoring a tar upload and before a reboot.

Before we get too deep into this, I am going to configure another ESP32 with my setup. To prove the target device is not failing.

- Thomas

TD-er · #309 Post by **TD-er** » 03 May 2025, 00:30

Is it possible the ack of a save/submit request may get lost in transit due to poor wifi and then the browser (or impatient user

) may be retrying the save/submit again?
My theory is that this may run into a race condition regarding save/load actions if this can happen.

ThomasB · #310 Post by **ThomasB** » 03 May 2025, 01:01

I configured another ESP32. It is a version with built-in antenna (attic unit has external antenna). EraseSDKwifi was performed, then cold boot.

Initial tests were without EasyFetch. I did not experience the lost settings problem.

But after I added EasyFetch the problem returned. So this is the smoking gun we needed.

Here's a screenshot of the file system, with EasyFetch loaded. Fetch.html.gz is an earlier version. Fetch1.html.gz is the latest version. Is memory usage OK, or is it too full with the three extra files?

screenshot1.jpg

- Thomas

TD-er · #311 Post by **TD-er** » 03 May 2025, 01:06

You can see on the sysinfo page how 'full' the file system is.
and if you like to see how the settings files are being used in charts, you can give the command 'meminfodetail' and after that every time you open the sysinfo page you will see extra in-depth info until you reboot.

ThomasB · #312 Post by **ThomasB** » 03 May 2025, 01:47

Here is a screenshot. There's a lot of info to digest. Rather than trust my amateur analysis, I will need to lean on you Devs. Is there enough free ram and flash to create a new config.dat? Because this sure seems like modified configurations are not being saved after I added those three EasyFetch files.

TD-er · #313 Post by **TD-er** » 03 May 2025, 01:51

that was more about the partitioning to validate the flash size and file system size.
The free space is still text-only.

ThomasB · #314 Post by **ThomasB** » 03 May 2025, 01:59

If you need any other information from me please let me know.

- Thomas

TD-er · #315 Post by **TD-er** » 03 May 2025, 02:11

Just the section regarding ESP Board and Storage of the sysinfo page.

This shows how fast the flash is running, what flash is found, and some info about how the flash might be connected to the ESP (in the ESP Board sections)

ThomasB · #316 Post by **ThomasB** » 03 May 2025, 02:26

Screenshot of ESP Board and Storage. I think I see the problem, only 112K free. I have a feeling it needs to be >131K to create the new dat file. ?

TD-er · #317 Post by **TD-er** » 03 May 2025, 02:30

Hmm 112kB free might be a problem here as the settings file is 128k and LittleFS needs to rewrite the entire file when making changes.

If this really is the issue, then we really must speed up splitting the settings file into separate smaller ones.

Can you try to reproduce save issues when working with at least 128k + 16k free space?

ThomasB · #318 Post by **ThomasB** » 03 May 2025, 02:35

I deleted the older unneeded EasyFetch file. Now 128K free. And the problem is gone.

- Thomas

TD-er · #319 Post by **TD-er** » 03 May 2025, 02:44

OK, so probably it is caused by "disk full".

So while you're going to buy an ESP with 8 or 16M flash, Ton and I have to think about how to fix this by splitting the settings file.
I think a quick fix can be to split it into 2 chunks.
For this I refer to the first chart of "Layout Settings File".
As you can see, there is a large area upto the 32k mark where you don't see any color.
From 32k to 96k there are 2 chunks of 1k per task.
The chart doesn't show what is stored from 96k to 128k, so if that's still empty, we should get rid of it.
Then the first part upto 32k does have at least 16k of unused space.

So I think the easiest fix for now would be to split it into 2 files and thus save about 48k in used space and also make saving quite a lot faster.

ThomasB · #320 Post by **ThomasB** » 03 May 2025, 06:05

Thanks for assisting with this unusual settings problem. Besides your proposed solution, perhaps a "disk full" warning message can be displayed when there's not enough room.

Now to return to the original issue. That is, the EasyFetch random stalls/hangs.

- Thomas

chromo23 · #321 Post by **chromo23** » 05 May 2025, 12:34

ThomasB wrote: ↑03 May 2025, 06:05 Now to return to the original issue. That is, the EasyFetch random stalls/hangs.

I just uploaded a newer version of easyfetch (20250505/1) addressing this issue.
You should get an update notification. If you press OK it downloads a new fetch.html.gz file and reloads the page.

ThomasB · #322 Post by **ThomasB** » 05 May 2025, 14:57

Thank you for the new release. It's installed and running.

On the previous version I typically saw a reboot every 15-20 minutes. So let's see how the latest release does. Fingers are crossed that it is a winner.

- Thomas

TD-er · #323 Post by **TD-er** » 05 May 2025, 15:33

We are also looking into other issues which might be ESPEasy related.
So if this isn't the winner... yet, it might be when we finally found out what bothers Safari loading pages.

ThomasB · #324 Post by **ThomasB** » 05 May 2025, 15:53

It's been running for an hour. No reboots. No stalls. I'm optimistic that we might have a winner.

I looked at the patch. The reset interval is now 5.5 Sec. So it seems TD-er's hunch was on target.

- Thomas

chromo23 · #325 Post by **chromo23** » 05 May 2025, 17:01

ThomasB wrote: ↑05 May 2025, 15:53 I looked at the patch. The reset interval is now 5.5 Sec. So it seems TD-er's hunch was on target.

Yes it was. I also added a check that requests do not start to overlap and keep piling up...
If nothing else is coming up I can make this "new" easyfetch the default version and start to write a small documentation (

my favorite part

)

TD-er · #326 Post by **TD-er** » 05 May 2025, 17:15

Great!

Still I will continue making some work-around to other causes for waits/stalls with other browsers like Chrome/Firefox.

ThomasB · #327 Post by **ThomasB** » 05 May 2025, 21:30

It's been about four hours and it has rebooted twice. That's not bad.

The reboots started after I enabled EasyFetch charting. I will try to confirm it is the cause, but this will take a couple days.

- Thomas

chromo23 · #328 Post by **chromo23** » 05 May 2025, 21:54

ThomasB wrote: ↑05 May 2025, 21:30 The reboots started after I enabled EasyFetch charting.

Hmmmm.. that's interesting since the charts have nothing to do with easyfetch accessing ESPEasy.
Another mystery that needs to be solved

I will wait here patiently for your findings....

Edit: In the info section of the /tools tab you can find the reboot reason. Would be nice to know and maybe helps to find out what's happening...

TD-er · #329 Post by **TD-er** » 05 May 2025, 22:20

How much space is left on the file system?

ThomasB · #330 Post by **ThomasB** » 05 May 2025, 23:43

In the info section of the /tools tab you can find the reboot reason. Would be nice to know and maybe helps to find out what's happening...

Reset Reason: CPU0: Software reset CPU CPU1: Software reset CPU
Last Action before Reboot: PLUGIN_READ: Task FURNACE

How much space is left on the file system?

Without EasyFetch Charting:
Free RAM: 166080 [byte] (147008 - )
Heap Max Free Block: 110592 [byte]
Free Stack: 6644 [byte] (2340 - )
Little FS Size: 316 [kB] (128 kB free)

With Charting:
Free RAM: 173720 [byte] (147008 - )
Heap Max Free Block: 110592 [byte]
Free Stack: 6644 [byte] (2340 - )
Little FS Size: 316 [kB] (128 kB free)

- Thomas

TD-er · #331 Post by **TD-er** » 06 May 2025, 00:53

What plugin is used for the task "FURNACE" ?

Also I think 128k is not enough. Especially on a fragmented file system.

I will check to see what I can do with at least the last 32k.
Maybe I can truncate it.

And if it is possible, you can also do this yourself using some tool like dd.
Just download the config.dat file, cut of the last 32k and upload the file again to ESPEasy's file system
Make sure to not change anything before rebooting after upload of the file.

Please first download a backup as a tar file.

ThomasB · #332 Post by **ThomasB** » 06 May 2025, 04:00

The tasks are defined here:
download/file.php?id=9375

EDIT:
I truncated and reloaded the config.dat file. I removed the last 32kB bytes. So config.dat is now 98304 bytes and device has 160kB free.
Little FS Size: 316 [kB] (160 kB free)

Now running without any EasyFetch charting. Will test this tomorrow with charting enabled and see if the reboots return.

BTW, when EasyFetch charting is enabled and dashboard is visible, ESPEasy's various pages are slow to load. Normally about 1-2 secs, the load times become about 2-5 secs, and system load increases to 100%. Maybe this will be a helpful bit of info.

- Thomas

chromo23 · #333 Post by **chromo23** » 06 May 2025, 09:12

ThomasB wrote: ↑06 May 2025, 04:00 BTW, when EasyFetch charting is enabled and dashboard is visible, ESPEasy's various pages are slow to load. Normally about 1-2 secs, the load times become about 2-5 secs, and system load increases to 100%. Maybe this will be a helpful bit of info.

I try to test now under similar conditions, got my old ESP32 out of the drawer and installed:
ESP_Easy_mega_20250506_climate_ESP32_4M316k_LittleFS_ETH May 6 2025

First of all it is slow from the beginning. +5 seconds when saving and sometimes also slow when changing tabs +2sec (all without easyfetch installed)
I now added some tasks, enabled some charts and let it run for a while to see if i can reproduce any of your issues...

For the short time its running i couldn't neither see an increase of system load nor a slower load time of the ESPEasy tabs/pages
But it is of course impossible for me to simulate your sophisticated setup

ef2t1.jpg

Ath · #334 Post by **Ath** » 06 May 2025, 09:36

chromo23 wrote: ↑06 May 2025, 09:12 First of all it is slow from the beginning. +5 seconds when saving and sometimes also slow when changing tabs +2sec (all without easyfetch installed)
I now added some tasks, enabled some charts and let it run for a while to see if i can reproduce any of your issues...

Have you executed the EraseSDKWiFi command on the unit? As most likely it had older calibration data stored, that might not work nice with the latest (LittleFS) builds

TD-er · #335 Post by **TD-er** » 06 May 2025, 09:38

Thomas, maybe you can show a screenshot of the timingstats page after running this for a while so we can see where most of the time is spent on the ESP.

chromo23 · #336 Post by **chromo23** » 06 May 2025, 09:46

Ath wrote: ↑06 May 2025, 09:36 Have you executed the EraseSDKWiFi command on the unit? As most likely it had older calibration data stored, that might not work nice with the latest (LittleFS) builds

Good point.. it got it bit faster but writing to the config.dat still takes ages...

e1.png

TD-er · #337 Post by **TD-er** » 06 May 2025, 09:55

Yep, that's being worked on as it is also rather space-inefficient as Thomas has experienced.

We're thinking of splitting the settings file into smaller chunks (probably 8k each as that's the flash block size)

chromo23 · #338 Post by **chromo23** » 06 May 2025, 12:49

ThomasB wrote: ↑05 May 2025, 21:30 It's been about four hours and it has rebooted twice. That's not bad.

The reboots started after I enabled EasyFetch charting. I will try to confirm it is the cause, but this will take a couple days.

I was running easyfetch now for 4 hours with the ESPEasy UI open and the side-nav (see picture above).
I did not experience any slowdowns, reboots or network errors. All was running smoothly and as expected.

So either it is not related to easyfetch or there is a very rare condition which i didn't see yet...

BTW: Easyfetch only runs in an active browser window.
When changing to another tab or the browser window is closed (obviously) easyfetch pauses and is not doing any requests to the ESPEasy device.

TD-er · #339 Post by **TD-er** » 06 May 2025, 14:13

Hmm that may also be some kind of clue...
What if you switch back to have the browser active again.
Could this lead to a 'surge' of requests, or at least 2 nearly simultaneous requests?

I know the ESPEasy log viewer may sometimes show a burst of activity when you switch back to the tab with the log viewer.

Also I wonder if we might also see something else in Thomas' timingstats like some plugin which may take a considerable amount of time.
For example the MQTT controller is known to be reported as last activity right before a crash.
Sending to a controller is done from PLUGIN_READ, so it may also be the lead cause of crashes here on Thomas' setup.

Who is online