maybe easier to use Dashboard replacement

[ThomasB]

Hmm. The mega_20250224_climate_ESP32_4M316k release has broken all the values in one of my Dummy Device plugins. Reverting back to the Dec 2024 release restores them.

Scratching my head. I will have to look closely at the rules to see if I can find a reason for this. But it might be a new ESPEasy bug.

At least it's not a dashboard problem.

EDIT: It's not a rule problem. So I've reverted easyfetch back to 20250221/efc6 since that release works with the Dec 2024 ESPEasy. If I learn anything useful I will report to the ESPEasy devs.

- Thomas

[TD-er]

Hmm might be related to another reported issue about RTC stored values.
Does it only show 'messed up' after a reboot, or also when updating values?

[chromo23]

Hmm might be related to another reported issue about RTC stored values.

That was the first thought i had....

If I learn anything useful I will report to the ESPEasy devs.

If there is anything to it please open a new thread.

So I've reverted easyfetch back to 20250221/efc6

You shouldn't have to. Easyfetch should handle this issue gracefully and ignore the broken json

[TD-er]

If this build (finally) will succeed, can you test to see if the same RTC issues are still present?
https://github.com/letscontrolit/ESPEas ... 3686812891

It is using a newer SDK (for LittleFS builds)

Only try on accessible units on your desk...

[ThomasB]

I use five Dummy Device plugins, mostly because they allow me to assign useful names to vars. One of the Dummy's (all of its three values) won't accept data that I store. All remain at zero. So probably not related to RTC.

Only try on accessible units on your desk...

The device is mounted in the attic. Not readily accessible.

You shouldn't have to. Easyfetch should handle this issue gracefully and ignore the broken json

Unfortunately it didn't handle it. The GUI was a blank screen. Reverting back ESPEasy AND EasyFetch, as described, was needed.

I haven't had time to further debug this. I am juggling other projects at the moment.

- Thomas

[ThomasB]

I haven't had time to further debug this. I am juggling other projects at the moment.

I found the problem with the latest ESPEasy test build. Reported here:
viewtopic.php?t=10670

- Thomas

[ThomasB]

TD-er fixed the ESPEasy bug. So I was able to continue testing efc_chart3.

Here's what I found:
1. Charting does not update in real-time. Requires page refresh to see new graphed data.
2. Page refreshing often causes the ESPEasy to hang for several minutes. This does not occur if EasyFetch's charting feature is not used.
3. Only one instance of EasyFetch is possible. Additional browsers get an empty page with hamburger and house icons at the bottom.

- Thomas

[TD-er]

Do you also need a Climate ESP32 SPIFFS build for this?

If so, then I can make you a test build including the Math fix and some sysvar parsing tweaks + my work-in-progress for the WiFi rewrite.

[ThomasB]

I thought the build you gave me today was a Climate ESP32 SPIFFS. If not, what did you provide?

- Thomas

[TD-er]

Just made a test build:
ESP_Easy_mega_20250311_climate_ESP32_4M316k_ETH

ESP_Easy_mega_20250311_normal_ESP32_4M316k_LittleFS_ETH

And just to be sure, only use it on nodes that can be re-flashed via USB.
This is based on the WiFi code complete rewrite I'm working on for a while now.

The LittleFS build does have some fixes regarding the 'connected' state of a client accessing the ESP via http etc.

[TD-er]

I thought the build you gave me today was a Climate ESP32 SPIFFS. If not, what did you provide?

- Thomas

It was a SPIFFS Climate build, but based on the current 'mega' branch code + the fix for calculations in rules.

This latest build I just linked to DropBox links is based on highly experimental WiFi code.
A work-in-progress.

[ThomasB]

Oh, it sounds tempting. But the device is mounted in the attic crawl space. Not practical to do a USB recovery if it is bricked. So I have to wait until others have tried it out.

- Thomas

[TD-er]

And you don't have a 2nd ESP32 to test?

[chromo23]

TD-er fixed the ESPEasy bug. So I was able to continue testing efc_chart3.

Here's what I found:
1. Charting does not update in real-time. Requires page refresh to see new graphed data.
2. Page refreshing often causes the ESPEasy to hang for several minutes. This does not occur if EasyFetch's charting feature is not used.
3. Only one instance of EasyFetch is possible. Additional browsers get an empty page with hamburger and house icons at the bottom.

- Thomas

Thats odd as I can not reproduce any of these issues and have right now no idea what would cause this. But let me think about it....
Here is what i see when a chart is enabled and running two instances at the same time (one in chrome one in firefox):

2g.gif (149.28 KiB) Viewed 9909 times

It was only when I opened a third instance on my phone that I had problems with failed requests. But let's not forget that this is a microcontroller, and as such it has limited resources.

[chromo23]

Here's what I found:

I found something too.
The check for broken JSON wasn't that robust so with any error in JSON-requests easyfetch was falling back to "standard" JSON and therefore not getting the chart data.
I should be fixed with 20250311/efc_chart1. This means you have to also update the index.htm
I hope it fixes some of your issues...

[TD-er]

Also there is distinct difference currently between LittleFS and SPIFFS builds as the LittleFS builds (the fairly recent test builds) do have a fix to properly determine whether a client is still connected.
Without this fix, making multiple connections (also frequent connections from the same client) would easily fail.

This will sadly not be fixed for the (ESP32-xx) SPIFFS builds, so we probably need to switch to LittleFS-only builds for ESP32-xx builds and thus the transition should be made as easy as possible.
The TAR backup feature Ton made and the fact the WiFi credentials are also stored outside the file system will make this hopefully a fairly simple task to do.
However it must be documented and thoroughly tested and made very clear why this is needed.

[ThomasB]

And you don't have a 2nd ESP32 to test?

Aliexpress has filled my stock with ESP32 boards. But they don't have any of my sensors on them. Aren't they necessary to make this a useful test?

I should be fixed with 20250311/efc_chart1

I will try it out.

This will sadly not be fixed for the (ESP32-xx) SPIFFS builds, so we probably need to switch to LittleFS-only builds for ESP32-xx builds and thus the transition should be made as easy as possible.

SPIFFS has been good to me. It will be missed. I will postpone the transition to LittleFS for a time when my calendar is empty.

- Thomas

[TD-er]

And you don't have a 2nd ESP32 to test?

Aliexpress has filled my stock with ESP32 boards. But they don't have any of my sensors on them. Aren't they necessary to make this a useful test?

I should be fixed with 20250311/efc_chart1

I will try it out.

This will sadly not be fixed for the (ESP32-xx) SPIFFS builds, so we probably need to switch to LittleFS-only builds for ESP32-xx builds and thus the transition should be made as easy as possible.

SPIFFS has been good to me. It will be missed. I will postpone the transition to LittleFS for a time when my calendar is empty.

- Thomas

The sysinfo plugin is very well capable in delivering fluctuating values without anything connected to it.
For example the internal temperature, free memory, RSSI, TX-power.
Those are the ones I always use with my nodes here on my desk which do not have anything else connected to them.

I for sure understand the hesitation to wait updating to LittleFS...
As a matter of fact, I did postpone this for the last 2.5 - 3 years to not let go of SPIFFS in builds.
But now it has come to a point where specific bugfixes will no longer be backported to a platform build which has SPIFFS code and will not be too large to fit on the 4M flash.

[ThomasB]

I installed EasyFetch 20250311/efc_chart1. I saw the chart window update, with a few seconds delay between each update. So that is a win.

But after a minute or so the ESP became unresponsive. After several more minutes I was able to reach the default ESPEasy GUI. But each time I tried to view EasyFetch it became unresponsive again.

ESPEasy does not reboot, it just stops responding for a while. Browser reports "The connection was reset." When it is unresponsive the device still acknowledges IP pinging.

In case you see anything important, I've attached the json from the device index that I chart. EasyFetch charts the Delta-T and CFM data in it.

stats_json.zip

Device Stats

(1.29 KiB) Downloaded 68 times

I am using ESPEasy SPIFFS build mega_20250310 ESP32_4M316k. Which build release do you use for your EasyFetch development? And is it SPIFFS or LittleFS?

Edit: The most reliable version of EasyFetch for me is the 20250221/efc6 release. So I have reverted back to it.

- Thomas

[chromo23]

I saw the chart window update, with a few seconds delay between each update. So that is a win.

In fact the update interval in easyfetch is 2 seconds. So this delay is normal.

But after a minute or so the ESP became unresponsive. After several more minutes I was able to reach the default ESPEasy GUI. But each time I tried to view EasyFetch it became unresponsive again.

This is really strange.
Easyfetch works the same way, the /devices tab of ESPEasy works. It fetches the JSON of the device and your browser draws some nice tiles and puts the data in it.
If ther is an issue, than the /devices page in ESPEasy should also become unresponsive after a while.
Easyfetch is even more conservative with the update interval (2 sec vs. 1sec)

In case you see anything important, I've attached the json from the device index that I chart. EasyFetch charts the Delta-T and CFM data in it.

Unfortunately this doesn't help me. If easyfetch displays something, than the JSON is fine. The content of the developer console would be more helpful to see the errormessages.

Edit: The most reliable version of EasyFetch for me is the 20250221/efc6 release. So I have reverted back to it.

I think I will stop the update notifications for you, as they can be quite annoying

[ThomasB]

No need to stop the notifications, 20250221/efc6 came out before you added that feature.

If ther is an issue, than the /devices page in ESPEasy should also become unresponsive after a while.

I have not had the issue while in the native ESPEasy pages. Only occurs when I visit EasyFetch page. And turning off the graph did not solve it.

- Thomas

[chromo23]

I have not had the issue while in the native ESPEasy pages. Only occurs when I visit EasyFetch page. And turning off the graph did not solve it.

I have just tried to run easyfetch on several devices with charts enabled for a few minutes. I have not experienced any errors or unresponsiveness so far.

Whenever you find time to show me the contents of your browser's developer console, I'd appreciate it.
Edit: But take your time. From Thursday on I am gone for one month. Maybe we'll accidentally bump into each other

[TD-er]

@Thomas:
When testing a continuous ping to the ESP device, are the ping times constant?
Can you check on the inspect window of your browser on the "network" tab to see how long it typically takes for your JSON loads to complete?

[ThomasB]

When testing a continuous ping to the ESP device, are the ping times constant?

While on ESPEasy's main page, ping times vary from 43mS to 500mS. And I sometimes see a timeout, which is concerning. RSSI is -43dBm, so signal strength is very good.

Can you check on the inspect window of your browser on the "network" tab to see how long it typically takes for your JSON loads to complete?

Ranges from 395 to 962mS. Avg somewhere in the middle. 8.87kb transferred. But this is the older "working" EasyFetch version.

Whenever you find time to show me the contents of your browser's developer console, I'd appreciate it.

Let's wait until you are back from your trip. Have safe travels.

- Thomas

[ThomasB]

@TD-er, I loaded your test build (20250312_normal_ESP32_4M316k_LittleFS_ETH) on a spare ESP32. No problems switching from SPIFFS to LittleFS. Good news: It is running fine.

@chromo23, Your latest version (20250311/efc_chart1) is working well with the ESPEasy test build while charting some System Info values. I can perform page refreshes without causing a page hang too. While watching the charting, I observed a short period (~6 secs) where EasyFetch became unresponsive, but it recovered on its own.

I observed json load times for fetch.html.gz ranging from 263ms to 805mS, Xfr size 13.03kb. The efc_min.js time is 0mS and it is 0 byte length.

Further testing: The target device is not easily accessed (in attic). Plus it needs a different build configuration (climate). So I haven't upgraded it with this new configuration to further test things out. Let's wait to do that after TD-er's latest wifi code receives more thumbs-up reviews from other testers.

TL;DR The latest code revisions are working! Thank you TD-er and chromo23 for helping out.

- Thomas

[TD-er]

For that spare ESP, you could start testing with this GH Actions test build regarding the WiFi rewrite:
https://github.com/letscontrolit/ESPEas ... 3866749190

There are still some issues left for me to fix, but I already have it running on several nodes in my test setup.
But keep in mind it is still best to not run it on remote nodes.

N.B. Ethernet is probably not yet working as it should, or at least not tested at all...

Preferrably test LittleFS builds on ESP32-xx
And you could also try it on some ESP8266 nodes if you still have those.

[ThomasB]

Can you add ESP32 LittleFS Climate to your actions build?

BTW, I don't have the hardware to test Ethernet builds.

- Thomas

[Ath]

Can you add ESP32 LittleFS Climate to your actions build?

I've added PR #5283 to have an ESP32 Climate LittleFS build.

[TD-er]

And one highly experimental WiFi rewrite pull request build is here: https://github.com/letscontrolit/ESPEas ... 3875023945

[ThomasB]

I will start a new topic for discussions about the WiFi improvement code testing.

Edit: viewtopic.php?t=10695

- Thomas

[chromo23]

I found some time to hopefully get rid of most of the bugs and everything should work smoothly.

I have attached the index.htm.gz for uploading to your devices.

Right click on the tiles to make changes or on mobile devices the config mode can be activated by long pressing the side-nav button (three horizontal lines).
Just click around and change the settings. A second click on another tile or outside the tiles closes the context menu and shows the changes, and a third click outside the tiles or on the bottom left X button closes the settings menu without saving.
Note that pressing the save button writes a small file to the device, so don't do it too often

Have fun....

index.htm.gz

(12.38 KiB) Downloaded 59 times

[ThomasB]

Thanks for the new release. It still hangs on my mega_20250310_climate_ESP32_4M316k device. I need to crawl into the attic and remove the device so I can convert to LittleFS using the latest ESPEasy test build. I should be able to do that in a few days.

- Thomas

[ThomasB]

I visited the attic and pulled the ESP32 device. Then it was erased and updated with the latest ESPEasy (mega_20250429_climate_ESP32_4M316k_LittleFS_ETH) via TD-er's webflasher. Lastly I went to your github test repo and installed the latest fetch.html.gz.

But unfortunately the hang problem remains. If I stay on ESPEasy's web pages everything is fine. But if I visit the EasyFetch dashboard the device works for a brief period, then stops communicating. Pinging still functions during the hang and I get a page reset warning in the browser if the page is refreshed. After anywhere from 2-15 minutes, the device reboots, and works again for a short time. Rinse and repeat. This stalling occurs often, which makes operation difficult.

After several hours of poking and prodding, I've made a few observations. Unfortunately I did not have any ah-ha moments that will help with the diagnosis. Here's a summary:

1. I initially reloaded my old EasyFetch configuration (main_efc.json). This did not work at all because all I saw was a mostly blank dashboard page with the home and hamburger icons at the bottom. Deleting the config file restored the default dashboard tiles.

2. If the main_efc.json file is removed, EasyFetch never creates a permanent stall or reboot. I will randomly see it stall for a few seconds, but it always recovers without a reboot. Unfortunately this is not a good solution because I want to customize the layout.

3. I currently have 13 device plugins that report data. During troubleshooting I added a system info device to help debug, for a 14 device total. With this extra plugin, anytime I save the EasyFetch layout, the new plugin is deleted. I can consistently repeat this fault. I tried adding it to the empty #13, as wells as the #16 Task locations; Both experience this deletion issue.

4. As mentioned, I have a lot of plugins on this device. Perhaps this complexity is involved in my sad situation? Below is a screenshot of my plugin configuration.

HVAC Monitor Screenshot

Screenshot.jpg (295.85 KiB) Viewed 80 times

[TD-er]

How long does it take to load the JSON?
What if this takes roughly 1 second and the page refresh of this loading attempt is also 1 sec?

Also did you perform a RF calibration reset as you updated the device from IDF <= 5.3 to IDF 5.4
N.B. not all devices need this, as the differences in RF calibration seem to be related to the corner cases of antenna tuning.

Anyway, you could try to use the inspect window in your browser to see how long the JSON loading takes.

[ThomasB]

Yikes, I see some fetch.html.gz load times that are extremely close to 1000mS, which is the page refresh time. So I think you are on to something. That is to say, maybe the reload/refresh needs to be 1500mS?

BTW, when the hang occurs the inspector's "Transfer" column begins reporting NS_BINDING_ABORTED every second.

I didn't perform EraseSDKwifi because I read the github discussion for the new ESPEasy test release. There was some info about how the wifi calibration data would be erased automatically to ensure connection success. So I didn't bother with the extra step. But since you asked, I performed the reset. It seems better now, but still experiences the hangs.

- Thomas

[TD-er]

Extending the reload time is something that needs to be done in the EasyFetch.
Maybe start some timer after the JSON was received and wait for a second before attempting a new fetch of the JSON.
This way you have a rate limiter regardless of the load times.

[TD-er]

I did remove the auto erase as there appears to be a bug in the ESP-IDF SDK which may cause an ESP to completely hang.
So I removed it, as on those boards that are affected by this (not all seem to hang. oddly enough) you would never get the ESP back online...
Ask me how I know

It didn't seem to be that good of an user experience, so that's why I disabled the automatic erase of RF calibration.