maybe easier to use Dashboard replacement

[TD-er]

Also keep in mind, when charting is enabled, the added JSON for the chart data may not be that much. But after some time you may have upto 250 samples per taskvalue you added the checkbox for + 250 timestamps per task you enabled any charting for.

So after a while, this can be as many as 1250 elements in various tasks.

Maybe this chart info should be fetched per task.

Can you open an URL like this, to see how much data it is and how long it takes?
http://192.168.10.229/json?showpluginstats=1

[ThomasB]

After about 10 mins the loop times are much longer. About 8 secs.

See attached zip file for json.

[chromo23]

Is it possible that when the chart data might contain invalid JSON, the EasyFetch is doing strange stuff?

No, invalid JSON would throw an error. (I had this issue with older builds where the JSON + pluginstats would generate an invalid JSON)

I just downloaded Thomas JSON file, which is quite large indeed, and put it on my ESP32 and changed the path of easyfetch to grep that file.
This seems to work fine:

json_thomas.png

I am also doing another test where I have added a lot of tasks with stats enabled to generate as much JSON data as Thomas has, to see if there is a problem with generating the JSON on the ESP...

[chromo23]

Here are my findings:
It definitely adds to the CPU load and loading times for the JSON is going up.
But not as bad as for Thomas device. Must be a mixture of easyfetch and kind + amount of tasks and rules processing i guess.

jsonme.png

stats.png

screenshotme.png

[Ath]

It's been running for a few hours (without EasyFetch charting). Had only one reboot.

Hmm, that's a bit worrisome, I wouldn't expect any reboots, unless maybe you have very complicated rules that overload the unit?
The EraseSDKWifi command has been applied, AFAIR?

[TD-er]

Just did some tweaks to generate JSON.
On a very good WiFi connection, the load times are roughly 100 kByte/sec of JSON on ESP32-S3 (with 8M PSRAM) and slightly less than half that speed on ESP32-classic.

I will make a commit of these changes, so we can see if it makes some difference on Thomas' setup.
However I think this approach of fetching all data is the wrong way of doing this.

For example do we need 250 data points on a small dashboard button?
Shouldn't we just decimate this, or fetch the last N samples?

Also we know what the likely interval is per task, so maybe it is better to just fetch the chart-data per task?
Or just fetch the new chart samples since timestamp X and let the browser append it to some chart data buffer.

All of these options like getting decimated values or N last samples of new samples since timestamp X are things that need to be added to ESPEasy, but I'm open for ideas as I am sure the current approach is just the wrong one.

[chromo23]

On a very good WiFi connection, the load times are roughly 100 kByte/sec of JSON on ESP32-S3 (with 8M PSRAM) and slightly less than half that speed on ESP32-classic.

But than it isn't probably much faster than it already is on a ESP32 classic. In my screenshot you can see around 70k/s

or example do we need 250 data points on a small dashboard button?

A small dashboard button only calls /json
Only when a chart is enabled in easyfetch it starts to call /json?showpluginstats=1.
Splitting up the chart data from the regular JSON seem to be a good approach.
e.g. calling /json?chart=<tasknr>
With this it would at least reduce the number of samples in JSON to the ones that are really needed

[chromo23]

On a very good WiFi connection, the load times are roughly 100 kByte/sec of JSON on ESP32-S3 (with 8M PSRAM) and slightly less than half that speed on ESP32-classic.

But than it isn't probably much faster than it already is on a ESP32 classic. In my screenshot you can see around 70k/s

or example do we need 250 data points on a small dashboard button?

A small dashboard button only calls /json
Only when a chart is enabled in easyfetch it starts to call /json?showpluginstats=1.
Splitting up the chart data from the regular JSON seem to be a good approach.
e.g. calling /json?chart=<tasknr>
With this it would at least reduce the number of samples in JSON to the ones that are really needed

[TD-er]

On a very good WiFi connection, the load times are roughly 100 kByte/sec of JSON on ESP32-S3 (with 8M PSRAM) and slightly less than half that speed on ESP32-classic.

But than it isn't probably much faster than it already is on a ESP32 classic. In my screenshot you can see around 70k/s
[...]

Was this just serving a JSON file or was it generated?
I extrapolated it from a 20 - 40k generated JSON.
So the 100-ish msec WiFi beacon interval may play a large role in the error of my estimates.

Also not all ESP32-boards are equal. The flash on some is faster than on others, even though all should work at the same clock frequency.

[chromo23]

Was this just serving a JSON file or was it generated?

See the screenshots of my last posts (#353 and #354)....
The 77k JSON from Thomas was on the FS of my ESP and took around 400-500ms (json1.json)
The timing of the generated JSON you can seen in the screenshot of the following post

[ThomasB]

Keep in mind that there are several sensors, including two one-wire and three I2C devices. Plus a NeoPixel. Those will probably be impractical to simulate in your test setup.

I'm having problems running with charting enabled. It tends to stall/hang a lot, especially if I interact with the dashboard. And then I have to wait several minutes for a reboot. Sometimes I give up waiting and I go to the garage and toggle the breaker.

I was hoping to find some significant clue, a proverbial smoking gun. But I don't see a pattern and it all looks random to me. But I can say that the latest EasyFetch works reasonable well for me if I don't use charting.

There were a lot of previous request for me to provide some specific info. Not sure what you need now.

or example do we need 250 data points on a small dashboard button? Shouldn't we just decimate this, or fetch the last N samples?

In my application a lot of chart history is helpful. I currently see about 50 minutes of charted data. Any less would be undesirable and more would be ideal.

- Thomas

[ThomasB]

FYI, the problem seems to be spreading to the forum. For the last two days I've been getting a lot of stalling and hangs while using the forum.

Now putting on a face mask to reduce further spreading of a possible virus.

- Thomas

[TD-er]

[...]

or example do we need 250 data points on a small dashboard button? Shouldn't we just decimate this, or fetch the last N samples?

In my application a lot of chart history is helpful. I currently see about 50 minutes of charted data. Any less would be undesirable and more would be ideal.

- Thomas

That's why I suggested to decimate the historic data.
Like averaging 4 or 5 samples into 1 to show.

I don't see how it can be useful to have more samples than the width of the chart.

Another option could be that EasyFetch is giving the last received sample timestamp in the request, so I only send what's new since.
Then the viewer could combine this info in the browser window and we reduce the amount of data to render into JSON significantly.

Can you try this test build: https://github.com/letscontrolit/ESPEas ... 4887247773
Hopefully it does speed up rendering JSON a bit.
Though I'm not sure this is the right solution, to just speed-up generating way too much data.

[TD-er]

FYI, the problem seems to be spreading to the forum. For the last two days I've been getting a lot of stalling and hangs while using the forum.

Now putting on a face mask to reduce further spreading of a possible virus.

- Thomas

Do you think you can prevent AI from spreading by wearing a face mask?
If so, then I will put on like 10 masks at least. (the forum is being hit by 1000's of AI-searchbots continously for days.)