Fileupload prevents saving of rules

[ralftopas]

Dear all,

i have the following problem:
I use an ESP-Wroom-32 (38 Pinout)
FW: ESP_Easy_mega_20230623_display_ESP32_4M316k Jun 23 2023
Every time i upload an BMP image via filebrowser I am unable to store any changes in one of the four rules.
After deleting the image all four rules are "cleared".
This does not happen in case of a different filetype upload.

Anyone with the same issue?

BAckground: i have some waterleven sensors and wanted to draw the filling level over an image on my attached ILI9341 display...

Kind regards
Ralf!

[Ath]

Hi Ralf,

What size is the BMP, or more interestingly, what space is still available after you have uploaded the BMP?
The ESP32 4M configurations only have 316 kB (gross, real is ca. 300 kB) storage available, and ~140 kB is already in use for the settings, so the remaining space isn't that much. And also, some free blocks are required, to be able to save to a file, so at least a couple of kB has to stay free for that.
So you should better either use very small BMPs, use an ESP32 with an SD-card reader (I think that's available in the ESP32 Display build) and store the BMPs there, or use an ESP32 with 16 MB Flash (or ESP32-S3 with 8MB Flash) and a MAX build that allows 1MB or 8MB (units with 16 MB Flash only) of available file-storage on the built-in flash.

[ralftopas]

Hi,
BMP file size is 177kB
That should be the reason.
I'll try an SD Card Reader (included in my display)...

Thanks!

[Ath]

BMP file size is 177kB
That should be the reason.

Yes, that's larger than available space, and as the SPIFFS file system, used on the internal flash, isn't particularly robust, uploading too large a file may cause other files to be overwritten
We would like to migrate to the LittleFS file system (as used on most of the MAX builds and a few other builds), but there is no in-place upgrade possible without complex manual intervention, so that's an issue to be tackled...

I'll try an SD Card Reader (included in my display)...

That's probably the best solution, just configure the SD-card CS pin on the Hardware tab (once it's connected, other SPI pins go in parallel to the display pins, don't forget the MISO pin...), and after a reboot it should be accessible. There is a special button on the Tools page, aptly named 'SD Card', that'll allow you to upload/manage files there. The "tft,bmp" command will find the file on the SD card as if it was on local flash

[ralftopas]

SD card works fine.
Rules can be stored without problems, but interstingly bmp files are not displayed...

The log file gives this response:
482251: ACT : tft,r,5,33,305,200,white
482282: ACT : tft,bmp,6,34,grundstueck3.bmp
482334: showBmp: Done.
482355: Command unknown: tft,bmp,6,34,grundstueck3.bmp

Any ideay?
Thank you in advance!

Ralf

[Ath]

Hmm, is the spelling and upper/lowercase of the filename correct?
Also, as the documentation states, the bitmap has to be a 24 bit colored bitmap, else it won't work. (Quick check: use a hex editor or Notepad++ to open the file on Windows, and check if the first 2 characters of the file are 'BM' or '42 4d' in hex notation)

Edit: I see that the message "showBmp: Done" is displayed, either wrong or right, I'll change that to show an error if the file signature is not correct.

Edit2: I've created a PR (Pull request) #4767 that has an improvement on this behavior.

[ralftopas]

Hi,
upper/lowercase (if case sensitive?) ist correct. (copy & pasted the filnema multiple times from SD-card file reader tool under advanced tools.
Bitmap is definitely 24bit. checked this too.

Exactly, after reading "showBmp: Done" i was a bit confused.
This should only be displayed in case of correct call of this function.
But the next message "command unknown" ist strange.

Thanks for your help and submitted request!

[Ath]

Exactly, after reading "showBmp: Done" i was a bit confused.
This should only be displayed in case of correct call of this function.
But the next message "command unknown" ist strange.

The 'command unknown' message is the result of the bmp not being according to expectations by the code, hence the change in the code.
Can you shorten the name of the file to 8.3 format, and see if that improves the behavior?

[Ath]

upper/lowercase (if case sensitive?) ist correct. (copy & pasted the filnema multiple times from SD-card file reader tool under advanced tools.
Bitmap is definitely 24bit. checked this too.

In this screenshot, I've marked the 2 bytes that indicate the color-depth of the image. The supported values there are either '18 00' or '01 00' (for 1 bit color-depth).
You can also recognize the '42 4d' (BM) marker at the start of the file.

I remember I had to jump some hoops to actually obtain a 24 bit bmp file, as many paint/graphics programs claim the file to be 24 bit colors, while they actually are 32 bit... (the bytes then would be '20 00').

I'll add another message to the code to signal that situation.

Edit: I see an addition requirement, not stated very clear: Only single-plane, uncompressed images are supported (This is a piece of code I found on the interwebz, and adapted to work with our code-base...)

Edit2: This GH Actions run has that extra error message, can you test that once it's finished building? (You will need to be logged in to your Github account (free) to be able to download the artifacts)

[Ath]

This is one of the bmp files I used as a background during testing. You could try that (zipped to get it unharmed to you, and the forum doesn't allow bmp files as attachment ) it is 221 by 233 pixels

[ralftopas]

Hi,

sorry for the delayed response...

Shortening of the filname got the same log result & error.
Using your image could not resolve the problem...

69310: ACT : tft,bmp,50,50,day_bg.bmp
69361: showBmp: Done.
69383: Command unknown: tft,bmp,50,50,day_bg.bmp

[Ath]

Did you install the updated build from the Actions run I referred to? That should give a more clear error if it can't handle the content of the bmp.

[TD-er]

And can you read back the image from the flash using the file browser of ESPEasy?
Like download it and compare the file's checksum with the file you uploaded.

[ralftopas]

Hi,
just updated the FW.

21074: ACT : tft,bmp,50,50,'day_bg.bmp'
21156: showBmp: File signature error.
21178: Command unknown: tft,bmp,50,50,'day_bg.bmp'

[ralftopas]

Downloading files from SD card results in a 0kb File on my HDD
I will try uploading the image directly via a sd cardreader instead of uploading via webfrontend...

[Ath]

What's the size of your SD-card? Not sure what the max. working card-size would be. Tested mostly using a 2 GB card.

[ralftopas]

16GB...
Manual upload via SD Card reeader failed.

[ralftopas]

Hi,
I ordered a smaller SD card. As soon as I have tested, I’ll give you a feedback!

All the best!

[TD-er]

Maybe the problem is here the difference between SD cards and SDHC.
Upto 2 GB you had the old SD standard, starting from 4 GB you needed to have SDHC cards.
But there are lots of examples online where people only get SDHC cards formatted as 2 GB when you order 2 GB cards (or less...)

Maybe you can also try to format the card as FAT and not as FAT32.
FAT can only be used upto 2 GB, so you may need to repartition the card also to be able to format it as FAT.

[ralftopas]

Next try.

Got a 1GB SD (not SDHC) and formatted as FAT and FAT32.
Upload via ESPEasy frontend and via CardReader.
Checksum is correct.
Still the same error...

Needs to be an SDHC Card?

The Same with your testimage. Same error...

Any suggestions?

[Ath]

The release that was created on 22-08-2023 has the changes I added with extra log-info when it has an issue with the bmp file. Do you have that installed?
And what is the error message that is now in the log when displaying the bmp?

[ralftopas]

Yes,

update was performed.
after the manual update i jus installed the actual official releas, too.

Still this error:
87693: ACT : tft,clear
87806: ACT : tft,bmp,50,50,gs3.bmp
87881: showBmp: File signature error.
87921: Command unknown: tft,bmp,50,50,gs3.bmp

[Ath]

Can you share that gs3.bmp file? (best is to put it in a .zip file and attach that below the message via the Attachments tab) Then I'll be able to investigate.
It doesn't seem to have the correct signature (BM in the first 2 bytes of the file), according to the message in the log.

[ralftopas]

Sure!

[Ath]

Downloaded, extracted and uploaded gs3.bmp to a WS32-SC01 unit (display: ST7796, 320x480 pixels) I have connected at the moment, and issued the command st77xx,bmp,50,50,gs3.bmp, and this is wat it shows:

PXL_20230824_194316128.jpg (968.15 KiB) Viewed 14949 times

and in the log:

Code: Select all

>st77xx,bmp,50,50,gs3.bmp
124562 : Info   : showBmp: bitmap w:305, h:197, dpt:24, colors:16777216, cmp:0, pl:1, x:50, y:50
124564 : Info   : showBmp: x:50, y:50, dw:480, dh:320
124928 : Info   : showBmp: Done.

OK

Looking good to me

[TD-er]

As a question to both...
How is this file uploaded to the SD card?
Just to make sure whatever is written using ESPEasy can be read by it.
And also that ESPEasy should be able to read something written to the file system by another platform (e.g. Windows)

Can you also check whether it makes a difference if ESPEasy is rebooted after uploading the file?

[Ath]

Hmm, I've uploaded it to the internal flash, as the WT32-SC01 doesn't have an outside accessible SD slot, will try tomorrow with another unit that has an SD-card slot, or open this one.

[Ath]

@ralftopas Did you remove the failed upload from the internal flash of the unit? Possibly a file with the same name but size 0 is still there. You can use the File browser button on the Tools page to remove that file. The file is first searched on internal storage and if not found, then on the SD card, if and when available/enabled.

[ralftopas]

Did you remove the failed upload from the internal flash of the unit?

Indeed, that's a good question.
After using the SD Reader this is possible.
Unfortunately i am back on saturday and will check imediately.

The file seems not corrupted after you were able to display...

@TD-er:
File was uploaded via card reader (Laptop) and via web frontend. Both ways did not work out...
And yes, reboot was performed after every upload.

Again: Thank you all for your great support and ideas!

[TD-er]

You're welcome and thanks for actually using the SD card feature as it isn't being used that often (or hardly anybody experiences errors )

[ralftopas]

So,

on internal flash the old file was already deleted.
What i tested:

i uploaded a very tiny image without exceeding the internal flash (15kb, just a green rectangle)
When this file is stored on the internal flash it works.
Log shows:
89576: ACT : tft,bmp, 50,50, test.bmp
89636: showBmp: bitmap w:100, h:50, dpt:24, colors:16777216, cmp:0, pl:1, x:50, y:50
89678: showBmp: x:50, y:50, dw:320, dh:240
89925: showBmp: Done.

after deleting the file from the internal flash and uploading it to the SD card via web frontend this occurs (without editing the rules commands):
23020: ACT : tft,bmp, 50,50, test.bmp
23129: showBmp: File signature error.
23172: Command unknown: tft,bmp, 50,50, test.bmp

It seems the file is found on the SD card but can't be read from it...
It seems there is a problem reading the file correctly from the SD card...

[Ath]

Last Friday I've found an issue when opening a file from SD-card, that I've opened PR #4776 for to fix. A build is available from this Actions run.
Can you please test that, to see if that maybe fixes your issue?

[ralftopas]

Same error...

239538: ACT : tft,bmp, 50,50, test.bmp
239588: showBmp: File signature error.
239604: Command unknown: tft,bmp, 50,50, test.bmp

[Ath]

I've saved your other .bmp on a 256 MB partition of an SD card (reformatted the FAT partition of a Raspberry Pi boot SD ) via a card-reader and put it in a M5Stack (16 MB unit).
Then again used the ili9342,bmp,0,0,gs3.bmp command, and this is the result:
Log:

Code: Select all

>ili9342,bmp,0,0,gs3.bmp
193941 : Info   : showBmp: bitmap w:305, h:197, dpt:24, colors:16777216, cmp:0, pl:1, x:0, y:0
193943 : Info   : showBmp: x:0, y:0, dw:240, dh:320
198353 : Info   : showBmp: Done.

Screen:

PXL_20230827_195108668.jpg (687.94 KiB) Viewed 14832 times

Maybe you can re-partition/format your SD-card to a smaller size, to see if that helps.

In the meantime I'll repartition my SD to a 2 GB FAT partition (it's a 16 GB card) and re-test.

[Ath]

I've tried these options:

Code: Select all

Partition  Size     Result
---------------------------
FAT        256MB    Works
FAT        2GB      Works
exFAT      2GB      file not found (SD seems empty in SD Browser)
FAT32      2GB      Works

Uploaded a different file via the web UI to the SD-card, and that also works, except for the exFAT formatted test, that's just not working at all.

Don't know how we can proceed from here, as in your situation it just doesn't seem to work from SD-card, and here it works as intended

[ralftopas]

Thanks again!

yes, i think my SD cards are the problem.
I'll try later a reformating and give it a new try...
Tanks!

[ralftopas]

After some testing i found the following issue:
- Reading directories and files from SD card via webfrontend works
- upload works
- taking the sd card out of the slot and putting it into my laptop works fine. files can be read and displayed correctly
- trying to download the file or showing the file via browser leads to an error or revelas in a 0kb size image

-> i guess reading the image via rules leads to a 0kb file, reuslting in the known error, that the file ist corrupted.
Therefore, i think the file is correct stored on the SD card but can not be read properly.
Any suggestion what can cause this?

[Ath]

- trying to download the file or showing the file via browser leads to an error or revelas in a 0kb size image

Most of your remarks are clear, but the one I quoted here is not, what is the "browser" you are talking about here, as you also state: "- Reading directories and files from SD card via webfrontend works" ?

[ralftopas]

Have a look:

[Ath]

Have you been able to download the gs3.bmp file from the SD card successfully via that browser? You could try to right-click the file-link, and select Save Link As... (or your browser/language specific version of that) instead of left-clicking the link.

As said before, I'm out of clues, as over here it is all working as intended, on multiple hardware.

[ralftopas]

That is exactly the problem.
Right click -.> Save as downloads a 0kb file, while shown as 180506 byte big file in the webbrowser (see image below).
I guess that reading the file from the sd via rules leads to the aforementioned error, because the image can't be read properly.
Inserting the SD card into my laptop makes it possible to display the image correct...

very strange.

[TD-er]

Sounds like the SD is being accessed in a different mode compared to how it is accessed in the PC.
SD cards can be accessed using 1, 2 or 4 bits at the same time.
However not all cards have all modi implemented.
The only thing I really don't get is why the FAT can be read consistently and the files can't.

Maybe there are other differences too, like how pages are being opened and closed, which isn't supported in the Arduino lib?

Can you look for some SD card diagnostics tools to see whether you have an SD or SD-HC card?
Or maybe even other types?

Or maybe there are other differences I'm not even aware of, like that the card should tell the controller something about voltages? (just thinking out loud)
So maybe some diagnostics tool may show lots of capabilities we can then try to verify or debunk whether the controller you have even supports them and also the library.

[Ath]

That is exactly the problem.
Right click -.> Save as downloads a 0kb file, while shown as 180506 byte big file in the webbrowser (see image below).
I guess that reading the file from the sd via rules leads to the aforementioned error, because the image can't be read properly.
Inserting the SD card into my laptop makes it possible to display the image correct...

very strange.

And what happens if you click on the gs3.bmp filename? It should show the image in the webbrowser.

As said, it all works fine here, so there must be some technical difference we're not aware of yet. Loading the directory-list in the SD-Card browser is also reading from the device, so it's at least partially working

Your last list shows also some relatively large files on the card. Can those be removed, the card formatted and then only the bmp file uploaded?, so the file is most likely at the beginning of the storage area, instead of (possibly) somewhere across some unknown/invisible border. (This would probably indicate a bug in the Arduino FAT implementation used)

[ralftopas]

Thanks again for your ideas!
i'll try a diagnostic tool this evening, reformat the card and try only one file on the card.
i'll report asap.

greetings!

[TD-er]

Maybe also try to format the card in a non-PC, like a camera?
Those typically try to use a more basic approach in formatting SD cards.

[ralftopas]

ok:
chkdsk finds no errors.
formatting the SD via my camera does not solve the problem.

when i upload an image via espeasy webfrontend and put the sd card into my laptop i can read/show the image without any errors.
but espeasy seems ti be unable to read the file.

is there a possibility to read images from a webserver/NAS etc. to avaoid storing it local?

another option is to use an additional sd card reader instead the included reader in my display-..

[TD-er]

Is it possible to exchange the ESP in your project with a 16M flash version?
If so then I guess that's by far the easiest option right now as Ton can't reproduce your SD-card issues.
So it is unclear whether it is something in software or hardware or very specific to the combination of your board and the SD-card(s) you're using.
Right now it is like finding a needle in a haystack, but you have to find the haystack first.

[ralftopas]

I think using an esp32 with 16mb flash or another external sd card reader module are the only ways to solve this problem...

does any one have a link to a esp32 with 38 pins with a 16mb flash? there are different esp wrom versions for example, but most of them have only 4mb flash integrated...
any experience or a good retailer?

thx!

[Ath]

I think using an esp32 with 16mb flash or another external sd card reader module are the only ways to solve this problem...

does any one have a link to a esp32 with 38 pins with a 16mb flash? there are different esp wrom versions for example, but most of them have only 4mb flash integrated...
any experience or a good retailer?

I've been testing on this type of unit (links to the vendor I got it from, no other relation then being a satisfied customer )
And I used the SD-card slot that's installed on the board.
And also tested on this M5Stack unit (though that's rather expensive ) that has an SD-card slot built-in.
I assume they'll ship to Germany too

[Nippy]

Same problem as mentioned above...

4" ST7769 480x320 | SD Cardreader onboard
ESP32 D1 Mini | 4MB | ESP32-WROOM-32 | ESP32-D0WDQ6



If i upload an image to the SD Card

Code: Select all

Command output:
Command unknown: st77xx,bmp,0,0,gs3.bmp

Code: Select all

LOG:
436181: HTTP: st77xx,bmp,0,0,gs3.bmp
436220: showBmp: file not found
436222: Command unknown: st77xx,bmp,0,0,gs3.bmp

But if i upload it on the internal memory, it works halfway

Code: Select all

Command output: ok

Code: Select all

LOG:
577436: HTTP: st77xx,bmp,0,0,gs3.bmp
577452: showBmp: bitmap w:305, h:197, dpt:24, colors:16777216, cmp:0, pl:1, x:0, y:0
577453: showBmp: x:0, y:0, dw:480, dh:320
580630: showBmp: Done.

IMG_20231014_133648.jpg (39.56 KiB) Viewed 13688 times

Other graphics i tried or created have also errors on the top part