Please help with 1MB module

Moderators: grovkillen, Stuntteam, TD-er

Post Reply
Message
Author
tim
Normal user
Posts: 52
Joined: 05 Dec 2015, 11:24

Please help with 1MB module

#1 Post by tim » 08 Feb 2020, 13:44

Today i download the ESPEasy 20200204 release where the static IP crash is fixed.
Testing on a Wemos D1 Mini and this version works ok again.

Now i want to test this version on an LSC smartplug that has an ESP8285 module.
It now runs ESPCoreRules but i want to see how it will work with ESPEasy and compare

I was reading the old wiki and readthedocs website but did not find how to create a small image that can be uploaded on the 8285 module with OTA.
The size is standard more then 800 kB

Using Arduino IDE 1.8.5 and this setting:

#define PLUGIN_BUILD_MINIMAL_OTA

the size will be 629188 bytes and is still too big.

I feel stupid because this must be easy i guessed. :mrgreen: :mrgreen: :mrgreen: :mrgreen: :mrgreen:
Please help with instructions what setting i need to make

TD-er
Core team member
Posts: 2117
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Please help with 1MB module

#2 Post by TD-er » 08 Feb 2020, 22:24

You can download the nightly build and see if there is an 1M OTA version included based on the more recent core library.
If it isn't then the builds we make ourselves are also too big for the 1M 2-step OTA and thus it isn't that easy to get it small enough :)

FanOfHue
Normal user
Posts: 92
Joined: 06 Oct 2018, 10:08

Re: Please help with 1MB module

#3 Post by FanOfHue » 09 Feb 2020, 16:56

TD-er wrote:
08 Feb 2020, 22:24
If it isn't then the builds we make ourselves are also too big for the 1M 2-step OTA and thus it isn't that easy to get it small enough :)
Same issue here on one of my test ESP01 modules. Instead of trying to get a smaller image, i think we could also try to gain some more space on ESP01 modules and ESP8285 based hardware.

I demand OTA so i decided to tweak ESPEasy source a bit and use 64kB SPIFFS instead of default 128kB:

Code: Select all

#define PLUGIN_BUILD_MINIMAL_OTA
#define TASKS_MAX                        8
#define DAT_OFFSET_TASKS                 4096
#define DAT_OFFSET_CONTROLLER            (DAT_OFFSET_TASKS + (DAT_TASKS_DISTANCE * TASKS_MAX))
#define DAT_OFFSET_CUSTOM_CONTROLLER     (DAT_OFFSET_CONTROLLER + (DAT_CONTROLLER_SIZE * CONTROLLER_MAX))
#define CONFIG_FILE_SIZE                 28672
and change this check:
if (SpiffsSectors() < 32)
into:
if (SpiffsSectors() < 16)

This way i can easily switch between ESPEasy and ESPCoreRules by just using OTA.
(ESPCoreRules and ESPEasy can be switched without reformatting or reconfiguring)

ESPCoreRules can still do one-step OTA, but for ESPEasy a two-step OTA is needed. But with 64kB SPIFFS, the max size is a lot larger:
1M_uploader.png
1M_uploader.png (12.45 KiB) Viewed 836 times
And then it seems to fit with a lot smaller config file:
ESPEasy64kSPIFFS.png
ESPEasy64kSPIFFS.png (24.22 KiB) Viewed 836 times
This has been verified to work on release 20200204 in my lab, sending data on HTTP to Domoticz, but likely unsupported.

But maybe ESPEasy developers could think of a similar solution that has official support?

TD-er
Core team member
Posts: 2117
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Please help with 1MB module

#4 Post by TD-er » 09 Feb 2020, 17:52

I tried the same once, with 64k SPIFFS, but that was causing lots of issues.

FanOfHue
Normal user
Posts: 92
Joined: 06 Oct 2018, 10:08

Re: Please help with 1MB module

#5 Post by FanOfHue » 09 Feb 2020, 18:29

TD-er wrote:
09 Feb 2020, 17:52
I tried the same once, with 64k SPIFFS, but that was causing lots of issues.
I started out with 32768 and also had serious issues. Net free space caused other dat files to truncate and the config.dat did not store any data. But all issues were solved with 28672 size.
Test unit is running fine. even added the P253 plugin and it's now part of the UDP message bus network.

I think the config.dat size can even be reduced by setting nr of controllers to one or two. Or less tasks.
Just calculating, 4k basic + 8k for 4 tasks and 2*2*1k for one controller: 16kB (right?)

Could also skip notification.dat (who uses this?)

Also, recompiled the uploader using old Arduino core 2.0.0. and it offers even more upload space. Sketch is only 251k:
1M_uploader_core200.png
1M_uploader_core200.png (12.57 KiB) Viewed 822 times

TD-er
Core team member
Posts: 2117
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Please help with 1MB module

#6 Post by TD-er » 09 Feb 2020, 18:48

Run "meminfodetail" as command on the ESP and when you then open the sysinfo page, you can see the layout of the settings file graphically represented.
Changing the number of tasks will render the settings incompatible with other nodes.
Saving settings on a small SPIFFS will later lead to issues as the file system needs a number of free (unallocated) blocks to operate.
With a 64k SPIFFS you are already at the limit of the number of free blocks and thus any fragmentation of the file will keep pages occupied so you don't have 2 blocks (of 16 or 32 pages) where all pages are either free or marked as not used.

FanOfHue
Normal user
Posts: 92
Joined: 06 Oct 2018, 10:08

Re: Please help with 1MB module

#7 Post by FanOfHue » 09 Feb 2020, 21:42

So we're living on the edge here :mrgreen:
Some changes to save another 10kB:

Code: Select all

#define TASKS_MAX                        4
#define CONTROLLER_MAX                   3
#define DAT_OFFSET_TASKS                 4096
#define DAT_TASKS_DISTANCE               2048
#define DAT_CONTROLLER_SIZE              1024
#define DAT_OFFSET_CONTROLLER            (DAT_OFFSET_TASKS + (DAT_TASKS_DISTANCE * TASKS_MAX))
#define DAT_OFFSET_CUSTOM_CONTROLLER     (DAT_OFFSET_CONTROLLER + (DAT_CONTROLLER_SIZE * CONTROLLER_MAX))
#define CONFIG_FILE_SIZE                 18432
Even tried to set it to one task and one controller, but the "static_assertions" build into the source code won't let me change some parameters. Are there some (hidden) dependencies to disallow setting those to 1 ??

It also seems that the 1MB minimum OTA does not have the notification feature, but the notification.dat file is still present and it can't be deleted from the webgui. Could save another 4kB.

SPIFFS now has 16kB free. I don't expect many updates on SPIFFS filesystem. This firmware will typically run on a simple switch like many currently available Wifi Smart-Plugs with ESP8285 modules. They can only do one thing (on/off) and i think they only need a one-time config to setup anyway. Much more important for me is the ability to change firmware through OTA. (for the LSC smart plug, it is not so easy to get access to serial!!)

TD-er
Core team member
Posts: 2117
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Please help with 1MB module

#8 Post by TD-er » 10 Feb 2020, 11:27

You can also have a look at the webserver code.
We have a lot of defines to strip out pages, which may save quite a bit in code size.

tim
Normal user
Posts: 52
Joined: 05 Dec 2015, 11:24

Re: Please help with 1MB module

#9 Post by tim » 10 Feb 2020, 17:15

Thanks for discussion, but this all looks not really easy and standard solution for me
Also no other users comments on this, so must be the only one here on this forum
Tried TuyaConvert and you have Tasmota running so easy
So not waiting anymore, i moved over to Tasmota with better support for the Tuya modules

martinus
Normal user
Posts: 1
Joined: 15 Feb 2020, 16:57

Re: Please help with 1MB module

#10 Post by martinus » 19 Feb 2020, 18:08

It is sad to see users move to alternate firmwares because they experience that things as no longer as 'EASY' as they used to be. While ESP Easy Mega was initially intended to be used on all 1M+ flash modules it seems tough to keep it running on those 1M modules with OTA still enabled.
While only the early adopters like me are likely still using those ancient ESP-01 modules, the ESP8285 is still worth while to support as some companies still fit them into recent products.

Instead of reducing SPIFFS size, there may be another escape route that i have recently used to upgrade my 'old' ESP-01 modules using OTA:

I used an old Core-2.0.0. compiled BasicOTA sketch (with my Wifi credentials as well as a static IP)
The firmware does not use SPIFFS to keep the size as small as possible: 219 kB

Using a two-step approach with this binary, we can keep SPIFFS at 128 kB and OTA upload binaries to a max size of 655 kB.

My current bin is 639 kB for the ESP-01 and Sonoff. Just compiled using the latest ESPEasy release. Plugins C001, P001 and some custom stuff as well.
I could update it using two-step OTA without issues:

1) Use the ESP Easy web-gui to upload the BasicOTA binary.
2) Use the ESPOTA tool to upload a new ESPEasy binary.

Only change required is that ESPEasy does not seem to offer OTA upload on the web-gui because it calculates free space based on this:

Code: Select all

  #define SMALLEST_OTA_IMAGE 276848
And that needs to be changed then.

Noticing this discussion, i just felt the need to step in and add my two cents...
But i don't know if Gijs wants to have a look at it...

regards,

Martinus

User avatar
grovkillen
Core team member
Posts: 3439
Joined: 19 Jan 2017, 12:56
Location: Hudiksvall, Sweden
Contact:

Re: Please help with 1MB module

#11 Post by grovkillen » 19 Feb 2020, 18:45

martinus wrote:
19 Feb 2020, 18:08
It is sad to see users move to alternate firmwares because they experience that things as no longer as 'EASY' as they used to be.

...

Noticing this discussion, i just felt the need to step in and add my two cents...
But i don't know if Gijs wants to have a look at it...

regards,

Martinus
I understand your position but please understand ours too. We are working our butts of for this. We took over the project and needed to feel like we knew it inside out. Lots of functionality was tangled up with parts of the code which made code duplication more normal than an exception. Along this refactoring a lot of new plugins has been added. The core has grown too, as well as some dependency libraries. The result is a very competent firmware boarderlining a OS.

Please support us in this quest, we are not going to make things happen if we don't feel like anyone is accepting our vision.
ESP Easy Flasher [flash tool and wifi setup at flash time]
ESP Easy Webdumper [easy screendumping of your units]
ESP Easy Netscan [find units]
Official shop: https://firstbyte.shop/
Sponsor ESP Easy, we need you :idea: :idea: :idea:

FanOfHue
Normal user
Posts: 92
Joined: 06 Oct 2018, 10:08

Re: Please help with 1MB module

#12 Post by FanOfHue » 19 Feb 2020, 21:48

martinus wrote:
19 Feb 2020, 18:08
I used an old Core-2.0.0. compiled BasicOTA sketch
Thanks for this simple solution :) Still a workaround but not as worse as mine :oops:
Now i don't have to tweak the TASK settings for my LSC smart plug anymore

@moderator: May i suggest to close this topic because:
1) I think no one else has the issue so we may be fighting over nothing here
2) The original topic starter already left the building anyway
3) Discussion now seems to turn into an off-topic one

Post Reply

Who is online

Users browsing this forum: No registered users and 21 guests