Please help with 1MB module

Moderators: grovkillen, Stuntteam, TD-er

Post Reply
Message
Author
tim
Normal user
Posts: 52
Joined: 05 Dec 2015, 11:24

Please help with 1MB module

#1 Post by tim » 08 Feb 2020, 13:44

Today i download the ESPEasy 20200204 release where the static IP crash is fixed.
Testing on a Wemos D1 Mini and this version works ok again.

Now i want to test this version on an LSC smartplug that has an ESP8285 module.
It now runs ESPCoreRules but i want to see how it will work with ESPEasy and compare

I was reading the old wiki and readthedocs website but did not find how to create a small image that can be uploaded on the 8285 module with OTA.
The size is standard more then 800 kB

Using Arduino IDE 1.8.5 and this setting:

#define PLUGIN_BUILD_MINIMAL_OTA

the size will be 629188 bytes and is still too big.

I feel stupid because this must be easy i guessed. :mrgreen: :mrgreen: :mrgreen: :mrgreen: :mrgreen:
Please help with instructions what setting i need to make

TD-er
Core team member
Posts: 2791
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Please help with 1MB module

#2 Post by TD-er » 08 Feb 2020, 22:24

You can download the nightly build and see if there is an 1M OTA version included based on the more recent core library.
If it isn't then the builds we make ourselves are also too big for the 1M 2-step OTA and thus it isn't that easy to get it small enough :)

FanOfHue
Normal user
Posts: 96
Joined: 06 Oct 2018, 10:08

Re: Please help with 1MB module

#3 Post by FanOfHue » 09 Feb 2020, 16:56

TD-er wrote:
08 Feb 2020, 22:24
If it isn't then the builds we make ourselves are also too big for the 1M 2-step OTA and thus it isn't that easy to get it small enough :)
Same issue here on one of my test ESP01 modules. Instead of trying to get a smaller image, i think we could also try to gain some more space on ESP01 modules and ESP8285 based hardware.

I demand OTA so i decided to tweak ESPEasy source a bit and use 64kB SPIFFS instead of default 128kB:

Code: Select all

#define PLUGIN_BUILD_MINIMAL_OTA
#define TASKS_MAX                        8
#define DAT_OFFSET_TASKS                 4096
#define DAT_OFFSET_CONTROLLER            (DAT_OFFSET_TASKS + (DAT_TASKS_DISTANCE * TASKS_MAX))
#define DAT_OFFSET_CUSTOM_CONTROLLER     (DAT_OFFSET_CONTROLLER + (DAT_CONTROLLER_SIZE * CONTROLLER_MAX))
#define CONFIG_FILE_SIZE                 28672
and change this check:
if (SpiffsSectors() < 32)
into:
if (SpiffsSectors() < 16)

This way i can easily switch between ESPEasy and ESPCoreRules by just using OTA.
(ESPCoreRules and ESPEasy can be switched without reformatting or reconfiguring)

ESPCoreRules can still do one-step OTA, but for ESPEasy a two-step OTA is needed. But with 64kB SPIFFS, the max size is a lot larger:
1M_uploader.png
1M_uploader.png (12.45 KiB) Viewed 8146 times
And then it seems to fit with a lot smaller config file:
ESPEasy64kSPIFFS.png
ESPEasy64kSPIFFS.png (24.22 KiB) Viewed 8146 times
This has been verified to work on release 20200204 in my lab, sending data on HTTP to Domoticz, but likely unsupported.

But maybe ESPEasy developers could think of a similar solution that has official support?

TD-er
Core team member
Posts: 2791
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Please help with 1MB module

#4 Post by TD-er » 09 Feb 2020, 17:52

I tried the same once, with 64k SPIFFS, but that was causing lots of issues.

FanOfHue
Normal user
Posts: 96
Joined: 06 Oct 2018, 10:08

Re: Please help with 1MB module

#5 Post by FanOfHue » 09 Feb 2020, 18:29

TD-er wrote:
09 Feb 2020, 17:52
I tried the same once, with 64k SPIFFS, but that was causing lots of issues.
I started out with 32768 and also had serious issues. Net free space caused other dat files to truncate and the config.dat did not store any data. But all issues were solved with 28672 size.
Test unit is running fine. even added the P253 plugin and it's now part of the UDP message bus network.

I think the config.dat size can even be reduced by setting nr of controllers to one or two. Or less tasks.
Just calculating, 4k basic + 8k for 4 tasks and 2*2*1k for one controller: 16kB (right?)

Could also skip notification.dat (who uses this?)

Also, recompiled the uploader using old Arduino core 2.0.0. and it offers even more upload space. Sketch is only 251k:
1M_uploader_core200.png
1M_uploader_core200.png (12.57 KiB) Viewed 8132 times

TD-er
Core team member
Posts: 2791
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Please help with 1MB module

#6 Post by TD-er » 09 Feb 2020, 18:48

Run "meminfodetail" as command on the ESP and when you then open the sysinfo page, you can see the layout of the settings file graphically represented.
Changing the number of tasks will render the settings incompatible with other nodes.
Saving settings on a small SPIFFS will later lead to issues as the file system needs a number of free (unallocated) blocks to operate.
With a 64k SPIFFS you are already at the limit of the number of free blocks and thus any fragmentation of the file will keep pages occupied so you don't have 2 blocks (of 16 or 32 pages) where all pages are either free or marked as not used.

FanOfHue
Normal user
Posts: 96
Joined: 06 Oct 2018, 10:08

Re: Please help with 1MB module

#7 Post by FanOfHue » 09 Feb 2020, 21:42

So we're living on the edge here :mrgreen:
Some changes to save another 10kB:

Code: Select all

#define TASKS_MAX                        4
#define CONTROLLER_MAX                   3
#define DAT_OFFSET_TASKS                 4096
#define DAT_TASKS_DISTANCE               2048
#define DAT_CONTROLLER_SIZE              1024
#define DAT_OFFSET_CONTROLLER            (DAT_OFFSET_TASKS + (DAT_TASKS_DISTANCE * TASKS_MAX))
#define DAT_OFFSET_CUSTOM_CONTROLLER     (DAT_OFFSET_CONTROLLER + (DAT_CONTROLLER_SIZE * CONTROLLER_MAX))
#define CONFIG_FILE_SIZE                 18432
Even tried to set it to one task and one controller, but the "static_assertions" build into the source code won't let me change some parameters. Are there some (hidden) dependencies to disallow setting those to 1 ??

It also seems that the 1MB minimum OTA does not have the notification feature, but the notification.dat file is still present and it can't be deleted from the webgui. Could save another 4kB.

SPIFFS now has 16kB free. I don't expect many updates on SPIFFS filesystem. This firmware will typically run on a simple switch like many currently available Wifi Smart-Plugs with ESP8285 modules. They can only do one thing (on/off) and i think they only need a one-time config to setup anyway. Much more important for me is the ability to change firmware through OTA. (for the LSC smart plug, it is not so easy to get access to serial!!)

TD-er
Core team member
Posts: 2791
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Please help with 1MB module

#8 Post by TD-er » 10 Feb 2020, 11:27

You can also have a look at the webserver code.
We have a lot of defines to strip out pages, which may save quite a bit in code size.

tim
Normal user
Posts: 52
Joined: 05 Dec 2015, 11:24

Re: Please help with 1MB module

#9 Post by tim » 10 Feb 2020, 17:15

Thanks for discussion, but this all looks not really easy and standard solution for me
Also no other users comments on this, so must be the only one here on this forum
Tried TuyaConvert and you have Tasmota running so easy
So not waiting anymore, i moved over to Tasmota with better support for the Tuya modules

martinus
Normal user
Posts: 81
Joined: 15 Feb 2020, 16:57

Re: Please help with 1MB module

#10 Post by martinus » 19 Feb 2020, 18:08

It is sad to see users move to alternate firmwares because they experience that things as no longer as 'EASY' as they used to be. While ESP Easy Mega was initially intended to be used on all 1M+ flash modules it seems tough to keep it running on those 1M modules with OTA still enabled.
While only the early adopters like me are likely still using those ancient ESP-01 modules, the ESP8285 is still worth while to support as some companies still fit them into recent products.

Instead of reducing SPIFFS size, there may be another escape route that i have recently used to upgrade my 'old' ESP-01 modules using OTA:

I used an old Core-2.0.0. compiled BasicOTA sketch (with my Wifi credentials as well as a static IP)
The firmware does not use SPIFFS to keep the size as small as possible: 219 kB

Using a two-step approach with this binary, we can keep SPIFFS at 128 kB and OTA upload binaries to a max size of 655 kB.

My current bin is 639 kB for the ESP-01 and Sonoff. Just compiled using the latest ESPEasy release. Plugins C001, P001 and some custom stuff as well.
I could update it using two-step OTA without issues:

1) Use the ESP Easy web-gui to upload the BasicOTA binary.
2) Use the ESPOTA tool to upload a new ESPEasy binary.

Only change required is that ESPEasy does not seem to offer OTA upload on the web-gui because it calculates free space based on this:

Code: Select all

  #define SMALLEST_OTA_IMAGE 276848
And that needs to be changed then.

Noticing this discussion, i just felt the need to step in and add my two cents...
But i don't know if Gijs wants to have a look at it...

regards,

Martinus

User avatar
grovkillen
Core team member
Posts: 3607
Joined: 19 Jan 2017, 12:56
Location: Hudiksvall, Sweden
Contact:

Re: Please help with 1MB module

#11 Post by grovkillen » 19 Feb 2020, 18:45

martinus wrote:
19 Feb 2020, 18:08
It is sad to see users move to alternate firmwares because they experience that things as no longer as 'EASY' as they used to be.

...

Noticing this discussion, i just felt the need to step in and add my two cents...
But i don't know if Gijs wants to have a look at it...

regards,

Martinus
I understand your position but please understand ours too. We are working our butts of for this. We took over the project and needed to feel like we knew it inside out. Lots of functionality was tangled up with parts of the code which made code duplication more normal than an exception. Along this refactoring a lot of new plugins has been added. The core has grown too, as well as some dependency libraries. The result is a very competent firmware boarderlining a OS.

Please support us in this quest, we are not going to make things happen if we don't feel like anyone is accepting our vision.
ESP Easy Flasher [flash tool and wifi setup at flash time]
ESP Easy Webdumper [easy screendumping of your units]
ESP Easy Netscan [find units]
Official shop: https://firstbyte.shop/
Sponsor ESP Easy, we need you :idea: :idea: :idea:

FanOfHue
Normal user
Posts: 96
Joined: 06 Oct 2018, 10:08

Re: Please help with 1MB module

#12 Post by FanOfHue » 19 Feb 2020, 21:48

martinus wrote:
19 Feb 2020, 18:08
I used an old Core-2.0.0. compiled BasicOTA sketch
Thanks for this simple solution :) Still a workaround but not as worse as mine :oops:
Now i don't have to tweak the TASK settings for my LSC smart plug anymore

@moderator: May i suggest to close this topic because:
1) I think no one else has the issue so we may be fighting over nothing here
2) The original topic starter already left the building anyway
3) Discussion now seems to turn into an off-topic one

happytm
Normal user
Posts: 105
Joined: 15 Aug 2016, 17:53

Re: Please help with 1MB module

#13 Post by happytm » 02 Mar 2020, 18:03

It is sad to see users move to alternate firmwares because they experience that things as no longer as 'EASY' as they used to be. While ESP Easy Mega was initially intended to be used on all 1M+ flash modules it seems tough to keep it running on those 1M modules with OTA still enabled.
While only the early adopters like me are likely still using those ancient ESP-01 modules, the ESP8285 is still worth while to support as some companies still fit them into recent products.
Welcome back. You are absolutely right about firmware not being Easy anymore. I was the user for ESPEasy from beginning. After you left the firmware has gone downhill with lot of stability issues. There seems to be efforts to over promising good documentation and stable version for last 2 years but none of that has happened yet.Every couple of months there is promise of very good UI or documentation is around the corner but it does not come true. You have to just look at open issues at github Repo. Meanwhile Tasmota and ESPHome firmware have made big progress and many people I follow have moved on to these 2 firmware.
I used an old Core-2.0.0. compiled BasicOTA sketch (with my Wifi credentials as well as a static IP)
The firmware does not use SPIFFS to keep the size as small as possible: 219 kB

Using a two-step approach with this binary, we can keep SPIFFS at 128 kB and OTA upload binaries to a max size of 655 kB.

My current bin is 639 kB for the ESP-01 and Sonoff. Just compiled using the latest ESPEasy release. Plugins C001, P001 and some custom stuff as well.
I could update it using two-step OTA without issues:

1) Use the ESP Easy web-gui to upload the BasicOTA binary.
2) Use the ESPOTA tool to upload a new ESPEasy binary.

Only change required is that ESPEasy does not seem to offer OTA upload on the web-gui because it calculates free space based on this:
CODE: SELECT ALL

#define SMALLEST_OTA_IMAGE 276848
And that needs to be changed then.
Can you please post & link your version of code somewhere so I can use it?

Thanks.

happytm
Normal user
Posts: 105
Joined: 15 Aug 2016, 17:53

Re: Please help with 1MB module

#14 Post by happytm » 02 Mar 2020, 19:30

I want to acknowledge and thank couple of users who have made real efforts to go back 2-3 years and revive old stable code:

1. https://github.com/enesbcs/ESPEasyRetro
2. https://github.com/SmartNodeRules/ESPCoreRules

I use RPIEasy @ https://github.com/enesbcs/rpieasy for my main controller now a days. It is based on ESPEasy firmware but code is written from scratch in python. It is very stable code and just works without any issue. Here is a developer who created this firmware by himself with a full time job as far as I know within 1 year time.

Thanks.

TD-er
Core team member
Posts: 2791
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Please help with 1MB module

#15 Post by TD-er » 03 Mar 2020, 09:11

happytm wrote:
02 Mar 2020, 18:03
[...]

Welcome back. You are absolutely right about firmware not being Easy anymore. I was the user for ESPEasy from beginning. After you left the firmware has gone downhill with lot of stability issues. There seems to be efforts to over promising good documentation and stable version for last 2 years but none of that has happened yet.Every couple of months there is promise of very good UI or documentation is around the corner but it does not come true. You have to just look at open issues at github Repo. Meanwhile Tasmota and ESPHome firmware have made big progress and many people I follow have moved on to these 2 firmware.
[...]
It is a bit sad to read, but I do (partly) agree to what you're saying here.

To keep the issues list open, is a choice of mine and it may be used as an indication of how good or bad a product is.
If you look at the Tasmota issue list, there is almost no open issue as they get closed quite aggressively.
The reason for closing may be one of "try-this-and-that-close-for-now", or "duly noted => close".

As I don't have any other means of keeping track of the todo list, it does remain open.
I guess keeping them open may lead to frustration as users get the idea nothing gets done with their issue, but closing may have the same feeling.
The only way it can be done right is by fixing it immediately, but there's simply not enough man power to do so.
There seems to be efforts to over promising good documentation and stable version for last 2 years but none of that has happened yet
Well, I was under the impression the versions since Sept were quite stable, with lots of reports of uptimes way over 100 days again.
And the documentation on https://espeasy.readthedocs.io/en/latest/ is improving quite a lot.
So please let me know how I can improve there.

User avatar
ThomasB
Normal user
Posts: 696
Joined: 17 Jun 2018, 20:41
Location: USA

Re: Please help with 1MB module

#16 Post by ThomasB » 03 Mar 2020, 18:39

Regarding the OTA support for the older 1MB modules, I do miss that. My workaround has been to swap the 1MB Flash chip with 4MB. For sure this is not a practical solution for most users. And Sonoff has killed the Flash upgrade idea by "improving" some of their modules with the crippled ESP8285 chip.
After you left the firmware has gone downhill with lot of stability issues.
The ongoing stability issues have hurt ESPEz's reputation. But thanks to TD-er, the most recent releases have fixed the stability problems in my ESPEasy installations. Others have reported the same good news.
Every couple of months there is promise of very good UI or documentation is around the corner but it does not come true.
Regarding UI improvements, I'm not convinced this is worth the effort. At least not yet.

But well maintained documentation (with useful examples) is required to ensure a good user experience. Without this the user base will be limited to gear heads that don't mind reading source code to get the info they need. Instead, we should insist that all operational changes and/or new plugin's include useful user-ready documentation on the wiki.

The reality is that this is an open source (community driven) project. We have only ourselves to blame (or praise) on the overall experience it offers.

- Thomas

TD-er
Core team member
Posts: 2791
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Please help with 1MB module

#17 Post by TD-er » 03 Mar 2020, 22:15

ThomasB wrote:
03 Mar 2020, 18:39
Regarding the OTA support for the older 1MB modules, I do miss that. My workaround has been to swap the 1MB Flash chip with 4MB. For sure this is not a practical solution for most users. And [...]
Have you seen the most recent merges?
Last merges I made today brought the build size of the minimal OTA builds below the max again, even for core 2.6.3

I've been busy last few days.

User avatar
ThomasB
Normal user
Posts: 696
Joined: 17 Jun 2018, 20:41
Location: USA

Re: Please help with 1MB module

#18 Post by ThomasB » 04 Mar 2020, 06:02

Last merges I made today brought the build size of the minimal OTA builds below the max again, even for core 2.6.3
Nice! I wanted to check the minimal OTA build using your mega-20200301 release. I have a Sonoff running mega-20200222, but this recent version has OTA disabled. So my experiment will have to wait until I get the device out of the attic and serial flash it to OTA-compatible firmware.

- Thomas

FanOfHue
Normal user
Posts: 96
Joined: 06 Oct 2018, 10:08

Re: Please help with 1MB module

#19 Post by FanOfHue » 04 Mar 2020, 08:53

Of course this is all hobby, so who are we to judge the developers in any way?
(it could be different if we'd payed for it ;) )

But just some free advice from me :mrgreen: :mrgreen: :mrgreen: :

- Stop working on code cleanup/restrucure for now because it's still a lot of work with risks of new instabilities
- Stop working on the new UI for now. ESP Easy has a perfect GUI already IMHO
- Try to focus ONLY on stability, check the real showstoppers from the issue list and declare a CHANGE FREEZE for a few months
- When the stable release candidate is out and being verified by end users:
---- focus on real issues only during a few months
---- focus on your documentation backlog
- Publish the new stable and then move forward with the project for the next cycle.

In the next cycle:
- Cleanup the issue list, prioritize
- Work on the new GUI
- Keep up the good work

TD-er
Core team member
Posts: 2791
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Please help with 1MB module

#20 Post by TD-er » 04 Mar 2020, 10:13

FanOfHue wrote:
04 Mar 2020, 08:53
- Stop working on code cleanup/restrucure for now because it's still a lot of work with risks of new instabilities
Well this is really needed to get the code size down as you may have seen in the last few days.
It is a lot of work and it is done in small steps at a time, so it is controllable with true focus on stability.
Also a lot of the movement into .h/.cpp files is meant to make things more stable.
What was happening is that some code was compiled and linked such that there were multiple instances of global variables with the same name.
Not only does that take extra resources, but also it lead to hard to reproduce bugs.
FanOfHue wrote:
04 Mar 2020, 08:53
- Stop working on the new UI for now. ESP Easy has a perfect GUI already IMHO
Grovkillen is mainly working on the UI and only every now and then I add a new JSON entry for him to get access to functionality for the GUI.
The release of ESPEasy will not wait for the new UI, that will be a separate release.

The reason why we want it though, is to make it possible to split UI and core functionality.
This will make the binaries much smaller (100 - 150k) and also it does allow to work on them separately.
For future plans it also allows to handle multiple nodes in one UI, but that's future planning.
FanOfHue wrote:
04 Mar 2020, 08:53
- Try to focus ONLY on stability, check the real showstoppers from the issue list and declare a CHANGE FREEZE for a few months
See first part of the reply.
The focus is mainly on working towards a release.
If you look at the last few months of commits, there have been only a few new features added and some of those were needed for paid projects I work on.
As you may know, my job is to make rapid prototypes for sensor applications and for that I do use ESPEasy as my toolbox.
So every project I get, will result in improvements and some additions to the ESPEasy code.


As far as I can see now, the main issues are:
- Strange issues with DNS lookups => lead to crashes => probably core lib issue.
- In some circumstances the ESP may take minutes before the first network activity is successful.
- Switch plugin is still acting strange and way too complicated
- Dallas plugin is acting strange in some specific setups.
- Some issues with MQTT controllers. Domoticz MQTT is working very stable, so may have to setup OpenHAB here to see what may be happening.
- Include 1MB modules again.


If those issues are either fixed or made perfectly clear what is causing them (so we can list it as a known issue), then I would call it a release.

FanOfHue wrote:
04 Mar 2020, 08:53
- When the stable release candidate is out and being verified by end users:
---- focus on real issues only during a few months
---- focus on your documentation backlog
- Publish the new stable and then move forward with the project for the next cycle.

In the next cycle:
- Cleanup the issue list, prioritize
- Work on the new GUI
- Keep up the good work
Agree.

TD-er
Core team member
Posts: 2791
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Please help with 1MB module

#21 Post by TD-er » 05 Mar 2020, 12:19

Hmm I don't get it.
Last few builds made by Travis were all below 614384 bytes for the minimal OTA builds and now last night's build:

Code: Select all

FAILED: too large 617696 > 614384 Bytes. (minimal_core_261_ESP8266_1M_OTA)
FAILED: too large 617696 > 614384 Bytes. (minimal_core_261_ESP8285_1M_OTA)
FAILED: too large 618416 > 614384 Bytes. (minimal_core_261_sdk3_ESP8266_1M_OTA)
FAILED: too large 618400 > 614384 Bytes. (minimal_core_261_sdk3_ESP8285_1M_OTA)
Edit:
Hmm tested a few builds here and also looked at the last Travis outputs.
It seems like the reported "Flash" size is later appended by roughly 4k.
The final bin size is also the size of the sketch reported, so maybe it is later appending the boot loader?

This is really a bummer :(

Post Reply

Who is online

Users browsing this forum: Bing [Bot] and 13 guests