Smaller bin files - more stripdown options in custom.h?

[moelski]

Hi !

Today I did some research regarding OTA ...
At the moment my compiled build takes ~666244 bytes. If I use a 128kb Spiffs (64 kb are not enough ? ) there are 892912 Bytes left.
And my build only contains 1 Controller und 4 Devices ...

I also compiled a EspBuddy (https://github.com/soif/EspBuddy) and ended with a ESPEasyUploaderMega.OTA.ino.bin.gz of ~257kb ...
Ok let´s do some math ...

892912 - 666244 = ~226kb

So I just wonder if you can add some additional stripdown options There are very interesting parts in EspEasy, but in a daily usage you won´t need them at all ...

Here is my (unfinished) list where you could save space ...
* Complete disable the rules section. If you don´t need the rules it takes a lot of space in the firmware ...
* Option to disable the favicon
* Option to strip down the CSS data (it could look a little bit more ugly, but it will still work )
* Option to store the CSS in the SPIFFS. 5kb are no problem and left in Spiffs
* Option to disable Copy to Github and Copy to Clipboard Option
* sysinfo page is really nice, but in daily usage more or less useless
* Show Json -> not needed in daily usage
* Timing Stats -> not needed in daily usage
* System variables page -> not really needed if you don´t use rules
* I2C Scan -> Nice, but also not really needed
* Option to disable Inter-ESPEasy Network at all
* Option to disable Syslog
* Option to disable NTP sync
* Maybe some other things we can disable in WebStaticData.h
* Disable Autorefresh of values at the devices tab which uses JavaScript
* Disable githublogo

The above option should be options / defines in the custom.h

So it would be really nice if the firmware size is more scaleable with the custom.h config.
That said OTA updates would be possible again using the 2-step OTA methode.

What do you think about these suggestions ?

[TD-er]

A lot of those already are wrapped in #ifdef or #ifndef statements and thus can be excluded from the build.

[moelski]

Hmm ok.
But it´s not really userfriendly to have spreaded defines all over the project.
Do you think it´s possible to get them all in one spot like the custom.h?

As said bevor I come from Tasmota. There you have one file and can customize the features und devices.
Very handy

Sorry to bother you with this topic.

[TD-er]

The #ifdef checks are there so you can define them in the Custom.h file if you like.


For example the sysinfo page you mentioned, is wrapped in #ifdef WEBSERVER_SYSINFO

A quick search of that define in the code base points to the define_plugin_sets.h file with this section:

Code: Select all

/******************************************************************************\
 * WebServer pages   **********************************************************
\******************************************************************************/
// FIXME TD-er: Make useful selections for these pages to be included. (e.g. view only)

#ifndef WEBSERVER_CUSTOM_BUILD_DEFINED
    #ifndef WEBSERVER_TIMINGSTATS
        #define WEBSERVER_TIMINGSTATS
    #endif
    #ifndef WEBSERVER_SYSVARS
        #define WEBSERVER_SYSVARS
    #endif
    #ifndef WEBSERVER_NEW_UI
    //    #define WEBSERVER_NEW_UI
    #endif
    #ifndef WEBSERVER_I2C_SCANNER
        #define WEBSERVER_I2C_SCANNER
    #endif
    #ifndef WEBSERVER_FAVICON
        #define WEBSERVER_FAVICON
    #endif
    #ifndef WEBSERVER_LOG
        #define WEBSERVER_LOG
    #endif
    #ifndef WEBSERVER_GITHUB_COPY
        #define WEBSERVER_GITHUB_COPY
    #endif
    #ifndef WEBSERVER_ROOT
        #define WEBSERVER_ROOT
    #endif
    #ifndef WEBSERVER_ADVANCED
        #define WEBSERVER_ADVANCED
    #endif
    #ifndef WEBSERVER_CONFIG
        #define WEBSERVER_CONFIG
    #endif
    #ifndef WEBSERVER_CONTROL
        #define WEBSERVER_CONTROL
    #endif
    #ifndef WEBSERVER_CONTROLLERS
        #define WEBSERVER_CONTROLLERS
    #endif
    #ifndef WEBSERVER_DEVICES
        #define WEBSERVER_DEVICES
    #endif
    #ifndef WEBSERVER_DOWNLOAD
        #define WEBSERVER_DOWNLOAD
    #endif
    #ifndef WEBSERVER_FACTORY_RESET
        #define WEBSERVER_FACTORY_RESET
    #endif
    #ifndef WEBSERVER_FILELIST
        #define WEBSERVER_FILELIST
    #endif
    #ifndef WEBSERVER_HARDWARE
        #define WEBSERVER_HARDWARE
    #endif
    #ifndef WEBSERVER_PINSTATES
        #define WEBSERVER_PINSTATES
    #endif
    #ifndef WEBSERVER_RULES
        #define WEBSERVER_RULES
    #endif
    #ifndef WEBSERVER_SETUP
        #define WEBSERVER_SETUP
    #endif
    #ifndef WEBSERVER_SYSINFO
        #define WEBSERVER_SYSINFO
    #endif
    #ifndef WEBSERVER_TOOLS
        #define WEBSERVER_TOOLS
    #endif
    #ifndef WEBSERVER_UPLOAD
        #define WEBSERVER_UPLOAD
    #endif
    #ifndef WEBSERVER_WIFI_SCANNER
        #define WEBSERVER_WIFI_SCANNER
    #endif
#endif

In short, just define WEBSERVER_CUSTOM_BUILD_DEFINED to make sure the default set is not defined and then you can define the pages you want to have.

[moelski]

Ok, will look at that and try.

Thx. so far.

[moelski]

Hi !

Ok after my Sonoff controller is almost running I tried a little bit at this topic. Getting the BIN file so smal that OTA will work.

Attached is my custom.h for reference ...

I started with a basic build which includes the following:
* my new Plugin
* DS1820 Support
* Input Switch Support
* my new Controller for Sonoff Adapter in ioBroker

If I compile that sketch I end up with 663904 bytes using the Arduino IDE.

Now I added your WEBSERVER_CUSTOM_BUILD_DEFINED stuff and removed some pages:

Code: Select all

        //#define WEBSERVER_TIMINGSTATS
        //#define WEBSERVER_SYSVARS
        //#define WEBSERVER_NEW_UI
        //#define WEBSERVER_I2C_SCANNER
        //#define WEBSERVER_FAVICON
        //#define WEBSERVER_LOG
        //#define WEBSERVER_GITHUB_COPY
        //#define WEBSERVER_DOWNLOAD
        //#define WEBSERVER_FILELIST
        //#define WEBSERVER_PINSTATES
        //#define WEBSERVER_RULES
        //#define WEBSERVER_SYSINFO
        //#define WEBSERVER_UPLOAD
        //#define WEBSERVER_WIFI_SCANNER

If I compile again I end up with 619316 bytes. That´s 44,5kb less ...

This means if you use Spiffs with 128kb you have 892912 bytes of flash for the sketch.
892912 - 619316 = 273596 bytes left for OTA ...

Now I compiled the EspBuddy from https://github.com/soif/EspBuddy
I compiled the sketch ESPEasyUploaderMega.OTA by myself and got a bin file with 367216 bytes.
That´s to big for OTA but there is this compression stuff
I used 7zip to get the ESPEasyUploaderMega.OTA bin file compressed to gzip with highest compression.
The result is 262551 bytes.

Let´s bring it together ...
892912 - 619316 (EspEasy) - 262551 (compressed OTA) = 11kb left
I did not test this if it really works, but at least I hope it

But there are a couple of things which come to my mind when I done this ...
* There are still some parts which should be configurable.
One example is the SC16IS752 chip ... It´s a huge lib and to be honest I don´t know any device which uses such a I2C serial bridge.
* From the Sysinfo page I got this: SPIFFS Size: 110 kB (32 kB free). That means there are 32kb unused flash ... That´s a lot if you remember that removing the WEBSERVER_xxxx stuff only gets 44,5kb back ...
So can you think of a reduced Spiffs layout? Maybe reducing the devices and controller to get the settings stored within a 64kb partition? This would free up a lot space in spiffs / flash.
* Last but not least I think it should be more easy for people to get thier own compiled version of EspEasy.
At the moment you have to setup a lot of things ... Enable custom.h .. Include the parts for the Webserver defines ...
Wouldn´t it be nice if there is one place to configure the whole stuff? And beside that it would also be a good idea to add some documentation to the custom.h.
At the moment you have this ...

Code: Select all

#define WEBSERVER_CUSTOM_BUILD_DEFINED
        #define WEBSERVER_TIMINGSTATS

That´s fine if you are know what you are doing. But something like this would be more helpful to a normal user:

Code: Select all

#define WEBSERVER_CUSTOM_BUILD_DEFINED		// Enable custom selection of compiled Webpages
#define WEBSERVER_TIMINGSTATS			// Enable Webpage "TimingStats"					~adds 5kb Flash

Something like that ... Maybe I can help at this topic ?


Some notes after testing ...
Ok I figured out that OTA is default disabled ... And enabling it will add 20kb of extra size to the bin file.
And if I enable the OTA in EspEasy I get the following error:
OTA : Error (will reboot): Begin Failed
I used this command to start OTA:

Code: Select all

espota.exe -i 192.168.30.204 -p 18266 --auth= -f ESPEasyUploaderMega.OTA.ino.bin.gz

But anyway ... Now the bin file is to big for using OTA. That´s not good news at all

[TD-er]

I will look in the defines wrapping the I2C serial bridge as that's a valid point. (if it is indeed included while not needed)

About the compressed feature.
Remember you cannot simply compress it and think that's the amount of space you gain.
You have to keep in mind, at least one of the sketches should fit in uncompressed form.

- 2-step-OTA compressed + your sketch uncompressed
- 2-step-OTA uncompressed + your sketch compressed

Both combinations must fit.


About the SPIFFS free size.
SPIFFS needs 2 blocks (of 32 pages each 256 bytes) free to perform its operations and when writing you will create some fragmentation.
So 32 k free is somewhat of a minimum to be able to change stuff in the file system.
There is however a lot of slack space (all zeroes) at the end of the config.dat file, so there is room to gain 30k of space.

[moelski]

Hi !

I will look in the defines wrapping the I2C serial bridge as that's a valid point.

I saw on github that there is a define for the I2c serial bridge.

Code: Select all

#define DISABLE_SC16IS752_Serial

in custom.h did not remove the lib in my first test this morning.

Both combinations must fit.

I know but had no time to do further tests.
But it should work if I compress the BIN with 7zip to a gz file while using the compression rate ultra?
I just want to be sure to not produce any compressed BIN files which are not compatible with OTA.

There is however a lot of slack space (all zeroes) at the end of the config.dat file, so there is room to gain 30k of space.

And this could be used ? 30k is a lot ...
btw ... Placing the CSS stuff in Spiffs will reduce the flash size of about 5kb.
Maybe some other stuff from WebStaticData.h could be stored in Spiffs if there is no way to use the 30kb as additional flash space.

Anyway ... It would be awesome to get EspEasy to a BIN size of ~600kb.
Look at tasmota ... They have tons of plugins in the main bin file and itś size is ~597kb with the latest release.

[TD-er]

Sadly the #define DISABLE_SC16IS752_Serial
cannot be in the Custom.h as it must be present at the start of the compile.
So either you must define it in your PlatformIO.ini (as I did for minimal_OTA labelled builds) or in the pre_custom_xxx.py file.
For Arduino IDE you must make sure it is defined in your platform.local.txt as described here: https://espeasy.readthedocs.io/en/lates ... efinitions

I will have a look at the drawbacks of reducing the config.dat file size.
I don't expect it to be an issue, at least when using 12 task.

[moelski]

Sadly the #define DISABLE_SC16IS752_Serial
cannot be in the Custom.h as it must be present at the start of the compile.

Bad news
This way EspEasy get´s more and more scattered.

No chance to centralize all build Settings in one file?

[TD-er]

Well I also attempted to do it in the pre_custom_xxx.py files (see tools/pio/) but it is quite hard to describe string like parameters in a Python environment which will end up as defines in C++ code.

I managed to do so with build environment parameters like the build filename and the build environment liks Git branch.
But certain things like this is not possible (as far as I know)

Code: Select all

#define BUILD_GIT           "MyBuild: "  __DATE__ " "  __TIME__

Also parameters that don't need to be a string must be dealt with differently like these:

Code: Select all

#define DEFAULT_AP_IP       192, 168, 4, 1                           // Enter IP address (comma separated) for AP (config) mode
#define DEFAULT_USE_STATIC_IP                false 

That makes it either complex in the Python file, or in the processing of that file.

So right now you only need to keep 2 files for yourself to define your build:
- tools/pio/pre_custom_xxxx.py
- Custom.h

I think that's quite a decent trade off.