Page 1 of 1

Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 03 Jun 2017, 14:59
by Martinus
If you have uploaded the newer stock Mega ESP Easy firmware to 1MB modules like the ESP-01 or the Sonoff, you will no longer be able to use OTA because the firmware is a lot larger (533k) and there's not enough free space to hold a temp copy of a newer upload. You can use a workaround by uploading a smaller image and then upload a normal size image after that.

We have provided a very small sketch to use this workaround:
https://www.letscontrolit.com/wiki/inde ... g_firmware

Martinus

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 16 Apr 2018, 16:41
by grumpy
Where can I find this small sketch?

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 16 Apr 2018, 17:28
by grovkillen

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 02 Aug 2018, 02:03
by RichEO
I was having major slowdown when I had an MQTT controller enabled. So I decided to update to a newer version.

I uploaded the small sketch, the upload said it completed successfully but the device dropped off my network.

I have power cycled the device (a sonoff SV) and it is not showing up on my router.

Has anyone experienced this? It is inside my gate motor so in a few days I will open it up and reflash it, but maybe before then anyone has any ideas what is wrong?

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 03 Aug 2018, 03:59
by kenkoknz
Hi @grovkillen,

I flashed a sonoff(1m) with normal 1044 bin (716k) and a esp01 (puya) with dev puya 1024bin(785k) just using the normal flashESP8266, and it all seems to work normally. I can access both OTA to cnfig etc. Can you clarify if the special sketch is needed as both the files are over 533k and seems to be working fine with normal flasher?
Thanks,
Ken

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 03 Aug 2018, 06:31
by grovkillen
Current releases are too big for OTA (2 step procedure for 1Mb units). We need to rethink how to support those units. They need to be physically attached using cable and serial programming to be updated. Sorry for any inconveniences but hopefully we can think of something.

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 03 Aug 2018, 08:32
by kenkoknz
Hi,
This is really just a minor inconvenience, there are possible lots more urgent things on your plate to get a stable release 2. Connecting a cable for firmware upgrade is not too much of a hassle, I think we are just getting a bit lazy...

Keep up the good work,

Cheers,
Ken

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 20 Aug 2018, 02:04
by sheppy
I hope a working OTA solution can be found, I have 38 ESP's, many are in difficult to access locations, and I have just discovered a possible DHT Humidity bug in the January Sketch I used

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 26 Aug 2018, 20:47
by ewaldharmsen
I also look forward to be able to update OTA my devices!

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 31 Aug 2018, 16:27
by ewaldharmsen
In the mean time: home can I make a custom build with only the plug-ins included I need?

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 31 Aug 2018, 18:44
by grovkillen
I think we're beyond the point of OTA even with no plugins in the compilation. I might be wrong though.

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 31 Aug 2018, 23:51
by TD-er
Please have a look at the file define_plugin_sets.h and PlatformIO.ini.
Those two make it clear how the different builds are made together.
It should be possible to perform OTA on 1 M modules, when using 2-step flash. But some changes have to be made.

- filesize of the config,dat file has to be reduced to something like 36k (rest is not used)
- Flash layout may have to be changed to 1M.64k (current is 1M.128k)

At least the last step will render existing installations invalid.

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 01 Sep 2018, 12:48
by Methuselah
grovkillen wrote: 03 Aug 2018, 06:31 Current releases are too big for OTA (2 step procedure for 1Mb units). We need to rethink how to support those units. They need to be physically attached using cable and serial programming to be updated. Sorry for any inconveniences but hopefully we can think of something.
What would be helpful is a configure script, a-la linux's "configure" which automatically enables base functionality (based on the type of device the user specifies) and then prompts the user to add in optional modules for those features s/he is going to use. For example I do not use the MQTT functionality whatsoever and could easily eliminate that from a build.

Unfortunately, the size of this project is only going to grow, as more and more functionality is added in... eventually it will get to the point where the builds will be too big for the 1MB modules, and then a whole bunch of people will be orphaned.

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 01 Sep 2018, 14:55
by grovkillen
Methuselah wrote: 01 Sep 2018, 12:48
grovkillen wrote: 03 Aug 2018, 06:31 Current releases are too big for OTA (2 step procedure for 1Mb units). We need to rethink how to support those units. They need to be physically attached using cable and serial programming to be updated. Sorry for any inconveniences but hopefully we can think of something.
What would be helpful is a configure script, a-la linux's "configure" which automatically enables base functionality (based on the type of device the user specifies) and then prompts the user to add in optional modules for those features s/he is going to use. For example I do not use the MQTT functionality whatsoever and could easily eliminate that from a build.

Unfortunately, the size of this project is only going to grow, as more and more functionality is added in... eventually it will get to the point where the builds will be too big for the 1MB modules, and then a whole bunch of people will be orphaned.
We have much in plan for the upcoming autumn and next year will be great. 1MB users are not going to be left in the dust.

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 15 Sep 2018, 08:00
by Djoe
I have try to update a sonoff basic (1m) like this :

- I have build my own firmware with a size of 580kb approximately
- I have uploaded it in the sonoff
- Now I can upload ESPEasyUploaderMega_1024 through OTA, the first step
- The sonoff reboot and I select the same custom firmware (580k)

But after the reboot the new firmware not start... The wiki say that we can upload max 604k, so I think my firmware is okay ?

Is someone have already use ESPEasyUploaderMega_1024 successfully ?

Thank you and have a good day!

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 02 Oct 2018, 11:12
by neoseb
Hi all

Very interested by this topic and by knowing how to build my custom lighter firmware for my Sonoff so that I will be able to update them as they will not be easily accessible in the coming weeks (they'll be used for my covers!)

Is there a tutorial somewhere in the web? Didn't fond anything yet.

Thank you for your help.

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 02 Oct 2018, 11:55
by Djoe
neoseb wrote: 02 Oct 2018, 11:12 Is there a tutorial somewhere in the web? Didn't fond anything yet..
You can have a look to :

https://www.letscontrolit.com/wiki/inde ... platformio

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 17 Jan 2020, 09:05
by riker1
grovkillen wrote: 03 Aug 2018, 06:31 Current releases are too big for OTA (2 step procedure for 1Mb units). We need to rethink how to support those units. They need to be physically attached using cable and serial programming to be updated. Sorry for any inconveniences but hopefully we can think of something.
Hi is this still valid?

So i do not understand why there is a firmware with
ESPEasy_2step_UploaderMega_1024.bin
which can not be used?

Did I miss anything?

when I call firmwareupdate via gui.

Code: Select all

URI: /update
Method: GET
Arguments: 0
So finally on 1 MB devices it is not working? right?

would be very usefull

thanks a lot T

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 17 Jan 2020, 10:13
by grovkillen
Well I think it is working work OTA now but since I'm doing 100% development of the new GUI as of the last 4-5 month I haven't been testing any new firmware stuff. Perhaps @TD-er can shed some light?

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 17 Jan 2020, 15:48
by TD-er
As some of you may know, I've also been way too busy the last month in moving to a new house (last build was > 1 month ago :( )
In the mean time there has been a new addition to the core lib which allows for flashing gzip'ed images via OTA, which I really want to add.
It would be a really great addition to extend support for 1M modules.

It isn't there yet, but I will try to add it in a few weeks time.
Then I can also determine the true gain in size of the compressed binaries as it is really hard to predict the amount of compression we can expect.

In short the current situation:

We have in the nightly builds a set of so called "minimal_OTA" versions.
These are meant to be flashed to 1M modules in order to be able to perform OTA updates.

The 1M flash is split in 2 parts:
- roughly 128k SPIFFS filesystem
- 1M - 128k = 873k for active sketch + OTA image

The smallest 2-step OTA image we have is roughly 273k, which leaves 600k for the "active sketch". (N.B. all numbers out of my head)
So if the sketch is =< 600k, you still have room for a small "2-step OTA image" which can be used for only uploading the new image.

My hope is that the compression can shrink this 2-step image, so we have more room for the active sketch.

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 12 Jul 2020, 18:00
by lumajo
I have ugraded ESP01 to 2M Flash. (they where sent wrongly to my, but I still gave it a try) However, it also says I am not able to make a OTA upgrade. Why? There must be enough room. The flash size is correctly recognized.
Any Idea? However, it works with the 1M workaround.

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 12 Jul 2020, 18:06
by TD-er
The flash chip is partitioned in a partition for the sketch + OTA and the SPIFFS file system
A build for 1M flash is only using the first 1M of the flash chip, regardless its size.
But if you have 2M flash, it makes sense to use a 2M build (there are some 2M builds in the nightly builds, but not a lot)
With a 2M build, the SPIFFS is roughly 500k, which leaves 1500 k for the sketch + OTA, so you could use roughly 750 kB builds, or a few 100 k larger builds if you flash .bin.gz files. (only supported since core 2.7.0)

There are also ESP8285 chips with 2M, although I have never seen them myself.

N.B. if you change builds between 1M build and 2M build, the file system will be erased so you will loose all stored settings.

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 12 Jul 2020, 21:59
by lumajo
Thanks a Lot, i will give it a try.

So I used ESP_Easy_mega_20200703_normal_WROOM02_2M256 to flash the ESP01, which works quite well.
It now says OTA yes, and it works...
However I do not know the differences to the normal one (w/Vcc). The time server ntp does not work here. So the time always stays on 1970-1-1

So maybe there is a bug here?

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 13 Jul 2020, 15:24
by TD-er
If you don't enter a host for the NTP server, then it will use a random one from pool.ntp.org.
Only requirement then is that it has access to a gateway and DNS server. So please check those are correct.
In the sysinfo page both are reported.

VCC build is about being able to measure its own voltage on the analog input pin.
If you need to read an analog value from some sensor, you don't need the VCC feature.

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 12 Jan 2021, 08:10
by slom
Hi,
I also have this error when I try update firmware through www:

URI: /update
Method: GET
Arguments: 0

Is any possibility to do this in this way?
Currently, I have mega-20191208 and I try update my sonoff 4CH

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 12 Jan 2021, 12:51
by TD-er
What is the size of the currently flashed 'sketch' ?
You can see it on the sysinfo page at the bottom.
If it is > 600 kB, you cannot update via OTA, not even when using the 2-step OTA.

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 13 Jan 2021, 21:04
by chromo23
TD-er wrote: 12 Jan 2021, 12:51 What is the size of the currently flashed 'sketch' ?
You can see it on the sysinfo page at the bottom.
If it is > 600 kB, you cannot update via OTA, not even when using the 2-step OTA.
But the line with "OTA 2-step Needed: true" is misleading then. Right?!
For me, this implies that a two-step OTA is possible.
Bildschirmfoto 2021-01-13 um 20.59.13.png
Bildschirmfoto 2021-01-13 um 20.59.13.png (85.88 KiB) Viewed 17241 times

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 14 Jan 2021, 11:49
by TD-er
The line above states it is not possible right now.
The check for "2-step OTA needed" is simply this:

Code: Select all

const bool     otaPossible                = freeSketchSpace >= smallestOtaImageSizeNeeded;
use2step = freeSketchSpace < currentSketchSize; // Assume the new image has the same size.
On 4M units, the free sketch space is more than the max. sketch size, so you never need to use 2-step OTA.
On 2M units, the space for the active sketch + OTA sketch is roughly 1.7 MB, so it is possible the running sketch may take more than half this 1.7 MB and then you may need to perform 2-step OTA, but it will always be possible as the smallest 2-step OTA image + max sketch size is still less than this 1.7 MB
On 1M units, you always need 2-step, but is not always possible as the max. 2-step OTA builds must be smaller than ~600 kB.
You can still flash larger images on 1M units, but then you will not be able to perform an OTA update.

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 14 Jan 2021, 13:46
by chromo23
Thank you for clarification!

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 07 Aug 2022, 14:19
by chromo23
TD-er wrote: 17 Jan 2020, 15:48 In the mean time there has been a new addition to the core lib which allows for flashing gzip'ed images via OTA, which I really want to add.
It would be a really great addition to extend support for 1M modules.
Are you still planning to implement this?
I recently flashed some 1m Tasmota devices with the minimal Tasmota image and then flashed my gzipped espeasy binaries.
It would be great to be able to do this with espeasy too :)

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 07 Aug 2022, 14:52
by TD-er
What is the size of the Tasmota 2-step bin?

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 08 Aug 2022, 09:24
by chromo23
The sensor.bin was installed, then i flashed the minimal.bin
Bildschirmfoto 2022-08-08 um 09.17.02.png
Bildschirmfoto 2022-08-08 um 09.17.02.png (69.04 KiB) Viewed 8604 times
my espeasy.bin is 680kb and compressed 460kb....

I think tasmota doesn’t use spiffs on 1m devices (if i understood correctly) and therefore has more space to upload the minimal.bin in the first place...

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 08 Aug 2022, 11:36
by TD-er
Yep, that's possible.
I know Tasmota wasn't using any filesystem for a long time, so I can imagine why they would continue doing so on 1M devices.
Maybe we can do the same for the settingsfile on 1M modules, using some storage layer like the EEPROM class does.
Still you need to have 1 of both uncompressed on the flash.
So I'm not sure how much we gain by updating the 2-step bin to support compressed bins.
Stripping away the filesystem part may help though as it gives quite a lot of overhead, both in .bin size and needed slack space.

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 08 Aug 2022, 22:07
by chromo23
I am not familiar with the flash layout but would it be possible to have a feature where you can repartition the filessystem or even delete the spiffs partition entirely to make space for OTA.
Something like a checkbox and after a reboot spiffs is gone and you are able to update....

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 08 Aug 2022, 23:00
by TD-er
Sure you can recreate the filesystem, which will be done right after boot and ESPEasy cannot find a proper file system.
But how do you expect the 2-step OTA bin file to connect to WiFi then?

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 08 Aug 2022, 23:26
by Ath
chromo23 wrote: 08 Aug 2022, 22:07 ...to have a feature where you can repartition the filessystem or even delete the spiffs partition entirely to make space for OTA.
Something like a checkbox and after a reboot spiffs is gone and you are able to update....
See "Allow OTA without size-check" on the Tools/Advanced page... :o

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 09 Aug 2022, 10:28
by chromo23
TD-er wrote: 08 Aug 2022, 23:00 But how do you expect the 2-step OTA bin file to connect to WiFi then?
I have no answer to that because that is beyond my horizon. You are the expert. :D
Ath wrote: 08 Aug 2022, 23:26 See "Allow OTA without size-check" on the Tools/Advanced page...
I tried but it didn’t work so i was not sure what it actually does...

this is the device:
size.png
size.png (8.48 KiB) Viewed 8518 times
The minimal 2step file is aroud 280kb
and this is what i get even when "Allow OTA without size-check" is activated:
upderror.png
upderror.png (11.07 KiB) Viewed 8518 times
So it doesn’t seem to work as expected or my expectations are wrong...

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 09 Aug 2022, 10:40
by Ath
And I assume you have rebooted after enabling that setting, as the note suggests?

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 09 Aug 2022, 14:27
by chromo23
Ath wrote: 09 Aug 2022, 10:40 And I assume you have rebooted after enabling that setting, as the note suggests?
yes, serveral times...

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 09 Aug 2022, 14:36
by Ath
Ah, then you have probably tried to OTA-upgrade a minimal build with another minimal build, but as 1 uncompressed (the installed) image and 1 compressed bin file added are still > 1MB so that still can't fit. That flash-size check is still applied, only the FileSystem part is ignored.

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 09 Aug 2022, 21:58
by chromo23
No, i did not use a minimal build. I used a custom build (650k) and then tried the 2step build (ESPEasy_2step_UploaderMega_1024.bin).
I didn't know that the latter one is compressed...
So i guess serial flash is the way to go :)

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 09 Aug 2022, 22:07
by Ath
AFAIK, that ESPEasy 2 step uploader doesn't support compressed bin files (yet), as it built with a too old Arduino framework version. So it isn't compressed itself either...

Maybe you could try to use the Tasmota 2 step uploader to upgrade ESPEasy? (Not sure how well that could go)

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 09 Aug 2022, 22:08
by chromo23
TD-er wrote: 17 Jan 2020, 15:48 The 1M flash is split in 2 parts:
- roughly 128k SPIFFS filesystem
- 1M - 128k = 873k for active sketch + OTA image

The smallest 2-step OTA image we have is roughly 273k, which leaves 600k for the "active sketch". (N.B. all numbers out of my head)
So if the sketch is =< 600k, you still have room for a small "2-step OTA image" which can be used for only uploading the new image.
I still don´t get what "Allow OTA without size-check" is for then?
It states:
When enabled, OTA updating can overwrite the filesystem and settings!
Shouldn´t there be enough space for the 2step bin after enabling this? Or what does it actually do

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 09 Aug 2022, 22:17
by Ath
The 2 step OTA needs access to the settings to read the WiFi AP and password, to be able to connect to WiFi, so you can access it to upload the new build. So the currently Installed running build + (in the future) compressed 2 step uploader should fit in that 875 kB.

Overwriting the FS is acceptable if you can directly install a new build, that has the UI Setup elements for configuring WiFi etc. and then upload a previously saved config.dat. It was added on request of a user that could handle that use-case.

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 09 Aug 2022, 22:23
by chromo23
Ahh... i understand. Thanks a lot for the clarification!

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 09 Aug 2022, 22:25
by chromo23
Ath wrote: 09 Aug 2022, 22:17 The 2 step OTA needs access to the settings to read the WiFi AP and password, to be able to connect to WiFi, so you can access it to upload the new build.
Does it though? I mean a simple ap would be also ok...

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 09 Aug 2022, 22:31
by TD-er
chromo23 wrote: 09 Aug 2022, 22:25 [...]

Does it though? I mean a simple ap would be also ok...
But it is not included in that 2-step OTA bin, so if you flash it, it will not allow you to start an AP.

Re: Mega (V2.0.0) OTA firmware updates on 1MB modules

Posted: 09 Aug 2022, 22:55
by chromo23
Ok. I think i got the whole picture… thanks to you as well!