ESP32Custom Platformio OTA update error

[PandCall]

Environement
ESP32
PlatformIO
Project Task custom_ESP32_4M316K

-----------------------------------------------------------------------
The first ESP flash on the COM18 pot is no problem,Everything works fine
Now I'd like to update with the OTA function to add sensors, for example.

I've changed this part in my platformio.ini file.

[espota]
upload_protocol = espota
; each flag in a new line
; Do not use port 8266 for OTA, since that's used for ESPeasy p2p
upload_flags_esp8266 =
--port=18266
upload_flags_esp32 =
--port=3232
build_flags = -DFEATURE_ARDUINO_OTA=1
upload_port = 192.168.69.75

and Custom.h file
#define FEATURE_ARDUINO_OTA 1 //enables the Arduino OTA capabilities

After my first connection

Info Board

OTA-Board.png (111.18 KiB) Viewed 12622 times

During the upload procedure I get this message

OTA update message
Warning! We have just detected `upload_port` as IP address or host name of ESP device. `upload_protocol` is switched to `espota`.
Please specify `upload_protocol = espota` in `platformio.ini` project configuration file.

I think it's just information.

But then I get a message telling me that the ESP on the IP hasn't responded and the upload isn't done.

Uploading .pio\build\custom_ESP32_4M316k\ESP_Easy_mega_20230715_custom_ESP32_4M316k.bin
13:22:11 [DEBUG]: Options: {'esp_ip': '192.168.69.75', 'host_ip': '0.0.0.0', 'esp_port': 3232, 'host_port': 36700, 'auth': '', 'image': '.pio\\build\\custom_ESP32_4M316k\\ESP_Easy_mega_20230715_custom_ESP32_4M316k.bin', 'spiffs': False, 'debug': True, 'progress': True, 'timeout': 10}
13:22:11 [INFO]: Starting on 0.0.0.0:36700
13:22:11 [INFO]: Upload size: 1098640
Sending invitation to 192.168.69.75 ..........
13:22:11 [ERROR]: No response from the ESP
*** [upload] Error 1
The terminal process "C:\.platformio\penv\Scripts\platformio.exe 'run', '--target', 'upload', '--environment', 'custom_ESP32_4M316k', '--upload-port', '192.168.69.75'" terminated with exit code: 1.
* Terminal will be reused by tasks, press any key to close it.

In platformio I've set the ports to Auto or forced the IP, but that doesn't change anything.

If I reconnect the ESP to the COM18 port, I have no problem updating the ESP and my new sensors.

Two questions
Is OTA updating possible from platformio?
If yes
Did I forget anything in my configuration to be able to remotely update ESPs?

Last test:
If I use the web interface to update the BIN file
.pio\build\custom_ESP32_4M316k\ESP_Easy_mega_20230715_custom_ESP32_4M316k.bin
It works fine, the ESP is updated with my new sensors.

Update Success! Rebooting...

Best regards

[TD-er]

I have never used OTA update from PlatformIO myself.
So it is very well possible I may have forgotten something in the PIO config.

[PandCall]

Don't hesitate if you'd like me to try them out for you.

There's nothing urgent at the moment, uploading the binary works with the Web interface, it's just one more step, nothing serious.

[Ath]

[-]For ESP8266 you seem to have set the port number to 18266, but the max possible portnr is 65535, so your setting is not valid[-]

Edit: I was probably wearing my glasses upside down when I wrote that, missing a number on my side, sorry

[PandCall]

Hello Ath,
I don't understand the meaning of what you're telling me.
It probably makes sense, but...

You're telling me that the port is greater than what is allowed 65534
That my ESP8266 port is 18266
For me, 18266 is well below 65534

And I haven't changed these ports, that's what's in the Custom.h example file, but as you say, you haven't specially tested this function, so you probably have a good reason for telling me that.

And since I'm using an ESP32, port 3232 should be used instead?

Dans le doute, j'ai testé le port
upload_flags_esp8266 =
--port=3231
upload_flags_esp32 =
--port=3232
build_flags = -DFEATURE_ARDUINO_OTA=1
upload_port = 192.168.69.75

Mais cela ne change rien.

Uploading .pio\build\custom_ESP32_4M316k\ESP_Easy_mega_20230716_custom_ESP32_4M316k.bin
09:35:23 [DEBUG]: Options: {'esp_ip': '192.168.69.75', 'host_ip': '0.0.0.0', 'esp_port': 3232, 'host_port': 19897, 'auth': '', 'image': '.pio\\build\\custom_ESP32_4M316k\\ESP_Easy_mega_20230716_custom_ESP32_4M316k.bin', 'spiffs': False, 'debug': True, 'progress': True, 'timeout': 10}
09:35:23 [INFO]: Starting on 0.0.0.0:19897
09:35:23 [INFO]: Upload size: 1110560
Sending invitation to 192.168.69.75 ..........
09:35:23 [ERROR]: No response from the ESP
*** [upload] Error 1

I'm going to look at the ESPTOOL parameters because this line seems strange to me
Starting on 0.0.0.0:19897

Regards

[TD-er]

I think Ton was tired due to the weather as 18266 is indeed < 65535


It looks like you need to specify the

Code: Select all

upload_flags = ${espota.upload_flags_esp32}

For ESP32 builds
And for ESP8266 builds:

Code: Select all

upload_flags = ${espota.upload_flags_esp8266}

This in your platformio.ini files for your custom builds.

[PandCall]

Hello,
Merci poir ces infos,
Not sure exactly where to place this line.
But I tested this

[platformio]
default_envs = custom_ESP32_4M316k

[espota]
upload_protocol = espota
; each flag in a new line
; Do not use port 8266 for OTA, since that's used for ESPeasy p2p
upload_flags_esp8266 =
--port=18266
upload_flags_esp32 =
upload_flags = ${espota.upload_flags_esp32}
--port=3232
build_flags = -DFEATURE_ARDUINO_OTA=1
upload_port = 192.168.69.75

I'm having trouble understanding the line

upload_flags_esp32 =
and
upload_flags = ${espota.upload_flags_esp32}

Which seem very similar.


But always the upload error.

Configuring upload protocol...
AVAILABLE: cmsis-dap, esp-bridge, esp-prog, espota, esptool, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa
CURRENT: upload_protocol = esptool
Uploading .pio\build\custom_ESP32_4M316k\ESP_Easy_mega_20230717_custom_ESP32_4M316k.bin
09:22:18 [DEBUG]: Options: {'esp_ip': '192.168.69.75', 'host_ip': '0.0.0.0', 'esp_port': 3232, 'host_port': 19904, 'auth': '', 'image': '.pio\\build\\custom_ESP32_4M316k\\ESP_Easy_mega_20230717_custom_ESP32_4M316k.bin', 'spiffs': False, 'debug': True, 'progress': True, 'timeout': 10}
09:22:18 [INFO]: Starting on 0.0.0.0:19904
09:22:18 [INFO]: Upload size: 1102480
Sending invitation to 192.168.69.75
09:22:18 [INFO]: Waiting for device...
09:22:28 [ERROR]: No response from device
*** [upload] Error 1

Regards

[TD-er]

You can't refer to items in the same block where it is being referred to.

So this must be placed in the block for custom_ESP32_4M316k (assuming that's the one you're building)

Code: Select all

[platformio]
default_envs = custom_ESP32_4M316k

[espota]
upload_protocol = espota
; each flag in a new line
; Do not use port 8266 for OTA, since that's used for ESPeasy p2p
upload_flags_esp8266 =
--port=18266
upload_flags_esp32 =
--port=3232
build_flags = -DFEATURE_ARDUINO_OTA=1
upload_port = 192.168.69.75

And somewhere in the block you're building:

Code: Select all

[env:custom_ESP32_4M316k]
....
upload_flags = ${espota.upload_flags_esp32}

N.B. the ... are just to indicate that there's other stuff in that block you should leave as it is.

[thalesmaoa]

Hi there, I try to follow it. However, I can't make it work. Any help on how to debug it?

[TD-er]

Hi there, I try to follow it. However, I can't make it work. Any help on how to debug it?

What are you trying to achieve?

[thalesmaoa]

Sorry @TD-er!
I'm trying to develop a new controller using CAN for ESP32.
Thus, I'm using PlatformIO, in order to speed up test, I want to upload my code using OTA to ESPEasy directly from pio.

I have enabled it in Custom.h file. Also, I followed your config from the last post.
However, it states that can't accept invitation and never gets uploaded.

I'm out of clue how to debug the OTA from pio.

[Ath]

AFAICS, there is no OTA Upload option available in the PlatformIO menu, unless you can show me where to find it

If you have your ESP connected to your PC via USB, then the Upload menu-item of any of the build environments, will build and upload that binary via USB to the ESP. No need to configure anything extra for that (in fact I think you should revert any change you made there). See the second image from this link, where Upload is just below the Build option.

OTA upload is performed via the ESPEasy web UI from the Tools/Update Firmware button.

[thalesmaoa]

Hi Ath, thank you for following. I can be wrong, but it seems that there is.

If you follow the post and TD-er suggestion, it does try to upload using OTA feature.
There is also PIO doc regarding it.
https://docs.platformio.org/en/latest/p ... ota-update

I'm currently using the web interface. I just wanna speed up my debug process. Also, the ESP is in the field and it is complicated to connect the serial converter to upload it every time.

Code: Select all

[env:normal_ESP32_4M316k_CAN]
extends                   = env:normal_ESP32_4M316k
build_flags               = ${env:normal_ESP32_4M316k.build_flags}
                            -DFEATURE_CAN=1

upload_flags = ${espota.upload_flags_esp32}
upload_port = 192.168.200.97

During upload process I get:

Code: Select all

Timestamp: 2024-08-14 15:03:31.710009
Configuring upload protocol...
AVAILABLE: cmsis-dap, esp-bridge, esp-prog, espota, esptool, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa
CURRENT: upload_protocol = esptool
Uploading .pio/build/normal_ESP32_4M316k_CAN/ESP_Easy_mega_20240814_normal_ESP32_4M316k_CAN.bin
15:03:31 [DEBUG]: Options: {'esp_ip': '192.168.200.97', 'host_ip': '0.0.0.0', 'esp_port': 3232, 'host_port': 10390, 'auth': '', 'image': '.pio/build/normal_ESP32_4M316k_CAN/ESP_Easy_mega_20240814_normal_ESP32_4M316k_CAN.bin', 'spiffs': False, 'debug': True, 'progress': True, 'timeout': 10}
15:03:31 [INFO]: Starting on 0.0.0.0:10390
15:03:31 [INFO]: Upload size: 1486192
Sending invitation to 192.168.200.97 ..........
15:05:12 [ERROR]: No response from the ESP
*** [upload] Error 1

It tries, but ESPEasy doesn't reply.

[Ath]

Hmm, is the "Enable Arduino OTA" setting on Tools/Advanced checked? As besides the feature being included in the build, it also has to be enabled... If you have to enable it, it's best to reboot the unit after saving the setting, as AFAICS, it is (only) started at boot.

[thalesmaoa]

I'm doing something wrong now.

In Advanced setting I only have
Allow OTA without size-check



Enabling this should be enough to show this option?

[TD-er]

Do you have LIMIT_BUILD_SIZE also defined?
If so, then FEATURE_ARDUINO_OTA will also be set to 0

[thalesmaoa]

Hi TD-er. No, I'm not.
From your question I realized that I was forgetting the CUSTOM_H flag. Now everything works. Thanks a lot.
Just will figure now how to see the log in pio.lol

Code: Select all

[env:normal_ESP32_4M316k_CAN]
extends                   = env:normal_ESP32_4M316k
build_flags               = ${env:normal_ESP32_4M316k.build_flags}
                            -DFEATURE_CAN=1
                            -DUSE_CUSTOM_H

upload_flags = ${espota.upload_flags_esp32}
upload_port = 192.168.200.97

[TD-er]

Good to know it is now working.

And I'm also curious about this CAN feature you're adding.

[thalesmaoa]

It is working. However, we are now changing the implementation following MQTT Import to CAN Import.
I will share it soon.

[Ath]

Hi TD-er. No, I'm not.
From your question I realized that I was forgetting the CUSTOM_H flag. Now everything works. Thanks a lot.
Just will figure now how to see the log in pio.lol

Code: Select all

[env:normal_ESP32_4M316k_CAN]
extends                   = env:normal_ESP32_4M316k
build_flags               = ${env:normal_ESP32_4M316k.build_flags}
                            -DFEATURE_CAN=1
                            -DUSE_CUSTOM_H

upload_flags = ${espota.upload_flags_esp32}
upload_port = 192.168.200.97

Why do you modify a normal env if there are also Custom envs that _have_ that USE_CUSTOM_H define by default? When using VSCode with PlatformIO there is no need to modify platformio.ini for creating a Custom build
That's all explained in the Starter Guide for local development

[thalesmaoa]

Hi Ath. Yes! There is a custom definition already. Also, I haven't modify the already defined env, I've built a new one.

I did edit the platformio_esp32_envs.ini

Code: Select all

[env:normal_ESP32_4M316k_CAN]
extends                   = env:normal_ESP32_4M316k

In fact, I've done this for two main reasons.

The first one is better understood if I present my canbus structure.
I have a consumer (master/gateway) node (1). It uses Ethernet (not wifi) as connection. It receives data from producers (slave) and publishes using MQTT. During debug, I have to upload my code to (minimum) two devices. It's lot easier to just build and upload with one click.

Second reason is that, everytime pio rescan the envs devices, it presents a giant list. It was really annoying keep looking for the custom one in that entire list. I just commented them and left the two devices that I need.

I'm sure that is not recommend, but like that, I can speed things up

PS: pio takes a lot of time to compile. I couldn't find the reason yet.

[TD-er]

Just as a tip regarding PlatformIO...
Next to the "Project Tasks" label are 4 icons.
The first one is "Pick Project Environment".
If you click that one, there will appear a text search field in the middle of the VS code window in which you can search/select your specific project.
Select this and you will be much less annoyed by the quirky PlatformIO behavior of selecting the right one from a list of ~160 envs which will also collapse and redraw as soon as you selected the right one.

And about the compile time....
When on Windows, you have 2 options to reduce compile time:
Option 1:
- make sure the virusscanner is not looking at the build folders and where you have your git repo.
- Disable any other tool which scans for file changes or excluse these folders from those searches (e.g. Dropbox, OneDrive, Opened Git browser tool)

Option 2:
Install Ubuntu using WSL2 on Windows.
Make sure you have your git repo located on a folder in your / mount (not windows drive mount) and make sure the folder where WSL is installed is excluded from any virusscanner.
Then in VS code you can install the WSL extension as described here: https://code.visualstudio.com/docs/remote/wsl-tutorial
If you need direct access to an USB device in WSL2, you can follow these steps: https://stackoverflow.com/questions/694 ... erial-port

The most gain in build speed can be expected when using option2.
Just did a quick test using VScode and building via WSL2:

Code: Select all

Environment                         Status    Duration
----------------------------------  --------  ------------
max_ESP32c6_16M8M_LittleFS_CDC_ETH  SUCCESS   00:01:57.796

This took about 10 minutes on Windows.
It is the exact same computer as WSL with Ubuntu is running within Windows.

[TD-er]

By the way, I strongly recommend you use LittleFS along with the latest ESP-IDF5.x
A lot (!!!) has changed in the IDF core since 4.4 (used in SPIFFS builds) and wouldn't be surprised if also stuff in CAN bus implementation has changed.
Better implement it against the current SDK instead of having to redo it later.