SDK migration headaches

[martinus]

I thought it would be nice to spend this free time on migrating my IDE environment from Arduino 2.x (IDF 4.x) to 3.x (IDF 5.x).
In the past, not so much to do, but this upgraded seems to come with a lot of backwards incompatibilities
Just to name a few compile issues:

• Direct flash routines are changed (like spi_flash_erase_sector)

• ESPNOW starts to complain on the receive callback

• M5Stack display library fails

• Servo32 library fails

and likely more to come when I proceed. I think I'm going to postpone this a bit...

I also guess it's time to upgrade some of my libraries as they are quite old (based on the concept: 'If it ain't broke, don't fix it')
I guess the current ESPEasy developers must have also spend quite some time to implement this upgrade?
(not sure if there's a list somewhere? The formal list of breaking changes is incomplete and does not inform on incompatible library versions)

[TD-er]

I have done this migration while the Arduino guys also developed Arduino 3.0.
So all-in-all, it has taken about 18 months (not 18 months of work, but you get the idea)
Since PlatformIO did not have a platform, Jason2866 and I used our own platform builds.
At some point, it became clear there was some dispute going on between Espressif and PlatformIO and that there would be no official PlatformIO build for Arduino3.0/ESP-IDF5.x
This led to us opening the Platform builds we were using for over 2 years at that time under the name PioArduino.

You can look at the ESPeasy Platformio.ini files to see which flags and platform builds I am using.

N.B. for SPIFFS builds I am using ESP-IDF5.1 and for LittleFS builds ESP-IDF5.3 right now.
You can also use the pioarduino release builds, but those will result in quite a bit larger builds right now as there are lots of encryption cyphers linked in.

Another thing you should look into is direct GPIO access, which has become a lot (!!) slower when using digitalRead or digitalWrite.
Like a factor 100x - 1000x slower.
So you can also have a look at the direct GPIO lib I included in ESPEasy's lib folder.

[martinus]

I'm still on the Arduino IDE (1.8.19) for all my projects, as it has been my loyal companion for the last 15 years or so.
My sketches are not so complex (My 'legacy' ESPEasy version is just 42 ino files, including 28 plugins), but I can imagine that current ESPEasy maybe too much for the Arduino IDE to cope with.

I also moved from SPIFFS to littleFS a few months ago, because SPIFFS support will be dropped in the future.

I've also tried the Arduino 2.0 but it lacks the portable mode so I'll stick with the old one for now.
Not sure if the issues on IDF-5.x are due to using Arduino 1.8.19. Currently trying ESP32 Arduino Release 3.0.7.

[TD-er]

Just keep in mind that quite a lot of libraries have not yet been adapted to the latest SDK versions.
ESPEasy and Tasmota are the only 'larger' firmwares which have made the transition.

[martinus]

Just keep in mind that quite a lot of libraries have not yet been adapted to the latest SDK versions.
ESPEasy and Tasmota are the only 'larger' firmwares which have made the transition.

I basically use the same libraries as the formal ESPEasy project. So the solution to my issues is likely to be found somewhere in the ESP Easy repository.
And if I'm lucky, you may have already done the dirty work
Only thing I'm not sure about is the use of ESPNOW. Was that ever implemented?

[TD-er]

There is a pending PR in which I have made some ESP-NOW code.
However this still does have issues as it can mess up its mesh network structure resulting in packets 'running' in circles.

Also there was some issue where querying the used wifi network resulted in getting the wrong channel reported when using ESP-NOW and even checking the RSSI of the WiFi adapter would result in strange ESP-NOW behavior.

I plan on continuing that PR after I'm done refactoring the WiFi/network code.
Now since Arduino3.x there is actual structure in the network code and a lot (!!) of bugs have been fixed either in Arduino or IDF code.
So I want to get rid of the extremely complex ESPEasy network/WiFi code which is the result of 7 years of patching and creating work-arounds of bugs in the closed source code.
I already have made quite a lot of progress, but still am not done with it.


Just to give you some heads up of possible hours of debugging:

- Mac address cannot be queried for unless the interface has been started.
- Use AP interface for nodes which are "always on" as these do not "sleep" which greatly improves the responsiveness of devices trying to send to it. (you can only query its MAC after starting, or 'compute' it from the STA interface)
- Just to be sure, use channel 1 and nothing else.
- Do not assume your ESP-NOW still has the peer set, thus either check for the peer or set it before sending
- I found the 'broadcast' to be working unreliable, but maybe this has changed now.

[martinus]

I have done this migration while the Arduino guys also developed Arduino 3.0.
So all-in-all, it has taken about 18 months (not 18 months of work, but you get the idea)

Well thanks to you and others to do the dirty groundwork for me.
I guess because of that it took me only a few hours to be able to compile my project on Arduino Release 3.x with IDF 5.x without errors.

The issues were mainly on the plugins:

• OneWire: Upgrade from 2.3.7 to 2.3.8 did the job

• IRRemote8266:2.7.4 to 2.8.6 did not help, used the one from espeasy repo

• ServoESP32: used the one from espeasy repo

• M5Stack: Manually patched some lines of code

• ESPNOW: moved #include <esp_now.h> to Globals.h (unknown why, but it works that way)

• Wireguard: The library that I use is not updated yet, but I could manually patch it to work

Now I can start actual tests to see if it also really works on a module.

It's a bit sad to see that they did so many breaking changes and several libraries seem abandoned by the respective maintainers.
And typically normal users just update their environment because they believe it is always good to use the most recent platform and libraries, and then run into unrecoverable issues.
Most of the workaround that I've seen, only advice to revert to Arduino 2.x platform.

[Ath]

• IRRemote8266:2.7.4 to 2.8.6 did not help, used the one from espeasy repo

There's an open PR #2040 over there, that I created after fixing the ESPEasy code, but the maintainers feel the scope of my PR is too big, so they are reluctant to merge it

It's a bit sad to see that they did so many breaking changes and several libraries seem abandoned by the respective maintainers.
And typically normal users just update their environment because they believe it is always good to use the most recent platform and libraries, and then run into unrecoverable issues.
Most of the workaround that I've seen, only advice to revert to Arduino 2.x platform.

I'm afraid that's kind of the defacto standard in OSS, after initial creation of (usually) good stuff, it's often abandoned because of the effort it takes to keep stuff up to date. For C++ & Arduino libraries it's not as bad as for Javascript/Typescript libraries though, where they seem to expire with every underwear change

[TD-er]

I would love it if you told me what has to be changed for Wireguard to be used.
I plan on adding Wireguard to ESPEasy as it does remove the need to open up ports or complex setups with VPN tools.
I only did some copy lib, include some files to make sure it would be used and a test build and this built fine, but wasn't actually using it.
I did however use a pending PR on a fork as that was told to be working in some comments.

So maybe I don't need to change anything and it will "Just work"

[TD-er]

Oh and for ESP-NOW, the Arduino repo now also has some code to "Arduino-ize" it.
Never tried it as Jason2866 already removed that lib from the PlatformIO package we've been using and it did add quite a lot to the build size. Probably includes more than needed.

[martinus]

I would love it if you told me what has to be changed for Wireguard to be used.
I plan on adding Wireguard to ESPEasy as it does remove the need to open up ports or complex setups with VPN tools.
I only did some copy lib, include some files to make sure it would be used and a test build and this built fine, but wasn't actually using it.
I did however use a pending PR on a fork as that was told to be working in some comments.
So maybe I don't need to change anything and it will "Just work"

Maybe it depends on where you have got your library. I'm using Wireguard for half a year now on my ESP8266 and ESP32 modules that run from my second home (chalet on campsite). Using Wireguard Addon on Home Assistant as server. It is really cool because at the remote site, the ESP modules just connect to a Wifi hotspot and then establishes a WireGuard connection to phone home.

I got my library from here
https://github.com/ciniml/WireGuard-ESP32-Arduino
It relies on tcpip_adapter and works for IDF 4.x but not on IDF 5.x, at least not using Arduino IDE
This library also looks abandoned, no updates for three years

The changes were inspired from this library that is not for Arduino but it has the right fixes in Wireguardif.c
https://github.com/trombik/esp_wireguar ... eguardif.c

Mainly this piece of code:

Code: Select all

#if defined(CONFIG_WIREGUARD_ESP_NETIF)
	struct netif* underlying_netif = NULL;
	char lwip_netif_name[8] = {0,};

	err = esp_netif_get_netif_impl_name(esp_netif_get_handle_from_ifkey("WIFI_STA_DEF"), lwip_netif_name);
	if (err != ESP_OK) {
		ESP_LOGE(TAG, "esp_netif_get_netif_impl_name: %s", esp_err_to_name(err));
		result = ERR_IF;
		goto fail;
	}
	underlying_netif = netif_find(lwip_netif_name);
	if (underlying_netif == NULL) {
		ESP_LOGE(TAG, "netif_find: cannot find WIFI_STA_DEF");
		result = ERR_IF;
		goto fail;
	}
#elif defined(CONFIG_WIREGUARD_ESP_TCPIP_ADAPTER)
	void *underlying_netif = NULL;
	err = tcpip_adapter_get_netif(TCPIP_ADAPTER_IF_STA, &underlying_netif);
	if (err != ESP_OK) {
		ESP_LOGE(TAG, "tcpip_adapter_get_netif: %s", esp_err_to_name(err));
		result = ERR_IF;
		goto fail;
	}
#endif

Essentially replacing this:

Code: Select all

tcpip_adapter_get_netif(TCPIP_ADAPTER_IF_STA, &underlying_netif);

with this:

Code: Select all

err = esp_netif_get_netif_impl_name(esp_netif_get_handle_from_ifkey("WIFI_STA_DEF"), lwip_netif_name);

[martinus]

I'm afraid that's kind of the defacto standard in OSS, after initial creation of (usually) good stuff, it's often abandoned because of the effort it takes to keep stuff up to date.

So it is very nice that this did not happen with ESPEasy after I 'abandoned' it. Well, at least I did a proper handover first.

[TD-er]

Yep, still going strong
When I want to implement all ideas I now have for ESPEasy, I will be busy at least till my pension (I'm < 50 now) let alone all new ideas I will get in the upcoming years

[martinus]

Well everything was running fine on Arduino 3.0.7 using IDF 5.1.4. But moving to 3.1.0 with IDF 5.3 is another crash experience.

It seems they have enabled CONFIG_LWIP_TCPIP_CORE_LOCKING, leading to another painfully breaking change to existing network related libraries.
The Arduino core team has likely fixed all the libraries that are part of the core, but at least using the Wireguard lib goes directly into a core panic:

Code: Select all

Initializing WireGuard...
assert failed: netif_add /IDF/components/lwip/lwip/src/core/netif.c:286 (Required to lock TCPIP core functionality!)

It looks like we have to add mutex locks into each piece of code that calls the TCPIP stack API's, like discussed in this topic:
https://github.com/espressif/arduino-esp32/issues/10526

[martinus]

I was looking for a ready made new library for WireGuard but I could find one.
I did find this one that is used by ESPHome:
https://github.com/droscy/esp_wireguard ... eguardif.c
But also that one does not have patches for 3.1.0, so I decided to start the challenge on my own.

It took half a day with a lot of trial and error, but I finally have the WireGuard library working again on Arduino 3.1.0 with IDF 5.3.2.
You need to add a lot of Mutex wrappers on net_if_xxx calls, upd_recv and more...
Not even sure if that is the proper way, I got the inspiration from looking at the latest "AsyncUDP" library that is part of the 3.1.0 package.

I hope that the core devs are not planning to do a lot more breaking changes with every update

[TD-er]

Is the wireguard implementation acting as an interface?
It is now also possible to set a default interface, though I don't know whether it uses the gateway assigned to an interface or other other way around.

[martinus]

Is the wireguard implementation acting as an interface?

I don't know, have not checked the code in detail. AFAIK it just binds to an existing interface, current library is even fixed to only use STA interface. I've seen a PR on supporting the AP interface as well, but I don't need that.

[martinus]

Upgraded several ESP32 (classic) to the latest 5.3.2 IDF and for now, it looks ok.

I have one test unit based on ESP32-C3. That one has an issue with MQTT for some unknown reason.
Exact same sketch, compiled on IDF 5.1 runs fine, compiled on 5.3, cannot connect to MQTT server

@Gijs, did you encounter specific issues with the C3 when using IDF 5.3 related to networking?

[TD-er]

Yep, there was an issue, but it was fixed (by me) right before the latest release of Arduino days before Christmas.
Problem was uninitialized data in NetworkClient.

You can try one of the LittleFS builds of the last ESPEasy builds to see if it does work on your setup.
See: https://td-er.nl/ESPEasy/

[martinus]

I think the plan to upgrade my entire environment (source code, platforms, modules) during the Christmas/new year holidays is going to be successful in the end
Today is the last day of my holiday and I've managed to fix the last showstopper, being MQTT on the C3 module. I've tried the pubsubclient from ESPEasy repo but that one also failed on me.
During testing and using serial.prints with a small delay I accidentally discovered that a "delay(1)" solved the connection issue.

So now I'm using pubsubclient 2.7 stock with only one modification in the connect code:

Code: Select all

            while (!_client->available()) {
                unsigned long t = millis();
                if (t-lastInActivity >= ((int32_t) this->socketTimeout*1000UL)) {
                    _state = MQTT_CONNECTION_TIMEOUT;
                    _client->stop();
                    return false;
                }
                delay(1); // This seems needed on ESP-C3 otherwise it will not connect !!!
            }

It works, but IMHO it should not be needed because we have RTOS running on C3. I guess the fix somehow works because the delay also returns to the RTOS scheduler.
What's even more strange, when I started an MQTT connection manually from either web or telnet, it works with the default pubsubclient library without delay(1).
So for some strange reason, it works OOTB if any other client is connected at the same time.

Puzzled, but at least it's all working now on IDF 5.3.2 !

[TD-er]

This doesn't sound too odd to me, as the C3 is a single core.
So what you're doing by calling delay() in Arduino is you let the Arduino SDK code check for WiFi events to be processed.
I always got the impression those C-versions were a bit more prone to connection issues and missing events.

In the WiFi code of ESPEasy, there are some delay() calls right after any call to something WiFi related just for this reason.

I suspect your code may be doing some DNS resolve at that point, which I know is still being a bit flaky at best.

[martinus]

This doesn't sound too odd to me, as the C3 is a single core.
So what you're doing by calling delay() in Arduino is you let the Arduino SDK code check for WiFi events to be processed.

Yes that somehow has to be related. But on preemptive RTOS, the scheduler should timeslice both Wifi task and loopTask, at least each millisecond. There should not be any reason to delay or yield to handover cpu time to the wifi task on RTOS.

Even more strange, this patch was not needed on IDF5.1. They must have changed something related in the most recent IDF to result into this issue.

[martinus]

Anyway, I'm a happy camper now, also on the C3 with just a single core.
I don't notice any slower responses on the C3 compared with the classic ESP32.

RTOS_C3.png (13.08 KiB) Viewed 1547 times