Failed to build "normal" in new release. 2023(deque)

[Rini52]

Hi
Help with build new falsh from default new ESPEasy.git . I did a fresh fork from the latest release and got the error

TypeError: can only concatenate deque (not "list") to deque:

Running vscode/platformio on unbuntu Ubuntu 22.04.2 LTS (fully updated). Other 'non' ESPEasy vscode/pio builds no problem.

Thanks.
Rini

Below the last part of the terminal messages form the build

---->
-------------------------------
Compile time defines
File System: SPIFFS
PROGNAME: ESP_Easy_mega_20230417_normal_ESP8266_4M1M
BOARD_NAME: Espressif Generic ESP8266 ESPEasy 4M Flash 1M FS
BUILD_PLATFORM: Linux-5.15.0-69-generic-x86_64-with-glibc2.35
GIT_HEAD: pygit2_not_installed
CDN_URL: https://cdn.jsdelivr.net/gh/letscontrol ... sy/static/
BUILD_VERSION: 20442
BUILD_UNIXTIME: 1681734397
BUILD_RFC1123: Mon, 17 Apr 2023 12:26:37 GMT
-------------------------------
Flash configuration
--flash-size: 4MB
--flash-freq: 40000000L
--flash-mode: dout
CONFIGURATION: https://docs.platformio.org/page/boards ... board.html
PLATFORM: Espressif 8266 (2.6.3) > Espressif Generic ESP8266 ESPEasy 4M Flash 1M FS
HARDWARE: ESP8266 80MHz, 80KB RAM, 1015.93KB Flash
PACKAGES:
- framework-arduinoespressif8266 @ 2.7.4+sha.2843a5a
- tool-esptool @ 1.413.0 (4.13)
- tool-esptoolpy @ 1.30000.201119 (3.0.0)
- toolchain-xtensa @ 2.40802.200502 (4.8.2)
Converting ESPEasy.ino
LDF: Library Dependency Finder -> https://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ strict
Found 89 compatible libraries
Scanning dependencies...
Dependency Graph
|-- AM2320
|-- AS_BH1750
|-- Adafruit HMC5883 Unified @ 1.2.0
|-- Adafruit Motor Shield V2 Library @ 1.0.4
|-- Adafruit NeoPixel @ 1.10.6
|-- Adafruit Unified Sensor @ 1.0.2
|-- ArduinoJson @ 6.19.4
|-- ESPeasySerial @ 2.0.16
|-- GPIO_DirectAccess
|-- HLW8012 @ 1.1.1
|-- PZEM-004T-v30 @ 1.0.0
|-- MechInputs @ 0.0.1
|-- Regexp @ 0.1.0
|-- SDM @ 2.0.0
|-- SPI @ 1.0
|-- TinyGPSPlus @ 1.0.2
|-- WakeOnLan @ 1.1.6
|-- SerialSensors @ 0.0.1
|-- Blynk @ 1.0.1
|-- Servo(esp8266) @ 1.0.2
|-- ESP8266WiFi @ 1.0
|-- ArduinoOTA @ 1.0
|-- ESP8266HTTPClient @ 1.2
|-- RTClib @ 1.13.0
|-- Wire @ 1.0
|-- Adafruit BME680 Library @ 1.1.1
|-- Adafruit MPR121 @ 1.0.0
|-- Adafruit SGP30 Sensor @ 1.0.5
|-- Adafruit SI1145 Library @ 1.1.1
|-- Adafruit TCS34725 @ 1.0.1
|-- ESP8266_SSD1306 @ 3.2.7
|-- CHT16K33 LED Matrix Library @ 0.0.1
|-- LiquidCrystal_I2C
|-- MD_MAX72XX @ 3.3.0
|-- MD_Parola @ 3.5.6
|-- MFRC522
|-- SparkFun APDS9960 RGB and Gesture Sensor @ 1.4.2
|-- SparkFun_AS7265x
|-- XPT2046_Touchscreen
|-- FrogmoreScd30 @ 1.0
|-- IthoCC1101 @ 1.0.0
|-- Multi Channel Relay Arduino Library @ 1.1.0
|-- ShiftRegister74HC595_NonTemplate @ 1.3.1
|-- Adafruit NeoMatrix @ 1.3.0
|-- RAK12019_LTR390_UV_Light @ 1.0.1
|-- SparkFun SCD4x Arduino Library @ 1.0.4
|-- htcw_ip5306 @ 0.1.0
|-- Adafruit seesaw Library @ 1.6.3
|-- NewPing @ 1.9.6
|-- Adafruit TSL2591 Library @ 1.0.2
|-- ccronexpr @ 1.0.0
|-- Adafruit GFX Library @ 1.11.3
|-- Adafruit ILI9341 ESPEasy @ 1.5.10
|-- LOLIN_EPD @ 1.0.0
|-- VL53L0X @ 1.3.0
|-- SparkFun VL53L1X 4m Laser Distance Sensor @ 1.2.9
|-- SparkFun MAX1704x Fuel Gauge Arduino Library @ 1.0.4
|-- Adafruit_ST77xx @ 1.7.3
|-- ITG3205 @ 1.0
|-- SparkFun ADXL345 Arduino Library @ 1.0.0
|-- NeoPixelBus @ 2.6.9
|-- Adafruit PCD8544 Nokia 5110 LCD library @ 2.0.1
|-- PubSubClient @ 2.7
|-- RN2xx3 Arduino Library @ 1.0.2
|-- I2Cdevlib-Core
|-- CircularBuffer @ 1.3.3
|-- DNSServer @ 1.1.1
|-- ESP8266HTTPUpdateServer @ 1.0
|-- ESP8266WebServer @ 1.0
|-- ESP8266mDNS @ 1.2
TypeError: can only concatenate deque (not "list") to deque:
File "/home/rini/.platformio/penv/lib/python3.10/site-packages/platformio/builder/main.py", line 188:
env.SConscript("$BUILD_SCRIPT")
File "/home/rini/.platformio/packages/tool-scons/scons-local-4.5.2/SCons/Script/SConscript.py", line 598:
return _SConscript(self.fs, *files, **subst_kw)
File "/home/rini/.platformio/packages/tool-scons/scons-local-4.5.2/SCons/Script/SConscript.py", line 285:
exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
File "/home/rini/.platformio/platforms/espressif8266@2.6.3/builder/main.py", line 238:
target_elf = env.BuildProgram()
File "/home/rini/.platformio/packages/tool-scons/scons-local-4.5.2/SCons/Util/envs.py", line 242:
return self.method(*nargs, **kwargs)
File "/home/rini/.platformio/penv/lib/python3.10/site-packages/platformio/builder/tools/piobuild.py", line 61:
env.ProcessProjectDeps()
File "/home/rini/.platformio/packages/tool-scons/scons-local-4.5.2/SCons/Util/envs.py", line 242:
return self.method(*nargs, **kwargs)
File "/home/rini/.platformio/penv/lib/python3.10/site-packages/platformio/builder/tools/piobuild.py", line 163:
plb.env.BuildSources(
File "/home/rini/.platformio/packages/tool-scons/scons-local-4.5.2/SCons/Util/envs.py", line 242:
return self.method(*nargs, **kwargs)
File "/home/rini/.platformio/penv/lib/python3.10/site-packages/platformio/builder/tools/piobuild.py", line 352:
nodes = env.CollectBuildFiles(variant_dir, src_dir, src_filter)
File "/home/rini/.platformio/packages/tool-scons/scons-local-4.5.2/SCons/Util/envs.py", line 242:
return self.method(*nargs, **kwargs)
File "/home/rini/.platformio/penv/lib/python3.10/site-packages/platformio/builder/tools/piobuild.py", line 298:
new_node = callback(new_node)
File "/home/rini/ESPEasy/tools/pio/generate-compiletime-defines.py", line 107:
CPPDEFINES=env["CPPDEFINES"]
=================================================== [FAILED] Took 8.63 seconds ===================================================

Environment Status Duration
------------------- -------- ------------
normal_ESP8266_4M1M FAILED 00:00:08.634
============================================== 1 failed, 0 succeeded in 00:00:08.634 ==============================================

* The terminal process "platformio 'run', '--environment', 'normal_ESP8266_4M1M'" terminated with exit code: 1.
* Terminal will be reused by tasks, press any key to close it.

[TD-er]

Hmm, maybe the required Python packages are not installed?

Can you also test using the build script in the tools folder?
That's what I always use to build on Linux.
It does create a Python virtual env using Python 3.10, will install the required Python packages in the virtual env and try to build the separate PlatformIO envs.
You can also specify in the command options of that script whether you want to build only ESP32/esp82xx and where to build it.

[Rini52]

Thanks,

Running the ¨build_ESPeasy.sh" script in the tool directory indeed build the ¨normal_ESP8266" without any problem. So I will flash it and assume it will work okay.
Followed the instructions in the platformio and assumed it would build correctly. Did not realize I had to go to the tool directory to build it in ubuntu. Sorry for the trouble and thanks for your remarks.
The idea was to check if I could get "OPENHAB_MQTT" working for a nodo RfLink board. At the moment I´ḿ using espRLINKMQTT ,that works okay, but has the same issues (with MesH wifi networks) we had also with ESPEasy some years ago. These were solved by you and the interface of ESPEasy is much better/clearer than espRFLINKMQTT.
Unfortunately I have to go a long way to understand how to get that parsing of the RFLINK commands into your MQTT code since my C+ is a little rusty..

Anyway thanks for getting the first hurdle solved.

Rini

[TD-er]

The build script hint was mainly to make sure nothing is forgotten.
Good to know that still works, so nothing has been changed in Ubuntu which may break our build setup/tools.

[Rini52]

TD,

The build script worked okay , but it took a long time since it was building all the esp8266 versions. A small modification in the grep part help me to select the one i wanted to build.
However I also found out that my platformio version was already on 6.1.7 putting it back to 6.1.6 resolved the problem and I could build via the normal custom setup. Resetiing was not that a liitle trickky you had to use the "Platformio-ide: Use Builtin PIOCore" setting in Platformio to false.
But I'm happy, I can use the normal Espeasy clone in Ubuntu 22.04 now!

[TD-er]

OK, last time I checked (today actually) the 6.1.7 release was not yet released.
There were however issues with fetching the required Python libs today as this was nearly all day unavailable: https://files.pythonhosted.org/packages

I will keep an eye on build issues regarding PIO 6.1.7, so thanks for reporting.

[Rini52]

Don't know how I got it. Uninstalled the Pio in vscode and reinstalled it. You get the build in version 6.1.7rc1 ! . Just don't know how to get the current version 6.1.6 as the build in core. But unchecking the use of the builtin will automatically use the system default of 6.1.6 .

[jbarea]

However I also found out that my platformio version was already on 6.1.7 putting it back to 6.1.6 resolved the problem

Hi, I have the same problem, I can't build EspEasy anymore. I checked different computers and different versions. Unfortunately PIO updates automatically and now the core is 6.1.7.
I tried to use the setting Use Builtin PIOCore to false, but it doesn't work for me since it goes to version 5.1.1 or so. My question is, how can I put back PIOCore to 6.1.6 version?

[jbarea]

However I also found out that my platformio version was already on 6.1.7 putting it back to 6.1.6 resolved the problem

Hi, I have the same problem, I can't build EspEasy anymore. I checked different computers and different versions. Unfortunately PIO updates automatically and now the core is 6.1.7.
I tried to use the setting Use Builtin PIOCore to false, but it doesn't work for me since it goes to version 5.1.1 or so. My question is, how can I put back PIOCore to 6.1.6 version?

Ok, I found the way: in the terminal console $ pip install platformio=6.1.6

Now everything is ok.

[TD-er]

The latest code does allow to build again using PIO 6.1.7
I did spend all Tuesday and Wednesday to fix it.

[Micha_he]

Can we hope, that a "xxx_Normal_1M_xxx"-build will also be back or is there still a (size-)problem with it?

[TD-er]

See here for a very quick look of what's currently being built:
https://td-er.nl/ESPEasy/20230510/

(als also showing off the newly generated web flasher config )

Edit:
Nope, the normal 1M build doesn't seem to be included...
I will look into that too.

[TD-er]

Can we hope, that a "xxx_Normal_1M_xxx"-build will also be back or is there still a (size-)problem with it?

See this GH Actions build: https://github.com/letscontrolit/ESPEas ... 4962339871
N.B. code is not yet tested, just to let you know I'm working on reducing the build sizes again

Would be really appreciated if you could test it, but please do it on modules which can be flashed easily again and doesn't matter if their settings get corrupted etc.

[Rini52]

Hi TD,

In the mean time I went back to arduino-ide for the 0230509 branch. I found the following problems with 3 includes in the src directory...

C018.Config.h
C018_data_struct.h
P025.data_struct.ccp

all had their relative path wrong.

You have to change #include /src to-> #include ../../src

Also had problems with building against core 3.1.2 all gave an error like :
/home/rini/arduino-1.8.19/portable/sketchbook/libraries/ESPEasySerial/ESPEasySoftwareSerial.cpp:47:16: error: expected initializer before 'espeasy_sws_isr_0'
47 | void IRAM_ATTR espeasy_sws_isr_0() { ObjList[0]->rxRead(); };

Building with core 2.7.4 no problem....

I don't know how to get it included via push/pull github. Hopefully it helps you building a better future

Regards,
Rini

[Ath]

I've created PR #4666 to fix the #include issues.

Will have a look at the IRAM_ATTR issue with core 3.1.2 later, most likely the existing workaround that was in place is removed at some point.

[TD-er]

Maybe this isn't properly processed in Arduino IDE:

Code: Select all

  #ifndef CORE_POST_3_0_0
    #define IRAM_ATTR ICACHE_RAM_ATTR
  #endif

Or you're are trying to compile against the latest ESP8266 Arduino code?

[Rini52]

Td,

Below the plotform.local.txt file I created in hardware for the arduino-ide. Yes I just went to the newest Arduino_ide-1 version: 1.8.19. Will try the 1.18.15 I have still laying around...

--->
compiler.minsize=
compiler.c.extra_flags=-DUSE_CUSTOM_H -DPUYA_SUPPORT=0 -DCONFIG_FREERTOS_ASSERT_DISABLE -DCONFIG_LWIP_ESP_GRATUITOUS_ARP -DCONFIG_LWIP_GARP_TMR_INTERVAL=30 -DCORE_POST_2_5_0 -DCORE_POST_2_6_0 -DCORE_POST_3_0_0 -DNDEBUG -DICACHE_FLASH -DWEBSERVER_RULES_DEBUG=0 -DVTABLES_IN_FLASH -mtarget-align -Werror=unused-variable {compiler.minsize}
compiler.c.elf.extra_flags=-DUSE_CUSTOM_H -DPUYA_SUPPORT=0 -DCONFIG_FREERTOS_ASSERT_DISABLE -DCONFIG_LWIP_ESP_GRATUITOUS_ARP -DCONFIG_LWIP_GARP_TMR_INTERVAL=30 -DCORE_POST_2_5_0 -DCORE_POST_2_6_0 -DCORE_POST_3_0_0 -DNDEBUG -DICACHE_FLASH -DWEBSERVER_RULES_DEBUG=0 -DVTABLES_IN_FLASH -mtarget-align -Werror=unused-variable {compiler.minsize}
compiler.S.extra_flags=-DUSE_CUSTOM_H -DPUYA_SUPPORT=0 -DCONFIG_FREERTOS_ASSERT_DISABLE -DCONFIG_LWIP_ESP_GRATUITOUS_ARP -DCONFIG_LWIP_GARP_TMR_INTERVAL=30 -DCORE_POST_2_5_0 -DCORE_POST_2_6_0 -DCORE_POST_3_0_0 -DNDEBUG -DICACHE_FLASH -DWEBSERVER_RULES_DEBUG=0 -DVTABLES_IN_FLASH -mtarget-align -Werror=unused-variable {compiler.minsize}
compiler.cpp.extra_flags=-DUSE_CUSTOM_H -DPUYA_SUPPORT=0 -DCONFIG_FREERTOS_ASSERT_DISABLE -DCONFIG_LWIP_ESP_GRATUITOUS_ARP -DCONFIG_LWIP_GARP_TMR_INTERVAL=30 -DCORE_POST_2_5_0 -DCORE_POST_2_6_0 -DCORE_POST_3_0_0 -DNDEBUG -DICACHE_FLASH -DWEBSERVER_RULES_DEBUG=0 -DVTABLES_IN_FLASH -mtarget-align -Werror=unused-variable {compiler.minsize}
compiler.ar.extra_flags=
compiler.objcopy.eep.extra_flags=
compiler.elf2hex.extra_flags=
<---

[TD-er]

The problem isn't the Arduino IDE, but Espressif's ESP8266/Android version.

The older SDK versions used "ICACHE_RAM_ATTR" to indicate in what memory of the ESP a function should be stored.
Starting SDK 3.x this was renamed to IRAM_ATTR as that's the same for ESP32 code.

[Rini52]

Indeed in two libraries the IRAM_ATTR were (re)defined in the 3.1.2 core.
1) ESPEasySoftwareSerial.cpp
2) GPIO_Direct_Access.cpp

Just commenting them out like below, did the trick.
#ifndef CORE_POST_3_0_0
// #define ICACHE_RAM_ATTR __attribute__((noinline, section("\".iram.text." __FILE__ "." __ICACHE_STRINGIZE(__LINE__) "." __ICACHE_STRINGIZE(__COUNTER__) "\"")))
// #define IRAM_ATTR ICACHE_RAM_ATTR
#endif

Also found another missing include in htcw_ip5306. This is located in another directory (include).


The above resulted in a compilation without errors. But the implication is outside my knowledge and don't know the impact of it will cause. The custom flash firmware created is with core 3.1.2 , and runs ok on my ESP8266. But did not test GPIO pins etc...

I also included/defined another custom platform 4.1.0. in platformio in vscode. This also included the core 3.1.2 and compiles ok.. Problem with Platformio setup it takes me ages to find out were what is set or included.

Regards,
Rini

[Ath]

Problem with Platformio setup it takes me ages to find out were what is set or included.

When following the guidelines for local development (using PlatformIO), all is working as intended (and tested), without the need to search for these kind of settings...

[Rini52]

Dear

I tried to follow that indeed. But had to go to a steep git,vscode, and platformio learning curve , not to mention EspEasy itself. If you'r a ms guru oke, I guess your familiar with all the particulars. However I wanted to include the core 3.1.2 and the only way for me was to find out how the you had glued all these libs etc together. Aapparently I had to define a new platform set to achieve my goal and used your 4.1.0 example. This finally worked to get me core 3.1.2 included. So its not that straightforward as you might think for someone occasionally doing some odd things.
Its much more straightforward in Arduino. Just copy the lib and src directory to ..../portable update the custom.h/platform.txt and set core version in my case and your off to go.
But I assume i will loose this discussion from the professionals!!!!!

Anyway thanks for all the help from your lot to get things going....

[Ath]

If you're not into software development, why are you trying to switch to Arduino 3.1.2 framework? If there where pressing reasons to do so, it would have been done already... Biggest disadvantage of the 3.x line is that it has a bigger flash-footprint, leaving less room for plugins and other features.

Its much more straightforward in Arduino. Just copy the lib and src directory to ..../portable update the custom.h/platform.txt and set core version in my case and your off to go.
But I assume i will loose this discussion from the professionals!!!!!

That 'simple' process has stopped me from using the Arduino IDE, that IMHO is also using a rather dated UI and UX, and I didn't even know how to use git (and github) when I started my ESPEasy journey, a couple of years ago. The easy proces of "install git and PIO, fork & clone the repo, press the build button and a .bin roles out" was a real improvement over the old Arduino procedure.

But we've recently started work on a web-based UI for building and uploading personalized ESPEasy binaries, that'll get some mass during the coming months.

[TD-er]

Why did you need a build based on the core 3.1.2 code?
Just curious, as I like to know what makes it a preferred choice compared to what we're having right now.

[Rini52]

TD,

The reason was i ended up with disconnecting boards while using ASUS mesh routers and connected to the central unit. So I hoped that a new core would help to achieve a stable connection. I found out that I needed to remove the IMGB setting (multi to uni broadcast) on the ASUS router to keep it connected. Maybe in combination with this new firmware(core3.1.2.) I don't know, but it is now connected already for a week, so good hope now. I put in a rule to check the MQTT connection if it is offline. Also it rebooted oke and connected without a problem, This didn't happen with my old build.

But the mayor problem lays in the combination ASUS mesh and ESP8266, I assume ,because all other devices work oke.

Once again thanks for all your effort putting in this project.
Rini

[Rini52]

Hi ath

"If you're not into software development, why are you trying to switch to Arduino 3.1.2 framework? If there where pressing reasons to do so, it would have been done already... Biggest disadvantage of the 3.x line is that it has a bigger flash-footprint, leaving less room for plugins and other features"

The arduino build firmware seems smaller that the "410" build with Platformio. The Arduine leaves FRee RAM: 21144 vs Vscode/Platformio :13224 Free Ram.

RIni

[TD-er]

"free ram" highly depends on the included plugins in a build and also which features are included.
For example some "collections" builds may even give you more free memory compared to a "normal" build, since some features are disabled in those "collections" builds and thus even though they have more plugins included, there may be more free RAM.

But I will look into the ESP8266 core 3.1.x build.
As far as I know, the "beta" builds would always use the latest SDK, but maybe I missed some here.
Will look at it tomorrow.

A few tricks to get Asus AP's working with ESPEasy:

- Disable "Band Steering" in the access point.
- Set ESPeasy to only allow using 802.11g or b. (tools->Advanded page: "Force WiFi B/G")
- Check the checkbox "Enable SDK WiFi Auto Reconnect:" in ESPeasy

You can also test using the latest 20230508 build: https://td-er.nl/ESPEasy/
Or click on "latest" at the bottom of that page for builds of yesterday's state of the code.

[Rini52]

TD

Thanks for your reply. However;

->
A few tricks to get Asus AP's working with ESPEasy:

- Disable "Band Steering" in the access point.
- Set ESPeasy to only allow using 802.11g or b. (tools->Advanded page: "Force WiFi B/G")
- Check the checkbox "Enable SDK WiFi Auto Reconnect:" in ESPeasy
_<

Did all the above and more but it did not help. Only switching of the IGMP-snooping worked.

I loaded your new build en tested/included the above. Same result no connection if IGMP is on.

Thanks for the new flash included in your link. It work okay also in linux. Unfortunatly its not included in FireFox which is my default browser but in Chrome it went perfect, also the terminal.
Great improvement of the esptool, which is always tricky...

Rini

[TD-er]

Yep, it won't work in Firefox, since it is lacking the API access to serial ports which is offered by Edge and Chrome.

[TD-er]

I was just thinking about what you had to turn off to get ESPEasy to run....
IGMP snooping is about multicast traffic.
The only thing I can think of that may need multicast in some form is mDNS.

ESPEasy p2p uses UDP and may send it to the broadcast address, however this is not in the multicast IP range.
So I really don't understand why this would make a difference.

But if I do understand your posts correctly, you still need to disable IGMP snooping on your network device(s) in order to keep ESPEasy connected, even when running the latest ESP8266 core code?

[Rini52]

" But if I do understand your posts correctly, you still need to disable IGMP snooping on your network device(s) in order to keep ESPEasy connected, even when running the latest ESP8266 core code?"

yes correct, IGMP-snooping needs to disabled to connect to my central ASUS router.

[Rini52]

to clearify

I disable it in the ASUS-Mesh (RT-AX92U) router.

[Rini52]

Td,

I don't use p2p in my setup in Espeasy. Only serial,systeminfo and ping device and MQTT controller active to check the connection.

Attached is part of a discussion on snbforum about multicast and IGMP-snooping in connection to ASUS routers. Hence the switching off of IGMP.....

Hope you can make some sense of it. But in the mean time its working oke and I don't have ITV or streaming. My wife has Apple hardware, but that seems to work okay with viewing Netflix etc...

Regards,
Rini