Page 4 of 5

Re: Hardware Watchdog Reboots

Posted: 05 Sep 2019, 18:11
by ThomasB
Today Hackaday published information on three of the latest ESP32/8266 exploit vulnerabilities. It seems that hackers can easily cause a WiFi crash. Geez, what a surprise.
https://hackaday.com/2019/09/05/esp8266 ... fi-hacked/

I'm wondering if Expressif's recently patched library will have any affect on our WiFi related crash issues. Even if it doesn't, the information provides more evidence that Expressif's programming bugs will continue to torture us. Ideally, they should publicly release the source files to their cores so the ESP community can help fix the problems.

- Thomas

Re: Hardware Watchdog Reboots

Posted: 05 Sep 2019, 19:45
by TD-er
I just read the same post on HAD.
The session hijack is one that sounds dangerous, but as far as I know we don't even support that specific protocol yet.
The others are active attacks to just crash a node.
Currently you don't need a neighbor with a grudge against your toys for that :(

Re: Hardware Watchdog Reboots

Posted: 06 Sep 2019, 10:36
by TD-er
See also the (15 days old) issue here: https://github.com/esp8266/Arduino/issues/6436
Core 2.6.0 should be safe. No sure if our latest build already has the fix for it though.

Core 2.5.2 is still vulnerable.

Re: Hardware Watchdog Reboots

Posted: 18 Sep 2019, 06:55
by dynamicdave
Just checked one of my Wemos D1 Minis and found it had been running for over 18-days without a reboot.
This one has capacitors (large and small) fitted to the +5V and +3.3V power rails.
ScreenShot095.png
ScreenShot095.png (20.02 KiB) Viewed 45578 times

Re: Hardware Watchdog Reboots

Posted: 18 Sep 2019, 09:51
by TD-er
And a fairly recent build I assume?

Re: Hardware Watchdog Reboots

Posted: 20 Sep 2019, 11:09
by Tobbes
Hello,
I have also trouble with the Hardware Watchdog Reboots.
I use a Counter to measure the water consumption in my house. Every turn around from the water meter is counted via a light barrier from the ESP8266.
Tryed a Wemos D1 and a Nodemcu and both has several reboots over a day. The power plug is a good 2Ah from an Ipad. The device must run over 24h and then I reboot it per rule to reset the counter.
A minute before I reboot, the device send me an email and the values are send to Thingspeak. So the firmware cannot chosen very old, because this features are from the near past.

I tryed the firmware from "mega-20190827" and set "Force WiFi No sleep" to true. The repeater for the wlan connection is directly placed next to the esp8266.
How can I improve the setup to achieve a 24hours running device?

@dynamicdave, would you be so kind to show a photo or sketch from the capacitor hack you use? Maybe this will help in my case too.

Thanks a lot for your work.
Tobbes

Re: Hardware Watchdog Reboots

Posted: 20 Sep 2019, 11:11
by TD-er
Pulse counter may have other issues, since it is using interrupts on a GPIO pin.
So the WDT reboots here may have a different cause than WiFi.

Re: Hardware Watchdog Reboots

Posted: 20 Sep 2019, 11:46
by georgep
Tobbes wrote: 20 Sep 2019, 11:09 ...
The repeater for the wlan connection is directly placed next to the esp8266.
How can I improve the setup to achieve a 24hours running device?
...
I have seen odd things happen with my WiFi if my ESP devices choose to connect to my repeater rather than to my main Wifi [router].

Different repeaters seem to work in different ways - mine (a TP-Link device) repeats the same SSID but the MAC addresses of connected devices (the first three parts) are changed by the repeater; like some weird form of NAT at the MAC address level that I'd sever seen until I bought this repeater!

In my environment I have seen this cause some issues:

1) A connected device can get a different IP address each time it reconnects depending on whether it connects to the repeater or to the router. Depending on your DHCP server you can overcome this by having the server assign the same IP to the two different MAC addresses of each device.

2) If the above "fix" is not applied I have seen some devices rapidly switch between 2 IP addresses (I have no idea why this happens).
As far as I can tell the client sends a DHCP request, but not for the address that it is currently using but for the one it had last time; the server refuses this and offers a different address from the pool, which the client accepts. The next time the client tries to renew (often only a minute later, even with a lease time of several days???) it again appears to ask for the "wrong" address and the loop repeats over and over with the server switching the client between 2 addresses from the pool!!

I've not had time to investigate this in any detail and can only see from the DHCP server logs that something "isn't right" and so the above may not be 100% accurate.

Could you perhaps switch off your repeater and see if the reboots continue in the same way? You may have to relocate the ESP if possible?

Just my experiences and thoughts :) ...
George

Re: Hardware Watchdog Reboots

Posted: 23 Sep 2019, 09:37
by Tobbes
Hello and thanks a lot for your input.

I tryed to run the Wemos D1 with capatitors - unfortunatelly without success.
Also a dedicated wlan for the wemos endet in several reboots.

So I setup a tasmota counter. This firmware works fine over 24h as a counter - but I would prefer Espeasy because of the features as Thinkspeak and email sending...

Best regards
Tobbes

Re: Hardware Watchdog Reboots

Posted: 28 Sep 2019, 17:35
by dynamicdave
Just a quick update...

Just checked one of my Wemos nodes and was surprised that it had gone for 29-days without a reboot !!!

Re: Hardware Watchdog Reboots

Posted: 28 Sep 2019, 17:39
by TD-er
Good to hear it is not only my own nodes having again double digits in days uptime :)

We're moving in the right direction again... finally.

Still, not all nodes I have are that stable, so we're not there yet.

Re: Hardware Watchdog Reboots

Posted: 29 Sep 2019, 02:26
by Sasch600xt
Hello all,

2019 i had big trouble to get my nodes stable.

i have a nodemcu with normal 4mb build from 27.08.2019 and a few nodes using still 30.07.2019.

But now ALL my nodes running pretty stable.
At least for me was the gamechanger to set my router and AP´s to a FIX wifi channel and do not let them change the channel automatic.

Hope that helps :)
Have a great day !
uptime.jpg
uptime.jpg (66.07 KiB) Viewed 45280 times

Re: Hardware Watchdog Reboots

Posted: 29 Sep 2019, 08:18
by dynamicdave
All my nodes (Wemos D1 Minis) have fixed IPs.

They all run Home Assistant (OpenHAB) MQTT.

I'll have to check and try fixing the WiFi channel and see if I can get a length of time that matches @Sasch600xt

Re: Hardware Watchdog Reboots

Posted: 29 Sep 2019, 10:45
by TD-er
Sascha, I am curious, how many times did that node reconnect to WiFi?

Also curious, do you run any MQTT controllers?

Re: Hardware Watchdog Reboots

Posted: 29 Sep 2019, 13:39
by georgep
Just popping in with my latest status too - this is the 'best' uptime for me so far and I have another unit with the same "test build" with a similar uptime.
My other units are not so stable...

Image
~~~~
Image
~~~~
Image


George

Re: Hardware Watchdog Reboots

Posted: 29 Sep 2019, 19:34
by Sasch600xt
@ TD-er
wifi.jpg
wifi.jpg (313.93 KiB) Viewed 45232 times
i run openHab MQTT ,16 IOs INPUT, 16 IOs OUTPUT and a few generic data (uptime, rssi) every 60 seconds by MQTT.

If you want i can send you my config file and rule sets ?

Hope that helps

By the way, i made some PCBs for my nodes including capacitors for better stable conditions.
So they are not "floating in the air".

Re: Hardware Watchdog Reboots

Posted: 29 Sep 2019, 22:30
by TD-er
Good to know there are other nodes out there having multiple WiFi disconnects and MQTT and still running for multiple weeks.

Re: Hardware Watchdog Reboots

Posted: 30 Sep 2019, 09:44
by dynamicdave
Hi Sasch,
Could I take you up on your offer to send/publish your config settings as I'd like to setup one of my spare Wemos D1 Mini as a test vehicle.

Regards, David.

PS:
Oops - should have opened my eyes and look in the thread above.

Re: Hardware Watchdog Reboots

Posted: 30 Sep 2019, 15:46
by Sasch600xt
@TD-er

yes, but at the moment i am afraid to change my wifi channel back to "auto".
Next reboot and i will try. Would be intresting to see reboots increase after wifi channel set to "auto"




@dynamicdave

if you want send me an privat message with your email adress and i´ll send you my config file and my rule set.
So you can make an 1:1 copy on your spare node.

Re: Hardware Watchdog Reboots

Posted: 30 Sep 2019, 21:21
by TD-er
Well, don't tempt the system if it is working fine now.
I have a few of my nodes connected to an AP that does change its channel every now and then.
I even disconnect the units every now and then from the web interface of the AP and now have it attached direct to a metal plate, with another AP right next to it to cause as much interference as possible.
So the worst case scenarios are being tested :)

Re: Hardware Watchdog Reboots

Posted: 02 Nov 2019, 13:41
by Sasch600xt
Hello all,

what i reconized in the past is:

After i flashed new firmware it seems like my dream comes true in case of stability and no reboot.

But after a few weeks it started to be weak again.
So after i flashed a new firmware the ESPs (5 pieces) where online for about 2 weeks.
Then a reboot after a week, then after 2 days and now i do not even get a full 24 hour.

So this is what i can see at the devices.

Hope that helps and can be improved in future.
I need at least a full day without reboot.

Have a great Day

Re: Hardware Watchdog Reboots

Posted: 02 Nov 2019, 15:25
by Drum
Just out of curiosity, are the devices consistent, rebooting at similar times or are they all over the place? Have you tried a control with something like the old stable V120?
Do you have a syslog server you can send everything to see what is happening?

Re: Hardware Watchdog Reboots

Posted: 02 Nov 2019, 17:10
by TD-er
Do you use fixed WiFi channel?
What if you also reboot your router/AP as soon as you realize the ESP nodes reboot for the first time after a long time.

Re: Hardware Watchdog Reboots

Posted: 03 Nov 2019, 18:50
by Sasch600xt
no, not all at the same time. The devices reboot kinda randomly.

And yes, i still use fixed wifi channels.

Have a great day

Re: Hardware Watchdog Reboots

Posted: 09 Nov 2019, 15:42
by Sasch600xt
i have 2 enviroments:

1: Home / Fritzbox Wifi

2: Company / Unifi Wifi

I have a big different between wd reboots between the 2 enviroments.

A few days ago i swaped 2 devices from home to company and visa vers.

So the one wich was stable at home runs only 100 - 500 minutes untill reboot.

The one wich was unstable at the company runs since 3 days now without a reboot


So i have the tiny idea it has something to do with Unifi APs.

The unifi controller is a BIG tool, has anyone expierance with Unifi and could give me an advice what settings i could check or change ?

Re: Hardware Watchdog Reboots

Posted: 09 Nov 2019, 16:55
by TD-er
I don't have Unifi tools here, so I cannot test it.
Also I don't think you can adjust how the company WiFi is configured, which leaves it to just try and get the ESPEasy settings right.

I assume you have tried to:
- Disable ECO mode
- Enable WiFi no sleep
- B/G only mode (if it is allowed on your company network)

Re: Hardware Watchdog Reboots

Posted: 09 Nov 2019, 18:24
by Sasch600xt
oop....no

i did not have a look for those settings ever :(

Eco Mode was disabled at all of my nodes.
Wifi no sleep and B/G only mode i activated now.

Some of my nodes had the box for "Periodical send Gratuitous ARP" checked and others not.
So i activated this on all my nodes for testing now even if i dont know exactley what it do.

These are Settings for Wifi with Unifi. Do you see something i would have to change ?
The whole SSID "Hopp-Technik" ist just for the ESps. So we can change everything here without disturbing the other SSIDs.
unifi_config.png
unifi_config.png (74.73 KiB) Viewed 44331 times

Re: Hardware Watchdog Reboots

Posted: 09 Nov 2019, 19:28
by TD-er
Not sure what the renewal of the keyring does to the ESP stability, so maybe you can play with this setting?
The most important setting (which I did not see on your screenshot) is to set the WiFi channel to a fixed channel.

Re: Hardware Watchdog Reboots

Posted: 09 Nov 2019, 19:55
by Sasch600xt
a fixed channel is a problem at the company because i have 4 APs there.

And all 4 at the same channel causes trouble.
With one router at home sure no problem.

but i had also the 4 APs running at the same channel at the company for about 2 weeks.
No improvement of the stability there..

Re: Hardware Watchdog Reboots

Posted: 09 Nov 2019, 23:47
by Sasch600xt
i fixed the channels again.

every AP has his own channel now and is fixed.

With the new Firmware 20191108 i have the problem now that sometimes after a reset the device do not connect to wifi
untill i toggle power.

Re: Hardware Watchdog Reboots

Posted: 10 Nov 2019, 10:23
by TD-er
Sasch600xt wrote: 09 Nov 2019, 23:47 [...]
With the new Firmware 20191108 i have the problem now that sometimes after a reset the device do not connect to wifi
until i toggle power.
What core lib version?

Re: Hardware Watchdog Reboots

Posted: 10 Nov 2019, 12:19
by Sasch600xt
"What core lib version?"

How i find out ?

or do you taking about the firmware ?
I am using normal 1M4M always.

Re: Hardware Watchdog Reboots

Posted: 10 Nov 2019, 13:30
by georgep
Sasch600xt wrote: 10 Nov 2019, 12:19 "What core lib version?"

How i find out ?
Look for this section in Tools / Info ::

Firmware
Build:⋄ 20104 - Mega
System Libraries:⋄ ESP82xx Core 2.6.0-dev stage, NONOS SDK 2.2.2-dev(38a443e), LWIP: 2.1.2 PUYA support
Git Build:⋄ mega-20191104
Plugins:⋄ 46 [Normal]
Build Md5: dca44f27362cb6df4e6b5d17109a34d
Md5 check: passed.
Build Time:⋄ Nov 4 2019 03:20:41
Binary Filename:⋄ ESP_Easy_mega-20191104_normal_core_260_sdk222_alpha_ESP8266_4M1M

Re: Hardware Watchdog Reboots

Posted: 10 Nov 2019, 13:38
by Sasch600xt
Firmware
Build:⋄ 20104 - Mega
System Libraries:⋄ ESP82xx Core 2_5_2, NONOS SDK 2.2.1(cfd48f3), LWIP: 2.1.2 PUYA support
Git Build:⋄ mega-20191108
Plugins:⋄ 46 [Normal]
Build Md5: a95cf6ec0fd893adaaea7e09180f196
Md5 check: passed.
Build Time:⋄ Nov 8 2019 03:19:01
Binary Filename:⋄ ESP_Easy_mega-20191108_normal_ESP8266_4M1M.bin


is that means it is not enough to update the ESPs with new firmware, i have from time to time start from scratch and flash it with USB cable ?
Or is there another was to update core ?

Re: Hardware Watchdog Reboots

Posted: 10 Nov 2019, 13:49
by TD-er
The core library is selected at the build of the firmware.
As you can see in the nightly builds ZIP file, there are several versions in there and some have "core_260" or something similar in the name.
You can upload via OTA, from the webpage. (Tools => Update FIrmware)
Just make sure you select the right flash layout (4M1M) or else the settings and rules will be erased.

But right now, I am not sure the issue you're seeing will be fixed with the last build. (you may try though)
Please let me know if it has been fixed when running core 2.6.0

Re: Hardware Watchdog Reboots

Posted: 10 Nov 2019, 13:58
by Sasch600xt
i did an update to :

ESP_Easy_mega-20191108_normal_core_260_sdk222_alpha_ESP8266_4M1M.bin

so lets see next hours...

At home i have with fritzbox wifi between 40 and 100 days already (5 devices)

Here at the company i do not even get 24 hours out of it :(

Re: Hardware Watchdog Reboots

Posted: 10 Nov 2019, 15:08
by TD-er
Yep, Fritzbox WiFi is exceptionally stable and also really well tested here at my house :)
I have 3x FritzBox 1750e extender (setup as accesspoint, not extender) and a FritzBox 7581 as modem/router (dual-link VDSL).

I've been using FritzBox for over 12 years now and am really happy with them.
In January I have to move to a temporal house and there's no VDSL, so I have to temporary switch to another cable provider, which means I have to switch to some other router.
That's something I really will miss.

Re: Hardware Watchdog Reboots

Posted: 11 Nov 2019, 08:17
by riker1
Sasch600xt wrote: 09 Nov 2019, 15:42 i have 2 enviroments:

1: Home / Fritzbox Wifi

2: Company / Unifi Wifi

I have a big different between wd reboots between the 2 enviroments.

A few days ago i swaped 2 devices from home to company and visa vers.

So the one wich was stable at home runs only 100 - 500 minutes untill reboot.

The one wich was unstable at the company runs since 3 days now without a reboot


So i have the tiny idea it has something to do with Unifi APs.

The unifi controller is a BIG tool, has anyone expierance with Unifi and could give me an advice what settings i could check or change ?

Hi, I am running UNIFI as well but also having problems. Not really with diconnects but with network connection itself. Can not reach unifi connected ESP from other devices within the network. reported an error and it is confirmed be Unifi. but no solution provided yet- looks like it is a problem with their ACL rules in the bridging module.

https://help.ubnt.com/hc/en-us/requests/1870288?page=4

Maybe this is giving some hints....

T

Re: Hardware Watchdog Reboots

Posted: 11 Nov 2019, 12:18
by Sasch600xt
The link is gone ?

oops
The page you were looking for doesn't exist
You may have mistyped the address or the page may have moved

Take me back to the home page

Re: Hardware Watchdog Reboots

Posted: 11 Nov 2019, 13:35
by waspie
I believe that's a request that's personal to his account.

I have Unifi APs as well, I'll be interested to see what this is about...

Re: Hardware Watchdog Reboots

Posted: 12 Nov 2019, 20:31
by riker1
Sasch600xt wrote: 11 Nov 2019, 12:18 The link is gone ?

oops
The page you were looking for doesn't exist
You may have mistyped the address or the page may have moved

Take me back to the home page

sorry, I forgot, yes it is a support ticket work in progress....unfortunaltely unifi is very slow in solving problems.....very sad

Re: Hardware Watchdog Reboots

Posted: 12 Nov 2019, 20:45
by riker1
waspie wrote: 11 Nov 2019, 13:35 I believe that's a request that's personal to his account.

I have Unifi APs as well, I'll be interested to see what this is about...
Hi request is mainly that due to some acl rules clients do not communicate correct.


Had a support engineer with teamviewer logged to record the issue as they did not believe me.

Error is confirmed but not solved.

impact is: can not control esp devices via laptop or mobile logged to the same network ssid

I am really anoyed....I constant try to follow up , but unifi is mainly ignoring this....I expected a better service....support....still hoping so...

will keep you updated.

Re: Hardware Watchdog Reboots

Posted: 12 Nov 2019, 22:31
by TD-er
So the problem occurs when you are connected to the same AP with your laptop and the ESP module?

Re: Hardware Watchdog Reboots

Posted: 13 Nov 2019, 09:55
by riker1
Hi

yes mainly, clients connected to unifi ap do not always can be connected.

even sometimesI can only ping them from the compiuter where unifi controller is hosted.

I also checked from ssh into unifi ap and con not ping other connected devices on different APs.

really strange

still waiting from answer by unifi ....

Re: Hardware Watchdog Reboots

Posted: 13 Nov 2019, 14:21
by TD-er
I know some AP's have a so called 'guest' mode, where the connected clients are also not able to send messages to eachother.
Routing packets between clients directly is then blocked.
Maybe the Unify also has something similar as an option somewhere?

But maybe we can split this Unify AP discussion into a separate topic, as it probably is not really related to the HW Watchdog reboots.

Re: Hardware Watchdog Reboots

Posted: 13 Nov 2019, 15:58
by waspie
got it, well, that's not an issue i've ever had with mine so I'll be content if it stays that way.

thanks for the update.

Re: Hardware Watchdog Reboots

Posted: 21 Nov 2019, 17:42
by Rampler
My 5 cent to the reboot problem:
I have had reboot problems for at least one year !!
Now, after using "ESP82xx Core 2_6_0, NONOS SDK 2.2.2-dev(bb83b9b), LWIP: 2.1.2 PUYA support" my ESP's stays up for now. (5 days)
I use at all 9 ESP8266 with different access points and different sensor's. All 9 ESP's stays up !!!

Re: Hardware Watchdog Reboots

Posted: 22 Nov 2019, 00:56
by TD-er
What build is it? You only list the core library.

Re: Hardware Watchdog Reboots

Posted: 22 Nov 2019, 09:11
by Rampler
ESP_Easy_mega-20191116_normal_core_260_sdk222_alpha_ESP8266_4M1M.bin

Re: Hardware Watchdog Reboots

Posted: 26 Nov 2019, 17:55
by roadsnail
Have just registered here to confirm Rampler's observation.

I too have experienced reboot problems and just assumed my ESP8266s (a mixture of D1 mini clones and a few Sonoff S20 modified with temperature sensor) would continue to have reboot issues.

I have been following this thread with interest for a while and decided to download and upgrade 2 of my D1 minis with some success!
Two of them have been running continuously now for over 27 days. The build is:-
ESP_Easy_mega-20191028_normal_core_260_sdk222_alpha_ESP8266_4M1M
With System libraries:
ESP82xx Core 2.6.0-dev stage, NONOS SDK 2.2.2-dev(38a443e), LWIP: 2.1.2 PUYA support

I have just loaded the same build onto a third D1 mini, and I will monitor progress. It is far too early to tell if the third has also improved with regard to random reboots.

My Sonoff S20's were running (with seemingly random reboots) a build dating back to Mar 15 2019. I could not determine any obvious reason for the reboots. Sometimes they would
run a few days, others a few hours, so I have been living with the issue.

However in order to try to improve reliability of the S20's, I have downloaded and loaded a very recent 1M build
ESP_Easy_mega-20191123_normal_ESP8266_1M.bin
The system libraries are:
ESP82xx Core 2_6_1, NONOS SDK 2.2.2-dev(38a443e), LWIP: 2.1.2 PUYA support

It's far too early to arrive at any conclusions, but the three I have upgraded have at least survived 24 hours with no issues.
(The old build version from erlier this year sometimes rebooted a couple of times a day!).

I will report back as time progresses.

Lastly, thanks to the ESPEasy team and their perseverance in fixing this reboot issue.