Serious instability using MQTT Import

Moderators: grovkillen, Stuntteam, TD-er

Post Reply
Message
Author
Wiki
Normal user
Posts: 413
Joined: 23 Apr 2018, 17:55
Location: Germany

Serious instability using MQTT Import

#1 Post by Wiki » 16 Mar 2020, 23:48

Since yesterday afternonn I am importing eight datasets over MQTT Import. Since the moment I actvated the MQTT Import the device - Wemos D1, ESP_Easy_mega-20191208_dev_ESP8266_4M1M.bin - reboots 8-15 times/day.

Code: Select all

Boot:	Manual reboot (18)
Reset Reason:	Exception
Last Task:	Const Interval timer, id: 2
SW WD count:	0
The device has had an uptime of >50 days before reboot, last reboot before has been a power cycle due to a problem with the Nextion. Any ideas?

Plugins (DS18B20 deactivated, broken cable):
MQTTImport01.JPG
MQTTImport01.JPG (95.63 KiB) Viewed 9738 times
MQTTImport0.JPG
MQTTImport0.JPG (79.65 KiB) Viewed 9738 times
MQTTImport1.JPG
MQTTImport1.JPG (76.62 KiB) Viewed 9738 times
MQTTImport2.JPG
MQTTImport2.JPG (74.28 KiB) Viewed 9738 times

Code: Select all

pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Serious instability using MQTT Import

#2 Post by TD-er » 17 Mar 2020, 00:44

Those exceptions look like you're running out of memory. Is that possible?
You also have some controllers. Maybe lower the queue depth on them?

Wiki
Normal user
Posts: 413
Joined: 23 Apr 2018, 17:55
Location: Germany

Re: Serious instability using MQTT Import

#3 Post by Wiki » 17 Mar 2020, 00:58

I wasn't aware to have a memory problem:

Code: Select all

Local Time:	2020-03-17 01:00:30
Uptime:	0 days 0 hours 15 minutes
Load:	29.60% (LC=2588)
CPU Eco Mode:	false
Free Mem:	11096 (4176 - duringDataTX)
Free Stack:	3600 (976 - sendContentBlocking)
Heap Max Free Block:	8368
Heap Fragmentation:	25%
Boot:	Manual reboot (20)
Reset Reason:	Exception
Last Task:	Task Device timer, id: 1
SW WD count:	0
To give a reasonable comment I am too noob.

Controllers:
Domoticz MQTT
Generice HTTP

I will lower the queue depth and will report.

Code: Select all

pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Serious instability using MQTT Import

#4 Post by TD-er » 17 Mar 2020, 01:14

Code: Select all

11096 (4176 - duringDataTX)
That's quite low (4k free) and that means the largest block of free memory is probably a lot smaller.
So allocating a string to send is probably failing in a not so elegant way.

Wiki
Normal user
Posts: 413
Joined: 23 Apr 2018, 17:55
Location: Germany

Re: Serious instability using MQTT Import

#5 Post by Wiki » 17 Mar 2020, 01:39

Eehm, and what can I do to allocate more free memory? I'm lost at the moment.....

Code: Select all

pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $

Wiki
Normal user
Posts: 413
Joined: 23 Apr 2018, 17:55
Location: Germany

Re: Serious instability using MQTT Import

#6 Post by Wiki » 17 Mar 2020, 01:44

Disabled one of the two MQTT Import plugins:

Code: Select all

Free Mem:	11200 (7656 - sendContentBlocking)
Free Stack:	3680 (1312 - sendContentBlocking)

Code: Select all

pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Serious instability using MQTT Import

#7 Post by TD-er » 17 Mar 2020, 01:46

See also the description of the controller parameters here: https://espeasy.readthedocs.io/en/lates ... parameters

Things that may keep long strings in the buffers/queue are:
- Large number of items in the queue (lower is better for memory, but less than number of expected messages is probably causing lost messages)
- Minimum send interval too large (10 msec is fine for most hosts)
- Too large timeout (100 msec is probably fine for local network, only use longer for online services)


Read (send) less often or maybe use a relative prime interval between sources of data (e.g. 3 / 5 / 7 as ratio) and low number of items in the queue.
This will cause some messages to get lost, but due to their relative prime interval they will eventually send data at a moment no other is sending so then you can get away with a shorter queue.

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Serious instability using MQTT Import

#8 Post by TD-er » 17 Mar 2020, 01:50

Wiki wrote: 17 Mar 2020, 01:44 Disabled one of the two MQTT Import plugins:

Code: Select all

Free Mem:	11200 (7656 - sendContentBlocking)
Free Stack:	3680 (1312 - sendContentBlocking)
That's a lot better in lowest free mem (7656 bytes free)
And it is also been when sending data of the web page, which you would normally not do when the node is running.

Current implementation of the MQTT import uses its own PubSubClient.
I don't know why it was done like this, but I think all MQTT thingies (MQTT import, MQTT controllers) should be able to use the same PubSubClient and thus save a bit (more than a bit :) ) of resources.

Wiki
Normal user
Posts: 413
Joined: 23 Apr 2018, 17:55
Location: Germany

Re: Serious instability using MQTT Import

#9 Post by Wiki » 17 Mar 2020, 01:55

Thank you for your fast reaction. Reconfiguered both controllers to max queue depth 3 and 10ms Minimu Send Interval.

I will report....

Code: Select all

pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Serious instability using MQTT Import

#10 Post by TD-er » 17 Mar 2020, 01:58

Well don't expect any reply from me soon as I will now go look for my bed as it is 2am here in the Netherlands and my personal alarm clock is probably awake in 4 - 5 hours from now.

Wiki
Normal user
Posts: 413
Joined: 23 Apr 2018, 17:55
Location: Germany

Re: Serious instability using MQTT Import

#11 Post by Wiki » 17 Mar 2020, 01:59

And with the changes, both tasks MQTT Import reenabled:

Code: Select all

Free Mem:	10512 (7096 - sendContentBlocking)
Free Stack:	3632 (1104 - sendContentBlocking)
Jo, looks much better, at least if you know where to look at, and seems to stay at this value. These data were more or less a mysterium for me - up to now.

Code: Select all

pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $

Wiki
Normal user
Posts: 413
Joined: 23 Apr 2018, 17:55
Location: Germany

Re: Serious instability using MQTT Import

#12 Post by Wiki » 17 Mar 2020, 02:00

good n8

Code: Select all

pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $

Wiki
Normal user
Posts: 413
Joined: 23 Apr 2018, 17:55
Location: Germany

Re: Serious instability using MQTT Import

#13 Post by Wiki » 17 Mar 2020, 15:11

Preliminary report:
The values seem to stay stable on

Code: Select all

Free Mem:	9864 (5064 - ruleMatch2)
Free Stack:	3280 (864 - sendContentBlocking)
Max queue depth reduced to 3 in Domoticz MQTt controller as well as in Generic HTTP.

Should I open an issue as reminder for future releases?

Code: Select all

pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Serious instability using MQTT Import

#14 Post by TD-er » 17 Mar 2020, 15:14

Well, the re-use of the PubSubClient in all is already on my todo list.
So you don't have to add a new issue for it as I probably can't find it to close when I fix it :)

Wiki
Normal user
Posts: 413
Joined: 23 Apr 2018, 17:55
Location: Germany

Re: Serious instability using MQTT Import

#15 Post by Wiki » 22 Mar 2020, 01:20

OK, stability seems to be back again with the change of the settings as posted before:
Stabilität1.JPG
Stabilität1.JPG (30.62 KiB) Viewed 9451 times
But the Mem info still makes me nervous:
Stabilität2.JPG
Stabilität2.JPG (30.26 KiB) Viewed 9451 times
I will see and report.

Code: Select all

pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Serious instability using MQTT Import

#16 Post by TD-er » 22 Mar 2020, 02:10

Do you have log (serial log) enabled?
If so, please try to disable serial log.

Wiki
Normal user
Posts: 413
Joined: 23 Apr 2018, 17:55
Location: Germany

Re: Serious instability using MQTT Import

#17 Post by Wiki » 22 Mar 2020, 10:59

Nope, the Nextion is connected to the serial port

Code: Select all

pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Serious instability using MQTT Import

#18 Post by TD-er » 22 Mar 2020, 11:26

Well then I don't think you can do a lot until the PubSubClient object is shared among all MQTT instances (controller/MQTTimport)

This will then also remove the limit of a single active MQTT controller. (as long as you connect to the same broker)

Wiki
Normal user
Posts: 413
Joined: 23 Apr 2018, 17:55
Location: Germany

Re: Serious instability using MQTT Import

#19 Post by Wiki » 22 Mar 2020, 17:35

Well I can tell you that I am so lucky to have in my smarthome network some devices running a firmware called ESPEasy Mega. They are pretty powerful concerning communication with each other. I chose one of them to overtake one of the two MQTT Import and forced it to directly send the data via http onto the Nextion display.

Done, I think, and so I can wait for your update of the various MQTT clients.

I know, I could feed the Nextion from my Domoticz, but I try to be as indipendent as possible from any server device because replacing a Wemos D1 is pretty much easier than to setup a toasted server.

Thank you anyway, you helped me a lot.

Code: Select all

pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $

Post Reply

Who is online

Users browsing this forum: No registered users and 25 guests