mhz-19B "skip unstable" function ?

Moderators: grovkillen, Stuntteam, TD-er

Post Reply
Message
Author
User avatar
pwassink
Normal user
Posts: 60
Joined: 31 Oct 2016, 08:33
Location: Vorden NL

mhz-19B "skip unstable" function ?

#1 Post by pwassink » 11 Nov 2019, 12:50

Device = Esp8266-Nodemcu v2 Running at (official) Git Build: mega-20191108 Normal 4mb

I have a couple of mhz-19A and B type Co2 sensors in use here for quite a while now,
and now i have seen unexpected behavior of a 0-5000 Ppm mhz-19B model here.

It is in a home environment and will vary between 380 and approx 2500 Ppm depending the conditions, number of people in the room and the local weather normally.

The config of the MH17 * sensor has the option "skip unstable" activated but it records and sends to domoticz suddenly peaking (faulty) levels of >34000 Ppm,.
Seeing that, and the fact it is a "digital" sensor i thought ok, this one is clearly confused / damaged and i replaced the sensor with an tested and wellworking MHZ19B variant.
During the initial outdoor re-calibration and the wrong (too high) values i saw several occasions of the ridiculous values again on this completely other sensor.
stats out of the plugin device (still outside the house) config now are : Checksum (pass/fail/reset): 10/6/1
and sometimes it reports on the device-config as an mhz-19A model which it is not ..

Is my interpretation of the configurable option "skip unstable" wrong or is this a bug in that function, maybe ask at config the range of the sensor and redirect values way outside of the range to /dev/null instead of sending them an garble the detail off the Co2 statistics ?

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: mhz-19B "skip unstable" function ?

#2 Post by TD-er » 11 Nov 2019, 13:21

The "skip unstable" is maybe not the most descriptive option.
For the MH-Z19A version, the sensor did report a value named "U" which may be interpreted as "uncertainty".
For U = 64, the value is stable.
For U < 64, the value was suddenly changed (e.g. blowing on the sensor)

The initial implementation of the plugin for this sensor did just ignore these values, but I found out how to still use them.

For the "MH-Z19B" version, this U value was no longer there.


If the sensor does report suddenly high values, do you also see the counter for CRC errors increase (on the plugin page, with more recent builds) ?
How is the sensor positioned?
Can direct sunlight hit the sensor?

Also I would strongly advice not to perform your own calibrations, since the chance you're doing it right is very very small.

User avatar
pwassink
Normal user
Posts: 60
Joined: 31 Oct 2016, 08:33
Location: Vorden NL

Re: mhz-19B "skip unstable" function ?

#3 Post by pwassink » 11 Nov 2019, 14:19

abt that U value included a image of the deviceconfig from the confirmed mhz-19B 0-5000 Ppm variant sensor i use:
Image

Sensor is in a very well ventilated housing, on its side, both openings are free to breath without direct sunlight, the housing is now inside a halfopen bicicle-shed where the wind can go through freely.

It still has a lot of faults it seems : Checksum (pass/fail/reset): 21/19/4 (and that after reboot around 12oclock)
More recent version ? i am running : Binary Filename: ESP_Easy_mega-20191108_normal_ESP8266_4M1M.bin ?? that is the latest

About the "calibration" i mentioned, new and directly out of the box and inside the house most of these mhz-19* sensors will report co2 values much too high.

if you place them outside and not in direct sunlight (which can cause similar high Co2 reports also) they will find their <400 Ppm base level in a couple of hours.
That "calibration" just takes time, and when it has reached that status i place them in a room with another already working sensor to verify the readouts.

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: mhz-19B "skip unstable" function ?

#4 Post by TD-er » 11 Nov 2019, 14:27

Sorry, I meant the "S" value. (I must look more into the code before answering...)

From the factory, these sensors should be calibrated, so there is no us in offsetting the factory calibration by something that's very likely not the right 400 ppm environment.

If you have a lot of read errors, then I guess that may be the reason you're seeing such strange values.
packets with a checksum error from the sensor are ignored, but if they happen in the direction to the ESP, they may also happen during communication to the sensor.
So I guess that's undefined behavior.

The reading of these sensors should be near error-free.
I have 2 of these units here, running for 100'000+ samples with maybe 1 or 2 checksum errors, using software serial.
So if you see a lot more errors, you should first look into the communication to the sensor.


And about the "more recent builds" I meant a build of the last 6 months or so.
I guess I missed the build number you did mention in your initial post.

User avatar
pwassink
Normal user
Posts: 60
Joined: 31 Oct 2016, 08:33
Location: Vorden NL

Re: mhz-19B "skip unstable" function ?

#5 Post by pwassink » 12 Nov 2019, 11:41

Well,

Did some additional reading, looked at the plugin code, enabled syslog functon for this Esp, changed the device-filter from skipped-unstable to Fast_Response and captured this sequence a bit later, look a see: Ppm value measured, filtered and the Ppm value it actually sends out is a bit different ?

# Syslog segment :

<5>1 2019-11-12T11:27:08.730751+01:00 hub08 EspEasy - - - EspEasy: WD : Uptime 1355 ConnectFailures 0 FreeMem 14960 WiFiStatus 3
<5>1 2019-11-12T11:27:11.843780+01:00 hub08 EspEasy - - - EspEasy: MHZ19: Raw PPM: 418 Filtered PPM value: 427 Temp/S/U values: 11/0/34433.00
<5>1 2019-11-12T11:27:11.843780+01:00 hub08 EspEasy - - - EspEasy: EVENT: Co2-Hub8#PPM=427.00
<5>1 2019-11-12T11:27:11.844302+01:00 hub08 EspEasy - - - EspEasy: ACT : GPIO,2,1
<5>1 2019-11-12T11:27:11.844302+01:00 hub08 EspEasy - - - EspEasy: SW : GPIO 2 Set to 1
<5>1 2019-11-12T11:27:11.844858+01:00 hub08 EspEasy - - - EspEasy: ACT : GPIO,15,0
<5>1 2019-11-12T11:27:11.844858+01:00 hub08 EspEasy - - - EspEasy: SW : GPIO 15 Set to 0
<5>1 2019-11-12T11:27:11.845408+01:00 hub08 EspEasy - - - EspEasy: ACT : GPIO,13,0
<5>1 2019-11-12T11:27:11.845408+01:00 hub08 EspEasy - - - EspEasy: SW : GPIO 13 Set to 0
<5>1 2019-11-12T11:27:11.845921+01:00 hub08 EspEasy - - - EspEasy: EVENT: Co2-Hub8#Temperature=11.00
<5>1 2019-11-12T11:27:11.869871+01:00 hub08 EspEasy - - - EspEasy: EVENT: Co2-Hub8#U=34433.00
<5>1 2019-11-12T11:27:11.894054+01:00 hub08 EspEasy - - - EspEasy: Domoticz: Sensortype: 6 idx: 102 values: 427.00;11.00;34433.00

A bit further in the logging i've found several Unknown responses from the Mhz-!9 and then i found this :

# Syslog segment :
5>1 2019-11-12T11:27:11.843780+01:00 hub08 EspEasy - - - EspEasy: MHZ19: Raw PPM: 418 Filtered PPM value: 427 Temp/S/U values: 11/0/34433.00
<5>1 2019-11-12T11:27:11.843780+01:00 hub08 EspEasy - - - EspEasy: EVENT: Co2-Hub8#PPM=427.00

A bit further down the syslog

# Syslog segment :
5>1 2019-11-12T12:00:08.791827+01:00 hub08 EspEasy - - - EspEasy: WD : Uptime 1388 ConnectFailures 0 FreeMem 15008 WiFiStatus 3
<5>1 2019-11-12T12:00:11.788813+01:00 hub08 EspEasy - - - EspEasy: MHZ19: Raw PPM: 33200 Filtered PPM value: 16813 Temp/S/U values: 11/64/2593.00
<5>1 2019-11-12T12:00:11.788813+01:00 hub08 EspEasy - - - EspEasy: EVENT: Co2-Hub8#PPM=16813.00


It seems that the Esp is sending out the U value in this measurement as being the PPM value once in a while ? That number is the one that's got mixed up and send into Domoticz in my humble opinion .

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: mhz-19B "skip unstable" function ?

#6 Post by TD-er » 12 Nov 2019, 18:16

Hmm, looks like it indeed.
But what I don't understand is how it does accept the packet with such a mixed up value.

The filtering is a rolling average.
For example old value = 100, new value = 120.
The filter does take the difference (120 - 100 = 20) and divides that by some value (e.g. 5).
So the new value to report is then: (100 + (120-100)/5) = 104.
With a fast response the divide value is lower (e.g. 1 or 2) and with a slow response the value is is larger (e.g. 10 or 20)

Does the ESP node have other active sensors?
You may want to look at the timingstats page to see if there are some other plugins or controllers which need quite a long time.

User avatar
pwassink
Normal user
Posts: 60
Joined: 31 Oct 2016, 08:33
Location: Vorden NL

Re: mhz-19B "skip unstable" function ?

#7 Post by pwassink » 12 Nov 2019, 19:11

Your idea is something that crossed my mind also, mainly because this is the only esp that suffers from this problem.

This esp has a Si7021 for temp/hum, a BH1750 for Lux, the MHZ-19B for Co2 and a Framed Oled with 4 lines of data in "slow" mode.
Also it has 4 "remote p2p" devices from which some values are also on the Oled (outside temp) .

With the old rules1 setting ive created a "trafficlight" which uses a couple of if > nummeric-level statements to switch the three Gpio Leds according to different co2-levels it will light up green, yellow or red ad predefined levels.

I'am not sure how to interpret the timing page exactly, will add e screen capture of it, and place it below.

Image

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: mhz-19B "skip unstable" function ?

#8 Post by TD-er » 12 Nov 2019, 19:14

700 msec for the OLED framed is way too long.
Can you test with much faster scrolling?
Still the packets should not be accepted when the checksum is incorrect. This means that either the checksum is correct by accident, but that should not be so likely.

User avatar
pwassink
Normal user
Posts: 60
Joined: 31 Oct 2016, 08:33
Location: Vorden NL

Re: mhz-19B "skip unstable" function ?

#9 Post by pwassink » 12 Nov 2019, 19:29

Correction, Oled has 2 x 4 lines of data to display and 4 lines in between are empty - not used..

Changed the scrollrate to Fast at your request now, unit is back on the bench now, waiting for the first correct Co2-value,
Timings page is now like this : Oled values seems decreased a bit.

Image

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: mhz-19B "skip unstable" function ?

#10 Post by TD-er » 12 Nov 2019, 19:34

Stil 300+ msec on average is a lot.
Is this the fastest scrolling?
If so, then we must for sure have a good look at that plugin.
Someone is already having a look at the plugin: https://github.com/letscontrolit/ESPEasy/pull/2706
I will test that PR later this evening, to see how it may have been improved.

User avatar
pwassink
Normal user
Posts: 60
Joined: 31 Oct 2016, 08:33
Location: Vorden NL

Re: mhz-19B "skip unstable" function ?

#11 Post by pwassink » 12 Nov 2019, 19:44

That last screen print was running on Oled mode "very-fast", changed that to "Instant" now which seems the fastest ?

Oled timing-page value now :

P_36_Display - OLED SSD1306/SH1106 Framed READ 10 0.02 174.722 290.892 402.140

So still around 400 Ms ?

User avatar
pwassink
Normal user
Posts: 60
Joined: 31 Oct 2016, 08:33
Location: Vorden NL

Re: mhz-19B "skip unstable" function ?

#12 Post by pwassink » 13 Nov 2019, 09:36

Updated this Esp hub 08 to the new release with Oled fixes #2706 this morning

P_36_Display - OLED SSD1306/SH1106 Framed READ 11 0.02 138.500 142.876 148.757
P_36_Display - OLED SSD1306/SH1106 Framed ONCE_A_SECOND 655 1.00 0.592 0.952 3.085
P_36_Display - OLED SSD1306/SH1106 Framed TEN_PER_SECOND 6526 9.97 0.028 0.030 0.064
P_36_Display - OLED SSD1306/SH1106 Framed EVENT_OUT 33 0.05 0.034 0.037 0.049
P_36_Display - OLED SSD1306/SH1106 Framed UDP_IN 32 0.05 0.011 0.015 0.020
P_36_Display - OLED SSD1306/SH1106 Framed FIFTY_PER_SECOND 32364 49.46 0.015 0.021 0.050

Seems faster, reboot after flash was faster also, no browser timeouts when the update finished/rebooted for the first time, saw that on several nodes .

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: mhz-19B "skip unstable" function ?

#13 Post by TD-er » 13 Nov 2019, 14:18

Still, I could add a check for the range of the read PPM value.
Not sure if I can read the currently active range of the sensor, but I guess we can deduct it from the range of the U value whether a value is plausible.

User avatar
pwassink
Normal user
Posts: 60
Joined: 31 Oct 2016, 08:33
Location: Vorden NL

Re: mhz-19B "skip unstable" function ?

#14 Post by pwassink » 14 Nov 2019, 12:02

Timing Oled seems better with these fixes , see some improvement on the bogus-values but still get them sent to domoticz.

Counter of the Esp hub08 running for >27 hours now
Software version : ESP_Easy_mega-20191113_normal_ESP8266_4M1M.bin
Checksum (pass/fail/reset): 30/28/40

Looked at the syslog of this esp a bit more closer, grepped around in it and found some possible clues.

Still have many unknown response entries from the Mhz-19B readouts, most of them go wrong, just a sample :

<5>1 2019-11-14T08:37:05.434912+01:00 hub08 EspEasy - - - EspEasy: MHZ19: Unknown response: ff 86 3 1c c6 80 45 41 cf
<5>1 2019-11-14T08:38:05.432706+01:00 hub08 EspEasy - - - EspEasy: MHZ19: Unknown response: ff a6 3 13 56 0 4d 1 dc
<5>1 2019-11-14T08:39:05.430377+01:00 hub08 EspEasy - - - EspEasy: MHZ19: Unknown response: ff 86 13 1d 4e 8 45 5 ce
<5>1 2019-11-14T08:40:05.440594+01:00 hub08 EspEasy - - - EspEasy: MHZ19: Unknown response: ff 86 23 3b 46 10 45 9 d8
<5>1 2019-11-14T08:41:05.428169+01:00 hub08 EspEasy - - - EspEasy: MHZ19: Unknown response: ff 96 13 36 4e 0 45 5 b5

And in counts :
[/share/MD0_DATA/Qbackup] # cat syslogmessages |grep 'MHZ19: Unknown response:' |wc -l
2681
This happens quite often as you can see :-)

Find a quite irregular pattern in which the Mhz-19b gets (re-)initialised many times :

[/share/MD0_DATA/Qbackup] # cat syslogmessages |grep 'MHZ19: Init OK' |wc -l
78

the reports that go > domoticz are irregular, measurement readout frequency for the Esp of the sensor is 1 minute,
in which time window the sensor itself does several red blinks and mesurements, when "marked correctly" & processed and send trough
that is quite irregular, sometimes they are many minutes apart. Bogus values still exist, those lines extracted look like this (happened 12 times) :

<5>1 2019-11-12T12:44:53.477929+01:00 hub08 EspEasy - - - EspEasy: Domoticz: Sensortype: 6 idx: 102 values: 8627.00;74.00;11809.00
<5>1 2019-11-12T13:43:08.480430+01:00 hub08 EspEasy - - - EspEasy: Domoticz: Sensortype: 6 idx: 102 values: 8618.00;11.00;21505.00
<5>1 2019-11-12T14:48:23.651471+01:00 hub08 EspEasy - - - EspEasy: Domoticz: Sensortype: 6 idx: 102 values: 8603.00;10.00;6689.00
<5>1 2019-11-12T18:15:38.948604+01:00 hub08 EspEasy - - - EspEasy: Domoticz: Sensortype: 6 idx: 102 values: 8612.00;9.00;28417.00
<5>1 2019-11-13T02:58:00.603700+01:00 hub08 EspEasy - - - EspEasy: Domoticz: Sensortype: 6 idx: 102 values: 9600.00;30.00;9760.00
<5>1 2019-11-13T03:16:00.585207+01:00 hub08 EspEasy - - - EspEasy: Domoticz: Sensortype: 6 idx: 102 values: 9397.00;29.00;26368.00
<5>1 2019-11-13T03:40:15.604311+01:00 hub08 EspEasy - - - EspEasy: Domoticz: Sensortype: 6 idx: 102 values: 9318.00;29.00;10784.00
<5>1 2019-11-13T05:22:45.796202+01:00 hub08 EspEasy - - - EspEasy: Domoticz: Sensortype: 6 idx: 102 values: 9018.00;28.00;13344.00
<5>1 2019-11-13T05:36:00.828635+01:00 hub08 EspEasy - - - EspEasy: Domoticz: Sensortype: 6 idx: 102 values: 8961.00;28.00;13600.00
<5>1 2019-11-13T07:08:30.995012+01:00 hub08 EspEasy - - - EspEasy: Domoticz: Sensortype: 6 idx: 102 values: 8881.00;27.00;15904.00
<5>1 2019-11-13T10:21:37.480815+01:00 hub08 EspEasy - - - EspEasy: Domoticz: Sensortype: 6 idx: 102 values: 9917.00;32.00;28960.00

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: mhz-19B "skip unstable" function ?

#15 Post by TD-er » 14 Nov 2019, 12:11

Maybe the reset of the stats is also a bit confusing:

Code: Select all

Checksum (pass/fail/reset): 30/28/40
The pass/fail counter should not be reset when the sensor is reset.
It does hide the true number of failures.

I think something is not right at all with this sensor's communications.
Maybe you can test it for an hour or so with only the CO2 sensor enabled, just to see whether proper communication is possible?
How long is the connection between the ESP and the CO2 sensor?
Does it run on 5V or 3.3V?
When using wires, try to twist all wires (GND RX TX) to make sure it will not pick up noise.

User avatar
pwassink
Normal user
Posts: 60
Joined: 31 Oct 2016, 08:33
Location: Vorden NL

Re: mhz-19B "skip unstable" function ?

#16 Post by pwassink » 14 Nov 2019, 17:29

the reset of the stats is also a bit confusing ? yes it is ..

Sensor-fault , that is why i did replace this sensor to begin with, problem stays so it has to be the combination of components and must be Esp or Sw related.

Had some thoughts on this again and for a while, dit produce > 20 esp-sensor boxes exactly like this one (ok except the oled/leds) but same setup, wire routing, components, short wiring in the identical plastic housing.
none of them did suffer these same issues running on a bit older software (out of my head they were all running the mid march 2018 mega-version ).

I will make some checks, and look for a reason for this distortions, (regulator oscillations, ripple, noise, missing of broken ceramic suppressing / decoupling capacitors etc later, might also be an outside cause or noise from RF or field strength related, will do some additional tests on that point on this unit ..

When the box is on the bench i will takeout the nodemcu and Co2-sensor and wire them directly and test with new (short) wiring

Unit is wired with short & twisted wiring, my skillset is mainly based on field RF/HF engineering, practically seen i will add decoupling capacitors on almost anything before even connecting the sensor itself, sort of second nature :-)

Another thing which is even more confusing for me : This same unit was running for more than a day outside in the shed runnimng on a wellbuilt, clean and completely different Usb power supply outside the house and quite far away from most man made noises i just remember.

Anyway, will start measuring the power supply and look at the data lines for distortion, regulator or other noises and or rf of field related noises anyway just to be sure ..

Will be continued ..

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: mhz-19B "skip unstable" function ?

#17 Post by TD-er » 14 Nov 2019, 22:30

You can also try to test with an old firmware version, just to make sure it isn't a software thing.

User avatar
pwassink
Normal user
Posts: 60
Joined: 31 Oct 2016, 08:33
Location: Vorden NL

Re: mhz-19B "skip unstable" function ?

#18 Post by pwassink » 16 Nov 2019, 13:11

Did not made any analysis of the esp and noise yet.

While upgrading to mega *1116 for the rest of the test net i did downgrade the fauling Hub08
to the version : ESP_Easy_mega-20190805_normal_core_241_ESP8266_4M.bin
Which was the oldest i had online available on this desktop, lets see if that changes anything

just to be sure

User avatar
pwassink
Normal user
Posts: 60
Joined: 31 Oct 2016, 08:33
Location: Vorden NL

Re: mhz-19B "skip unstable" function ?

#19 Post by pwassink » 18 Nov 2019, 13:13

Just looked at the syslogdata of my hub08 esp , since last Saturday running on
Binary Filename: ESP_Easy_mega-20190805_normal_core_241_ESP8266_4M.bin

No glitches have occured since the downgrade, works as a charm and as designed ..

Will continue to monitor this Esp, but this is an outcome i did not expect to happen ..

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: mhz-19B "skip unstable" function ?

#20 Post by TD-er » 18 Nov 2019, 15:01

Can you also test with a core 2.5 (or 2.6?) build from around that time?

User avatar
pwassink
Normal user
Posts: 60
Joined: 31 Oct 2016, 08:33
Location: Vorden NL

Re: mhz-19B "skip unstable" function ?

#21 Post by pwassink » 18 Nov 2019, 19:37

I can, after 48 hours no faults appeared on hub08.

Now it is running Binary Filename:⋄ ESP_Easy_mega-20190805_normal_core_252_ESP8266_4M.bin

And well see how this one behaves, also for approx 48 Hours ?

i think we have found something, at 21:50 the first unreal value was sent
And it did continue giving faulty values over and over during this day
so there is a indication that C02 behaviour is changing witht used core ..
Last edited by pwassink on 20 Nov 2019, 08:23, edited 1 time in total.

User avatar
pwassink
Normal user
Posts: 60
Joined: 31 Oct 2016, 08:33
Location: Vorden NL

Re: mhz-19B "skip unstable" function ?

#22 Post by pwassink » 20 Nov 2019, 08:23

Updated to version :

Binary Filename: ESP_Easy_mega-20190805_test_core_260_sdk2_alpha_ESP8266_4M.bin

Lets find out how that is behaving then

Has been running since yesterday evening, also same issues, already more than 15 occurances of "out of sensor spec" co2 values

So it seems core related ?

cant tell why it is only occurring at one node though, will see if i can build a clone with my available parts and see if that one behaves equally ?

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: mhz-19B "skip unstable" function ?

#23 Post by TD-er » 20 Nov 2019, 09:39

It looks like it is a bit more sloppy in timings.
If the signal level of that specific setup is a bit less optimal, the these issues have a higher chance to happen.
For example if the shape of the digital signal is not square, but more rounded, then accurate timing is more important.
This can be caused by higher capacitance over the data lines.
For example longer cables, or data cables tighter together, or solder joints almost touching each other.
Maybe the used ESP is also a bit different (solder joints almost touching?)

User avatar
pwassink
Normal user
Posts: 60
Joined: 31 Oct 2016, 08:33
Location: Vorden NL

Re: mhz-19B "skip unstable" function ?

#24 Post by pwassink » 21 Nov 2019, 18:01

if so, be aware that the setup itself is stable, even in in a housing, the usb (powersupply) cable is fastened and not wiggeling.

So all wiring, solderjoints, sensors, esp itself is fixed, does not change, cant vary in any way.

Still there is a sw (core) related and reproducible fault, it had ran faultless since midnight, when i put it back on Binary Filename: ESP_Easy_mega-20190805_normal_core_241_ESP8266_4M.bin

i've put the 1113 version back on it this esp via the airwaves this morning (without any mechanical issues) and is has been sending crappy measurements again and again.

Will have a closer look later on, but i think there is at least something weird based on the core-software too

Post Reply

Who is online

Users browsing this forum: No registered users and 29 guests