Read and process binary data from serial port

[chromo23]

Hello experts,

I recently bought an air quality monitor on Aliexpress because... well, it was cheap and I am curious.
I wanted to know what sensors and WiFi module it is using and if i could "convert" it to ESPEasy.
The sensors are: an unknown PM2.5 sensor, an unknown VOC sensor and an AHT30
The data is gathered by an STC 8G1K08 Microcontroller and send to an Tuya CB3S (pinout: https://docs.libretiny.eu/boards/cb3s/)
Photos: Some other person already disassembled the sensor: https://kaspars.net/blog/tuya-smart-box ... ir-quality

So no easy "ESPEasyfying" (at least not the easy way). I wanted to compare this with my slightly more expensive sensors, but the data from this thing is written to a cloud once an hour, which is quite a long time.
By analyzing the serial output of the STC, I found that it sends binary data (9600 8N1). Each value is 15 bytes long and is sent in chunks of 30 to 60 bytes.
The values are stored in bytes 13 to 14 and are int16. The type of value is stored in byte 7.

Example: 55 AA 03 07 00 08 12 02 00 04 00 00 00 CB F4 (where 00CB = 203 which is the temperature*10)
(helpful site for analyzing: https://www.scadacore.com/tools/program ... converter/)

Question:
Is it possible to gather and convert the data in ESPEasy? I already tried the Serial Proxy and Server plugins. But they seem to ignore the incoming data (They do work in general. I checked that with a different device).

[TD-er]

First of all, the board isn't an Espressif board, so you need to replace it with some other board.

The pinout doesn't seem to be 'signal compatible' with an ESP-12F, as the bottom pins (on the image in the first link) are connected to the flash on the ESP and here they use it for I2C

I don't know about the other pins, have to check.
At least they didn't use a lot of the pads to connect the MCU, so it will be relatively easy to remove the module and wire up an ESP module.

Not sure about the value of the PM2.5 sensor as it doesn't have a fan, so there is no way to tell how much airflow there is inside the air chaimber.
Also 'nice' touch of them to use white for GND, blue for 5V and red for the sensor's TX.
Maybe the "P" (3rd from the left) pin of the sensor is giving some pulse output related to the particle counts?
Still it is hard to correlate it with actual concentrations.

It is a good design to have the temp/hum sensor mounted on a small PCB edge like they did, however I would have chosen to put it at the bottom as the heat generated by the other parts will rise to the top, so the measured temperature will be too high and thus the humidity too low.

I do like the diagonally placed diode as they clearly found another way to cut costs after designing it

[chromo23]

First of all, the board isn't an Espressif board, so you need to replace it with some other board.

I don`t want to replace it because i want to keep the "tuya" functionality.
My goal is to add an ESP which also receives the data sent by the STC.
So the question still is: Is there a way to read and analyze the binary data with the capabilities (like existing plugins) of ESPEasy or do i need to write some extra code?

[chromo23]

r I would have chosen to put it at the bottom as the heat generated by the other parts will rise to the top,

Then you are using it wrong.. you have to hang it form the cable

[TD-er]

Well you could read along with the serial data, however there isn't any plugin yet which can easily decode a serial stream.
You can also try to hook into the I2C data bus, however 2 masters on the same I2C bus might yield strange results.

[TD-er]

Looks like this is the PM2.5 sensor: https://www.made-in-china.com/showroom/ ... ensor.html

https://www.luftmysensor.com/pm-sensor/ ... ensor.html

Or the 'more advanced' GDS07 version differentiating between PM1.0, PM2.5, PM10
https://www.luftmysensor.com/pm-sensor/ ... odule.html

Maybe when looking for the more 'advanced' version, you can find some protocol info.
I think they are all using exactly the same protocol
And it somewhat looks familiar with the 55 AA start. (0101 0101 1010 1010 to signal a wake-up and help detecting baud rate)
Maybe they just use the same protocol as Plantower does for their PMSx003 sensors? Or the SDS011?

[TD-er]

My first guess about the protocol:
55 AA preamble
03 07 = 775 (PM25 ?)
00 08 = 8
12 02 = 4610 (46.1% ?)
00 04 = 4
00 00 = 0
00 CB = 203 (20.3 C ?)
F4 checksum

I guess it would make sense to have some kind of generic plugin, where you can select the decoding schema as done on the link you gave:
https://www.scadacore.com/tools/program ... converter/
And then select the byte offset, nr of bytes and what type it is.
In the formula field you then can correct the values like "%value%/100"

[chromo23]

Maybe when looking for the more 'advanced' version, you can find some protocol info.

I actually don`t care about the different protocols of each sensor as the data of all sensors is gathered and sent by the STC MCU.
And this i already analyzed.

These sensors are not worth getting any extra attention

Well you could read along with the serial data, however there isn't any plugin yet which can easily decode a serial stream.

Could you give me a short overview where and how i would start with this?
What code can i use? Is there a plugin that acts similar and can be reused/rewritten?

That`s how the data chunks look:

Code: Select all

55 AA 03 07 00 08 12 02 00 04 00 00 00 D1 FA 
55 AA 03 07 00 08 13 02 00 04 00 00 02 2D 59

55 AA 03 07 00 08 12 02 00 04 00 00 00 D1 FA 
55 AA 03 07 00 08 13 02 00 04 00 00 02 31 5D 

55 AA 03 07 00 08 14 02 00 04 00 00 00 07 32 
55 AA 03 07 00 08 15 02 00 04 00 00 00 10 3C 
55 AA 03 07 00 08 16 02 00 04 00 00 01 6B 99 
55 AA 03 07 00 08 02 02 00 04 00 00 00 06 1F

55 AA 03 07 00 08 14 02 00 04 00 00 00 04 2F 
55 AA 03 07 00 08 15 02 00 04 00 00 00 0D 39 
55 AA 03 07 00 08 16 02 00 04 00 00 01 69 97 
55 AA 03 07 00 08 02 02 00 04 00 00 00 09 22

55 AA 03 07 00 08 12 02 00 04 00 00 00 D0 F9 
55 AA 03 07 00 08 13 02 00 04 00 00 02 31 5D 

55 AA 03 07 00 08 14 02 00 04 00 00 00 05 30 
55 AA 03 07 00 08 15 02 00 04 00 00 00 0F 3B 
55 AA 03 07 00 08 16 02 00 04 00 00 01 69 97 
55 AA 03 07 00 08 02 02 00 04 00 00 00 0A 23

My first guess about the protocol:
55 AA preamble
03 07 = 775 (PM25 ?)
00 08 = 8
12 02 = 4610 (46.1% ?)
00 04 = 4
00 00 = 0
00 CB = 203 (20.3 C ?)
F4 checksum

Almost. As you can see byte 7 is only changing. 12 = temp, 13 = hum; 14= formaldehyd; 15 = voc, 16 = (fake)co2 and 2 = pm2.5

Edit: i made the new lines and spaces as the data is sent as a stream but i used empty lines to show the delay between the sendings

[chromo23]

55 AA preamble
03 07 = ?(not changing)
00 08 = ?(not changing)
12 02 = Type (temp)
00 04 = 4(not changing)
00 00 = 0(not changing)
00 CB = Value (203)
F4 checksum

[TD-er]

Ah so it is kinda verbose, where the 12 - 16 is some kind of unit of measure enum and each sentence is only a single datapoint.
That makes it extremely simple to do.

You could take a look at the recent Victron plugin.
However that takes in ASCII data.

Or use the MHZ19 plugin, or the SDS011.
Those typically only parse a single sentence and need to detect the start of a message.

Is the checksum already known?

A really simple parser could be something like this:

Code: Select all

bool loop(uint8_t& unitOfMeasure, uint16_t& value) {
  char nextExpectedChar = 0x55;
  uint32_t timeout = millis() + 15; // need to pick some value, 9600 baud is roughly 1 msec per byte
  int offset = -1;
  while (!timeoutReached(timeout)) {
    char c = serial.read();
    if (offset >= 0 && offset < 14) {
      ++offset;
      switch (offset) {
        case 6: { unitOfMeasure = c; break; }
        case 12: { value = c; break; }
        case 13: { value  <<= 8; value += c; return true; }
      }
    } else if (offset >= 15) {
      nextExpectedChar = 0x55;
    } else {
      if (c == nextExpectedChar) {
        if (nextExpectedChar == 0xAA) { offset = 2; }
        nextExpectedChar = (nextExpectedChar == 0x55) ? 0xAA : 0x55;
      }
    }
  }
  return false;
}

Just a quick write down here in the forum edit window, no idea if offsets are correct, nor if braces are balanced...
And this assumes at least 15 bytes are available in the read buffer of the serial port.

[chromo23]

each sentence is only a single datapoint.

Exactly

where the 12 - 16 is some kind of unit of measure enum

See one post before...

Is the checksum already known?

Last byte: CheckSum8 Modulo 256

Just a quick write down here in the forum edit window, no idea if offsets are correct, nor if braces are balanced...

I will have a look. Thank you!

[TD-er]

Maybe it could even be done with less complexity, by simply using a switch statement over the offset and just not increment the offset until the right char was found.

[TD-er]

Simpler approach:

Code: Select all

bool loop(uint8_t& unitOfMeasure, uint16_t& value) {
  uint32_t timeout = millis() + 15; // need to pick some value, 9600 baud is roughly 1 msec per byte
  int offset = 0;
  while (!timeoutReached(timeout)) {
    if (serial.available()) {
      char c = serial.read();
      switch (offset) {
        case 0: if (c == 0x55) { ++offset; ) break;
        case 1: if (c == 0xAA) { ++offset; ) else { offset = 0;} break;
        case 6: { unitOfMeasure = c; break; }
        case 12: { value = c; break; }
        case 13: { value  <<= 8; value += c; return true; }
      }
      if (offset >= 2) ++offset;
      if (offset > 13) offset = 0;
    }
  }
  return false;
}

[chromo23]

Thanks again.
I am not quite sure how to integrate this code? I think I need to use "ESPeasySerial" but otherwise the structure is still a mystery to me.
Can you give me a very simple structural overview of how to make use of ESPeasySerial in the context of writing a plugin for ESPEasy with the given function?

Edit: Just found this on one of the Tuya sites. There are some extensive write-ups of this protocol...

tuya prot.png (85.67 KiB) Viewed 4086 times

https://images.tuyacn.com/smart/aircond ... ya-MCU.pdf

[TD-er]

All plugins for serial devices do have a wrapper for ESPEasySerial.

The P054 DMX512 plugin is probably the smallest plugin in code.

That could be used as a template.

I would check in the PLUGIN_FIFTY_PER_SECOND to see if there is enough data (serial.available()) and then use the loop() code I showed.
You don't even need to check for a timeout then.

So the 50/sec call is then something like this:

Code: Select all

case PLUGIN_FIFTY_PER_SECOND: {
  char c = serial.peek();
  while (c!= 0x55 & serial.available()) {
    serial.read();
    c = serial.peek();
  }
  if (serial.available() >= 14) {
    // parse your data, note that the 0x55 is the first byte present
    ...
    
    switch (unitOfMeasure) {
      case 12: {
        // temp
        UserVar.setFloat(event->TaskIndex, 0, value / 10.0f);       
        break;
      }
      case 13: {
        // Humidity
        UserVar.setFloat(event->TaskIndex, 1, value / 10.0f);       
        break;
      }
      case 14: {
        // formaldehyd
        UserVar.setFloat(event->TaskIndex, 2, value / 10.0f);       
        break;
      }
  }

  break;
}

When this yields a valid packet, you can store the read value in the UserVar struct.
The UnitOfMeasure can then be used to compute a taskvalueindex.

Just realize you only have 4 taskvalues, so you have to make a choice which one to use.
First I would just make this a hard coded choice and later you can add output selectors

[chromo23]

Just realize you only have 4 taskvalues, so you have to make a choice which one to use.

Apart the fact that all the readings (except those probably from the aht30) will only give a direction rather than exact data, it is the readings around the VOC sensor that can be ignored. CO2 is completely useless and looking at the graphs, formaldehyde correlates exactly with the VOC values.

The pinout doesn't seem to be 'signal compatible' with an ESP-12F, as the bottom pins (on the image in the first link) are connected to the flash on the ESP and here they use it for I2C

Actually it would be a perfect candidate as only VCC, GND, TX and 3 other pins are used (which are not at the bottom, one of them is the LED and one is a Button). And only these pins are soldered...

[Ath]

Actually it would be a perfect candidate as only VCC, GND, TX and 3 other pins are used (which are not at the bottom, one of them is the LED and one is a Button). And only these pins are soldered...

Better use an o-scope to check the levels, and use a 5V to 3.3V level converter to protect your ESP

[chromo23]

Better use an o-scope to check the levels, and use a 5V to 3.3V level converter to protect your ESP

It's all 3.3v