ESP Easy Ser2net no message in MQTT (P020)

Message

fredduin · #1 Post by **fredduin** » 25 Aug 2022, 14:49

Hello,

I want to use MQTT from serial P1 slimmemeter.
Ser2net is working, I see the message/characters in telnet
For test I send messages/character with Putty/Com port

Other plugin is working with MQTT

Binary Filename:⋄ ESP_Easy_mega_20220809_normal_ESP8266_4M1M
Controller Plugin 005: Home Assistant (openHAB) MQTT

Syslog
Aug 25 12:41:28 ESP-Easy-P1-20 EspEasy: Ser2Net : data send!
Aug 25 12:41:28 ESP-Easy-P1-20 EspEasy: Ser2Net : data send!
Aug 25 12:41:28 ESP-Easy-P1-20 EspEasy: Ser2Net : data send!

Is this an issue?

Many thanks

Ath · #2 Post by **Ath** » 25 Aug 2022, 15:21

What MQTT controller have you configured in ESPEasy?

fredduin · #3 Post by **fredduin** » 25 Aug 2022, 16:32

Controller Plugin 005: Home Assistant (openHAB) MQTT

Config

TD-er · #4 Post by **TD-er** » 25 Aug 2022, 17:00

Are you sure the ESP is receiving the data from the P1 port?
I know there was something with P1 like it was an inverted signal?

There is also P044, which is essentially the Ser2Net plugin, but then tailored for P1.
Maybe you want to use that one for initial testing to see it is working?

Also which MQTT topics are you expecting the messages to be sent to?
Maybe also check with a tool like MQTT Explorer to see if it might be sent to an unexpected topic?
As far as I know, this plugin was not (yet) adapted to support sending task values of type String.

Ath · #5 Post by **Ath** » 25 Aug 2022, 17:04

AFAICS, HA has support for the network interface/protocol provided by P1 monitor: https://www.home-assistant.io/integrations/p1_monitor/
I'm not a HA user, so haven't tried this at home...

Ath · #6 Post by **Ath** » 25 Aug 2022, 17:06

This project might also help: https://infi.nl/nieuws/hobbyproject-sli ... aar-later/ (I'm just guessing you're able to read Dutch

)

fredduin · #7 Post by **fredduin** » 25 Aug 2022, 18:22

@TD-er
For test I do not send data from P1 (I am aware about inverting signal)
I send data from a USB-COM port for test.

The P044 has no "Send to controller" = no MQTT

I have a Tasmota P1 sensor working, but I prefer ESP-Easy

I am working on a small local project measuring voltage and send the data (MQTT) to a central server
For 1 Fase I use a smart adapter with Tasmota and MQTT. (works perfect)
For 3 fase I am looking for a P1 sensor with MQTT (simple cheap ESP12/ESP32/NodeMCU)

@Ath see my PCB, it can also be also used for SDS011 (fijnstofmeter + temp .. etc more than 45 sensor are build this year Amersfoort/Leusden/Woudenberg)

TD-er · #8 Post by **TD-er** » 25 Aug 2022, 18:58

What does Tasmota send?
Is it the complete P1 telegram?

If so, then maybe you could look into the SerialProxy plugin.
This one does send the full string to a MQTT controller.

fredduin · #9 Post by **fredduin** » 26 Aug 2022, 12:00

Test message Hello from...

TD-er · #10 Post by **TD-er** » 26 Aug 2022, 14:52

OK, so Tasmota does wrap the entire string into a JSON message.

Can you have a look at the SerialProxy plugin, to see if that's working for you?

I did have a quick look at the code for the P1 plugin and I think we can make a small modification to it to allow to send the received data as an eventvalue.
Or send it to a MQTT controller.
But it also seems like it offers a socket for some client to connect to and if it isn't connected it won't process the data from the serial port but just discard it.
So maybe it needs some slightly more changes to send it as a string to a controller...

fredduin · #11 Post by **fredduin** » 26 Aug 2022, 16:30

I have installed the SerialProxy, it is not working yet
Need some RegEx rules!?

The new P020 Serial Server plugin has "Process events without client" feature.

Ath · #12 Post by **Ath** » 26 Aug 2022, 16:50

When you enable that setting, and set Event processing to Generic, you will see events like !Serial#(your incoming message is here) that you can handle in the Rules:

Code: Select all

on !Serial* do
  Publish,whatever/topic/you/need/%eventvalue0%
endon

This will include the entire string received, from %eventvalue0%

NB: Untested...

TD-er · #13 Post by **TD-er** » 26 Aug 2022, 16:52

SerialProxy is a bit special as it will send to the first enabled MQTT controller.

N.B. it requires a line termination character like \n
If there is none from the P1 meter, then it will never end waiting for a complete telegram.
Not sure if the P1 telegram has such a line ending.

fredduin · #14 Post by **fredduin** » 26 Aug 2022, 18:16

I have data in MQTT (plugin P087)

fredduin · #15 Post by **fredduin** » 28 Aug 2022, 17:00

I get data from the P1 meter.(plugin P087 SerialProxy)
After a few strings from the P1 meter data is missing in MQTT Explorer

On the Ser2Net you can change the RX buffer to 1024
The P1 test message is 883 characters /38 Lines

No errors in syslog

Ath · #16 Post by **Ath** » 28 Aug 2022, 17:17

fredduin wrote: ↑28 Aug 2022, 17:00 On the Ser2Net you can change the RX buffer to 1024
The P1 test message is 883 characters /38 Lines

No errors in syslog

Does that config change fix the issue, or is data still missing?

fredduin · #17 Post by **fredduin** » 28 Aug 2022, 17:54

Yes,it fix the data missing on Ser2Net (not MQTT)
In telnet I see the last line when buffer is 1024
On buffer is 256 it stops at line 10

TD-er · #18 Post by **TD-er** » 28 Aug 2022, 22:10

Could be the buffers are not that large on the SerialProxy.
Will look at the code to see if this might be the issue.

TD-er · #19 Post by **TD-er** » 28 Aug 2022, 22:14

Yep, it seems to be set to something like 500 bytes.

fredduin · #20 Post by **fredduin** » 29 Aug 2022, 15:33

Is this a simple fix to change?

Ath · #21 Post by **Ath** » 29 Aug 2022, 20:02

There are at least these 2 possible approaches here: (1 should be enough too fix this though)
a) Enable receiving data, and generate events, for P044 "Communication - P1 WiFi Gateway", as that already correctly handles receiving P1 data, similar to P020 "Communication - Serial Server"
b) Add a size the buffer on P087 "Communication - Serial Proxy", but that would probably need some fine-tuning to properly handle the P1 data 'quirks'

I prefer to go for option a) as that is already working correctly with P1 data when using a network connection.

fredduin · #22 Post by **fredduin** » 30 Aug 2022, 17:26

Ok, who makes a Pull request?

Ath · #23 Post by **Ath** » 31 Aug 2022, 20:05

I'll do the PR, I did the same feature for P020, can do that again.

TD-er · #24 Post by **TD-er** » 31 Aug 2022, 21:29

Ath wrote: ↑31 Aug 2022, 20:05 I'll do the PR, I did the same feature for P020, can do that again.

<sniff><sniff><sniff>
Hmm do I smell some code duplication?
<sniff><sniff><sniff>

Ath · #25 Post by **Ath** » 31 Aug 2022, 22:21

TD-er wrote: ↑31 Aug 2022, 21:29 Hmm do I smell some code duplication?

I'll see if I can merge the plugins so we can deprecate/sunset P044.

TD-er · #26 Post by **TD-er** » 31 Aug 2022, 23:03

Or make a common plugin struct which can have a validator object as optional member.
Hint... that could be re-used even more.

fredduin · #27 Post by **fredduin** » 11 Oct 2022, 12:17

Is there a PR?

Ath · #28 Post by **Ath** » 11 Oct 2022, 12:53

I've merged the features of P044 into P020, in this PR, while still keeping P044 available as a plugin (it does convert its settings, so can't revert...).
There is probably 1 feature still missing though, for P1 data processing; an event is generated, but that doesn't include the data (yet), as the message can be quite large (buffer is 2kB), so could be unstable in low memory conditions. I'll see if I can add an option to enable that, though.

Ath · #29 Post by **Ath** » 11 Oct 2022, 21:33

I've added an option in the plugin to pass the data in the event, so you could use a rule like this to publish to a MQTT topic:

Code: Select all

on SerialServer#Data* do // Include * to process string values
  Publish,"your/topic/goes here",`%eventvalue0%` // Wrap in quotes because of possible separators in data
endon

giorgino · #30 Post by **giorgino** » 24 Oct 2022, 21:14

Hi everybody! I've got the same problem (or similar, sorry I'm newbie)

I'm reading a serial data (from solar VE.DIRECT) and ser2net does its job, so I read it on telnet at the selected 1236 port, but I need to publish the data on mqtt (than I will read it from HmeAssistant and publish on my dash board)

So the mqtt doesnt' post nothing.
I only set "SEND to CONTROLLER 1" where 1 is the MQTT broker used also for other topics (an working good)
Where am I wrong???
thankyou

Ath · #31 Post by **Ath** » 24 Oct 2022, 21:34

giorgino wrote: ↑24 Oct 2022, 21:14 So the mqtt doesnt' post nothing.
I only set "SEND to CONTROLLER 1" where 1 is the MQTT broker used also for other topics (an working good)

- What version of ESPEasy are you using (.bin file installed please)
- Have you added the rule I posted a few messages up? You need to adjust the eventname, the Event processing option should be set to Generic, and the rule should then look like this:

Code: Select all

on !Serial* do // Include * to process string values
  Publish,"your/topic/goes here",`%eventpar%` // Wrap in quotes because of possible separators in data
endon

NB: Untested!

giorgino · #32 Post by **giorgino** » 24 Oct 2022, 23:42

hi, thank for your reply!

I'm using
ESP_Easy_mega_20211105_normal_ESP32_4M316k
mega-20211105_c79d675

giorgino · #33 Post by **giorgino** » 24 Oct 2022, 23:47

please let me know about "your/topic/goes here"

the serial gets strings like these:

---------------------------
PID 0xA053
FW 159
SER# HQ213436EP4
V 6380
I 0
VPV 30
PPV 0
CS 0
MPPT 0
OR 0x00000001
ERR 0
--------------------------

I need to extract "V ", "6380" and some other like VPV, PID etc
the MQTT should be something like this: "ESP_easy/input/vict1"

Ath · #34 Post by **Ath** » 25 Oct 2022, 08:49

giorgino wrote: ↑24 Oct 2022, 23:42 hi, thank for your reply!

I'm using
ESP_Easy_mega_20211105_normal_ESP32_4M316k
mega-20211105_c79d675

Can you please use a more recent release, preferably from the latest Github Actions build, here (you will need a (free) Github account to be able to download the Binaries.zip file)

Ath · #35 Post by **Ath** » 25 Oct 2022, 08:50

giorgino wrote: ↑24 Oct 2022, 23:47 please let me know about "your/topic/goes here"
...
the MQTT should be something like this: "ESP_easy/input/vict1"

You have answered yourself already

giorgino · #36 Post by **giorgino** » 25 Oct 2022, 22:46

thanks, it works

Code: Select all

on !Serial* do // Include * to process string values
  Publish,'ESP_Easy/input/vict1',%eventvalue%
endon

Now, a step forward:

the mqtt publish strings, scrambled:

!Serial#
PID 0xA053
FW 159
SER# HQ2

then:
!Serial#
CS 0
MPPT 0
OR 0x00000001
ERR 0
LOAD ON
IL 0
H19 66
H20 1
H21 10
H22 0
H23 4
HSDS 33

or:
!Serial#
PPV 0
CS 0
MPPT 0
OR 0x00000001
ERR 0
LOAD ON
IL 0
H19 66
H20 1
H21 10
H22 0
H23 4
HSDS 33
Checksum

HOW can I collect only some data?example H20 (value=1) and H19 (value=66)

giorgino · #37 Post by **giorgino** » 25 Oct 2022, 23:51

P.S. update OTA to latest

giorgino · #38 Post by **giorgino** » 06 Dec 2022, 22:09

so now I can't move:
data flows but they are mixed. How can I extract only some strings?

giorgino · #39 Post by **giorgino** » 06 Dec 2022, 22:12

is there any debug tools to learn the syntax of RULES ?

Ath · #40 Post by **Ath** » 06 Dec 2022, 22:17

giorgino wrote: ↑06 Dec 2022, 22:09 so now I can't move:
data flows but they are mixed. How can I extract only some strings?

This should get you going: https://espeasy.readthedocs.io/en/lates ... terpreting

But for learning to work with rules, you can also start reading at the top of that page

TD-er · #41 Post by **TD-er** » 06 Dec 2022, 22:18

giorgino wrote: ↑06 Dec 2022, 22:12 is there any debug tools to learn the syntax of RULES ?

On the latest builds, there is syntax highlighting and code completion in the rules edit dialog when using a desktop browser.

chromo23 · #42 Post by **chromo23** » 06 Dec 2022, 22:33

And here (https://raw.githack.com/chromoxdor/Easy ... rcode.html) you find the online code-editor.
If you click there on the ? , you´ll find some basic instructions.
If you click on the "!" you´ll have code-editor and rules documentation side by side.

giorgino · #43 Post by **giorgino** » 07 Dec 2022, 02:22

question: the only event I can get is %eventValue% that give me back my value.

%Eventname% and %eventpar% doesn't give me back nothing. Am I right? depends from SER2NET?

eventvalueN doesn't works too

Ath · #44 Post by **Ath** » 07 Dec 2022, 07:43

giorgino wrote: ↑24 Oct 2022, 23:42 I'm using
ESP_Easy_mega_20211105_normal_ESP32_4M316k
mega-20211105_c79d675

You need a newer ESPEasy release to be able to use %eventname% and/or %eventpar%, try a merge build after the 20221105 release please
You can also test the PR build that adds some new features: https://github.com/letscontrolit/ESPEas ... 3610186170

Edit: Removed 1 link

TD-er · #45 Post by **TD-er** » 07 Dec 2022, 09:37

For ESP32 please try a later GitHub Actions build as the 20221105 build did have some funky issues with ESP32.
For example this one: https://github.com/letscontrolit/ESPEas ... 3634254620

giorgino · #46 Post by **giorgino** » 13 Dec 2022, 21:28

works go ahead: extracting and publishing over mqtt than got from nodered in HA and posted in my dashboard. BUT..... lock !!!

A new problem now: the mqtt sometimes seems to "lock" and the data flow to the broker stops to arrive.

Than, later, it restarts seem randomly. Any idea? is there any "overflows" or "overbuffer" in espeasy ?

(I'm really a newbie...)

Ath · #47 Post by **Ath** » 13 Dec 2022, 23:00

What options did you enable in your Device configuration page?

Can you collect the logs (info level is fine) by opening the Tools/Log page?

Have you increased the RX Buffer size? Not sure if the 1kB current max. size is large enough, though it may also be the cause of any crashes.

What other plugins are enabled?

giorgino · #48 Post by **giorgino** » 13 Dec 2022, 23:05

Ath wrote: ↑07 Dec 2022, 07:43
giorgino wrote: ↑24 Oct 2022, 23:42 I'm using
ESP_Easy_mega_20211105_normal_ESP32_4M316k
mega-20211105_c79d675
You need a newer ESPEasy release to be able to use %eventname% and/or %eventpar%, try a merge build after the 20221105 release please
You can also test the PR build that adds some new features: https://github.com/letscontrolit/ESPEas ... 3610186170

Edit: Removed 1 link

pls do I need to compile ? isn't available the bin for OTA?

giorgino · #49 Post by **giorgino** » 13 Dec 2022, 23:15

Ath wrote: ↑13 Dec 2022, 23:00 What options did you enable in your Device configuration page?

Can you collect the logs (info level is fine) by opening the Tools/Log page?

Have you increased the RX Buffer size? Not sure if the 1kB current max. size is large enough, though it may also be the cause of any crashes.

What other plugins are enabled?

OPTIONS? only 1 device: Communication - Serial Server

Have you increased the RX Buffer size? it was 256B, now I try increasing to 1024B

Other plugins? Ii'm using only the seriale and the MQTT

Controller Queue
Minimum Send Interval:
100
[ms]
Max Queue Depth:
10
Max Retries:
10
Full Queue Action:
Ignore New
Allow Expire:
De-duplicate:
Check Reply:
Ignore Acknowledgement
Client Timeout:
100
[ms]

giorgino · #50 Post by **giorgino** » 13 Dec 2022, 23:32

hey something happend: it was "locked", with MQTT explorer muted, than I connect PUTTY to the telnet port.

data stars to flow on the monitor of Putty and at the same time mqtt explorer starts to reporte the flow !

Is this useful for your debug?

----------------the flag!!!!! connected client?????

Who is online