MQTT Import topic subscribe fields too short

Moderators: grovkillen, Stuntteam, TD-er

Post Reply
Message
Author
nibiru
New user
Posts: 6
Joined: 17 Oct 2021, 22:16
Location: Ukraine

MQTT Import topic subscribe fields too short

#1 Post by nibiru » 17 Oct 2021, 22:34

Hello to all!
I am using a mqtt broker maqiatto.com, which has a required topic prefix - username, which is an email address of user.
The topic import field in MQTT Import plugin has a limit of 40 characters, which is very small.
Is it possible to somehow increase this limit, or set a common prefix for all topics separately?
Thank you very much!
fr_1.png
fr_1.png (23.4 KiB) Viewed 12765 times

TD-er
Core team member
Posts: 8607
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: MQTT Import topic subscribe fields too short

#2 Post by TD-er » 17 Oct 2021, 23:26

Not without changing the code of this plugin.
But maybe you can for example set something in a system variable, like you're doing in the %sysname%?
See the sysvars page for inspiration.

User avatar
Ath
Normal user
Posts: 3386
Joined: 10 Jun 2018, 12:06
Location: NL

Re: MQTT Import topic subscribe fields too short

#3 Post by Ath » 17 Oct 2021, 23:36

I'm still working on a couple of enhancements for this plugin, including a re-worked settings structure. I can add a generic prefix for all topics, as changing all input lengths seems less useful, and harder to upgrade existing settings.
It still needs thorough testing, as I'm fixing some memory allocation issues, but thats progressing nicely now. The PR is already taking quite some time, but that should not block you from testing, once I complete testing myself. Will report here when that test-build is available.
/Ton (PayPal.me)

User avatar
ThomasB
Normal user
Posts: 1064
Joined: 17 Jun 2018, 20:41
Location: USA

Re: MQTT Import topic subscribe fields too short

#4 Post by ThomasB » 18 Oct 2021, 00:58

But maybe you can for example set something in a system variable, like you're doing in the %sysname%?
+1 That seems like a simple solution.

Or, how about using the Home Assistant (openHAB) MQTT controller instead of the MQTT Import plugin? It is a generic MQTT controller and its subscribe strings are much longer (128 chars).

- Thomas

nibiru
New user
Posts: 6
Joined: 17 Oct 2021, 22:16
Location: Ukraine

Re: MQTT Import topic subscribe fields too short

#5 Post by nibiru » 18 Oct 2021, 01:08

ThomasB wrote: 18 Oct 2021, 00:58
Or, how about using the Home Assistant (openHAB) MQTT controller
Yes. i use this controller.
But maybe I misunderstand the control mechanism. Isn't a MQTT Import plugin the only way to transfer data from mqtt broker to ESP board?
fr_3.png
fr_3.png (18.24 KiB) Viewed 12745 times
fr_2.png
fr_2.png (19.61 KiB) Viewed 12745 times

User avatar
ThomasB
Normal user
Posts: 1064
Joined: 17 Jun 2018, 20:41
Location: USA

Re: MQTT Import topic subscribe fields too short

#6 Post by ThomasB » 18 Oct 2021, 01:13

Isn't a MQTT Import plugin the only way to transfer data from mqtt broker to ESP board?
I'm using the Home Assistant (openHAB) MQTT controller to receive MQTT commands. I use it to execute ESPEasy event rules.

EDIT: I just tried setting a system var in case that is what you need to do. Works great.
Here is an example of a publish statement:

Code: Select all

publish: TOPIC_TEMPLATE/cmd
message: let,1,99
Check %v1% (page Tools->System Variables) and you should see the "99" value in it.

- Thomas

TD-er
Core team member
Posts: 8607
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: MQTT Import topic subscribe fields too short

#7 Post by TD-er » 18 Oct 2021, 09:21

One other thing...
It looks like the hostname you're using is not on your local network.
Thus better increase the timeout from 100 msec to 500 or more.

nibiru
New user
Posts: 6
Joined: 17 Oct 2021, 22:16
Location: Ukraine

Re: MQTT Import topic subscribe fields too short

#8 Post by nibiru » 18 Oct 2021, 19:21

Hello everyone!
Was very surprised by the speed of response in this community as well as the warm welcome !
TD-er wrote: 18 Oct 2021, 09:21 Thus better increase the timeout from 100 msec to 500 or more
100 ms is send interval, not client timeout.
ThomasB wrote: 18 Oct 2021, 01:13 publish: TOPIC_TEMPLATE/cmd
message: let,1,99
Wow, its works, thanks. Also tested this method, and it also works!
fr_4.png
fr_4.png (6.95 KiB) Viewed 12689 times
But...
This method assumes special formatting of the values that need to be entered. It is not very convenient to control the board using a mobile MQTT Dashboard app. The MQTT import plugin works much more conveniently. Then there are 2 types of topics - for sending values and receiving confirmation (feedback).

Now I'm using MQTT import plugin, but with short, non-human-readable topic names. But that's okay. I was also lucky that the username (the email address) was not so long :)
Ath wrote: 17 Oct 2021, 23:36 Will report here when that test-build is available
I will be happy to take part in testing.
Ath wrote: 17 Oct 2021, 23:36 and harder to upgrade existing settings
I needed to switch from one broker to another and had to rewrite the prefixes in the plugin, controller settings, and also in the rules.
Thus, if it were possible to set the user prefix globally, it would be cool!

Thank you all very much for the answers, and for such a powerful firmware for esp!

Roman.

User avatar
ThomasB
Normal user
Posts: 1064
Joined: 17 Jun 2018, 20:41
Location: USA

Re: MQTT Import topic subscribe fields too short

#9 Post by ThomasB » 18 Oct 2021, 20:27

Thus, if it were possible to set the user prefix globally, it would be cool!
A theoretical solution would be to add string type vars to ESPEasy. Besides general purpose use, they could be used as string substitutes in the application you mentioned.

For example, initialize the string var at boot in the rules, like this:

Code: Select all

on System#Boot do
  let,str1,"yourname@gmail.com/" + %sysname% + "/"
endof
Now any place %str1% is used it would expand to yourname@gmail.com/ESP_Easy/

Just thinking out loud here. Maybe @TD-er or @Ath have better ideas. Or maybe this functionality already exists?

- Thomas

TD-er
Core team member
Posts: 8607
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: MQTT Import topic subscribe fields too short

#10 Post by TD-er » 18 Oct 2021, 21:14

Nope, does not (yet) exist... But I think the %strN% syntax analog to %vN% would be a good idea.
The string variant for "let" would then be like "letstr".

User avatar
Ath
Normal user
Posts: 3386
Joined: 10 Jun 2018, 12:06
Location: NL

Re: MQTT Import topic subscribe fields too short

#11 Post by Ath » 23 Oct 2021, 15:10

nibiru wrote: 18 Oct 2021, 19:21
Ath wrote: 17 Oct 2021, 23:36 Will report here when that test-build is available
I will be happy to take part in testing.
Hi Roman,

Reporting back after applying improvements to the code in my PR. Can you fetch your build from the Github Actions, or do you want me to provide a build for you? If the latter, then please specify what build you are currently using (bin filename please, didn't find clues in this thread) or the ESP type (ESP32/ESP8266), size of flash and a list of plugins you actually use, so I can create a Custom build for you.

Edit: The Github Actions build failed because of .bin size issues (as it does often), so the second option is the best for now ;)
/Ton (PayPal.me)

nibiru
New user
Posts: 6
Joined: 17 Oct 2021, 22:16
Location: Ukraine

Re: MQTT Import topic subscribe fields too short

#12 Post by nibiru » 23 Oct 2021, 15:48

Ath wrote: 23 Oct 2021, 15:10 bin filename please
Hello, Ath!
I'm using ESP_Easy_mega_20211005_normal_ESP8266_4M1M.
Using:
Generic - MQTT Import
Generic - Dummy Device
Environment - DS18b20
Generic - System Info

I would be glad if you provide me with a test build.
Thanks!

Roman.

User avatar
Ath
Normal user
Posts: 3386
Joined: 10 Jun 2018, 12:06
Location: NL

Re: MQTT Import topic subscribe fields too short

#13 Post by Ath » 23 Oct 2021, 16:38

This is the 'normal' build from my development branch:
ESP_Easy_mega_20211023_normal_ESP8266_4M1M.zip
(606.32 KiB) Downloaded 203 times
And a preview of the documentation: (not yet including the Topic prefix field, that my next task :))
/Ton (PayPal.me)

nibiru
New user
Posts: 6
Joined: 17 Oct 2021, 22:16
Location: Ukraine

Re: MQTT Import topic subscribe fields too short

#14 Post by nibiru » 23 Oct 2021, 18:17

Ath wrote: 23 Oct 2021, 16:38 build from my development branch
Already flashed and tested. It is working!
fr_5.png
fr_5.png (26.89 KiB) Viewed 12517 times
Now I am thinking about applying the new features of the plugin (filter). Now filtering (or rather normalization of incoming values) is done using rules. But by using the new features, I maybe can save a little space in the rules.

Many thanks for the provided build and new plugin features!

Roman.

User avatar
Ath
Normal user
Posts: 3386
Joined: 10 Jun 2018, 12:06
Location: NL

Re: MQTT Import topic subscribe fields too short

#15 Post by Ath » 24 Oct 2021, 13:17

nibiru wrote: 23 Oct 2021, 18:17 Now I am thinking about applying the new features of the plugin (filter). Now filtering (or rather normalization of incoming values) is done using rules. But by using the new features, I maybe can save a little space in the rules.
Major advantage of filtering is that the topic is 'dropped' if it doesn't pass the filter, and the rules aren't parsed at all then. Even that is a big speed/load improvement.
nibiru wrote: 23 Oct 2021, 18:17 Many thanks for the provided build and new plugin features!
You're welcome 8-)
/Ton (PayPal.me)

User avatar
ThomasB
Normal user
Posts: 1064
Joined: 17 Jun 2018, 20:41
Location: USA

Re: MQTT Import topic subscribe fields too short

#16 Post by ThomasB » 24 Oct 2021, 19:00

@Ath, The new features are well thought out. Fantastic documentation!

- Thomas

User avatar
Ath
Normal user
Posts: 3386
Joined: 10 Jun 2018, 12:06
Location: NL

Re: MQTT Import topic subscribe fields too short

#17 Post by Ath » 24 Oct 2021, 19:25

ThomasB wrote: 24 Oct 2021, 19:00 @Ath, The new features are well thought out. Fantastic documentation!

- Thomas
Thnx :D
/Ton (PayPal.me)

User avatar
dynamicdave
Normal user
Posts: 257
Joined: 30 Jan 2017, 20:25
Location: Hampshire, UK

Re: MQTT Import topic subscribe fields too short

#18 Post by dynamicdave » 07 Jan 2022, 08:44

ThomasB wrote: 18 Oct 2021, 20:27
Thus, if it were possible to set the user prefix globally, it would be cool!
A theoretical solution would be to add string type vars to ESPEasy. Besides general purpose use, they could be used as string substitutes in the application you mentioned.

For example, initialize the string var at boot in the rules, like this:

Code: Select all

on System#Boot do
  let,str1,"yourname@gmail.com/" + %sysname% + "/"
endof
Now any place %str1% is used it would expand to yourname@gmail.com/ESP_Easy/

Just thinking out loud here. Maybe @TD-er or @Ath have better ideas. Or maybe this functionality already exists?

- Thomas
Has anything happened regarding this excellent suggestion?
Also on a related topic/question... has there been any development with sending a string via MQTT event and collecting it with %eventstring%

The reason I ask is... it would great if a string variable could be used in a 'publish' command.
e.g. replace "garage" with %str1%

Code: Select all

On publishReading Do
  publish homie/reading,'{"node_number":[my#nodeID],"location":"garage","ssid":"%ssid%","update_interval":[timer#value]}'
EndOn

User avatar
Ath
Normal user
Posts: 3386
Joined: 10 Jun 2018, 12:06
Location: NL

Re: MQTT Import topic subscribe fields too short

#19 Post by Ath » 07 Jan 2022, 09:45

I have a longstanding open PR #3424 that adds a 'Prefix for all topics' feature (see this screenshot from the PR)

There is no %eventstring% support yet, other things have gotten priority, and the desire to add string handling as a generic feature isn't speeding things up either.

For replacing 'garage' with a variable, it would depend on the number of options, but the {substring:<startpos>:<endpos>:<string>} function could help here (documentation)
/Ton (PayPal.me)

User avatar
dynamicdave
Normal user
Posts: 257
Joined: 30 Jan 2017, 20:25
Location: Hampshire, UK

Re: MQTT Import topic subscribe fields too short

#20 Post by dynamicdave » 07 Jan 2022, 10:49

Thanks for your swift response.
It's not an issue as I can handle it the way I'm doing it now.
Much better for the 'Team' to concentrate on higher priority features/issues.

TD-er
Core team member
Posts: 8607
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: MQTT Import topic subscribe fields too short

#21 Post by TD-er » 07 Jan 2022, 10:57

Adding labels to variables isn't that hard to do, but it may take up more memory and resources in parsing.
Right now variables are stored as a std::map<int,double>
But this can be changed to std::map<String,double>
It also needs some adaptations to the parser, but that's about it.
Then you could use %v_garagetimer% for example as a variable holding a double value.

Storing strings is a different matter.
Not only will it easily blow up in used memory resources, as those variables may tend to live quite long.
But it also needs quite a lot of work to extend the "let" command (or add a "letstr") and where it can be used.
Maybe that last part isn't as hard as I am afraid it will be as things will be replaced by their string equivalent before parsing anyway, but I think I might be missing some corner cases then.
However, the more I think about it, the more powerful it appears to be to have string operations as you could even make dynamic program flows.
For example you can even place a string variable on a rules line and expect it to be executed if it contains rules code.
But that's also the dangerous part. What if you get some string value as an event value via MQTT or HTTP and it contains a call to perform a factory reset?

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest