MQTT Import topic subscribe fields too short

[nibiru]

Hello to all!
I am using a mqtt broker maqiatto.com, which has a required topic prefix - username, which is an email address of user.
The topic import field in MQTT Import plugin has a limit of 40 characters, which is very small.
Is it possible to somehow increase this limit, or set a common prefix for all topics separately?
Thank you very much!

fr_1.png (23.4 KiB) Viewed 18675 times

[TD-er]

Not without changing the code of this plugin.
But maybe you can for example set something in a system variable, like you're doing in the %sysname%?
See the sysvars page for inspiration.

[Ath]

I'm still working on a couple of enhancements for this plugin, including a re-worked settings structure. I can add a generic prefix for all topics, as changing all input lengths seems less useful, and harder to upgrade existing settings.
It still needs thorough testing, as I'm fixing some memory allocation issues, but thats progressing nicely now. The PR is already taking quite some time, but that should not block you from testing, once I complete testing myself. Will report here when that test-build is available.

[ThomasB]

But maybe you can for example set something in a system variable, like you're doing in the %sysname%?

+1 That seems like a simple solution.

Or, how about using the Home Assistant (openHAB) MQTT controller instead of the MQTT Import plugin? It is a generic MQTT controller and its subscribe strings are much longer (128 chars).

- Thomas

[nibiru]

Or, how about using the Home Assistant (openHAB) MQTT controller

Yes. i use this controller.
But maybe I misunderstand the control mechanism. Isn't a MQTT Import plugin the only way to transfer data from mqtt broker to ESP board?

fr_3.png (18.24 KiB) Viewed 18655 times

fr_2.png (19.61 KiB) Viewed 18655 times

[ThomasB]

Isn't a MQTT Import plugin the only way to transfer data from mqtt broker to ESP board?

I'm using the Home Assistant (openHAB) MQTT controller to receive MQTT commands. I use it to execute ESPEasy event rules.

EDIT: I just tried setting a system var in case that is what you need to do. Works great.
Here is an example of a publish statement:

Code: Select all

publish: TOPIC_TEMPLATE/cmd
message: let,1,99

Check %v1% (page Tools->System Variables) and you should see the "99" value in it.

- Thomas

[TD-er]

One other thing...
It looks like the hostname you're using is not on your local network.
Thus better increase the timeout from 100 msec to 500 or more.

[nibiru]

Hello everyone!
Was very surprised by the speed of response in this community as well as the warm welcome !

Thus better increase the timeout from 100 msec to 500 or more

100 ms is send interval, not client timeout.

publish: TOPIC_TEMPLATE/cmd
message: let,1,99

Wow, its works, thanks. Also tested this method, and it also works!

fr_4.png (6.95 KiB) Viewed 18599 times

But...
This method assumes special formatting of the values that need to be entered. It is not very convenient to control the board using a mobile MQTT Dashboard app. The MQTT import plugin works much more conveniently. Then there are 2 types of topics - for sending values and receiving confirmation (feedback).

Now I'm using MQTT import plugin, but with short, non-human-readable topic names. But that's okay. I was also lucky that the username (the email address) was not so long

Will report here when that test-build is available

I will be happy to take part in testing.

and harder to upgrade existing settings

I needed to switch from one broker to another and had to rewrite the prefixes in the plugin, controller settings, and also in the rules.
Thus, if it were possible to set the user prefix globally, it would be cool!

Thank you all very much for the answers, and for such a powerful firmware for esp!

Roman.

[ThomasB]

Thus, if it were possible to set the user prefix globally, it would be cool!

A theoretical solution would be to add string type vars to ESPEasy. Besides general purpose use, they could be used as string substitutes in the application you mentioned.

For example, initialize the string var at boot in the rules, like this:

Code: Select all

on System#Boot do
  let,str1,"yourname@gmail.com/" + %sysname% + "/"
endof

Now any place %str1% is used it would expand to yourname@gmail.com/ESP_Easy/

Just thinking out loud here. Maybe @TD-er or @Ath have better ideas. Or maybe this functionality already exists?

- Thomas

[TD-er]

Nope, does not (yet) exist... But I think the %strN% syntax analog to %vN% would be a good idea.
The string variant for "let" would then be like "letstr".

[Ath]

Will report here when that test-build is available

I will be happy to take part in testing.

Hi Roman,

Reporting back after applying improvements to the code in my PR. Can you fetch your build from the Github Actions, or do you want me to provide a build for you? If the latter, then please specify what build you are currently using (bin filename please, didn't find clues in this thread) or the ESP type (ESP32/ESP8266), size of flash and a list of plugins you actually use, so I can create a Custom build for you.

Edit: The Github Actions build failed because of .bin size issues (as it does often), so the second option is the best for now

[nibiru]

bin filename please

Hello, Ath!
I'm using ESP_Easy_mega_20211005_normal_ESP8266_4M1M.
Using:
Generic - MQTT Import
Generic - Dummy Device
Environment - DS18b20
Generic - System Info

I would be glad if you provide me with a test build.
Thanks!

Roman.

[Ath]

This is the 'normal' build from my development branch:

ESP_Easy_mega_20211023_normal_ESP8266_4M1M.zip

(606.32 KiB) Downloaded 430 times

And a preview of the documentation: (not yet including the Topic prefix field, that my next task )

Generic - MQTT Import — ESP Easy 2.1-beta1 documentation.zip

(269.74 KiB) Downloaded 307 times

[nibiru]

build from my development branch

Already flashed and tested. It is working!

fr_5.png (26.89 KiB) Viewed 18427 times

Now I am thinking about applying the new features of the plugin (filter). Now filtering (or rather normalization of incoming values) is done using rules. But by using the new features, I maybe can save a little space in the rules.

Many thanks for the provided build and new plugin features!

Roman.

[Ath]

Now I am thinking about applying the new features of the plugin (filter). Now filtering (or rather normalization of incoming values) is done using rules. But by using the new features, I maybe can save a little space in the rules.

Major advantage of filtering is that the topic is 'dropped' if it doesn't pass the filter, and the rules aren't parsed at all then. Even that is a big speed/load improvement.

Many thanks for the provided build and new plugin features!

You're welcome

[ThomasB]

@Ath, The new features are well thought out. Fantastic documentation!

- Thomas

[Ath]

@Ath, The new features are well thought out. Fantastic documentation!

- Thomas

Thnx

[dynamicdave]

Thus, if it were possible to set the user prefix globally, it would be cool!

A theoretical solution would be to add string type vars to ESPEasy. Besides general purpose use, they could be used as string substitutes in the application you mentioned.

For example, initialize the string var at boot in the rules, like this:

Code: Select all

on System#Boot do
  let,str1,"yourname@gmail.com/" + %sysname% + "/"
endof

Now any place %str1% is used it would expand to yourname@gmail.com/ESP_Easy/

Just thinking out loud here. Maybe @TD-er or @Ath have better ideas. Or maybe this functionality already exists?

- Thomas

Has anything happened regarding this excellent suggestion?
Also on a related topic/question... has there been any development with sending a string via MQTT event and collecting it with %eventstring%

The reason I ask is... it would great if a string variable could be used in a 'publish' command.
e.g. replace "garage" with %str1%

Code: Select all

On publishReading Do
  publish homie/reading,'{"node_number":[my#nodeID],"location":"garage","ssid":"%ssid%","update_interval":[timer#value]}'
EndOn

[Ath]

I have a longstanding open PR #3424 that adds a 'Prefix for all topics' feature (see this screenshot from the PR)

There is no %eventstring% support yet, other things have gotten priority, and the desire to add string handling as a generic feature isn't speeding things up either.

For replacing 'garage' with a variable, it would depend on the number of options, but the {substring:<startpos>:<endpos>:<string>} function could help here (documentation)

[dynamicdave]

Thanks for your swift response.
It's not an issue as I can handle it the way I'm doing it now.
Much better for the 'Team' to concentrate on higher priority features/issues.

[TD-er]

Adding labels to variables isn't that hard to do, but it may take up more memory and resources in parsing.
Right now variables are stored as a std::map<int,double>
But this can be changed to std::map<String,double>
It also needs some adaptations to the parser, but that's about it.
Then you could use %v_garagetimer% for example as a variable holding a double value.

Storing strings is a different matter.
Not only will it easily blow up in used memory resources, as those variables may tend to live quite long.
But it also needs quite a lot of work to extend the "let" command (or add a "letstr") and where it can be used.
Maybe that last part isn't as hard as I am afraid it will be as things will be replaced by their string equivalent before parsing anyway, but I think I might be missing some corner cases then.
However, the more I think about it, the more powerful it appears to be to have string operations as you could even make dynamic program flows.
For example you can even place a string variable on a rules line and expect it to be executed if it contains rules code.
But that's also the dangerous part. What if you get some string value as an event value via MQTT or HTTP and it contains a call to perform a factory reset?