json or xml parser feature request

Moderators: grovkillen, Stuntteam, TD-er

Post Reply
Message
Author
User avatar
chromo23
Normal user
Posts: 821
Joined: 10 Sep 2020, 16:02
Location: germany

json or xml parser feature request

#1 Post by chromo23 » 21 Apr 2021, 09:08

I am using espeasy for quite a long time now on many devices and since now i had no need for a broker/gateway/server.
I want to keep it that way. Not because i am unable to do so but i like it simple.
One thing i am missing is the possibility to receive data from somewhere far away. Like weather data from a weather service oder from another station.
Therefore the simplest solution in my opinion would be a simple json or xml parser.
Is it something other people need too and therefore worth implementing ?

User avatar
Ath
Normal user
Posts: 3417
Joined: 10 Jun 2018, 12:06
Location: NL

Re: json or xml parser feature request

#2 Post by Ath » 21 Apr 2021, 09:40

Is this PR, at least partially, what you have in mind? https://github.com/letscontrolit/ESPEasy/pull/3424

Biggest challenge is the lack of available memory on the ESP8266 units, IMHO, that makes it very hard when receiving large(r) messages.
/Ton (PayPal.me)

User avatar
chromo23
Normal user
Posts: 821
Joined: 10 Sep 2020, 16:02
Location: germany

Re: json or xml parser feature request

#3 Post by chromo23 » 21 Apr 2021, 09:53

I am no expert but requesting simple json messages doesn´t seem like a big memory eater.


Example output from a thingspeak channel.

Code: Select all

{"created_at":"2021-04-21T07:38:11Z","entry_id":36611,"field1":null,"field2":null,"field3":"19.11","field4":"55.37","field5":null,"field6":null,"field7":"1013.81","field8":"4.06"}
i request this by:

Code: Select all

http://api.thingspeak.com/channels/978931/feeds/last
i thought about it like

Code: Select all

do every minute a url request
write field1 to dummy device
i don’t have large messages in mind.

User avatar
Ath
Normal user
Posts: 3417
Joined: 10 Jun 2018, 12:06
Location: NL

Re: json or xml parser feature request

#4 Post by Ath » 21 Apr 2021, 10:10

chromo23 wrote: 21 Apr 2021, 09:53 i thought about it like

Code: Select all

do every minute a url request
write field1 to dummy device
Currently there is no 'GetFromHTTP' command, that would have to be implemented first. And handling strings in rules is also a bit of a challenge, so that would either be quite specific, or needs some revision of ESPEasy internals (that has been thought/talked about before, but isn't picked up yet)

With the PR I linked implemented, you could receive that via MQTT to process on a unit.
Then also events can be generated, either for a selected field or for all fields received.

chromo23 wrote: 21 Apr 2021, 09:53 i don’t have large messages in mind.
Well, that's how we all start :D :D :D
/Ton (PayPal.me)

User avatar
chromo23
Normal user
Posts: 821
Joined: 10 Sep 2020, 16:02
Location: germany

Re: json or xml parser feature request

#5 Post by chromo23 » 21 Apr 2021, 10:22

Thank you for the information!

MQTT is no option.... because the i would also need a server for it.
a url request is much simpler because you only need the esp and internet of course.
You can put the esp where ever you want and you don’t need to configure something extra.
Also i personally find the json way easier to share data with other esps that are not in the same network or even with friends.
E.g.: I have made an esp with a display and a sensor for a friend so he easily can see the inside temperature. But he would like to also see the outside temp.
since he is in a big city there is no need for an extra outside sensor. he could get the data from openweather
If i would do it the mqtt way i have to install a sever at his place and also have to maintain it. (Waste of energy an resources) :)

(One could ask... why is he not using his phone to see the outside temperature? answer: some people are not able or it’s harder for them...)

(Edit: for me personally i associate espeasy with being able to use an esp almost standalone. the ability of using a broker is a nice addition but was never the reason for me using espeasy.)
Last edited by chromo23 on 21 Apr 2021, 10:43, edited 1 time in total.

User avatar
Ath
Normal user
Posts: 3417
Joined: 10 Jun 2018, 12:06
Location: NL

Re: json or xml parser feature request

#6 Post by Ath » 21 Apr 2021, 10:42

And another issue is that ESPEasy (at least on ESP8266) doesn't have the resources available (and hasn't been implemented because of that), to process HTTPS connections, and it is becoming more and more of a challenge to find external services that still accept HTTP requests.

Sharing your data with 'external parties' will be a challenge for the same reasons, I've seen other solutions that use stuff like an OpenHAB server or similar that has support for HTTPS, to send out local data, or use a MQTT server, either on a Raspberry Pi or out on the internet, to collect sensor data, to make that available to third parties.
/Ton (PayPal.me)

User avatar
chromo23
Normal user
Posts: 821
Joined: 10 Sep 2020, 16:02
Location: germany

Re: json or xml parser feature request

#7 Post by chromo23 » 21 Apr 2021, 10:48

Ath wrote: 21 Apr 2021, 10:42 And another issue is that ESPEasy (at least on ESP8266) doesn't have the resources available (and hasn't been implemented because of that), to process HTTPS connections, and it is becoming more and more of a challenge to find external services that still accept HTTP requests.
Good Point.
All services i use can handle http (thingspeak and openweather). but it might be possible that that will change.
Ath wrote: 21 Apr 2021, 10:42 Sharing your data with 'external parties' will be a challenge for the same reasons,
Sharing means in this context: If you have the url, you have the data. Thats only a matter of copy’n’paste ;)

User avatar
Ath
Normal user
Posts: 3417
Joined: 10 Jun 2018, 12:06
Location: NL

Re: json or xml parser feature request

#8 Post by Ath » 21 Apr 2021, 10:56

chromo23 wrote: 21 Apr 2021, 10:48 Sharing means in this context: If you have the url, you have the data. Thats only a matter of copy’n’paste ;)
Allowing access to the ESP directly from the i-net is a big no-no, as there is hardly any security, and the unit can probably be hacked quite easily, even for/by non-malicious hackers.
/Ton (PayPal.me)

User avatar
chromo23
Normal user
Posts: 821
Joined: 10 Sep 2020, 16:02
Location: germany

Re: json or xml parser feature request

#9 Post by chromo23 » 21 Apr 2021, 11:00

You understood me wrong i guess...

I have an esp that sends data do thingspeak via sendtohttp.....

you want my data:

Code: Select all

http://api.thingspeak.com/channels/978931/feeds/last.json
you want to process it on your esp:

Code: Select all

do every minute a url request of http://api.thingspeak.com/channels/978931/feeds/last.json
write field1 to dummy device
Edit:
if you use this data to control something vital or hazardous it could be n issue... but just for displaying data its fine i guess
the esp doesn’t present itself to the internet and the hacking potential for a url request is small.. a hacker can only change the numbers on the server side
Last edited by chromo23 on 21 Apr 2021, 11:29, edited 3 times in total.

User avatar
chromo23
Normal user
Posts: 821
Joined: 10 Sep 2020, 16:02
Location: germany

Re: json or xml parser feature request

#10 Post by chromo23 » 21 Apr 2021, 11:13

Maybe implementing this in the core ist not a good idea for the reasons you stated.
It could be an optional thingspeak or openweather only plugin.
Maybe i have to start diving deeper into the rabbit hole and do it. But.....realistically .....i don´t...unfortunately ..... if only days and individual lifetime would be longer :D

User avatar
Ath
Normal user
Posts: 3417
Joined: 10 Jun 2018, 12:06
Location: NL

Re: json or xml parser feature request

#11 Post by Ath » 21 Apr 2021, 11:33

chromo23 wrote: 21 Apr 2021, 11:13 Maybe implementing this in the core ist not a good idea for the reasons you stated.
It could be an optional thingspeak or openweather only plugin.
I'm thinking of a somewhat generic plugin that will support the GetJSONFromHTTP command, with optional credentials and similar processing options that I proposed in the PR for P037. It will involve some refactoring of the code, but code re-use should be very well possible.
/Ton (PayPal.me)

User avatar
chromo23
Normal user
Posts: 821
Joined: 10 Sep 2020, 16:02
Location: germany

Re: json or xml parser feature request

#12 Post by chromo23 » 21 Apr 2021, 11:42

That would be it i guess.
I don`t like to ask..but could you do this. :oops:

It feels uncomfortable to ask because i could be the only one in need of this.
Last edited by chromo23 on 21 Apr 2021, 12:07, edited 1 time in total.

User avatar
Ath
Normal user
Posts: 3417
Joined: 10 Jun 2018, 12:06
Location: NL

Re: json or xml parser feature request

#13 Post by Ath » 21 Apr 2021, 11:45

chromo23 wrote: 21 Apr 2021, 11:42 That would be it i guess.
I don`t like to ask..but could you do this. :oops:
I can, but I won't give an ETA 8-)
/Ton (PayPal.me)

User avatar
chromo23
Normal user
Posts: 821
Joined: 10 Sep 2020, 16:02
Location: germany

Re: json or xml parser feature request

#14 Post by chromo23 » 21 Apr 2021, 11:51

I can, but I won't give an ETA 8-)

This is something i would never ask in this context....only if i would be your boss :D

Post Reply

Who is online

Users browsing this forum: No registered users and 15 guests