obscure problem with sendtohtttp (and thingspeak)

Moderators: grovkillen, Stuntteam, TD-er

Message
Author
burdal1
Normal user
Posts: 3
Joined: 30 Nov 2019, 09:42

obscure problem with sendtohtttp (and thingspeak)

#1 Post by burdal1 » 30 Nov 2019, 11:05

Hi, i have problem with sendtohttp since mega-20190813. I use sendtohttp in rules to update thingspeak channel once per minute. For test purpose i have running MinWeb server at bur.dubya.net:8000.

Code: Select all

SendToHTTP api.thingspeak.com,80,/update?api_key=<secret>&field1=10&field2=1
SendToHTTP bur.dubya.net,8000,/update?api_key=<secret>&field1=10&field2=1

With mega-20190809: it works - thingspeak channel is updated and in MiniWeb server logs i can find request with request path: update?api_key=<secret>&field1=10&field2=1
With mega-20190813: doesn't work - thingspeak channel is NOT updated BUT in MiniWeb server logs i can found correct request
With mega-20191103: same situation as with mega-20190813

with mega-20191130 i used

Code: Select all

SendToHTTP api.thingspeak.com,80,"/update?api_key=<secret>&field1=10&field2=1"
SendToHTTP bur.dubya.net,8000,"/update?api_key=<secret>&field1=10&field2=1"

and the results are same a mega-20190813. In MiniWeb server everythings look ok but Thingspeak channel is not updated.

It is not important if i use "normal" or "test" version. it works equally.
Do you have any idea where is the problem? I can't find the differences between mega-20190813 and mega-20190809. Thanks.

TD-er
Core team member
Posts: 2791
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: obscure problem with sendtohtttp (and thingspeak)

#2 Post by TD-er » 30 Nov 2019, 11:14

I do know that ThingSpeak does require to accept the response from the server.
That's a change they made at the beginning of this year.
But that doesn't really explain why older builds are still working with sendtohttp.

By the way I removed your API keys from your post.

burdal1
Normal user
Posts: 3
Joined: 30 Nov 2019, 09:42

Re: obscure problem with sendtohtttp (and thingspeak)

#3 Post by burdal1 » 30 Nov 2019, 11:19

TD-er wrote:
30 Nov 2019, 11:14
By the way I removed your API keys from your post.
Thanks.

Is here any alternative? I need send over 4 values from different sensor in one request. I can use dummy device+thingspeak controller, but dummy device can collect only 4 values.

TD-er
Core team member
Posts: 2791
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: obscure problem with sendtohtttp (and thingspeak)

#4 Post by TD-er » 30 Nov 2019, 12:23

Well you can copy various values to a dummy using taskvalueset and then call for taskrun to "execute" the dummy.
If you set the thingspeak controller to that dummy, then you should be able to send the values.
4 should be enough then, right?
Otherwise you can use multiple dummy tasks. Just make sure to set the minimum send interval of the controller to an appropriate value for thingspeak if you're using the free version.

burdal1
Normal user
Posts: 3
Joined: 30 Nov 2019, 09:42

Re: obscure problem with sendtohtttp (and thingspeak)

#5 Post by burdal1 » 30 Nov 2019, 13:04

TD-er wrote:
30 Nov 2019, 12:23
4 should be enough then, right?
Otherwise you can use multiple dummy tasks.
This is the problem, i need send more than 4 values in one request - at the same time with one timestamp. I use the highcharts graph to display the data and i need display all field from one moment together.

Image

User avatar
iron
Normal user
Posts: 108
Joined: 24 Sep 2016, 08:37
Location: Veria, Greece
Contact:

Re: obscure problem with sendtohtttp (and thingspeak)

#6 Post by iron » 14 Dec 2019, 19:44

This works fine on a 20190805 unit

Code: Select all

SendToHTTP,api.thingspeak.com,80,/update?api_key=<mykey>&field1=[Thermometer#Temperature]&field2=[Thermometer#Humidity]
When I upgrade the same unit to 20191208 the same does not work.

1. Enabled /disabled the the parsing tolerance option : does not work allthough log shows trigger and "sendtohttp"
2. I quoted as described by TD-er : does not work allthough log shows trigger and "sendtohttp"

Code: Select all

SendToHTTP,api.thingspeak.com,80,"/update?api_key=<mykey>&field1=[Thermometer#Temperature]&field2=[Thermometer#Humidity]"
3. mix and match quotes and enable / disable parsing tolerance : does not work allthough log shows trigger and "sendtohttp"

Downgrading back to 20190805 with the original code everything works

https://thingspeak.com/channels/33034

-D
-D

TD-er
Core team member
Posts: 2791
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: obscure problem with sendtohtttp (and thingspeak)

#7 Post by TD-er » 14 Dec 2019, 20:09

Is there a " in the string?
You can also try one of the other 2 quotes.
You can use the ` ' or "

Oh and by the way, it can also be a problem with sendtohttp in combination with thingspeak.

User avatar
iron
Normal user
Posts: 108
Joined: 24 Sep 2016, 08:37
Location: Veria, Greece
Contact:

Re: obscure problem with sendtohtttp (and thingspeak)

#8 Post by iron » 14 Dec 2019, 20:31

TD-er wrote:
14 Dec 2019, 20:09
Is there a " in the string?
You can also try one of the other 2 quotes.
You can use the ` ' or "

Oh and by the way, it can also be a problem with sendtohttp in combination with thingspeak.
There are no quotes in the initial string.

I tried quoting both single and double quotes, still no go.

Makes me wonder why it works on the older version... Perhaps the parsed string is not as it should be ?
Will try a quoted version to one of my servers to verify the final string received once it gets parsed

-D
-D

MarcelM
Normal user
Posts: 16
Joined: 15 Dec 2019, 14:41

Re: obscure problem with sendtohtttp (and thingspeak)

#9 Post by MarcelM » 15 Dec 2019, 15:50

Same obscure problem here with version 20191130 en 20191208 (normal and test version).
I want to sends a serie of 8 fields in one SendToHTTP request using Rules.
It makes no difference using quotes or no quotes, that is not the problem here.
On my local TcpListener tool, the GET request is send correctly. Also the HTTP headers look good, except the Accept: */* is missing, I don't know if it is a problem.

I also tried [Generic HTTP Advanced controller] in the newest test version 20191208. This one operates fine, it allows also to send HTTP POST requests and you can manipulate HTTP headers.
But, this controller can only send values from one task, thus max 4 values. It does not support syntax like field1=[DHT22#Temp]&field2=[SDS011#PM10].

Thanks to this thread, I found that version 20190809 handles the SendToHTTP requests correctly, so I decided to downgrade my ESP.
I hope that the mentioned problems will be solved soon.

TD-er
Core team member
Posts: 2791
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: obscure problem with sendtohtttp (and thingspeak)

#10 Post by TD-er » 15 Dec 2019, 16:22

MarcelM wrote:
15 Dec 2019, 15:50
Same obscure problem here with version 20191130 en 20191208 (normal and test version).
I want to sends a serie of 8 fields in one SendToHTTP request using Rules.
It makes no difference using quotes or no quotes, that is not the problem here.
On my local TcpListener tool, the GET request is send correctly. Also the HTTP headers look good, except the Accept: */* is missing, I don't know if it is a problem.

I also tried [Generic HTTP Advanced controller] in the newest test version 20191208. This one operates fine, it allows also to send HTTP POST requests and you can manipulate HTTP headers.
But, this controller can only send values from one task, thus max 4 values. It does not support syntax like field1=[DHT22#Temp]&field2=[SDS011#PM10].

Thanks to this thread, I found that version 20190809 handles the SendToHTTP requests correctly, so I decided to downgrade my ESP.
I hope that the mentioned problems will be solved soon.
Do you have a sample of the request which has the ?

Code: Select all

Accept: */*
I think you meant this header: https://developer.mozilla.org/en-US/doc ... ers/Accept
Apparently if no Accept: field is present, then it is assumed that text/plain and text/html are accepted.
What is the returned type when sending to Thingspeak?
But, this controller can only send values from one task, thus max 4 values.
You mean in one request?

MarcelM
Normal user
Posts: 16
Joined: 15 Dec 2019, 14:41

Re: obscure problem with sendtohtttp (and thingspeak)

#11 Post by MarcelM » 15 Dec 2019, 22:49

You mean in one request?
Yes I mean in one request

Concering the SendToHTTP request.
I found out that there is no difference in the output data stream of ESP version 20191208 and version 20190809. They are both exactly the same (except for the User agent version date of course).
The old version works fine with api.thingspeak.com, while the new version does not. Below the dump of the http request on Tcplisten level:

GET /update?api_key=XXXXXXXXXXXXXXXX&field1=14.80&field2=51.70&field3=4.27&field4=14.24&field5=0.00 HTTP/1.1
Host: 192.168.1.11:3000
User-Agent: ESP Easy/20103/Aug 9 2019 02:17:37
Connection: close

Note that: 92.168.1.11:3000 is my own local TcpListener.
There is no "Accept:" header, probably not necessary. The old ESP version works fine without Accept header.

I think it is a timing problem. The new ESP version closes the http connection directly as soon as the call has been executed.
For local network connections ok, but not for long distance connections / slow servers.

TD-er
Core team member
Posts: 2791
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: obscure problem with sendtohtttp (and thingspeak)

#12 Post by TD-er » 15 Dec 2019, 23:17

Can you test the test build linked here: https://github.com/letscontrolit/ESPEasy/pull/2820

This one includes an Accept header.

TD-er
Core team member
Posts: 2791
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: obscure problem with sendtohtttp (and thingspeak)

#13 Post by TD-er » 16 Dec 2019, 00:20

I also added a new commit to allow the SendToHttp command to wait for an acknowledgement.

For a new test build, see https://www.dropbox.com/s/azwe162gmxptl ... 0.zip?dl=0
See the commit: https://github.com/letscontrolit/ESPEasy/pull/2820

User avatar
iron
Normal user
Posts: 108
Joined: 24 Sep 2016, 08:37
Location: Veria, Greece
Contact:

Re: obscure problem with sendtohtttp (and thingspeak)

#14 Post by iron » 16 Dec 2019, 09:28

TD-er wrote:
16 Dec 2019, 00:20
I also added a new commit to allow the SendToHttp command to wait for an acknowledgement.

For a new test build, see https://www.dropbox.com/s/azwe162gmxptl ... 0.zip?dl=0
See the commit: https://github.com/letscontrolit/ESPEasy/pull/2820
I just tested 20191208-17-PR_2820
Double Quotes work with or without "Tolerant last parameter" enabled
Single quote works only with Tolerant last parameter" enabled

I also tested the parsed string on the previous version (mega-20191208) that did not work and the parsed string returns OK to the server so it must had been something in the "Accept header and/or ACK" that you introduced in this release.

Perhaps we should also document the need for quotes at

https://espeasy.readthedocs.io/en/latest
https://espeasy.readthedocs.io/en/lates ... g-examples

Thank you
-D
-D

MarcelM
Normal user
Posts: 16
Joined: 15 Dec 2019, 14:41

Re: obscure problem with sendtohtttp (and thingspeak)

#15 Post by MarcelM » 16 Dec 2019, 12:20

I also just just tested the new version 2820. Great!!
It communicates correct with api.thinkspeak.com.
I saw also that Accept has been added, see my TCPlistener below.

GET /update?api_key=XXXXXXXXXXXXXXXX&field1=20.40&field2=49.90&field3=96.67&field4=0 HTTP/1.1
Host: 192.168.1.11:3000
Accept: */*;q=0.1
User-Agent: ESP Easy/20104/Dec 16 2019 00:46:46
Connection: close

Problem solved, thanks!

MarcelM
Normal user
Posts: 16
Joined: 15 Dec 2019, 14:41

Re: obscure problem with sendtohtttp (and thingspeak)

#16 Post by MarcelM » 16 Dec 2019, 16:48

Great that SendToHttp in version 2820 has been Solved.

I am testing now the ESP MQTT interface with Thingspeak.
I found that the ESP MQTT interface has a similar problem. The MQTT messages are not received by Thingspeak.
I send the following message with Rules to MQTT, using the ESP (openHAB) MQTT controller to set the server, user and password.

Publish channels/931292/publish/XXXXXXXXXXXXXXXX,&field1=[DHT22#Temp]&field2=[DHT22#Hum]&field3=[Light#Lux]&field4=[PIR#State]&status=MQTTPUBLISH

The message is correctly received in my local MQTT Mosquitto server. I use the MQTT.fx to analyze the message. Both topic and data are ok.
While sending to mqqt.thingspeak.com nothing is received.

When I send the same message (topic and data) from my local MQTT.fx client to mqtt.thingspeak.com, the message is received correctly by Thingspeak.

Conclusion, I think that the ESP MQTT connection is closed to fast after the executing the "Publish" command
Possible is this a similar problem as SendToHttp?

TD-er
Core team member
Posts: 2791
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: obscure problem with sendtohtttp (and thingspeak)

#17 Post by TD-er » 16 Dec 2019, 21:27

MQTT messages are sent via the PubSubClient library.
This should keep the connection open, so whatever the problem may be, it is not the same.

You could try to increase the timeout. I think the default is 100 msec, which may be a bit too low for online services.
Just increase it to something about 300 - 1000 msec.
Usually 3x the average ping time is fine.

MarcelM
Normal user
Posts: 16
Joined: 15 Dec 2019, 14:41

Re: obscure problem with sendtohtttp (and thingspeak)

#18 Post by MarcelM » 16 Dec 2019, 22:59

Thanks
I have the timeout on max 1000ms. In advanced settings the message interval at 15000ms.
I tried several other settings without any success.

With the MQTT.fx client, I can send MQTT messages to Thingsspeak without any problem, with QoS = 0 and retained = false
Can you see the QoS and the Retained settings in the ESP PubSubClient library?

The ESP logwindow says:
275999: ACT : Publish channels/931292/publish/XXXXXXXXXXXX,field1=18.10&field2=54.80&field3=0.00&field4=0&status=MQTTPUBLISH
276002: Command: Publish
276006: ACT : timerset,1,60
276007: Command: timerset
276024: MQTT : Intentional reconnect
276508: MQTT : Broker C005 connection failed (12/0)

User avatar
ThomasB
Normal user
Posts: 696
Joined: 17 Jun 2018, 20:41
Location: USA

Re: obscure problem with sendtohtttp (and thingspeak)

#19 Post by ThomasB » 17 Dec 2019, 00:52

With the MQTT.fx client, I can send MQTT messages to Thingsspeak without any problem, with QoS = 0 and retained = false
Can you see the QoS and the Retained settings in the ESP PubSubClient library?
The Controller is hardcoded with QoS=0. MQTT Retain Msg is a user setting in Tools=>Advanced.

- Thomas

TD-er
Core team member
Posts: 2791
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: obscure problem with sendtohtttp (and thingspeak)

#20 Post by TD-er » 17 Dec 2019, 08:47

ThomasB wrote:
17 Dec 2019, 00:52
The Controller is hardcoded with QoS=0. MQTT Retain Msg is a user setting in Tools=>Advanced.
Maybe this should be moved to the controller settings?

User avatar
ThomasB
Normal user
Posts: 696
Joined: 17 Jun 2018, 20:41
Location: USA

Re: obscure problem with sendtohtttp (and thingspeak)

#21 Post by ThomasB » 17 Dec 2019, 17:42

I send the following message with Rules to MQTT, using the ESP (openHAB) MQTT controller to set the server, user and password.
Do you mean you are using the OpenHAB MQTT Controller and Publish rules to speak to Thingspeak? I didn't know that was supported, so learned something new here. Be sure to set the MQTT controller's Check Reply: Check Acknowledgement (as required by Thingspeak).
Maybe this should be moved to the controller settings?
Not a critical thing, but would be a nice touch.

- Thomas

User avatar
iron
Normal user
Posts: 108
Joined: 24 Sep 2016, 08:37
Location: Veria, Greece
Contact:

Re: obscure problem with sendtohtttp (and thingspeak)

#22 Post by iron » 17 Dec 2019, 18:55

ThomasB wrote:
17 Dec 2019, 17:42
I send the following message with Rules to MQTT, using the ESP (openHAB) MQTT controller to set the server, user and password.
Do you mean you are using the OpenHAB MQTT Controller and Publish rules to speak to Thingspeak? I didn't know that was supported, so learned something new here. Be sure to set the controller's Check Reply: Check Acknowledgement (as required by Thingspeak).
Maybe this should be moved to the controller settings?
Not a critical thing, but would be a nice touch.

- Thomas
The majority (if not all) of the Home Assistant and openHAB MQTT implementations is Mosquitto so it is standard MQTT broker although called "Home Assistant (openHAB) MQTT" in the ESPEasy menu.

I have it connected to cloudmqtt and Kappelt as well.

-D
-D

MarcelM
Normal user
Posts: 16
Joined: 15 Dec 2019, 14:41

Re: obscure problem with sendtohtttp (and thingspeak)

#23 Post by MarcelM » 17 Dec 2019, 22:23

Do you mean you are using the OpenHAB MQTT Controller and Publish rules to speak to Thingspeak? I didn't know that was supported, so learned something new here. Be sure to set the controller's Check Reply: Check Acknowledgement (as required by Thingspeak).
Yes,I have to do so.
Or is there another way to specify a MQTT broker.
I am forced to choose a predefined MQTT controller (Domoticz, OpenHAB or PiDome) in order to specify a MQTT broker with credentials. But I don't want their predefined messages.

May be I miss the point.
With the MQTT “Publish” command in Rules, you can compose your own topics and messages. I like this command, but I don't know where to specify the Broker associated with the Publish command.

User avatar
ThomasB
Normal user
Posts: 696
Joined: 17 Jun 2018, 20:41
Location: USA

Re: obscure problem with sendtohtttp (and thingspeak)

#24 Post by ThomasB » 18 Dec 2019, 01:36

To get a better understanding of the issue, I setup thingspeak on the OpenHAB MQTT controller. After three hours of poking around I had to abandon the investigation (ran out of free time).

I tested on the latest Mega PR (mega-20191208). I confirmed that the MQTT client does not successfully connect to the mqtt.thingspeak.com broker. As a Hail Mary experiment I changed the Controller Hostname entry to tcp://mqtt.thingspeak.com and the ESPEasy weblog's MQTT: Failed To Connect to Broker error went away. But no joy. Just to be clear, adding a "tcp://" preamble was a desparate thing to try but is improper host name syntax. So don't do it.

At this point my only hunch is that perhaps the MQTT controller's 1000mS max client timeout is too short. But that's just a wild guess.

Long story short, the Openhab MQTT controller does not seem to be compliant with thingspeak. Hopefully a MQTT controller guru shows up and takes a look.

- Thomas

TD-er
Core team member
Posts: 2791
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: obscure problem with sendtohtttp (and thingspeak)

#25 Post by TD-er » 18 Dec 2019, 09:40

MarcelM wrote:
17 Dec 2019, 22:23
Yes,I have to do so.
Or is there another way to specify a MQTT broker.
I am forced to choose a predefined MQTT controller (Domoticz, OpenHAB or PiDome) in order to specify a MQTT broker with credentials. But I don't want their predefined messages.

May be I miss the point.
With the MQTT “Publish” command in Rules, you can compose your own topics and messages. I like this command, but I don't know where to specify the Broker associated with the Publish command.
It is a bit of a hack, but the Publish command does need an enabled MQTT controller (just any), as it uses the settings of the "first enabled MQTT controller".

Maybe tingspeak's MQTT controller does have some kind of redirect and does seem to change IP address based on locality?
Each controller does have the option to use DNS for the server it needs to connect to.
But to overcome the blocking call to a DNS query, it does cache the IP-address.
This cache is not based on the TTL value, as I cannot get hold of that value, but it does cache it until it is unable to connect to the host.
What if it can resolve the DNS to an IP and that host does appear to be online, but not accepting a connection as it should redirect?
Then this caching does not work. Also I wonder if pubsubclient can handle redirects.

So maybe you can find some hostname for a local MQTT broker?

MarcelM
Normal user
Posts: 16
Joined: 15 Dec 2019, 14:41

Re: obscure problem with sendtohtttp (and thingspeak)

#26 Post by MarcelM » 18 Dec 2019, 12:18

@ThomasB,
The poking around for hours I recognize. Because sometimes I had a unique situation that only one message did arrive, I was happy for a short moment, but could not reproduce the situation afterwards.

@TD-er
I think the implementation of the ESP MQTT controller has been made too complex, probably to prevent blocking.
A simple nodejs code snippet communicates seamlessly with Thinsgspeak, also NodeRed and MQTT.fx do so.

Thanks, I stop MQTT testing for the moment.
I will use SendToHttp command in stead, that problem has been solved now.

User avatar
ThomasB
Normal user
Posts: 696
Joined: 17 Jun 2018, 20:41
Location: USA

Re: obscure problem with sendtohtttp (and thingspeak)

#27 Post by ThomasB » 18 Dec 2019, 18:25

I'm not ready to give up on the fight. So more investigation today.
Maybe tingspeak's MQTT controller does have some kind of redirect and does seem to change IP address based on locality?
I suspected issues with the domain/DNS too. So I also tested using the thingspeak MQTT broker's IP address rather than domain name. Pinging found that there's several available IP's, but periodic tests with MQTTlens client has shown that the IP I use remains valid during the ongoing tests.

@TD-er: I found an issue that will prevent the MQTT connection to ThingSpeak. Their Publish troubleshooting guide (https://www.mathworks.com/help/thingspe ... blish.html) specifically states:
Do not set CleanSession flag to 0. If you set the CleanSession flag to 0 on your MQTT client, any connection request to the ThingSpeak™ MQTT broker is rejected with a CONNACK status of "0x05 Connection Refused, not authorized".
Unfortunately, the MQTT controller code sets the CleanSession flag to 0. I see this was a workaround (applied last May) for a PubSubClient bug, per https://github.com/knolleary/pubsubclient/issues/458.

The previous PR's did not specify the CleanSession value, so they fail too. It would be grand to have a [Testing] PR with CleanSession=true. I'm currently not setup to compile, so I'm hoping for someone to build a [Testing] bin for me to try.

- Thomas

MarcelM
Normal user
Posts: 16
Joined: 15 Dec 2019, 14:41

Re: obscure problem with sendtohtttp (and thingspeak)

#28 Post by MarcelM » 19 Dec 2019, 00:00

nice that you find a clue.
Indeed in MQTT.fx client the CleanSession is also set to true.

User avatar
ThomasB
Normal user
Posts: 696
Joined: 17 Jun 2018, 20:41
Location: USA

Re: obscure problem with sendtohtttp (and thingspeak)

#29 Post by ThomasB » 19 Dec 2019, 00:17

I'm currently not setup to compile, so I'm hoping for someone to build a [Testing] bin for me to try.
Reply to self: I was able to get Platformio to behave, so I have the ability to compile ESPEasy again.
nice that you find a clue.
I was hoping that the solution would be as easy as setting the cleanSession flag. But doing that did not bring any joy. I also tried longer client timeouts and still cannot connect to the thingspeak broker.

I added log debug messages that allow me to confirm the connection parameters; Everything on my side looks valid.

At the moment I'm at a standstill. Maybe someone has an idea on what could be preventing the connection. So toss your ideas at me and I will see what I can do.

- Thomas

User avatar
ThomasB
Normal user
Posts: 696
Joined: 17 Jun 2018, 20:41
Location: USA

Re: obscure problem with sendtohtttp (and thingspeak)

#30 Post by ThomasB » 19 Dec 2019, 01:40

Good news, I got it working. The basic patch affects one file, Controller.ino.

The LWTTopic & LWTMessageDisconnect strings must be NULL. Also cleanSession must be updated as discussed earlier. See below:

Code: Select all

boolean cleanSession = true;
MQTTresult = 
	MQTTclient.connect(clientid.c_str(),
        SecuritySettings.ControllerUser[controller_idx],
        SecuritySettings.ControllerPassword[controller_idx],
        //LWTTopic.c_str(),
        0,
        willQos,
        willRetain,
        //LWTMessageDisconnect.c_str(),
        0,
        cleanSession);
Of course this patch breaks OpenHAB compatibility. So if it was added to an official PR then a more graceful patch is needed that allows the user to select this special mode. Could be as simple as a controller checkbox that says "Use ThingSpeak Compatibility mode."

I'm not a ThingSpeak user so the patch isn't something I need. I'm passsing the info to those that want a fix and have the ability to self compile.

- Thomas

User avatar
grovkillen
Core team member
Posts: 3607
Joined: 19 Jan 2017, 12:56
Location: Hudiksvall, Sweden
Contact:

Re: obscure problem with sendtohtttp (and thingspeak)

#31 Post by grovkillen » 19 Dec 2019, 09:52

I just talked to TD-er, he will make this patch official later today or tomorrow.
ESP Easy Flasher [flash tool and wifi setup at flash time]
ESP Easy Webdumper [easy screendumping of your units]
ESP Easy Netscan [find units]
Official shop: https://firstbyte.shop/
Sponsor ESP Easy, we need you :idea: :idea: :idea:

TD-er
Core team member
Posts: 2791
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: obscure problem with sendtohtttp (and thingspeak)

#32 Post by TD-er » 19 Dec 2019, 14:52

Good find here.
I will add a few flags to the MQTT controller settings, to address the parts of not sending some LWT strings and clean session.

It will be a bit larger change than a single file, but it will keep the compatibility for all MQTT controllers.


MarcelM
Normal user
Posts: 16
Joined: 15 Dec 2019, 14:41

Re: obscure problem with sendtohtttp (and thingspeak)

#34 Post by MarcelM » 19 Dec 2019, 17:41

Nice work :!:
How can I test 2825

User avatar
ThomasB
Normal user
Posts: 696
Joined: 17 Jun 2018, 20:41
Location: USA

Re: obscure problem with sendtohtttp (and thingspeak)

#35 Post by ThomasB » 19 Dec 2019, 17:59

I just talked to TD-er, he will make this patch official later today or tomorrow.
@grovkillen & @TD-er: Thank you both for your ongoing development work and helping out with this issue. Merry Christmas!

- Thomas

User avatar
grovkillen
Core team member
Posts: 3607
Joined: 19 Jan 2017, 12:56
Location: Hudiksvall, Sweden
Contact:

Re: obscure problem with sendtohtttp (and thingspeak)

#36 Post by grovkillen » 19 Dec 2019, 18:40

ThomasB wrote:
19 Dec 2019, 17:59
I just talked to TD-er, he will make this patch official later today or tomorrow.
@grovkillen & @TD-er: Thank you both for your ongoing development work and helping out with this issue. Merry Christmas!

- Thomas
Thank you and the community for keeping us on our toes :)
ESP Easy Flasher [flash tool and wifi setup at flash time]
ESP Easy Webdumper [easy screendumping of your units]
ESP Easy Netscan [find units]
Official shop: https://firstbyte.shop/
Sponsor ESP Easy, we need you :idea: :idea: :idea:

TD-er
Core team member
Posts: 2791
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: obscure problem with sendtohtttp (and thingspeak)

#37 Post by TD-er » 19 Dec 2019, 21:37

MarcelM wrote:
19 Dec 2019, 17:41
Nice work :!:
How can I test 2825
I just started to make a test build for this PR.


User avatar
ThomasB
Normal user
Posts: 696
Joined: 17 Jun 2018, 20:41
Location: USA

Re: obscure problem with sendtohtttp (and thingspeak)

#39 Post by ThomasB » 19 Dec 2019, 22:45

I just installed the 2825 PR (via OTA) and now my NodeMCU won't boot. That's all I know at the moment, I'll dig deeper into it later today.

Proceed with caution.

EDIT: Boot loop problem. Snippet from the serial log:

Code: Select all

INIT : Booting version:  (ESP82xx Core 2_6_2, NONOS SDK 2.2.2-dev(38a443e), LWIP: 2.1.2 PUYA support)
69 : Info  : INIT : Free RAM:30480
70 : Info  : INIT : Warm boot #63 Last Task: Background Task - Restart Reason: Exception
72 : Info  : FS   : Mounting...
97 : Info  : FS   : Mount successful, used 77810 bytes of 957314
550 : Info  : CRC  : program checksum       ...OK�U69 : Info  : 

INIT : Booting version:  (ESP82xx Core 2_6_2, NONOS SDK 2.2.2-dev(38a443e), LWIP: 2.1.2 PUYA support)
70 : Info  : INIT : Free RAM:30480
71 : Info  : INIT : Warm boot #64 Last Task: Background Task - Restart Reason: Exception
73 : Info  : FS   : Mounting...
98 : Info  : FS   : Mount successful, used 77810 bytes of 957314
551 : Info  : CRC  : program checksum       ...OK�U69 : Info  : 
I used serial flash and reverted back to the previous version. So no harm done.

- Thomas

TD-er
Core team member
Posts: 2791
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: obscure problem with sendtohtttp (and thingspeak)

#40 Post by TD-er » 19 Dec 2019, 23:07

Hmm that's strange.
I did upload it to a node I have, so it was tested slightly more than "it compiles".
Will have a look at it tomorrow, but would be great if some people could test it also with a node that can be flashed over serial.

MarcelM
Normal user
Posts: 16
Joined: 15 Dec 2019, 14:41

Re: obscure problem with sendtohtttp (and thingspeak)

#41 Post by MarcelM » 20 Dec 2019, 11:55

I tested PR 2825 on ESP8266 Wemos d1 mini.
I flashed it with serial FlashESP8266.exe
No problems here.

MQTT with Thinsgpeaks works, great!!!

TD-er
Core team member
Posts: 2791
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: obscure problem with sendtohtttp (and thingspeak)

#42 Post by TD-er » 20 Dec 2019, 13:50

Did you create a new controller setting after flashing?
I assume Thomas has had some settings present when testing.

MarcelM
Normal user
Posts: 16
Joined: 15 Dec 2019, 14:41

Re: obscure problem with sendtohtttp (and thingspeak)

#43 Post by MarcelM » 20 Dec 2019, 14:16

after flashing, I enabled the MQTTT Domoticz controller and sets the new settings:
Will Retain: false
Clean Session true

And of course I changed code in the Rules from:

SendToHTTP api.thingspeak.com,80,/update?api_key=XXXX&field1=[DHT22#Temp]&field2=[DHT22#Hum]&field3=[Light#Lux]&field4=[PIR#State]

to
Publish channels/931292/publish/XXX,field1=[DHT22#Temp]&field2=[DHT22#Hum]&field3=[Light#Lux]&field4=[PIR#State]&status=MQTTPUBLISH

TD-er
Core team member
Posts: 2791
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: obscure problem with sendtohtttp (and thingspeak)

#44 Post by TD-er » 20 Dec 2019, 14:18

OK, so the problem experienced by Thomas could be related to settings not correct initialized.
Even when the settings are not valid, (random data), the ESP should not crash on it.

User avatar
ThomasB
Normal user
Posts: 696
Joined: 17 Jun 2018, 20:41
Location: USA

Re: obscure problem with sendtohtttp (and thingspeak)

#45 Post by ThomasB » 20 Dec 2019, 17:32

I've done more evaluation to sniff out the problem. I tried with the MQTT controller disabled and with it removed/deleted. In all cases the OTA flash ends with the boot loop.

Perhaps my bad results are related to the firmware bin and enabled plugins. Here's a short summary:
* Firmware bin: ESP_Easy_mega-20191208-19-PR_2825_test_ESP8266_4M1M_VCC (also tried ESP_Easy_mega-20191208-19-PR_2825_normal_ESP8266_4M1M)
* Installed devices: Switch Input, Gyro-MPU 6050 (disabled), Display-Nextion (swapped serial), Generic-System Info
* Board: Lolin NodeMCU V3
* Other: NTP enabled, old rules engine, Force WiFi B/G

Which PR_2825 firmware bin file are you guys using? What are your devices?

- Thomas

TD-er
Core team member
Posts: 2791
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: obscure problem with sendtohtttp (and thingspeak)

#46 Post by TD-er » 20 Dec 2019, 20:33

Well, I have not used the bin from that zip, but flashed right from the IDE and later created the test build based on the PR.
I can test with uploading the bin itself.

User avatar
ThomasB
Normal user
Posts: 696
Joined: 17 Jun 2018, 20:41
Location: USA

Re: obscure problem with sendtohtttp (and thingspeak)

#47 Post by ThomasB » 21 Dec 2019, 00:52

Well, I have not used the bin from that zip, but flashed right from the IDE
That seemed like a good thing to try. So I self-compiled PR_2825 using test_ESP8266_4M1M build configuration. Flashed it directly from PIO's IDE. The dreaded boot loop still occurs.

I disabled the Nextion plugin so I could use serial log. Here's the boot messages:

Code: Select all

ets Jan  8 2013,rst cause:2, boot mode:(3,7)

load 0x4010f000, len 1384, room 16 
tail 8
chksum 0x2d
csum 0x2d
vbc204a9b
~ld
�U69 : Info  : 

INIT : Booting version:  (ESP82xx Core 2_6_2, NONOS SDK 2.2.2-dev(38a443e), LWIP: 2.1.2 PUYA support)
70 : Info  : INIT : Free RAM:30480
71 : Info  : INIT : Warm boot #1 Last Task: Background Task - Restart Reason: Exception
73 : Info  : FS   : Mounting...
98 : Info  : FS   : Mount successful, used 77810 bytes of 957314
550 : Info  : CRC  : program checksum       ...OK
587 : Info  : CRC  : SecuritySettings CRC   ...OK
587 : Info  : CRC  : binary has changed since last save of Settings
667 : Info  : INIT : Free RAM:27392
668 : Info  : INIT : I2C
668 : Info  : INIT : SPI not enabled
951 : Info  : INFO : Plugins: 79 [Normal] [Testing] (ESP82xx Core 2_6_2, NONOS SDK 2.2.2-dev(38a443e), LWIP: 2.1.2 PUYA support)
953 : Info  : EVENT: System#Wake
fpm close 3 
1212 : Info  : WIFI : Set WiFi to STAmode : sta(68:c6:3a:d6:f1:a9)
add if0

1245 : Info  : WIFI  : Start network scan
1246 : Info  : WIFI : Connecting MYROUTER attempt #0Fatal exception 9(LoadStoreAlignmentCause):
epc1=0x40292c00, epc2=0x00000000, epc3=0x402bbc7d, excvaddr=0x3fff1a2e, depc=0x00000000

Exception (9):
epc1=0x40292c00 epc2=0x00000000 epc3=0x402bbc7d excvaddr=0x3fff1a2e depc=0x00000000

>>>stack>>>

ctx: cont
sp: 3fff2e20 end: 3fff30a0 offset: 01a0
3fff2fc0:  3fff16bc 3fff6924 3fff0eac 3fff16dc
3fff2fd0:  3fff16bc 3fff169c 00000020 40101873
3fff2fe0:  3fff16bc 3fff6924 00000002 40235728  
3fff2ff0:  40293620 00000000 40293620 00000000
3fff3000:  40293620 00000000 40293620 00000000
3fff3010:  3fff16bc 3fff169c 3fff0c50 4024f27c  
3fff3020:  3fff6900 0028002f 80ff3050 40285a9d
3fff3030:  402e8b24 3ffe8724 3fff1a24 3fff1f64
3fff3040:  3fff0ea9 3ffe8724 3fff1a24 402704e0  
3fff3050:  3fff6300 000b000f 8029005d 40291e64
3fff3060:  3fff63e4 0072007f 00efeffe feefeffe
3fff3070:  feefeffe feefeffe feefeffe 3fff30e8  
3fff3080:  3fffdad0 00000000 3fff30a8 402874fc
3fff3090:  feefeffe feefeffe 3ffe875c 401007d9
<<<stack<<<

I expect that the problem will go away if I clear the flash and start over from a factory fresh state. But hopefully we can find a way to avoid that in case the boot loop problem affects other users.

I don't have a dog in this fight (not a thingspeak user). But I'm available to help out. So if you need me to do any special troubleshooting then let me know.

- Thomas

MarcelM
Normal user
Posts: 16
Joined: 15 Dec 2019, 14:41

Re: obscure problem with sendtohtttp (and thingspeak)

#48 Post by MarcelM » 21 Dec 2019, 22:38

I used ESPEasy_mega-20191208-19-PR_2825.zip downloaded from:
[quot]
A test build for this PR: https://www.dropbox.com/s/iazr1ou3dmenf ... 5.zip?dl=0
[/qout]

I flashed the file ESP_Easy_mega-20191208-19-PR_2825_normal_ESP8266_4M1M.bin from the bin directory.
no problems with Wemos D1 mini

User avatar
iron
Normal user
Posts: 108
Joined: 24 Sep 2016, 08:37
Location: Veria, Greece
Contact:

Re: obscure problem with sendtohtttp (and thingspeak)

#49 Post by iron » 22 Dec 2019, 17:51

I confirm no issues updating a test device from the 20191208-17-PR_2820 to 20191208-19-PR_2825

-D
-D

User avatar
iron
Normal user
Posts: 108
Joined: 24 Sep 2016, 08:37
Location: Veria, Greece
Contact:

Re: obscure problem with sendtohtttp (and thingspeak)

#50 Post by iron » 22 Dec 2019, 18:30

Unfortunately sendtohttp (thingspeak) does not work for me now no matter the "SendToHTTP wait for ack:" is enabled or not.
I tested with double quotes, either "Tolerant last parameter:" is enabled or not.

-D
-D

Post Reply

Who is online

Users browsing this forum: No registered users and 17 guests