How to send Sensor Data from Easy ESP to my localhost with generic HTTP?

[groovy]

replace %bmp085% with %tskname% and name your task "bmp085", because %bmp085% is no recognized variable, but %tskname% is.

I'm new to this firmware too, just trying to get it work. Additionally, I'm no programmer, so I can't help with your backend.

[rai]

@groovy

replace %bmp085% with %tskname% and name your task "bmp085", because %bmp085% is no recognized variable, but %tskname% is.

that's what i did. my BMP085 sensor has the taskname bmp085!

sorry groovy, but from my point of view this can't be, because what happened if you have more than 1 task, like i do? I have 3 devices = 3 tasks:

display - this is task 1 with task name display
bmp085 - this is task 2 with task name bmp085
si7021 - this is task 32 with task name si7021

How can ESP Easy assign to the 3 different tasks 3 different values with one and the same %tskname% variable?

[rai]

sorry,
but i don't make any progress with my application because i cannot figure out how ESP Easy is naming/renaming the task- and value names behind the scenes. As result of that, i cannot compose the correct URL in menu "Tools" "Advanced" "Publish Template:". If i do not know the correct "Publish Template:" string with the included and correct named variable and value names, i cannot call up from my localhost the corresponding name/value pairs with my PHP $_GET[""] function.

is there really no one out there who can tell me how to use correctly the ESP Easy Generic HTTP settings/protocol for sending sensor data to a localhost, and how to request this sensor data from the localhost with PHP scripting langugae to store the requested data on the localhost? Maybe one of the brilliant developer of Easy ESP?

If i have a look to the number of views of my thread so far, i guess there must be a common interest in this matter.?.
Many thanks for all answers so far, especially to groovy. Still waiting for a practical solution.

[Martinus]

Using the default template:

Code: Select all

demo.php?name=%sysname%&task=%tskname%&valuename=%valname%&value=%value%

On an ESP with name "ESPDemo" and running a DHT11 sensortask with name "DHT11" and default valuenames,
this would result in two http requests fired at your controller:

Code: Select all

demo.php?name=ESPDemo&task=DHT11&valuename=Temperature&value=21
demo.php?name=ESPDemo&task=DHT11&valuename=Humidity&value=65

Your demo.php file should process the valuepairs provided in the querystring and do something with them, like storing to a MySQL database.
Remember that each sensorvalue is processed as a single request. A BME280 would send 3 requests.

[rai]

@Martinus
Thanks for your answer. But if i adapt your example URL to my speficic application, i cannot call the data with a http request from my localhost with a PHP file. I guess i do not understand the "caller/receiver" concept of ESPs "Generic HTTP" protocol.

@all
How i have explained in detail in the very beginning of this thread, i have another sensor (Si7021) and a OLED-Display (SH1106 1.3") connected to my ESP, and all values are displayed correctly on the OLED!. But to make it short and easy for a helpful answer concening my"Generic HPPT" localhost problem, following i only show once again my setup for just one sensor, namely for the sensor BMP085:

- in menu "Config" in field "Name" i wrote "buero" - so my ESP is named buero, and the IP address from my ESP is 192.168.178.66!
- in menu "Config" in select field "Protocol:" i choosed "Generic HTTP".
- in menu "Config" in select field "Locate Controller:" i choosed "Use IP address".
- in menu "Config" in field "Controller IP:" i wrote "192.168.178.21", which is the IP-Address from my localhost!

- in menu "Devices" i made my sensor settings for "Temperature & Pressure - BMP085" as following:
Device: "Temperature & Pressure - BMP085"
Name: bmp085
Value Name 1: temp (for the Temperature)
Value Name 2: press (for the air pressure)
For both fields i choosed "Decimals: 1"

- in menu "Tools" "Advanced" in field "Publis Template:" i tried to adapt "Martinus" default URL as follwing:
update.php?name=%sysname%&task=%tskname%&valuename=%valname%&value=%value%

Then i wrote in PHP a short "update.php" script to get the valuepairs out of the query string send by my ESP "buero" for further data processing and storing the data to a MySQL database or to a *.csv file as following:

Code: Select all

<?php
//refresh the sensor data every 5 seconds
echo '<meta http-equiv="refresh" content="5">';

// my ESP variable for the temperature is "temp", so grab the temperature value from this variable
$temp = $_GET["temp"];
// show me the temperature
echo $temp;

// my ESP variable for the air pressure is "press", so grab the air pressure value from this variable
$press = $_GET["press"];
// show me the air pressure
echo $press;
?>

This very simple "update.php" file i placed for test purposes only into the root of my localhost (IP address 192.168.178.21) and i try to get my sensor data by calling the URL "192.168.178.21/update.php". This doesn't work at all. I do not get any server error message, but i do not get any sensor data as well.

@ Martinus and @ all:

I guess i missed the whole "Generic HTTP" ESP Easy protocol concept on how ESP Easy is sending the sensor data to the localhost and how the localhost then will request and receive the requested sensor data.
So who ever want to help me please take for your answer, instead of pseudo code, my above mentioned ESP system name with my a. m. ESP sensor and value names to show me the correct "Publish Template:" URL string. And please don't forget to show me a short example script for my PHP file to make a correct request to my ESP for the sensor data.

If i not only can display, but store the data values as well on my localhost for further data mining and data graphing, the Easy ESP framework would be the solution for my application. Please help to get me running, and for that many thanks in advance!

[groovy]

First of all, set the "web log level" under tools in the advanced section to 4 and see, what the resulting URL-string actually is.

[rai]

@ groovy
thanks, but what do you mean with

First of all, set the "web log level" under tools in the advanced section to 4 and see, what the resulting URL-string actually is.

As you proposed I set the "web log level" to 4, but i cannot see any changes in the URL-String at all. Where and how should i see a change in the resulting URL?
The URL from my ESP stays always the same, namely at 192.168.178.66, and it only changes with the change of menu position, like:

- http://192.168.178.66/ -> if i click on the "Main" menu button
- http://192.168.178.66/config -> if i click on the "Config" menu button
- http://192.168.178.66/devices -> if i click on the "Devices" menu button
- http://192.168.178.66/tools-> if i click on the "Tools" menu button
- http://192.168.178.66/advanced -> if i click on the "Advanced" menu button

There is no visible URL from my http://192.168.178.66 ESP-system with any name/value pairs which sends any data to my http://192.168.178.21 localhost server, what raises for me the question:

does the word "Controller" in menu "Config" really mean my localhost server (192.168.178.21) or is it the name for the ESP itself with IP-address 192.168.178.66? If so, then i have to set the IP-address to my ESP IP-address 192.168.178.66, instead of IP-address 192.168.178.21 for my localhost?!
I tested that without any success by setting the "Controller" IP-address from 192.168.178.21 (localhost) to my ESP IP-address 192.168.178.66!
groovy: how did you setup your data request?

Beside the correct setup of my ESP "Publish Template" URL string, my main problem is the PHP script (e.g. update.php) to make the correct request to my ESP for the sensor data. Any helpful suggestions?

[groovy]

Setting the "web log level" to 4 and watching the log (tools -> log), you should see something like this:

Code: Select all

102334782 : DHT : Humidity: 38.70
102334782 : EVENT: 30#v1=26.60
102334782 : EVENT: 30#v2=38.70
102334783 : HTTP : Delay 976 ms
102335759 : HTTP : connecting to 192.168.2.81 using port 80
102335763 : /shc/index.php?app=shc&a&ajax=pushsensorvalues&spid=3&sid=30&type=2&v1=26.60
102335965 : HTTP : closing connection
102336965 : HTTP : connecting to 192.168.2.81 using port 80
102336968 : /shc/index.php?app=shc&a&ajax=pushsensorvalues&spid=3&sid=30&type=2&v2=38.70
102337170 : HTTP : closing connection

As you can see, the variables are translated to the right values and sent to the server.

[rai]

Hello again,

@ groovy
thanks for your advice

Setting the "web log level" to 4 and watching the log (tools -> log), you should see something like this:

Your help has taken me a step further!


To make things easier, i now installed a BME280 sensor, because this sensor gives me all the data i need, namely Temperature, Air Pressure, and Humidity. Now i only have to take care about 1 task! I left my "Config" settings as mentioned in my previous post, but changed my "Tools" "Advanced" "Publish Template:" setting to:

"esp_easy/demo.php?task=%tskname%&valuename=%valname%&value=%value%"

I had to change the path because i made some changes in my localhost file structure.

In the next step i've scripted following PHP cURL code

Code: Select all

<?php
echo '<meta http-equiv="refresh" content="1">';

// Get cURL resource
$curl = curl_init();
// Set some options - we are passing in a useragent too here
curl_setopt_array($curl, array(
    CURLOPT_RETURNTRANSFER => 1,
    CURLOPT_URL => 'http://192.168.178.66/esp_easy/demo.php?task=bme280&valuename=Temperature&value=value',
    CURLOPT_USERAGENT => 'Codular Sample cURL Request'
));
// Send the request & save response to $resp
$resp = curl_exec($curl);
echo $resp;
// Close request to clear up some resources
curl_close($curl);
?>

into my "demo.php" file on my localhost server (192.168.178.21), which sends a cURL request for my sensor data to my ESP Easy (192.168.178.66).

With the a. m. settings and with the a. m. PHP cURL script i now can see in my Easy ESP log all my sensor data send to my server 192.168.178.21 (localhost) as following:

151991 : HTTP : Delay 820 ms
152811 : HTTP : connecting to 192.168.178.21 using port 80
152816 : /esp_easy/demo.php?task=bme280&valuename=Temperature&value=23.6
153213 : HTTP : closing connection
154213 : HTTP : connecting to 192.168.178.21 using port 80
154240 : /esp_easy/demo.php?task=bme280&valuename=Humidity&value=50.7
154680 : HTTP : closing connection
155680 : HTTP : connecting to 192.168.178.21 using port 80
155685 : /esp_easy/demo.php?task=bme280&valuename=Pressure&value=1021.6
156146 : HTTP : closing connection

But how to get my sensor data out of my cURL script? The data should be in variable $resp (update.php)!?. But this variable only contains information about the cURL request like:

URI: /esp_easy/demo.php Method: GET Arguments: 3 NAME:task VALUE:bme280 NAME:valuename VALUE:Temperature NAME:value VALUE:value

I figured out, that the cURL request has to go back to my ESP Easy (192.168.178.66), and not to my localhost (192.168.178.21), as i thought before! As soon as i set the request in my "demo.php" to my localhost IP address 192.168.178.21 the "demo.php" page hangs up, the sensor data in my ESP log stops reading, and the apache server port 80 goes crazy, because the port changes constantly to undefined values!

Resume so far:

- I figured out the "Controller" and the "Publish Template:" settings
- i figured out a HTTP cURL request (demo.php) which keeps my ESP sending and my localhost pulling my sensor data

But i still don't know how to get my sensor data out of the URL send from my ESP (192.168.178.66) to my localhost (192.168.178.21)!

groovy,
i feel sorry, that i cannot help you with your problem. If i look into your thread, i can see that you have the same information problem like i have. More or less we are talking to ourself!
I really hope, that another user, or maybe one of the developers, will help to find a solution for my "Generic HTTP" problem. There must be a solution for sending ESP sensor data to a private server (localhost).

Who ever can help, i kindly ask: please help! Many thanks in advance.

[raptor]

Hello rai,

I will try to explain you in details how I get temperature and humidity values from my DHT11 sensor and write it to file or MySQL database on localhost.
First, go to config page and set these parameters

Untitled1.png (8.71 KiB) Viewed 37186 times

where Controller IP is your localhost ip, Controller Port is your localhost port and sensor delay to 60 sec. (in my case is 300 sec.)

Next, go to Tools->Advanced and set Publish Template like this:
esp/upload.php?name=%sysname%&task=%tskname%&valuename=%valname%&value=%value%
If your upload.php file is in C:/xampp/htdocs/esp_easy/ your publish template must be like this:
esp_easy/upload.php?name=%sysname%&task=%tskname%&valuename=%valname%&value=%value%

Reboot your module.

Go to localhost and create upload.php with this code:

<?php

//EXPERIMENTAL USE ONLY!

$date = date('l jS \of F Y h:i:s A');
error_log($date."\n", 3, "C:/xampp/htdocs/esp_easy/test_log.txt");

$valuename = $_GET['valuename'];
$value = $_GET['value'];

switch ($valuename) {
case 'Temperature':
$temp = $value;
error_log('Temperature - '.$temp.' C'."\n", 3, "C:/xampp/htdocs/esp_easy/test_log.txt");
break;
case 'Humidity':
$humidity = $value;
error_log('Humidity - '.$humidity.' %'."\n", 3, "C:/xampp/htdocs/esp_easy/test_log.txt");
break;
}

?>

This code get temp and humidity values from HTTP GET request and write them in test_log.txt

Now when esp module get sensor data it will make HTTP GET request to the upload.php on your localhost for every value of every sensor. I have one DHT11 sensor with two values (temperature and humidity), so when my module get sensor data it will send two HTTP GET request to upload.php one for temp and one for humidity. If you have 3 sensors with two values each you will have 6 HTTP GET requests to upload.php and etc...
The idea is that upload.php listen for HTTP GET requests with sensor values from Espeasy and when Espeasy make HTTP GET request to upload.php, upload.php get and processing this data.

This is how my Apache access log looks like:

192.168.100.2 - - [25/Oct/2016:21:26:32 +0300] "GET /esp/upload.php?name=SonoffExperiment&task=TempHumidity&valuename=Temperature&value=25.00 HTTP/1.1" 200 - "-" "-"
192.168.100.2 - - [25/Oct/2016:21:26:33 +0300] "GET /esp/upload.php?name=SonoffExperiment&task=TempHumidity&valuename=Humidity&value=61.00 HTTP/1.1" 200 - "-" "-"
192.168.100.2 - - [25/Oct/2016:21:31:33 +0300] "GET /esp/upload.php?name=SonoffExperiment&task=TempHumidity&valuename=Temperature&value=25.00 HTTP/1.1" 200 - "-" "-"
192.168.100.2 - - [25/Oct/2016:21:31:34 +0300] "GET /esp/upload.php?name=SonoffExperiment&task=TempHumidity&valuename=Humidity&value=62.00 HTTP/1.1" 200 - "-" "-"
192.168.100.2 - - [25/Oct/2016:21:36:32 +0300] "GET /esp/upload.php?name=SonoffExperiment&task=TempHumidity&valuename=Temperature&value=25.00 HTTP/1.1" 200 - "-" "-"
192.168.100.2 - - [25/Oct/2016:21:36:33 +0300] "GET /esp/upload.php?name=SonoffExperiment&task=TempHumidity&valuename=Humidity&value=62.00 HTTP/1.1" 200 - "-" "-"
192.168.100.2 - - [25/Oct/2016:21:41:33 +0300] "GET /esp/upload.php?name=SonoffExperiment&task=TempHumidity&valuename=Temperature&value=25.00 HTTP/1.1" 200 - "-" "-"
192.168.100.2 - - [25/Oct/2016:21:41:34 +0300] "GET /esp/upload.php?name=SonoffExperiment&task=TempHumidity&valuename=Humidity&value=62.00 HTTP/1.1" 200 - "-" "-"
192.168.100.2 - - [25/Oct/2016:21:46:33 +0300] "GET /esp/upload.php?name=SonoffExperiment&task=TempHumidity&valuename=Temperature&value=25.00 HTTP/1.1" 200 - "-" "-"
192.168.100.2 - - [25/Oct/2016:21:46:34 +0300] "GET /esp/upload.php?name=SonoffExperiment&task=TempHumidity&valuename=Humidity&value=61.00 HTTP/1.1" 200 - "-" "-"

You can see that espeasy make HTTP GET request for every value of my DHT11 sensor.

This is my test_log.txt file:

Tuesday 25th of October 2016 09:26:32 PM
Temperature - 25.00 C
Tuesday 25th of October 2016 09:26:33 PM
Humidity - 61.00 %
Tuesday 25th of October 2016 09:31:33 PM
Temperature - 25.00 C
Tuesday 25th of October 2016 09:31:34 PM
Humidity - 62.00 %
Tuesday 25th of October 2016 09:36:32 PM
Temperature - 25.00 C
Tuesday 25th of October 2016 09:36:33 PM
Humidity - 62.00 %
Tuesday 25th of October 2016 09:41:33 PM
Temperature - 25.00 C
Tuesday 25th of October 2016 09:41:34 PM
Humidity - 62.00 %
Tuesday 25th of October 2016 09:46:33 PM
Temperature - 25.00 C
Tuesday 25th of October 2016 09:46:34 PM
Humidity - 61.00 %

This is some images from my configuration:

Untitled2.png (28.12 KiB) Viewed 37186 times

Untitled3.png (21.46 KiB) Viewed 37186 times

Untitled4.png (20.65 KiB) Viewed 37186 times

I'm beginner in IoT world. I hope I've helped you in some way. (sorry for my english )

[sexton]

Thank you Raptor.Where there was a dark void the is now light.

[raptor]

I'm glad to be helpful

[rai]

Hello again,
since i was out of home for a couple of days i couldn't get back earlier to my thread. Sorry for that, but finally i found a sufficiant solution for me.

@ raptor
Thanks a lot for your answer. I could understand every word of your post, so your english is OK. Your answer really gives some clear application advices in this matter.

Anyway, i had already found my own solution, which i will herewith present and explain step by step.
First try:

My apache "access log" showed up the correct GET request as following:
192.168.178.66 - - [22/Oct/2016:15:26:53 +0200] "GET /webdev/myhome/update.php?sl=buero&s=bme280&st=Temperature&sv=21.9 HTTP/1.1" 200 1790 "-" "-"
192.168.178.66 - - [22/Oct/2016:15:26:54 +0200] "GET /webdev/myhome/update.php?sl=buero&s=bme280&st=Humidity&sv=51.7 HTTP/1.1" 200 1790 "-" "-"
192.168.178.66 - - [22/Oct/2016:15:26:56 +0200] "GET /webdev/myhome/update.php?sl=buero&s=bme280&st=Pressure&sv=1015.0 HTTP/1.1" 200 1790 "-" "-"

This "access.log" clearly shows, that my ESP Easy sends the correct values. I didn't want to hack my server "acces.log" file, so i updated my "httpd.conf" file as following:
Go to section

<IfModule log_config_module>
..........
..........
..........
</IfModule>

and make a new common LogFormat entry like

LogFormat "%t %q" common

and a new path and log-filename entry like

CustomLog "D:/xampp/htdocs/home/logs/espeasy-data.log" common

All together it looks similar to something like

<IfModule log_config_module>
..........
LogFormat "%t %q" common
..........
CustomLog "D:/xampp/htdocs/home/logs/espeasy-data.log" common
..........
</IfModule>

This will save the ESP Easy Data in path "D:/xampp/htdocs/home/logs/" in file "espeasy-data.log" as following:

[22/Oct/2016:16:16:44 +0200] ?name=Buero&task=bme280&valuename=Temperature&value=21.2
[22/Oct/2016:16:16:45 +0200] ?name=Buero&task=bme280&valuename=Humidity&value=49.9
[22/Oct/2016:16:16:46 +0200] ?name=Buero&task=bme280&valuename=Pressure&value=1032.5

This is much shorter and more convinient as the "access.log" part! To save the data like this for further data-handling would be sufficiant.

Second and final try:
But for the long run i've looked for a more professional way to save my data into a CSV-File (Comma Separated Value). I ended up with following solution, which now represents my final version for the handling of reading and writing my sensor data to my localhost (XAMPP Version 5.6.24). The basic settings for "Generic HTTP", "Devices" (BME280) and "Publish Template:" are all the same as described before in this thread!

My sensor data is accessible in following PHP GET variables (which are all ESP Easy standard variable names):

$_GET['name'];
$_GET['task'];
$_GET['valuename'];
$_GET['value'];

The data, stored in this GET variables, will be parsed as described in the following "get_data.php" file:

Code: Select all

<?php

// Unix Zeitstempel
$unix_stamp = time();
// Datum und Uhrzeit
$date_and_time = date('d.m.Y-H:i:s');

// Sensor Ort
$location = $_GET['name'];
// Sensortyp
$sensor = $_GET['task'];
// Wertname (entweder Temperatur oder Luftdruck oder Luftfeuchte)
$valuename = $_GET['valuename'];
// Messwert von Temperatur oder Luftdruck oder Luftfeuchte
$value = $_GET['value'];

// Pfad zur CSV-Datendatei
$csv_path = "data/";
// Dateiname der CSV-Datendatei
$csv_filename = $location . ".csv";
// Pfad und CSV-Dateiname
$csv_path_and_filename = $csv_path.$csv_filename;

// Kopfzeile - Spaltennamen der CSV-Datendatei
$csv_header = "Nr.\t;Sensor-Ort\t;Unix Timestamp\t;Datum u. Uhrzeit\t;Sensortyp\t;Temperatur C\t;Luftfeuchte %\t;Luftdruck mb (NN)\n";

// Wenn die CSV-Datendatei nicht existiert, dann erstelle sie mit der Kopfzeile neu
if (!file_exists($csv_path_and_filename)) {
	$datei = fopen($csv_path_and_filename,"w");
	fwrite($datei, $csv_header);
	fclose($datei);	
}
// Zeilenanzahl der CSV-Datendatei ermitteln - Achtung: 0-basiert (Null-basiert)!
// Kopfzeile ist die Zeile 0. Die Zeile 1 fängt also wie gewuenscht mit 1 an!
$anzahl = count(file($csv_path_and_filename));
  
// Die nacheinander folgenden Sensorwerte auslesen und in eigene Variablen einlesen
switch ($valuename) {
	case 'Temperature':
		$temp = $value;
	 	error_log($anzahl."\t;"."\t".$location."\t;"."\t".$unix_stamp."\t;".$date_and_time.';'."\t".$sensor."\t;"."\t".$temp."\t\t;", 3, $csv_path_and_filename);
		break;
	case 'Humidity':
		$humidity = $value;
		error_log("\t".$humidity."\t\t;", 3, $csv_path_and_filename);
		break;
	case 'Pressure':
		$pressure = $value;
		error_log("\t".$pressure."\n", 3, $csv_path_and_filename);
		break;
	Default:
		error_log("Messfehler!", 3, $csv_path_and_filename);		
}	

?>

Sorry, but due to lack of time i will leave the comments in german language. 

OK, the result is a nice and clean csv data file with a clear data structure inside. ESP Easy will update the sensor data every 5 minutes, and my CSV data now looks like:

Nr. ;Sensor-Ort ;Unix Timestamp ;Datum u. Uhrzeit ;Sensortyp ;Temperatur C ;Luftfeuchte % ;Luftdruck mb (NN)
1 ; Buero ; 1477600137 ;27.10.2016-22:28:57; bme280 ; 24.2 ; 51.0 ; 1034.6
2 ; Buero ; 1477600438 ;27.10.2016-22:33:58; bme280 ; 24.1 ; 50.8 ; 1034.7
3 ; Buero ; 1477600738 ;27.10.2016-22:38:58; bme280 ; 24.1 ; 50.6 ; 1034.7
4 ; Buero ; 1477601039 ;27.10.2016-22:43:59; bme280 ; 24.1 ; 51.0 ; 1034.7
5 ; Buero ; 1477601339 ;27.10.2016-22:48:59; bme280 ; 24.2 ; 51.5 ; 1034.6
6 ; Buero ; 1477601640 ;27.10.2016-22:54:00; bme280 ; 24.2 ; 50.6 ; 1034.7
7 ; Buero ; 1477601940 ;27.10.2016-22:59:00; bme280 ; 24.1 ; 50.0 ; 1034.7

Please note:
others as shown above, due to the "\t" signs (Tabulator) in the "get_data.php" file all columns/values of the head line and all following data lines are set exactly under each other (in the stored CSV data file)!

The main goal is, that there are tons of PHP CSV functions for further data handling! And if your CSV data file gets to big, you easyly can migrate to a MySQL Data Base!

If nobody comes up with a better/easier way of how to do it, i will stay with my solution as described before. For each room now i have a little cheap WiFi sensor for measuring and displaying Temperature, Humidity and Air Pressure as well as for further data handling/research grabbed from my "Big Data" stored on my localhost.

There are now round about 1100 views for this thread. So there must be a certain common interest in this "Generic HTTP" matter. That's the reason why i presented my final solution to the public. I would be glad, if the two solutions (raptor or mine) in this thread would be for any help for some people in the future.

[rai]

Hello,

in addition to my yesterdays post please find following screenshot of my csv datafile created with my "get_data.php" file.



The above screenshot shows, what i meant in my yesterdays post with

Please note:
... all columns/values of the head line and all following data lines are set exactly under each other (in the stored CSV data file)!

P. S.
Sorry. The image (sreenshot) i put here will only be displayed for about 15-30 minutes. After this short time the image is gone, and only the word "Image" is left (see above). I tried it with images stored on "Dropbox" or "MS One Drive". Both won't work, and I don't know why!?