OK, here we go, next step of my tiny project: Trip recorder.
You need:
- a GPS device, mine is a
Neo6M, others are supported, too. I tok the Neo6M because I ave positive experiences in other automotive projects. Supported devices are
documented in the ESPEasy readthedocs
- a Wemos D1
- another Wemos D1 and the Nextio display
- In my case ESP_Easy_mega_20200608_custom_ESP8266_4M2M.bin to get the cache controller and some more space to store data
On the road I get the current GPS readings including some calculated values:
- GPS6.JPG (88.3 KiB) Viewed 123088 times
The fields names are self explaining.
Using the last custom build
Code: Select all
ESP_Easy_mega_20200608_custom_ESP8266_4M2M
ESPEasy gives me the experimental cache controller which is able to write some data into the flash:
- GPS3.JPG (58.19 KiB) Viewed 123088 times
Defining some devices:
- GPS2.JPG (98.26 KiB) Viewed 123088 times
- GPS4.JPG (68.22 KiB) Viewed 123088 times
- GPS5.JPG (79.34 KiB) Viewed 123088 times
- GPS7.JPG (77.59 KiB) Viewed 123088 times
is the prerequisit for what I am doing now.
Since the last release the GPS plugin sends an event depending on the travelled distance including the calculated distance since last event. I take this value to add it to the total distance travelled, write it into the dummy device and send the values to the cache controller which writes it into the flash:
Code: Select all
On GPS#travelled Do
if %eventvalue%<500 //just make sure there's no misreading of the GPS sensor
let 1,[Strecke#Strecke]+%eventvalue% //adding the distance
TaskValueSet 2,1,%v1% //setting dummy values
TaskValueSet 2,2,[GPS#lat]
TaskValueSet 2,3,[GPS#long]
TaskValueSet 2,4,[GPS#alt]
TaskRun 2 //run the task to write the data into the flash using the cache controller
let 1,[GPS#spd]*3.6 /calculating speed in km/h
SendTo 28,'Di,page2.t7.txt="[int#1]km/h"'
SendTo 28,'Di,page2.t1.txt="Long:[GPS#long]°"'
SendTo 28,'Di,page2.t2.txt="Lat:[GPS#lat]°"'
let 1,[GPS#alt]
SendTo 28,'Di,page2.t3.txt="Höhe:[int#1]m"'
SendTo 28,'Di,page2.t4.txt="Speed:[GPS#spd]m/s"'
let 1,[Strecke#Strecke]/1000
SendTo 28,'Di,page2.t5.txt="Trip:%v1%km"' //displaying the total distance since last trip reset
EndIf
EndOn
Hitting the "Trip Reset" button on the display generates an event on the device to reset trip time as well as trip distance:
Code: Select all
On TripReset Do
SendTo 28,'Di,page2.t5.txt="Trip:0.0km"'
SendTo 28,'Di,page2.t7.txt="0km/h"'
TaskValueSet 2,1,0
TaskValueSet 3,1,%uptime%
TaskValueSet 3,2,0
SendTo 28,'Di,page2.t6.txt="Zeit:%c_m2dhm%([Strecke1#Tripzeit])"'
TaskValueSet 2,2,[GPS#lat]
TaskValueSet 2,3,[GPS#long]
TaskValueSet 2,4,[GPS#alt]
TaskRun 2
EndOn
For calculating the trip time since last trip reset I am using the uptime of the device, calculating the difference betwen starttime and current uptime, writing it into the dummy variable and send it every minute to the Nextion display:
Code: Select all
On Clock#Time=All,**:** Do
SendTo 28,'Di,page1.t10.txt="%systm_hm%"' //sending the GPS-depending time to the zime fields of the various Nextion pages
SendTo 28,'Di,page2.t10.txt="%systm_hm%"'
SendTo 28,'Di,page3.t1.txt="%systm_hm%"'
let 1,%uptime%-[Strecke1#Startzeit] // calculating the trip time
TaskValueSet 3,2,%v1% //setting the dummy value
SendTo 28,'Di,page2.t6.txt="Zeit:%c_m2dhm%([Strecke1#Tripzeit])"' //and sending it to Nextion display
EndOn
Some more lines of code make the life easier. Because I am working with testing and experimental parts of ESPEasy some reboot are most likely. The values of the dummy devices will survive a reboot, but the uptime will be reset to zero. So I want to assure the surviving of the trip time:
Code: Select all
On System#Boot Do
//Warmstart überleben Tripzeit
let 1,0-[Strecke1#Tripzeit]
TaskValueSet 3,1,%v1%
TaskValueSet 3,2,0
EndOn
The Nextion will not run 24/7 so it is useful to inform the GPS device that the Nextion display has been started. On System#Boot the Nextion device generates an event on the GPS device which will send the currrent display data:
Code: Select all
On NextionInit Do
let 1,[GPS#spd]*3.6
SendTo 28,'Di,page2.t7.txt="[int#1]km/h"'
SendTo 28,'Di,page2.t1.txt="Long:[GPS#long]°"'
SendTo 28,'Di,page2.t2.txt="Lat:[GPS#lat]°"'
let 1,[GPS#alt]
SendTo 28,'Di,page2.t3.txt="Höhe:[int#1]m"'
SendTo 28,'Di,page2.t4.txt="Speed:[GPS#spd]m/s"'
let 1,[Strecke#Strecke]/1000
SendTo 28,'Di,page2.t5.txt="Trip:%v1%km"'
EndOn
So whatfor am I doing all this stuff except for online information during driving?
@TD-er was so kind topost:
TD-er wrote: ↑30 May 2020, 09:48
[...]
On the file system I have placed this .htm file (use .htm as extension, not .html)
Via the browser you can then open this .htm file and export all recorded samples to a .csv file
A stored task takes 24 bytes, so storing 2 dummy tasks like this takes 48 bytes per interval.
On a build with 4M flash divided in 1M filesystem, you can store roughly 750k of files.
Code: Select all
<html>
<script src="https://cdn.jsdelivr.net/npm/lodash@4.17.11/lodash.min.js"></script>
<script>
const TASKS_MAX = 12;
const VARS_PER_TASK = 4;
function sleep(ms) {
return new Promise(resolve => setTimeout(resolve, ms));
}
class DataParser {
constructor(data) {
this.view = new DataView(data);
this.offset = 0;
this.bitbyte = 0;
this.bitbytepos = 7;
}
pad(nr) {
while (this.offset % nr) {
this.offset++;
}
}
bit(signed = false, write = false, val) {
if (this.bitbytepos === 7) {
if (!write) {
this.bitbyte = this.byte();
this.bitbytepos = 0;
} else {
this.byte(signed, write, this.bitbyte);
}
}
if (!write) {
return (this.bitbyte >> this.bitbytepos++) & 1;
} else {
this.bitbyte = val ? (this.bitbyte | (1 << this.bitbytepos++)) : (this.bitbyte & ~(1 << this.bitbytepos++));
}
}
byte(signed = false, write = false, val) {
this.pad(1);
const fn = `${write ? 'set' : 'get'}${signed ? 'Int8' : 'Uint8'}`;
const res = this.view[fn](this.offset, val);
this.offset += 1;
return res;
}
int16(signed = false, write = false, val) {
this.pad(2);
let fn = signed ? 'Int16' : 'Uint16';
const res = write ? this.view[`set${fn}`](this.offset, val, true) : this.view[`get${fn}`](this.offset, true);
this.offset += 2;
return res;
}
int32(signed = false, write = false, val) {
this.pad(4);
let fn = signed ? 'Int32' : 'Uint32';
const res = write ? this.view[`set${fn}`](this.offset, val, true) : this.view[`get${fn}`](this.offset, true);
this.offset += 4;
return res;
}
float(signed = false, write = false, val) {
this.pad(4);
const res = write ? this.view.setFloat32(this.offset, val, true) : this.view.getFloat32(this.offset, true);
this.offset += 4;
return res;
}
bytes(nr, signed = false, write = false, vals) {
const res = [];
for (var x = 0; x < nr; x++) {
res.push(this.byte(signed, write, vals ? vals[x] : null));
}
return res;
}
ints(nr, signed = false, write = false, vals) {
const res = [];
for (var x = 0; x < nr; x++) {
res.push(this.int16(signed, write, vals ? vals[x] : null));
}
return res;
}
longs(nr, signed = false, write = false, vals) {
const res = [];
for (var x = 0; x < nr; x++) {
res.push(this.int32(signed, write, vals ? vals[x] : null));
}
return res;
}
floats(nr, signed = false, write = false, vals) {
const res = [];
for (var x = 0; x < nr; x++) {
res.push(this.float(write, vals ? vals[x] : null));
}
return res;
}
string(nr, signed = false, write = false, val) {
if (write) {
for (var i = 0; i < nr; ++i) {
var code = val.charCodeAt(i) || '\0';
this.byte(false, true, code);
}
} else {
const res = this.bytes(nr);
return String.fromCharCode.apply(null, res).replace(/\x00/g, '');
}
}
}
const parseConfig = (data, config, start) => {
const p = new DataParser(data);
if (start) p.offset = start;
const result = {};
config.map(value => {
const prop = value.length ? value.length : value.signed;
_.set(result, value.prop, p[value.type](prop, value.signed));
});
return result;
}
/*
const fileFormat = [
[...Array(1000)].map((x, i) => ({ prop: `samples[${i}].values`, type:'floats', length: VARS_PER_TASK })),
[...Array(1000)].map((x, i) => ({ prop: `samples[${i}].timestamp`, type: 'longs', signed: false })),
[...Array(1000)].map((x, i) => ({ prop: `samples[${i}].controllerIndex`, type: 'byte' })),
[...Array(1000)].map((x, i) => ({ prop: `samples[${i}].taskIndex`, type: 'byte' })),
[...Array(1000)].map((x, i) => ({ prop: `samples[${i}].sensorType`, type: 'byte' })),
[...Array(1000)].map((x, i) => ({ prop: `samples[${i}].valueCount`, type: 'byte' })),
];
*/
const fileFormat = [
{ prop: 'values', type:'floats', length: VARS_PER_TASK },
{ prop: 'timestamp', type: 'int32', signed: false },
{ prop: 'controllerIndex', type: 'byte' },
{ prop: 'taskIndex', type: 'byte' },
{ prop: 'sensorType', type: 'byte' },
{ prop: 'valueCount', type: 'byte' },
];
/*
loadConfig = () => {
return fetch('http://192.168.1.182/cache_json').then(response => response.arrayBuffer()).then(async response => {
document.body.innerText = parseConfig(response, fileFormat);
});
}
*/
loadConfig = async () => {
const floatvalues = {};
const info = await fetch('/cache_json').then(response => response.json());
let csv = info.columns.join(';') + '\n';
for (var j = 0; j < (VARS_PER_TASK * TASKS_MAX); j++) {
// TODO make "unused" value configurable
floatvalues[j] = 0;
}
// TODO must also read partial files (< 24k)
var maxFileNr = info.files.length;
for (var filenr = 0; filenr < maxFileNr; filenr++) {
var elem = document.getElementById("bar");
var width = Math.round(100.0 * (filenr / (maxFileNr - 1 )));
elem.style.width = width + '%';
elem.innerHTML = width * 1 + '%';
const binary = await fetch(info.files[filenr]).then(response => response.arrayBuffer()).then(async response => {
const samples = {};
var arrayLength = response.byteLength / 24;
//1000;//samples.length;
[...Array(arrayLength)].map((x, i) => {
samples[i] = parseConfig(response, fileFormat, 24 * i);
});
// TODO Fetch number of samples.
for (var i = 0; i < arrayLength; i++) {
var floatIndex = VARS_PER_TASK * samples[i].taskIndex;
samples[i].values.forEach(item => {
floatvalues[floatIndex] = item;
floatIndex++;
});
// TODO quick fix to remove damaged samples due to writing to closed files.
if (samples[i].timestamp > 1500000000) {
const utc_date = new Date(samples[i].timestamp * 1000);
csv += samples[i].timestamp + ';' + utc_date.toISOString() + ';' + samples[i].taskIndex;
for (var j = 0; j < (VARS_PER_TASK * TASKS_MAX); j++) {
csv += ';' + floatvalues[j];
}
csv += '\n';
}
}
await sleep(100); // Wait to prevent the ESPeasy node from rebooting.
});
}
// document.body.innerText = csv;
// document.body.innerHTML = `<a href='data:text/plain;charset=utf-8,${encodeURIComponent(csv)}' download='test.csv'>click me</a>`;
const a = document.createElement('a');
const aText = document.createTextNode('Download');
a.href = window.URL.createObjectURL(new Blob([csv]), {type: 'text/csv'});
a.download = 'test.csv';
a.appendChild(aText);
a.classList.add("button");
document.getElementById("downloadLink").appendChild(a);
// document.body.appendChild(a);
}
</script>
<style>
#progress {
width: 100%;
background-color: #ddd;
}
#bar {
width: 0%;
height: 30px;
background-color: #4CAF50;
text-align: center;
line-height: 30px;
color: white;
}
h1, h2 {
font-size: 16pt;
margin: 8px 0;
}
h1, h2 {
color: #07D;
}
* {
box-sizing: border-box;
font-family: sans-serif;
font-size: 12pt;
margin: 0;
padding: 0;
}
.button {
background-color: #07D;
border: none;
border-radius: 4px;
color: #FFF;
margin: 4px;
padding: 4px 16px;
}
</style>
<body>
<h2>ESPeasy cache to CSV</h2>
<BR>
<button class="button" type="button" onclick="loadConfig()">Fetch cache files</button>
<div id="progress">
<div id="bar">0%</div>
</div>
<BR>
<p id="downloadLink"></p>
</body>
</html>
Using the cache controller and the html-code I get a semicolon seperated .csv file which - with a little bit of work - I can convert them to a .gpx format using
GPSBabel.
At first I have to cleanup the data extracted from the devices flash using the html-code of @TD-er, I am doing this using Excel. You have to assure, that the excel-produced "," inside the exported .csv have are converted to ".". I am doing this currently simply using wordpad. Works. Up to now I was concentrated purely on the geodata Latitude and Longitude, much more is possible. Out of the .csv file I build a tab-delimited file with a header setting, structure looks like this:
Code: Select all
No Latitude Longitude
0 xx.xxxxxxx y.yyyyyyyyy
.
.
.
.
.
This structure can be imported by GPSBabel using the universal csv format:
- GPS8.JPG (92.06 KiB) Viewed 123088 times
)
