I want to replace my 16 years old Tiny Tiger onboard "computer" of my caravan which I designed decades ago to track my travels in 100m detail by storing geo points, set start/end points of my trips, observe battery capacity, ambient temperatures aso. It has done its job pretty well over the years, stored the data in its internal flash, gave me the possibility to follow my trips by fetching the data over serial port with my laptop.
Readthedocs talks about a future plan to implement an event named GPS#Travelled, which would be essentially important for me for the replacement of my pretty old grandpa Tiny Tiger. Is the event already included in the current testing builds?
If yes, is there any thinkable way to save the positions/dates/times depending on the event to a sd card?
GPS#Travelled already included? [SOLVED]
Moderators: grovkillen, Stuntteam, TD-er
GPS#Travelled already included? [SOLVED]
Last edited by Wiki on 08 Jun 2020, 17:10, edited 1 time in total.
Code: Select all
pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $
Re: GPS#Travelled already included?
For the record, I'm not using the GPS plugin or the SD card controller. But since no one else has chimed in, here's some cheering from the sidelines:
So at face value I would say that your project idea has merit. Unless someone else says otherwise, the only way to know for sure is to try it out.
- Thomas
From a quick review of the P082 source code it doesn't appear that GPS#Travelled is implemented. But it can read positional data and output values after N meters traveled (via user setting). So you could set it to 100M and the GPS data would be sent whenever this distance occurs, just as you needed.Is the event already included in the current testing builds?
It seems to me that the SD Card controller should be able to store the plugin data values that you specify.If yes, is there any thinkable way to save the positions/dates/times depending on the event to a sd card?
So at face value I would say that your project idea has merit. Unless someone else says otherwise, the only way to know for sure is to try it out.
- Thomas
Re: GPS#Travelled already included?
Thank you, Thomas. Your comment ecourages me to give this project a try. Maybe, it could be the first automotive project using ESPEasy? I didn't find anything similar.
Yesterday I have ordered a Neo 6M, I do have experiences already with this nice, pretty fast locking GPS sensor in another automotive project using a Raspberry Pi.
I will report.....
Yesterday I have ordered a Neo 6M, I do have experiences already with this nice, pretty fast locking GPS sensor in another automotive project using a Raspberry Pi.
I will report.....
Code: Select all
pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $Re: GPS#Travelled already included?
Sorry, was occupied the last 2 days, so have not been online much.
ThomasB is correct, you can set a threshold for meters travelled.
My setup:
- Interval of GPS is set to a high interval of 15 minutes (can be any suitable value)
- Distance Update Interval is set to 100m
What this does:
- If the vehicle does not move, it will send out a new value every 900 sec.
- When moving, the GPS plugin sends an update every 100 meter travelled.
In the rules I trigger capturing other sensor values based on the new GPS update and call taskrun on the dummy tasks (task 7 and 8) to send them to a controller.
As controller I use the (experimental) cache controller, which stores the values on the file system.
On the file system I have placed this .htm file (use .htm as extension, not .html)
Via the browser you can then open this .htm file and export all recorded samples to a .csv file
A stored task takes 24 bytes, so storing 2 dummy tasks like this takes 48 bytes per interval.
On a build with 4M flash divided in 1M filesystem, you can store roughly 750k of files.
ThomasB is correct, you can set a threshold for meters travelled.
My setup:
- Interval of GPS is set to a high interval of 15 minutes (can be any suitable value)
- Distance Update Interval is set to 100m
What this does:
- If the vehicle does not move, it will send out a new value every 900 sec.
- When moving, the GPS plugin sends an update every 100 meter travelled.
In the rules I trigger capturing other sensor values based on the new GPS update and call taskrun on the dummy tasks (task 7 and 8) to send them to a controller.
As controller I use the (experimental) cache controller, which stores the values on the file system.
Code: Select all
on gps#long do
TaskValueSet 7,1,[S8#co2]
TaskValueSet 7,2,[S8#T]
TaskValueSet 7,3,[bat#V]
TaskValueSet 7,4,[sys#uptime]
TaskValueSet 8,1,[sys#rssi]
TaskValueSet 8,2,[bme#T]
TaskValueSet 8,3,[bme#H]
TaskValueSet 8,4,[bme#P]
TaskRun 7
TaskRun 8
endon
Via the browser you can then open this .htm file and export all recorded samples to a .csv file
A stored task takes 24 bytes, so storing 2 dummy tasks like this takes 48 bytes per interval.
On a build with 4M flash divided in 1M filesystem, you can store roughly 750k of files.
Code: Select all
<html>
<script src="https://cdn.jsdelivr.net/npm/lodash@4.17.11/lodash.min.js"></script>
<script>
const TASKS_MAX = 12;
const VARS_PER_TASK = 4;
function sleep(ms) {
return new Promise(resolve => setTimeout(resolve, ms));
}
class DataParser {
constructor(data) {
this.view = new DataView(data);
this.offset = 0;
this.bitbyte = 0;
this.bitbytepos = 7;
}
pad(nr) {
while (this.offset % nr) {
this.offset++;
}
}
bit(signed = false, write = false, val) {
if (this.bitbytepos === 7) {
if (!write) {
this.bitbyte = this.byte();
this.bitbytepos = 0;
} else {
this.byte(signed, write, this.bitbyte);
}
}
if (!write) {
return (this.bitbyte >> this.bitbytepos++) & 1;
} else {
this.bitbyte = val ? (this.bitbyte | (1 << this.bitbytepos++)) : (this.bitbyte & ~(1 << this.bitbytepos++));
}
}
byte(signed = false, write = false, val) {
this.pad(1);
const fn = `${write ? 'set' : 'get'}${signed ? 'Int8' : 'Uint8'}`;
const res = this.view[fn](this.offset, val);
this.offset += 1;
return res;
}
int16(signed = false, write = false, val) {
this.pad(2);
let fn = signed ? 'Int16' : 'Uint16';
const res = write ? this.view[`set${fn}`](this.offset, val, true) : this.view[`get${fn}`](this.offset, true);
this.offset += 2;
return res;
}
int32(signed = false, write = false, val) {
this.pad(4);
let fn = signed ? 'Int32' : 'Uint32';
const res = write ? this.view[`set${fn}`](this.offset, val, true) : this.view[`get${fn}`](this.offset, true);
this.offset += 4;
return res;
}
float(signed = false, write = false, val) {
this.pad(4);
const res = write ? this.view.setFloat32(this.offset, val, true) : this.view.getFloat32(this.offset, true);
this.offset += 4;
return res;
}
bytes(nr, signed = false, write = false, vals) {
const res = [];
for (var x = 0; x < nr; x++) {
res.push(this.byte(signed, write, vals ? vals[x] : null));
}
return res;
}
ints(nr, signed = false, write = false, vals) {
const res = [];
for (var x = 0; x < nr; x++) {
res.push(this.int16(signed, write, vals ? vals[x] : null));
}
return res;
}
longs(nr, signed = false, write = false, vals) {
const res = [];
for (var x = 0; x < nr; x++) {
res.push(this.int32(signed, write, vals ? vals[x] : null));
}
return res;
}
floats(nr, signed = false, write = false, vals) {
const res = [];
for (var x = 0; x < nr; x++) {
res.push(this.float(write, vals ? vals[x] : null));
}
return res;
}
string(nr, signed = false, write = false, val) {
if (write) {
for (var i = 0; i < nr; ++i) {
var code = val.charCodeAt(i) || '\0';
this.byte(false, true, code);
}
} else {
const res = this.bytes(nr);
return String.fromCharCode.apply(null, res).replace(/\x00/g, '');
}
}
}
const parseConfig = (data, config, start) => {
const p = new DataParser(data);
if (start) p.offset = start;
const result = {};
config.map(value => {
const prop = value.length ? value.length : value.signed;
_.set(result, value.prop, p[value.type](prop, value.signed));
});
return result;
}
/*
const fileFormat = [
[...Array(1000)].map((x, i) => ({ prop: `samples[${i}].values`, type:'floats', length: VARS_PER_TASK })),
[...Array(1000)].map((x, i) => ({ prop: `samples[${i}].timestamp`, type: 'longs', signed: false })),
[...Array(1000)].map((x, i) => ({ prop: `samples[${i}].controllerIndex`, type: 'byte' })),
[...Array(1000)].map((x, i) => ({ prop: `samples[${i}].taskIndex`, type: 'byte' })),
[...Array(1000)].map((x, i) => ({ prop: `samples[${i}].sensorType`, type: 'byte' })),
[...Array(1000)].map((x, i) => ({ prop: `samples[${i}].valueCount`, type: 'byte' })),
];
*/
const fileFormat = [
{ prop: 'values', type:'floats', length: VARS_PER_TASK },
{ prop: 'timestamp', type: 'int32', signed: false },
{ prop: 'controllerIndex', type: 'byte' },
{ prop: 'taskIndex', type: 'byte' },
{ prop: 'sensorType', type: 'byte' },
{ prop: 'valueCount', type: 'byte' },
];
/*
loadConfig = () => {
return fetch('http://192.168.1.182/cache_json').then(response => response.arrayBuffer()).then(async response => {
document.body.innerText = parseConfig(response, fileFormat);
});
}
*/
loadConfig = async () => {
const floatvalues = {};
const info = await fetch('/cache_json').then(response => response.json());
let csv = info.columns.join(';') + '\n';
for (var j = 0; j < (VARS_PER_TASK * TASKS_MAX); j++) {
// TODO make "unused" value configurable
floatvalues[j] = 0;
}
// TODO must also read partial files (< 24k)
var maxFileNr = info.files.length;
for (var filenr = 0; filenr < maxFileNr; filenr++) {
var elem = document.getElementById("bar");
var width = Math.round(100.0 * (filenr / (maxFileNr - 1 )));
elem.style.width = width + '%';
elem.innerHTML = width * 1 + '%';
const binary = await fetch(info.files[filenr]).then(response => response.arrayBuffer()).then(async response => {
const samples = {};
var arrayLength = response.byteLength / 24;
//1000;//samples.length;
[...Array(arrayLength)].map((x, i) => {
samples[i] = parseConfig(response, fileFormat, 24 * i);
});
// TODO Fetch number of samples.
for (var i = 0; i < arrayLength; i++) {
var floatIndex = VARS_PER_TASK * samples[i].taskIndex;
samples[i].values.forEach(item => {
floatvalues[floatIndex] = item;
floatIndex++;
});
// TODO quick fix to remove damaged samples due to writing to closed files.
if (samples[i].timestamp > 1500000000) {
const utc_date = new Date(samples[i].timestamp * 1000);
csv += samples[i].timestamp + ';' + utc_date.toISOString() + ';' + samples[i].taskIndex;
for (var j = 0; j < (VARS_PER_TASK * TASKS_MAX); j++) {
csv += ';' + floatvalues[j];
}
csv += '\n';
}
}
await sleep(100); // Wait to prevent the ESPeasy node from rebooting.
});
}
// document.body.innerText = csv;
// document.body.innerHTML = `<a href='data:text/plain;charset=utf-8,${encodeURIComponent(csv)}' download='test.csv'>click me</a>`;
const a = document.createElement('a');
const aText = document.createTextNode('Download');
a.href = window.URL.createObjectURL(new Blob([csv]), {type: 'text/csv'});
a.download = 'test.csv';
a.appendChild(aText);
a.classList.add("button");
document.getElementById("downloadLink").appendChild(a);
// document.body.appendChild(a);
}
</script>
<style>
#progress {
width: 100%;
background-color: #ddd;
}
#bar {
width: 0%;
height: 30px;
background-color: #4CAF50;
text-align: center;
line-height: 30px;
color: white;
}
h1, h2 {
font-size: 16pt;
margin: 8px 0;
}
h1, h2 {
color: #07D;
}
* {
box-sizing: border-box;
font-family: sans-serif;
font-size: 12pt;
margin: 0;
padding: 0;
}
.button {
background-color: #07D;
border: none;
border-radius: 4px;
color: #FFF;
margin: 4px;
padding: 4px 16px;
}
</style>
<body>
<h2>ESPeasy cache to CSV</h2>
<BR>
<button class="button" type="button" onclick="loadConfig()">Fetch cache files</button>
<div id="progress">
<div id="bar">0%</div>
</div>
<BR>
<p id="downloadLink"></p>
</body>
</html>
Re: GPS#Travelled already included?
Wow, thats cute, looks as if this is exactly what I would need.
Hints (for me):
My nodes never ever will have a direct access to the Inet. May I download the lodash.min.js as a file to the filesystem of the node and use it from there? And if yes (I am not very familiar with html programming and java stuff and so on), how do I access it out of the html code?
Where do I get the experimental cash controller from?
Hints (for me):
My nodes never ever will have a direct access to the Inet. May I download the lodash.min.js as a file to the filesystem of the node and use it from there? And if yes (I am not very familiar with html programming and java stuff and so on), how do I access it out of the html code?
Where do I get the experimental cash controller from?
Code: Select all
pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $Re: GPS#Travelled already included?
Experimental cache controller is included in the custom builds, or you make it yourself of course.
The .js file will be downloaded by your browser and used there.
You can save it to your node, but that will take valuable space.
If your mobile (or whatever you need to connect) is also connected to the internet (e.g. via 4G) then it will download the needed .js and it will be kept there for a long time in your browser cache.
The JavaScript will query the node for extra information before decoding the .bin files.
It asks the node for the names of the tasks to generate the first line of the CSV file.
Then it asks for the file names and starts downloading them.
The decoding is done in the browser, so you don't use the ESP's resources.
The .js file will be downloaded by your browser and used there.
You can save it to your node, but that will take valuable space.
If your mobile (or whatever you need to connect) is also connected to the internet (e.g. via 4G) then it will download the needed .js and it will be kept there for a long time in your browser cache.
The JavaScript will query the node for extra information before decoding the .bin files.
It asks the node for the names of the tasks to generate the first line of the CSV file.
Then it asks for the file names and starts downloading them.
The decoding is done in the browser, so you don't use the ESP's resources.
Re: GPS#Travelled already included?
OK, Gotit and got it working.
Hey guys, its playtime now....
Hey guys, its playtime now....
Code: Select all
pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $Re: GPS#Travelled already included?
@TD-er:
Where should I place my observations for you as reminder concerning bugs / wishes / needs in GPS plugin and Cache Controller?
Where should I place my observations for you as reminder concerning bugs / wishes / needs in GPS plugin and Cache Controller?
Code: Select all
pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $-
grovkillen
- Core team member
- Posts: 3621
- Joined: 19 Jan 2017, 12:56
- Location: Hudiksvall, Sweden
- Contact:
Re: GPS#Travelled already included?
That would be GitHub
ESP Easy Flasher [flash tool and wifi setup at flash time]
ESP Easy Webdumper [easy screendumping of your units]
ESP Easy Netscan [find units]
Official shop: https://firstbyte.shop/
Sponsor ESP Easy, we need you
ESP Easy Webdumper [easy screendumping of your units]
ESP Easy Netscan [find units]
Official shop: https://firstbyte.shop/
Sponsor ESP Easy, we need you
Re: GPS#Travelled already included?
Yep Github is the place for those requests for future bugs 
Re: GPS#Travelled already included?
OK, configuration is working so far. Thanks a lot @TD-er for the solution provided. Its nice to get the data directly into Excel for further processing.
I will describe my caravan project in a seperate thread.
I followed the recommendation to add a rule triggered by GPS#long, configuered the distance update interval to 100m and am writing +100m to a dummy value if the rule is triggered. But the recorded distance differs around 20-30% from the in reality travelled distance.
Any suggestions?
I will describe my caravan project in a seperate thread.
I followed the recommendation to add a rule triggered by GPS#long, configuered the distance update interval to 100m and am writing +100m to a dummy value if the rule is triggered. But the recorded distance differs around 20-30% from the in reality travelled distance.
Any suggestions?
Code: Select all
pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $Re: GPS#Travelled already included?
I gave a few suggestions on the Github issue you opened.
Just the link for those finding this thread: https://github.com/letscontrolit/ESPEasy/issues/3099
Just the link for those finding this thread: https://github.com/letscontrolit/ESPEasy/issues/3099
Re: GPS#Travelled already included? [SOLVED]
I got a preliminary build from TD-er with event GPS#travelled and as eventvalue the calcuulated distancein meters. Works perfectly, better and more accurate than expected.
You have to take in mind, that the distance only can be calculated as a straight line between two geo points. So there will be a slight difference between calculated and really moved distance. But in my first test this difference is neglectible.
Thanks @TD-er, pretty nice job.
You have to take in mind, that the distance only can be calculated as a straight line between two geo points. So there will be a slight difference between calculated and really moved distance. But in my first test this difference is neglectible.
Thanks @TD-er, pretty nice job.
Code: Select all
pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $Re: GPS#Travelled already included? [SOLVED]
I guess the error depends on the driving speed.
If you're driving at 30 m/sec (over 100 km/h) the road will be almost straight as you simply cannot make sharp corners.
When driving in a town at less than 10 m/sec (30 km/h), the corners will be a lot more sharp, meaning you can have a much larger error.
But on the other hand, when driving slower, you will not drive a large distance, so in total the highway kilometers will be a lot more than the ones in a town.
If you're driving at 30 m/sec (over 100 km/h) the road will be almost straight as you simply cannot make sharp corners.
When driving in a town at less than 10 m/sec (30 km/h), the corners will be a lot more sharp, meaning you can have a much larger error.
But on the other hand, when driving slower, you will not drive a large distance, so in total the highway kilometers will be a lot more than the ones in a town.
Who is online
Users browsing this forum: No registered users and 12 guests