GPS#Travelled already included? [SOLVED]

Moderators: grovkillen, Stuntteam, TD-er

Post Reply
Message
Author
Wiki
Normal user
Posts: 413
Joined: 23 Apr 2018, 17:55
Location: Germany

GPS#Travelled already included? [SOLVED]

#1 Post by Wiki » 29 May 2020, 01:44

I want to replace my 16 years old Tiny Tiger onboard "computer" of my caravan which I designed decades ago to track my travels in 100m detail by storing geo points, set start/end points of my trips, observe battery capacity, ambient temperatures aso. It has done its job pretty well over the years, stored the data in its internal flash, gave me the possibility to follow my trips by fetching the data over serial port with my laptop.

Readthedocs talks about a future plan to implement an event named GPS#Travelled, which would be essentially important for me for the replacement of my pretty old grandpa Tiny Tiger. Is the event already included in the current testing builds?

If yes, is there any thinkable way to save the positions/dates/times depending on the event to a sd card?
Last edited by Wiki on 08 Jun 2020, 17:10, edited 1 time in total.

Code: Select all

pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $

User avatar
ThomasB
Normal user
Posts: 1064
Joined: 17 Jun 2018, 20:41
Location: USA

Re: GPS#Travelled already included?

#2 Post by ThomasB » 29 May 2020, 19:03

For the record, I'm not using the GPS plugin or the SD card controller. But since no one else has chimed in, here's some cheering from the sidelines:
Is the event already included in the current testing builds?
From a quick review of the P082 source code it doesn't appear that GPS#Travelled is implemented. But it can read positional data and output values after N meters traveled (via user setting). So you could set it to 100M and the GPS data would be sent whenever this distance occurs, just as you needed.
If yes, is there any thinkable way to save the positions/dates/times depending on the event to a sd card?
It seems to me that the SD Card controller should be able to store the plugin data values that you specify.

So at face value I would say that your project idea has merit. Unless someone else says otherwise, the only way to know for sure is to try it out.

- Thomas

Wiki
Normal user
Posts: 413
Joined: 23 Apr 2018, 17:55
Location: Germany

Re: GPS#Travelled already included?

#3 Post by Wiki » 30 May 2020, 01:41

Thank you, Thomas. Your comment ecourages me to give this project a try. Maybe, it could be the first automotive project using ESPEasy? I didn't find anything similar.

Yesterday I have ordered a Neo 6M, I do have experiences already with this nice, pretty fast locking GPS sensor in another automotive project using a Raspberry Pi.

I will report.....

Code: Select all

pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: GPS#Travelled already included?

#4 Post by TD-er » 30 May 2020, 09:48

Sorry, was occupied the last 2 days, so have not been online much.

ThomasB is correct, you can set a threshold for meters travelled.

My setup:
- Interval of GPS is set to a high interval of 15 minutes (can be any suitable value)
- Distance Update Interval is set to 100m

What this does:
- If the vehicle does not move, it will send out a new value every 900 sec.
- When moving, the GPS plugin sends an update every 100 meter travelled.

In the rules I trigger capturing other sensor values based on the new GPS update and call taskrun on the dummy tasks (task 7 and 8) to send them to a controller.
As controller I use the (experimental) cache controller, which stores the values on the file system.

Code: Select all

on gps#long do
  TaskValueSet 7,1,[S8#co2]
  TaskValueSet 7,2,[S8#T]
  TaskValueSet 7,3,[bat#V]
  TaskValueSet 7,4,[sys#uptime]
  TaskValueSet 8,1,[sys#rssi]
  TaskValueSet 8,2,[bme#T]
  TaskValueSet 8,3,[bme#H]
  TaskValueSet 8,4,[bme#P]
  TaskRun 7
  TaskRun 8
endon
On the file system I have placed this .htm file (use .htm as extension, not .html)
Via the browser you can then open this .htm file and export all recorded samples to a .csv file
A stored task takes 24 bytes, so storing 2 dummy tasks like this takes 48 bytes per interval.
On a build with 4M flash divided in 1M filesystem, you can store roughly 750k of files.

Code: Select all


<html>
<script src="https://cdn.jsdelivr.net/npm/lodash@4.17.11/lodash.min.js"></script>
<script>
const TASKS_MAX = 12;
const VARS_PER_TASK = 4;

function sleep(ms) {
  return new Promise(resolve => setTimeout(resolve, ms));
}

class DataParser {
    constructor(data) {
        this.view = new DataView(data);
        this.offset = 0;
        this.bitbyte = 0;
        this.bitbytepos = 7;
    }

    pad(nr) {
        while (this.offset % nr) {
            this.offset++;
        }
    }

    bit(signed = false, write = false, val) {
        if (this.bitbytepos === 7) {
            if (!write) {
                this.bitbyte = this.byte();
                this.bitbytepos = 0;
            } else {
                this.byte(signed, write, this.bitbyte);
            }
        }
        if (!write) {
            return (this.bitbyte >> this.bitbytepos++) & 1;
        } else {
            this.bitbyte = val ? (this.bitbyte | (1 << this.bitbytepos++)) : (this.bitbyte & ~(1 << this.bitbytepos++));
        }
    }

    byte(signed = false, write = false, val) {
        this.pad(1);
        const fn = `${write ? 'set' : 'get'}${signed ? 'Int8' : 'Uint8'}`;
        const res = this.view[fn](this.offset, val);
        this.offset += 1;
        return res;
    }

    int16(signed = false, write = false, val) {
        this.pad(2);
        let fn = signed ? 'Int16' : 'Uint16';
        const res =  write ? this.view[`set${fn}`](this.offset, val, true) : this.view[`get${fn}`](this.offset, true);
        this.offset += 2;
        return res;
    }

    int32(signed = false, write = false, val) {
        this.pad(4);
        let fn = signed ? 'Int32' : 'Uint32';
        const res =  write ? this.view[`set${fn}`](this.offset, val, true) : this.view[`get${fn}`](this.offset, true);
        this.offset += 4;
        return res;
    }
    float(signed = false, write = false, val) {
        this.pad(4);
        const res =  write ? this.view.setFloat32(this.offset, val, true) : this.view.getFloat32(this.offset, true);
        this.offset += 4;
        return res;
    }
    bytes(nr, signed = false, write = false, vals) {
        const res = [];
        for (var x = 0; x < nr; x++) {
            res.push(this.byte(signed, write, vals ? vals[x] : null));
        }
        return res;
    }
    ints(nr, signed = false, write = false, vals) {
        const res = [];
        for (var x = 0; x < nr; x++) {
            res.push(this.int16(signed, write, vals ? vals[x] : null));
        }
        return res;
    }
    longs(nr, signed = false, write = false, vals) {
        const res = [];
        for (var x = 0; x < nr; x++) {
            res.push(this.int32(signed, write, vals ? vals[x] : null));
        }
        return res;
    }
    floats(nr, signed = false, write = false, vals) {
        const res = [];
        for (var x = 0; x < nr; x++) {
            res.push(this.float(write, vals ? vals[x] : null));
        }
        return res;
    }
    string(nr, signed = false, write = false, val) {
        if (write) {
            for (var i = 0; i < nr; ++i) {
                var code = val.charCodeAt(i) || '\0';
                this.byte(false, true, code);
            }
        } else {
            const res = this.bytes(nr);
            return String.fromCharCode.apply(null, res).replace(/\x00/g, '');
        }
    }
}

const parseConfig = (data, config, start) => {
    const p = new DataParser(data);
    if (start) p.offset = start;
    const result = {};
    config.map(value => {
        const prop = value.length ? value.length : value.signed;
        _.set(result, value.prop, p[value.type](prop, value.signed));
    });
    return result;
}
/*
const fileFormat = [
    [...Array(1000)].map((x, i) => ({ prop: `samples[${i}].values`, type:'floats', length: VARS_PER_TASK  })),
    [...Array(1000)].map((x, i) => ({ prop: `samples[${i}].timestamp`, type: 'longs', signed: false  })),
    [...Array(1000)].map((x, i) => ({ prop: `samples[${i}].controllerIndex`, type: 'byte'  })),
    [...Array(1000)].map((x, i) => ({ prop: `samples[${i}].taskIndex`, type: 'byte'  })),
    [...Array(1000)].map((x, i) => ({ prop: `samples[${i}].sensorType`, type: 'byte'  })),
    [...Array(1000)].map((x, i) => ({ prop: `samples[${i}].valueCount`, type: 'byte'  })),
];
*/
const fileFormat = [
    { prop: 'values', type:'floats', length: VARS_PER_TASK },
    { prop: 'timestamp', type: 'int32', signed: false },
    { prop: 'controllerIndex', type: 'byte' },
    { prop: 'taskIndex', type: 'byte' },
    { prop: 'sensorType', type: 'byte' },
    { prop: 'valueCount', type: 'byte' },
];

/*
loadConfig = () => {
    return fetch('http://192.168.1.182/cache_json').then(response => response.arrayBuffer()).then(async response => {
        document.body.innerText = parseConfig(response, fileFormat);
    });
}
*/

loadConfig = async () => {
    const floatvalues = {};
    const info = await fetch('/cache_json').then(response => response.json());
	let csv = info.columns.join(';') + '\n';
	for (var j = 0; j < (VARS_PER_TASK * TASKS_MAX); j++) {
		// TODO make "unused" value configurable
		floatvalues[j] = 0;
	}
	
	// TODO must also read partial files (< 24k)
	var maxFileNr = info.files.length;
	for (var filenr = 0; filenr < maxFileNr; filenr++) {
	    var elem = document.getElementById("bar");
		var width = Math.round(100.0 * (filenr / (maxFileNr - 1 )));
		elem.style.width = width + '%'; 
        elem.innerHTML = width * 1 + '%';
		const binary = await fetch(info.files[filenr]).then(response => response.arrayBuffer()).then(async response => { 
			const samples = {};
			var arrayLength = response.byteLength / 24;
			//1000;//samples.length;
			
			[...Array(arrayLength)].map((x, i) => {
				samples[i] = parseConfig(response, fileFormat, 24 * i);
			});
			
			// TODO Fetch number of samples.
			for (var i = 0; i < arrayLength; i++) {
			  var floatIndex = VARS_PER_TASK * samples[i].taskIndex;
			  samples[i].values.forEach(item => {
				floatvalues[floatIndex] = item;
				floatIndex++;		  
			  });
			  // TODO quick fix to remove damaged samples due to writing to closed files.
			  if (samples[i].timestamp > 1500000000) {
			      const utc_date = new Date(samples[i].timestamp * 1000);
				  csv += samples[i].timestamp + ';' + utc_date.toISOString() + ';' + samples[i].taskIndex;
				  for (var j = 0; j < (VARS_PER_TASK * TASKS_MAX); j++) {
					csv += ';' + floatvalues[j];
				  }
				  csv += '\n';  
			  }
			}
			await sleep(100); // Wait to prevent the ESPeasy node from rebooting.			
		});
	}
//	document.body.innerText = csv;
//    document.body.innerHTML = `<a href='data:text/plain;charset=utf-8,${encodeURIComponent(csv)}' download='test.csv'>click me</a>`;

	const a = document.createElement('a');
	const aText = document.createTextNode('Download');
	a.href = window.URL.createObjectURL(new Blob([csv]), {type: 'text/csv'});
	a.download = 'test.csv';
	a.appendChild(aText);
	a.classList.add("button");
	document.getElementById("downloadLink").appendChild(a);
//	document.body.appendChild(a);
}
</script>

<style>
#progress {
  width: 100%;
  background-color: #ddd;
}

#bar {
  width: 0%;
  height: 30px;
  background-color: #4CAF50;
  text-align: center;
  line-height: 30px;
  color: white;
}

h1, h2 {
    font-size: 16pt;
    margin: 8px 0;
}

h1, h2 {
    color: #07D;
}

* {
    box-sizing: border-box;
    font-family: sans-serif;
    font-size: 12pt;
    margin: 0;
    padding: 0;
}

.button {
    background-color: #07D;
    border: none;
    border-radius: 4px;
    color: #FFF;
    margin: 4px;
    padding: 4px 16px;
}

</style>
<body>

<h2>ESPeasy cache to CSV</h2>
<BR>

<button class="button" type="button" onclick="loadConfig()">Fetch cache files</button>

<div id="progress">
  <div id="bar">0%</div>
</div>
<BR>
<p id="downloadLink"></p>
</body>
</html>

Wiki
Normal user
Posts: 413
Joined: 23 Apr 2018, 17:55
Location: Germany

Re: GPS#Travelled already included?

#5 Post by Wiki » 30 May 2020, 14:53

Wow, thats cute, looks as if this is exactly what I would need.

Hints (for me):

My nodes never ever will have a direct access to the Inet. May I download the lodash.min.js as a file to the filesystem of the node and use it from there? And if yes (I am not very familiar with html programming and java stuff and so on), how do I access it out of the html code?

Where do I get the experimental cash controller from?

Code: Select all

pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: GPS#Travelled already included?

#6 Post by TD-er » 30 May 2020, 16:44

Experimental cache controller is included in the custom builds, or you make it yourself of course.

The .js file will be downloaded by your browser and used there.
You can save it to your node, but that will take valuable space.
If your mobile (or whatever you need to connect) is also connected to the internet (e.g. via 4G) then it will download the needed .js and it will be kept there for a long time in your browser cache.

The JavaScript will query the node for extra information before decoding the .bin files.
It asks the node for the names of the tasks to generate the first line of the CSV file.
Then it asks for the file names and starts downloading them.
The decoding is done in the browser, so you don't use the ESP's resources.

Wiki
Normal user
Posts: 413
Joined: 23 Apr 2018, 17:55
Location: Germany

Re: GPS#Travelled already included?

#7 Post by Wiki » 30 May 2020, 18:23

OK, Gotit and got it working.

Hey guys, its playtime now....

Code: Select all

pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $

Wiki
Normal user
Posts: 413
Joined: 23 Apr 2018, 17:55
Location: Germany

Re: GPS#Travelled already included?

#8 Post by Wiki » 30 May 2020, 21:00

@TD-er:

Where should I place my observations for you as reminder concerning bugs / wishes / needs in GPS plugin and Cache Controller?

Code: Select all

pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $

User avatar
grovkillen
Core team member
Posts: 3621
Joined: 19 Jan 2017, 12:56
Location: Hudiksvall, Sweden
Contact:

Re: GPS#Travelled already included?

#9 Post by grovkillen » 30 May 2020, 21:26

That would be GitHub
ESP Easy Flasher [flash tool and wifi setup at flash time]
ESP Easy Webdumper [easy screendumping of your units]
ESP Easy Netscan [find units]
Official shop: https://firstbyte.shop/
Sponsor ESP Easy, we need you :idea: :idea: :idea:

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: GPS#Travelled already included?

#10 Post by TD-er » 30 May 2020, 22:36

Yep Github is the place for those requests for future bugs :)

Wiki
Normal user
Posts: 413
Joined: 23 Apr 2018, 17:55
Location: Germany

Re: GPS#Travelled already included?

#11 Post by Wiki » 03 Jun 2020, 00:12

OK, configuration is working so far. Thanks a lot @TD-er for the solution provided. Its nice to get the data directly into Excel for further processing.

I will describe my caravan project in a seperate thread.

I followed the recommendation to add a rule triggered by GPS#long, configuered the distance update interval to 100m and am writing +100m to a dummy value if the rule is triggered. But the recorded distance differs around 20-30% from the in reality travelled distance.

Any suggestions?

Code: Select all

pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: GPS#Travelled already included?

#12 Post by TD-er » 03 Jun 2020, 09:04

I gave a few suggestions on the Github issue you opened.
Just the link for those finding this thread: https://github.com/letscontrolit/ESPEasy/issues/3099

Wiki
Normal user
Posts: 413
Joined: 23 Apr 2018, 17:55
Location: Germany

Re: GPS#Travelled already included? [SOLVED]

#13 Post by Wiki » 08 Jun 2020, 17:16

I got a preliminary build from TD-er with event GPS#travelled and as eventvalue the calcuulated distancein meters. Works perfectly, better and more accurate than expected.

You have to take in mind, that the distance only can be calculated as a straight line between two geo points. So there will be a slight difference between calculated and really moved distance. But in my first test this difference is neglectible.

Thanks @TD-er, pretty nice job.

Code: Select all

pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: GPS#Travelled already included? [SOLVED]

#14 Post by TD-er » 08 Jun 2020, 19:28

I guess the error depends on the driving speed.
If you're driving at 30 m/sec (over 100 km/h) the road will be almost straight as you simply cannot make sharp corners.
When driving in a town at less than 10 m/sec (30 km/h), the corners will be a lot more sharp, meaning you can have a much larger error.
But on the other hand, when driving slower, you will not drive a large distance, so in total the highway kilometers will be a lot more than the ones in a town.

Post Reply

Who is online

Users browsing this forum: Ahrefs [Bot] and 34 guests