Let's Control It

The biggest problem I currently have is that it's difficult to test hardware and controllers after we've made changes or improvements. A possible solution for this is to create a test-setup that contains a lot of common sensors.

Because I might get a financial grant to support ESPEasy (more on this later), it's possible for me to order all kinds of hardware and create: THE RIG(tm).

The rig will consist of several common ESP8266's, and a ESP8285. Connected to these ESP's will be a lot of the most commonly used sensors and other hardware. Some ESP's might be connected to each other to test some other things like GPIO or pulsecounters. All the ESP's wil be connected to a server so they can be flashed and configured easy. Serial output will also be logged.

When this setup is complete it's also possible to create a bunch of automated regressions tests: For example compare the outputs of several temperature and humidity sensors. The plan is to automated most of the tests and let the rig run 24/7 to test stability and reliability.

After this I might setup a bunch of controllers so we also can test these as well.

Having a test-rig makes it possible to do some changes that have a huge impact, and test if everything still works. Currently we sometimes have to wait for other people to test stuff and report problems.

Maybe its usefull to give some developers access to test-machine that is connected to these ESP's, so that they can test stuff as well.

What do you guys think of this? Any suggestions?

Edwin

Good idea
+1 +1

If you need sensors i can provide some from my shelf

Regards
Shardan

Great idea and great use of the financial support. I hope the funding get through soon.

psy0rz wrote: ↑28 May 2017, 15:02 The biggest problem I currently have is that it's difficult to test hardware and controllers after we've made changes or improvements. A possible solution for this is to create a test-setup that contains a lot of common sensors.

Because I might get a financial grant to support ESPEasy (more on this later), it's possible for me to order all kinds of hardware and create: THE RIG(tm).

The rig will consist of several common ESP8266's, and a ESP8285. Connected to these ESP's will be a lot of the most commonly used sensors and other hardware. Some ESP's might be connected to each other to test some other things like GPIO or pulsecounters. All the ESP's wil be connected to a server so they can be flashed and configured easy. Serial output will also be logged.

When this setup is complete it's also possible to create a bunch of automated regressions tests: For example compare the outputs of several temperature and humidity sensors. The plan is to automated most of the tests and let the rig run 24/7 to test stability and reliability.

After this I might setup a bunch of controllers so we also can test these as well.

Having a test-rig makes it possible to do some changes that have a huge impact, and test if everything still works. Currently we sometimes have to wait for other people to test stuff and report problems.

Maybe its usefull to give some developers access to test-machine that is connected to these ESP's, so that they can test stuff as well.

What do you guys think of this? Any suggestions?

Edwin

Which standard ESP setup's are planned? I wouldn't make much sense to have multiple setup around ESP-01 and other simular setup around the ESP8266.
It would make sense to choose a standard ESP core processor like the ESP-12F and different periferal chipsets like memory (1M/4M), serial (F330) or power measurement (hlw8012) chipset's.
A good base setup is paramount, additional sensors are then easy to add. I also would name sensor chipsets (like BME280) instead of sensors. It's impossible to test every possible setup, focus on one of each chipset combination and let users tell you which variations are working (same sensor chipset but different manufactor).

+1 for the rig!!

+1 Great idea!

LisaM wrote: ↑28 May 2017, 17:22 Which standard ESP setup's are planned? I wouldn't make much sense to have multiple setup around ESP-01 and other simular setup around the ESP8266.
It would make sense to choose a standard ESP core processor like the ESP-12F and different periferal chipsets like memory (1M/4M), serial (F330) or power measurement (hlw8012) chipset's.
A good base setup is paramount, additional sensors are then easy to add. I also would name sensor chipsets (like BME280) instead of sensors. It's impossible to test every possible setup, focus on one of each chipset combination and let users tell you which variations are working (same sensor chipset but different manufactor).

I didnt plan anything specific yet, these are just ideas floating around in my head

Although you are right, dont forget different hardware is still different: Even if the core processor and peripheral chipsets are the same, we still may have weird software crashes due to hardware issues. (power regulators, wifi interference etc)

I would prefer to start with the most commonly used stuff on https://www.letscontrolit.com/wiki/inde ... P_Hardware .


I also talked about sensors, but I think we have to look at it differently: We want to test our plugins. If a plugin needs a Device, then we need such a device. If one plugin can work with more than one Device type (DHT11/DHT22 is a nice example), we probably just only need one such device.

For plugins like the pulse counter, we can just use another ESP to simulate the pulses. (via pwm gpio or something) We dont need an actual device for that.

Even better: the pulse counter and gpio pwm will test eachother that way.


Also we probably dont need every esp/device combination we can think off. I would prefer if we could get away with just one type of each ESP and a minimal set of devices to test the most commonly used plugins.


Keep in mind I want to test our software: So while it might be nice to compare the actual performance of say a DHT11 and 22, this is a different goal. It requires a different setup than the software testing ri (controlled environment conditions, calibrated equipment etcetc), and different skillsets.

psy0rz,

Because I might get a financial grant to support ESPEasy (more on this later)

I like to donate too.
Is there somewhere a Donate button like Domoticz has?
http://www.domoticz.com/downloads/

psy0rz & Dev. team.
Thanks for all the time spent and the great result.

Domosapiens

This test rig a really good idea and will probably save you a lot of time.

Keep us posted !

Testrigs are essential for a developer. Had one before, but looked like this
And that was only half of it, forgot to make a picture of the NodeMCU testbed... (looked a bit better but not much, considering all the jump wires)
All sensors where attached at multiple units, but back in those days we did not have that many sensors yet...

Martinus, I'd love to get some behind the scene story of how it began and what not. I would like to have a history section in the wiki

Martinus that looks awesome! It pure art.

Regarding donations: Might be a good idea. But there are multiple people involved. For example Remco is paying/sponsoring/hosting our forum , wiki and other stuff.

So i cant just decide this. Ill discuss it with remco and martinus to figure out how to do this.

Martinus wrote: ↑29 May 2017, 20:41 Testrigs are essential for a developer. Had one before, but looked like this

Testlab.jpg

And that was only half of it, forgot to make a picture of the NodeMCU testbed... (looked a bit better but not much, considering all the jump wires)
All sensors where attached at multiple units, but back in those days we did not have that many sensors yet...

Chaos reigns! If you want to bring ESPEasy to the next level, this won't do...

I started working on the rig: First i'm building scripts to do things like compiling/erasing flash/flashing/wifi setup/serial connections.

I will try to make scripts reusable and build a clean modular rig.

Eventually the scripts are able to do everything automaticly on as many ESP's as we have. So, also things like configuring via http.

Anyway, i already found a bug: erasing the flash and reflashing it with espeasy will not correctly restart the unit after factory formatting. I'll fix that first. (a manual reset will fix it but i its annoying and confusing)

By scripting everything we get a regression test and i'm able to detect and reproduce these kinds of errors more easy.

Domosapiens wrote: ↑29 May 2017, 01:01 psy0rz,

Because I might get a financial grant to support ESPEasy (more on this later)

I like to donate too.
Is there somewhere a Donate button like Domoticz has?
http://www.domoticz.com/downloads/

psy0rz & Dev. team.
Thanks for all the time spent and the great result.

Domosapiens

Well you can now, i have a patreon: https://www.patreon.com/psy0rz

$1 each month would already be very nice.

psy0rz wrote: ↑19 Jan 2018, 18:00

Domosapiens wrote: ↑29 May 2017, 01:01 psy0rz,

Because I might get a financial grant to support ESPEasy (more on this later)

I like to donate too.
Is there somewhere a Donate button like Domoticz has?
http://www.domoticz.com/downloads/

psy0rz & Dev. team.
Thanks for all the time spent and the great result.

Domosapiens

Well you can now, i have a patreon: https://www.patreon.com/psy0rz

$1 each month would already be very nice.

Yay!

I like to reward the team for all their work, but ..
Sorry, this is too complicated and too much waste.
- PayPal redirection does not work
- 21% VAT goes to ???
Please open just a PayPal account for gifts.

Domosapiens wrote: ↑19 Jan 2018, 19:48 I like to reward the team for all their work, but ..
Sorry, this is too complicated and too much waste.
- PayPal redirection does not work
- 21% VAT goes to ???
Please open just a PayPal account for gifts.

Where did you find the Vat info? Opening a PayPal account for gifts is a good idea as well. But I like Patreon so no need to scratch that.

The uPyEasy testrig components are starting to arrive:
For the ESP32
For the ESP-12F

On these boards i'll place the most common sensors to test protocols, like the ds18b20 for onewire, bme280 for i2c, microswitch for switch, neopixel for gpio and a SD card for SPI. All of the boards are placed in plastic boxes and use usb which are attached to an usb-hub, while the usb-hub is attached to unix build platform. Stacked, organized and neatly stored, just like a rig should be!

The only gift i accept is your time.

Wow Lisa. Great boards. Made them yourself?

grovkillen wrote: ↑20 Jan 2018, 16:55 Wow Lisa. Great boards. Made them yourself?

Ali made them for me... https://www.aliexpress.com/item/7x9CM-7 ... 46336.html

LisaM wrote: ↑20 Jan 2018, 18:17

grovkillen wrote: ↑20 Jan 2018, 16:55 Wow Lisa. Great boards. Made them yourself?

Ali made them for me... https://www.aliexpress.com/item/7x9CM-7 ... 46336.html

Ohhh nice!!!!

If only they would fit in the 70x50 el-cheapo cases, then it would be perfect.

Edit:

Strike that.... they exist
https://www.aliexpress.com/item/5x7CM-5 ... 262.FxlnfN

How do you power these? What power regulator do you use?

LisaM wrote: ↑20 Jan 2018, 18:17

grovkillen wrote: ↑20 Jan 2018, 16:55 Wow Lisa. Great boards. Made them yourself?

Ali made them for me... https://www.aliexpress.com/item/7x9CM-7 ... 46336.html

Thank you! I've been looking for something like that. Ordered 10 straight up

TD-er wrote: ↑20 Jan 2018, 20:12 If only they would fit in the 70x50 el-cheapo cases, then it would be perfect.

Edit:

Strike that.... they exist
https://www.aliexpress.com/item/5x7CM-5 ... 262.FxlnfN

How do you power these? What power regulator do you use?

I've these 5x7 boards also, i use them for experimenting.

Bigger board:
Regulator 5v: https://www.aliexpress.com/item/5pcs-22 ... 35430.html with 3.3v step-down: https://www.aliexpress.com/item/1-pcs-2 ... 12156.html

Smaller board:
dual 5v/3.3v power supply: https://www.aliexpress.com/item/220V-to ... 49666.html (Edited: See remarks below!)

TTL-Serial converter: https://www.aliexpress.com/item/Free-Sh ... 32750.html

Domosapiens wrote: ↑19 Jan 2018, 19:48 I like to reward the team for all their work, but ..
Sorry, this is too complicated and too much waste.
- PayPal redirection does not work
- 21% VAT goes to ???
Please open just a PayPal account for gifts.

Done: https://www.paypal.me/psy0rz

I also have around 10 types of cryptowallets, so that would be possible as well.

LisaM wrote: ↑20 Jan 2018, 15:25 The uPyEasy testrig components are starting to arrive:

esp32 testboard.jpg
For the ESP32

ESP-12F testboard.jpg
For the ESP-12F

On these boards i'll place the most common sensors to test protocols, like the ds18b20 for onewire, bme280 for i2c, microswitch for switch, neopixel for gpio and a SD card for SPI. All of the boards are placed in plastic boxes and use usb which are attached to an usb-hub, while the usb-hub is attached to unix build platform. Stacked, organized and neatly stored, just like a rig should be!

The only gift i accept is your time.

Looks nice!

Did you start building on an automated testsuite? Perhaps some files i'm working on might be usefull, some are pretty generic:

Low level stuff: Handles things like communication via serial, reboot via DTR lines, build, flash, wifi config, serial/wifi pinging, post data via http etc: https://github.com/letscontrolit/ESPEas ... st/node.py This needs a PATH environment variable to the platformio command.

Controller emulation: https://github.com/letscontrolit/ESPEas ... lleremu.py This starts some threads to handle HTTP requests, line based protocols and MQTT (via a broker). Currently it "emulates" domoticz http/mqtt, so I can verify if a sensor transmits the correct data. Will add the other controllers soon.

All the actual tests are in the testXXX files, created in such a way I can easily copy/paste them for new tests. HTTP post-data can be just copypasted from a Chromium network tab to a http_post() call.

To run a test, just start the specific file. I will add a wrapper later to run all tests in sequence. Maybe later automaticly against every every new commit?

espcli is a console interface to call stuff from node.py and others. I often use "./espcli node 0 bfs" to build/flash and open a serial console.

I didnt had the time to look into uPyEasy yet (sounds great, much better than trying to debug C , but would it be feasible to work on a test framework together?

Perhaps generalize some parts a bit more so that we both can make our own seperate tests for espeasy and upyeasy? (in a seperate test-repository) I already send you an invite to the ESPEasy slack channel a while ago, if you didt receive it let me know.

Edwin

Done: https://www.paypal.me/psy0rz

LisaM wrote: ↑20 Jan 2018, 21:21 Smaller board:
dual 5v/3.3v power supply: https://www.aliexpress.com/item/220V-to ... 49666.html

I never ordered such small PS as I though 200mA is not enough to run an ESP8266 in a stable way (even worse with ESP32 I guess).
Do you confirm they are able to run stable as much as the HiLink you linked above? (which I use 99% of the times as proven very very reliable)?

papperone wrote: ↑21 Jan 2018, 10:04

LisaM wrote: ↑20 Jan 2018, 21:21 Smaller board:
dual 5v/3.3v power supply: https://www.aliexpress.com/item/220V-to ... 49666.html

I never ordered such small PS as I though 200mA is not enough to run an ESP8266 in a stable way (even worse with ESP32 I guess).
Do you confirm they are able to run stable as much as the HiLink you linked above? (which I use 99% of the times as proven very very reliable)?

For a single ESP8266/ESP32 chip (like ESP-01, or ESP32 with BT off) it's enough (barely): http://bbs.esp32.com/viewtopic.php?t=2662 / http://bbs.espressif.com/viewtopic.php?t=133, anything else it's:
HLK 3.3V https://www.aliexpress.com/item/HLK-PM0 ... 95801.html + Step Up 3.3V to 5Vhttps://www.aliexpress.com/item/8PCS-1- ... 942507.htm
or... HLK 5v + Step down to 3.3V (when using CO2 IR sensors for example since they eat a lot of 5V power, up to 150ma!)
If you want to be on the safe side, just buy/use the HLK since that always works and don't cost that much more like you already said.

I have everything and use a different combination for each project...

Ps. i'll do a duration test on my scope, just don't know when i'll have time for that.

psy0rz wrote: ↑21 Jan 2018, 04:59 Looks nice!

Did you start building on an automated testsuite? Perhaps some files i'm working on might be usefull, some are pretty generic:

Low level stuff: Handles things like communication via serial, reboot via DTR lines, build, flash, wifi config, serial/wifi pinging, post data via http etc: https://github.com/letscontrolit/ESPEas ... st/node.py This needs a PATH environment variable to the platformio command.

Controller emulation: https://github.com/letscontrolit/ESPEas ... lleremu.py This starts some threads to handle HTTP requests, line based protocols and MQTT (via a broker). Currently it "emulates" domoticz http/mqtt, so I can verify if a sensor transmits the correct data. Will add the other controllers soon.

All the actual tests are in the testXXX files, created in such a way I can easily copy/paste them for new tests. HTTP post-data can be just copypasted from a Chromium network tab to a http_post() call.

To run a test, just start the specific file. I will add a wrapper later to run all tests in sequence. Maybe later automaticly against every every new commit?

espcli is a console interface to call stuff from node.py and others. I often use "./espcli node 0 bfs" to build/flash and open a serial console.

I didnt had the time to look into uPyEasy yet (sounds great, much better than trying to debug C , but would it be feasible to work on a test framework together?

Perhaps generalize some parts a bit more so that we both can make our own seperate tests for espeasy and upyeasy? (in a seperate test-repository) I already send you an invite to the ESPEasy slack channel a while ago, if you didt receive it let me know.

Edwin

I haven't decided on any (software) framework for uPyEasy so far, i welcome all suggestions! So far i only ordered the hardware, since it takes a few weeks to arrive (pcb's did arrive, like most sensors). Now i have almost all components, need to find time to put everything together (and continue to work on uPyEasy, and finished my other project, and work so i get paid, and .... )

I looked around abit for standard testsoftware, but then i decided to build something myself.

Most standard unit testing is made to test libraries and functions on the native system. We want to actually build and flash firmware and then http post and receive sensor data, while emulating actual controllers. There might be something that can do it, but as usual with python: its easier and quicker to implement something ourself for this specific usecase.

In case you find an existing framework that doesnt have a huge learning case and is better then what I have right know, let me know. For now this works and i'm actually starting to benefit from it.

LisaM wrote: ↑20 Jan 2018, 18:17

grovkillen wrote: ↑20 Jan 2018, 16:55 Wow Lisa. Great boards. Made them yourself?

Ali made them for me... https://www.aliexpress.com/item/7x9CM-7 ... 46336.html

I screwed up the first board, just couldn't get the esp32 onto the board without bridging two or more pins. For now i gave up on it.
Ok, i have to rethink/find out how to solder the esp32/esp8266 onto such a board...

LisaM wrote: ↑22 Jan 2018, 22:34

LisaM wrote: ↑20 Jan 2018, 18:17

grovkillen wrote: ↑20 Jan 2018, 16:55 Wow Lisa. Great boards. Made them yourself?

Ali made them for me... https://www.aliexpress.com/item/7x9CM-7 ... 46336.html

I screwed up the first board, just couldn't get the esp32 onto the board without bridging two or more pins. For now i gave up on it.
Ok, i have to rethink/find out how to solder the esp32/esp8266 onto such a board...

Sorry to hear that... hot air gun + soldering paste should do the trick, essentially smd soldering.

https://www.ebay.de/i/282663336169?chn=ps
that one works for me

LisaM wrote: ↑22 Jan 2018, 22:34

LisaM wrote: ↑20 Jan 2018, 18:17

grovkillen wrote: ↑20 Jan 2018, 16:55 Wow Lisa. Great boards. Made them yourself?

Ali made them for me... https://www.aliexpress.com/item/7x9CM-7 ... 46336.html

I screwed up the first board, just couldn't get the esp32 onto the board without bridging two or more pins. For now i gave up on it.
Ok, i have to rethink/find out how to solder the esp32/esp8266 onto such a board...

I ordered a bunch. Hopefully I get more lucky.