"SmartNodeRules" concept idea

[FanOfHue]

I have started to work on a "SmartNodeRules" idea, further explained here: https://github.com/SmartNodeRules/Documentation/wiki

I just finished creating some prototypes to demonstrate how this could work in my own home and so far it seems to work as planned. A plugin for ESPEasy is also available to let it become a node on this network. The concept has been set with KISS and YAGNI in mind (https://en.wikipedia.org/wiki/You_aren% ... na_need_it)

It could be nice to have a RaspBerryPI edition prototype, but my skills on Linux are mainly non existing. and i don't own a RPI (yet).

Would like to hear if someone else likes the idea. If you already own a fancy Home Automation controller, you likely don't need this. But I only have a simple Hue bridge which is limited to lighting stuff. My older Arduino hardware has now been migrated to this concept and I'm using two shortcuts on my smartphone to control everything. One is the Hue app, the other is a web shortcut to the main page on one of the smartnode units. It looks like i don't need much more than this right now.

[dynamicdave]

Hi,
That sounds like a brillant idea - how much of it is workable at the moment???

I have a home automation system running on a couple of RPi-3s using Node-RED and MQTT.
Around the house I have a number of WeMos D1 Minis (re-flashed with ESP Easy) that send various readings (temp, humdiity, pressure, switch settings) via MQTT to NR.
I also have a number of SonOff-S20 mains switches that communicate with NR via Tasmota.
Also some Yeelights and Alexa devices that "talk" to my system.

The current system works really well and has been very reliable for a couple of years.

I have a spare RPi, that I use for development work, that I could devote to checking out your SmartNodeRules.

Looking forward to hearing more of your project development.

Kind regards and Happy Christmas from Dynamic Dave in the UK.

[ThomasB]

I use OpenHab for home control, so I don't require a standalone solution. But there are ESP Easy that would benefit from your concept. So I just stopped by to show my support.

- Thomas

[tim]

I'm using ArduinoEasy for a while with Domoticz. Could be nice to have a small webgui with power control buttons to control this unit directly. Maybe in case Domoticz has crashed like it did a while ago when the SD card in the RPI suddenly died.

ArduinoEasy does not have such an option and it seems there's no development on that version anymore because it is listed as archived on the wiki. My Arduino Mega unit has a double relay board attached so i only need 4 buttons.

But can the ArduinoMegaRules version work with Domoticz?
The wiki says that it works without a centralized Home Automation controller, but can't we have both options, so with and or without?

[FanOfHue]

I'm using ArduinoEasy for a while with Domoticz. Could be nice to have a small webgui with power control buttons to control this unit directly. Maybe in case Domoticz has crashed like it did a while ago when the SD card in the RPI suddenly died.

ArduinoEasy does not have such an option and it seems there's no development on that version anymore because it is listed as archived on the wiki. My Arduino Mega unit has a double relay board attached so i only need 4 buttons.

But can the ArduinoMegaRules version work with Domoticz?
The wiki says that it works without a centralized Home Automation controller, but can't we have both options, so with and or without?

Well it could be controlled by anything that can send plain UDP text messages to trigger an event on one of the nodes. I don't know if Domoticz can do that?
Maybe I should update the wiki because it was not intended to work as standalone exclusively.

But http control might also be an option as the webserver is still there for that kind of basic tasks like processing events.

[FanOfHue]

I use OpenHab for home control, so I don't require a standalone solution. But there are ESP Easy that would benefit from your concept. So I just stopped by to show my support.

- Thomas

Thanks for taking the time to respond. I presume that you're using MQTT with OpenHAB? I have a testversion ready that can use both UDP as MQTT as a generic broacast type message bus. Running Mosquitto on Windows, just to verify the workings. Maybe MQTT is a better choice for a message bus, but in case you only have two nodes, you could fallback to bare UDP peer to peer.

But maybe I'm the only one here that thinks that way ? Everyone uses fancy controllers?

[FanOfHue]

Hi,
That sounds like a brillant idea - how much of it is workable at the moment???

I have a home automation system running on a couple of RPi-3s using Node-RED and MQTT.
Around the house I have a number of WeMos D1 Minis (re-flashed with ESP Easy) that send various readings (temp, humdiity, pressure, switch settings) via MQTT to NR.
I also have a number of SonOff-S20 mains switches that communicate with NR via Tasmota.
Also some Yeelights and Alexa devices that "talk" to my system.

The current system works really well and has been very reliable for a couple of years.

I have a spare RPi, that I use for development work, that I could devote to checking out your SmartNodeRules.

Looking forward to hearing more of your project development.

Kind regards and Happy Christmas from Dynamic Dave in the UK.

I guess that you have also noticed the RPIEasy project? So it's not needed to build something new. But a universal message bus would still be on my wish list to exchange event between multiple IoT platforms.

[ThomasB]

Everyone uses fancy controllers?

I suspect that many ESPEasy users {that have elaborate home automation} are using fancy controllers. My automation devices utilize Amazon Echo speech and a lot of rules, so OpenHab is essential to me. But your concept is ideal for those that have less demanding applications.

- Thomas

[tim]

On Arduino Mega:

Sending this http://<ip>/control?cmd=gpio,12,1

I get this error:

Unknown or restricted command!

Could you have a look at it?

[FanOfHue]

On Arduino Mega:

Sending this http://<ip>/control?cmd=gpio,12,1

I get this error:

Unknown or restricted command!

Could you have a look at it?

That should be solved on github now. In the mean time, a new feature has been added to show nodes in groups. A new command:

Config,Group,<group name>

has been introduced. Th main page now shows a list of groups based on the nodelist. Click on a group to show all nodes within that group.

[happytm]

I looked at you wiki page and it is interesting concept. I sometimes need minimum setup for small project.we have ongoing discussion at https://github.com/enesbcs/rpieasy/issues/73 regarding ESP-Now protocol.Please share your thoughts there.I am trying to understand few things from your wiki.

- Does this implementation support dummy sensor ? For me dummy sensor plugin makes rule engine more powerful.
- are any other plugin supported ? if so how to implement them ? I would like to have both IR plugins standard in any implementation.

Thanks

[FanOfHue]

I looked at you wiki page and it is interesting concept. I sometimes need minimum setup for small project.we have ongoing discussion at https://github.com/enesbcs/rpieasy/issues/73 regarding ESP-Now protocol.Please share your thoughts there.I am trying to understand few things from your wiki.

- Does this implementation support dummy sensor ? For me dummy sensor plugin makes rule engine more powerful.
- are any other plugin supported ? if so how to implement them ? I would like to have both IR plugins standard in any implementation.

Thanks

I've created that project because I needed a small core for private development of plugins. Size is only 350k so I can OTA without a double step on the older 1M modules. It also leaves a lot of free RAM to use for plugins.

It's in stable production for approx. 6 months now and has not required any updates since. It does not work with devices but instead all is scripted using the ESP Easy rule engine. I guess the dummy device is used to store measurements, but this project just uses named variables that can be used within the rule engine. Plugins need to be modified because the configuration will be command based instead of a gui.

So in short: it's just a network core with a web-based script editor and a rule engine.

[happytm]

Understood.

Thanks.

[TD-er]

I really like the idea.

The ESPeasy p2p protocol is getting some rewrite to share plugin data among nodes.
If you need any changes/additions or have some good ideas, please let me know.
This is just something I was thinking about also, but was not yet able to make it due to time and time lost in the track of WDT reboots.

Edit:
If you like, I can also add this plugin to the main builds: https://github.com/SmartNodeRules/ESPEa ... eRules.ino

[FanOfHue]

I really like the idea.

The ESPeasy p2p protocol is getting some rewrite to share plugin data among nodes.
If you need any changes/additions or have some good ideas, please let me know.
This is just something I was thinking about also, but was not yet able to make it due to time and time lost in the track of WDT reboots.

Edit:
If you like, I can also add this plugin to the main builds: https://github.com/SmartNodeRules/ESPEa ... eRules.ino

In my home, all is now driven from simple textbased UDP messages. Philips Hue is controlled through a Windows .net version of SmartNodeRules. (because the windows node can run SSL very fast. ESP32 can also do it, but it takes a few seconds to establish the SSL connection)
Any node can control Hue lights and sockets using the Windows Node as an SSL bridge to Hue.

The ESP build-in P2P protocol was a binary format version and i decided to use just plain text messages on a broadcast UDP port instead of unicast.
Much easier and a lot faster. I've also added a very basic 'confirmation' option, but i've not used it yet.

I admit that the protocol is not secure at all, but it only lives in the dedicated VLAN where all IOT devices are connected.
Maybe it would need some form of encryption with a preshared key to make it more secure but still keep it simple to operate.

[TD-er]

[...]
The ESP build-in P2P protocol was a binary format version and i decided to use just plain text messages on a broadcast UDP port instead of unicast.
Much easier and a lot faster. I've also added a very basic 'confirmation' option, but i've not used it yet.
[...]

About the broadcast messages. Do you miss some every now and then on nodes connected to WiFi?
One of the things I really want to change is the way the p2p protocol is sending packets to any other node using unicast.
That's at least O(n^2) and maybe even worse.
But I've noticed sometimes broadcast messages get lost on some of the WiFi connected nodes.
Even unicast messages sometimes get lost.

[FanOfHue]

Would be great if the plugin is added and maybe other people could have use cases for it.

I currently use the messagebus for sensor data that is repeated every minute so i guess that some missed messages get unnoticed. Looking at the sensor graphs, i don't see any gaps.

I've defined a confirmation format that could be used for sending commands. with a very basic queue/retry/confirm 'protocol'
That might compensate for using the UDP protocol that is unreliable by design (but very easy to implement, fast and lightweigth)

What do you guys think of my simple MessageBus 'protocol' that uses 100% plain text messages without a predefined structure?

Only reserved these special formats:

1) To announce a node on the messagebus:
MSGBUS/Hostname=<nodename>

2) To force other nodes to announce themselve immediatly:
MSGBUS/Refresh

3) To get confirmation of a specific target node, use the > and < token as the first character in the message:
><nodename>/<msg>
(the > token gets stripped by the receiving code before the message is transferred to the event processing engine. So there's no difference receiving it with or without a confirmation request)

Sample confirmation transmission on the UDP bus:
>OutdoorNode/Lights On
will be confirmed on the UDP bus as:
<OutdoorNode/Lights On
and the event for the rule engine will be like:
OutdoorNode/Lights On

For all my sensors, i use a well-known structure that is also typically used in MQTT:

Outdoor/Temperature=8.5
Outdoor/Lux=23000

[dynamicdave]

I'd agree with that - I would be interested to try out SmartNodeRules.

[FanOfHue]

I've just updates several repositories on github. Main update is the addition of the group feature on the Node List. Because the list of nodes has grown rapidly in my home, i can now group devices and the main landing web page first shows the discovered groups. Selecting a group will list all nodes inside that group.

groups.png (8.01 KiB) Viewed 32827 times

[happytm]

Can you please give little how to guide so I can setup my esp8266 device using your firmware. I looked at wiki pages but it does not have information on how to get started.I was able to compile & upload for esp8266 but in serial console I get this:

⸮ogl`orǛ⸮g#⸮ls⸮⸮gb⸮d⸮⸮dd⸮⸮$`⸮⸮n⸮
SPIFFS?
sd

I was looking for access point on my network but I do no see any ?

Thanks

[FanOfHue]

Can you please give little how to guide so I can setup my esp8266 device using your firmware. I looked at wiki pages but it does not have information on how to get started.I was able to compile & upload for esp8266 but in serial console I get this:

⸮ogl`orǛ⸮g#⸮ls⸮⸮gb⸮d⸮⸮dd⸮⸮$`⸮⸮n⸮h
SPIFFS?
sd

I was looking for access point on my network but I do no see any ?

Thanks

Because i migrated from ESP Easy to my own mini core, setup feature has never been build, other than auto-creating a boot file with SSID and key.
I assume you are using a fresh ESP module. My firmware expects an existing SPIFFS partition with an existing security.dat file.

For now, easiest way is to install and setup Wifi connection using ESP Easy Mega edition and then upload the SmartNodeRules bin file. IP address will default to DHCP settings.

I still use ESP Easy to start with on new modules and then replace the firmware.

[happytm]

Thanks for your clarification. I am trying to build on work you have done so far.

[happytm]

I tried as you described process in earlier post. I also uploded boot.txt and rules.txt files from your wiki page using espeay. I get following serial data but no gui at fixed ip:

Thanks



273 : WIFI : Connecting HTM1 attempt #0
274 : IP : Static IP : 10.1.10.70 GW: 10.1.10.1 SN: 255.255.255.0 DNS: 10.1.10.1
276 : WIFI : Not configured in Station Mode!!: HTM1
1612 : WD : Uptime 0 ConnectFailures 0 FreeMem 23216 WiFiStatus 0
4151 : WIFI : Connected! AP: HTM1 (00:27:22:E2:F0:52) Ch: 7 Duration: 3775 ms
4152 : IP : Static IP : 10.1.10.70 GW: 10.1.10.1 SN: 255.255.255.0 DNS: 10.1.10.1
4153 : WIFI : Static IP: 10.1.10.70 (IR-Extended-0) GW: 10.1.10.1 SN: 255.255.255.0 duration: 103 ms
Update: ESPCoreRules.ino.d1_mini.bin
sleep disable
.........................................................................................................................................................................................Update Success: 377600
Rebooting...

ets Jan 8 2013,rst cause:2, boot mode:(3,6)

load 0x4010f000, len 1384, room 16
tail 8
chksum 0x2d
csum 0x2d
vbb28d4a3
@cp:0
ld

SPIFFS?

[happytm]

I was able to hard code some settings and esp device now connected as station on local network. But there is still no GUI and no serial communication. I am able to communicate via telnet.

Please help.

Thanks

[FanOfHue]

It looks like the SPIFFS partition does not mount for some reason. That's a major requirement to continue. Did you use exactly the same compiler settings for ESP Easy and ESPCoreRules?

[happytm]

Yes I did upload both bin file one after another within a minute from same computer. I think spiff is working because I hardcoded my access point and password and it saved it for me.

Thanks

[happytm]

I wiped whole esp with 4 mb blank and also tried with another esp device but still same problem.I also tried 192.168.4.1 ip after associating with ESP's AP but I get nothing in chrome's console or it does not show any source code which is odd. It is up with AP SSID but it get stuck after spiffs? serial print.The code upto that point is:

if (SPIFFS.begin())
{
event = "System#Config";
if (SPIFFS.exists(FILE_BOOT)){
rulesProcessing(FILE_BOOT, event);
}
else{
if (SPIFFS.exists(FILE_SECURITY)){
LoadSettings(); // use security.dat
}
else
{
SecuritySettings.WifiSSID[0] = 0;
SecuritySettings.WifiKey[0] = 0;
strcpy(SecuritySettings.WifiAPKey, "configesp");
}
fs::File f = SPIFFS.open(FILE_BOOT, "w");
f.println(F("on System#Config do"));
f.print(F(" Config,WifiSSID,"));
f.println(SecuritySettings.WifiSSID);
f.print(F(" Config,WifiKey,"));
f.println(SecuritySettings.WifiKey);
f.println(F("endon"));
}

rulesProcessing(FILE_RULES, event);

WiFi.persistent(false); // Do not use SDK storage of SSID/WPA parameters
WifiConnect(3);
WebServerInit();
WiFi.mode(WIFI_STA);
}
else
{
Serial.println("SPIFFS?");
delay(1);
}

[FanOfHue]

If you hardcoded the Wifi SSID it does not depend on SPIFFS. As long as it prints "SPIFFS?" on serial, the system is unable to mount SPIFFS.
From my experience it means that there is no formatted SPIFFS partition or it is in the wrong area. For Wemos D1 mini, i use 4M (1M SPIFFS) setting on both ESP Easy and ESPCoreRules. ESPEasy does the SPIFFS formatting and ESPCoreRules can just mount it normally. It will then read the security.dat file and convert it into a basic boot.t
xt file. But all that will fail if the SPIFFS mount fails.

I could add the initial setup stuff from ESP Easy R120 to make life easier, but i'm on holiday right now, so that will take a while before i can get to it.

you could move this line:
WebServerInit();
outside of the Spiffs.begin check and the webserver should be running. But without SPIFFS you would not be able to store any files.

[FanOfHue]

I had the latest dev ESPCoreRules code on my laptop. I added a "reset" command that should format and create the files.
Commit to github just now. This version also has serial debug enabled so you could see a bit more on serial.

You could try this:

Upload the latest code
Boot the ESP, wait until it's done.
type command: reset
wait until format is done
type command: reboot
AP should start and you should be able to connect on 192.168.4.1 to edit the boot file and reboot to connect as AP-STA

good luck!

[happytm]

Success at last.I am sorry I was using 2M spiff all this time not realizing it matters.Switched to 1M spiffs and all of sudden everything is fine.

I will now try latest commit.

Thank you for being so patient.

[tim]

I really like the idea.

The ESPeasy p2p protocol is getting some rewrite to share plugin data among nodes.
If you need any changes/additions or have some good ideas, please let me know.
This is just something I was thinking about also, but was not yet able to make it due to time and time lost in the track of WDT reboots.

Edit:
If you like, I can also add this plugin to the main builds: https://github.com/SmartNodeRules/ESPEa ... eRules.ino

When will it be part of official build?

[TD-er]

[...]
When will it be part of official build?

That's hard to say right now.
At the moment we're running into all kinds of build issues regarding build size, so adding more plugins is not going to help much then.

[FanOfHue]

[...]
When will it be part of official build?

That's hard to say right now.
At the moment we're running into all kinds of build issues regarding build size, so adding more plugins is not going to help much then.

What if you only add the one plugin so you can link SmartNodeRules network to ESPEasy network like some sort of gateway. What will the size be then? I guess it's less than 500k (?). ESPCoreRules on my doorsensor is only 350k.

[TD-er]

As you may have seen on GitHub, I've been very busy trying to free up space and thinking of other sets of plugins to combine, so we have free space for other additions.
Can you add this suggestion here: https://github.com/letscontrolit/ESPEasy/issues/2590

[FanOfHue]

As you may have seen on GitHub, I've been very busy trying to free up space and thinking of other sets of plugins to combine, so we have free space for other additions.
Can you add this suggestion here: https://github.com/letscontrolit/ESPEasy/issues/2590

Would be great if letscontrolit could provide a website where users can request their build-on-demand firmware.
Just a page with radiobuttons for controllers and plugins and a giant "BUILD NOW" button.
Preset build sets will likely never satisfy all users.

[ManS-H]

As you may have seen on GitHub, I've been very busy trying to free up space and thinking of other sets of plugins to combine, so we have free space for other additions.
Can you add this suggestion here: https://github.com/letscontrolit/ESPEasy/issues/2590

Would be great if letscontrolit could provide a website where users can request their build-on-demand firmware.
Just a page with radiobuttons for controllers and plugins and a giant "BUILD NOW" button.
Preset build sets will likely never satisfy all users.

Great idea.

[grovkillen]

Please remember that Let'sControlIt is an umbrella for all the different firmwares and hardware. ESP Easy is it's own ecosystem = we might have to have a espEasy.com site for it rather than I section of this forum.

[TD-er]

As you may have seen on GitHub, I've been very busy trying to free up space and thinking of other sets of plugins to combine, so we have free space for other additions.
Can you add this suggestion here: https://github.com/letscontrolit/ESPEasy/issues/2590

Would be great if letscontrolit could provide a website where users can request their build-on-demand firmware.
Just a page with radiobuttons for controllers and plugins and a giant "BUILD NOW" button.
Preset build sets will likely never satisfy all users.

You have seen the recent development where I made an initial start for this where users can define their own Custom.h file and build a version using Vagrant?