ESPEasyDevelopment

From Let's Control It
Jump to navigation Jump to search

WIP.gifYou may hear some construction noise in the background...

Welcome to the developers documentation page!

To aid in contributions to the project, it may come in handy to have some background information. Still very limited, but we have to start somewhere (and we did not start from the beginning...)


General concept

ESP Easy is build upon the Arduino ESP8266 core (check out on https://github.com/esp8266). This code package provides the Arduino way of working on an ESP8266 module with many ported libraries.

ESP Easy consists of a framework that provides basic functionality and a collection of controller and device plugins. Controller plugins (_Cxxxx files) provide communication to Home Automation platform like Domoticz, OpenHAB, etc. Device plugins (_Pxxx files) provide communication to a wide collection of sensor devices and some actuators. Because ESP Easy was designed as a "wireless multi-sensor" project, the actuator part is limited.

Remember that ESP Easy is primarily a sensor device. It handles measurements and stores output in numerical floating point variables that can be send over the network.

Data structures

We have a lot of data structures in ESP Easy. The most important ones:

Settings - Holds the majority of system configurations
SecuritySettings - Holds things like Wifi SSID, passwords
ExtraTaskSettings - Holds the current actual copy of additional task settings into RAM

Important: Most of the data within ESP Easy is always accessible with exception to ExtraTaskSettings. Because most structures use global RAM they put a heavy load on resources. Tasks have a lot of string data and we can't afford to keep this data in RAM for all 12 tasks! So there's only one copy available in ExtraTaskSettings. Before working on some task, the configuration needs to be loaded from flash. The framework handles this before a call is made to a device plugin.

Device plugins

In many cases, it is quite easy to add support for a specific device. The framework periodically calls each device plugin with a different type of function type. The minimum set:

PLUGIN_DEVICE_ADD - Describes device characteristics to the framework
PLUGIN_GET_DEVICENAME - Returns the device name to the framework
PLUGIN_GET_DEVICEVALUENAMES - Returns the default value names to the framework
PLUGIN_READ - This is called when the framework reads a device (sensor)

Because ESP Easy is basically a sensor device, everything is hooked upon a large floating point variable array (UserVar[]). It provides a maximum of 4 variables for each task. A device can be loaded more than once, so each 'instance' has it's own set of variables. The event->BaseVarIndex variable points to the first available variable so you don't have to calculate this yourself.

Two other useful calls, often used in switch devices that use a fast polling technique to read data:

PLUGIN_TEN_PER_SECOND - This is called 10 times per second for speedy response in polling scenario's
PLUGIN_ONCE_A_SECOND - This is called once per second for not so speedy stuff

Remember that we have a single threaded model so don't spend to much time handling things within the plugin as it will stall all other processing!

The easiest way to create a new plugin is just by copying one that looks a bit similar to the new device. Renumber all XXX numbered references to a new unique id (check the plugin playground) to avoid conflicts with the original.

Remember that you can only store floating point numbers as output from your device. Also note that the floating point precision is limited. For most application this will not lead to issues but storing a large 32 bit number like RFID tags will get rounded. As a workaround, you can store this into two floating point variables as 16 bit parts. Selecting SENSOR_TYPE_LONG will handle this within the framework. But it supports only one long value!

Plugin configuration

Basic configuration is already provided by the framework, like names, GPIO selection, port selection, etc. So maybe you don't have to worry about this at all. You just have to read the required variables if you need them:

Settings.TaskDevicePin1[event->TaskIndex] - The first GPIO pin selected within the task
Settings.TaskDevicePin2[event->TaskIndex] - The second GPIO pin selected within the task
Settings.TaskDevicePort[event->TaskIndex] - The port in case the device has multiple in/out pins

If you need some custom device configuration, the framework provides 8 variables to store these custom settings. They are 16 bit so the value can range between -32767 and +32768. They can be used in this way:

Settings.TaskDevicePluginConfig[event->TaskIndex][x] - Custom setting

Where x can be 0 to 7. If a 16 bit var does not fit your purpose, there are also 4 float and long variables available:

Settings.TaskDevicePluginConfigFloat[event->TaskIndex][x] - Custom setting
Settings.TaskDevicePluginConfigLong[event->TaskIndex][x] - Custom setting

In case you have a custom config, of course the framework does not know this custom stuff. So you need to expand the web gui with additional fields. The framework has two calls to handle this:

PLUGIN_WEBFORM_LOAD - code to be executed when the web form loads, typically a HTML input box
PLUGIN_WEBFORM_SAVE - code to be executed when the form is submitted, typically read the field and store into the settings variable

PLUGIN_READ

This is the most interesting call to plugins. It's based on tasks, so the user has to setup one or more tasks using this plugin. Remember that the plugin can be used more than once and the frameworks holds configuration data for each task using this plugin. The variable "event->TaskIndex" holds a pointer to the current active task called by the framework. So if you need to read a GPIO pin, you need the actual pin configured for this current task like this: digitalRead(Settings.TaskDevicePin1[event->TaskIndex]);

PLUGIN_WRITE

Some devices like actuators can have commands to set their state. When the ESP Easy framework was in it's early development stage, it was decided that plugins can add 'control commands' to ESP Easy and we moved basic GPIO output handling from the framework to _P001_Switch.

It is important to understand that this WRITE call within plugins, adds 'commands' to ESP Easy but they are not in any way related to tasks (!!). When a command is received within the framework, it does not check tasks but simply runs the command through all compiled plugins until it finds a match (plugin returns true). This means that within the PLUGIN_WRITE code block, you can't address any of the task related stuff directly as there's no relation to tasks.

If you want to access task related its up to the plugin to implement a method to determine the corresponding task. One way to do it is by using the taskname as a selector:

case PLUGIN_WRITE:
 {
   String command = parseString(string, 1);
   if (command == F("demo"))
   {
     String taskName = parseString(string, 2);
     int8_t taskIndex = getTaskIndexByName(taskName);
     if (taskIndex != -1)
     {
       success = true;
   // from here you can use the taskIndex to acces all stuff from the task that is referenced!
       // show something on serial to prove that this works:
       Serial.println(ExtraTaskSettings.TaskDeviceName);
       Serial.println(Settings.TaskDevicePluginConfig[taskIndex];
     }
   }
   break;
 }



Controller plugins

The framework calls the selected controller during init phase and each time data from a sensor must be send. The minimum set:

CPLUGIN_PROTOCOL_ADD - Describes controller characteristics to the framework
CPLUGIN_GET_DEVICENAME - Returns the controller name to the framework
CPLUGIN_PROTOCOL_SEND - This is were the magic happens, sending data to your Home Automation platform.

Data transfer between devices and controllers is handled though the UserVar[] array and a special data structure "event". This data structure holds additional information needed to process the sensor data.

event->TaskIndex - The index to the task that made the request
event->BaseVarIndex - points to the actual position in the uservar array for this send request
event->idx - holds the IDX value if needed
event->sensorType - provides information on what type of sensor made the request
event->String1 - in case of MQTT, this holds the topic
event->String2 - in case of MQTT, this holds the payload

And you have access to other task related information like:

ExtraTaskSettings.TaskDeviceName - the active task name that made the request
ExtraTaskSettings.TaskDeviceValueNames[x] - An array of value names for this task
ExtraTaskSettings.TaskDeviceValueDecimals[x] - Corresponding decimal rounding requested

Configuration storage

The project needs to store it's configuration and this will be done in the on board flash memory chip. In the beginning we have used the emulated EEPROM but we have abandoned that because of it's limitations and permanent RAM usage. We moved over to SPIFFS but that also introduced a rather large decrease of available RAM. So we moved over to reading/writing directly to flash memory. For convenience, we use the same area that is normally reserved for SPIFFS, so we don't have to worry about locating a suitable area. This also means that we can't use SPIFFS and compiling without reservation for SPIFFS wont work with ESP Easy.

With a minimum of 64k SPIFFS reservation and the available emulated EEPROM area, we have a total of 17 sectors available, 4k each.

This is our current layout:

Sector 0	Global settings struct
Sector 1-3	Task settings (1kB per task, first 512 bytes for system, 512 remaining can be used for custom task config)
Sector 4-6	Reserved for future expansion for Task settings
Sector 7	Custom controller config
Sector 8	Security settings
Sector 9	CSS config
Sector 10	Rules
Sector 11-16	future use

When saving the configuration, sectors 0-7 (32k) are saved to file. This is because sector 8 contains security data. Because CSS and rules are plain text, they can be saved in other ways. Sector 0-7 contains binary data.

The framework handles most of the work so you don't have to worry about this yourself. But the layout can explain why things are limited and that's something you need to be aware of if you start to use custom data structures for configuration.

A controller plugin can have custom configuration up to 4096 bytes. A device plugin can only use 512 bytes for it's custom configuration.

In most cases you don't need a custom config, but the framework provides these system calls to easy things if you need it:

To load/save data structures from/into flash when using a device plugin:

 LoadCustomTaskSettings(event->TaskIndex, (byte*)&customConfig, sizeof(customConfig));
 SaveCustomTaskSettings(event->TaskIndex, (byte*)&customConfig, sizeof(customConfig));

To load/save data structures from/into flash when using a controller plugin:

 LoadCustomControllerSettings((byte*)&customConfig, sizeof(customConfig));
 SaveCustomControllerSettings((byte*)&customConfig, sizeof(customConfig));

In these samples, the 'customConfig' is a data structure that can hold a collection of variables.


Networking protocols

The framework provides basic handling of several generic and well known networking protocols so the basic setup is already present. The framework runs a webserver, can connect to a MQTT broker and listens to generic UDP traffic. These will be explained in more detail below:

HTTP support

The framework starts a webserver at boot and controls the administration web gui. The web gui can be expanded from within device and controller plugins when needed. Special control commands can be send to the webserver and they will be run against all device plugins. Use the PLUGIN_WRITE call to catch the command and process required action within the plugin.

Controller plugins control their own local webclient to connect to Home Automation servers or webservices.

MQTT support

The framework connects to a configured MQTT broker at boot and catches any messages for topics that it's subscribed to. The currently selected controller plugin receives a CPLUGIN_PROTOCOL_RECV call and MQTT data is passed using the event struct.

Controller plugins can simply publish data on the existing broker connection.

UDP support

The framework listens to a user defined UDP port for internal system messages.

Network logging

Logging to a syslog server is handled by the framework. Within plugins, you only need to use the addLog() system call.