CANBus dev

[thalesmaoa]

Hi! My friend and I are working with a CANBus controller to build a reliable network for sending and receiving data.
I plan to share the development progress so far to get some feedback and suggestions. Also, this will help avoid messing up the code before making a pull request.

The basic structure:
Controller -> A new CAN controller is responsible for sending information to the bus.

image1.png (59.17 KiB) Viewed 5878 times

image2.png (58.51 KiB) Viewed 5878 times

A hardware config:

image3.png (63.7 KiB) Viewed 5878 times

The message structure is as follows:
[TaskIndex] [ValueIndex (in case of Task with 4 values)] [float byte1] [float byte2] [float byte3] [float byte4]

Each node uses a Plugin called CAN Import:

image4.png (75.48 KiB) Viewed 5878 times

image5.png (54.96 KiB) Viewed 5878 times

With plugin, I can redirect the messages using MQTT.
I still need to clean the code and solve some minor bugs before a pull request.

However, I'm now stuck in the best way to redirect commands to node in the network
As an example:
I need to activate a "relay" of "node65" (CanBus). The gateway is "node1".
I need "node1" to subscribe to a topic to receive the info. For that, I'm using "Generic - MQTT Import".
Thus, I can send values to the "node1". I was expecting to use it do "Send to Controller".
However, it does not have the option send to controller.
Dead end.

Another option is to develop a command:
CANSend, ....
With that, I can use mqtt, http or whatever. But I'm not sure if I should deep into commands. Also, I will probably need to change something in my controller.

The motivation is to build a system with almost 120 sensors (EspEasy). But I'm sure the wifi is gonna be weak and I should keep it reliable. This is way I need a gateway with ethernet cable. The controller must operate despite of wifi connection.
I'm planing to limit gateways to 32 devices since these is the max tasks.

[Ath]

Let's start with a practical tip: DO NOT place your images on an external service, within a couple of weeks/months these will be deleted by the external service. Please use the Attachments tab below the Reply edit field, and place them where you want in your message using the Place inline button next to the image. It will then be placed where your cursor was last in the text. The images will be available for as long as the forum will be available

Then about the concept of a Controller in ESPEasy:
- A Controller sends out data from tasks to an external service/host.
- Each task can be configured to use that Controller for output, if so desired.

Your CAN bus Controller looks like what we call a Plugin that can be configured in a Task, as plugins are used to collect data, that can be (optionally) processed, and sent to a Controller, like MQTT, HTTP etc., for further processing and storage.
In the Plugin configuration (Devices tab), per task any hardware settings can be stored; only generic/global stuff, like I2C, SPI, Ethernet etc. configuration, should be on the Hardware page.

If you need data from any task to be sent via the CAN bus to other devices, it can also be a Controller, in ESPEasy terms, but then the specific hardware configuration should be in the Controller page, and not the global Hardware page. Any CAN bus plugins that need that same configuration should then use that Controller configuration, or use services exposed by the Controller for retrieving data, instead of repeating those settings, similar to how the MQTT-related commands use the first enabled MQTT Controller to send out data to an MQTT Server.

[TD-er]

I think it looks more like how we're doing p2p communication.

I don't know what can be sent via CANbus. As far as I know it is more like an arbitrary length of bytes right?
So maybe you should take a look at how stuff is done for C013.

I think the CAN bus should be able to do more then just sensor data, but maybe also commands or events.

[thalesmaoa]

DO NOT place your images on an external service

Ok. Fixed in the original post.

Your CAN bus Controller looks like what we call a Plugin that can be configured in a Task,

Sorry, I'm a bit confused. Probably, my fault of not being clear. I do have a controller just to send data. In the other end, I just did a CAN Import to view data and redirect to MQTT. Is that what you mean?

only generic/global stuff, like I2C, SPI, Ethernet etc. configuration, should be on the Hardware page.

Perfect. I think I'm on the correct path.

onfiguration should be in the Controller page, and not the global Hardware page

Do you mean, TX and RX should be in hardware, but nodeID should be in the controller?

I think it looks more like how we're doing p2p communication.

Sure. It is really similar. I will look for it.

I think the CAN bus should be able to do more then just sensor data, but maybe also commands or events.

Good. We are trying events now, but commands, I'm still confused.

I appreciate feedback. It gave me a lot to think of.

[Ath]

Your CAN bus Controller looks like what we call a Plugin that can be configured in a Task,

Sorry, I'm a bit confused. Probably, my fault of not being clear. I do have a controller just to send data. In the other end, I just did a CAN Import to view data and redirect to MQTT. Is that what you mean?

Yes, your CAN import plugin seems like what I'd expect for receiving CAN data. A few questions I have are below.

only generic/global stuff, like I2C, SPI, Ethernet etc. configuration, should be on the Hardware page.

Perfect. I think I'm on the correct path.

Not sure if CAN bus will be promoted to a core protocol like SPI or I2C yet, that's why it could be best to have the hardware configuration in the Controller, as that seems to be kind of single-instance, and the receivers to make use of that (more questions below, though).

onfiguration should be in the Controller page, and not the global Hardware page

Do you mean, TX and RX should be in hardware, but nodeID should be in the controller?

For now, IMHO, all 'global' functional and all technical configuration could be in the Controller, for the receiver plugins to use.

I think it looks more like how we're doing p2p communication.

Sure. It is really similar. I will look for it.

I think the CAN bus should be able to do more then just sensor data, but maybe also commands or events.

Good. We are trying events now, but commands, I'm still confused.

I appreciate feedback. It gave me a lot to think of.

Handling commands in ESPEasy can be in 3 places: Global, Plugins and Controller (though currently all Controller commands are 'hiding' in the Global command handler). Events are generated from several places, determined by some cause that might need attention, and can be handled in Rules. From there on the user should be able to take care of the necessary actions to handle the cause (data received, specific situations like alerts and/or errors , etc.)

I've been reading a bit on CAN bus and found this rather interesting 'CAN bus for dummies/newbies'-like article: https://www.csselectronics.com/pages/ca ... o-tutorial

A few questions:
- There seem to be a few somewhat different versions of CAN bus, what version is supported here, 2.0, 2.0A, 2.0B, FD, XL? Especially the supported payload is different, from 8 to 2048 bytes, and that can be somewhat of a challenge for MCUs. Shouldn't the protocol version be part of the functional configuration?
- The payload data can be packed/encoded quite specific depending on the device that's sending it. How do you expect to handle that? Shouldn't that be configurable?
- You named your controller SJA1000, does that mean it's specific for the SJA1000 CAN bus controller chip? If so, are there other controller-chips to be supported/expected?
- Multiple CAN buses (as mentioned in the link above): Do you expect to handle multiple CAN buses in a single ESPEasy unit?
- What other areas than automotive are currently using CAN bus, that triggered you to add this feature to ESPEasy? As I'm not sure how much/often ESPEasy is used in the automotive sector

Probably I'll have more questions later

[thalesmaoa]

There seem to be a few somewhat different versions of CAN bus, what version is supported here, 2.0, 2.0A, 2.0B, FD, XL? Especially the supported payload is different, from 8 to 2048 bytes, and that can be somewhat of a challenge for MCUs. Shouldn't the protocol version be part of the functional configuration?

CAN and ESP32 are not so famous as other buses over the community. I've done a lot of things with CAN and used CANOpen for different applications. I will try to answer based only in my opinion and due to problems I've faced. CAN is extremely reliable and easy to use (physical layer).

Not all drivers (libraries) that we find online works well. Many of them has bugs. The ones that works, does not implement FD. The idea is to reuse them, and not reimplement.

Bellow FD, the only difference is the id length, which is something we should evaluate. For the first implementation, the idea is keep it up to 254 and get community feedback.

This is the main reason to keep it as simple as possible.

The payload data can be packed/encoded quite specific depending on the device that's sending it. How do you expect to handle that? Shouldn't that be configurable?

Yes. This is why CAN is so enigmatic. Despite of some standards. You can implement whatever you want. I'm just making the definition as I go . This is why I've put:

Task index - Value index - Var Type - 4 byte value - empty last byte.

Of course, this is just the first proposition.

- You named your controller SJA1000, does that mean it's specific for the SJA1000 CAN bus controller chip? If so, are there other controller-chips to be supported/expected?

The name is not good and should be changed. In fact, it uses the native TWAI driver. I will change the name.

- Multiple CAN buses (as mentioned in the link above): Do you expect to handle multiple CAN buses in a single ESPEasy unit?

No. Just looking for a EspEasy CAN standard . I know that soon enough people will try to plug EspEasy in their cars and expect to read a lot of stuffs. This is a lot more complicated. First goal should be: from EspEasy to EspEasy.

What other areas than automotive are currently using CAN bus, that triggered you to add this feature to ESPEasy? As I'm not sure how much/often ESPEasy is used in the automotive sector

Industry. Despite of different protocols and physical layers named Modbus RS485 or TCP, Profibus, Ethernet, EitherCAT, Ethernet/IP and so on. Industry also uses CAN and CANOpen.
It is open and wasn't created by some big company.

There are a lot of topics here questioning Modbus and Serial. The oldest industry protocol is still there, but I has huge latency. I have a lot of headache to work with multiple nodes.

From my perspective, CAN using CANOpen is the best one. I do use Ethernet and TCP a lot. However, there are some places that you just don't have room, you need to wire 20 sensors with one cable, you need reliability and there are a lot of noise. With CAN you can reach up to 100m and 500kbps. No need to check erros and priority are already defined at hardware level. I use two pairs of twisted cables. One with 24V and the other with communication. Something similar to PoE, but easier. I can use isolated transceivers and they are not expensive.

This is all based in my opinion. IoT is growing using wifi, but wifi is not reliable. I do have a lot of EspEasy in the field. Also some tuya, sonoff and others. Not talking about hardware, but I don't trust wireless Why not improve it as a second layer of communication? CAN is native!

[Ath]

OK, I like the approach you take here, start as simple as possible (but it's never simple ) and work up from there.
Let's see what you have so far in a PR. We'll have remarks and questions where needed that's how it goes.

What Controller and plugin ID's are you using? we have a 'register' for that, over here: https://github.com/letscontrolit/ESPEasy/issues/3839

[thalesmaoa]

Hi, returning to the topic. Commands work now, and I have to clean the code before a PR. I will perform it in the next couple weeks.

However, I want to hear from you about your perspective.
Do you see ESPEasy for indutrial application? Is this more focused on domotics and hobbits?

I should first clarify my question.
At some point, I've read in this forum someone requesting to bypass wifi.
Some want to use it without wifi connection. From the devs replies, I understood that is not the goal of ESPEasy and it's core is wifi. There were also suggestions of using AP mode or using another ESPEasy as AP.

I will try to put some ESPEasy in the field, but I'm also apprehensive about wifi. For some reason, if my network fails, my sensors will stop. CAN will not work due to the lack of wifi and so on. Despite of having two layers of communication, I'm hostage of the first wifi connection.

Does that make sense? Should I keep ESPEasy as target?

[TD-er]

Not sure where you've read about the goal being WiFi.
I have spent a lot (!!!) of time on various implementations of Ethernet modules. Now all ESP32-variants (all except ESP32-C2) support Ethernet.
I even had to make pull requests for Arduino and ESP-IDF to add support for some like the M5Stack lite W5500 PoE modules since those lack a few pins.
Also ESPEasy is the only one (next to Tasmota as we use the same patched library) to support those Dingtian DIN rail modules with JL1101 Ethernet.
Those are not supported in ESP-IDF/Arduino.

I don't see ESPEasy as "just for Domotica".
I think "the others" like ESPHome, Tasmota, Espruna, are more aimed at typical domotica usage.
I try to aim at a more scientific and/or industrial use, as can also be seen on the more elaborate analytics options in the UI and in the communications options.

ESPHome is more like "single task, preconfigured for Home Assistant"
Tasmota is mainly aimed at supporting (lots!) of of-the-shelve modules, however with their Berry language they can add quite complex stuff without a lot of hassle.

Espruna I honestly have never used myself, so can't say much about that.

N.B. I do have quite a lot of talks with the Tasmota people as Jason2866 and I almost daily talk about issues regarding ESP-IDF/Arduino and PlatformIO integration.
Right now, I think Tasmota and ESPEasy are the only two "larger" firmwares with support for ESP-IDF5.1, which is required for SPI Ethernet modules and IPv6. (simply because there is no official PlatformIO platform package for ESP-IDF5.x/Arduino3.x and Jason and I have worked on this for over a year now, with more to come in the near future)

Soon I will also add other networking protocols, which are now made possible since the huge network code refactor done by me-no-dev for Arduino 3.0.4 (3.0.5 will be released in a few days, I have already test builds ready also with ESP-IDF5.3)


TL;DR
I am more than interested in supporting more 'industrial' protocols like CAN.

[thalesmaoa]

I'm really happy to read this. Thank you very much.
I'm comfortable now of what I'm doing and to share it.

I'm sure I've made some mistake stating that wifi is priority. Let me clarify, maybe you can show me a possible solution.

I've made a thermocouple board using ESPEasy and CANBus. I dont use ethernet for this board. At total, there are 16 boards and 1 ethernet CANBus gateway. Since the board is inside a metal enclosure, I have weak wifi signal. After a power outage (reboot), I start read temperature only after the first wifi connection is established.

I need it to start reading temperature data and send to controller immediately after reboot, since my CAN gateway has a ethernet connection and has instant connection.

Is it possible to bypass the first wifi connection?

[TD-er]

If you can work with delayed sending the data, then you could also (as in duplicate) send it to the "Cache Controller" and use an extra task "Cache reader" to send the stored data when you have a connection to a MQTT controller for instance.

I am about to look into the whole network stack in ESPEasy as it has grown to be extremely complex over the years.
Now with the redesign of the network code in ESP-IDF, I can finally make a more clear and concise design.
I also plan on adding some preferred network order setting. (Arduino 3.0.4 finally has options to set preferred gateway/interface when multiple interfaces have a connection)

This will probably also include the ESPEasy-NOW (mesh) p2p layer I have been working on for about 5 years now.

A rough layout of the use case scenarios I have laid out for myself:

Code: Select all

WiFi connect scenarios ESPEasy
------------------------------

N.B. start of AP in below scenarios can be disabled in settings.

Scenario 1: Reconnect/warm boot
- Turn on WiFi
- Retry last known SSID/BSSID/channel (with "Use Last Connected AP from RTC" set)
Failed: -> Cold boot scenario


Scenario 2: Cold boot with known credentials
- Collect known credentials
- Perform scan
- Sort known credentials based on scan
- Loop:
  - Select credentials[N}
  - Try connect
  - Failed:
    - ++N
    - If no more credentials:
      - start AP
      - If no-one connected to AP:  restart scenario 2


Scenario 3: No known credentials
- Perform scan (to speed up setup page)
- start AP (when scan completes)

Scenario 4: LAN available
- If connected, try getting IP
- If not connected or IP timeout: Scenarios 1...3


Scenario 5: Mesh enabled
- Try scenario 1...4
- If still not connected: Mesh only mode


Scenario 6: Mesh only mode
- WiFi AP (mesh-only) mode
- Periodically scan for service AP
- If no other mesh nodes are found, perform a scan to try and locate other mesh nodes.


 

WiFi- and Mesh modes
--------------------

WiFi Modes:
- WiFi STA                -> Connect to other AP
- WiFi AP      (No mesh)  -> Allow user to connect for configuration
- WiFi AP+STA  (No mesh)  -> Allow user to connect for configuration + test connecting to (newly) configured AP
- WiFi AP      (mesh-only)-> No visible AP, use AP radio for mesh
- WiFi AP+STA  (mesh-GW)  -> No visible AP, use AP radio for mesh + STA mode to connect to AP (mesh gateway)



Mesh modes:
- Gateway        -> Is connected to IP network + access to MQTT broker
- Mesh-only node -> Only communicates via Mesh + occasionally tries to connect to service AP.

Mesh Gateway WiFi:
- has credentials
- Connected to AP & MQTT broker
- Use "WiFi AP+STA (mesh)" mode

Mesh Gateway Ethernet:
- Connected to network & MQTT broker
- Only use "WiFi AP (mesh)" WiFi mode


Mesh-only Node WiFi:
- NO credentials || NOT Connected to AP
- Has credentials:  AP+STA mode to scan for known network, but not (yet) connected
- NO credentials:  Only use "WiFi AP (mesh)" WiFi mode



Specifics per WiFi mode:
- STA mode:
  - Used for scanning
  - Connect to other AP
  - Is not 'listening' all the time => unsuitable for mesh
  - Capable of connecting to other AP using 802.11n
  - Cannot set channel, as this is dictated by AP.
- AP mode/STA+AP mode:
  - AP is used to let other STA devices connect to.
  - AP is always 'listening' -> perfect for mesh
  - AP-mode can set a channel, STA+AP mode should use a channel dictated by another AP the STA interface is connected to.
  - STA+AP mode cannot connect to other AP using 802.11n, only 802.11b or g
  

However keep in mind this will not be done in the next month or so as I've also other things which need time and there are only 24h in a day.