Collect data from ESP on SD card in router

[mackowiakp]

I would like to mount 6 ESPs with different sensors in a certain location. Measurements should be stored on the router's SD card as flat files (CSV for example). The router (all ESP`s are connected to that router over WiFi) would be equipped with OpenWRT (Asus) and would transmit data via LTE to a central server using RSYNC over SSH. The router would have a Huawei LTE modem connected, with the option of installing a 32 GB SD card, for collecting measurements.
Of course, RPi can be used instead (and this is the case at the moment), but RPI requires proper shutdown, soldering, casing construction, etc.
However, I wonder how to transfer data to the router so that they save to the SD card. Generic HTTP would probably have to be used. Well, what about the router side? Inserting the entire WEB server will take up most of the router's resources and is not needed for anything. I uploaded "clean" OpenWRT on it, without even a GUI (LuCI or other).
Hence my question. How to solve the problem of writing data to the SD card in the router (mounted as a filesystem in some directory) and the data would be transferred from ESP devices. Any idea?

[dynamicdave]

What's wrong with sending the sensor data from the ESPs, via MQTT with a rule-set, to a Raspberry Pi running Node-RED ??
You can then do all sorts of things with the data... store it on the Pi's SD card (local file), write it to a database (local or remote) or send it to other places.

[mackowiakp]

Nothing wrong. Its very good solution. But I want to reduce amount of equipment. I have to use router because necessity to connect external WiFi and GSM antennas because of coverage and necessity of connection three other units over wired Ethernet. If OpenWRT router will be able to collect data from ESP`s there is one unit less. And no necessity to print housing, UPS, maintenance, external hardware and so, etc
I have to add that all this equipment is installed in distant location so I want to keep it ASAP (As Simple As Possible). Mainly because of maintenance.

[Friedrich]

I ran into a similar situation when testing a remote site for temperature over 2-3 days. I wanted a quick and easy setup to log data collected from three ESP modules onto a SD card, avoiding the use of a Pi exactly because of the problems resulting from improper shutdown. As quick solution I found a working MQTT broker running out of the box on a ESP 12e board : https://github.com/martin-ger/esp_mqtt. To actually log the data I used another ESP module with SD card running a MQTT client in micropython.
Surely these are two more devices, but it was setup up and running in less than half an hour, at less power costs than a Pi and worked like a charm. I had only about a dozen values per minute, but I guess for higher loads micropython might quickly prove itself as bottleneck.

[TD-er]

Not sure how the data is being written to the SD card.
Not all SD card solutions have some kind of wear-leveling active and you are probably writing to a card formatted with a FAT32 filesystem.
Both are a perfect receipt for writing often to just a few sectors on the SD card and wearing down the card in weeks or maybe months.

In theory you could write new bytes to the same sector and so appending to the same binary data and just counting it as a single write.
But I don't know what filesystem supports that and if there is any it is probably not implemented in ESP-like libraries.

A Raspberry Pi does use proper caching to reduce the number of writes to the flash, but this puts an extra restriction on it for proper shutdown or else you will corrupt data.

Some routers support http access to a filesystem mounted from an USB device.
So you could then use some SSD with USB connector as proper filesystem (not using an USB stick, which is often nothing other than a micro-SD USB reader with a micro-SD card in it) with proper wear leveling.

[mackowiakp]

Not all SD card solutions have some kind of wear-leveling active and you are probably writing to a card formatted with a FAT32 filesystem.
Both are a perfect receipt for writing often to just a few sectors on the SD card and wearing down the card in weeks or maybe months.

Yeah. As Freddie Mercury sang "Too much writes will kill You..." (or something like this). Knowing that, I use only "high endurance" SD cards and USB mem stick. Additionally, I worked at least half a year to reduce amount of writes to system SD card. All temporary files are moved to RAM (/dev/shm), even including NTP drift file and other similar. Logs are enabled only if needed. All disk operations are provided in RAM too. So total amount of data written to SD is less than 50Kb per day. FAT16 boot partition is mounted as read-only (by default is RW). I use nano-pi , without GPU so I removed all X related files from installation.
All data are stored on 16 or 32 GB "high endurance" USB mem. Total amount of data/measurements on such stick is less than 250 MB (max 7 days old).
All partitions (except boot) are EXT4 type.
Moreover, my agreements with customers stands that such sticks should be change each year. Because it is a consumable item and it is worn like tires in a car.
So I never had problems with any SD or USB mem during past 5 years.You can believe me because at this moment it is over 200 pcs of such units.
But maintenance is a horror, so I am looking for simpler solution (ESP + OpenWRT router).

Some routers support http access to a filesystem mounted from an USB device.

Nooo. You are genius!!! After all, the solution is very easy. OpenWRT (as a kind of Linux) has the ability to implement WEB-Dav and access to the SD card/USB mem via HTTP using Generic HTTP from ESP. "Simple as a flail service" - as we say. That I didn't think of it myself... Shame...

Here is a small advice for everyone who has Domoticz (or similar) installed on RPi. The most dangerous "killer files" for SD are temporary database files. They need to be moved to RAM (symlink in Domoticz startup script). Loss of power is not a problem. These files are always created as new when the database is started, even if old ones exists.

[Drum]

I have used the Raspberry pi - Mosquito - NodeRed - Sqlite (and Pihole) setup for a good 3 years, never had any problems with improper shutdown due to power outage (We do have frequent power outages).
I only have sandisk or samsung SD cards, so that is what is in my Pis. I did replace the card once, out of an abundance of caution, never had any issues, and recently upgraded to a Pi 3+ and SSD, works great, and no problems so far from power outages. I have almost 9 million entries in the sqlite database(not recommended) on that Raspberry Pi, most were before the SSD upgrade so the SD cards I have used are pretty reliable I think. I am not worried too much about the power usage, it is certainly a lot less than the other computers and servers I have running. It might even work on a Pi Zero w. With a minute or 2 of battery power so the wifi goes down before the power fails to the Pi....

From what I understand of writing to a SD Card in the router, and I could be wrong, it sounds like you will have a lot of small files. If this is the case check if there is a limit on the number of files the card can hold, I believe there is a limit based the number and length of the file name. Also I think you could have the same problem with the SD card in the router if the power fails while it is being written to.

[TD-er]

Most primitive file systems had at least a limit of the number of file entries in the root directory.
Also having lots of small files does hurt performance quite a bit.