Rules running out of space

Moderators: grovkillen, Stuntteam, TD-er

Post Reply
Message
Author
Wiki
Normal user
Posts: 413
Joined: 23 Apr 2018, 17:55
Location: Germany

Rules running out of space

#1 Post by Wiki » 16 Mar 2020, 17:44

For some 1 1/2 years I have set up a "Central Information Device". Means, in the living room I have a wallmounted Nextion display which displays several informations about the status and values of my (roughly >30) several ESPEasy and Tasmota devices.

Coding the tiny Wemos D1 ESPEasy to display everything I need (and, of course, things that nobody needs, but which are pretty nice) works. But meanwhile I am running out of space with four rules settings limited each to 2048 bytes.

I already shortened every event name, every task name, I have moved as many logics as possible towards the sending devices and towards the Nextion display. In the codes of the rules, there are no indents anymore, no comments, only using capital letters gives a minimum optical structure. But nevertheless, without any indents and comments the code looks like an absolute unreadable bowl of spaghetti.

Is there any way to extend the nuber of rules or to enlarge the size of the rules?

Any help is appreciated

Code: Select all

pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $

User avatar
grovkillen
Core team member
Posts: 3621
Joined: 19 Jan 2017, 12:56
Location: Hudiksvall, Sweden
Contact:

Re: Rules running out of space

#2 Post by grovkillen » 16 Mar 2020, 17:57

You could upload bigger files but the parser might not be as fast as you want.
ESP Easy Flasher [flash tool and wifi setup at flash time]
ESP Easy Webdumper [easy screendumping of your units]
ESP Easy Netscan [find units]
Official shop: https://firstbyte.shop/
Sponsor ESP Easy, we need you :idea: :idea: :idea:

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Rules running out of space

#3 Post by TD-er » 16 Mar 2020, 19:49

Jimmy made a fix for the posting of rules when editing.
The bug is that sometimes the rules may get corrupted when saving.
This bug does occur sometimes when saving rules as "small" as 1400 bytes. This is part of the reason we still stuck with 2048 bytes as that's a rule size that will be able to save almost always.

The Fix Jimmy made (not yet merged) does also allow for much larger rules and I have tried them.
However it does make rules processing very slow.

There is a work-around for this, called the "new rules engine" which can be enabled in the Advanced settings page.
This does split the rules into lots of files, a single on... do block per file.
It allows for faster rules processing and also removes the limit of the rules as we now have it, but editing is a bit tedious, which is the reason we have not yet made it the default.

But you can try it yourself.
Just make sure to keep a copy of the existing rules before switching the rules engine as it does not allow to edit the old rule files in this mode.

Wiki
Normal user
Posts: 413
Joined: 23 Apr 2018, 17:55
Location: Germany

Re: Rules running out of space

#4 Post by Wiki » 16 Mar 2020, 21:55

Ok, I was not aware that it is possible to upload rules > 2048 byte.

Due to the fact, that it is currently not possible - in my experience never ever - to save rules > 1300bytes I am editing my rules as txt file and uploading them to bypass the online editor since almost a year, meanwhile this is routine for me. I will test it with bigger files, thanks for the hint.

Btw.: I already tried the new rules engine: not useable, sorry.

Code: Select all

pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Rules running out of space

#5 Post by TD-er » 16 Mar 2020, 22:29

Wiki wrote: 16 Mar 2020, 21:55 [...]
Btw.: I already tried the new rules engine: not useable, sorry.
I know. That's why it isn't the default right now.

Wiki
Normal user
Posts: 413
Joined: 23 Apr 2018, 17:55
Location: Germany

Re: Rules running out of space

#6 Post by Wiki » 16 Mar 2020, 23:02

I'm sorry for grovkillen, it seems he is in progress with the new rules engine. But the way they are oriented right now may be useable for a set of 3-5 events. In my special case to feed a 7.0" Nextion with the desired huge mass of information - no way, never ever.

I have put all of my existing rules for the special device mentioned above togehter into one rule set and uploaded it to a playground device (8073 characters). The only hint I can recognize up to now is the the remark in the web editor:

Code: Select all

Filesize exceeds web editor limit!
What else do I have to expect beside slower performance - any experiences?

What does
it does make rules processing very slow
mean? Processing will take minutes?

Code: Select all

pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $

Wiki
Normal user
Posts: 413
Joined: 23 Apr 2018, 17:55
Location: Germany

Re: Rules running out of space

#7 Post by Wiki » 16 Mar 2020, 23:11

Additional: If you need a tester for further enhancements of rules processing I would like to offer me.

I would need an image with plugins: Nextion, DS18B20, Thermocouple MAX6675, Dummy, MQTT Import, Controller MQTT Domoticz & Generic HTTP.

The only risc I have testing whatever you need would be a decreasing WAF (I'm talking about our living room functionality, i.e. switching the lights, surround sound booster etc) - and I will be able to deal with it.

Code: Select all

pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Rules running out of space

#8 Post by TD-er » 17 Mar 2020, 00:22

No, Jimmy wasn't working on the rules stuff.
I made him do it, as this can only be fixed in JavaScript and I don't speak JS.
Not sure what it is with JS, but that language freaks me out. Maybe it is the operators with more than 2 =='s, or maybe it is just me.
The very few times I had to do something in JavaScript I felt like I was going mad.
So that's why Jimmy had to offer some time to make a script to generate a file to upload from within the browser.


About the rules parsing with large files...
Problem is the way how the SPIFFS filesystem does handle file access and also how we parse the files.
I think the only way to make it truly fast in handling large files is to index the files and generate some index of where all the blocks start, or simply split the files into smaller chunks when they are saved.
Also the string-compares are quite expensive, so that could be made more efficient by "pre-compiling" the commands into their enum equivalent values.

Well that does take quite some time to do and it is not my highest priority right now.
I guess the simplest way could be to simply add more files (e.g. 8 instead of 4) and it will also be faster if you have the ones occurring most often at the start of a file, preferably the 1st file.

Wiki
Normal user
Posts: 413
Joined: 23 Apr 2018, 17:55
Location: Germany

Re: Rules running out of space

#9 Post by Wiki » 17 Mar 2020, 00:50

Now I have put my merged four rules into rules1.txt, works like a charme, no recognizable hesitation in reaction hitting buttons or displaying received data. I am happy, merging the rules gave me instantly the possibility to put in a visible structure and eliminate some doubled functions and inconsistencies - what a shame. The rules file shrinked (including added indents and some lean comments) from 8k to about 6.5k.

Maybe a recommendation: Don't waste your time in an online editor. There are so many editors on the market able to display yout coding structure in multicolor. Just give the opportunity to have a look at the rules coding and let the coding be done outside of the firmware. Just give an advice, that huge rule files will reduce performance. Could reduce image size, too.

At least: you are still my heroes.

But: Why nowhere is documented, that the file size is limited just by the behaviour of the online editor and some misbehaviour in performance? Could have saved me a lot of time. Or did I miss something?

Code: Select all

pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Rules running out of space

#10 Post by TD-er » 17 Mar 2020, 01:09

Well the limit was not intentionally set, but more like it used to be highly impractical to use it for larger files.
The rules parsing has been improved a lot in the last 2 years.
First Martinus made quite a huge improvement in rules parsing by eliminating some very silly extra work done in the code.
And I also moved in some tweaks in the code to make it faster.
So the limit has not been very strict but it is now doable to use longer rules.

Just try to have the most frequently used events at the top of the file, so the parsing can stop as soon as possible.

Wiki
Normal user
Posts: 413
Joined: 23 Apr 2018, 17:55
Location: Germany

Re: Rules running out of space

#11 Post by Wiki » 17 Mar 2020, 01:34

Thanks for the hints, I will follow them and up to now I am really happy.

YOU MADE MY DAY

Code: Select all

pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Rules running out of space

#12 Post by TD-er » 17 Mar 2020, 13:10

AnnekeH wrote: 17 Mar 2020, 12:03 You can try to use a txt-file for large objects or upload smaller files
Can you explain a bit more what you mean? (as uploading files was already discussed)

Wiki
Normal user
Posts: 413
Joined: 23 Apr 2018, 17:55
Location: Germany

Re: Rules running out of space

#13 Post by Wiki » 18 Mar 2020, 01:38

Putting the whole rules code into rules1.txt works fine with a rules file of roughly 6.5kbyte. Having a visible structure enabled me to reduce the overall size by aroound 2k. Up to now I did not notice any loss of performance.
Interesting: Trying to open the page for rules2 using the web editor results in a broken connection, but the device is still up and running, still accessible on the other pages.

Uploading the rules2.txt works fine and the rules are parsed correctly. Works for me this way, I am used to edit the rules offline anyway. The experience forcing me to use this way was, that I do have correct and actual versions of the rules on my server - including useful things like backups with follow up of versions.

Code: Select all

pi@raspberrypi:~ $ man woman
No manual entry for woman
pi@raspberrypi:~ $

Umendi
New user
Posts: 8
Joined: 16 Jan 2021, 08:56

Re: Rules running out of space

#14 Post by Umendi » 10 Apr 2021, 23:58

Hi there... a newbie here :)

I thought that the limitation was due to the max lengh of the URL (2,048 characters), as everytime I use the online editor the URL changes to include the full code included on the rule I am editing.

Wiki, I understand that by merging the rules into a single file you mean basically to copy/paste all code into 1 file. Right?
How did you upload the rules1.txt to the ESPEASY?

The only way I found to do it was when flashing the device as there is an option to upload the 4 files for the rules (but I do not want to reflash it and lose all my current configuration).

PS: a huge THANKS to all of you that create/participate in the ESPEasy project. In two months I have created a portable CO2 sensor box, a door/window/drawer alarm and a postal mail detector that emails me when I get physical mail. Currently developing an tiny Escape Box (hence the need of lots of rules...)

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Rules running out of space

#15 Post by TD-er » 11 Apr 2021, 00:08

With the latest builds you can save much larger rules from the web interface.
The JavaScript running in the browser will actually call the upload URL.

Umendi
New user
Posts: 8
Joined: 16 Jan 2021, 08:56

Re: Rules running out of space

#16 Post by Umendi » 11 Apr 2021, 09:36

I am using this release ESP_Easy_mega_20210223_test_beta_ESP8266_4M1M (and Chrome's last built)
I know being a test_beta it is not stable but I will needed the Gyro MPU6050 device, which AFAIK is only available in test releases

Both old rules engine and the new rules engine show the same max: "Current size: xxxx characters (Max 2048)"
If I surpass this limit, the alert message "Data was not saved, exceeds web editor limit! rules/test/test" is displayed.

I am doing something incorrectly?
How can I upload the txt file with the rules directly?

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Rules running out of space

#17 Post by TD-er » 11 Apr 2021, 20:40

Hmm the file should be saved anyway. I have tested it before and rules upto 6 - 7 k were working fine (did not test larger ones)
The mentioned file limit is a left-over from before we saved the rules using JavaScript via the upload URL.

Umendi
New user
Posts: 8
Joined: 16 Jan 2021, 08:56

Re: Rules running out of space

#18 Post by Umendi » 13 Apr 2021, 00:18

I did further testing:

old rules engine
ESP_Easy_mega_20201130_normal_ESP8266_4M1M
Current size: 2373 characters (Max 2048)Filesize exceeds web editor limit!
After surpassing the 2048 limit, the character count does not reflect the actual amount of characters but all data is still saved as you commented.
I managed to add up to 16000 characters before crashing the ESP8266 (sudden reboot)

new rules engine:
ESP_Easy_mega_20210223_test_beta_ESP8266_4M1M
Current size: 2048 characters (Max 2048)
one more character and it shows the message Data was not saved, exceeds web editor limit! rules/test/test
(and it is indeed not saved)
However I can add lots of events, so no problem.

Thanks for all your help

TD-er
Core team member
Posts: 8643
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Rules running out of space

#19 Post by TD-er » 13 Apr 2021, 08:31

Just to be clear, what do you mean by "new rules engine" ?
There is also a flag in the Tools->Advanced settings which has this name (and should soon be replaced by "experimental rules, do not use")

Post Reply

Who is online

Users browsing this forum: Google [Bot] and 34 guests