Rules running out of space

Moderators: grovkillen, TD-er, Stuntteam

Post Reply
Message
Author
Wiki
Normal user
Posts: 223
Joined: 23 Apr 2018, 17:55

Rules running out of space

#1 Post by Wiki » 16 Mar 2020, 17:44

For some 1 1/2 years I have set up a "Central Information Device". Means, in the living room I have a wallmounted Nextion display which displays several informations about the status and values of my (roughly >30) several ESPEasy and Tasmota devices.

Coding the tiny Wemos D1 ESPEasy to display everything I need (and, of course, things that nobody needs, but which are pretty nice) works. But meanwhile I am running out of space with four rules settings limited each to 2048 bytes.

I already shortened every event name, every task name, I have moved as many logics as possible towards the sending devices and towards the Nextion display. In the codes of the rules, there are no indents anymore, no comments, only using capital letters gives a minimum optical structure. But nevertheless, without any indents and comments the code looks like an absolute unreadable bowl of spaghetti.

Is there any way to extend the nuber of rules or to enlarge the size of the rules?

Any help is appreciated

User avatar
grovkillen
Core team member
Posts: 3582
Joined: 19 Jan 2017, 12:56
Location: Hudiksvall, Sweden
Contact:

Re: Rules running out of space

#2 Post by grovkillen » 16 Mar 2020, 17:57

You could upload bigger files but the parser might not be as fast as you want.
ESP Easy Flasher [flash tool and wifi setup at flash time]
ESP Easy Webdumper [easy screendumping of your units]
ESP Easy Netscan [find units]
Official shop: https://firstbyte.shop/
Sponsor ESP Easy, we need you :idea: :idea: :idea:

TD-er
Core team member
Posts: 2620
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Rules running out of space

#3 Post by TD-er » 16 Mar 2020, 19:49

Jimmy made a fix for the posting of rules when editing.
The bug is that sometimes the rules may get corrupted when saving.
This bug does occur sometimes when saving rules as "small" as 1400 bytes. This is part of the reason we still stuck with 2048 bytes as that's a rule size that will be able to save almost always.

The Fix Jimmy made (not yet merged) does also allow for much larger rules and I have tried them.
However it does make rules processing very slow.

There is a work-around for this, called the "new rules engine" which can be enabled in the Advanced settings page.
This does split the rules into lots of files, a single on... do block per file.
It allows for faster rules processing and also removes the limit of the rules as we now have it, but editing is a bit tedious, which is the reason we have not yet made it the default.

But you can try it yourself.
Just make sure to keep a copy of the existing rules before switching the rules engine as it does not allow to edit the old rule files in this mode.

Wiki
Normal user
Posts: 223
Joined: 23 Apr 2018, 17:55

Re: Rules running out of space

#4 Post by Wiki » 16 Mar 2020, 21:55

Ok, I was not aware that it is possible to upload rules > 2048 byte.

Due to the fact, that it is currently not possible - in my experience never ever - to save rules > 1300bytes I am editing my rules as txt file and uploading them to bypass the online editor since almost a year, meanwhile this is routine for me. I will test it with bigger files, thanks for the hint.

Btw.: I already tried the new rules engine: not useable, sorry.

TD-er
Core team member
Posts: 2620
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Rules running out of space

#5 Post by TD-er » 16 Mar 2020, 22:29

Wiki wrote:
16 Mar 2020, 21:55
[...]
Btw.: I already tried the new rules engine: not useable, sorry.
I know. That's why it isn't the default right now.

Wiki
Normal user
Posts: 223
Joined: 23 Apr 2018, 17:55

Re: Rules running out of space

#6 Post by Wiki » 16 Mar 2020, 23:02

I'm sorry for grovkillen, it seems he is in progress with the new rules engine. But the way they are oriented right now may be useable for a set of 3-5 events. In my special case to feed a 7.0" Nextion with the desired huge mass of information - no way, never ever.

I have put all of my existing rules for the special device mentioned above togehter into one rule set and uploaded it to a playground device (8073 characters). The only hint I can recognize up to now is the the remark in the web editor:

Code: Select all

Filesize exceeds web editor limit!
What else do I have to expect beside slower performance - any experiences?

What does
it does make rules processing very slow
mean? Processing will take minutes?

Wiki
Normal user
Posts: 223
Joined: 23 Apr 2018, 17:55

Re: Rules running out of space

#7 Post by Wiki » 16 Mar 2020, 23:11

Additional: If you need a tester for further enhancements of rules processing I would like to offer me.

I would need an image with plugins: Nextion, DS18B20, Thermocouple MAX6675, Dummy, MQTT Import, Controller MQTT Domoticz & Generic HTTP.

The only risc I have testing whatever you need would be a decreasing WAF (I'm talking about our living room functionality, i.e. switching the lights, surround sound booster etc) - and I will be able to deal with it.

TD-er
Core team member
Posts: 2620
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Rules running out of space

#8 Post by TD-er » 17 Mar 2020, 00:22

No, Jimmy wasn't working on the rules stuff.
I made him do it, as this can only be fixed in JavaScript and I don't speak JS.
Not sure what it is with JS, but that language freaks me out. Maybe it is the operators with more than 2 =='s, or maybe it is just me.
The very few times I had to do something in JavaScript I felt like I was going mad.
So that's why Jimmy had to offer some time to make a script to generate a file to upload from within the browser.


About the rules parsing with large files...
Problem is the way how the SPIFFS filesystem does handle file access and also how we parse the files.
I think the only way to make it truly fast in handling large files is to index the files and generate some index of where all the blocks start, or simply split the files into smaller chunks when they are saved.
Also the string-compares are quite expensive, so that could be made more efficient by "pre-compiling" the commands into their enum equivalent values.

Well that does take quite some time to do and it is not my highest priority right now.
I guess the simplest way could be to simply add more files (e.g. 8 instead of 4) and it will also be faster if you have the ones occurring most often at the start of a file, preferably the 1st file.

Wiki
Normal user
Posts: 223
Joined: 23 Apr 2018, 17:55

Re: Rules running out of space

#9 Post by Wiki » 17 Mar 2020, 00:50

Now I have put my merged four rules into rules1.txt, works like a charme, no recognizable hesitation in reaction hitting buttons or displaying received data. I am happy, merging the rules gave me instantly the possibility to put in a visible structure and eliminate some doubled functions and inconsistencies - what a shame. The rules file shrinked (including added indents and some lean comments) from 8k to about 6.5k.

Maybe a recommendation: Don't waste your time in an online editor. There are so many editors on the market able to display yout coding structure in multicolor. Just give the opportunity to have a look at the rules coding and let the coding be done outside of the firmware. Just give an advice, that huge rule files will reduce performance. Could reduce image size, too.

At least: you are still my heroes.

But: Why nowhere is documented, that the file size is limited just by the behaviour of the online editor and some misbehaviour in performance? Could have saved me a lot of time. Or did I miss something?

TD-er
Core team member
Posts: 2620
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Rules running out of space

#10 Post by TD-er » 17 Mar 2020, 01:09

Well the limit was not intentionally set, but more like it used to be highly impractical to use it for larger files.
The rules parsing has been improved a lot in the last 2 years.
First Martinus made quite a huge improvement in rules parsing by eliminating some very silly extra work done in the code.
And I also moved in some tweaks in the code to make it faster.
So the limit has not been very strict but it is now doable to use longer rules.

Just try to have the most frequently used events at the top of the file, so the parsing can stop as soon as possible.

Wiki
Normal user
Posts: 223
Joined: 23 Apr 2018, 17:55

Re: Rules running out of space

#11 Post by Wiki » 17 Mar 2020, 01:34

Thanks for the hints, I will follow them and up to now I am really happy.

YOU MADE MY DAY

TD-er
Core team member
Posts: 2620
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Rules running out of space

#12 Post by TD-er » 17 Mar 2020, 13:10

AnnekeH wrote:
17 Mar 2020, 12:03
You can try to use a txt-file for large objects or upload smaller files
Can you explain a bit more what you mean? (as uploading files was already discussed)

Wiki
Normal user
Posts: 223
Joined: 23 Apr 2018, 17:55

Re: Rules running out of space

#13 Post by Wiki » 18 Mar 2020, 01:38

Putting the whole rules code into rules1.txt works fine with a rules file of roughly 6.5kbyte. Having a visible structure enabled me to reduce the overall size by aroound 2k. Up to now I did not notice any loss of performance.
Interesting: Trying to open the page for rules2 using the web editor results in a broken connection, but the device is still up and running, still accessible on the other pages.

Uploading the rules2.txt works fine and the rules are parsed correctly. Works for me this way, I am used to edit the rules offline anyway. The experience forcing me to use this way was, that I do have correct and actual versions of the rules on my server - including useful things like backups with follow up of versions.

Post Reply

Who is online

Users browsing this forum: No registered users and 6 guests