How should bugs be reported?

Moderators: grovkillen, Stuntteam, TD-er

Post Reply
Message
Author
namirda
Normal user
Posts: 53
Joined: 22 Jan 2016, 17:09

How should bugs be reported?

#1 Post by namirda » 27 Feb 2016, 04:45

Hi,

I've been playing around trying to add some of my own modules to ESP Easy and I have maybe found a bug.

I believe there is a small but devious bug in LoadTaskSettings in module misc.ino. It was not updating ExtraTaskSettings.TaskIndex for the non SPIFFS case. My proposed update is below. How should bugs be reported?

Code: Select all

void LoadTaskSettings(byte TaskIndex)  
{
  if (ExtraTaskSettings.TaskIndex == TaskIndex)
    return;
    
#if FEATURE_SPIFFS
  LoadFromFile((char*)"config.txt", 4096 + (TaskIndex * 1024), (byte*)&ExtraTaskSettings, sizeof(struct ExtraTaskSettingsStruct));
#else
  LoadFromFlash(4096 + (TaskIndex * 1024), (byte*)&ExtraTaskSettings, sizeof(struct ExtraTaskSettingsStruct));
#endif

ExtraTaskSettings.TaskIndex = TaskIndex; // store active index

}

tozett
Normal user
Posts: 734
Joined: 22 Dec 2015, 15:46
Location: Germany

Re: How should bugs be reported?

#2 Post by tozett » 27 Feb 2016, 09:03

good question. i think. one could get lost in the forum if it grows, it could get a jungle :P
maybe it can be of some help to sort things out and mark solved problems, some kind as help for others..
or use some sort of system like this one, i mentioned earlier :P
http://www.esp8266.nu/forum/viewtopic.p ... =240#p4347
which looks some kind like this:
Image

Martinus

Re: How should bugs be reported?

#3 Post by Martinus » 01 Mar 2016, 08:17

namirda wrote:Hi,

I've been playing around trying to add some of my own modules to ESP Easy and I have maybe found a bug.

I believe there is a small but devious bug in LoadTaskSettings in module misc.ino. It was not updating ExtraTaskSettings.TaskIndex for the non SPIFFS case. My proposed update is below. How should bugs be reported?

Code: Select all

void LoadTaskSettings(byte TaskIndex)  
{
  if (ExtraTaskSettings.TaskIndex == TaskIndex)
    return;
    
#if FEATURE_SPIFFS
  LoadFromFile((char*)"config.txt", 4096 + (TaskIndex * 1024), (byte*)&ExtraTaskSettings, sizeof(struct ExtraTaskSettingsStruct));
#else
  LoadFromFlash(4096 + (TaskIndex * 1024), (byte*)&ExtraTaskSettings, sizeof(struct ExtraTaskSettingsStruct));
#endif

ExtraTaskSettings.TaskIndex = TaskIndex; // store active index

}
You are right that there's a bug with this function call. But the value is not stored properly in the first place.
I you would read the real flash contents it does not show the right value. So the bug starts with saving.
Loading should then automatically be fixed as the whole struct is copied back with memcpy anyway.

Thanks for pointing out the issue!

But what bad behavior comes from this bug? Everything stills seems to work?
I guess only the 'load caching' fails because of this, causing some extra delay.

I will address this in R83 because it does needs a fix.

namirda
Normal user
Posts: 53
Joined: 22 Jan 2016, 17:09

Re: How should bugs be reported?

#4 Post by namirda » 06 Mar 2016, 06:18

Hi Martinus,

Thanks for the reply. I came across the bug while trying to develop a new plugin. I was trying to give appropriate error messages using constructions similar to some of yours such as :

Code: Select all

      
      LoadTaskSettings(event->TaskIndex);
      String log = F("200 Device ");
       log+=ExtraTaskSettings.TaskDeviceName;
       log+=" Value ";
       log += ExtraTaskSettings.TaskDeviceValueNames[0];
       log += " - Initially set to 0 [OFF]";
       addLog(LOG_LEVEL_INFO, log);
I was getting surprising results and discovered the error I mentioned previously in LoadTaskSettings. As you mentioned, there are other problems as well and I think some of these may be related to frequent use of dynamic memory allocation.

I have been looking a bit further into the code and notice that String variables are used heavily throughout ESPEasy - and not always in the safest way. A particularly bad example can be found in the module WebServer.ino where you will find the following call to PluginCall :

Code: Select all

PluginCall(PLUGIN_WEBFORM_LOAD, &TempEvent, reply);
'reply' is a String variable to which successive strings have been dynamically added until it is about 4700 characters long by the time PluginCall is invoked!! Each of these appends copies the original string to a temporary variable, appends the new string and then copies back which results in a very fragmented stack. Some plugins ( for example P012 ) make several more appends to 'reply' which add an additional 1500 characters to this already enormous string!! As new functionality has been added to ESPEasy in recent releases, 'reply' has grown until, in my case at least, it results in a stack crash every time the P012 webform is loaded.

I can easily workaround this problem for now as follows :

Code: Select all

String Treply;
PluginCall(PLUGIN_WEBFORM_LOAD, &TempEvent, Treply);
reply+=Treply;
This shortens the string which is passed to PluginCall thereby reducing the fragmentation and avoids the stack crash. However, it is not a final solution.

You probably know better than I do that String operations allocate memory dynamically and in ways that are hard to predict. Dynamic memory allocation almost always causes memory fragmentation which will eventually lead to a stack crash. It's just a matter of time! For embedded software, it is best to avoid String variables altogether - and certainly huge ones like 'reply'. Better to allocate appropriate char arrays wherever possible.

The problem for ESPEasy is that removing String variables would also mean changing the calling signature of all plugins and I can see that you probably don't want to do that. As a stopgap measure, I think that eliminating the enormous strings used in WebServer (or at least breaking them up into smaller strings) would help a lot.

I hope you don't take the above as criticism - it is meant to be helpful. In general I really like the software and have found it to be really flexible and easily expandable - I wish I had thought of it!

Thanks for your help

N

Martinus

Re: How should bugs be reported?

#5 Post by Martinus » 06 Mar 2016, 11:12

When I started on ESP Easy, it was 95% focus on converting sensor plugins and the plugin framework from the Nodo project and 5% on having a web gui to configure the stuff. So one can expect some (or more) design flaws in the web parts of the code.

The plugin call originally had a char array, but I deliberately changed it to a string object to be more flexible. Besides that, the used Arduino SDK is also full of string objects in most of the web handling. Probably have to live with that for now. I was under the impression that the string object is passed by reference and it should not be an issue when passed to a function as it would only take over the object address. But that may be a wrong interpretation on how the string class works. Or maybe I'm passing it the wrong way.

The 'reply' string holds the entire webpage to be returned, based on the very first SDK way to handle simple web requests. I guess this can be done more efficient by writing in chunks to the web client somehow and avoid to hold an entire copy of the dynamically created web page in memory. But never had the time to look into this. And as my ESP Easy units are running fine without crashes, it never got priority so far.

namirda
Normal user
Posts: 53
Joined: 22 Jan 2016, 17:09

Re: How should bugs be reported?

#6 Post by namirda » 07 Mar 2016, 00:24

Hi Martinus,

Fair enough - I would certainly have done the same.

I have played around with various options today and have found that the fragmentation problem goes away completely if you preallocate the maximum storage requirements for the variable 'reply' in function 'handle_devices' in module WebServer. All you need to do is add one line after the declaration of the variable 'reply'

Code: Select all

String reply="";
reply.reserve(10000);
This preallocates 10000 bytes for 'reply' and thus removes all fragmentation problems - the 10000 bytes are released on exit from function 'handle_devices' . I wish all problems were as easily solved!

10000 is probably more than is required but you would know that much better - maybe 8000 would be sufficient?

N

Martinus

Re: How should bugs be reported?

#7 Post by Martinus » 09 Mar 2016, 09:32

Although I understand the possible heap fragmentation in using large string objects, I'm unable to reproduce any effects from this. How did you detect/investigate the memory fragmentation? Some of my units are running for a month now and still display the same amount of free RAM. And if I display free mem at the beginning and end of handle_devices, it always starts with the same amount of free mem.

Any tips on how to consistently reproduce the memory fragmentation, so fixes can be verified to work?

namirda
Normal user
Posts: 53
Joined: 22 Jan 2016, 17:09

Re: How should bugs be reported?

#8 Post by namirda » 09 Mar 2016, 22:14

Hi Martinus,

I believe that the WebServer is the main offender when it comes to dynamic memory allocation - particularly modules P012 (Display - LCD2004) and P023 (Display - OLED SSD1306) which have the most extensive setup options due to the custom task settings for those tasks.

From the main menu in ESPEasy Webserver, go to option 'Devices' and add the two devices mentioned above. Add a few other devices as well just for good measure! Now attempt to edit/submit the settings of the LCD2004 and the OLED alternately and after a couple of iterations I think you will find that you suffer a heap crash. If you monitor Freemem as you do this you can see what is happening. On my setup with 8 devices including 1 LCD it happens immediately on the first edit.

After adding the preallocation statement which I mentioned in my previous post there are no more crashes and freemem stays at similar levels after each edit.

Please let me know what you find.

DonJuanito
New user
Posts: 7
Joined: 16 Mar 2016, 20:01

Re: How should bugs be reported?

#9 Post by DonJuanito » 18 Mar 2016, 18:58

Hi. Sorry to hijack this thread, but the title was exactly what I was asking myself, and I hate forums where the same question is asked over and over in different threads...

I found what I think is a small bug int the _P005_DHT.ino sketch. As some other users, I was unable to maintain a constant flow of successful readings from a DHT11/22 sensor. With some ones, almost all the readings failed ("nan" value is returned).
After reading through the code and comparing it with the DHT11 datasheet I have, it seems that pulling high the communication IO for more than 40us at the communication init sequence is not the right thing to do. In fact, 40us is the MAXIMUM time the IO line should be pulled up by the controller, with 20us being the minimum.

So I changed the "delayMicroseconds(40);" line to "delayMicroseconds(20);" and ALL my sensors are successfully read each and every time.

This code line is around line number 98 in the R83 code on Github:

Code: Select all

        // DHT start condition, pull-down i/o pin for 18ms
        digitalWrite(Plugin_005_DHT_Pin, LOW);              // Pull low
        delay(18);
        digitalWrite(Plugin_005_DHT_Pin, HIGH);             // Pull high
        delayMicroseconds(20); // was 40. With 40, sensor returns NaN most of the time...
        pinMode(Plugin_005_DHT_Pin, INPUT);                 // change pin to input
I tested this modification successfully on 2 ESP12 based boards and 1 ESP01 board, with 4 DHT11 and 1 DHT22...

Can perhaps someone with more knowledge than me check this and modify the Github trunk if it seems OK?

Thanks for the Easy ESP development, it's fun to play with these boards ;)

Post Reply

Who is online

Users browsing this forum: Google [Bot] and 1 guest