uPyEasy v0.3.0 released

Moderators: Voyager, BertB, grovkillen, Stuntteam, LisaM

Post Reply
Message
Author
LisaM
Normal user
Posts: 513
Joined: 11 Apr 2017, 18:29

Re: uPyEasy v0.3.0 released

#21 Post by LisaM » 01 Jun 2018, 18:31

AndrewJ wrote: 28 May 2018, 21:58 Hi Lisa,
I see on Github that you've been hard at work on a new update (9ebffbd) of uPyEasy!! Thank you for all your effort on this project!

I'm still keen to get uPyEasy working on my STM32F405 AliExpress clones, or at least the one which still seems to work reliably.
So I updated my local repo and set about building the firmware.

Initially I had errors from a couple of "missing" files in the /unix directory (copy.py and types.py) which I was able to fix, also I needed to create new links for these into my micropython-stm32 file system before it would work.

uPyEasy then started up ok and ran for a short time. Some screens are working (MaIn, Config, Hardware - Hardware is much more responsive! But i did find some issues....
- uptime is a large negative number of days, and some hours and minutes, bit of a time-warp
- Controller page: I tried 'Add' but got a blank browser screen, and in the log a Critical exception OSError(2)
- Devices: I got the first bit, to select a plugin, but clicking Submit gave an error "connection reset" and in the log NameError ('local variable rererenced before assignment')
- Advanced: tried to change NTP, Submit gave a MemoryError. It seems to be gradually using up the memory while in use.

Please keep up the good work!! I'm really keen to have a working Ethernet device, as potentially it should be faster and more reliable than wifi. Is there anything I can do to help?

Cheers
Andrew
Several bug fixes later...
I have everything working on my side, the source has been updated to 000df1a. Please update your side and try again.

But.. i do have some of the same memory problems as you have. If running on a stm32, please run it using: upyeasy.main(loglevel=0) and keep an eye on the memory!
If the memory level drops to about 7-8K, then you might start getting memory errors due to memory fragmentation. Workaround is to do a single action, like adding a controller, and then reboot.
This is happening due to the new deepcopy's (although now replaced in some parts by ordereddict init's) which make things more reliable but also consume more memory. I need to do a second round of memory optimization to free up memory.
A memory map can be seen when clicking on tools/info menu in the browser, then on your usb console a complete memory map is appearing when running in debug mode!

I'm not going to able to fix the memory problems shortly, due to the fact that i'm back home to the good old USA for the next few weeks.

AndrewJ
Normal user
Posts: 229
Joined: 14 Feb 2017, 12:38

Re: uPyEasy v0.3.0 released

#22 Post by AndrewJ » 02 Jun 2018, 12:32

LisaM wrote: 01 Jun 2018, 18:31
AndrewJ wrote: 28 May 2018, 21:58 Hi Lisa,
I see on Github that you've been hard at work on a new update (9ebffbd) of uPyEasy!! Thank you for all your effort on this project!

I'm still keen to get uPyEasy working on my STM32F405 AliExpress clones, or at least the one which still seems to work reliably.
So I updated my local repo and set about building the firmware.

Initially I had errors from a couple of "missing" files in the /unix directory (copy.py and types.py) which I was able to fix, also I needed to create new links for these into my micropython-stm32 file system before it would work.

uPyEasy then started up ok and ran for a short time. Some screens are working (MaIn, Config, Hardware - Hardware is much more responsive! But i did find some issues....
- uptime is a large negative number of days, and some hours and minutes, bit of a time-warp
- Controller page: I tried 'Add' but got a blank browser screen, and in the log a Critical exception OSError(2)
- Devices: I got the first bit, to select a plugin, but clicking Submit gave an error "connection reset" and in the log NameError ('local variable rererenced before assignment')
- Advanced: tried to change NTP, Submit gave a MemoryError. It seems to be gradually using up the memory while in use.

Please keep up the good work!! I'm really keen to have a working Ethernet device, as potentially it should be faster and more reliable than wifi. Is there anything I can do to help?

Cheers
Andrew
Several bug fixes later...
I have everything working on my side, the source has been updated to 000df1a. Please update your side and try again.

But.. i do have some of the same memory problems as you have. If running on a stm32, please run it using: upyeasy.main(loglevel=0) and keep an eye on the memory!
If the memory level drops to about 7-8K, then you might start getting memory errors due to memory fragmentation. Workaround is to do a single action, like adding a controller, and then reboot.
This is happening due to the new deepcopy's (although now replaced in some parts by ordereddict init's) which make things more reliable but also consume more memory. I need to do a second round of memory optimization to free up memory.
A memory map can be seen when clicking on tools/info menu in the browser, then on your usb console a complete memory map is appearing when running in debug mode!

I'm not going to able to fix the memory problems shortly, due to the fact that i'm back home to the good old USA for the next few weeks.
Hi Lisa,
Thanks for the update! :D
Yes, I'll try the latest update and monitor the memory. I'll try to look for things which decrease the memory a lot. (I do have the impression - it is only an impression - that memory is sometimes used while "nothing" is happening, no actions from the browser!)
As you're going to be away, can you suggest any strategy to help me narrow down where memory is getting used up? If I can do that, it may make it easier for you when you return.
Cheers
Andrew

LisaM
Normal user
Posts: 513
Joined: 11 Apr 2017, 18:29

Re: uPyEasy v0.3.0 released

#23 Post by LisaM » 02 Jun 2018, 18:31

Hi Andrew,

My biggest concern is the memory fragmentation due to constant creation/destruction of objects!
This is explained more in details here: https://docs.micropython.org/en/latest/ ... ained.html
The memory map mentioned in the post above is a great help to see how the memory is consumed over time, if you would take a look at these i would be very greatfull!

One of the things i’ve done so far is the DB cache, so it doesn’t create/destroy DB records as before for the most used DB tables anymore. Stuff like that can do miracles for ram limited systems like the STM32. The ESP32 has a bigger ram supply, so it doesnt happen there as quickly as with the STM32. But the STM32 has way faster response times for both I2C/SPI/UART and web calls, so for more professional systems the STM32 is prefered...

Cheers,

Lisa

AndrewJ
Normal user
Posts: 229
Joined: 14 Feb 2017, 12:38

Re: uPyEasy v0.3.0 released

#24 Post by AndrewJ » 02 Jun 2018, 19:45

LisaM wrote: 02 Jun 2018, 18:31 Hi Andrew,

My biggest concern is the memory fragmentation due to constant creation/destruction of objects!
This is explained more in details here: https://docs.micropython.org/en/latest/ ... ained.html
The memory map mentioned in the post above is a great help to see how the memory is consumed over time, if you would take a look at these i would be very greatfull!

One of the things i’ve done so far is the DB cache, so it doesn’t create/destroy DB records as before for the most used DB tables anymore. Stuff like that can do miracles for ram limited systems like the STM32. The ESP32 has a bigger ram supply, so it doesnt happen there as quickly as with the STM32. But the STM32 has way faster response times for both I2C/SPI/UART and web calls, so for more professional systems the STM32 is prefered...

Cheers,

Lisa
Hi Lisa,
Thanks for that, it helps me to understand the problem better. The information you linked contains another link to a section about "the heap", which uses the phrase "when the object goes out of scope". Can you elaborate what this means in practical terms, with an example or two from uPyEasy if possible? It might give me a better understanding what to look for.
Cheers
Andrew

AndrewJ
Normal user
Posts: 229
Joined: 14 Feb 2017, 12:38

Re: uPyEasy v0.3.0 released

#25 Post by AndrewJ » 12 Jun 2018, 19:18

LisaM wrote: 02 Jun 2018, 18:31 Hi Andrew,

My biggest concern is the memory fragmentation due to constant creation/destruction of objects!
This is explained more in details here: https://docs.micropython.org/en/latest/ ... ained.html
The memory map mentioned in the post above is a great help to see how the memory is consumed over time, if you would take a look at these i would be very greatfull!

One of the things i’ve done so far is the DB cache, so it doesn’t create/destroy DB records as before for the most used DB tables anymore. Stuff like that can do miracles for ram limited systems like the STM32. The ESP32 has a bigger ram supply, so it doesnt happen there as quickly as with the STM32. But the STM32 has way faster response times for both I2C/SPI/UART and web calls, so for more professional systems the STM32 is prefered...

Cheers,

Lisa
Hi Lisa,
I updated with the new version.
Running ok on start-up.
Made and saved some changes in Config. Free men dropped to about 17000 but recovered a bit. (Sorry I'm away from my computer so can't give you exact numbers or full log.)
Tried to go into Controller page. Free men plummeted! Down below 4000! Without even adding a controller. Upyeasy froze.

Looks like Controllers code is prime candidate for some GC memory collect statements?

I'm going to be busy for next couple of weeks so unlikely to be able to get much further till then.

Cheers Andrew.

fluppie
Normal user
Posts: 89
Joined: 09 Oct 2015, 13:23

Re: uPyEasy v0.3.0 released

#26 Post by fluppie » 17 Jun 2018, 23:38

After some fiddling got it working :-). I have a rev0 module:

Code: Select all

ets Jun  8 2016 00:22:57

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
ets Jun  8 2016 00:22:57

rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:4464
ho 0 tail 12 room 4
load:0x40078000,len:0
load:0x40078000,len:11816
entry 0x4007a9fc
I (278) cpu_start: Pro cpu up.
I (278) cpu_start: Single core mode
I (278) heap_init: Initializing. RAM available for dynamic allocation:
I (282) heap_init: At 3FFAE6E0 len 00001920 (6 KiB): DRAM
I (288) heap_init: At 3FFC5120 len 0001AEE0 (107 KiB): DRAM
I (294) heap_init: At 3FFE0440 len 00003BC0 (14 KiB): D/IRAM
I (300) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
I (307) heap_init: At 4008E0E8 len 00011F18 (71 KiB): IRAM
I (313) cpu_start: Pro cpu start user code
I (33) cpu_start: Starting scheduler on PRO CPU.
I (101) modsocket: Initializing
I (3051) wifi: wifi firmware version: c202b34
I (3051) wifi: config NVS flash: enabled
I (3051) wifi: config nano formating: disabled
I (3061) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE
I (3071) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE
I (3081) wifi: Init dynamic tx buffer num: 32
I (3081) wifi: Init data frame dynamic rx buffer num: 64
I (3081) wifi: Init management frame dynamic rx buffer num: 64
I (3091) wifi: wifi driver task: 3ffd1194, prio:23, stack:4096
I (3101) wifi: Init static rx buffer num: 10
I (3101) wifi: Init dynamic rx buffer num: 0
I (3101) wifi: wifi power manager task: 0x3ffd5de4 prio: 21 stack: 2560
I (3171) phy: phy_version: 383.0, 79a622c, Jan 30 2018, 15:38:06, 0, 0
I (3171) wifi: mode : null
I (3171) wifi: mode : sta (24:0a:c4:83:1e:b8)
I (3171) wifi: STA_START
I (3901) wifi: n:6 0, o:1 0, ap:255 255, sta:6 0, prof:1
I (4461) wifi: state: init -> auth (b0)
I (5461) wifi: state: auth -> init (2)
I (5461) wifi: n:6 0, o:6 0, ap:255 255, sta:6 0, prof:1
I (5461) wifi: STA_DISCONNECTED, reason:2
I (5591) wifi: n:6 0, o:6 0, ap:255 255, sta:6 0, prof:1
I (5591) wifi: state: init -> auth (b0)
I (6591) wifi: state: auth -> init (2)
I (6591) wifi: n:6 0, o:6 0, ap:255 255, sta:6 0, prof:1
I (6591) wifi: STA_DISCONNECTED, reason:2
I (6711) wifi: n:6 0, o:6 0, ap:255 255, sta:6 0, prof:1
I (6711) wifi: state: init -> auth (b0)
I (6711) wifi: state: auth -> assoc (0)
I (6721) wifi: state: assoc -> run (10)
I (6761) wifi: connected with WiFi-2.4-24E7, channel 6
I (6761) network: event 4
I (8041) event: sta ip: 192.168.1.25, mask: 255.255.255.0, gw: 192.168.1.1
I (8041) network: GOT_IP
I (9721) wifi: pm start, type:0

2018-06-17 23:36:29 [info] uPyEasy-uPyEasy: Main: Async loop, log level: 2
2018-06-17 23:36:29 [info] uPyEasy-uPyEasy: Main: uPyEasy running in Station mode
2018-06-17 23:36:29 [info] uPyEasy-uPyEasy: Main: uPyEasy Main Async Loop on IP adress: 192.168.1.25:80

Looks like I cannot read the analog inputs, is that correct?

LisaM
Normal user
Posts: 513
Joined: 11 Apr 2017, 18:29

Re: uPyEasy v0.3.0 released

#27 Post by LisaM » 29 Jun 2018, 18:33

AndrewJ wrote: 02 Jun 2018, 19:45 the phrase "when the object goes out of scope".
Hi Andrew,

It means that the object is no longer used by the function, because the function ended. Objects are allocated inside the function and when the functions end, the objects memory should be released.

Cheers,

Lisa

LisaM
Normal user
Posts: 513
Joined: 11 Apr 2017, 18:29

Re: uPyEasy v0.3.0 released

#28 Post by LisaM » 29 Jun 2018, 18:34

fluppie wrote: 17 Jun 2018, 23:38 Looks like I cannot read the analog inputs, is that correct?
why not? Are you getting an input error or so?

cheers,

Lisa

AndrewJ
Normal user
Posts: 229
Joined: 14 Feb 2017, 12:38

Re: uPyEasy v0.3.0 released

#29 Post by AndrewJ » 30 Jun 2018, 08:59

LisaM wrote: 29 Jun 2018, 18:33
AndrewJ wrote: 02 Jun 2018, 19:45 the phrase "when the object goes out of scope".
Hi Andrew,

It means that the object is no longer used by the function, because the function ended. Objects are allocated inside the function and when the functions end, the objects memory should be released.

Cheers,

Lisa
Hi Lisa,
Welcome back on the forum!
Thanks for your reply, it helps, although I'm still getting my head around the basics of micropython/python - could you possibly elaborate on the difference between an "object" and a "function", or perhaps give an example of each within uPyEasy please?
BTW, I saw a thread on the Micropython forum recently about memory problems, don't know if you have seen it already. https://forum.micropython.org/viewtopic.php?f=2&t=4912

Cheers
Andrew

LisaM
Normal user
Posts: 513
Joined: 11 Apr 2017, 18:29

Re: uPyEasy v0.3.0 released

#30 Post by LisaM » 08 Jul 2018, 16:45

AndrewJ wrote: 30 Jun 2018, 08:59 Thanks for your reply, it helps, although I'm still getting my head around the basics of micropython/python - could you possibly elaborate on the difference between an "object" and a "function", or perhaps give an example of each within uPyEasy please?
BTW, I saw a thread on the Micropython forum recently about memory problems, don't know if you have seen it already. https://forum.micropython.org/viewtopic.php?f=2&t=4912

Cheers
Andrew
Hi Andrew,

This tutorial explains about Python objects and garbage collection: http://www.tutorialspoint.com/python/py ... bjects.htm
I saw the memory thread, haven't got to implementing it. I'm busy with something big.

Cheers,

Lisa

AndrewJ
Normal user
Posts: 229
Joined: 14 Feb 2017, 12:38

Re: uPyEasy v0.3.0 released

#31 Post by AndrewJ » 08 Jul 2018, 21:08

LisaM wrote: 08 Jul 2018, 16:45
AndrewJ wrote: 30 Jun 2018, 08:59 Thanks for your reply, it helps, although I'm still getting my head around the basics of micropython/python - could you possibly elaborate on the difference between an "object" and a "function", or perhaps give an example of each within uPyEasy please?
BTW, I saw a thread on the Micropython forum recently about memory problems, don't know if you have seen it already. https://forum.micropython.org/viewtopic.php?f=2&t=4912

Cheers
Andrew
Hi Andrew,

This tutorial explains about Python objects and garbage collection: http://www.tutorialspoint.com/python/py ... bjects.htm
I saw the memory thread, haven't got to implementing it. I'm busy with something big.

Cheers,

Lisa
Hi Lisa,
Thanks for that link, I'll study it and compare some of the code in uPyEasy.
Good luck with the something big!
Andrew

LisaM
Normal user
Posts: 513
Joined: 11 Apr 2017, 18:29

Re: uPyEasy v0.3.0 released

#32 Post by LisaM » 09 Jul 2018, 13:33

AndrewJ wrote: 08 Jul 2018, 21:08 Hi Lisa,
Thanks for that link, I'll study it and compare some of the code in uPyEasy.
Good luck with the something big!
Andrew
If it all works, things will start to fly... ;)

AndrewJ
Normal user
Posts: 229
Joined: 14 Feb 2017, 12:38

Re: uPyEasy v0.3.0 released

#33 Post by AndrewJ » 09 Jul 2018, 14:16

LisaM wrote: 09 Jul 2018, 13:33
AndrewJ wrote: 08 Jul 2018, 21:08 Hi Lisa,
Thanks for that link, I'll study it and compare some of the code in uPyEasy.
Good luck with the something big!
Andrew
If it all works, things will start to fly... ;)
That sounds exciting!
Best of luck :D

Post Reply

Who is online

Users browsing this forum: No registered users and 10 guests