New directory and file structure V024

Moderators: Voyager, BertB, grovkillen, Stuntteam, LisaM

Post Reply
Message
Author
AndrewJ
Normal user
Posts: 229
Joined: 14 Feb 2017, 12:38

Re: New directory and file structure V024

#11 Post by AndrewJ » 16 Apr 2018, 22:55

Hi Lisa
Sorry to hit you with yet another question.... :roll:
As you said earlier, I still need to do the work around steps for micropython for STM32. Fine.
But in the overall new file structure, I have "two micropythons" (micropython, micropython-stm32). The second is where I did the work around.

So.... presumably it is also the second one (micropython-stm32) which I should use when doing the following steps - yes?

Code: Select all

- make a modules directory (in ports esp32 or stm32)
- goto modules dir 
- make a symlink to stm32 modules : ln -s upyeasy/modules/stm32/* .
- make a symlink to src: ln -s upyeasy/src upyeasy
And, another question, presumably it is also in micropython-stm32 (not micropython) that I should do the build of uPyEasy firmware for STM32 - yes?

Sorry if I'm being slow here!! I just want to be sure I'm not going off the straight-and-narrow.

Cheers
Andrew

AndrewJ
Normal user
Posts: 229
Joined: 14 Feb 2017, 12:38

Re: New directory and file structure V024

#12 Post by AndrewJ » 20 Apr 2018, 09:24

AndrewJ wrote: 16 Apr 2018, 22:12
AndrewJ wrote: 16 Apr 2018, 20:57
LisaM wrote: 16 Apr 2018, 20:42
Hi Andrew,

1. Yes, this is a problem in the STM32 micropython firmware, not in uPyEasy.
2. Yes and no (no copy needed!):
- make a modules directory (in ports esp32 or stm32)
- goto modules dir
- make a symlink to stm32 modules : ln -s upyeasy/modules/stm32/* .
- make a symlink to src: ln -s upyeasy/src upyeasy
3. Yes, upyeasy modules will be added to the firmware because the entire stm32/modules dir will be added (symlinks followed and files included).

Cheers,

Lisa
Hi Lisa,
Thanks for that! I'll give it a go. :)
I'm still getting my head round the changed filestructure, but once I've been right the way through it should come easier.

I guess the benefit of the new structure is that all the separate parts, (upyeasy, micropython, micropython-lib etc etc) are together and can be kept updated individually?

I'll let you know how it goes!
Andrew
Hi again Lisa,
I just noticed that there is a new update on Github for uPyEasy, with the changes to the links in modules, so they refer to /unix. So I did "git pull" to update my local repo (to commit fb7e90d)
Unfortunately there seem to be five folder links in stm32 which are broken, these are
- json
- picoweb
- umodbus
- urllib
- utemplate
In the /unix folder, these folders are not present.
HTH.
Andrew
Bump :)

AndrewJ
Normal user
Posts: 229
Joined: 14 Feb 2017, 12:38

Re: New directory and file structure V024

#13 Post by AndrewJ » 20 Apr 2018, 09:25

AndrewJ wrote: 16 Apr 2018, 22:55 Hi Lisa
Sorry to hit you with yet another question.... :roll:
As you said earlier, I still need to do the work around steps for micropython for STM32. Fine.
But in the overall new file structure, I have "two micropythons" (micropython, micropython-stm32). The second is where I did the work around.

So.... presumably it is also the second one (micropython-stm32) which I should use when doing the following steps - yes?

Code: Select all

- make a modules directory (in ports esp32 or stm32)
- goto modules dir 
- make a symlink to stm32 modules : ln -s upyeasy/modules/stm32/* .
- make a symlink to src: ln -s upyeasy/src upyeasy
And, another question, presumably it is also in micropython-stm32 (not micropython) that I should do the build of uPyEasy firmware for STM32 - yes?

Sorry if I'm being slow here!! I just want to be sure I'm not going off the straight-and-narrow.

Cheers
Andrew
Bump :)

LisaM
Normal user
Posts: 513
Joined: 11 Apr 2017, 18:29

Re: New directory and file structure V024

#14 Post by LisaM » 20 Apr 2018, 18:54

AndrewJ wrote: 16 Apr 2018, 22:55 Hi Lisa
Sorry to hit you with yet another question.... :roll:
As you said earlier, I still need to do the work around steps for micropython for STM32. Fine.
But in the overall new file structure, I have "two micropythons" (micropython, micropython-stm32). The second is where I did the work around.

So.... presumably it is also the second one (micropython-stm32) which I should use when doing the following steps - yes?

Code: Select all

- make a modules directory (in ports esp32 or stm32)
- goto modules dir 
- make a symlink to stm32 modules : ln -s upyeasy/modules/stm32/* .
- make a symlink to src: ln -s upyeasy/src upyeasy
And, another question, presumably it is also in micropython-stm32 (not micropython) that I should do the build of uPyEasy firmware for STM32 - yes?

Sorry if I'm being slow here!! I just want to be sure I'm not going off the straight-and-narrow.

Cheers
Andrew
Sorry, excellent weather over here.. ;)
In micropython-stm32 you make uPyEasy for the STM32, these steps you do over there are:
- make a modules directory : mkdir micropython-stm32/ports/stm32/modules
- goto micropython-stm32/ports/stm32/modules dir
- make a symlink to stm32 modules : ln -s upyeasy/modules/stm32/* . (upyeasy is where your source root tree is)
- make a symlink to src: ln -s upyeasy/src upyeasy

cd ..
make MICROPY_PY_WIZNET5K=5500 BOARD=PYBV3 USE_PYDFU=0 deploy

AndrewJ
Normal user
Posts: 229
Joined: 14 Feb 2017, 12:38

Re: New directory and file structure V024

#15 Post by AndrewJ » 20 Apr 2018, 21:24

LisaM wrote: 20 Apr 2018, 18:54
AndrewJ wrote: 16 Apr 2018, 22:55 Hi Lisa
Sorry to hit you with yet another question.... :roll:
As you said earlier, I still need to do the work around steps for micropython for STM32. Fine.
But in the overall new file structure, I have "two micropythons" (micropython, micropython-stm32). The second is where I did the work around.

So.... presumably it is also the second one (micropython-stm32) which I should use when doing the following steps - yes?

Code: Select all

- make a modules directory (in ports esp32 or stm32)
- goto modules dir 
- make a symlink to stm32 modules : ln -s upyeasy/modules/stm32/* .
- make a symlink to src: ln -s upyeasy/src upyeasy
And, another question, presumably it is also in micropython-stm32 (not micropython) that I should do the build of uPyEasy firmware for STM32 - yes?

Sorry if I'm being slow here!! I just want to be sure I'm not going off the straight-and-narrow.

Cheers
Andrew
Sorry, excellent weather over here.. ;)
In micropython-stm32 you make uPyEasy for the STM32, these steps you do over there are:
- make a modules directory : mkdir micropython-stm32/ports/stm32/modules
- goto micropython-stm32/ports/stm32/modules dir
- make a symlink to stm32 modules : ln -s upyeasy/modules/stm32/* . (upyeasy is where your source root tree is)
- make a symlink to src: ln -s upyeasy/src upyeasy

cd ..
make MICROPY_PY_WIZNET5K=5500 BOARD=PYBV3 USE_PYDFU=0 deploy
Hi Lisa,
Thanks for that, that makes it clear. :)

How about the missing folders and files in /unix (post #10) ?
Unfortunately there seem to be five folder links in stm32 which are broken, these are
- json
- picoweb
- umodbus
- urllib
- utemplate
In the /unix folder, these folders are not present.
HTH.
Thanks in advance. :)

Andrew

LisaM
Normal user
Posts: 513
Joined: 11 Apr 2017, 18:29

Re: New directory and file structure V024

#16 Post by LisaM » 23 Apr 2018, 19:22

Hi andrew,

All missing Unix dir's are fixed!
I also added some new features for ESP32 uPyEasy, like auto switch to AP mode when the wifi connection is not available at startup. v0.2.5 is about to be released.

Cheers,

Lisa

AndrewJ
Normal user
Posts: 229
Joined: 14 Feb 2017, 12:38

Re: New directory and file structure V024

#17 Post by AndrewJ » 23 Apr 2018, 19:43

LisaM wrote: 23 Apr 2018, 19:22 Hi andrew,

All missing Unix dir's are fixed!
I also added some new features for ESP32 uPyEasy, like auto switch to AP mode when the wifi connection is not available at startup. v0.2.5 is about to be released.

Cheers,

Lisa
Great news! Thanks, Lisa, for all your hard work on this. :)
Andrew

AndrewJ
Normal user
Posts: 229
Joined: 14 Feb 2017, 12:38

Re: New directory and file structure V024

#18 Post by AndrewJ » 24 Apr 2018, 08:25

LisaM wrote: 23 Apr 2018, 19:22 Hi andrew,

All missing Unix dir's are fixed!
I also added some new features for ESP32 uPyEasy, like auto switch to AP mode when the wifi connection is not available at startup. v0.2.5 is about to be released.

Cheers,

Lisa
Hi again Lisa,
Thx, that worked fine, I pulled the latest update from Github to my local repo, and all the links are good now. :D :D
Now to have a go at building the whole thing for STM32! :P
Cheers
Andrew

AndrewJ
Normal user
Posts: 229
Joined: 14 Feb 2017, 12:38

Re: New directory and file structure V024

#19 Post by AndrewJ » 25 Apr 2018, 16:19

AndrewJ wrote: 24 Apr 2018, 08:25
LisaM wrote: 23 Apr 2018, 19:22 Hi andrew,

All missing Unix dir's are fixed!
I also added some new features for ESP32 uPyEasy, like auto switch to AP mode when the wifi connection is not available at startup. v0.2.5 is about to be released.

Cheers,

Lisa
Hi again Lisa,
Thx, that worked fine, I pulled the latest update from Github to my local repo, and all the links are good now. :D :D
Now to have a go at building the whole thing for STM32! :P
Cheers
Andrew
Hi Lisa,
I updated my repo and got all the new filesystem set up and made the symlinks to refer to the upyeasy STM32 'src' and ''modules' files, and was able to build successfully. :)

(I did find errors complaining that two files were missing in my latest micropython-stm32. These were 'lib/axtls/ssl/version.h' and 'lib/axtls/ssl/tls1.h'
I got round this by copying these files from my previous micropython-stm32.)

To get it to build, I ended up using "make MICROPY_PY_WIZNET5K=5500 BOARD=PYBV3" (without "USE_PYDFU=0 deploy") as with the full line I got errors ("cannot find -laxtls") and then found that the full command worked ok to download to my STM32 board.

So, long story short, I've got firmware onto my STM32 with the new file system. :D :D

There is a "but" however.... :(
When I start uPyEasy on the board, it gets to the line "Init: rule table" then throws an error:

Code: Select all

2015-01-01 10:57:44 [debug] uPyEasy: Init: config Table
2015-01-01 10:57:44 [debug] uPyEasy: Init: network Table
2015-01-01 10:57:44 [debug] uPyEasy: Init: protocol Table
2015-01-01 10:57:44 [debug] uPyEasy: Init: controller Table
2015-01-01 10:57:44 [debug] uPyEasy: Init: hardware Table
2015-01-01 10:57:44 [debug] uPyEasy: Init: dxpin Table
2015-01-01 10:57:44 [debug] uPyEasy: Init: dxmap Table
2015-01-01 10:57:44 [debug] uPyEasy: Init: plugin Table
2015-01-01 10:57:44 [debug] uPyEasy: Init: pluginstore Table
2015-01-01 10:57:44 [debug] uPyEasy: Init: device Table
2015-01-01 10:57:44 [debug] uPyEasy: Init: service Table
2015-01-01 10:57:44 [debug] uPyEasy: Init: notification Table
2015-01-01 10:57:44 [debug] uPyEasy: Init: advanced Table
2015-01-01 10:57:44 [debug] uPyEasy: Init: script Table
2015-01-01 10:57:44 [debug] uPyEasy: Init: rule Table
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "upyeasy/__init__.py", line 40, in main
  File "upyeasy/init.py", line 158, in init
  File "upyeasy/db.py", line 43, in public
  File "upyeasy/db.py", line 1, in <listcomp>
  File "filedb.py", line 96, in get
  File "filedb.py", line 96, in get
ValueError: syntax error in JSON
>>> 

I've had a look at the code and googled the error but I'm afraid I'm stuck!
Any ideas where I'm going wrong, please? :?:

Cheers
Andrew

AndrewJ
Normal user
Posts: 229
Joined: 14 Feb 2017, 12:38

Re: New directory and file structure V024

#20 Post by AndrewJ » 25 Apr 2018, 22:58

Hi again Lisa,
Following on from post #19 above.
I had a go at debugging the error by adding some print lines ib filedb.py, in 'get' as follows:

Code: Select all

88   @classmethod
89    def get(cls):
90        # Return dict!
91        for dirent in uos.ilistdir("%s/%s" % (cls.__db__.name, cls.__table__)):
92            fname = dirent[0]
93            print("filedb.py line 93: fname = ", fname )
94            if fname[0] == "." or dirent[1] == 16384:
95                continue
96            with open(cls.fname(fname)) as f:
97                #print("filedb.py line 96: fname(fname) =", fname(fname) )
98                print("filedb.py line 97: f=", f)
99                print("filedb.py line 98: f.read() =", f.read() )
100              yield ujson.loads(f.read())
The output I get is like this....

Code: Select all

2015-01-01 00:00:00 [debug] uPyEasy: Init: service Table
2015-01-01 00:00:00 [debug] uPyEasy: Init: notification Table
2015-01-01 00:00:00 [debug] uPyEasy: Init: advanced Table
2015-01-01 00:00:00 [debug] uPyEasy: Init: script Table
2015-01-01 00:00:00 [debug] uPyEasy: Init: rule Table
filedb.py line 93: fname =  2000-01-01_00-00-001062930438
filedb.py line 97: f= <io.TextIOWrapper 2000e9b0>
filedb.py line 98: f.read() = �����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "upyeasy/__init__.py", line 40, in main
  File "upyeasy/init.py", line 158, in init
  File "upyeasy/db.py", line 43, in public
  File "upyeasy/db.py", line 1, in <listcomp>
  File "filedb.py", line 100, in get
  File "filedb.py", line 100, in get
ValueError: syntax error in JSON
>>> 
I'm not sure what the output from f.reat should look like, but I think the "yield ujson.loads... " line 100 is expecting something that looks like json format, which the print output doesn't look like to me (?).
Can you help me with next steps to debug this, please?

tia
Andrew

AndrewJ
Normal user
Posts: 229
Joined: 14 Feb 2017, 12:38

Re: New directory and file structure V024

#21 Post by AndrewJ » 27 Apr 2018, 20:16

Hi Lisa,
I still have the error mentioned in my posts above, would you be able to help, please? :)
I just found the latest uPyEasy update (f20076c) on Github and gave it a try, but I still get the "syntax error in JSON" as above.
I've pretty much run out of ideas at the moment.

Cheers
Andrew

AndrewJ
Normal user
Posts: 229
Joined: 14 Feb 2017, 12:38

Re: New directory and file structure V024

#22 Post by AndrewJ » 27 Apr 2018, 21:22

Hi Lisa
Another update (still on my STM32 pyboard clone).....
I thought a bit more and something made me look in the config files on my board (PYBFLASH/config).
In the subfolders, I found lots of files with the uPyEasy-style filenames, containing nothing but lots of characters like a 'y' with 2 dots above. I wonder if I corrupted the flash, perhaps by unplugging it without dismounting. :oops:
Anyway, I deleted all those files and rebooted. It got much further through the startup sequence...then displays the output of the print commands I put in filedb.py for debugging, then throws a similar error.

Code: Select all

(snipped)
2015-01-01 00:00:00 [debug] uPyEasy: Init: rule Table
2015-01-01 00:00:00 [debug] uPyEasy: GC MEM Free: 30,432
2015-01-01 00:00:00 [debug] uPyEasy: Init: Create Config Record
2015-01-01 00:00:00 [debug] uPyEasy: GC MEM Free: 30,288
2015-01-01 00:00:01 [debug] uPyEasy: Init: Create Network Record
2015-01-01 00:00:01 [debug] uPyEasy: GC MEM Free: 30,112
2015-01-01 00:00:02 [debug] uPyEasy: Init: Create Hardware Record
2015-01-01 00:00:02 [debug] uPyEasy: GC MEM Free: 30,080
2015-01-01 00:00:02 [debug] uPyEasy: Hal: hardwaredb init
2015-01-01 00:00:02 [debug] uPyEasy: GC MEM Free: 30,000
2015-01-01 00:00:02 [debug] uPyEasy: Hal: hardwaredb pyboard
2015-01-01 00:00:02 [debug] uPyEasy: GC MEM Free: 30,000
2015-01-01 00:00:02 [debug] uPyEasy: Hal: board info
2015-01-01 00:00:02 [debug] uPyEasy: GC MEM Free: 30,016
2015-01-01 00:00:02 [debug] uPyEasy: Hal: board pyboard
2015-01-01 00:00:02 [debug] uPyEasy: GC MEM Free: 30,000
2015-01-01 00:00:03 [debug] uPyEasy: Init: Create dxpin Record
2015-01-01 00:00:03 [debug] uPyEasy: GC MEM Free: 30,048
2015-01-01 00:00:04 [debug] uPyEasy: Hal: dxpins_init
2015-01-01 00:00:04 [debug] uPyEasy: GC MEM Free: 29,712
2015-01-01 00:00:04 [debug] uPyEasy: Hal: dxpins_init pyboard
2015-01-01 00:00:04 [debug] uPyEasy: GC MEM Free: 29,712
2015-01-01 00:00:04 [debug] uPyEasy: Hal: board info
2015-01-01 00:00:04 [debug] uPyEasy: GC MEM Free: 29,664
2015-01-01 00:00:04 [debug] uPyEasy: Hal: board pyboard
2015-01-01 00:00:04 [debug] uPyEasy: GC MEM Free: 29,648
2015-01-01 00:00:04 [debug] uPyEasy: STM32: init_pindb
2015-01-01 00:00:04 [debug] uPyEasy: GC MEM Free: 29,120
2015-01-01 00:00:05 [debug] uPyEasy: Init: Create advanced Record
2015-01-01 00:00:06 [debug] uPyEasy: GC MEM Free: 29,792
2015-01-01 00:00:06 [debug] uPyEasy: Hal: Init
2015-01-01 00:00:06 [debug] uPyEasy: GC MEM Free: 29,696
filedb.py line 93: fname =  2015-01-01_00-00-01859068510
filedb.py line 97: f= <io.TextIOWrapper 20010400>
filedb.py line 98: f.read() = {"fbkey": "", "ip": "", "gateway": "", "fbssid": "", "subnet": "", "ssid": "", "key": "", "timestamp": "2015-01-01_00-00-01859068510", "spi": 0, "dns": "", "mode": "STA", "cs": "", "rst": ""}
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "upyeasy/__init__.py", line 40, in main
  File "upyeasy/init.py", line 207, in init
  File "upyeasy/hal.py", line 46, in init_network
  File "upyeasy/hal.py", line 42, in init_network
  File "upyeasy/db.py", line 97, in getrow
  File "filedb.py", line 100, in get
  File "filedb.py", line 100, in get
ValueError: syntax error in JSON
>>> 
The print output from line 98 looks to me rather like a network config file, so I looked in PYBFLASH/config/network but no file is there. I also checked all the other config subfolders, no files there. I don't know where this info is coming from - perhaps the database?? But it appears that it is somehow not in the expected format.

I'll keep trying, but I really would value your input when you can, please ;)

TIA
Andrew

LisaM
Normal user
Posts: 513
Joined: 11 Apr 2017, 18:29

Re: New directory and file structure V024

#23 Post by LisaM » 28 Apr 2018, 12:41

AndrewJ wrote: 25 Apr 2018, 22:58 2015-01-01 00:00:00 [debug] uPyEasy: Init: rule Table
filedb.py line 93: fname = 2000-01-01_00-00-001062930438
filedb.py line 97: f= <io.TextIOWrapper 2000e9b0>
filedb.py line 98: f.read() = �����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "upyeasy/__init__.py", line 40, in main
File "upyeasy/init.py", line 158, in init
File "upyeasy/db.py", line 43, in public
File "upyeasy/db.py", line 1, in <listcomp>
File "filedb.py", line 100, in get
File "filedb.py", line 100, in get
ValueError: syntax error in JSON
>>>
Hi Andrew,

I suspect that your rule table config file has gone corrupt, it happens to me also sometimes on the STM32F405 platform. Just use the usb share and delete the rule table.
I also have optimized uPyEasy to use far less memory, switched from pre-loading to JIT, since i was running out of it on the STM32. The GC mem free debug was added to see which parts of uPyEasy are using memory, i'll make it a main() parameter so you can switch it on and off. It also provides the possibility to monitor memory leaks.

Ps. the STM is suffering from memory fragmentation, something i'm currently fixing also.

Cheers,

Lisa

AndrewJ
Normal user
Posts: 229
Joined: 14 Feb 2017, 12:38

Re: New directory and file structure V024

#24 Post by AndrewJ » 28 Apr 2018, 21:26

LisaM wrote: 28 Apr 2018, 12:41
Hi Andrew,

I suspect that your rule table config file has gone corrupt, it happens to me also sometimes on the STM32F405 platform. Just use the usb share and delete the rule table.
I also have optimized uPyEasy to use far less memory, switched from pre-loading to JIT, since i was running out of it on the STM32. The GC mem free debug was added to see which parts of uPyEasy are using memory, i'll make it a main() parameter so you can switch it on and off. It also provides the possibility to monitor memory leaks.

Ps. the STM is suffering from memory fragmentation, something i'm currently fixing also.

Cheers,

Lisa
Hi Lisa,
Thanks for your reply. It helps to know I'm not alone with this! I'll try deleting the rule table and report back.

Cheers
Andrew

Post Reply

Who is online

Users browsing this forum: No registered users and 10 guests