Major Software Damage to OS (UBUNTU 20.04)

Moderators: grovkillen, Stuntteam, TD-er

Post Reply
Message
Author
JimBobK27
New user
Posts: 9
Joined: 06 May 2021, 04:31

Major Software Damage to OS (UBUNTU 20.04)

#1 Post by JimBobK27 » 10 May 2021, 23:16

I downloaded the ESPeasy pgk for GitHUB, unzipped file to find a TON of files, mostly for systems that don't interest me at all. So, tried to find out HOW TO ... build just what I wanted... Executed the ESPeasy_build.sh founded in TOOLS.. It ran until it consumed every available disk space on root drive. After a reboot, found that the script had install a Python Vir. Environment!!!! My desktop was wiped out, along with all my Firefox database... But that's not all, it erased everything on my secondary disk drive, which held all my current work with IoT devices.. PLEASE make sure you warn user of this actions!!!! I would provide log files, but everything is GONE!!!!

I monitored the progress for 45 mins, everything appeared to be running ok, after 4 hrs. I came back to terminal to find a ton of messages about no disk space...

README file again were of little or no help in this script.....

TD-er
Core team member
Posts: 8644
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Major Software Damage to OS (UBUNTU 20.04)

#2 Post by TD-er » 10 May 2021, 23:47

I'm really sorry to hear you lost data.
But to be honest, there are quite a few protective measures taken to make sure the user environment was NOT to be tampered with.
It does indeed install a virtual env, just to make sure it is not tampering with your Python environment as it is rather important to leave that as it is in Ubuntu.

So this is what the script does:
- Install a virtual Python environment in ~/.venv/python3.8/
- Activate the virtual Python environment: source ~/.venv/python3.8/bin/activate
- Install platformio and all what is mentioned in the requirements.txt file
- Fetch the ESPEasy sources from GitHub
- Update platformio from inside the ESPEasy repository, to make sure the toolchain is loaded (only for this Python environment)
- Build all, only inside the ESPEasy directory (folder .pio)
- Collect all in ZIP files.


And above all, I also specifically made it all present in the start of the build script as configurable parameters:

Code: Select all

#!/bin/bash

# Build script to make a build in Linux.
# Tested on Ubuntu 18.04 from within Windows 10

# Needed packages:
# sudo apt-get update
# sudo apt-get upgrade
# sudo apt install python3-minimal virtualenv build-essential zip binutils
# For Python 3.8:
# sudo apt install -y software-properties-common
# sudo add-apt-repository ppa:deadsnakes/ppa
# sudo apt install -y python3.8


VENV=~/.venv/python3.8
SRC=~/GitHub/letscontrolit/ESPEasy
REPO=https://github.com/letscontrolit/ESPEasy.git
BRANCH=mega

PULL_REQ=0
DESCRIPTION=""
So really, what more should be in there?
The amount of data needed?
I do think that you may have ran out of disk space, but that could only really be harmful if you ran the build as root.
Otherwise, as a normal user, you would not have been able to fill up the complete file system as Ubuntu always reserves some space only to be used as root. (default of 5% I think)
Of course you can tweak that out, but still the default is some reserved space.

To summarize:
- All paths are configurable in the script and plainly made available at the top of the file, not scattered around
- Python virtual env is installed which should -as far as I know- not interfere with your system Python configuration. (please tell me if I'm wrong here, as I have used it like this for close to 10 years on lots and lots of systems)
- Builds are made ONLY inside the dir where you fetched/cloned ESPEasy (e.g.: gijs@espeasybuild:~/GitHub/letscontrolit/ESPEasy/.pio/build )
- PlatformIO does fetch packages and libraries into your own home dir: ~/.platformio/

Please tell me where we missed out big time here in not making things as safe as possible?
The only thing I could think of that could be better is that environments are cleaned after build, as they do leave quite a lot of files there which are not needed if you only build once.
On the other hand, if you build more than once, you may not want to download it all again every time.

My rough estimate is that the complete build environment takes like 12 GB if you do not clean it after building it.

TD-er
Core team member
Posts: 8644
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Major Software Damage to OS (UBUNTU 20.04)

#3 Post by TD-er » 11 May 2021, 09:21

I did think about this quite a bit last night and I really can't explain why your secondary drive was even touched by this script.
The only thing I can come up with is that the system may have become unstable by filling up the root partition.
This may cause all kinds of issues like other processes not able to update data structures, which may corrupt those. (e.g. a database like structure kept by Firefox)
But I really don't know how it could even touch data on another partition.
Maybe a hardware failure on one of the drives?

If you look at the build script, you can see it only clears the previous build dir (which will not be executed on a first build) and no other rm like command is given.

This also calls the before_deploy script, which does create a temp directory using:
TMP_DIST=`mktemp -d`

This directory is then populated with files, which is later zip'ed.
This temp dir is deleted:
rm -Rf ${TMP_DIST}/* 2>/dev/null

So the only thing I can imagine is that mktemp did not return a new directory in the /tmp folder?
But then I really don't understand what it may have returned that could lead to such a massive wipe of data.


This build script was created to perform the nightly builds and help users to create a build based on a pull request.
It has also been used for building using a Vagrant setup.
This is the first time I get a report of data getting wiped.

TD-er
Core team member
Posts: 8644
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Major Software Damage to OS (UBUNTU 20.04)

#4 Post by TD-er » 11 May 2021, 18:40

I think I understand what may have happened here...
If you don't have coreutils installed, you don't have the mktemp command.
So the temp dir is "" (empty) and thus
rm -Rf ${TMP_DIST}/* 2>/dev/null
will delete your entire system.

I will add a subdir to the TMP_DIST to make sure there is always a path present.
I also will add a check to the generated temp dir to see if it is not empty and also if write actions are permitted.

TD-er
Core team member
Posts: 8644
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Major Software Damage to OS (UBUNTU 20.04)

#5 Post by TD-er » 12 May 2021, 00:44

I am really sorry to realize what has happened to your setup, due to a missing installed package on your system.

So as soon as I had put my daughter to bed, I started working on improving the script (which was originally not intended to be used by end-users, only for automated builds)
Now, 5 hours later, I think (I hope...) I have included enough documentation and checks to make sure something like this may not happen in the future.

See the pull request: https://github.com/letscontrolit/ESPEasy/pull/3631

JimBobK27
New user
Posts: 9
Joined: 06 May 2021, 04:31

Re: Major Software Damage to OS (UBUNTU 20.04)

#6 Post by JimBobK27 » 13 May 2021, 00:06

Sorry for not getting back to monitor this post, but I spent almost 2 full days restoring my system.. (Never know how good your backup are until you try to restore)

I appreciate the pride you take in this project, but there are several things missing in the body all your work.. (As you stated, 10yrs of experience,, Mine is over 50yrs..)

First thing, anytime the a chance of the Newbie, doing something stupid... Warn him with a LARGE COMMENT at the top of the script... Like "WARNING WILL ROBINSON,
WARNING WILL ROBINSON!!
" You about to do something dangerous! Loss of data may result.....

I will try an recount my experience... First, I work from a laptop, very small root dev... I have a secondary dev. that I work from.. Root dev had only 3G free, hd2 had 180G.
I was operating as ROOT, yes, mine own fault!!!, but I have had problem with permissions with several of these packages... All I was trying to do was get a typical HELP output from the script... the "--help" option was parsed as , what I think was option 'p' a.k.a pull??? I think.... Not documented in header of script!!! Once started, I opt to watch the show, but then after 2 hours, I knew I was in trouble, but Control 'C' might cause more problems... First error on no disk space, just passed to the next and the next.....

The list of requirement was taken to be a part of 'auto configure' app... A comment in BOLD CAPS should say YOU NEED THESE OR ELSE!!!!

I'm back up and running, but very cautious of using your software!!!! USER BEWARE!!!

If you need help in documentation, which every programmer HATES!!! I might be able to donate my time to generate a "New Comer's Guide to the Universe"!!

TD-er
Core team member
Posts: 8644
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Major Software Damage to OS (UBUNTU 20.04)

#7 Post by TD-er » 13 May 2021, 00:51

Like I said, it was never intended to be harmful and I even had some checks in it to see if the directory was present. But as you have found out the (truly) hard way, those checks were far from enough.
Another thing is that the script was initially written to help me (not a user) to create test builds for users in a more reproducible way.
Later it got used by Travis builds and also in Vagrant build setups, but still not really meant for regular users to be used.

But now it is packed with options and even a -h option showing you all options and a 5 minutes pause to allow for a quick check of the given parameters (and a warning).

And about my "10 yrs of experience", that was just about the Python + virtualenv stuff.
I've been tinkering with sensors/motors and computers since the late '80s (I'm born in '76) when I connected all kinds of motors to my MSX printer port and tried to get things in sync with counting pulses on the joystick port from switches made from paperclips pushed up by Lego gears and pushing down on aluminium foil wrapped Lego pegs.
All on a computer which had more free RAM at boot than the ESP8266 has now.
So not a lot has changed since, only now I don't have to look up every word in the English dictionary anymore to understand texts :) (and phone bills are not that extravagant anymore...)


Well, I'm glad you didn't try this on your regular device or else you would have been in a lot more trouble.
Also I realized what could have happened here if it had gone wrong like you had on my "Ubuntu-on-Windows" as that environment mounts your Windows drives (all of them) on /mnt.
So in a way, I am glad you reported it and I could fix it.
Spent all afternoon yesterday to get it out ASAP so nobody else would end up with this mess.

JimBobK27
New user
Posts: 9
Joined: 06 May 2021, 04:31

Re: Major Software Damage to OS (UBUNTU 20.04)

#8 Post by JimBobK27 » 13 May 2021, 03:31

Thank you for your work... I hope I may have saved someone else from my fate!!
I will pull your new script and play with it, but NOT as root...

Just a question on the side, just how do you just compile, link, and flash just one target?? Make sure it is the only one set in dir .pio??

TD-er
Core team member
Posts: 8644
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Major Software Damage to OS (UBUNTU 20.04)

#9 Post by TD-er » 13 May 2021, 11:00

Right now the least complex way is via Visual Studio Code.
Or maybe if you have experience with tools like Vagrant, you can try to use that route. I have included a vagrant file.
This will build a version specific to your needs with a config file See: https://github.com/letscontrolit/ESPEasy/issues/2594

I guess the (infamous) build script can also use an option to build only custom builds. (I did add --esp32 and --esp82xx to specify only one of those subsets)

Maybe when you start testing, try running it first as another user, to make sure it cannot delete anything not owned by that user.

JimBobK27
New user
Posts: 9
Joined: 06 May 2021, 04:31

Re: Major Software Damage to OS (UBUNTU 20.04)

#10 Post by JimBobK27 » 17 May 2021, 01:11

I finally got around to testing your new script, but ran into trouble from the start...
Ran as me, not ROOT!! from my secondary disk drive (SD card).. Main problem seems to be the directory level... git clone places the pull, one layer down from script level...

Attached is a HACKED up copy of your new script, with my changes to make it work... Look for the tag, "JBK". I left the orginal code in tack, but commented out.

Still runs almost 10 hours, for what I thought was only DOCS and ESP82xx files... I am closer than I have been in a week..

Will keep you posted on progress. Were there any logs of the build generate? I can't find any... Otherwise, I would include them.

JimBobK (JBK)
Attachments
build_ESPEasy2.sh.zip
(3.85 KiB) Downloaded 144 times

TD-er
Core team member
Posts: 8644
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Major Software Damage to OS (UBUNTU 20.04)

#11 Post by TD-er » 17 May 2021, 08:37

Hmm 10 hours build time... That's a rather underpowered machine you're running.
I guess the SD-card is here to blame, as it is probably mounted with "write through" cache, as it is probably considered to be a 'removable' device.
Even on a 4 year old Intel NUC with a celeron, the complete build will not take over 2 hours.

I will have a look at your comments.

TD-er
Core team member
Posts: 8644
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Major Software Damage to OS (UBUNTU 20.04)

#12 Post by TD-er » 17 May 2021, 09:02

Just browsed through your changes.

"pip -> pip3" is a valid change, although since the Python virtual env with Python 3 should be active, there should be an alias for pip3.
So I doubt you have a virtual env active then and if not, you might alter the system Python packages, which is obviously not something that should happen.

Code: Select all

cd ${SCRIPT_DIR}
#cd ..       JBK
SRC=`pwd`/ESPEasy    # JBK
This I don't get...
SCRIPT_DIR should be the path to the build script regardless of where you start it from.

For example, if you cloned the repository into:
/home/user/GitHub/letscontrolit/ESPEasy
And run it from /home/user, the contents of SCRIPT_DIR should be /home/user/GitHub/letscontrolit/ESPEasy/tools

In this test I just added an echo ${SCRIPT_DIR} followed by an exit 0 like this:

Code: Select all

cd ${SCRIPT_DIR}
cd ..
SRC=`pwd`

echo ${SCRIPT_DIR}
exit 0
Result:

Code: Select all

gijs@DESKTOP-QLB7N8I:~/GitHub/TD-er/ESPEasy/tools$ ./build_ESPeasy.sh
/home/gijs/GitHub/TD-er/ESPEasy/tools
gijs@DESKTOP-QLB7N8I:~/GitHub/TD-er/ESPEasy/tools$ cd ../..
gijs@DESKTOP-QLB7N8I:~/GitHub/TD-er$ ESPEasy/tools/build_ESPeasy.sh
/home/gijs/GitHub/TD-er/ESPEasy/tools
So I don't understand why you needed to move one directory up instead of down.

JimBobK27
New user
Posts: 9
Joined: 06 May 2021, 04:31

Re: Major Software Damage to OS (UBUNTU 20.04)

#13 Post by JimBobK27 » 17 May 2021, 18:37

Here are the results of your short test script, on my system...

Code: Select all

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
lenny@lenny-Aspire-one-1-431:/media/lenny/SIDECARD/Software_Dev/Audrino_Work/IoT_work/ESPEasy/Build_Area$ bash ./Test_Script.sh 
/media/lenny/SIDECARD/Software_Dev/Audrino_Work/IoT_work/ESPEasy/Build_Area
/media/lenny/SIDECARD/Software_Dev/Audrino_Work/IoT_work/ESPEasy
------------- Dir List ------------------- Note! GIT placed pulled package one level down -----------------------------------------------------------------
lenny@lenny-Aspire-one-1-431:/media/lenny/SIDECARD/Software_Dev/Audrino_Work/IoT_work/ESPEasy/Build_Area$ ls
build_ESPeasy.sh  ESPEasy  Org_build_ESPeasy.sh  Pulled_build_ESPeasy.sh  Test_Script.sh
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
lenny@lenny-Aspire-one-1-431:/media/lenny/SIDECARD/Software_Dev/Audrino_Work/IoT_work/ESPEasy/Build_Area$ more Test_Script.sh 
====================================================================================
#!/bin/bash

SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"

cd ${SCRIPT_DIR}
cd ..
SRC=`pwd`

echo ${SCRIPT_DIR}
echo ${SRC}
exit 0
======================================================================================

TD-er
Core team member
Posts: 8644
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Major Software Damage to OS (UBUNTU 20.04)

#14 Post by TD-er » 17 May 2021, 18:52

Just added code tags to make it slightly better readable.

In this quick glance I can't see what is wrong with it.
The script is located in the tools dir, so to get the dir level where the repository resides is one up (cd ..)

So am I missing something?

JimBobK27
New user
Posts: 9
Joined: 06 May 2021, 04:31

Re: Major Software Damage to OS (UBUNTU 20.04)

#15 Post by JimBobK27 » 18 May 2021, 15:18

------------- Dir List ------------------- Note! GIT placed pulled package one level down -----------------------------------------------------------------
lenny@lenny-Aspire-one-1-431:/media/lenny/SIDECARD/Software_Dev/Audrino_Work/IoT_work/ESPEasy/Build_Area$ ls
build_ESPeasy.sh ESPEasy Org_build_ESPeasy.sh Pulled_build_ESPeasy.sh Test_Script.sh
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
If you were looking at my Dir path, you might have been fooled by the path: /ESPEasy/Build_Area.. The Build area is not within the pull from GIT.
The GREEN highlighted above is the pulled package from GIT. So, the very first error I got , requesting documents in script, was could not find requirements.txt file... ALL SYSTEMS STOP!!

I moved the script file up to the build area for easy access.. Does it have to be executed from the /tool directory within the GIT package? Could that have been the overall problem??

TD-er
Core team member
Posts: 8644
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Major Software Damage to OS (UBUNTU 20.04)

#16 Post by TD-er » 18 May 2021, 17:20

Yes I added some checks to make sure I was not making any assumptions about the path.
The build script is inside the tools dir, so I added a check to make sure we're in a known environment, thus I check for the presence of requirements.txt as that's also needed to make the Python virtual env.
And if that fails, we might as well stop before terrible things happen.

JimBobK27
New user
Posts: 9
Joined: 06 May 2021, 04:31

Re: Major Software Damage to OS (UBUNTU 20.04)

#17 Post by JimBobK27 » 21 May 2021, 01:31

Yes, me again..... tested your new script... Terminal output is below - Hopefully, readable..

Code: Select all

##############################################################################
#   Brand New "VIRGIN" test directory area
##############################################################################
lenny@lenny-Aspire-one-1-431:~$ cd /media/lenny/SIDECARD/Software_Dev/Audrino_Work/IoT_work/Test_Area
lenny@lenny-Aspire-one-1-431:/media/lenny/SIDECARD/Software_Dev/Audrino_Work/IoT_work/Test_Area$ ls
Build_ESPeasy.sh
------------------------------------------------------------------------------
lenny@lenny-Aspire-one-1-431:/media/lenny/SIDECARD/Software_Dev/Audrino_Work/IoT_work/Test_Area$ bash ./Build_ESPeasy.sh --help
File is not present in known folder.
Please checkout full ESPEasy repository using:
  git clone https://github.com/letscontrolit/ESPEasy.git
------------------------------------------------------------------------------
lenny@lenny-Aspire-one-1-431:/media/lenny/SIDECARD/Software_Dev/Audrino_Work/IoT_work/Test_Area$ git clone https://github.com/letscontrolit/ESPEasy.git
Cloning into 'ESPEasy'...
remote: Enumerating objects: 49327, done.
remote: Counting objects: 100% (2915/2915), done.
remote: Compressing objects: 100% (1027/1027), done.
remote: Total 49327 (delta 2037), reused 2596 (delta 1877), pack-reused 46412
Receiving objects: 100% (49327/49327), 348.30 MiB | 289.00 KiB/s, done.
Resolving deltas: 100% (35128/35128), done.
Updating files: 100% (3982/3982), done.
-------------------------------------------------------------------------------
lenny@lenny-Aspire-one-1-431:/media/lenny/SIDECARD/Software_Dev/Audrino_Work/IoT_work/Test_Area$ ls
Build_ESPeasy.sh  [color=#00BF40]ESPEasy[/color]
-------------------------------------------------------------------------------
lenny@lenny-Aspire-one-1-431:/media/lenny/SIDECARD/Software_Dev/Audrino_Work/IoT_work/Test_Area$ bash ./Build_ESPeasy.sh --help
File is not present in known folder.
Please checkout full ESPEasy repository using:
  git clone https://github.com/letscontrolit/ESPEasy.git
lenny@lenny-Aspire-one-1-431:/media/lenny/SIDECARD/Software_Dev/Audrino_Work/IoT_work/Test_Area$ 
#############################  END OF SHOW  ############################################
Yes, started with just the script, pulled from (or copied) GitHUB... This is how I always start working with any code project....
Error output on first try, copied example command... pasted to terminal window..
Waited for awhile, then I just did an up-arrow (linux last command function) -CR
Got error msg again....
I used 'ls' to show directory structure......
NOTE:I will be "Out of Touch" for the next three months.. I'm a NoMAD, and I need to relocate to mountains for cooler temps..
Access to internet is VERY limited... So, no rush on this effort...
Thanks for your assistance..

TD-er
Core team member
Posts: 8644
Joined: 01 Sep 2017, 22:13
Location: the Netherlands
Contact:

Re: Major Software Damage to OS (UBUNTU 20.04)

#18 Post by TD-er » 21 May 2021, 09:32

The script is checking if it is in the "tools" directory.
So after you cloned it, you should run tools/build_ESPeasy.sh

see this example:

Code: Select all

gijs@DESKTOP-QLB7N8I:~/GitHub/letscontrolit/ESPEasy$ tools/build_ESPeasy.sh --help
ESPEasy Build Script

Usage:
 build_ESPeasy.sh [-p <pull req. nr>] [-s <src dir>] [-d <description>] [--docs] [--esp32] [--esp82xx] [-h]
 build_ESPeasy.sh [-t <Git Tag>] [-s <src dir>] [-d <description>] [--docs] [--esp32] [--esp82xx] [-h]

 -h, -help,         --help          Display this help message
 -p, -pull-request, --pull-request  Build specific pull request NNNN (cannot be used with -t)
 -t, -tag,          --tag           Build based on Git tag (cannot be used with -p)
 -d, -description,  --description   Set description
 -s, -src-dir,      --src-dir       Display this help message
 --docs                             Build documentation
 --esp32                            Build all ESP32 PlatformIO project tasks
 --esp82xx                          Build all ESP8266/ESP8285 PlatformIO project tasks
   If no build target (docs/esp32/esp82xx) is given, all will be built

Examples:
  build_ESPeasy.sh -p NNNN                            Build specific pull request NNNN (cannot be used with -t)
  build_ESPeasy.sh -t mega-20210223                   Build based on Git tag (cannot be used with -p)
  build_ESPeasy.sh -d MyGreatNewFeature               Set description
  build_ESPeasy.sh -s ~/GitHub/letscontrolit/ESPEasy  Fetch source & build in set src dir


Required (Ubuntu) packages:
  sudo apt-get update
  sudo apt-get upgrade
  sudo apt install python3-minimal virtualenv build-essential zip binutils coreutils
For Python 3.8:
  sudo apt install -y software-properties-common
  sudo add-apt-repository ppa:deadsnakes/ppa
  sudo apt install -y python3.8
For Sphinx (convert):
  sudo apt install imagemagick


Summary:
  Pull Req     : 0
  Git Tag      :
  Description  :
  SRC dir      : /home/gijs/GitHub/letscontrolit/ESPEasy
  Python venv  : /home/gijs/GitHub/letscontrolit/ESPEasy/venv/python3.8
  TmpDir ZIP   : /home/gijs/GitHub/letscontrolit/ESPEasy/ESPEasy_collect_dist
  Build Docs   : 1
  Build ESP32  : 1
  Build ESP82xx: 1

      Use at your own risk!!
gijs@DESKTOP-QLB7N8I:~/GitHub/letscontrolit/ESPEasy$

Post Reply

Who is online

Users browsing this forum: No registered users and 28 guests