Search code examples
bashfilesystemsubuntu-14.04data-recoverydisaster-recovery

Deleted database accidentally by a bash script, rescue please

My developer committed a huge mistake and we cannot find our mongo database anyone in the server. Rescue please!!!

He logged into the server, and saved the following shell under ~/crontab/mongod_back.sh:

enter image description here

And then he run ./mongod_back.sh, then there were lots of permission denied, then he did Ctrl+C. Then the server shut down automatically.

He tried to restart the server, then he got an grub error:

enter image description here

He then contacted AliCloud, the engineer connected the disk to another working server, so that he could check the disk. Then, he realized that some folders have gone, including /data/ where the mongodb is!!!

1) We just don't understand how the bash could destroy the disk including /data/;

2) And of course, is it possible to get the /data/ back?

PS: he did not take a snapshot of the disk before.

Solution

  • Question 1

    1. We just don't understand how the bash could destroy the disk including /data/;

    Reason: $OUT_DIR was unset

    In bash and sh comments are written as # comment, not // comment.
    The following line will have the following effects

    someVariable=someValue // not a comment
    • Assign someValue to the environment variable someVariable, but only for this command. After that command the variable will go back to its old value, which is null in this case.
    • Execute the "command" // not a comment, that is the program // with the parameters not, a, and comment. Since // is just a directory (the same as /) this will cause an error message and nothing more.

    Right now this behavior might seem strange, but you may have already used it in well known idioms like IFS= read -r line or LC_ALL=C sort.

    Looking at your script the following lines probably caused the problem:

    OUT_DIR=/data/backup/mongodb/tmp // ...
...
rm -rf $OUT_DIR/*

    I'm sorry to bring this to you, but you basically executed rm -rf /* since $OUT_DIR expanded to the empty string.

    Potential Risk On Other Systems

    Even if $OUT_DIR wasn't empty the effect could have been the same since there is a // "comment" after rm. Consider the command

    rm -rf some // thing

    This is supposed to delete the three files/directories some, //, and thing. As already pointed out // is the same directory as /.

    However, most implementations of rm on Linux have a guard for this case and won't delete / so easily. On Ubuntu you will get the following warning (don't try this at home. Would suck if your rm differs.)

    $ rm -rf //
rm: it is dangerous to operate recursively on '//' (same as '/')
rm: use --no-preserve-root to override this failsafe

    Question 2

    1. And of cause, is it possible to get the /data/ back?

    This is off-topic for StackOverflow. However, you can find many answers to this question on other stackexchange sites.

    There are recovery tools you can try, but there is no guarantee that you can restore your data if you don't have a backup.