• We did not send an email asking for donations - please read this post.

Linux.org User-created Helpful Shell Scripts

dobeshow

New Member
Joined
Jun 1, 2022
Messages
9
Reaction score
8
Credits
68
Why not with rsync?

Even if you don't have more machines to keep in sync with the thumbdrive, off the top of my head I think I remember (or maybe it's a fabricated memory) that had some extra functionality and better performance.

DISCLAIMER: This is one of the first scripts i ever made and has the potential to shoot yourself in the foot. Error handling and security certainly need improvement.

Other than that it takes one (host) or more (directories) arguments and tries to sync the directories in 'dirs' from the remote host (which you need root access for). If an older snapshot (latest) is present, it only copies changes and hard links the rest. It needs the exclude list, otherwise it you end up with tons of stuff you normally don't want to back up. If you want a fresh snapshot, delete or rename 'latest'.
No idea if it runs in shells other than zsh. It only says bash in the code section to get the pretty colors.

Bash:
#!/bin/zsh
#
#[[ -f "$COLOR_FILE" ]] && source "$COLOR_FILE"

typeset -U hosts dirs
setopt pushdsilent

EXCLUDE_FROM_FILE="${ZSH}/misc/rsync_exclude-from"

if [[ ! -z "${1}" ]]; then
    hosts="${1}"
    shift
else
    hosts=(
        wb
        pi2
        pi3
    )
fi

dirs=(
    boot
    etc
    home
    root
    var
    opt
)


if [[ ! -z "${1}" ]]; then
    while [[ ! -z "${1}" ]]; do
        dirs+=( "${1}" )
        shift
    done
fi

echo $hosts
echo $dirs

check_wd() {
    if [[ ! -d "$WORKDIR" ]]; then
        echo "${WORKDIR} does not exit"
        echo "trying to create directory"
        mkdir ${WORKDIR} || return 1
    fi
    pushd ${WORKDIR} && echo "Diving into ${WORKDIR}" || return 1
}

sync_dir() {
    BACKUP_USER=$1
    BACKUP_HOST=$2
    BACKUP_DIR=$3
    echo
    echo "syncing ${BACKUP_USER}@${BACKUP_HOST}:/${BACKUP_DIR}"
    rsync -rlD -u -ptog --info=progress2 --exclude-from=${EXCLUDE_FROM_FILE}  --link-dest="${LINK}" "${BACKUP_USER}@${BACKUP_HOST}:/${BACKUP_DIR}" "${BACKUP_ROOT}/${BACKUP_HOST}/${DATE}"
}

for BACKUP_HOST in $hosts; do
    BACKUP_ROOT=/mnt/spin0/backup/hosts
    BACKUP_SNAP=latest
    WORKDIR="${BACKUP_ROOT}/${BACKUP_HOST}"
    LINK="${WORKDIR}/latest"
    DATE=$(date "+%Y-%m-%d__%T")
    echo "Starting backup on host: ${BACKUP_HOST}"
    check_wd
    if [[ ${BACKUP_HOST} == "bw" ]]; then
    dirs+=(
        mnt/fs.local/w10.system/Users
    )
    fi
   
    for BACKUP_DIR in $dirs; do
        sync_dir "root" "${BACKUP_HOST}" "${BACKUP_DIR}" || RC=1
    done
#    RC=$?
    if [[ -d "${DATE}" ]] && [[ $RC != 0 ]]; then
        rm -f latest
        ln -s "${DATE}" latest
    else
        echo "${message[ERROR]}: ${DATE} does not exist."
        echo "something went wrong. not linking."
    fi
    echo
done

EXCLUDE_FROM_FILE="${ZSH}/misc/rsync_exclude-from"
Code:
/var/log
*/*cache*
*/games
*/Games
*/swapfile
*/.local/share/Steam*

You should end up with a structure similar to this
Code:
├─ mnt                                     
║ │  ├─ fs.local                                                                                                                          ║
║ │  │  ├─ ba0                                                                                                                            ║
║ │  │  ├─ spin0                                                                                                                          ║
║ │  │  │  ├─ backup                                                                                                                      ║
║ │  │  │  │  └─ hosts                                                                                                                    ║
║ │  │  │  │     ├─ hexenpups                                                                                                             ║
║ │  │  │  │     ├─ pi0                                                                                                                   ║
║ │  │  │  │     │  └─ 2021-04-11__08:23:34                                                                                               ║
║ │  │  │  │     ├─ pi2                                                                                                                   ║
║ │  │  │  │     │  ├─ 2021-08-14__06:48:28                                                                                               ║
║ │  │  │  │     │  ├─ 2021-08-20__01:44:26                                                                                               ║
║ │  │  │  │     │  ├─ 2022-07-14__23:48:00                                                                                               ║
║ │  │  │  │     │  ├─ 2022-07-15__05:17:08                                                                                               ║
║ │  │  │  │     │  ├─ 2022-07-28__18:47:10
 
Last edited:


Tolkem

Well-Known Member
Joined
Jan 6, 2019
Messages
1,472
Reaction score
1,195
Credits
10,700
given the file has over 87,000 lines, it would be interesting to know if there are any problems with this script since i'm obviously not going to go through the whole thing and count as the end count wouldn't be accurate anyway.
Do note that most warnings and other stuff are usually safe to ignore; it can be anything, like the kernel has some module loaded for a piece of hardware your machine does not have, so the warning msg is triggered. Errors and failures might be worth investigating, but if your system works without any "apparent" issues, then you're probably fine by ignoring those as well.
 

Dart

Member
Joined
Jan 5, 2022
Messages
65
Reaction score
60
Credits
778
rsync is a awesome tool! I use it to do a daily synchronization of the phone books of 50 PBX (IP phone servers) and pull their backups to the sync server weekly. As a matter of fact, that runs tonight.

The first operation is to pull any new phone books and dump them in a folder. The second step is to push the entire folder back out to each PBX thus updating the local phone book. The beauty of it is, it only pulls whats changed, then you end up only sending changes back. It saves a mountain of time!!!

The weekly backups of their configuration takes a bit longer. It starts it in farthest point East and work my way back West. The backups start at 11PM (UTC-5) = 5AM in (UTC+1). Since they work backwards thru the time-zones, it gives "extra" time to complete the backups.
 

gvisoc

Well-Known Member
Joined
May 29, 2020
Messages
445
Reaction score
534
Credits
4,554
This one is a just-works work in progress. It will end up in GitHub when I feel it's OK, but worth sharing.

It is inspired by this Gist by Reilly Tucker Siemens.

The script signs kernel modules with your Machine Owner Key. If you use Secure Boot with your Linux installation, you will need to do this for any module not included in the official update repositories every time you update the Kernel. Examples are plenty: NVIDIA and VirtualBox modules, v4l2loopback, and many other.

In case of modules coming with dependencies, it will sign all modules in the directory of the module to sign. This is the case, for example, of VirtualBox; vboxdrv is the sort of main module, but comes with a handful more of them.

In summary:
  • Self-sudoes itself
  • It accepts the whole list of module names that you need to sign,
  • Signs the modules according to path, filenames and module management conventions for Fedora
  • Tries to restart all failed units due to unsigned modules automatically
  • As an exit code, gives you 0 if everything went well, 1 if any signing operations fail, 2 if the service restart failed, and 3 if both error conditions happen.
For example, an usage to sign the modules for Virtualbox and the Video4Linux2Loopback (the thingie that allows you to use the OBS' virtal camera, for instance), would be:

signmodules ~/Downloads/MOK.priv ~/Downloads/MOK.der vboxdrv v4l2loopback

Output:
Code:
$ signmodules ~/Downloads/MOK.priv ~/Downloads/MOK.der vboxdrv v4l2loopback
[sudo] password for gvisoc:
Passphrase for /home/gvisoc/Downloads/MOK.priv:
[signmodules] Processing all modules for 'vboxdrv':
[signmodules]     Unpacking '/lib/modules/5.19.4-200.fc36.x86_64/extra/VirtualBox/vboxdrv.ko.xz'...
[signmodules]     Signing '/lib/modules/5.19.4-200.fc36.x86_64/extra/VirtualBox/vboxdrv.ko'...
[signmodules]     Recompressing '/lib/modules/5.19.4-200.fc36.x86_64/extra/VirtualBox/vboxdrv.ko'
[signmodules]     Unpacking '/lib/modules/5.19.4-200.fc36.x86_64/extra/VirtualBox/vboxnetadp.ko.xz'...
[signmodules]     Signing '/lib/modules/5.19.4-200.fc36.x86_64/extra/VirtualBox/vboxnetadp.ko'...
[signmodules]     Recompressing '/lib/modules/5.19.4-200.fc36.x86_64/extra/VirtualBox/vboxnetadp.ko'
[signmodules]     Unpacking '/lib/modules/5.19.4-200.fc36.x86_64/extra/VirtualBox/vboxnetflt.ko.xz'...
[signmodules]     Signing '/lib/modules/5.19.4-200.fc36.x86_64/extra/VirtualBox/vboxnetflt.ko'...
[signmodules]     Recompressing '/lib/modules/5.19.4-200.fc36.x86_64/extra/VirtualBox/vboxnetflt.ko'
[signmodules] Processing all modules for 'v4l2loopback':
[signmodules]     Unpacking '/lib/modules/5.19.4-200.fc36.x86_64/extra/v4l2loopback/v4l2loopback.ko.xz'...
[signmodules]     Signing '/lib/modules/5.19.4-200.fc36.x86_64/extra/v4l2loopback/v4l2loopback.ko'...
[signmodules]     Recompressing '/lib/modules/5.19.4-200.fc36.x86_64/extra/v4l2loopback/v4l2loopback.ko'
[signmodules] Restarting any failed units after processing modules:
[signmodules]    Requested 'vboxdrv' restart with result 0
[signmodules]    Requested 'systemd-modules-load' restart with result 0
[signmodules] Done.

The script is on GitHub and has GPLv2 license:
 
Last edited:
OP
K

KGIII

Super Moderator
Staff member
Gold Supporter
Joined
Jul 23, 2020
Messages
7,349
Reaction score
6,334
Credits
59,574
I've made some improvements to my previous script entry,

It may be valuable to edit the first post, mention it in the first post, or remove the first post - so that the emphasis is given to the new/improved version.
 

gvisoc

Well-Known Member
Joined
May 29, 2020
Messages
445
Reaction score
534
Credits
4,554
It may be valuable to edit the first post, mention it in the first post, or remove the first post - so that the emphasis is given to the new/improved version.
I will mix them both in the second and "empty-edit" the first.
 

gvisoc

Well-Known Member
Joined
May 29, 2020
Messages
445
Reaction score
534
Credits
4,554

CrazedNerd

Well-Known Member
Joined
Mar 31, 2021
Messages
774
Reaction score
305
Credits
6,606
Here's the timer script discussed earlier in the thread (provided in the link by Tolkem) but with a small modification to make it have sound when the time runs out:

Code:
#!/bin/bash
 
if [ "$#" -lt "2" ] ; then
    echo "Incorrect usage ! Example:"
    echo './countdown.sh -d  "Jun 10 2011 16:06"'
    echo 'or'
    echo './countdown.sh -m  90'
    exit 1
fi
 
now=`date +%s`
 
if [ "$1" = "-d" ] ; then
    until=`date -d "$2" +%s`
    sec_rem=`expr $until - $now`
    echo "-d"
    if [ $sec_rem -lt 1 ]; then
        echo "$2 is already history !"
    fi
fi
 
if [ "$1" = "-m" ] ; then
    until=`expr 60 \* $2`
    until=`expr $until + $now`
    sec_rem=`expr $until - $now`
    echo "-m"
    if [ $sec_rem -lt 1 ]; then
        echo "$2 is already history !"
    fi
fi
 
_R=0
_C=7
tmp=0
percent=0
total_time=0
col=`tput cols`
col=$[ $col -5 ]

while [ $sec_rem -gt 0 ]; do
    clear
    date
    let sec_rem=$sec_rem-1
    interval=$sec_rem
    seconds=`expr $interval % 60`
    interval=`expr $interval - $seconds`
    minutes=`expr $interval % 3600 / 60`
    interval=`expr $interval - $minutes`
    hours=`expr $interval % 86400 / 3600`
    interval=`expr $interval - $hours`
    days=`expr $interval % 604800 / 86400`
    interval=`expr $interval - $hours`
    weeks=`expr $interval / 604800`
    echo "----------------------------"
    echo "Seconds: " $seconds
    echo "Minutes: " $minutes
    echo "Hours:   " $hours
    echo "Days:    " $days
    echo "Weeks:   " $weeks

    echo -n "["

    progress=$[$progress + 1]
    if [ $total_time -lt 1 ] ; then
        total_time=$[$hours * 3600 + $minutes * 60 + $seconds]
    fi
    
    printf -v f "%$(echo $_R)s>" ; printf "%s\n" "${f// /=}"
    _C=7
    tput cup 7 $col

    tmp=$percent
    percent=$[$progress * 100 / $total_time]
    printf "]%d%%" $percent
    change=$[$percent - $tmp]

    _R=$[ $col * $percent / 100 ]

    sleep 1
done

printf "\n"

spd-say "wrap it up"

make it say anything you want by replacing "wrap it up" with something else, i was surprised at what great english pronunciation spd-say has :)
 

Lord Boltar

Well-Known Member
Joined
Nov 24, 2020
Messages
1,903
Reaction score
1,310
Credits
14,148
rsync is a awesome tool! I use it to do a daily synchronization of the phone books of 50 PBX (IP phone servers) and pull their backups to the sync server weekly. As a matter of fact, that runs tonight.

The first operation is to pull any new phone books and dump them in a folder. The second step is to push the entire folder back out to each PBX thus updating the local phone book. The beauty of it is, it only pulls whats changed, then you end up only sending changes back. It saves a mountain of time!!!

The weekly backups of their configuration takes a bit longer. It starts it in farthest point East and work my way back West. The backups start at 11PM (UTC-5) = 5AM in (UTC+1). Since they work backwards thru the time-zones, it gives "extra" time to complete the backups.
Grsync is a front end to rsync it is great as well - I use it for uploading to SourceForge
 

CrazedNerd

Well-Known Member
Joined
Mar 31, 2021
Messages
774
Reaction score
305
Credits
6,606
Wow, i've been wondering how to do this for months, and today i figured it out...an interactive script that finds files containing certain text:

Code:
#this will look through every file in a working directory and
#find the text files that contain certain text
read -p "Display text files that contain what? " term

grep -l -I -s "$term" *
#options mean show files containing text of file, omit binary files,
#omit error messages

this is perhaps the most useful script i've ever made, not that it's impressive yet grep is a very handy tool
 

JasKinasis

Well-Known Member
Joined
Apr 25, 2017
Messages
1,563
Reaction score
2,224
Credits
11,524
It’s been posted before, but here’s my bash-based note management tool. Unimaginatively called note:

And a note related thread here on Linux.org:

It’s quite a simple script. It uses your systems default terminal based text editor.
And it stores all of your notes in ~/.note/notes/.

If you run note without parameters, it will list all of your notes.

Create a new note, or open/edit an existing note using:
Bash:
note filename
Where filename is the name of the file you want to create/open/edit.
It has tab completion for the filenames of your notes.

If you have espeak installed, it can read notes aloud to you.

You can list notes and search for notes containing certain words.

It has a few other useful features that are documented in the man page.

It also has a configuration script called notesettings, which allows you to set a few options:
- Preferred text editor,
- Preferred pager/viewer (less is used by default, I personally use w3m. But you could use more, or lynx etc).
- Whether to keep any temporary/backup files that might be generated by the editor. (Default is to delete backup files)

And it’s licensed under GNU GPL3 (code) and GNU FDL 1.3 (documentation)

When using the terminal, I find myself making lots of notes. I got bored of having to constantly type vim /path/to/note-file when creating notes.
And got annoyed when I couldn’t remember exactly where I’d saved a note, or what I’d called the file.

So I created note to keep all of my notes in one place. It’s quite an old project of mine and it’s pretty basic/simple, but I’ve used it pretty much every day since I started developing it.

I do have a few other scripts that might be useful. Like scripts to quickly launch Google/DuckDuckGo searches from the terminal.
I have a collection of bash scripts/functions/aliases for controlling cmus (terminal based music player mp3, flac, ogg, internet radio) via cmus-remote.

Probably a few others too!

I’ll have a dig through my HD and see what I can find. Then I’ll add them to my notabug git repo, clean them up a bit and then publish them!
 
Last edited:

wizardfromoz

Administrator
Staff member
Gold Supporter
Joined
Apr 30, 2017
Messages
7,783
Reaction score
6,608
Credits
28,627
This was the first bash script I ever wrote since adopting Linux exclusively, and a little over 8 years down the track I still use it nearly every time I run updates.

Code:
#!/bin/bash
sudo apt-get update
sudo apt-get -y upgrade
sudo apt-get -y dist-upgrade
sudo apt-get -y autoremove

I call it simply

upd

If you are not sure about the distinction between line 3 and line 4 there is this

https://itsfoss.com/apt-get-upgrade-vs-dist-upgrade/

You can just as easily assign it to an alias, I just prefer the bash script. It lives in /usr/local/bin, which is in my path.

I have different ones for Debian-based, RPM-based, Arch-based and Gentoo-based, so that all 78 of my stable of Linux are covered.

I just type in

upd

enter my password, and Bob's Your Uncle.

Wiz
 

CrazedNerd

Well-Known Member
Joined
Mar 31, 2021
Messages
774
Reaction score
305
Credits
6,606
This was the first bash script I ever wrote since adopting Linux exclusively, and a little over 8 years down the track I still use it nearly every time I run updates.

Code:
#!/bin/bash
sudo apt-get update
sudo apt-get -y upgrade
sudo apt-get -y dist-upgrade
sudo apt-get -y autoremove

I call it simply

upd

If you are not sure about the distinction between line 3 and line 4 there is this

https://itsfoss.com/apt-get-upgrade-vs-dist-upgrade/

You can just as easily assign it to an alias, I just prefer the bash script. It lives in /usr/local/bin, which is in my path.

I have different ones for Debian-based, RPM-based, Arch-based and Gentoo-based, so that all 78 of my stable of Linux are covered.

I just type in

upd

enter my password, and Bob's Your Uncle.

Wiz
What does line 5 do?
 
OP
K

KGIII

Super Moderator
Staff member
Gold Supporter
Joined
Jul 23, 2020
Messages
7,349
Reaction score
6,334
Credits
59,574

CrazedNerd

Well-Known Member
Joined
Mar 31, 2021
Messages
774
Reaction score
305
Credits
6,606
You can just as easily assign it to an alias,

Yeah aliases and scripts are basically the same thing, i figure you use a script instead of an alias if the commands take up more than one line and have comments just as a formatting/visibility thing.
 

CrazedNerd

Well-Known Member
Joined
Mar 31, 2021
Messages
774
Reaction score
305
Credits
6,606
OP
K

KGIII

Super Moderator
Staff member
Gold Supporter
Joined
Jul 23, 2020
Messages
7,349
Reaction score
6,334
Credits
59,574
I've shared this before, but we can do it here:

Code:
alias update="sudo apt update && sudo apt upgrade -y && sudo apt autoremove -y && sudo apt clean -y"

That's my upgrade alias for apt-using distros.
 
OP
K

KGIII

Super Moderator
Staff member
Gold Supporter
Joined
Jul 23, 2020
Messages
7,349
Reaction score
6,334
Credits
59,574
$100 Digital Ocean Credit
Get a free VM to test out Linux!

Linux.org Hosting Donations
Consider making a donation

Members online


Top