Starting scripts

Kevin Bach

New Member
Joined
Oct 11, 2017
Messages
11
Reaction score
10
Credits
0
I have managed servers for years, automatically starting BASH scripts and Perl scripts to rectify certain situations. Lately, I have had a rash of these not starting at all. My first idea was the thought that everything inside these scripts needed an absolute path, as something had changed in the evolution of Linux (Fedora, my fav distro), provoking the need for a lot more absolute paths. It seemed to help with some, but apart from picking through every single line of a few hundred scripts I have written over the years (a few hundred others that were retired but saved for reference), I just don't know where to start to diagnose this problem. Many of the problems started with rc.local lines being executed, but I have forked dozens of tiny baby BASH scripts, some are service monitoring (watchdogs of daemons).

(By the way, I hate the way the Linux OS has attempted to sabotage the rc.local concept in recent years.)

Any thoughts, musings, critiques, criticisms, comments, questions, concerns, would be received with an open mind.
Kevin
 


Hi @Kevin Bach and welcome to this venue :)

My experience in scripting relates to those "tiny baby BASH scripts" and a couple of aliases, not server-related, and so far only Debian-based.

However a few of our Members are very knowledgeable so hopefully you will make their acquaintance soon.:D

On your BTW I agree. I run at home typically 70 or more Distros - Debian-based, RPM-based (incl. Fedora), Gentoo-based and Arch-based ... I am always looking to run a couple of startup scripts and the way different Distros treat rc.local leaves me scratching my head. :confused:

Good luck, and I will follow this with interest.

Wizard
 
Thank you for the start on this thread. I have experimented briefly with Ubuntu because of the mass "you gotta try it," but I have recently spent more time with Debian by way of Raspbian, which is Debian for Raspberry Pi. It is the only Pi disto I've messed with the has the firmware update utility. I messed with Fedberry for a while but had to run home to Raspbian for the firmware update.

Hi @Kevin Bach and welcome to this venue :)

My experience in scripting relates to those "tiny baby BASH scripts" and a couple of aliases, not server-related, and so far only Debian-based.

However a few of our Members are very knowledgeable so hopefully you will make their acquaintance soon.:D

On your BTW I agree. I run at home typically 70 or more Distros - Debian-based, RPM-based (incl. Fedora), Gentoo-based and Arch-based ... I am always looking to run a couple of startup scripts and the way different Distros treat rc.local leaves me scratching my head. :confused:

Good luck, and I will follow this with interest.

Wizard
 
Without a doubt - the biggest change to start-up over the years has been the introduction of systemd, which replaced the old init system.

Bearing that in mind - is it possible that you need to enable /etc/rc.local compatibility in systemd? I have a feeling that it might not be enabled by default.

To check, you could try:
Code:
sudo systemctl status rc-local.service

In the output of that command there will be a field labelled "Active:"
If rc.local compatibility is enabled, it should say something like "active (exited)" - because it should already have ran at startup. If it is disabled, it will say "failed" with some error information.

To enable it, take a look at the following link, which contains detailed instructions on how to enable rc.local compatibility and probably explains it better than I ever could:
https://www.linuxbabe.com/linux-server/how-to-enable-etcrc-local-with-systemd

I suspect that the rc.local compatibility is disabled.
If it is already enabled, I'm not sure what to suggest.... :/
 
I have had hit or miss reliability from rc.local after the systemd enable. And these aren't the only scripts I have had trouble with. Truly this has been the most frustrating bug I've ever tracked since it seems to have no particular source and affects so many different scripts.

It did make me wonder if somehow default paths had been change inside the last two years.


Without a doubt - the biggest change to start-up over the years has been the introduction of systemd, which replaced the old init system.

Bearing that in mind - is it possible that you need to enable /etc/rc.local compatibility in systemd? I have a feeling that it might not be enabled by default.

To check, you could try:
Code:
sudo systemctl status rc-local.service

In the output of that command there will be a field labelled "Active:"
If rc.local compatibility is enabled, it should say something like "active (exited)" - because it should already have ran at startup. If it is disabled, it will say "failed" with some error information.

To enable it, take a look at the following link, which contains detailed instructions on how to enable rc.local compatibility and probably explains it better than I ever could:
https://www.linuxbabe.com/linux-server/how-to-enable-etcrc-local-with-systemd

I suspect that the rc.local compatibility is disabled.
If it is already enabled, I'm not sure what to suggest.... :/
 
Hmm.... It certainly sounds really annoying!

I don't really run any custom scripts on startup ATM, so I haven't noticed any problems. I'm not aware of any changes to any common paths offhand either. Of course, that doesn't mean that none have been changed! :/

There must be some commonality there though - unless there are several different problems - which would complicate things further.

Perhaps if you can debug one or two of the shorter/smaller errant scripts - it might give some clue as to the general cause of these failures.

From reading your other posts, you sound like you already know what you are doing, so I won't go into techniques for debugging shell scripts - I have a feeling I'd be preaching to the choir there. You've probably got much more experience than I have in that department.

Are there any short, concrete examples that you can give to demonstrate any of the problems you're having?
 
I was just thinking about this again this morning. I am wondering if a symbolic link to a script file is a part of the problem. I have had some luck improving the situation with absolute path names to programs called from the scripts. Also, there has been some recent moves of programs from /bin to /usr/bin (rather annoying but maybe necessary in the big picture of Linux). I haven't been able to pull a restart today on the server that I made a change on from symbolic link to a script, to the actual script file. After 5pm maybe today, about 3 hours from posting this note.
 
Another thought was the first line of the bash/sh scripts maybe needing to be changed
From:
#!/bin/bash
#!/bin/sh
To:
#!/usr/bin/bash
#!/usr/bin/sh
 
Another thought was the first line of the bash/sh scripts maybe needing to be changed
From:
#!/bin/bash
#!/bin/sh
To:
#!/usr/bin/bash
#!/usr/bin/sh

When I don't know what I'm talking about (like now).... I depend on Google to help me look foolish. So, with that preface, will this syntax for the shebang help in your case?

Code:
#!/usr/bin/env bash

This link says this, "The advantage of using env comes from the fact that when it calls the program, it has access to the $PATH variable set for the user; something not available within the shebang line."

Cheers! And welcome to the site!
 
A VERY Good suggestion, Antanere, as I have been wondering if PATH had been compromised, in some sort of stricter enforcement of something or other. (The preface literally made me laugh out loud.) If I didn't know better, I would turn hostilities on SELinux again (ripping it yet another new one) for somehow causing me EVEN MORE trouble than it's worth. (I should attend more of those insightful Linux conferences that would bore me to tears for my limited intellect.)

In the end game, it may be a combination of various things like this, beating down individual scripts each on their own, causing me to turn in paranoia to fix scripts already working just fine, as the new fixes will not fix or break them.
 
Hi @Kevin Bach - I have both Fedora 26 WS beta and Fedora 26 WS available to me if that is of any assistance?

I wondered what your output was for

Code:
echo $PATH
?

Also, Jas said at #6

...Are there any short, concrete examples that you can give to demonstrate any of the problems you're having?...

Would it assist to show us one or two of the problem scripts?

BTW - my output for the above command is

Code:
/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/home/chris/.local/bin:/home/chris/bin

... on Fedora.

I usually put my homemade scripts in /usr/local/bin/ ... which dependent on Distro may or may not be occupied already.

Cheers

Wizard

Love your avatar, reminds me of Seeley Booth always calling the scientists "squints" in Bones, lol.
 
I did do the
#!/usr/bin/env bash
startup and I have one bash script that never gets touched in the rc.local.

For the previously requested "concrete" example:
Line from rc.local on a Fedora 26 server:
#!/usr/bin/bash
/usr/local/etc/syncthing/rc.syncthing

So here's the script that never gets launched:

#!/usr/bin/env bash
/usr/local/etc/syncthing/syncthing -upgrade
/usr/local/etc/fork /usr/bin/nice /usr/local/etc/syncthing/syncthing -home "/usr/local/etc/syncthing/config" -no-browser -logflags="11" >> /usr/local/etc/syncthing/log.txt

And this very short script works fine on the command line.

The line:
echo $PATH

once inserted in this script produced:
/usr/local/etc:/usr/lib64/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin

I do notice that this is not the BASH path I have customized in /etc/profile.

Thoughts? Musings? Confusions? Critiques? Don't be afraid to insult me. I am wide-open to comments on this disturbing, annoying, frustrating, concerning issue.
Thanks for the comments! If I haven't already said so, this is the first forum I've posted to that actually had interactions. So thanks for that.
 
Well, again, I am out of my league here, but I'll toss out a few basics found with Google.... you will probably know instantly if anything may ring a bell in your situation.

1. Due to Fedora/systemd, path to rc.local is no longer in /etc --- should be /etc/rc.d/rc.local
2. rc.local must be executable (chmod + x)
3. Possible sleep (delay) needed in rc.local to initialize network or other service(s) first
4. Fix SELinux context (whatever that means) with this:
Code:
restorecon -v /etc/rc.d/rc.local
5. Be sure rc-local is started and enabled with these:
Code:
systemctl enable rc-local.service
systemctl start rc-local.service
systemctl status rc-local.service

These tips came from here and here.
 
I am continuing to go nuts over absolute path names, and I wrote a simple bash script a couple years ago to make sure rc.local (under /etc/rc.d) was accounted for with permissions and in the systemctl. And yet I remain disappointed that these same scripts I have written only work when I run them on the command line. Rc.local and cron and the "at" command just trip on them every time like they are broken. I hate it when I search (usually with my totally inept keywords) and I am the only one having a problem.

My warm regards to every linuxhead still following this thread. I am still on the case. I have to admit my hands are tied during the day with the office in production mode. I will keep you updated.

(PS One time about two weeks ago, I went out looking for an answer to a problem with DNF linux update, and when I found an answer and it worked, I went back to the forum page when I found the answer..... I wrote myself and forgot.)
 
Actually today I've been experimenting with the screen command so I may start the scripts (that keep going), but I can detach from them. I am quite the amateur with "screen" apparently.
 
I am continuing to go nuts over absolute path names, and I wrote a simple bash script a couple years ago to make sure rc.local (under /etc/rc.d) was accounted for with permissions and in the systemctl. And yet I remain disappointed that these same scripts I have written only work when I run them on the command line. Rc.local and cron and the "at" command just trip on them every time like they are broken. I hate it when I search (usually with my totally inept keywords) and I am the only one having a problem.

My warm regards to every linuxhead still following this thread. I am still on the case. I have to admit my hands are tied during the day with the office in production mode. I will keep you updated.

(PS One time about two weeks ago, I went out looking for an answer to a problem with DNF linux update, and when I found an answer and it worked, I went back to the forum page when I found the answer..... I wrote myself and forgot.)

Yes, getting Google (or others) to give good results can be trying sometimes. But I'll keep poking it to see if I can get something different for you to consider. I did some poking this afternoon after your latest posts, and a lot of the hits that come up seem to indicate systemd as a likely problem. Of course, systemd gets plenty of bashing (excuse the pun) from many quarters, no matter what the problem might be.

Anyway, a "new thing" (to me, but maybe you've tried it)... is to create your own "unit file." If you've not tried it before, then maybe make an example from one of your scripts that is known to fail, and see if this will launch it properly. I will try to copy and paste the entire post, but the source is here in case I mangle this.

You can create a Systemd unit file in /usr/lib/systemd/system/<service_name>.service. Here is a template:
Code:
[Unit]
Description=<description_string>

[Service]
WorkingDirectory=<working_directory>
Type=forking
ExecStart=/bin/bash <absolute_path_to_script>
KillMode=process

[Install]
WantedBy=multi-user.target
Replace anything in the angle brackets with your specific information. The 'WantedBy=multi-user.target' is the magic that tells Systemd to run your script on each start.

On the command line, tell Systemd to enable your service:

systemctl enable <service_name>.service
The next time you reboot your script should be run. Logs will be written to /var/log/messages.

Fedora has some basic documentation on unit files: Systemd unit files
 
I am narrowing the scope a bit and even after identifying a problem with the rc.local the previously cited syncthing script still only runs with from the command line. I guess the good news is there may not be a pattern of scripts in general not working. Nevertheless I know that absolute path names to compiled executables and subservient scripts has helped out a bit with some problems that were never a problem in previous years. I do greatly appreciate the extreme diligence in following this thread. I will continue to update as I make breakthroughs.
 

Members online


Top