Purchase Linux CDs / DVDs / Flash Drives at OSDisc.com

Welcome to Our Community

While Linux.org has been around for a while, we recently changed management and had to purge most of the content (including users). If you signed up before April 23rd please sign up again. Thanks!

  1. Kennedy Projects is proud to maintain and host Linux.org. Please contact us for any Linux or hosting related services if you find yourself in a jam. - Rob / KennedyProjects.com
    Dismiss Notice

Starting scripts

Discussion in 'General Linux' started by Kevin Bach, Oct 11, 2017.

  1. Kevin Bach

    Kevin Bach New Member

    Joined:
    Oct 11, 2017
    Messages:
    10
    Likes Received:
    9
    I have managed servers for years, automatically starting BASH scripts and Perl scripts to rectify certain situations. Lately, I have had a rash of these not starting at all. My first idea was the thought that everything inside these scripts needed an absolute path, as something had changed in the evolution of Linux (Fedora, my fav distro), provoking the need for a lot more absolute paths. It seemed to help with some, but apart from picking through every single line of a few hundred scripts I have written over the years (a few hundred others that were retired but saved for reference), I just don't know where to start to diagnose this problem. Many of the problems started with rc.local lines being executed, but I have forked dozens of tiny baby BASH scripts, some are service monitoring (watchdogs of daemons).

    (By the way, I hate the way the Linux OS has attempted to sabotage the rc.local concept in recent years.)



    Any thoughts, musings, critiques, criticisms, comments, questions, concerns, would be received with an open mind.
    Kevin
     
    wizardfromoz likes this.
  2. wizardfromoz

    wizardfromoz Well-Known Member

    Joined:
    Apr 30, 2017
    Messages:
    334
    Likes Received:
    342
    Hi @Kevin Bach and welcome to this venue :)

    My experience in scripting relates to those "tiny baby BASH scripts" and a couple of aliases, not server-related, and so far only Debian-based.

    However a few of our Members are very knowledgeable so hopefully you will make their acquaintance soon.:D

    On your BTW I agree. I run at home typically 70 or more Distros - Debian-based, RPM-based (incl. Fedora), Gentoo-based and Arch-based ... I am always looking to run a couple of startup scripts and the way different Distros treat rc.local leaves me scratching my head. :confused:

    Good luck, and I will follow this with interest.

    Wizard
     
  3. Kevin Bach

    Kevin Bach New Member

    Joined:
    Oct 11, 2017
    Messages:
    10
    Likes Received:
    9
    Thank you for the start on this thread. I have experimented briefly with Ubuntu because of the mass "you gotta try it," but I have recently spent more time with Debian by way of Raspbian, which is Debian for Raspberry Pi. It is the only Pi disto I've messed with the has the firmware update utility. I messed with Fedberry for a while but had to run home to Raspbian for the firmware update.

     
    wizardfromoz likes this.
  4. JasKinasis

    JasKinasis Active Member

    Joined:
    Apr 25, 2017
    Messages:
    110
    Likes Received:
    172
    Without a doubt - the biggest change to start-up over the years has been the introduction of systemd, which replaced the old init system.

    Bearing that in mind - is it possible that you need to enable /etc/rc.local compatibility in systemd? I have a feeling that it might not be enabled by default.

    To check, you could try:
    Code:
    sudo systemctl status rc-local.service
    In the output of that command there will be a field labelled "Active:"
    If rc.local compatibility is enabled, it should say something like "active (exited)" - because it should already have ran at startup. If it is disabled, it will say "failed" with some error information.

    To enable it, take a look at the following link, which contains detailed instructions on how to enable rc.local compatibility and probably explains it better than I ever could:
    https://www.linuxbabe.com/linux-server/how-to-enable-etcrc-local-with-systemd

    I suspect that the rc.local compatibility is disabled.
    If it is already enabled, I'm not sure what to suggest.... :/
     
    wizardfromoz likes this.
  5. Kevin Bach

    Kevin Bach New Member

    Joined:
    Oct 11, 2017
    Messages:
    10
    Likes Received:
    9
    I have had hit or miss reliability from rc.local after the systemd enable. And these aren't the only scripts I have had trouble with. Truly this has been the most frustrating bug I've ever tracked since it seems to have no particular source and affects so many different scripts.

    It did make me wonder if somehow default paths had been change inside the last two years.


     
  6. JasKinasis

    JasKinasis Active Member

    Joined:
    Apr 25, 2017
    Messages:
    110
    Likes Received:
    172
    Hmm.... It certainly sounds really annoying!

    I don't really run any custom scripts on startup ATM, so I haven't noticed any problems. I'm not aware of any changes to any common paths offhand either. Of course, that doesn't mean that none have been changed! :/

    There must be some commonality there though - unless there are several different problems - which would complicate things further.

    Perhaps if you can debug one or two of the shorter/smaller errant scripts - it might give some clue as to the general cause of these failures.

    From reading your other posts, you sound like you already know what you are doing, so I won't go into techniques for debugging shell scripts - I have a feeling I'd be preaching to the choir there. You've probably got much more experience than I have in that department.

    Are there any short, concrete examples that you can give to demonstrate any of the problems you're having?
     
    wizardfromoz likes this.
  7. Kevin Bach

    Kevin Bach New Member

    Joined:
    Oct 11, 2017
    Messages:
    10
    Likes Received:
    9
    I was just thinking about this again this morning. I am wondering if a symbolic link to a script file is a part of the problem. I have had some luck improving the situation with absolute path names to programs called from the scripts. Also, there has been some recent moves of programs from /bin to /usr/bin (rather annoying but maybe necessary in the big picture of Linux). I haven't been able to pull a restart today on the server that I made a change on from symbolic link to a script, to the actual script file. After 5pm maybe today, about 3 hours from posting this note.
     
  8. Kevin Bach

    Kevin Bach New Member

    Joined:
    Oct 11, 2017
    Messages:
    10
    Likes Received:
    9
    Another thought was the first line of the bash/sh scripts maybe needing to be changed
    From:
    #!/bin/bash
    #!/bin/sh
    To:
    #!/usr/bin/bash
    #!/usr/bin/sh
     
  9. atanere

    atanere Moderator
    Gold Supporter

    Joined:
    Apr 6, 2017
    Messages:
    550
    Likes Received:
    429
    When I don't know what I'm talking about (like now).... I depend on Google to help me look foolish. So, with that preface, will this syntax for the shebang help in your case?

    Code:
    #!/usr/bin/env bash
    This link says this, "The advantage of using env comes from the fact that when it calls the program, it has access to the $PATH variable set for the user; something not available within the shebang line."

    Cheers! And welcome to the site!
     
    wizardfromoz likes this.
  10. Kevin Bach

    Kevin Bach New Member

    Joined:
    Oct 11, 2017
    Messages:
    10
    Likes Received:
    9
    A VERY Good suggestion, Antanere, as I have been wondering if PATH had been compromised, in some sort of stricter enforcement of something or other. (The preface literally made me laugh out loud.) If I didn't know better, I would turn hostilities on SELinux again (ripping it yet another new one) for somehow causing me EVEN MORE trouble than it's worth. (I should attend more of those insightful Linux conferences that would bore me to tears for my limited intellect.)

    In the end game, it may be a combination of various things like this, beating down individual scripts each on their own, causing me to turn in paranoia to fix scripts already working just fine, as the new fixes will not fix or break them.
     
    atanere and wizardfromoz like this.
  11. wizardfromoz

    wizardfromoz Well-Known Member

    Joined:
    Apr 30, 2017
    Messages:
    334
    Likes Received:
    342
    Hi @Kevin Bach - I have both Fedora 26 WS beta and Fedora 26 WS available to me if that is of any assistance?

    I wondered what your output was for

    Code:
    echo $PATH
    ?

    Also, Jas said at #6

    Would it assist to show us one or two of the problem scripts?

    BTW - my output for the above command is

    Code:
    /usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/home/chris/.local/bin:/home/chris/bin
    ... on Fedora.

    I usually put my homemade scripts in /usr/local/bin/ ... which dependent on Distro may or may not be occupied already.

    Cheers

    Wizard

    Love your avatar, reminds me of Seeley Booth always calling the scientists "squints" in Bones, lol.
     
  12. Kevin Bach

    Kevin Bach New Member

    Joined:
    Oct 11, 2017
    Messages:
    10
    Likes Received:
    9
    I did do the
    #!/usr/bin/env bash
    startup and I have one bash script that never gets touched in the rc.local.

    For the previously requested "concrete" example:
    Line from rc.local on a Fedora 26 server:
    #!/usr/bin/bash
    /usr/local/etc/syncthing/rc.syncthing

    So here's the script that never gets launched:

    #!/usr/bin/env bash
    /usr/local/etc/syncthing/syncthing -upgrade
    /usr/local/etc/fork /usr/bin/nice /usr/local/etc/syncthing/syncthing -home "/usr/local/etc/syncthing/config" -no-browser -logflags="11" >> /usr/local/etc/syncthing/log.txt

    And this very short script works fine on the command line.

    The line:
    echo $PATH

    once inserted in this script produced:
    /usr/local/etc:/usr/lib64/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin

    I do notice that this is not the BASH path I have customized in /etc/profile.

    Thoughts? Musings? Confusions? Critiques? Don't be afraid to insult me. I am wide-open to comments on this disturbing, annoying, frustrating, concerning issue.
    Thanks for the comments! If I haven't already said so, this is the first forum I've posted to that actually had interactions. So thanks for that.
     
    wizardfromoz likes this.
  13. atanere

    atanere Moderator
    Gold Supporter

    Joined:
    Apr 6, 2017
    Messages:
    550
    Likes Received:
    429
    Well, again, I am out of my league here, but I'll toss out a few basics found with Google.... you will probably know instantly if anything may ring a bell in your situation.

    1. Due to Fedora/systemd, path to rc.local is no longer in /etc --- should be /etc/rc.d/rc.local
    2. rc.local must be executable (chmod + x)
    3. Possible sleep (delay) needed in rc.local to initialize network or other service(s) first
    4. Fix SELinux context (whatever that means) with this:
    Code:
    restorecon -v /etc/rc.d/rc.local
    5. Be sure rc-local is started and enabled with these:
    Code:
    systemctl enable rc-local.service
    systemctl start rc-local.service
    systemctl status rc-local.service
    These tips came from here and here.
     
    wizardfromoz likes this.
  14. Kevin Bach

    Kevin Bach New Member

    Joined:
    Oct 11, 2017
    Messages:
    10
    Likes Received:
    9
    I am continuing to go nuts over absolute path names, and I wrote a simple bash script a couple years ago to make sure rc.local (under /etc/rc.d) was accounted for with permissions and in the systemctl. And yet I remain disappointed that these same scripts I have written only work when I run them on the command line. Rc.local and cron and the "at" command just trip on them every time like they are broken. I hate it when I search (usually with my totally inept keywords) and I am the only one having a problem.

    My warm regards to every linuxhead still following this thread. I am still on the case. I have to admit my hands are tied during the day with the office in production mode. I will keep you updated.

    (PS One time about two weeks ago, I went out looking for an answer to a problem with DNF linux update, and when I found an answer and it worked, I went back to the forum page when I found the answer..... I wrote myself and forgot.)
     
    wizardfromoz and atanere like this.
  15. Kevin Bach

    Kevin Bach New Member

    Joined:
    Oct 11, 2017
    Messages:
    10
    Likes Received:
    9
    Actually today I've been experimenting with the screen command so I may start the scripts (that keep going), but I can detach from them. I am quite the amateur with "screen" apparently.
     
  16. atanere

    atanere Moderator
    Gold Supporter

    Joined:
    Apr 6, 2017
    Messages:
    550
    Likes Received:
    429
    Yes, getting Google (or others) to give good results can be trying sometimes. But I'll keep poking it to see if I can get something different for you to consider. I did some poking this afternoon after your latest posts, and a lot of the hits that come up seem to indicate systemd as a likely problem. Of course, systemd gets plenty of bashing (excuse the pun) from many quarters, no matter what the problem might be.

    Anyway, a "new thing" (to me, but maybe you've tried it)... is to create your own "unit file." If you've not tried it before, then maybe make an example from one of your scripts that is known to fail, and see if this will launch it properly. I will try to copy and paste the entire post, but the source is here in case I mangle this.

     
    wizardfromoz likes this.
  17. Kevin Bach

    Kevin Bach New Member

    Joined:
    Oct 11, 2017
    Messages:
    10
    Likes Received:
    9
    I am narrowing the scope a bit and even after identifying a problem with the rc.local the previously cited syncthing script still only runs with from the command line. I guess the good news is there may not be a pattern of scripts in general not working. Nevertheless I know that absolute path names to compiled executables and subservient scripts has helped out a bit with some problems that were never a problem in previous years. I do greatly appreciate the extreme diligence in following this thread. I will continue to update as I make breakthroughs.
     
    atanere and wizardfromoz like this.

Share This Page