Become a systemd guru

[dos2unix]

Most Linux's up until about the 2011-2012 time line used something called sysV init. Some people call it rc or rc.d init.
But about this time most of the major distro's started moving over to something called systemd. Fedora was the first, and then
the same year Debian and Ubuntu joined in. Many distro's used a hybrid approach for a while. Some services still used rc.init
files and some used the systemd.servce files. Some distro's still support backwards compatibility for sysV init, but it's becoming
more and more rare. Even the tradition UNIX's like Solaris and AIX have moved to systemd.

systemd is a lot more flexible and powerful, but there is a little bt of a learning curve. However once you get used to it, it's pretty
easy to use. Just about all major packages install systemd service files by default these days. Even if you download a package from
apache, postgresql, elasticsearch, nginx, docker, or just about anywhere these days... it will have a systemd service file.

You can see what systemd service files are installed on your Linux system by using this command.

systemctl list-unit-files

In my case, I have over 560 service files installed on my computer.
Unlike NetworkManager, which doesn't like you to manually edit the connection files by hand. Systemd actually wants you to edit
the service files, and it's encouraged to learn how. I do recommend backing up any systemd service that you edit, so that if you
break it, you have something to fall back to. Service files are "usually" kept under either /lib/systemd/system or in my case /usr/lib/systemd/system. Some packages install their service files under /etc/systemd/system. It's bad enough trying to get a the distro's to put the service
files in the same place, but what makes this harder, is all the 3rd party packages that aren't maintained by a specific distro.

.. to be continued.

 

[dos2unix]

How do you know where your specific systemd service file is located? Well I have nginx installed, so the command I use to find out is...

#~> systemctl cat nginx

# /usr/lib/systemd/system/nginx.service
[Unit]
Description=The nginx HTTP and reverse proxy server
After=network-online.target remote-fs.target nss-lookup.target
Wants=network-online.target

[Service]
Type=forking
PIDFile=/run/nginx.pid
# Nginx will fail to start if /run/nginx.pid already exists but has the wrong
# SELinux context. This might happen when running [ICODE]nginx -t[/ICODE] from the cmdline.
# https://bugzilla.redhat.com/show_bug.cgi?id=1268621
ExecStartPre=/usr/bin/rm -f /run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t
ExecStart=/usr/sbin/nginx
ExecReload=/usr/sbin/nginx -s reload
KillSignal=SIGQUIT
TimeoutStopSec=5
KillMode=mixed
PrivateTmp=true

[Install]
WantedBy=multi-user.target

# /usr/lib/systemd/system/service.d/10-timeout-abort.conf
# This file is part of the systemd package.
# See https://fedoraproject.org/wiki/Changes/Shorter_Shutdown_Timer.
#
# To facilitate debugging when a service fails to stop cleanly,
# TimeoutStopFailureMode=abort is set to "crash" services that fail to stop in
# the time allotted. This will cause the service to be terminated with SIGABRT
# and a coredump to be generated.
#
# To undo this configuration change, create a mask file:
#   sudo mkdir -p /etc/systemd/system/service.d
#   sudo ln -sv /dev/null /etc/systemd/system/service.d/10-timeout-abort.conf

[Service]
TimeoutStopFailureMode=abort

# /etc/systemd/system/nginx.service.d/php-fpm.conf
[Unit]
Wants=php-fpm.service

When you run this command, the first line returned will tell you where the file is located.
That first hashed line isn't actually a part of the file. It just tells you where the file is located.
systemd service files will always end with a .service filename extension. It is a requirement for systemd.
However this makes it easy to backup a service file, because I can name the backup something like
nginx.service.bkup, and systemd will ignore it.

In the service file any lines that begin with a comment hash "#" are ignored.

One thing to be aware of, is that some packages can edit the service file for other packages.
For example I also have php installed on this system. When I installed the php-fpm packages, it automatically
edited my service file to check for the php-fpm package. This was something the old sysV init startup files could
never do.

A good service files will normally have several "stanzas" or paragraphs or sections or whatever you want to call them.
The service file above has a [Unit] section, [Service] section, [Install] section, but wait.. there's another [Service]
section and another [Unit] section all in the same systemd service file. More about that later.

.. to be continued.

 

[Aristarchus]

SMF precedes sysemd. Solaris does not use systemd.

 

[dos2unix]

Notice in the service file above, in the first [Service] section, you see some lines that look like this.

ExecStartPre=/usr/bin/rm -f /run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t
ExecStart=/usr/sbin/nginx
ExecReload=/usr/sbin/nginx -s reload

These are the commands that get ran by systemctl. There isn't really a command called "systemd".
systemd is just the service that manages all the service files and processes associated with them.
This is a good thing, because in the old days under sysV init, it was possible for a process to lose it's
PID ( process ID ). This sometimes created something called "zombie" processes. They were called
zombies, because you couldn't kill them. Normally you would use a command like "kill 7632" to kill
process 7632. But how does this work if there's no process ID? It is still possible to have a zombie process
under systemd, but it's much more difficult to do it.

The main systemctl command to know are...
enable, cat, start, stop, restart, disable and mask. There are more, but these are the basics you need to know to get started.

What if I want to start nginx?

systemctl start nginx

It see documentation out on the internet that will say something like "systemctl start nginx.service".
That's fine, it works that way, I'm just lazy and like to type fewer characters. But wait a minute, I just typed
in this command, but my service didn't start. Most likely that's because you didn't have the service enabled.
Sometimes, the service will when I use that "start" command, but it doesn't automatically restart when I reboot the
computer. How do I fix that?

systemctl enable nginx

You need to "enable" the service. Now the service should automatically start when you reboot the computer.
Sometimes you need to enable a service before the service will start. What I want to stop a service?

systemctl stop postgresql

I pulled a sneaky fast one here. I changed the service from talking about nginx to talking about postgresql.
It doesn't matter, the syntax would be the same. "systemctl stop nginx". But I wanted you to see how
the syntax of systemd works. What I used I used the systemctl start command, but I'm not sure if the service
started or not? Well usually no news is good news. If you didn't get a error, it probably started. But how can you be sure?

systemctl status httpd

I pulled another fast one on you here, now I talking about apache/httpd. But the syntax is the same for nginx and postgresql
or any service. "systemctl status postgresql" or "systemctl status -l nginx". Notice I used status -l ( lowercase L ) in this
example. I like to use the -l, because it gives a little more information that just the status command alone.

But what if I don't want to completely stop a service. My boss doesn't like any downtime. But I needed to make a change
in my service file. No problem. Make the change to your service file, and then do a..

systemctl restart mysql

That does an immediate restart of the service, and no one should really notice any downtime.

.. to be continued.

 

[dos2unix]

What if I have a service installed, but I don't want it to automatically start when I reboot my computer?

systemctl disable httpd

In some cases, disabling a service isn't enough. I once tried to disable auditd by doing this, and it would not
quit restarting on every reboot. So I ended up "masking" the service.

systemctl mask nginx

I doubt you would ever need this for nginx, but at least now if you ever need to, you know how.
To turn the mask off.. "systemctl unmask nginx".

So now you know how to list service unit files. cat them, start, enable, mask, disable and stop them.

I may do an more advanced section if there is enough interest.

 

[dos2unix]

You can also use systemd to control your desktop environment Xwindows.
Assuming you have all the packages installed in order to run a Xwindows environment,
you can type the following, and this should automatically start Xwindows next time you
reboot your computer.

systemctl set-default graphical.target

The thing about Linux, is it seems there's always a dozen different ways to do something.
You can boot into the command line mode by editing your grub boot line, or you can start
Xwindows by typing "startx". But this article is about systemd, so I'm focusing on the systemd way to do it.

systemctl set-default multi-user.target

This will cause your system to boot into the command line mode after the next reboot.
You can always revert back to the previous command to start Xwindows assuming you
haven't broken anything in the meantime.

 

[dos2unix]

Let us create our little bash shell script here.
Something like

#!/bin/bash
# A script to print out a line to the console every hour
while true; do
  # Get the current time in hours and minutes
  time=$(printf "%(%H:%M)T\n")
  # Print out the line with the timestamp
  echo "[$time] KGIII, osprey, condobloke and f33dm3bits hate Linux!"
  # Sleep for one hour (3600 seconds)
  sleep 3600
done

Save this as /home/(mysuser)/linuxhaters.sh

Now this is just a simple example. I know you could write a cron job to run your script every hour.
But that isn't the point, we want to learn how to write a systemd service file to do this for us.
As mentioned in the first post of this thread, your systemd service files coud be a couple of different places,
but for the sake of this tutorial, we will assume they are /usr/lib/systemd/system

so we will make a new service file here. We could call it linuxhaters.service
It would look something like this.

[Unit]
Description=A service for running a bash script
After=network-online.target
Requires=network-online.target

[Service]
Type=simple
#ExecStart=/bin/bash /path/to/your/bash/script.sh
ExecStart=/bin/bash /home/dos2unix/linuxhaters.sh

[Install]
WantedBy=multi-user.target

This script doesn't need a network in order to run, but I wrote the service file so it
would wait until after the network comes online before it will run. Some executables do require
certain things to be running before they will start. It won't include all of the examples here,
but this is an example of how you would do it.
The service section is pretty straightforward. Usually it just needs a "ExecStart"
with a path to your shell script. This could a python script, or a java application, a binary file
or really any program that will execute. You still need to make sure the script has executable permission
flags set.

To enable this, you just type...

systemctl enable linuxhaters

That will make the script run on the next reboot. But what if I don't want to wait until the next reboot.
I want to see who the linux haters are right now! Then just type this.

systemctl start linuxhaters

If you like type a lot of characters, you could type "systemctl start linuxhaters.service".
If you are tired of having a message pop up every hour that tells you who the haters are, you can disable it
like this.

systemctl disable linuxhaters

.. and then your service ( which is really a bash shell script ) won't run after the next reboot.
If you wanted to stop this service immediately without having to reboot, just type this...

systemctl stop linuxhaters

It may seem difficult at first, but after you've done a few times, it gets to be pretty second nature and easy.

@KGIII @osprey @f33dm3bits @Condobloke