Consistent Issues Over Years With Linux SOLVED

Somehow we took that left turn at Albuquerque (Bugs Bunny reference) ...
When that happens I almost always have to do one of 2 things.
Either edit and fix the uuid in the /etc/fstab file with the correct uuid from the output of blkid <or> comment out the whole partition string that doesn't exist from looking at all partitions in gparted.
Which btw, is even harder to fix on a triple booted Linux box.

I am suspicious that this is something that the installer is not getting right during the installation. The installer appears to somehow mis-configure all of the other swap partitions during a regeneration cycle of the other swaps on a rig that has other Linux os installed.

The other issue is boot times after fresh installs that take up to 3 to 5 min's before log in.
Is systemd to blame?
Is this another Nvidia strikes again issue?
In you case, both are symptoms of the same error. Somehow you are getting an /etc/fstab out of your install - and apparently for a very long time - that does not equate to your actual installed hardware set; specifically your hard drives. That's what we need to dig into and that, more than anything that systemd does/does not, is the root cause of your boot times and your need to always "fix" fstab.

If the installed OS is looking at fstab, which it does, to mount drives and it doesn't find them, it's going to look next throughout the actual installed drives to find something to mount. That takes quite a while.

As I said before, it looks like you and I have the same install process, yet we have different results.

So first question:
Do you use UEFI or legacy BIOS?

Second question:
Are any of your "triple boot" Linux installs the result of a clone?
 


keep in mind that I don't have my Linux box here with me....so I am unable to experiment with my suggestions.....

I just spotted jglen's post above mine.....yes that is a good start.

Also, do you have systemctl installed ?

THIS is a bit dated but perhaps worth a read just to get the thought processes started....

Just as an "out there" thought.....can you ascertain if there any logs being created/run etc at boot up time...?
 
I really don't want to get rid of Mint because it's been very stable.

How long does your Mint take to boot?
I can't say because I don't have Linux Mint installed anymore since the new releases of Ubuntu 20.04 Focal Fossa was released.

From what I remember it was under 2 minutes and probably closer to 1.5 minutes from a cold start.

Take into account my computers are from 2010 and run dual core E7500 (2.93GHz) Core 2 Duo processors and 4.0GB DDR3 memory and mechanical hard drives.

-------------------------------------------------------------------------------------------------------------------

I don't see why everyone is so concerned with boot times never have and never will.

I get up in the morning power on my computer then make my coffee and go to the bathroom.

After the bathroom I stop and grab a cup of coffee and go to the computer which is ready to use.

Yeah I know too much information but it needed to be explained.

---------------------------------------------------------------------------

@Alexzee Post# 18 Any suggestions where to begin.

@JasKinasis posted in post #8 that he turned off and or removed some systemd packages that were not needed and that seemed to speed up his boot time.

It would be worth a look at imo.
 
Last edited by a moderator:
Somehow we took that left turn at Albuquerque (Bugs Bunny reference) ...

In you case, both are symptoms of the same error. Somehow you are getting an /etc/fstab out of your install - and apparently for a very long time - that does not equate to your actual installed hardware set; specifically your hard drives. That's what we need to dig into and that, more than anything that systemd does/does not, is the root cause of your boot times and your need to always "fix" fstab.

If the installed OS is looking at fstab, which it does, to mount drives and it doesn't find them, it's going to look next throughout the actual installed drives to find something to mount. That takes quite a while.

As I said before, it looks like you and I have the same install process, yet we have different results.

So first question:
Do you use UEFI or legacy BIOS?

Second question:
Are any of your "triple boot" Linux installs the result of a clone?
Two of my machines are UEFI but I have that disabled.
No, none of my installations are a result of a clone:-
 
keep in mind that I don't have my Linux box here with me....so I am unable to experiment with my suggestions.....

I just spotted jglen's post above mine.....yes that is a good start.

Also, do you have systemctl installed ?

THIS is a bit dated but perhaps worth a read just to get the thought processes started....

Just as an "out there" thought.....can you ascertain if there any logs being created/run etc at boot up time...?
I'll let you know in the morning if systemctl is installed.
I'm not on that machine tonight.
 
I can't say because I don't have Linux Mint installed anymore since the new releases of Ubuntu 20.04 Focal Fossa was released.

From what I remember it was under 2 minutes and probably closer to 1.5 minutes from a cold start.

Take into account my computers are from 2010 and run dual core E7500 (2.93GHz) Core 2 Duo processors and 4.0GB DDR3 memory and mechanical hard drives.

-------------------------------------------------------------------------------------------------------------------

I don't see why everyone is so concerned with boot times never have and never will.

I get up in the morning power on my computer then make my coffee and go to the bathroom.

After the bathroom I stop and grab a cup of coffee and go to the computer which is ready to use.

Yeah I know too much information but it needed to be explained.

---------------------------------------------------------------------------

@Alexzee Post# 18 Any suggestions where to begin.

@JasKinasis posted in post #8 that he turned off and or removed some systemd packages that were not needed and that seemed to speed up his boot time.

It would be worth a look at imo.
I don't know how to turn off systemd things.
I'll have to do some searching.
 
Jas @JasKinasis can probably elaborate a bit on what he did?

Alex, couple of quick questions -

  1. How much RAM?
  2. Do you have your setup such that when it boots you get a splash screen with maybe little green circles turning green to white to green, or have all the startup sequence spooling down the screen?
  3. Is there a reason for having the UEFI disabled?
I have to go for a little, but when back, I'll tell you how I run 60 to 100 Linux without any Swap partition. It may be slow, still, with 1 and a half arms, lol.

Chris
 
Jas @JasKinasis can probably elaborate a bit on what he did?

Alex, couple of quick questions -

  1. How much RAM?
  2. Do you have your setup such that when it boots you get a splash screen with maybe little green circles turning green to white to green, or have all the startup sequence spooling down the screen?
  3. Is there a reason for having the UEFI disabled?
I have to go for a little, but when back, I'll tell you how I run 60 to 100 Linux without any Swap partition. It may be slow, still, with 1 and a half arms, lol.

Chris
16 G's of RAM

I've got a white circle with the letters L and M that are green. The background spins till it boots to the desktop.

I've got UEFI disabled because this was a custom built rig and I didn't want to run into any issues.
 
16 G's of RAM
With 16GB of memory you don't need a swap file unless you allow your computer to go into hibernation.

Hibernation is usually not recommended because of known issues with waking up / coming out of hibernation.
 
Last edited by a moderator:
For me it was a case of looking at the output of:
Code:
systemd-analyze blame
and
Code:
systemd-analyze critical-chain
I took a look at the processes that were taking the longest and found that several of them were either NOT installed on my machine, or they WERE installed, but were excess to my requirements.
So for any that weren't installed - I manually removed all references to them from the systemd configs, which took a little while.

And with the services that were excess to requirement - I simply uninstalled them using apt and purging any configs - but I only did that after I'd checked that no other packages relied on any of them.

Using apt’s -s option (simulate) helped me to work that out. If removing any of the packages would have a larger impact on the system, a dry-run using -s would list any other packages that would also be removed.
Those that were safe to remove, were removed.

exim4 was the biggest bottleneck of the installed packages. I have no idea how exim4 got installed on my machine. I definitely haven't manually installed it. I have no need for a mail-transport daemon. Perhaps it was installed by something I've subsequently uninstalled? Or perhaps I'd accidentally installed it when I've used the --install-suggests option whilst installing something...IDK!

Either way, exim4 is gone now. I forgot what the other services were that I removed.
But after removing those few packages and any redundant/residual systemd configs - my boot times are much more acceptable. As can be seen from the screenshot in my previous post.

Removal of the unwanted configs was the tricky part. Once I identified things for removal, I tested by commenting out entries in configs, or moving entire config files out of /etc/systemd/.
After removing the necessary entries/files - I did a reboot.
For me the first reboot went smoothly and much more quickly.
And after rebooting and logging in - I ran the systemd-analyze, systemd-analyze blame and systemd-analyze --critical-chain commands and verified that all of the serious bottlenecks were no longer in the start-up.
So I guess I either did things correctly, or I got lucky! Ha ha!
 
If you wish to, you can edit /etc/default/grub, so that an existing line appears as follows

Code:
GRUB_CMDLINE_LINUX_DEFAULT="noquiet nosplash"

... or comment out the existing line (place a hash at beginning) so that you can revert easily if you wish, then add an additional line that simply says as I have in the code fields.

Then run update-grub and reboot.

Instead of the previous circles &c you will have output scroll past with a lot of OK's and maybe a warning or two, or even an error (usually in highlighted text).

With 16 GB (as I have found going from 8 GB) the scrolled output may move a bit too quickly for you to read properly.

In that case, in the installed distro environment, you can type at Terminal eg

Code:
dmesg | grep -i failed

# or

dmesg | grep -i warning

# or

dmesg | grep -i "start job"
a

That last one in particular is good for finding what is causing those 1 minute 30 seconds holdups - if they are swap-related, it is because Linux is looking for a previously-indicated suspend/resume device.

More tomorrow

Wiz
 
Here's all the things running on my system.

systemd-analyze blame
31.570s apt-daily.service
5.135s qemu-kvm.service
397ms apt-daily-upgrade.service
334ms systemd-logind.service
303ms vboxdrv.service
294ms lightdm.service
293ms plymouth-quit-wait.service
219ms dev-nvme0n1p2.device
162ms systemd-journal-flush.service
132ms udisks2.service
118ms snapd.service
78ms networking.service
75ms ufw.service
67ms NetworkManager.service
65ms apparmor.service
62ms networkd-dispatcher.service
59ms ubuntu-system-adjustments.service
57ms systemd-journald.service
55ms ModemManager.service
55ms upower.service
47ms systemd-udev-trigger.service
45ms keyboard-setup.service
42ms accounts-daemon.service

systemd-analyze critical-chain
The time after the unit is active or started is printed after the "@" character.
The time the unit takes to start is printed after the "+" character.

graphical.target @1min 30.603s
└─multi-user.target @1min 30.603s
└─getty.target @1min 30.603s
└─[email protected] @1min 30.603s
└─system-getty.slice @1min 30.602s
└─setvtrgb.service @1min 30.601s +1ms
└─systemd-user-sessions.service @1min 30.304s +2ms
└─network.target @1min 30.303s
└─NetworkManager.service @1min 30.235s +67ms
└─dbus.service @1min 30.230s
└─basic.target @1min 30.218s
└─sockets.target @1min 30.218s
└─snapd.socket @1min 30.218s +522us
└─sysinit.target @1min 30.210s
└─systemd-timesyncd.service @348ms +22ms
└─systemd-tmpfiles-setup.service @315ms +19ms
└─systemd-journal-flush.service @151ms +162ms
└─systemd-journald.service @93ms +57ms
└─systemd-journald-dev-log.socket @93ms
└─system.slice @89ms
lines 1-23

So far system-getty.slice, setvtrgb.service, getty.target, multi-user.target, system.slice and snapd.socket is not installed.

What service should I get rid of?
 
Last edited:
If you wish to, you can edit /etc/default/grub, so that an existing line appears as follows

Code:
GRUB_CMDLINE_LINUX_DEFAULT="noquiet nosplash"

... or comment out the existing line (place a hash at beginning) so that you can revert easily if you wish, then add an additional line that simply says as I have in the code fields.

Then run update-grub and reboot.

Instead of the previous circles &c you will have output scroll past with a lot of OK's and maybe a warning or two, or even an error (usually in highlighted text).

With 16 GB (as I have found going from 8 GB) the scrolled output may move a bit too quickly for you to read properly.

In that case, in the installed distro environment, you can type at Terminal eg

Code:
dmesg | grep -i failed

# or

dmesg | grep -i warning

# or

dmesg | grep -i "start job"
a

That last one in particular is good for finding what is causing those 1 minute 30 seconds holdups - if they are swap-related, it is because Linux is looking for a previously-indicated suspend/resume device.

More tomorrow

Wiz
dmesg | grep -i failed
[ 2.328916] ahci 0000:07:00.0: failed stop FIS RX (-16)
[ 2.808903] ata2.00: failed to enable Sense Data Reporting, Emask 0x1
[ 2.811162] ata2.00: failed to enable Sense Data Reporting, Emask 0x1
[ 5.536522] thermal thermal_zone0: failed to read out thermal zone (-61)
[ 10.358428] ccp 0000:0b:00.1: SEV: failed to get status. Error: 0x0
[ 95.418906] vboxdrv: module verification failed: signature and/or required key missing - tainting kernel

Looks like greek to me mate:-
 
For me it was a case of looking at the output of:
Code:
systemd-analyze blame
and
Code:
systemd-analyze critical-chain
I took a look at the processes that were taking the longest and found that several of them were either NOT installed on my machine, or they WERE installed, but were excess to my requirements.
So for any that weren't installed - I manually removed all references to them from the systemd configs, which took a little while.

And with the services that were excess to requirement - I simply uninstalled them using apt and purging any configs - but I only did that after I'd checked that no other packages relied on any of them.

Using apt’s -s option (simulate) helped me to work that out. If removing any of the packages would have a larger impact on the system, a dry-run using -s would list any other packages that would also be removed.
Those that were safe to remove, were removed.

exim4 was the biggest bottleneck of the installed packages. I have no idea how exim4 got installed on my machine. I definitely haven't manually installed it. I have no need for a mail-transport daemon. Perhaps it was installed by something I've subsequently uninstalled? Or perhaps I'd accidentally installed it when I've used the --install-suggests option whilst installing something...IDK!

Either way, exim4 is gone now. I forgot what the other services were that I removed.
But after removing those few packages and any redundant/residual systemd configs - my boot times are much more acceptable. As can be seen from the screenshot in my previous post.

Removal of the unwanted configs was the tricky part. Once I identified things for removal, I tested by commenting out entries in configs, or moving entire config files out of /etc/systemd/.
After removing the necessary entries/files - I did a reboot.
For me the first reboot went smoothly and much more quickly.
And after rebooting and logging in - I ran the systemd-analyze, systemd-analyze blame and systemd-analyze --critical-chain commands and verified that all of the serious bottlenecks were no longer in the start-up.
So I guess I either did things correctly, or I got lucky! Ha ha!
Thanks!

I'm in the process of going through those lists from the commands you posted to see what's installed.

BTW, systemctl is not installed and when I tried to install it the commandline returned that the pkg could not be found.

sudo apt install systemctl
[sudo]
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package systemctl
System-Product-Name:~$ dpkg -L systemctl
dpkg-query: package 'systemctl' is not installed
Use dpkg --info (= dpkg-deb --info) to examine archive files,
and dpkg --contents (= dpkg-deb --contents) to list their contents.
 
Here's all the things running on my system.

systemd-analyze blame
31.570s apt-daily.service
5.135s qemu-kvm.service
397ms apt-daily-upgrade.service
334ms systemd-logind.service
303ms vboxdrv.service
294ms lightdm.service
293ms plymouth-quit-wait.service
219ms dev-nvme0n1p2.device
162ms systemd-journal-flush.service
132ms udisks2.service
118ms snapd.service
78ms networking.service
75ms ufw.service
67ms NetworkManager.service
65ms apparmor.service
62ms networkd-dispatcher.service
59ms ubuntu-system-adjustments.service
57ms systemd-journald.service
55ms ModemManager.service
55ms upower.service
47ms systemd-udev-trigger.service
45ms keyboard-setup.service
42ms accounts-daemon.service

systemd-analyze critical-chain
The time after the unit is active or started is printed after the "@" character.
The time the unit takes to start is printed after the "+" character.

graphical.target @1min 30.603s
└─multi-user.target @1min 30.603s
└─getty.target @1min 30.603s
└─[email protected] @1min 30.603s
└─system-getty.slice @1min 30.602s
└─setvtrgb.service @1min 30.601s +1ms
└─systemd-user-sessions.service @1min 30.304s +2ms
└─network.target @1min 30.303s
└─NetworkManager.service @1min 30.235s +67ms
└─dbus.service @1min 30.230s
└─basic.target @1min 30.218s
└─sockets.target @1min 30.218s
└─snapd.socket @1min 30.218s +522us
└─sysinit.target @1min 30.210s
└─systemd-timesyncd.service @348ms +22ms
└─systemd-tmpfiles-setup.service @315ms +19ms
└─systemd-journal-flush.service @151ms +162ms
└─systemd-journald.service @93ms +57ms
└─systemd-journald-dev-log.socket @93ms
└─system.slice @89ms
lines 1-23

So far system-getty.slice, setvtrgb.service, getty.target, multi-user.target, system.slice and snapd.socket is not installed.

What service should I get rid of?


From reading the critical chain - system.slice to systemd-timesyncd.service are up in a few hundred milliseconds.
Then sysinit.target takes 1 minute and 30 seconds or so.
So that's one bottleneck. Between timesyncd.service and sysinit.target is a period of 1min 30.
Everything after sysinit.target is up in an additional four hundred milliseconds, or so.

So it looks like sysinit.target is the first problem. Something there is causing a problem.

Looking at the output of the blame command - the only thing taking a long time is the apt-daily service, which is taking up almost 32 seconds.
This is basically updating your package lists and may even be upgrading software behind the scenes. You might want to keep this enabled, if you want system upgrades to be done automatically in the background.

Personally, I prefer to update/upgrade manually - so I've completely masked that service using systemctl mask, which aliases it to /dev/null in the init - which will prevent it from running. So I haven't uninstalled it, but I've completely disabled it.

Regarding systemctl - FYI, it's part of the systemd package. So if you're running a distro that uses systemd - it should already be installed!
It needs to be ran as root, so it might be hidden from normal, unprivileged users. So use sudo to run it as root.
 
For a really head-scratching read

Code:
man systemctl

apt-daily.service acts as a trigger to call a script apt.systemd.daily which is housed in /usr/lib/apt/

This then runs unattended-upgrades, which can run at startup, at shutdown, or at a set interval. If it is at startup or shutdown, it can cause significant delays to those processes.

I have programmatically disabled u-a on all my debian-based distros, but I am with Jas and prefer to update/upgrade when I feel like it.

I run 12 Linux Mints in my stable currently, and startup times are 27 - 32 seconds, and shutdown times only a few seconds.

You can check for the presence of unattended-upgrades with logs in /var/log/unattended-upgrades/ or for a daily presence with

Code:
dmesg | grep -i unattended

Actually, I tell a lie - I have just spotted an unattended-upgrades-shutdown.log in Robolinux, so I'll slip over there and sort that bugger out. :)

I'll come back later with some screenies related to swap.

Later

Wizard
 
From reading the critical chain - system.slice to systemd-timesyncd.service are up in a few hundred milliseconds.
Then sysinit.target takes 1 minute and 30 seconds or so.
So that's one bottleneck. Between timesyncd.service and sysinit.target is a period of 1min 30.
Everything after sysinit.target is up in an additional four hundred milliseconds, or so.

So it looks like sysinit.target is the first problem. Something there is causing a problem.

Looking at the output of the blame command - the only thing taking a long time is the apt-daily service, which is taking up almost 32 seconds.
This is basically updating your package lists and may even be upgrading software behind the scenes. You might want to keep this enabled, if you want system upgrades to be done automatically in the background.

Personally, I prefer to update/upgrade manually - so I've completely masked that service using systemctl mask, which aliases it to /dev/null in the init - which will prevent it from running. So I haven't uninstalled it, but I've completely disabled it.

Regarding systemctl - FYI, it's part of the systemd package. So if you're running a distro that uses systemd - it should already be installed!
It needs to be ran as root, so it might be hidden from normal, unprivileged users. So use sudo to run it as root.
I appreciate you reading through the output and giving me details so thank you for that.

If this were your machine how would you deal with sysinit.target?

You said that system.slice to systemd-timesyncd.service is a bottleneck:-
Not sure how to address that either.
This is uncharted territory for me.

Is it the apt-daily service running to control updates and upgrades automatically to the system?
 
For a really head-scratching read

Code:
man systemctl

apt-daily.service acts as a trigger to call a script apt.systemd.daily which is housed in /usr/lib/apt/

This then runs unattended-upgrades, which can run at startup, at shutdown, or at a set interval. If it is at startup or shutdown, it can cause significant delays to those processes.

I have programmatically disabled u-a on all my debian-based distros, but I am with Jas and prefer to update/upgrade when I feel like it.

I run 12 Linux Mints in my stable currently, and startup times are 27 - 32 seconds, and shutdown times only a few seconds.

You can check for the presence of unattended-upgrades with logs in /var/log/unattended-upgrades/ or for a daily presence with

Code:
dmesg | grep -i unattended

Actually, I tell a lie - I have just spotted an unattended-upgrades-shutdown.log in Robolinux, so I'll slip over there and sort that bugger out. :)

I'll come back later with some screenies related to swap.

Later

Wizard
Thanks for explaining about what apt-daily service does.
How would I disable that service?

Running dmesg | grep -i unattended return nothing at all.

Going to the root file system and following the path or using the command-line to look at /var/log/unattended-upgrades/ returns no such file or directory:-
 
Running Systemctl looks like processes and information about the system.
Never used it before so I'm not familuar with it so I'll read the man page.
Thanks Wizard-:)

Screenshot_2020-05-22_14-28-01.png
 
When that happens I almost always have to do one of 2 things.
Either edit and fix the uuid in the /etc/fstab file with the correct uuid from the output of blkid <or> comment out the whole partition string that doesn't exist from looking at all partitions in gparted.
Which btw, is even harder to fix on a triple booted Linux box.

I am suspicious that this is something that the installer is not getting right during the installation. The installer appears to somehow mis-configure all of the other swap partitions during a regeneration cycle of the other swaps on a rig that has other Linux os installed.

The other issue is boot times after fresh installs that take up to 3 to 5 min's before log in.
Is systemd to blame?
Is this another Nvidia strikes again issue?

I stopped using UUIDs long ago. Mostly because UUID changes every time you make the slightest changes to the disks. Even flagging the device/the partition changes UUID and everything goes to hell. So I started using more clear partition labels (my fstab in the code tag):

Code:
/dev/sdb1    /             ext4          rw,relatime    0 1
/dev/sdc1 /media/1000GB ntfs-3g defaults,auto,nofail,x-systemd.device-timeout=10,uid=1000,gid=1000,umask=000 0 0
/dev/sda2 /media/390GB ntfs-3g defaults,auto,nofail,x-systemd.device-timeout=10,uid=1000,gid=1000,umask=000 0 0

Now, with this partition identification, it doesn't matter what other changes I make, for as long as these changes don't include rearranging of the devices.
You can get the device ID number (sdXY) by using the same command you use for UUID:

Code:
[rado@arch]: ~>$ sudo blkid
/dev/sda1: LABEL="Windows 7" BLOCK_SIZE="512" UUID="14562BB8562B9A0E" TYPE="ntfs" PARTUUID="d535e586-01"
/dev/sda2: LABEL="390GB" BLOCK_SIZE="512" UUID="547F53D87B122FBA" TYPE="ntfs" PARTUUID="d535e586-02"
/dev/sdb1: UUID="27edf17c-dd78-45d4-a426-9ab60da144bf" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="bfb8c7a5-01"
/dev/sdc1: LABEL="1000GB" BLOCK_SIZE="512" UUID="092138B830C24D80" TYPE="ntfs" PARTUUID="cd3e63b9-01"
/dev/sdd1: LABEL_FATBOOT="multiboot" LABEL="multiboot" UUID="075C-78A9" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="c03c7b4a-01"

As for your login problem - for the 4 years of using Linux (first Mint 18.3 and now Arch) I've never had such a problem, so this must be something local - related to your system only. Both Mint and Arch use systemd, so I doubt that's the cause. And on both systems I had (and still have) the proprietary driver of nvidia, so I doubt that's the cause either.
 


Top