Devuan Installation Unexpected Inconsistency You Will have To Run fsck Manually {Solved}

[Alexzee]

I'm running Devuan on my Samsung 250 GB nvme and today when booting up the distro boots to an all black screen with white letters complaining that there was a unexpected inconsistency and that fsck will have to be ran manually.

I did some reading and I understand that the drive can not be mounted and there are several ways to handle this. And the fsck can be scheduled as well for the next boot.

From Grub I could enter Rescue Mode and run fsck OR> I could boot into Slackware or my Debian 12 Mate install and run fsck as well.
Devuan is installed on /dev/nvme0n1.

Disk /dev/nvme0n1: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: Samsung SSD 970 EVO Plus 250GB        
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xd8468b80

Device         Boot     Start       End   Sectors   Size Id Type
/dev/nvme0n1p1           2048 486395903 486393856 231.9G 83 Linux
/dev/nvme0n1p2      486397950 488396799   1998850   976M  5 Extended
/dev/nvme0n1p5      486397952 488396799   1998848   976M 82 Linux swap / Solaris

This is a triple booted system on an Asus Tuf Gaming custom built desktop. Here's the other 2 HDD's.
Debian 12 Mate is installed to the 1 TB HDD /dev/sdb and Slackware is installed to the WD /dev/sda

Disk /dev/sda: 465.76 GiB, 500107862016 bytes, 976773168 sectors
Disk model: WDC WD5000AZLX-0
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 527DC993-23A2-442A-99F6-1F6D169ECE26

Device         Start       End   Sectors  Size Type
/dev/sda1       2048      6143      4096    2M Linux filesystem
/dev/sda2  972771328 976771071   3999744  1.9G Linux swap
/dev/sda3       6144   4200447   4194304    2G Linux swap
/dev/sda4    4200448 964696063 960495616  458G Linux filesystem

Partition table entries are not in disk order.


Disk /dev/sdb: 931.51 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: ST31000524NS  
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xfc500ed3

Device     Boot      Start        End    Sectors   Size Id Type
/dev/sdb1  *          2048 1917710575 1917708528 914.4G 83 Linux
/dev/sdb2       1917710576 1953520064   35809489  17.1G 82 Linux swap / Solaris

This has never been my experience in 10 plus years of running Linux..... so I'm asking what is the best approach?

Thanks in advance- Alex

 

[osprey]

@Alexzee wrote:

From Grub I could enter Rescue Mode and run fsck OR> I could boot into Slackware or my Debian 12 Mate install and run fsck as well.

You could do either of those things you have mentioned, with the relevant filesystem unmounted. Assuming the filesystem is ext4, it's possible to fsck interactively with a command like:

fsck /dev/nvme0n1p1

so the fsck command will stop at each problem and ask you whether to fix it or not.

Or, you can add the -y option which will say "yes" to each query, and try and fix the filesystem without user intervention.

However, running fsck from a live disk or rescue disk has the advantage of having the filesystem in question unmounted, but also that any running processes elsewhere and daemons in the system or on the machine are not interfering. It's safer in my view.

 

[Alexzee]

@Alexzee wrote:


You could do either of those things you have mentioned, with the relevant filesystem unmounted. Assuming the filesystem is ext4, it's possible to fsck interactively with a command like:

fsck /dev/nvme0n1p1

so the fsck command will stop at each problem and ask you whether to fix it or not.

Or, you can add the -y option which will say "yes" to each query, and try and fix the filesystem without user intervention.

However, running fsck from a live disk or rescue disk has the advantage of having the filesystem in question unmounted, but also that any running processes elsewhere and daemons in the system or on the machine are not interfering. It's safer in my view.

Thanks for the fast reply osprey!

To add the -y option do you mean to run the cmd this way?

fsck -y /dev/nvme0n1p1

The only Live USB that I have right now that's already ready to go is LMDE 6.
This desktop doesn't have a CDROM drive so will the Live USB of LMDE 6 do me justice?

 

[Alexzee]

Thanks for the fast reply osprey!

To add the -y option do you mean to run the cmd this way?

fsck -y /dev/nvme0n1p1

The only Live USB that I have right now that's already ready to go is LMDE 6.
This desktop doesn't have a CDROM drive so will the Live USB of LMDE 6 do me justice?

Oh, and another question:

How would I find out what corrupted the fs to begin with?

 

[wizardfromoz]

G'day Alex, I updated my Devuan 5 a week ago and had no problems, but I can swing into it a little later in my day and look for anything unusual.

If I find anything, I will report back.

How do you run your updates? Terminal or GUI?

Wiz

 

[Alexzee]

G'day Alex, I updated my Devuan 5 a week ago and had no problems, but I can swing into it a little later in my day and look for anything unusual.

If I find anything, I will report back.

How do you run your updates? Terminal or GUI?

Wiz

Hi mate

Glad to hear you are running Devuan:-

I update the system via the terminal.

 

[osprey]

Oh, and another question:

How would I find out what corrupted the fs to begin with?

Yes, the -y would be added as you have.

As to what has messed with the filesystem, there are a few possibilities.

If the machine has had a power interruption, even a small blip, whilst it's working, there can be a fault in writing to disk of one or more files by a program or a daemon (e.g. to a log file) so the journal for the file(s) doesn't have all the relevant info it needs, e.g. the info the directory file needs to point to where the file is on the disk. The fsck program can try and fix that.

There's the possibility of hardware issue where some bytes or sectors of the disk have become corrupted. It could be age, overuse, physical injury etc. The fsck command, or the e2fsck command, with the -c option can invoke the "badblocks" program that will mark unusable parts of the disk for non-use so the rest of the usable disk can still be used, with a lower capacity.

 

[Alexzee]

Yes, the -y would be added as you have.

As to what has messed with the filesystem, there are a few possibilities.

If the machine has had a power interruption, even a small blip, whilst it's working, there can be a fault in writing to disk of one or more files by a program or a daemon (e.g. to a log file) so the journal for the file(s) doesn't have all the relevant info it needs, e.g. the info the directory file needs to point to where the file is on the disk. The fsck program can try and fix that.

There's the possibility of hardware issue where some bytes or sectors of the disk have become corrupted. It could be age, overuse, physical injury etc. The fsck command, or the e2fsck command, with the -c option can invoke the "badblocks" program that will mark unusable parts of the disk for non-use so the rest of the usable disk can still be used, with a lower capacity.

Now that I think of it one of my applications was having performance issue's and the GUI wasn't displaying everything I wanted to use. When you mentioned a 'small blip' I quickly recalled that sometimes the lights flicker here in the house.

It's almost midnight so I'll have to fix the system tomorrow.

Can I run the -y flag and the -c option at the same time?

 

[osprey]

Now that I think of it one of my applications was having performance issue's and the GUI wasn't displaying everything I wanted to use. When you mentioned a 'small blip' I quickly recalled that sometimes the lights flicker here in the house.

It's almost midnight so I'll have to fix the system tomorrow.

Can I run the -y flag and the -c option at the same time?

If "fsck -y <partition>" is run, it may fix the filesystem so the -c operation may be unnecessary. So I'd run one at a time.

In my own case, I run my main computers on Uninterruptible Power Supplies (UPS) which provide a uniform current that smooths out the blips and brown-outs that sometimes occur here on the electrical mains. If there's a full mains power outage, the UPS can run for around half and hour so that's plenty of time to do a proper shutdown. That can also be automated, but here it's still manual.

 

[Alexzee]

A little while longer in the cave distressing and I'll be ready to fix my Devuan installation.

 

[Alexzee]

For those of you following this thread 'unexpected inconsistency' is a fancy way of saying a not so graceful shutdown.

One day last week I told the system to shut down and it halted and didn't shut down.
Holding down CTRL + F1 proved as useful so I could shut down the system with:

shutdown -h now

Upon the next reboot I was greeted with this message in the attached file:-

From the black screen with the complaint about the unexpected inconsistency and having to run fsck manually I simply plugged in the Live USB that I had of LMDE 6 and waited for it to boot to the desktop.

mint@mint:~$ sudo fsck -y /dev/nvme0n1p1
fsck from util-linux 2.38.1
e2fsck 1.47.0 (5-Feb-2023)
/dev/nvme0n1p1 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Inodes that were part of a corrupted orphan linked list found.  Fix? yes

Inode 12845062 was part of the orphaned inode list.  FIXED.
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences:  -37735445 -(53473241--53473243)
Fix? yes

Free blocks count wrong for group #1151 (8687, counted=8688).
Fix? yes

Free blocks count wrong for group #1631 (22756, counted=22759).
Fix? yes

Free blocks count wrong (56359388, counted=56355876).
Fix? yes

Inode bitmap differences:  -12589184 -12845062
Fix? yes

Free inodes count wrong for group #1536 (0, counted=1).
Fix? yes

Free inodes count wrong for group #1568 (8170, counted=8171).
Fix? yes

Free inodes count wrong (14903719, counted=14903646).
Fix? yes


/dev/nvme0n1p1: ***** FILE SYSTEM WAS MODIFIED *****
/dev/nvme0n1p1: 300706/15204352 files (0.6% non-contiguous), 4443356/60799232 blocks
mint@mint:~$ sudo fsck -c /dev/nvme0n1p1
fsck from util-linux 2.38.1
e2fsck 1.47.0 (5-Feb-2023)
Checking for bad blocks (read-only test): done                                               
/dev/nvme0n1p1: Updating bad block inode.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

/dev/nvme0n1p1: ***** FILE SYSTEM WAS MODIFIED *****
/dev/nvme0n1p1: 300706/15204352 files (0.6% non-contiguous), 4443356/60799232 blocks
mint@mint:~$

Rather than rebooting from the Live session I shut down the desktop.
I could have rebooted and pulled the USB as soon as my Asus Tuf Gaming splash came up:-

Once the desktop was off for a bit I unplugged the Live USB and rebooted.
Devuan booted right away from Grub!

It's nice to have my Devuan system up and running again.

Thanks osprey and wizardfromoz!

 

[Alexzee]

This is yet another reason to have a Live USB of Linux laying around.
It's great for rescuing Grub, what I did today and any other issues.

@wizardfromoz In your endeavors with running Devuan Daedalus have you had any performance issues?

 

[wizardfromoz]

Regrets on the delay in getting back here.

In your endeavors with running Devuan Daedalus have you had any performance issues?

Not to date, Alex.

I had one update pending, which I have applied as follows.

chris@devuanXfceHDD:~$ apt list --upgradable
Listing... Done
libuv1/stable-security 1.44.2-1+deb12u1 amd64 [upgradable from: 1.44.2-1]
N: There is 1 additional version. Please use the '-a' switch to see it
chris@devuanXfceHDD:~$ apt list --upgradable -a
Listing... Done
libuv1/stable-security 1.44.2-1+deb12u1 amd64 [upgradable from: 1.44.2-1]
libuv1/stable,now 1.44.2-1 amd64 [installed,upgradable to: 1.44.2-1+deb12u1]

Performance seems OK.

Wiz

 

[Alexzee]

From the Debian Security Team to my e-mail in-box today by:

Salvatore Bonaccorso​

It was discovered that the uv_getaddrinfo() function in libuv, an
asynchronous event notification library, incorrectly truncated certain
hostnames, which may result in bypass of security measures on internal
APIs or SSRF attacks.
For the oldstable distribution (bullseye), this problem has been fixed
in version 1.40.0-2+deb11u1.

For the stable distribution (bookworm), this problem has been fixed in
version 1.44.2-1+deb12u1.

I'm not entirely sure if this event had an effect on my Devuan install. That lib was in the update and was the only update as soon as the fs was fixed by running fsck manually.
Now I'm suspicious, as the application API's functionally failed just before the forced shutdown with elevated privileges.

 

[wizardfromoz]

Now that's interesting - I would check /etc/hosts and /etc/hostname for starters... mine are OK.

 

[Alexzee]

Now that's interesting - I would check /etc/hosts and /etc/hostname for starters... mine are OK.

I concur. It's late so I'm shutting down for the night.
I'll check those hostnames to address mappings for the TCP and post what I find after work tomorrow, thanks mate!

 

[Alexzee]

Putting out fires and de-stressing. Details tomorrow:-

 

[wizardfromoz]

...and de-stressing.

Beer works.

 

[Alexzee]

Beer works.

LOL!-

alex@debian-box:~ echo need a vacation!

 

[Alexzee]

Everything looks good in /etc/hosts.
Lot's of good information in the hosts.5 file.

I can see now how security for hostnames could be bypassed if an intruder knows how.
See the screenshots.

Sunny and 70 degrees so I'm off to enjoy the sunshine:-