dist-upgrade broke GRUB... I think?

jediwombat

New Member
Joined
Dec 11, 2018
Messages
14
Reaction score
4
Credits
0
Hi all,

I've just done an apt-get dist-upgrade on my Debian 9.6 server, and during the process, it attempted to update GRUB. This failed as it could not find the disk when it attempted to install. I told it not to install GRUB, thinking a simple grub-install /dev/sda1 would work after the dist-upgrade, but this returns a similar error.

As I understand it, I need to install GRUB over the top of itself to make sure it loads the new kernel on boot. I'm fuzzy on this part, as my previous GRUB already pointed to the right drive/partition/etc., so I thought it wouldn't need changing, but as the installer wanted to update it, I'm not confident I'm understanding this part correctly at all.

My setup is 4 3TB drives each with a BIOS partition, and the rest of the drive as a second partition. These second partitions are joined in a RAID5, which is presented to LVM as the PV.

I've found my existing GRUB on /dev/sda:
Code:
root@groth:~# dd if=/dev/sda bs=512 count=1 | xxd | grep -A2 -B2 GRUB
1+0 records in
1+0 records out
512 bytes copied, 9.5261e-05 s, 5.4 MB/s
00000160: 018e db31 f6bf 0080 8ec6 fcf3 a51f 61ff  ...1..........a.
00000170: 265a 7cbe 8e7d eb03 be9d 7de8 3400 bea2  &Z|..}....}.4...
00000180: 7de8 2e00 cd18 ebfe 4752 5542 2000 4765  }.......GRUB .Ge
00000190: 6f6d 0048 6172 6420 4469 736b 0052 6561  om.Hard Disk.Rea
000001a0: 6400 2045 7272 6f72 0d0a 00bb 0100 b40e  d. Error........

But when I try to do the grub-install on this drive:
Code:
root@groth:~# grub-install /dev/sda
Installing for i386-pc platform.
grub-install: error: disk `lvmid/8AMfWo-fbxG-Hn5u-zZ7u-Oezp-wwYC-NPhiFj/3xMcU1-qviE-c99a-m3T0-u1VO-1edY-Zt7PaP' not found.

I get the exact same error if I attempt to point it at /dev/sda1.

I'm not sure what other info people may need, but I'm happy to give what I need to if you ask. For now, the system is running, but I'm not at all confident my GRUB will boot the system, and I don't know how to test it without rebooting, which may fail.

tl;dr, three questions:
Do I need to do a new grub-install?
How can I make grub-install work?
How can I know if my current GRUB config is OK?

Thanks!
Benjamin.
 


@jediwombat welcome to linux.org :)

Sounds like a good Aussie name?

I know little about Servers, but we have some folks here who do

@JasKinasis @Rob @tchakatak and others, who might have some input. My mention of them will draw this to their attention.

I'll sign off here and then move your Thread over to General Server, where it may draw attention from Followers.

And I'll watch this thread with interest and see what this 5,000 year old Wizard can still learn.

BTW if your problem gets sorted or you find the time, swing over to Member Introductions and say hi, and meet some of The Gang :)

Cheers

Chris Turner
wizardfromoz
 
Thanks Wizard, I am indeed a fellow Australian!

Sorry for posting in the wrong subforum, and thanks for moving the thread, and pinging folks that might help :)

Cheers,
Benjamin.
 
Well, my sever crashed, and GRUB is indeed broken. I now get this prompt on boot:


Code:
error: invalid arch-independent ELF magic.
Entering rescue mode...
grub rescue>

I've booted to a Live CD and attempted to use the boot-repair tool, which it said was successful, but my boot still drops me at that same prompt.

My previous setup was a RAID5 BIOS boot partition, I think. I am not 100% on that.

How do I use the GRUB2 repair prompt? I've found some guides etc. which tell me what commands are available to me, but none of them seem accurate for my system.

I'm not really sure what to do from here. Any tips or suggestions would be greatly appreciated!
 
G'day Mate

Servers and RAID are not my bag, as I have mentioned, but I am no stranger to

grub rescue>

I am pretty sure I have a Grub Rescue doc that at least mentions RAID, I will scout around and see if I can find it and wrap my head around it sufficiently to walk you through it.

That's if the flamin' Cavalry does not arrive first to bail me/you out :)

Wiz
 
Did you do a BIOS upgrade/flash in recent times? Just brainstorming

Chris
I note

I've just done an apt-get dist-upgrade on my Debian 9.6 server,
 
Seems 2 out of 3 of the folks I pinged earlier have been unavailable for a couple of weeks, Murphy's Law. Not sure if @nuna is in town. Our Admin @Rob is predominantly CentOS and Fedora, but that is not to say he is not experienced, he is a man of many talents I am discovering. :)

This link here

https://help.ubuntu.com/community/Grub2/Installing - I have used the section starting

via chroot (you have to scroll a fair bit)

on a number of occasions, and it is still viable. Also mentions LVM and RAID. I would think it applicable to Debian, but don't quote me.

Sorry I have to bail for now, got the evening meal to cook.

NOTE for others:

It appears Benjamin is in BIOS-Legacy, not UEFI.

Cheers

Wizard
 
That looks exactly like the instructions I need! I've seen a few places were that exact document are referenced, but I've only found broken links etc. to it. Thanks! I'll update after I give that a go :)
 
Well, I was able to follow those instructions, so that's good. Unfortunately, no dice; I still boot to "error: invalid arch-independent ELF magic."

I note that those instructions advise installing grub onto /dev/sdX, not onto a partition. But, in my case of BIOS partitions at the start of each drive, ought I to install grub to /dev/sda1, for example?

In any case, I did get some errors during the grub-install and update-grub commands, inside the chroot. They're pasted here in the faint hope that they spark the answer in someone's mind. It does appear I may have a drive missing from my array, though. Could that be relevant?

I really appreciate all your help so far, Wizard :)

Benjamin.


Code:
root@ubuntu:/# grub-install /dev/sda
Installing for i386-pc platform.
grub-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
grub-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
Installation finished. No error reported.
root@ubuntu:/# mdadm --detail /dev/md0
/dev/md0:
       Version : 1.2
 Creation Time : Mon Jan  4 13:49:44 2016
    Raid Level : raid5
    Array Size : 8790398976 (8383.18 GiB 9001.37 GB)
 Used Dev Size : 2930132992 (2794.39 GiB 3000.46 GB)
  Raid Devices : 4
 Total Devices : 3
   Persistence : Superblock is persistent

 Intent Bitmap : Internal

   Update Time : Mon Dec 17 00:45:04 2018
         State : clean, degraded
Active Devices : 3
Working Devices : 3
Failed Devices : 0
 Spare Devices : 0

        Layout : left-symmetric
    Chunk Size : 512K

          Name : groth:0
          UUID : 022134c7:f9c04632:1c36d157:4c71e634
        Events : 216884

   Number   Major   Minor   RaidDevice State
      0       8        2        0      active sync   /dev/sda2
      -       0        0        1      removed
      2       8       50        2      active sync   /dev/sdd2
      3       8       66        3      active sync   /dev/sde2
root@ubuntu:/# grub-install /dev/sdd
Installing for i386-pc platform.
grub-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
grub-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
Installation finished. No error reported.
root@ubuntu:/# grub-install /dev/sde
Installing for i386-pc platform.
grub-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
grub-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
Installation finished. No error reported.
root@ubuntu:/# update-grub
Generating grub configuration file ...
/usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
/usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
/usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
/usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
/usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
/usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
Found background image: /usr/share/images/desktop-base/desktop-grub.png
/usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
Found linux image: /boot/vmlinuz-4.9.0-8-amd64
Found initrd image: /boot/initrd.img-4.9.0-8-amd64
/usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
/usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
/usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
Found linux image: /boot/vmlinuz-4.9.0-6-amd64
Found initrd image: /boot/initrd.img-4.9.0-6-amd64
Found linux image: /boot/vmlinuz-4.9.0-3-amd64
Found initrd image: /boot/initrd.img-4.9.0-3-amd64
/usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
mkdir: cannot create directory ‘/var/lib/os-prober/mount’: No such file or directory
mkdir: cannot create directory ‘/var/lib/os-prober/mount’: No such file or directory
mkdir: cannot create directory ‘/var/lib/os-prober/mount’: No such file or directory
mkdir: cannot create directory ‘/var/lib/os-prober/mount’: No such file or directory
mkdir: cannot create directory ‘/var/lib/os-prober/mount’: No such file or directory
Adding boot menu entry for EFI firmware configuration
done
root@ubuntu:/#
 
ought I to install grub to /dev/sda1, for example?

Morning Benjamin, I've just finished my 2nd coffee for the morning (Queensland) and am starting to resemble human :)

That is often a valid option in "mainstream" linux, I am not sure with the RAID/LVM scenario. However you have said earlier in your OP (OP can mean Original Poster, that is, you, and your Original Post)


thinking a simple grub-install /dev/sda1 would work after the dist-upgrade, but this returns a similar error.

So am I reading that right, that you have already tried that?

A little Googling under

grub-install: error: disk `lvmid/ not found

has yielded this

https://unix.stackexchange.com/questions/298068/system-unbootable-grub-error-disk-lvmid-not-found

Read it first and see if you think it has merit. Don't try using it yet until I clarify a couple of points.

Off for me 3rd java and I'll be back

Chris
 
(Wizard reappears in a puff of smoke, smile on his dial now that he is coffee'ed up)

The article cited is for Arch Linux but the conditions are similar. Do you think?

The two answers given were not voted up at all, but the reference to /etc/fstab got me thinking.

The commands syntax is mostly the same, except for update-grub2.

update-grub is an Ubuntu and Arch-Manjaro convention, and is a stub for

Code:
grub-mkconfig -o /boot/grub/grub.cfg
.

Anyway, take a look and see what you think. Any questions on fstab I can likely answer, unless there are differences with the RAID-LVM side of things.

Cheers

Wiz
 
Hm, that's possible; maybe the partitions aren't auto-mounting, so grub can't find anything to load.

I'll check that out when I get home.

Cheers mate,
Benjamin.
 
Sweet

upload_2018-12-17_11-18-34.jpeg


I'm a retired pensioner and usually on deck until about 6:30 Qld time (no savings) then I get the evening meal for my wife and me.

If my green "radar signal" is blinking I am close by.

Wiz
 
Doesn't look like my fstab is configured to use UUID; it's just mounting as per the LVM mapper path:


Code:
root@ubuntu:/mnt/etc# cat fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
/dev/mapper/vg_raid5-lv_slash /               ext4    errors=remount-ro 0       1
/dev/mapper/vg_raid5-lv_root /root           ext4    defaults        0       2
/dev/mapper/vg_raid5-lv_var /var            ext4    defaults        0       2
/dev/mapper/vg_raid5-lv_swap none            swap    sw              0       0
/dev/mapper/vg_raid5-lv_media /media    ext4    defaults    0    2
 
Cheers for that Mate, will take a look tomorrow, gotta fly :)

Chris
 
So, some kind of progress. I re-ran through those steps as I feared that last time I'd done update-grub, not update-grub2.

Here's the output I got (included the last two grub-install commands, for reference. Should I not be doing a grub-install /dev/sdd1? That would make sense to me but I'm probably thinking about it wrong):

Code:
root@ubuntu:/# grub-install /dev/sdd
Installing for i386-pc platform.
grub-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
grub-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
Installation finished. No error reported.
root@ubuntu:/# grub-install /dev/sde
Installing for i386-pc platform.
grub-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
grub-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
Installation finished. No error reported.
root@ubuntu:/# update-grub2
Generating grub configuration file ...
/usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
/usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
/usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
/usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
/usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
/usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
Found background image: /usr/share/images/desktop-base/desktop-grub.png
/usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
Found linux image: /boot/vmlinuz-4.9.0-8-amd64
Found initrd image: /boot/initrd.img-4.9.0-8-amd64
/usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
/usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
/usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
Found linux image: /boot/vmlinuz-4.9.0-6-amd64
Found initrd image: /boot/initrd.img-4.9.0-6-amd64
Found linux image: /boot/vmlinuz-4.9.0-3-amd64
Found initrd image: /boot/initrd.img-4.9.0-3-amd64
/usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
mkdir: cannot create directory ‘/var/lib/os-prober/mount’: No such file or directory
mkdir: cannot create directory ‘/var/lib/os-prober/mount’: No such file or directory
mkdir: cannot create directory ‘/var/lib/os-prober/mount’: No such file or directory
mkdir: cannot create directory ‘/var/lib/os-prober/mount’: No such file or directory
mkdir: cannot create directory ‘/var/lib/os-prober/mount’: No such file or directory
Adding boot menu entry for EFI firmware configuration
done
root@ubuntu:/#

On boot now, I no longer get to the grub-rescue prompt, but do just get dumped to the regular, garden-variety grub prompt.

I tried telling it to boot, but it wants me to manually load the kernel, and I'm not sure how to do that. Any ideas on what to do from the grub prompt, now that I can boot to it?

Cheers mate,
Benjamin.
 
More progress, I've figured out how to use the GRUB prompt, a bit, haha.

I ran the following and got the invalid magic number error.

Code:
set root=(lvm/vg_raid5-lv_slash)
set prefix=(lvm/vg_raid5-lv_slash)/boot/grub
linux /vmlinuz

error: invalid magic number.

I've googled this a bit but most of the threads I can find either aren't quite the same thing, or are "magically" resolved on systems that just started working again.

I'm considering trying the boot-repair tool again now that Grub seems to be in a different (better?) state, but I don't want to break things more than they are already broken :/

Any advice will be greatly appreciated!

Thanks :)
 
Yo Benjamin, how was your Christmas, hope it was good ? :)

This would be hilarious, if it was not so important for you - talk about the blind leading the blind o_O:rolleyes:

I'll digest your #17 Post in a mo', but just playing catch up

The bit from #14

/dev/mapper/vg_raid5-lv_slash / ext4 errors=remount-ro 0 1
/dev/mapper/vg_raid5-lv_root /root ext4 defaults 0 2
/dev/mapper/vg_raid5-lv_var /var ext4 defaults 0 2
/dev/mapper/vg_raid5-lv_swap none swap sw 0 0
/dev/mapper/vg_raid5-lv_media /media ext4 defaults 0 2

... new to me, but given that this is RAID, understandable.

...as I feared that last time I'd done update-grub, not update-grub2.

This is Debian so no need to reference grub2.

update-grub is a stub which calls on

Code:
grub-mkconfig -o /boot/grub/grub.cfg

to do its voodoo.

The Distro families other than Debian & derivatives (UBuntu, Mint &c) and Arch (Manjaro and Manjaro-derivatives only) do not use update-grub, they use the full command.

In some RPM-based Distros, they reference Grub2 in the syntax, and some also have a slightly different hierarchy on the placement of grub.cfg dependent on whether UEFI or Legacy based setup.

Does not apply here.

Should I not be doing a grub-install /dev/sdd1?

I would not have thought so normally, to the device itself is usually recommended.

BUT (Wizard's but usually follows closely behind him)

I have just read, in searching under Raid 5, that it may be the case that you have to install Grub to the root of the RAID, NOT to the MBR.

Do you want to try it?

Wiz
 
Hey Wizard, Christmas was good, how about yours? I ate about 3x my weight in food... and I'm not a light man!

I do want to try that, but can you please clarify what's meant by "root of the RAID"? Do you mean my slash mount, under LVM? Or the partition that's in the RAID, i.e. sd{a,b,c,d}2?

I'm off for a few hours now but I'll have a crack at that when I get back :)

Cheers mate,
Benjamin.
 
Crikey, I was hoping you would know

b_Zo_Rud6_-_Imgur.gif


I came across this from Ubuntu Forums

https://ubuntuforums.org/showthread.php?t=2167605

... and old mate "oldfred" has been around for years. I thought the /dev/mapper resembled your circumstances, see #2 there.

Google keywords I was using were

linux "raid5" mbr grub-install /dev/sdxy /dev/mapper

I can ping another three people if you don't have any joy, but it's hard to say with the season who's away or not.

Wiz
Avagudweegend

BTW I'm about 100kg and I reckon Santa twisted my arm to become 102kg. :)
 

Members online


Top