Purchase Linux CDs / DVDs / Flash Drives at OSDisc.com

Welcome to Our Community

While Linux.org has been around for a while, we recently changed management and had to purge most of the content (including users). If you signed up before April 23rd, 2017 please sign up again. Thanks!

dist-upgrade broke GRUB... I think?

Discussion in 'General Server' started by jediwombat, Dec 11, 2018.

  1. jediwombat

    jediwombat New Member

    Joined:
    Dec 11, 2018
    Messages:
    11
    Likes Received:
    1
    Hi all,

    I've just done an apt-get dist-upgrade on my Debian 9.6 server, and during the process, it attempted to update GRUB. This failed as it could not find the disk when it attempted to install. I told it not to install GRUB, thinking a simple grub-install /dev/sda1 would work after the dist-upgrade, but this returns a similar error.

    As I understand it, I need to install GRUB over the top of itself to make sure it loads the new kernel on boot. I'm fuzzy on this part, as my previous GRUB already pointed to the right drive/partition/etc., so I thought it wouldn't need changing, but as the installer wanted to update it, I'm not confident I'm understanding this part correctly at all.

    My setup is 4 3TB drives each with a BIOS partition, and the rest of the drive as a second partition. These second partitions are joined in a RAID5, which is presented to LVM as the PV.


    (Log in to hide this advertisement)


    I've found my existing GRUB on /dev/sda:
    Code:
    [email protected]:~# dd if=/dev/sda bs=512 count=1 | xxd | grep -A2 -B2 GRUB
    1+0 records in
    1+0 records out
    512 bytes copied, 9.5261e-05 s, 5.4 MB/s
    00000160: 018e db31 f6bf 0080 8ec6 fcf3 a51f 61ff  ...1..........a.
    00000170: 265a 7cbe 8e7d eb03 be9d 7de8 3400 bea2  &Z|..}....}.4...
    00000180: 7de8 2e00 cd18 ebfe 4752 5542 2000 4765  }.......GRUB .Ge
    00000190: 6f6d 0048 6172 6420 4469 736b 0052 6561  om.Hard Disk.Rea
    000001a0: 6400 2045 7272 6f72 0d0a 00bb 0100 b40e  d. Error........
    But when I try to do the grub-install on this drive:
    Code:
    [email protected]:~# grub-install /dev/sda
    Installing for i386-pc platform.
    grub-install: error: disk `lvmid/8AMfWo-fbxG-Hn5u-zZ7u-Oezp-wwYC-NPhiFj/3xMcU1-qviE-c99a-m3T0-u1VO-1edY-Zt7PaP' not found.
    I get the exact same error if I attempt to point it at /dev/sda1.

    I'm not sure what other info people may need, but I'm happy to give what I need to if you ask. For now, the system is running, but I'm not at all confident my GRUB will boot the system, and I don't know how to test it without rebooting, which may fail.

    tl;dr, three questions:
    Do I need to do a new grub-install?
    How can I make grub-install work?
    How can I know if my current GRUB config is OK?

    Thanks!
    Benjamin.
     
  2. wizardfromoz

    wizardfromoz Super Moderator
    Staff Member Gold Supporter

    Joined:
    Apr 30, 2017
    Messages:
    2,078
    Likes Received:
    1,923
    @jediwombat welcome to linux.org :)

    Sounds like a good Aussie name?

    I know little about Servers, but we have some folks here who do

    @JasKinasis @Rob @tchakatak and others, who might have some input. My mention of them will draw this to their attention.

    I'll sign off here and then move your Thread over to General Server, where it may draw attention from Followers.

    And I'll watch this thread with interest and see what this 5,000 year old Wizard can still learn.

    BTW if your problem gets sorted or you find the time, swing over to Member Introductions and say hi, and meet some of The Gang :)

    Cheers

    Chris Turner
    wizardfromoz
     
  3. jediwombat

    jediwombat New Member

    Joined:
    Dec 11, 2018
    Messages:
    11
    Likes Received:
    1
    Thanks Wizard, I am indeed a fellow Australian!

    Sorry for posting in the wrong subforum, and thanks for moving the thread, and pinging folks that might help :)

    Cheers,
    Benjamin.
     
    wizardfromoz likes this.
  4. jediwombat

    jediwombat New Member

    Joined:
    Dec 11, 2018
    Messages:
    11
    Likes Received:
    1
    Well, my sever crashed, and GRUB is indeed broken. I now get this prompt on boot:


    Code:
    error: invalid arch-independent ELF magic.
    Entering rescue mode...
    grub rescue>
    I've booted to a Live CD and attempted to use the boot-repair tool, which it said was successful, but my boot still drops me at that same prompt.

    My previous setup was a RAID5 BIOS boot partition, I think. I am not 100% on that.

    How do I use the GRUB2 repair prompt? I've found some guides etc. which tell me what commands are available to me, but none of them seem accurate for my system.

    I'm not really sure what to do from here. Any tips or suggestions would be greatly appreciated!
     
  5. wizardfromoz

    wizardfromoz Super Moderator
    Staff Member Gold Supporter

    Joined:
    Apr 30, 2017
    Messages:
    2,078
    Likes Received:
    1,923
    G'day Mate

    Servers and RAID are not my bag, as I have mentioned, but I am no stranger to

    grub rescue>

    I am pretty sure I have a Grub Rescue doc that at least mentions RAID, I will scout around and see if I can find it and wrap my head around it sufficiently to walk you through it.

    That's if the flamin' Cavalry does not arrive first to bail me/you out :)

    Wiz
     
  6. wizardfromoz

    wizardfromoz Super Moderator
    Staff Member Gold Supporter

    Joined:
    Apr 30, 2017
    Messages:
    2,078
    Likes Received:
    1,923
    Did you do a BIOS upgrade/flash in recent times? Just brainstorming

    Chris
    I note

     
  7. wizardfromoz

    wizardfromoz Super Moderator
    Staff Member Gold Supporter

    Joined:
    Apr 30, 2017
    Messages:
    2,078
    Likes Received:
    1,923
    Seems 2 out of 3 of the folks I pinged earlier have been unavailable for a couple of weeks, Murphy's Law. Not sure if @nuna is in town. Our Admin @Rob is predominantly CentOS and Fedora, but that is not to say he is not experienced, he is a man of many talents I am discovering. :)

    This link here

    https://help.ubuntu.com/community/Grub2/Installing - I have used the section starting

    via chroot (you have to scroll a fair bit)

    on a number of occasions, and it is still viable. Also mentions LVM and RAID. I would think it applicable to Debian, but don't quote me.

    Sorry I have to bail for now, got the evening meal to cook.

    NOTE for others:

    It appears Benjamin is in BIOS-Legacy, not UEFI.

    Cheers

    Wizard
     
  8. jediwombat

    jediwombat New Member

    Joined:
    Dec 11, 2018
    Messages:
    11
    Likes Received:
    1
    That looks exactly like the instructions I need! I've seen a few places were that exact document are referenced, but I've only found broken links etc. to it. Thanks! I'll update after I give that a go :)
     
  9. jediwombat

    jediwombat New Member

    Joined:
    Dec 11, 2018
    Messages:
    11
    Likes Received:
    1
    Well, I was able to follow those instructions, so that's good. Unfortunately, no dice; I still boot to "error: invalid arch-independent ELF magic."

    I note that those instructions advise installing grub onto /dev/sdX, not onto a partition. But, in my case of BIOS partitions at the start of each drive, ought I to install grub to /dev/sda1, for example?

    In any case, I did get some errors during the grub-install and update-grub commands, inside the chroot. They're pasted here in the faint hope that they spark the answer in someone's mind. It does appear I may have a drive missing from my array, though. Could that be relevant?

    I really appreciate all your help so far, Wizard :)

    Benjamin.


    Code:
    [email protected]:/# grub-install /dev/sda
    Installing for i386-pc platform.
    grub-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
    grub-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
    Installation finished. No error reported.
    [email protected]:/# mdadm --detail /dev/md0
    /dev/md0:
           Version : 1.2
     Creation Time : Mon Jan  4 13:49:44 2016
        Raid Level : raid5
        Array Size : 8790398976 (8383.18 GiB 9001.37 GB)
     Used Dev Size : 2930132992 (2794.39 GiB 3000.46 GB)
      Raid Devices : 4
     Total Devices : 3
       Persistence : Superblock is persistent
    
     Intent Bitmap : Internal
    
       Update Time : Mon Dec 17 00:45:04 2018
             State : clean, degraded
    Active Devices : 3
    Working Devices : 3
    Failed Devices : 0
     Spare Devices : 0
    
            Layout : left-symmetric
        Chunk Size : 512K
    
              Name : groth:0
              UUID : 022134c7:f9c04632:1c36d157:4c71e634
            Events : 216884
    
       Number   Major   Minor   RaidDevice State
          0       8        2        0      active sync   /dev/sda2
          -       0        0        1      removed
          2       8       50        2      active sync   /dev/sdd2
          3       8       66        3      active sync   /dev/sde2
    [email protected]:/# grub-install /dev/sdd
    Installing for i386-pc platform.
    grub-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
    grub-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
    Installation finished. No error reported.
    [email protected]:/# grub-install /dev/sde
    Installing for i386-pc platform.
    grub-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
    grub-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
    Installation finished. No error reported.
    [email protected]:/# update-grub
    Generating grub configuration file ...
    /usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
    /usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
    /usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
    /usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
    /usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
    /usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
    Found background image: /usr/share/images/desktop-base/desktop-grub.png
    /usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
    Found linux image: /boot/vmlinuz-4.9.0-8-amd64
    Found initrd image: /boot/initrd.img-4.9.0-8-amd64
    /usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
    /usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
    /usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
    Found linux image: /boot/vmlinuz-4.9.0-6-amd64
    Found initrd image: /boot/initrd.img-4.9.0-6-amd64
    Found linux image: /boot/vmlinuz-4.9.0-3-amd64
    Found initrd image: /boot/initrd.img-4.9.0-3-amd64
    /usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
    mkdir: cannot create directory ‘/var/lib/os-prober/mount’: No such file or directory
    mkdir: cannot create directory ‘/var/lib/os-prober/mount’: No such file or directory
    mkdir: cannot create directory ‘/var/lib/os-prober/mount’: No such file or directory
    mkdir: cannot create directory ‘/var/lib/os-prober/mount’: No such file or directory
    mkdir: cannot create directory ‘/var/lib/os-prober/mount’: No such file or directory
    Adding boot menu entry for EFI firmware configuration
    done
    [email protected]:/#
     
  10. wizardfromoz

    wizardfromoz Super Moderator
    Staff Member Gold Supporter

    Joined:
    Apr 30, 2017
    Messages:
    2,078
    Likes Received:
    1,923
    Morning Benjamin, I've just finished my 2nd coffee for the morning (Queensland) and am starting to resemble human :)

    That is often a valid option in "mainstream" linux, I am not sure with the RAID/LVM scenario. However you have said earlier in your OP (OP can mean Original Poster, that is, you, and your Original Post)


    So am I reading that right, that you have already tried that?

    A little Googling under

    grub-install: error: disk `lvmid/ not found

    has yielded this

    https://unix.stackexchange.com/questions/298068/system-unbootable-grub-error-disk-lvmid-not-found

    Read it first and see if you think it has merit. Don't try using it yet until I clarify a couple of points.

    Off for me 3rd java and I'll be back

    Chris
     
  11. wizardfromoz

    wizardfromoz Super Moderator
    Staff Member Gold Supporter

    Joined:
    Apr 30, 2017
    Messages:
    2,078
    Likes Received:
    1,923
    (Wizard reappears in a puff of smoke, smile on his dial now that he is coffee'ed up)

    The article cited is for Arch Linux but the conditions are similar. Do you think?

    The two answers given were not voted up at all, but the reference to /etc/fstab got me thinking.

    The commands syntax is mostly the same, except for update-grub2.

    update-grub is an Ubuntu and Arch-Manjaro convention, and is a stub for

    Code:
    grub-mkconfig -o /boot/grub/grub.cfg
    .

    Anyway, take a look and see what you think. Any questions on fstab I can likely answer, unless there are differences with the RAID-LVM side of things.

    Cheers

    Wiz
     
  12. jediwombat

    jediwombat New Member

    Joined:
    Dec 11, 2018
    Messages:
    11
    Likes Received:
    1
    Hm, that's possible; maybe the partitions aren't auto-mounting, so grub can't find anything to load.

    I'll check that out when I get home.

    Cheers mate,
    Benjamin.
     
  13. wizardfromoz

    wizardfromoz Super Moderator
    Staff Member Gold Supporter

    Joined:
    Apr 30, 2017
    Messages:
    2,078
    Likes Received:
    1,923
    Sweet

    upload_2018-12-17_11-18-34.jpeg

    I'm a retired pensioner and usually on deck until about 6:30 Qld time (no savings) then I get the evening meal for my wife and me.

    If my green "radar signal" is blinking I am close by.

    Wiz
     
  14. jediwombat

    jediwombat New Member

    Joined:
    Dec 11, 2018
    Messages:
    11
    Likes Received:
    1
    Doesn't look like my fstab is configured to use UUID; it's just mounting as per the LVM mapper path:


    Code:
    [email protected]:/mnt/etc# cat fstab
    # /etc/fstab: static file system information.
    #
    # Use 'blkid' to print the universally unique identifier for a
    # device; this may be used with UUID= as a more robust way to name devices
    # that works even if disks are added and removed. See fstab(5).
    #
    # <file system> <mount point>   <type>  <options>       <dump>  <pass>
    /dev/mapper/vg_raid5-lv_slash /               ext4    errors=remount-ro 0       1
    /dev/mapper/vg_raid5-lv_root /root           ext4    defaults        0       2
    /dev/mapper/vg_raid5-lv_var /var            ext4    defaults        0       2
    /dev/mapper/vg_raid5-lv_swap none            swap    sw              0       0
    /dev/mapper/vg_raid5-lv_media /media    ext4    defaults    0    2
     
  15. wizardfromoz

    wizardfromoz Super Moderator
    Staff Member Gold Supporter

    Joined:
    Apr 30, 2017
    Messages:
    2,078
    Likes Received:
    1,923
    Cheers for that Mate, will take a look tomorrow, gotta fly :)

    Chris
     
  16. jediwombat

    jediwombat New Member

    Joined:
    Dec 11, 2018
    Messages:
    11
    Likes Received:
    1
    So, some kind of progress. I re-ran through those steps as I feared that last time I'd done update-grub, not update-grub2.

    Here's the output I got (included the last two grub-install commands, for reference. Should I not be doing a grub-install /dev/sdd1? That would make sense to me but I'm probably thinking about it wrong):

    Code:
    [email protected]:/# grub-install /dev/sdd
    Installing for i386-pc platform.
    grub-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
    grub-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
    Installation finished. No error reported.
    [email protected]:/# grub-install /dev/sde
    Installing for i386-pc platform.
    grub-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
    grub-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
    Installation finished. No error reported.
    [email protected]:/# update-grub2
    Generating grub configuration file ...
    /usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
    /usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
    /usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
    /usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
    /usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
    /usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
    Found background image: /usr/share/images/desktop-base/desktop-grub.png
    /usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
    Found linux image: /boot/vmlinuz-4.9.0-8-amd64
    Found initrd image: /boot/initrd.img-4.9.0-8-amd64
    /usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
    /usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
    /usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
    Found linux image: /boot/vmlinuz-4.9.0-6-amd64
    Found initrd image: /boot/initrd.img-4.9.0-6-amd64
    Found linux image: /boot/vmlinuz-4.9.0-3-amd64
    Found initrd image: /boot/initrd.img-4.9.0-3-amd64
    /usr/sbin/grub-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
    mkdir: cannot create directory ‘/var/lib/os-prober/mount’: No such file or directory
    mkdir: cannot create directory ‘/var/lib/os-prober/mount’: No such file or directory
    mkdir: cannot create directory ‘/var/lib/os-prober/mount’: No such file or directory
    mkdir: cannot create directory ‘/var/lib/os-prober/mount’: No such file or directory
    mkdir: cannot create directory ‘/var/lib/os-prober/mount’: No such file or directory
    Adding boot menu entry for EFI firmware configuration
    done
    [email protected]:/#
    On boot now, I no longer get to the grub-rescue prompt, but do just get dumped to the regular, garden-variety grub prompt.

    I tried telling it to boot, but it wants me to manually load the kernel, and I'm not sure how to do that. Any ideas on what to do from the grub prompt, now that I can boot to it?

    Cheers mate,
    Benjamin.
     
  17. jediwombat

    jediwombat New Member

    Joined:
    Dec 11, 2018
    Messages:
    11
    Likes Received:
    1
    More progress, I've figured out how to use the GRUB prompt, a bit, haha.

    I ran the following and got the invalid magic number error.

    Code:
    set root=(lvm/vg_raid5-lv_slash)
    set prefix=(lvm/vg_raid5-lv_slash)/boot/grub
    linux /vmlinuz
    
    error: invalid magic number.
    I've googled this a bit but most of the threads I can find either aren't quite the same thing, or are "magically" resolved on systems that just started working again.

    I'm considering trying the boot-repair tool again now that Grub seems to be in a different (better?) state, but I don't want to break things more than they are already broken :/

    Any advice will be greatly appreciated!

    Thanks :)
     
  18. wizardfromoz

    wizardfromoz Super Moderator
    Staff Member Gold Supporter

    Joined:
    Apr 30, 2017
    Messages:
    2,078
    Likes Received:
    1,923
    Yo Benjamin, how was your Christmas, hope it was good ? :)

    This would be hilarious, if it was not so important for you - talk about the blind leading the blind o_O:rolleyes:

    I'll digest your #17 Post in a mo', but just playing catch up

    The bit from #14

    ... new to me, but given that this is RAID, understandable.

    This is Debian so no need to reference grub2.

    update-grub is a stub which calls on

    Code:
    grub-mkconfig -o /boot/grub/grub.cfg
    to do its voodoo.

    The Distro families other than Debian & derivatives (UBuntu, Mint &c) and Arch (Manjaro and Manjaro-derivatives only) do not use update-grub, they use the full command.

    In some RPM-based Distros, they reference Grub2 in the syntax, and some also have a slightly different hierarchy on the placement of grub.cfg dependent on whether UEFI or Legacy based setup.

    Does not apply here.

    I would not have thought so normally, to the device itself is usually recommended.

    BUT (Wizard's but usually follows closely behind him)

    I have just read, in searching under Raid 5, that it may be the case that you have to install Grub to the root of the RAID, NOT to the MBR.

    Do you want to try it?

    Wiz
     
  19. jediwombat

    jediwombat New Member

    Joined:
    Dec 11, 2018
    Messages:
    11
    Likes Received:
    1
    Hey Wizard, Christmas was good, how about yours? I ate about 3x my weight in food... and I'm not a light man!

    I do want to try that, but can you please clarify what's meant by "root of the RAID"? Do you mean my slash mount, under LVM? Or the partition that's in the RAID, i.e. sd{a,b,c,d}2?

    I'm off for a few hours now but I'll have a crack at that when I get back :)

    Cheers mate,
    Benjamin.
     
  20. wizardfromoz

    wizardfromoz Super Moderator
    Staff Member Gold Supporter

    Joined:
    Apr 30, 2017
    Messages:
    2,078
    Likes Received:
    1,923
    Crikey, I was hoping you would know

    [​IMG]

    I came across this from Ubuntu Forums

    https://ubuntuforums.org/showthread.php?t=2167605

    ... and old mate "oldfred" has been around for years. I thought the /dev/mapper resembled your circumstances, see #2 there.

    Google keywords I was using were

    linux "raid5" mbr grub-install /dev/sdxy /dev/mapper

    I can ping another three people if you don't have any joy, but it's hard to say with the season who's away or not.

    Wiz
    Avagudweegend

    BTW I'm about 100kg and I reckon Santa twisted my arm to become 102kg. :)
     

Share This Page