"crashed" as in hosed, wiped out, bricked, many unbootable.
It was RHEL and using LVM2. So I wanted to clone the root volume group to an external 2TB SSD. For the purpose of a backup or really RAID-1. I wanted to completely replicate the rhel volume group and then remove the cloned ssd VG from the primary VG. I used LVM for a long time and thought I could navigate it.
I found the following instructions which seemed reasonable:
I did it on RHEL but it's generic enough to work on any distro which runs LVM2. I mean the commands are exactly the same across distros.
It worked up to the point where I did the split, it did not split all the logical volumes, some just didn't split. So the external 2TB SSD and the VG on it got kind of stuck, married to the root vg on the internal PCIe/NVMe SSD. The box still ran. However, when I rebooted - I don't know why it does that - the device filename changed, e.g. from /dev/sdb to /dev/sdc and the root volume group would no longer activate. And the machine would go to the dreaded dracut prompt. I fought with it for days, trying to resolve and in the end said, screw and rebuilt it. It was just faster and easier to rebuild versus keeping fighting it. In the end, I got the scenario which is precisely what I was trying to avoid with the below process.
Mental Note:
1. Not all stuff you read online is correct (or applicable to you)
2. Test it first on a sandbox type test machine
3. LVM is the devil (NOT) but tempting to say it.
To reiterate, do not use the below, it has a bug in it somewhere (find it for bonus points) and I ran it verbatim, obviously adapting the correct device names/volume group names in my scenario.
It was RHEL and using LVM2. So I wanted to clone the root volume group to an external 2TB SSD. For the purpose of a backup or really RAID-1. I wanted to completely replicate the rhel volume group and then remove the cloned ssd VG from the primary VG. I used LVM for a long time and thought I could navigate it.
I found the following instructions which seemed reasonable:
I did it on RHEL but it's generic enough to work on any distro which runs LVM2. I mean the commands are exactly the same across distros.
It worked up to the point where I did the split, it did not split all the logical volumes, some just didn't split. So the external 2TB SSD and the VG on it got kind of stuck, married to the root vg on the internal PCIe/NVMe SSD. The box still ran. However, when I rebooted - I don't know why it does that - the device filename changed, e.g. from /dev/sdb to /dev/sdc and the root volume group would no longer activate. And the machine would go to the dreaded dracut prompt. I fought with it for days, trying to resolve and in the end said, screw and rebuilt it. It was just faster and easier to rebuild versus keeping fighting it. In the end, I got the scenario which is precisely what I was trying to avoid with the below process.
Mental Note:
1. Not all stuff you read online is correct (or applicable to you)
2. Test it first on a sandbox type test machine
3. LVM is the devil (NOT) but tempting to say it.
To reiterate, do not use the below, it has a bug in it somewhere (find it for bonus points) and I ran it verbatim, obviously adapting the correct device names/volume group names in my scenario.
Code:
# lvcreate -n test -L1g /dev/rootvg
..
# mkfs.ext3 -j -m0 /dev/rootvg/test
..
# mount /dev/rootvg/test /mnt/s
Copy some files on it to verify the integrity of the content later. Now let's make it a mirror. The --type mirror flag will enforce the use classic LVM mirror instead of newly RAID1 technology, --alloc anywhere lets you create mirror copies on VG wich contains a single PV (that is true in most situation). If your LV is big enough use -b flag to run converting in background.
# lvconvert --type mirror --alloc anywhere -m1 /dev/rootvg/test
rootvg/test: Converted: 0.00%
rootvg/test: Converted: 23.44%
rootvg/test: Converted: 46.88%
rootvg/test: Converted: 70.31%
rootvg/test: Converted: 93.75%
rootvg/test: Converted: 100.00%
# lvs -a -o +devices | egrep "LV|test"
LV VG Attr LSize .. ove Log Cpy%Sync Convert Devices
test rootvg mwi-aom--- 1.00g [test_mlog] 100.00 test_mimage_0(0),test_mimage_1(0)
[test_mimage_0] rootvg iwi-aom--- 1.00g /dev/sda5(87200)
[test_mimage_1] rootvg iwi-aom--- 1.00g /dev/sda5(36009)
[test_mlog] rootvg lwa-aom--- 4.00m /dev/sda5(66575)
Check that Cpy%Sync is 100% finished; both copies are in sync then. Now let's break the mirror !
# lvconvert --splitmirrors 1 --name testCopy /dev/rootvg/test
Logical volume test converted.
# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
..
test rootvg -wi-ao---- 1.00g
testCopy rootvg -wi-a----- 1.00g
# mount /dev/rootvg/testCopy /mnt/d
Verify content of /mnt/d if you do not trust LVM technology.
Cloning LV to external disk
This procedure can be useful for transferring data to another site, preparing DR, or migration. I will show the procedure on the KVM platform, but it is also suitable for the SAN environment. Although cloning itself can be done online without any problems, removing an external drive without reboot can be a non-trivial task. Commands for clean SAN LUN detaching could be found here: HOWTO LUNs on Linux using native tools
root@localhost:~ # pvs
PV VG Fmt Attr PSize PFree
/dev/vda2 rootvg lvm2 a-- 9.00g 0
root@localhost:~ # lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
slash rootvg -wi-ao---- 5.00g
swap rootvg -wi-ao---- 2.00g
var rootvg -wi-ao---- 2.00g
root@localhost:~ #
This is the initial setup for my RedHat 7 virtual machine. As you can see, the second partition on a single drive is used for "rootvg" which contains three logical volumes. I will add another disk (KVM allows you to do this online when using the VIRTIO disk; if you are going to add a SAN LUN, refer to the same HOWTO for the appropriate commands), then I will add this disk to the rootvg, will clone /var LV to the external disk, will split the rootvg for two VGs and finally export the external part.
So, a new 5g disk has been added to the running VM. It appears in /proc/partitions automatically, without additional scan commands:
root@localhost:~ # grep vd /proc/partitions
252 0 20971520 vda
252 1 262144 vda1
252 2 9440256 vda2
252 16 5242880 vdb
Add new disk to VG and clone desired LV to it:
root@localhost:~ # pvcreate /dev/vdb
Physical volume "/dev/vdb" successfully created.
root@localhost:~ # pvs
PV VG Fmt Attr PSize PFree
/dev/vda2 rootvg lvm2 a-- 9.00g 0
/dev/vdb lvm2 --- 5.00g 5.00g
root@localhost:~ # vgextend rootvg /dev/vdb
Volume group "rootvg" successfully extended
root@localhost:~ # pvs
PV VG Fmt Attr PSize PFree
/dev/vda2 rootvg lvm2 a-- 9.00g 0
/dev/vdb rootvg lvm2 a-- <5.00g <5.00g
root@localhost:~ # lvconvert --type mirror -m1 /dev/rootvg/var /dev/vdb
Logical volume rootvg/var being converted.
rootvg/var: Converted: 0.39%
rootvg/var: Converted: 22.46%
rootvg/var: Converted: 45.90%
rootvg/var: Converted: 72.66%
rootvg/var: Converted: 100.00%
root@localhost:~ #
Use -b (background) flag for big LV to clone. Check that cloning procedure finished (100% Copy state) by command:
root@localhost:~ # lvs -a -o +devices | egrep "LV|var"
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert Devices
var rootvg mwi-aom--- 2.00g [var_mlog] 100.00 var_mimage_0(0),var_mimage_1(0)
[var_mimage_0] rootvg iwi-aom--- 2.00g /dev/vda2(1792)
[var_mimage_1] rootvg iwi-aom--- 2.00g /dev/vdb(0)
[var_mlog] rootvg lwi-aom--- 4.00m /dev/vdb(512)
root@localhost:~ #
Make LV in consistent state. Let's think that you are exporting Oracle data, then make th LV in "begin hot backup" state and do not forget to flush caches by sync command.
Now, split mirror, split VG and export it:
root@localhost:~ # lvconvert --splitmirrors 1 --name varCopy /dev/rootvg/var /dev/vdb
Logical volume rootvg/var converted.
root@localhost:~ # lvs -a -o +devices | egrep "LV|var"
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert Devices
var rootvg -wi-ao---- 2.00g /dev/vda2(1792)
varCopy rootvg -wi-a----- 2.00g /dev/vdb(0)
root@localhost:~ # lvchange -an /dev/rootvg/varCopy
root@localhost:~ # lvs -a -o +devices | egrep "LV|var"
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert Devices
var rootvg -wi-ao---- 2.00g /dev/vda2(1792)
varCopy rootvg -wi------- 2.00g /dev/vdb(0)
root@localhost:~ # vgsplit rootvg migratevg /dev/vdb
New volume group "migratevg" successfully split from "rootvg"
root@localhost:~ # lvs -a -o +devices | egrep "LV|var"
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert Devices
varCopy migratevg -wi------- 2.00g /dev/vdb(0)
var rootvg -wi-ao---- 2.00g /dev/vda2(1792)
root@localhost:~ # pvs
PV VG Fmt Attr PSize PFree
/dev/vda2 rootvg lvm2 a-- 9.00g 0
/dev/vdb migratevg lvm2 a-- <5.00g <3.00g
root@localhost:~ # vgchange -an migratevg
0 logical volume(s) in volume group "migratevg" now active
root@localhost:~ # vgexport migratevg
Volume group "migratevg" successfully exported
root@localhost:~ # pvs
PV VG Fmt Attr PSize PFree
/dev/vda2 rootvg lvm2 a-- 9.00g 0
/dev/vdb migratevg lvm2 ax- <5.00g <3.00g
root@localhost:~ #
