Category: LVM

May 21

LVM resize root volume

Since this is a root/boot volume we are talking about anything on this page is VERY risky.  Be warned this post may provide you with a strategy or hints but you are completely responsible for your system.

Of course the best route to extend a root volume in an unmounted state is to boot of a Live CD.  In some cases and even more relevant a lot of cloud instances you do not have normal physical or virtualization methods to access server consoles.

I have been trying a couple ideas.  First one is using dracut to resize the root volume on system bootup.  I have had success with this method.  The second method is using pivot_root.  On this method it may be slightly easier on systemd servers but for my paritcular need I have an older Centos 6 flavor without systemd and so far I could not get pivot_root and resize online to work.

Below is notes on method 1 (dracut resize):

Specs:
VirtualBox VM CentOS6.4
Disk layout 15G
- /boot 200M ext4
- / LV 13G ext4
- /u01 LV 2G ext4

GOAL:
1. Resize / file system smaller without boot CD only SSH access. Using dracut bootup.
2. Extend /u01 with the additional space in the VG

	
[root@localhost ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroupSys-LogVolRoot
                       11G  2.1G  8.7G  20% /
tmpfs                 1.9G     0  1.9G   0% /dev/shm
/dev/sda1             194M   29M  155M  16% /boot
/dev/mapper/VolGroupSys-LogVolU01
                      2.1G   68M  2.0G   4% /u01
[root@localhost ~]# pvs
  PV         VG          Fmt  Attr PSize  PFree
  /dev/sda2  VolGroupSys lvm2 a--  14.80g 1.70g

[root@localhost ~]# vgs
  VG          #PV #LV #SN Attr   VSize  VFree
  VolGroupSys   1   2   0 wz--n- 14.80g 1.70g

[root@localhost ~]# lvs
  LV         VG          Attr      LSize  Pool Origin Data%  Move Log Cpy%Sync Convert
  LogVolRoot VolGroupSys -wi-ao--- 11.00g                                             
  LogVolU01  VolGroupSys -wi-ao---  2.11g    
  
# pwd
/usr/share/dracut/modules.d/95rootfs-block

# vi mount-root.sh
[..]
if [ -n "$root" -a -z "${root%%block:*}" ]; then
    ## custom code testing lvresize on root vol
    lvm vgchange -ay --config " global {locking_type=1} "
    rm -f /etc/lvm/lvm.conf
    e2fsck -C 0 -f /dev/VolGroupSys/LogVolRoot
    resize2fs -p -f /dev/VolGroupSys/LogVolRoot 8G
    lvm lvresize -f /dev/VolGroupSys/LogVolRoot -L 8G
    resize2fs -p -f /dev/VolGroupSys/LogVolRoot
    e2fsck -C 0 -f /dev/VolGroupSys/LogVolRoot

    mount -t ${fstype:-auto} -o "$rflags" "${root#block:}" "$NEWROOT" \
[..]
	
# dracut -f --install 'resize2fs e2fsck'

# reboot

Verify after the reboot

# pvs
  PV         VG          Fmt  Attr PSize  PFree
  /dev/sda2  VolGroupSys lvm2 a--  14.80g 4.70g

# vgs
  VG          #PV #LV #SN Attr   VSize  VFree
  VolGroupSys   1   2   0 wz--n- 14.80g 4.70g

# lvs
  LV         VG          Attr      LSize Pool Origin Data%  Move Log Cpy%Sync Convert
  LogVolRoot VolGroupSys -wi-ao--- 8.00g                                             
  LogVolU01  VolGroupSys -wi-ao--- 2.11g                                             

# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroupSys-LogVolRoot
                      7.9G  2.1G  5.8G  26% /
tmpfs                 1.9G     0  1.9G   0% /dev/shm
/dev/sda1             194M   30M  155M  16% /boot
/dev/mapper/VolGroupSys-LogVolU01
                      2.1G   68M  2.0G   4% /u01

** remove custom code from mount-root.sh

# dracut -f
# reboot

#################################################################
### EXTEND u01
#################################################################

# lvextend -L+2G /dev/VolGroupSys/LogVolU01 
  Extending logical volume LogVolU01 to 4.11 GiB
  Logical volume LogVolU01 successfully resized

# resize2fs /dev/VolGroupSys/LogVolU01 
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/VolGroupSys/LogVolU01 is mounted on /u01; on-line resizing required
old desc_blocks = 1, new_desc_blocks = 1
Performing an on-line resize of /dev/VolGroupSys/LogVolU01 to 1076224 (4k) blocks.
The filesystem on /dev/VolGroupSys/LogVolU01 is now 1076224 blocks long.

# vgs
  VG          #PV #LV #SN Attr   VSize  VFree
  VolGroupSys   1   2   0 wz--n- 14.80g 2.70g

# lvs
  LV         VG          Attr      LSize Pool Origin Data%  Move Log Cpy%Sync Convert
  LogVolRoot VolGroupSys -wi-ao--- 8.00g                                             
  LogVolU01  VolGroupSys -wi-ao--- 4.11g                                             

# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroupSys-LogVolRoot
                      7.9G  2.1G  5.8G  26% /
tmpfs                 1.9G     0  1.9G   0% /dev/shm
/dev/sda1             194M   29M  155M  16% /boot
/dev/mapper/VolGroupSys-LogVolU01
                      4.1G   69M  3.8G   2% /u01

#################################################################					  
### APPENDIX: 
#################################################################

** Use /var/messages/boot.log to check dracut failures on our custom code
[bash]
# more boot.log 
e2fsck 1.41.12 (17-May-2010)
Pass 1: Checking inodes, blocks, and sizes
_CentOS-6.4-x86_: |===                                             -  4.8%   
Pass 2: Checking directory structure                                           
Pass 3: Checking directory connectivity                                        
Pass 4: Checking reference counts
Pass 5: Checking group summary information
_CentOS-6.4-x86_: 76680/835584 files (0.1% non-contiguous), 578555/3328000 blocks
resize2fs 1.41.12 (17-May-2010)
Resizing the filesystem on /dev/VolGroupSys/LogVolRoot to 2621440 (4k) blocks.
Begin pass 3 (max = 102)
Scanning inode table          XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
The filesystem on /dev/VolGroupSys/LogVolRoot is now 2621440 blocks long.

  Read-only locking type set. Write locks are prohibited.
  Can't get lock for VolGroupSys
resize2fs 1.41.12 (17-May-2010)
Resizing the filesystem on /dev/VolGroupSys/LogVolRoot to 3328000 (4k) blocks.
Begin pass 1 (max = 22)
Extending the inode table     XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
The filesystem on /dev/VolGroupSys/LogVolRoot is now 3328000 blocks long.
[..]

** Howto check if lvm and resize2fs is in intramfs image

# lsinitrd /boot/initramfs-2.6.32-358.el6.x86_64.img | grep lvm
-rw-r--r--   1 root     root          657 Feb 22  2013 etc/udev/rules.d/64-lvm.rules
-r--r--r--   1 root     root         1286 Feb 22  2013 etc/udev/rules.d/11-dm-lvm.rules
drwxr-xr-x   2 root     root            0 May 21 13:27 etc/lvm
-rw-r--r--   1 root     root        37554 May 21 13:27 etc/lvm/lvm.conf
-rwxr-xr-x   1 root     root         2243 Feb 22  2013 sbin/lvm_scan
-r-xr-xr-x   1 root     root      1013336 May 21 13:27 sbin/lvm
[..]
-rwxr-xr-x   1 root     root          525 Feb 22  2013 cmdline/30parse-lvm.sh

# lsinitrd /boot/initramfs-2.6.32-358.el6.x86_64.img | grep resize2fs

#################################################################
#### Some links ...
#################################################################

Kernel panic on boot following "dracut Warning: LVM rootvg/rootlv not found": https://access.redhat.com/solutions/1282013
Rename LVM Volume Group Holding Root File System Volume: https://oraganism.wordpress.com/2013/03/09/rename-lvm-vg-for-root-fs-lv/
How to debug Dracut problems: https://fedoraproject.org/wiki/How_to_debug_Dracut_problems
Inject ephemeral disk into root disk: https://github.com/eucalyptus/eucalyptus/wiki/Inject-ephemeral-disk-into-root-disk
Convert an Existing System to Use Thin LVs: https://dustymabe.com/2013/09/07/convert-an-existing-system-to-use-thin-lvs/
Partition Resize fails on LVM: https://github.com/flegmatik/linux-rootfs-resize/issues/8
How to write a pre-mount startup script?: https://unix.stackexchange.com/questions/87814/how-to-write-a-pre-mount-startup-script

Is it possible to on-line shrink a EXT4 volume with LVM?: https://serverfault.com/questions/528075/is-it-possible-to-on-line-shrink-a-ext4-volume-with-lvm/528076
resize a Linux root partition while it's still mounted: http://www.ivarch.com/blogs/oss/2007/01/resize-a-live-root-fs-a-howto.shtml
How to shrink root filesystem without booting a livecd: https://unix.stackexchange.com/questions/226872/how-to-shrink-root-filesystem-without-booting-a-livecd?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa

[/bash]

Comments Off on LVM resize root volume
comments

Mar 02

LVM Removing VG with missing disk

If you removed a disk from a server and forgot to cleanly remove the configuration you may experience something similar.

Below shows what happens with lvremove:

# lvremove --force --verbose /dev/vg02/u01
  WARNING: Device for PV ore6TE-2ues-rc5q-mkCN-hEHE-C4Av-paaxEH not found or rejected by a filter.
    There are 1 physical volumes missing.
    Archiving volume group "vg02" metadata (seqno 2).
    Releasing logical volume "u01"
  Aborting vg_write: No metadata areas to write to!

Some people seem to have success with vgreduce commands but I ended up making a fake device and extending the VG in order to cleanup the LV.

# lvs
  WARNING: Device for PV ore6TE-2ues-rc5q-mkCN-hEHE-C4Av-paaxEH not found or rejected by a filter.
  LV   VG   Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  opt  vg01 -wi-ao---- 995.00g                                                    
  u01  vg02 -wi-----p- 248.00g                                                    

# dd if=/dev/zero of=/tmp/tmp.raw bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 0.0763176 s, 1.4 GB/s

# losetup -f
/dev/loop0

# losetup /dev/loop0 /tmp/tmp.raw 

# vgextend vg02 /dev/loop0
  WARNING: Device for PV ore6TE-2ues-rc5q-mkCN-hEHE-C4Av-paaxEH not found or rejected by a filter.
  WARNING: Device for PV ore6TE-2ues-rc5q-mkCN-hEHE-C4Av-paaxEH not found or rejected by a filter.
  Physical volume "/dev/loop0" successfully created.
  WARNING: Device for PV ore6TE-2ues-rc5q-mkCN-hEHE-C4Av-paaxEH not found or rejected by a filter.
  Volume group "vg02" successfully extended

# vgremove vg02 --force
  WARNING: Device for PV ore6TE-2ues-rc5q-mkCN-hEHE-C4Av-paaxEH not found or rejected by a filter.
  Logical volume "u01" successfully removed
  Volume group "vg02" successfully removed

# pvremove /dev/loop0
  WARNING: Device for PV ore6TE-2ues-rc5q-mkCN-hEHE-C4Av-paaxEH not found or rejected by a filter.
  Labels on physical volume "/dev/loop0" successfully wiped.

# losetup -d /dev/loop0  
# rm /tmp/tmp.raw 

# lvs
  LV   VG   Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  opt  vg01 -wi-ao---- 995.00g                                                    
# vgs
  VG   #PV #LV #SN Attr   VSize   VFree
  vg01   1   1   0 wz--n- 999.99g 4.99g
# pvs
  WARNING: Device for PV ore6TE-2ues-rc5q-mkCN-hEHE-C4Av-paaxEH not found or rejected by a filter.
  PV         VG   Fmt  Attr PSize   PFree
  /dev/sdb   vg01 lvm2 a--  999.99g 4.99g

Comments Off on LVM Removing VG with missing disk
comments

Nov 06

KVM VM Rename and Logical Volume Rename

I recently provisioned a Virtual Machine in a KVM Hypervisor with a typo in the name. Even worse I also created the storage LV with a typo. Here is how I fixed it. Off course shut the VM down first.

LV Rename

# lvrename VolumeGroup oldLVname newLVname
  Renamed "oldLVname" to "newLVname" in volume group "VolumeGroup"

Change reference to storage in VM XML

# virsh edit oldVMname
Domain oldVMname XML configuration edited.

# grep newLVname oldVMname.xml 
      

Recreate VM

# virsh dumpxml oldVMname > /tmp/newVMname.xml

# virsh undefine oldVMname
Domain newVMname has been undefined

# grep newVMname /tmp/newVMname.xml 
  newVMname
      

# virsh define /tmp/newVMname.xml 
Domain newVMname defined from /tmp/newVMname.xml

# virsh list --all
 Id Name                 State
----------------------------------
  7 aVMname            running
  - newVMname         shut off

# virsh start newVMname
Domain newVMname started

Comments Off on KVM VM Rename and Logical Volume Rename
comments

Nov 06

How to Rename your Logical Volume Group

Boot a Linux live CD. I used Fedora which I know has LVM enabled in the kernel at boot time.

# vgchange -a y
  5 logical volume(s) in volume group "locallvm" now active

# mkdir /mnt/a

# mount /dev/locallvm/root /mnt/a

** Figure out where your /boot physical partition lives. This one was sda2 but it could easily be /dev/sda1 also.

# mount /dev/sda2 /mnt/a/boot

# ls /mnt/a/boot/
abi-3.0.0-12-server         memtest86+.bin
config-3.0.0-12-server      memtest86+_multiboot.bin
grub                        System.map-3.0.0-12-server
initrd.img-3.0.0-12-server  vmcoreinfo-3.0.0-12-server
lost+found                  vmlinuz-3.0.0-12-server

# mount -o bind /sys /mnt/a/sys
** Note:  if –o dev and –o proc don’t work, use –o bind for the next two
# mount -o dev /dev /mnt/a/dev
# mount -o proc /proc /mnt/a/proc

# chroot /mnt/a /bin/bash

Run update-grub
** If /run/lock does not exist, create the directory or else update-grub will fail since /var/lock is linked to /run/lock.

Use ls /boot to find the most current kernel and then run mkinitramfs on it.

mkinitramfs -k -o /boot/initrd.img-3.2.0-29-generic -v 3.2.0-29-generic

This next step is important and you need to do it before you reboot. If necessary change /etc/fstab references to point to new VG name.

Type exit. Choose reboot system from menu. Eject the CD so we don’t boot off CD by default.

Comments Off on How to Rename your Logical Volume Group
comments