Libvirt and QCow2 Snapshots

This article is for one specific use case I had.  There is a lot more written around snapshots, qemu and libvirt.  For my use case a qcow2 formatted image and KVM virtual machine.  I wanted a way to capture machine state before an OS upgrade.  So that I can roll back or revert in case something goes wrong.  So for my case I do not care about live snapshots.  I can shut the machine down before taking a consistent snapshot of the image.

Specifically I am using what is called an internal qcow2 snapshot.

https://libvirt.org/formatsnapshot.html

# virsh snapshot-create-as debian1 pre-update
Domain snapshot pre-update created
# virsh snapshot-list debian1
 Name                 Creation Time             State
------------------------------------------------------------
 pre-update           2015-02-04 09:22:18 -0600 shutoff

# virsh snapshot-current debian1
<domainsnapshot>
  <name>pre-update</name>
  <state>shutoff</state>
  <creationTime>1423063338</creationTime>
  <memory snapshot='no'/>
  <disks>
    <disk name='vda' snapshot='internal'/>
    <disk name='hda' snapshot='no'/>
  </disks>
[..]

Start VM and make some changes…and…Shutdown

# virsh snapshot-revert debian1 pre-update

Now start VM and changes should be gone.

** Note when I created the snapshots using virsh they did not show up in virt-manager. Probably needs a refresh in virt-manager. Appears like a bug. When I create a new snapshot in virt-manager I can see both(virsh created and virt-manager created). Or cwhen I ompletely quit virt-manager and re-open I see all.

Deletion of an old snapshot:

# virsh snapshot-delete debian1 pre-update
Domain snapshot pre-update deleted

KVM virsh console on CentOS 7

Several articles on using the console feature of KVM guests exists already. This is just a quick note on a test I did using specifically Centos 7 which is using systemd now.

Console allows you to A) see bootup messages and B) login on the console of the guest without SSH.

As far as I can tell you only need to do two things. Add console=ttyS0 to the kernel being booted and in the guest XML switch virtio to serial for the console. Keep in mind this is if you are using the RHEL 7 type when creating the VM. If you were using a RHEL 6 type when creating the guest the console probably still were configured to use serial.

This KVM hypervisor is on Fedora 20 and using libvirt version 1.1.3.6.

To send messages from the kernel to the serial console add console=ttyS0:

# grep console=ttyS0 /boot/grub2/grub.cfg
        linux16 /vmlinuz-3.10.0-123.el7.x86_64 root=UUID=ba0f2424-e66e-4862-90ff-7dccb63339c2 ro rd.lvm.lv=centos/swap vconsole.font=latarcyrheb-sun16 rd.lvm.lv=centos/root crashkernel=auto  vconsole.keymap=us LANG=en_US.UTF-8 console=ttyS0

# grep ttyS0 /etc/securetty
ttyS0

Change guest console type to serial:

<console type='pty' tty='/dev/pts/6'>
      <source path='/dev/pts/6'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>

# virsh console centos7
Connected to domain centos7
CentOS Linux 7 (Core)
Kernel 3.10.0-123.el7.x86_64 on an x86_64

localhost login: 

Above should do it but if you want to see even the grub2 menu options you can try the following:

# grep serial /etc/default/grub 
GRUB_TERMINAL=serial
GRUB_SERIAL_COMMAND="serial --speed=38400 --unit=0 --word=8 --parity=no --stop=1"

# grep LINUX_DEFAULT /etc/default/grub 
GRUB_CMDLINE_LINUX_DEFAULT="console=tty0 console=ttyS0"

# grub2-mkconfig -o /boot/grub2/grub.cfg

Of course if you are setting grub DEFAULT options you don’t need to set anything for the kernel entries in /boot/grub2/grub.cfg like I did in step 1.

Note that when connecting to the console you might need to move your up or down arrows to see the actual menu entries. The terminal and curses might look a little flaky but it is there. I also set the grub TIMEOUT a little longer to wait at the menu entries.

Posted in KVM

Sun ZFS Storage Appliance Simulator on OVM or KVM

For those familiar with the excellent ZFS file system and the Sun(now Oracle) storage products built on ZFS, the ZFS storage appliance interface is very easy to use; and definitely worth considering when looking at when purchasing a SAN.

Oracle has a simulator virtual machine to try out the interface.  Unfortunately it only runs on Virtualbox which is fine for those running Virtualbox on a desktop.  If you would like to run it on something more accessible by multiple users(KVM or OVM); the Solaris based image has some issues running.

I recently got the Virtualbox image to run on OVM and subsequently also got it to work on KVM.  This is a quick guide how to get the Virtualbox image to run as a qcow2 image on a KVM hypervisor.
Update: Changed to qed format. If you don’t have qed, qcow2 worked for me also.

As I understand there was also a vmware image but it disappeared from the Oracle website.  I am not sure why Oracle does not publish at least OVM images or make an effort to run the simulator on OVM.  Maybe there is a good reason and it’s possible that Oracle want to discourage it being used other than on Virtualbox.  Really not sure.

Stage the Image:
Download the simulator (link should be on this page somewhere): http://www.oracle.com/us/products/servers-storage/storage/nas/zfs-appliance-software/overview/index.html

From the vbox-2011.1.0.0.1.1.8 folder copy the Sun ZFS Storage 7000-disk1.vmdk file to the KVM host and convert to qcow2.

** Note my first attempt I used qcow and not qcow2 format and had issues starting the image so make sure and convert to qcow2.

# qemu-img convert "Sun ZFS Storage 7000-disk1.vmdk" -O qed SunZFSStorage7000-d1.qed
# image: SunZFSStorage7000-d1.qed
file format: qed
virtual size: 20G (21474836480 bytes)
disk size: 1.9G
cluster_size: 65536

Create Guest:
Create KVM guest. Use ide disk for SunZFSStorage7000-disk1.qed and specify qed format.

# virsh dumpxml ZfsApp

  ZfsApp
...

</pre>
<address> </address>
<pre>
...

</pre>
<address> </address>
<pre>
...

Boot new Virtual Machine from the sol-11_1-text-x86.iso. Choose language etc. Select shell when menu appears.

Update ZFS Image:

Now import and mount the ZFS file system.  Find the correct device name and update bootenv.rc:

In my case the disk device name for the boot disk is c7d0.  I use format to see the disk device name and then find the correct slice for the root partition.  You can use “par” and “pr” commands in format to see partitions.  In my case we are after /dev/dsk/c7d0s0 and we need to find the correct entry in the /device tree.

# format
Searching for disks...done

AVAILABLE DISK SELECTIONS:
0. c7d0
/pci@0,0/pci-ide@1,1/ide@0/cmdk@0,0
Specify disk (enter its number): ^C

# ls -l /dev/dsk/c7d0s0
lrwxrwxrwx 1 root root 50 May 26 10:35 /dev/dsk/c7d0s0 -> ../../devices/pci@0,0/pci-ide@1,1/ide@0/cmdk@0,0:a

From above I found the exact device name: /devices/pci@0,0/pci-ide@1,1/ide@0/cmdk@0,0:a

Lets go update the bootenv.rc now.

# zpool import -f system

# zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
system 19.9G 2.03G 17.8G 10% 1.00x ONLINE -

# zfs list | grep root
system/ak-nas-2011.04.24.1.0_1-1.8/root 1.26G 14.4G 1.25G legacy

# mkdir /a
# mount -F zfs system/ak-nas-2011.04.24.1.0_1-1.8/root /a

# zfs set readonly=off system/ak-nas-2011.04.24.1.0_1-1.8/root

# cp /etc/path_to_inst /a/etc
# vi /a/boot/solaris/bootenv.rc
...
setprop boot /devices/pci@0,0/pci-ide@1,1/ide@0/cmdk@0,0:a

# tail -1 /a/boot/solaris/bootenv.rc
setprop boot /devices/pci@0,0/pci-ide@1,1/ide@0/cmdk@0,0:a

# bootadm update-archive -R /a
updating /a/platform/i86pc/boot_archive
updating /a/platform/i86pc/amd64/boot_archive

# cd /
root@solaris:~/# umount /a
root@solaris:~/# zpool export system

# init 0

On next boot lets make sure Solaris detect the hardware correct.  When you see grub booting edit the kernel boot line and add “-arvs”.  Then continue booting.

Probably only need to add “-r” but I did “-arvs” to see more and also get into single user mode in case I needed to do more.

Once in single user mode with prompt just reboot.

For me at this point the image was booting into the zfs appliance setup and I could configure it.  Also on KVM adding SATA disks worked and the zfs appliance interface could use them for pools.

Update 04.23.14
I recently tried to update a simulator running on KVM but the update did not want to complete. Not sure if too many versions elapsed or something it does not like about being under KVM. Anyhow I tried moving an up to date (ak-2013.06.05.1.1) image on Virtualbox to KVM and that did work.

Dell Drac Console Repeating Keys

I have had issues using Dell Drac’s on some servers. It appears to be related to high latency network connections.

Sometimes it is important to login to a Hypervisor Linux console and it is impossible if your keys repeat and you try to login as rrrooooottttt. Believe me I thought it was mildly amusing trying to get logged in like that but no dice. Hint: You can’t see the password.

It is possible this issue could be related to the Windows client.  I was using a Windows 7 client in a Virtualbox VM and I am not 100% sure but someone else was able to connect from a KVM client without the keys repeating. I played with a lot of options to fix this and some people had luck with setterm on the command line.

setterm -repeat off

However I need the repeating keys fixed before login and not after we logged in so above would not work for me. Also I could not get setterm to work, no matter where I tried inserting the command during OS startup. Options are:
– Change the getty for the virtual console in inittab or wherever the OS runs getty. In our case with Ubuntu upstart in /etc/init/tty[1|8].conf. The idea is to pass the –I variable to getty with a command string.
– Run setterm in rc.local or wherever newer flavors of Linux want to run custom command on startup.
– Kernel parameter with atkbd.reset= or something similar although it sounded like the kernel developers gave up on this idea. I am not sure.

I finally had some success by sending raw codes to the terminal direct ie not using setterm. So for instance sending the virtual console a keycode to change the foreground color to green as follow:

# echo -en "\\033[32m\\033[8]" > /dev/tty1

The following command which appears is for DEC terminals did work.

Turn on:

# echo -en "\\033[?8h" > /dev/tty1

Turn off:

# echo -en "\\033[?8l" > /dev/tty1

More codes here: http://linux.about.com/library/cmd/blcmdl4_console_codes.htm

So I ended up adding below to INKVM001 to give us a green and no key repeat virtual console on tty4:

# tail -1 console-setup.conf
 exec echo -en "\\033[32m\\033[8\\033[?8l" > /dev/tty4

To access different virtual console you just use Alt-F# when in text mode and Ctrl-Alt-F# when in a graphics mode. In this case the Drac menu can send the an Alt-F4 macro and you will have a safe (and green) login terminal which will not drive you innnnsssssannnnneeeee.

Other commands I was playing with just for reference:

# dpkg-reconfigure keyboard-configuration

setterm -blank 0 -powerdown 0 -powersave off > /dev/tty${index}

exec /sbin/getty -I `setterm -clear all -foreground yellow -bold -store` -8 38400 tty1

# kbdrate -d 1000
 Typematic Rate set to 10.9 cps (delay = 1000 ms)

Links:
http://www.kernel.org/doc/Documentation/kernel-parameters.txt
http://www.win.tue.nl/~aeb/linux/kbd/scancodes-12.html
http://luv.asn.au/overheads/virtualconsoles.html
http://linux.about.com/library/cmd/blcmdl4_console_codes.htm
http://www.linuxquestions.org/questions/linux-general-1/setting-terminal-colors-with-setterm-718290/
http://shallowsky.com/blog/linux/install/ctrl-key-debian-squeeze.html
https://bugzilla.redhat.com/show_bug.cgi?id=181457
http://www.ibiblio.org/pub/Linux/docs/HOWTO/Keyboard-and-Console-HOWTO

Python-ldap Query MS Active Directory

I use Python to pull Active Directory information sometimes. In my case mainly to report or view information but also to create files in a LDIF or PowerShell format.  These can be manually run on the Domain Controller later. For instance find all users in a Distribution List or Group and create a rule or ldif entry that can manually be executed line by line. Off course there is also ways with PowerShell and vbscript to do this, but I prefer Python for text manipulation and it is not too cumbersome for me to batch run these files manually later.

I noticed on Ubuntu 12.10 that my query failed with the following error:

ldap.LDAP_CONTROL_PAGE_OID,True,(page_size,'')
AttributeError: 'module' object has no attribute 'LDAP_CONTROL_PAGE_OID'

I found a comment from the developers saying that with python-ldap 2.4
“there have been changes in the API for LDAPv3 extended controls. Please see Demo/page_control.py (and Demo/paged_search_ext_s.py) how to use the simple paged control with 2.4.”

I found the source here: http://python-ldap.cvs.sourceforge.net/viewvc/python-ldap/python-ldap/Demo/page_control.py?view=log

I made a new script and tested as shown below:
** Note the following first:

  1. This is a “paged” query meaning you should not have issues if LDAP return a limted number of results.  More detail here: http://support.microsoft.com/kb/315071
  2. You need to install python-ldap of course.  apt-get install python-ldap should work on apt systems.
  3. Check comment about using 512 as userAccountControl.
DistributionList = "CN=IT Infrastructure,CN=Distribution Lists,CN=Users,DC=domain,DC=com"

url = "ldap://usdc101"
base = "dc=domain,dc=com"
#  search_flt = r'(objectClass=*)'
##  I used userAccountControl=512 but that would most certainly exclude some 
##  user accounts in your domain. For instance "66048 Enabled, Password Doesn't Expire"
##  values listed here: http://www.netvision.com/ad_useraccountcontrol.php
search_flt = r'(&(objectCategory=user) (userAccountControl=512) )'

page_size = 10

import ldap,pprint
from ldap.controls import SimplePagedResultsControl

searchreq_attrlist=["displayName","cn","distinguishedName","mail","memberOf"]

ldap.set_option(ldap.OPT_REFERRALS, 0)
l = ldap.initialize(url,trace_level=0)
l.protocol_version = 3
l.simple_bind_s("ADaccount@domain.com", "passsword")

req_ctrl = SimplePagedResultsControl(True,size=page_size,cookie='')

known_ldap_resp_ctrls = {
SimplePagedResultsControl.controlType:SimplePagedResultsControl,
}

# Send search request
msgid = l.search_ext(
base,
ldap.SCOPE_SUBTREE,
search_flt,
attrlist=searchreq_attrlist,
serverctrls=[req_ctrl]
)

pages = 0
i = 0
print "listing users in the list:" + DistributionList

while True:
  pages += 1
  rtype, rdata, rmsgid, serverctrls = l.result3(msgid,resp_ctrl_classes=known_ldap_resp_ctrls)

  for dn, entry in rdata:
    ##  Lets check if the user is a member of the AD List / Group
    try:
      membership = entry['memberOf']
    except:
      membership = 'none'
    if DistributionList in membership:
      i += 1
      print " \"%d\" | \"%s\" " % (i , dn)

  pctrls = [
    c
    for c in serverctrls
    if c.controlType == SimplePagedResultsControl.controlType
  ]
  if pctrls:
    if pctrls[0].cookie:
      # Copy cookie from response control to request control
      req_ctrl.cookie = pctrls[0].cookie
      msgid = l.search_ext(
        base,
        ldap.SCOPE_SUBTREE,
        search_flt,
        attrlist=searchreq_attrlist,
        serverctrls=[req_ctrl]
      )
    else:
      break
  else:
    print "Warning: Server ignores RFC 2696 control."
    break

l.unbind_s()

KVM VM Rename and Logical Volume Rename

I recently provisioned a Virtual Machine in a KVM Hypervisor with a typo in the name. Even worse I also created the storage LV with a typo. Here is how I fixed it. Off course shut the VM down first.

LV Rename

# lvrename VolumeGroup oldLVname newLVname
  Renamed "oldLVname" to "newLVname" in volume group "VolumeGroup"

Change reference to storage in VM XML

# virsh edit oldVMname
Domain oldVMname XML configuration edited.

# grep newLVname oldVMname.xml 
      

Recreate VM

# virsh dumpxml oldVMname > /tmp/newVMname.xml

# virsh undefine oldVMname
Domain newVMname has been undefined

# grep newVMname /tmp/newVMname.xml 
  newVMname
      

# virsh define /tmp/newVMname.xml 
Domain newVMname defined from /tmp/newVMname.xml

# virsh list --all
 Id Name                 State
----------------------------------
  7 aVMname            running
  - newVMname         shut off

# virsh start newVMname
Domain newVMname started