KVM & PXE: virtual machine setup automation

What is virtualization ?

Since the 1960s and mainframe computers, computer scientists have made huge progress on virtualization. However until late 1990s, this was out of fashion. Popek and Goldberg defined it as “an efficient and isolated duplicate of a real machine”. Nowadays, even if some are still reluctant to the idea, virtual machines are present everywhere in the IT world. Their purposes are various: development environment, production isolation, testing bed, etc. A lot of people and company are using them on daily basis. In some infrastructures, administrators are using hypervisors, operating system that are made specially to run virtual machine on it. We could discuss if it is a good or a bad to virtualize everything but this is not the subject here. A lot of software solutions are available to create virtual machines, all using different technologies and way of emulate computers: some very professional like VMWare, some more general public like VirtualBox, some generate a lot of hype like Docker. One of my favorite is KVM.

What is KVM ?

KVM stands for Kernel-based Virtual Machine, it is a software that turns a Linux kernel into an hypervisor and rely on QEMU to perform the hardware virtualization. We will use it to create and run our virtual machines. In order to easily manage our virtual machine, we will use Virt-Manager. That tool will provides us a GUI to interact with KVM, locally or remotely. We will use another too call Virsh too, which is a similar tool but with a CLI. These tools need to be interfaced with KVM: Lib-virt will handle that. It is a virtualization API used to provide interfaces with various virtualization solutions, including KVM.

This proof of concept will be setup on Debian 9. It should be faily easy to adapt it to recent Ubuntu version.

Goal

The purpose of this article is simple: find a way to create a virtual machine from scratch easily and automate the installation process. In my personal workflow, I create and trash a lot of Virtual Machine to test different deployments on my personal computer. It is very cumbersome to constantly create and setup a virtual machine. A solution to spend less time doing this could be to clone the virtual machine once setup at a blank state and start from there. But I thought of another solution, not very ideal but that could be interesting to test.

Solution

We will use KVM with Virt-Manager in order to create and manage our virtual machine. We will setup our network with a bridge to make it appear to other computers on your local network like a physical machine. Then we will setup a PXE boot with TFTP and Preseed to automate the operating system installation. Our objective will be to automate the setup of a Debian Jessie x64. This is an arbitrary choosen systems, you can adapt the PXE configuration to any operating systems but the installation automation rely on Preseed, which is only available on Debian, so you need to find a similar software for your system if you don’t use Debian-like system.

Requirements

First, let’s check if our processor is compatible.

egrep '^flags.*(vmx|svm)' /proc/cpuinfo

/proc/cpuinfo is a file used to describe the current processor. It contains a lot of useful informations. One of them is flags: this field contains abreviations that describe available processor features. Your processor needs to have smx or vmx if you want to use KVM. Simply because smx is a flag for AMD virtualization extention and vmx for Intel virtualization extension: if you don’t have one of those extentions, your processor is not capable to handle virtualization. If you have a recent cpu, it should not be an issue.

So if nothing is printed, your processor is not compactible. If you have something similar to this, you can continue:

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm arch_perfmon pebs bts nopl xtopology aperfmperf eagerfpu pni pclmulqdq dtes64 monitor vmx

Install KVM

If you met the previous requirements we should proceed and install KVM. Use your favourite package manager to install the following packages:

sudo apt-get install kvm qemu-kvm libvirt-bin virtinst virt-manager

Now we need to add our current user to the libvirtd group.

sudo usermod -a -G libvirtd $USER

If everything is alright, you can use KVM by now. Let’s try a simple command that will list every virtual machine you have created with Virsh. Since you have not create any, the list should empty. If there is no error, you can go on.

virsh -c qemu:///system list

The point of this article is not to show you how to use KVM or virt-manager, you can find a lot of ressources by browsing on the Internet or simply reading the documention.

Setup KVM network bridge

Before setting up the PXE, we need to change our network. By default, the virtual machine networking is set to NAT. Like your computer behind your Internet gateway, the virtual machine will be hidden behind the host IP to communicate with external network, and PXE cannot work in such configuration. That’s why we will configure our network card as bridge.

Some of the following commands will configure the eth0 interface, you can change eth0 by wlan0 if you use your Wifi interface to connect to your local network.

Open /etc/network/interfaces and make that changes :

auto lo
iface lo inet loopback

auto wlan0
iface wlan0 inet manual

auto eth0
iface eth0 inet manual

auto br0
iface br0 inet static
  bridge_ports eth0
  address 10.250.12.254
  netmask 255.255.255.0
  broadcast 10.250.12.255
  bridge_stp off
  bridge_fd 0
  bridge_maxwait 0

Then reload your networking service.

sudo systemctl restart networking

Now your virtual machines can use that bridge in order to be connected but they cannot connect to the Internet. You will need to allow package forwarding and enable masquerading. First, run this command:

sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

Then add the following line in /etc/sysctl.conf:

net.ipv4.conf.all.forwarding = 1

And reload it:

sysctl -p

You are now all set with KVM. Shall we continue ?

Preboot eXecution Environment

Now, you can manually create your virtual machine, mount an operating system ISO and install it. By the end of this installation, your virtual machine will appear as a physical host for other computers. But we don’t want to do it that way, we don’t want to have to choose the ISO every time we want to create a new virtual machine and go through each installation step manually. We need automation.

So let’s setup PXE to provide our virtual machine some netinstall images. PXE boot, or Pre-boot eXecution Environment, is an environment where the machine boots and downloads an operating system image from the network, so you don’t need to mount an ISO.

Before starting to write the PXE configuration, we need to setup the DHCP and the TFTP server. The first tool will provides dynamic IP for our virtual machines and tell them where to find the netinstall images and the second will give them access to these files. An HTTP server can be used instead of TFTP.

Setup DHCP server

You can install this dhcp server using your package manager.

sudo apt-get install isc-dhcp-server

Then edit the file /etc/dhcp/dhcpd.conf.

option domain-name "chafouin.org";
option domain-name-servers 8.8.8.8, 8.8.4.4;

default-lease-time 600;
max-lease-time 7200;
server-name "dhcp-server";

ddns-update-style none;

authoritative;

allow bootp;
allow booting;

subnet 10.250.12.0 netmask 255.255.255.0 {
  range 10.250.12.100 10.250.12.200;
  option subnet-mask 255.255.255.0;
  option broadcast-address 10.250.12.255;
  option routers 10.250.12.254;
  ping-check = 1;
  filename "pxelinux.0";
  next-server 10.250.12.254;
}

Our DHCP will provides network configurations to our virtual machines on our bridge. As you can see, I chose the ip range from 10.250.12.100 to 10.250.12.200 for our virtual machine, it should be enough. Our host is on 10.250.12.254 and will be our gateway. The ‘next-server’ field contains the TFTP server address and the filename "pxelinux.0" option will tell to the client to look for this file in order to boot on the PXE, we will see that later.

Setup TFTP server

As I said before. we need to install a TFTP server. It will provides the netinstall files to our machines. As usual, use your favourite package manager.

sudo apt-get install tftpd-hpa

Then edit the /etc/default/tftpd-hpa to add these changes:

# /etc/default/tftpd-hpa

TFTP_USERNAME="root"
TFTP_DIRECTORY="/srv/tftp"
TFTP_ADDRESS="0.0.0.0:69"
TFTP_OPTIONS="--secure"

The TFTP server is now ready to serve the image. We just need to prepare the path. Now you can get your netinstall files from your favourite distro official website or create a custom one. In our example, let’s get the Debian netinstall.

We should create some new folders where we will store our PXE configurations files and operating system images.

mkdir -p /srv/tftp/config/bootscreens /srv/tftp/config/debian /srv/tftp/installers/debian/jessie/amd64

Get the Debian netboot install from the official server and extract the files in our TFTP directory.

curl -O http://ftp.debian.org/debian/dists/jessie/main/installer-amd64/current/images/netboot/netboot.tar.gz
tar xvf netboot.tar.gz
mv debian-installer/amd64/pxelinux.0 /srv/tftp/pxelinux.0
mv debian-installer/amd64/pxelinux.cfg /srv/tftp
mv debian-installer/amd64/boot-screens/vesamenu.c32 /srv/tftp
mv debian-installer/amd64/initrd.gz debian-installer/amd64/linux /srv/tftp/installers/debian/jessie/amd64

We have our installer setup, now we can start to create our configuration.

Open a new file called /srv/tftp/pxelinux.cfg/default and paste the following content.

default vesamenu.c32

menu title PXE installer menu
menu background config/splash.png

menu hshift 0
menu vshift 7
menu width 50
menu margin 0

# Colors
menu color title  * #FFFFFFFF *
menu color border * #00000000 #00000000 none
menu color sel    * #ffffffff #76a1d0ff *
menu color hotsel 1;7;37;40 #ffffffff #76a1d0ff *
menu color tabmsg * #ffffffff #00000000 *
menu color help   37;40 #ffdddd00 #00000000 none

# Text position
menu helpmsgrow 15
menu cmdlinerow 16
#menu timeoutrow 16
menu tabmsgrow 14
menu tabmsg Choose the OS you want install
# No prompt, no timeout
prompt 0
timeout 100

# Deactivate escape keyboard and disallow options
noescape 1
allowoptions 0

include config/menu.cfg

This file contains PXE menus general parameters, read the coments to understand what it does. Then we need to describe our first menu.

Create the file /srv/tftp/config/menu.cfg

label local
    menu label Boot on local drive
    menu default
    localboot 0
menu begin debian
    menu title ^Debian Install
    include config/debian/menu.cfg
menu end

This file will define the menu displayed when you will start the PXE boot that will be used to select the operating system you want to install.

This first part with ‘label local’ will display an option to start the virtual machine directly on its attached hard drive. This option could be useful if you start a virtual machine that already have been installed.

The second part with ‘menu begin debian’ will display an option to open the Debian submenu, since we are trying to build something modular. This submenu will be descibe below. We will make only one configuration for Debian Jessie x64 but you can duplicate this configuration and customize it in order to build your own Debian version with x86, Wheezy or anything else.

Let’s write the second menu. It will contains the kernel files path and kernel command line options that PXE will load on our virtual machine. Finally open the file /srv/tftp/config/debian/menu.cfg.

label install
    menu label Debian Jessie ^64 bits
    kernel installers/debian/jessie/amd64/linux
    append vga=normal initrd=installers/debian/jessie/amd64/initrd.gz url=tftp://10.250.12.254/config/debian/./debian-jessie-preseed.cfg auto=true priority=critical interface=eth0
label separator
    menu label -----
label mainmenu
    menu label ^Back..
    menu exit

Here some explanations:

  • kernel : path to the kernel images
  • append : add options to the kernel command line

Append command explanations:

  • vga=normal : select the normal display mode on boot
  • initrd=installers/debian/jessie/amd64/initrd.gz : path to the initrd file
  • url=tftp://10.250.12.254/config/debian/./debian-jessie-preseed.cfg : url to a preconfiguration file to download
  • auto=true : start auto installer
  • priority=critical : stop any low priority questions from being asked
  • interface=eth0 : network interface setup before installation

Preseed

By now you should be capable to start create and start your virtual machine with virt-manager or virsh, navigate on the PXE interface, choose an operating system then install it manually. That’s great but not enought. We need a last step. We need more automation. Why? Because it saves your time and your money, reduces misconfiguration problems and it is less cumbersome. Here, we got 2 solutions to fix these issues: automate installation or clone already installed machine. We won’t use the cloning method. We will use Preseed, it will provides the installation automation we need here. Preseed is only available on Debian but you can find similar software for your OS, like Kickstart for CentOS or Remote Installation Service for Windows.

Let’s start to talk about the Preseed. this is a file used by the netboot installer in order to load the installation configuration. With that, no more user input is needed and the installation will be fully automated.

Let’s create a new file in /srv/tftp/config/debian/debian-jessie-preseed.cfg.

#### Contents of the preconfiguration file (for Debian Jessie)
### Localization
# Preseeding only locale sets language, country and locale.
d-i debian-installer/language string en
d-i debian-installer/country string FR
d-i debian-installer/locale string en_US.UTF-8

# Keyboard selection.
d-i keyboard-configuration/xkb-keymap select us

### Network configuration
# netcfg will pick eth0 as interface.
d-i netcfg/choose_interface select eth0

# Any hostname and domain names assigned from dhcp take precedence over
# values set here. However, setting the values still prevents the questions
# from being shown, even if values come from dhcp.
d-i netcfg/get_hostname string unassigned-hostname
d-i netcfg/get_domain string unassigned-domain

### Mirror settings
# If you select ftp, the mirror/country string does not need to be set.
#d-i mirror/protocol string ftp
d-i mirror/country string manual
d-i mirror/http/hostname string ftp.fr.debian.org
d-i mirror/http/directory string /debian
d-i mirror/http/proxy string

# Suite to install.
#d-i mirror/suite string testing
# Suite to use for loading installer components (optional).
#d-i mirror/udeb/suite string testing

### Account setup
# Skip creation of a root account (normal user account will be able to
# use sudo).
#d-i passwd/root-login boolean false
# Alternatively, to skip creation of a normal user account.
d-i passwd/make-user boolean false

# Root password, either in clear text
d-i passwd/root-password password toor
d-i passwd/root-password-again password toor
# or encrypted using an MD5 hash.
#d-i passwd/root-password-crypted password 5a3498520d8994fab848bf76299127b9

### Clock and time zone setup
# Controls whether or not the hardware clock is set to UTC.
d-i clock-setup/utc boolean true

# You may set this to any valid setting for $TZ; see the contents of
# /usr/share/zoneinfo/ for valid values.
d-i time/zone string Europe/Paris

# Controls whether to use NTP to set the clock during the install
d-i clock-setup/ntp boolean true
# NTP server to use. The default is almost always fine here.
d-i clock-setup/ntp-server string pool.ntp.org

### Partitioning
d-i partman-auto/method string lvm
d-i partman-lvm/device_remove_lvm boolean true
d-i partman-md/device_remove_md boolean true
d-i partman-lvm/confirm boolean true
d-i partman-lvm/confirm_nooverwrite boolean true
d-i partman-auto/choose_recipe select atomic

# This makes partman automatically partition without confirmation, provided
# that you told it what to do using one of the methods above.
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true

# This makes partman automatically partition without confirmation.
d-i partman-md/confirm boolean true
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true

## Controlling how partitions are mounted
# The default is to mount by UUID, but you can also choose "traditional" to
# use traditional device names, or "label" to try filesystem labels before
# falling back to UUIDs.
#d-i partman/mount_style select uuid

### Base system installation
# Configure APT to not install recommended packages by default. Use of this
# option can result in an incomplete system and should only be used by very
# experienced users.
d-i base-installer/install-recommends boolean false

### Apt setup
# You can choose to install non-free and contrib software.
d-i apt-setup/non-free boolean true
d-i apt-setup/contrib boolean true
d-i apt-setup/use_mirror boolean false
d-i apt-setup/services-select multiselect security, updates
d-i apt-setup/security_host string security.debian.org

### Package selection
tasksel tasksel/first multiselect minimal

# Individual additional packages to install
d-i pkgsel/include string openssh-server build-essential curl
# Whether to upgrade packages after debootstrap.
# Allowed values: none, safe-upgrade, full-upgrade
d-i pkgsel/upgrade select full-upgrade

# Some versions of the installer can report back on what software you have
# installed, and what software you use. The default is not to report back,
# but sending reports helps the project determine what software is most
# popular and include it on CDs.
popularity-contest popularity-contest/participate boolean false

### Boot loader installation
# Grub is the default boot loader (for x86). If you want lilo installed
# instead, uncomment this:
#d-i grub-installer/skip boolean true
# To also skip installing lilo, and install no bootloader, uncomment this
# too:
#d-i lilo-installer/skip boolean true


# This is fairly safe to set, it makes grub install automatically to the MBR
# if no other operating system is detected on the machine.
d-i grub-installer/only_debian boolean true

# This one makes grub-installer install to the MBR if it also finds some other
# OS, which is less safe as it might not be able to boot that other OS.
d-i grub-installer/with_other_os boolean true

# Due notably to potential USB sticks, the location of the MBR can not be
# determined safely in general, so this needs to be specified:
d-i grub-installer/bootdev  string /dev/vda

# Avoid that last message about the install being complete.
d-i finish-install/reboot_in_progress note

# This will prevent the installer from ejecting the CD during the reboot,
# which is useful in some situations.
d-i cdrom-detect/eject boolean false

# This will power off the machine instead of just halting it.
d-i debian-installer/exit/poweroff boolean true

#### Advanced options
### Running custom commands during the installation
# d-i preseeding is inherently not secure. Nothing in the installer checks
# for attempts at buffer overflows or other exploits of the values of a
# preconfiguration file like this one. Only use preconfiguration files from
# trusted locations! To drive that home, and because it's generally useful,
# here's a way to run any shell command you'd like inside the installer,
# automatically.

# This first command is run as early as possible, just after
# preseeding is read.
#d-i preseed/early_command string anna-install some-udeb
# This command is run immediately before the partitioner starts. It may be
# useful to apply dynamic partitioner preseeding that depends on the state
# of the disks (which may not be visible when preseed/early_command runs).
#d-i partman/early_command \
#       string debconf-set partman-auto/disk "$(list-devices disk | head -n1)"
# This command is run just before the install finishes, but when there is
# still a usable /target directory. You can chroot to /target and use it
# directly, or use the apt-install and in-target commands to easily install
# packages and run commands in the target system.
d-i preseed/late_command string \
in-target sed -i 's/PermitRootLogin without-password/PermitRootLogin yes/g' /etc/ssh/sshd_config

Everything is already commented, I don’t think you need more explanation. You should be able to create a new virtual machine and start it by now.

Choose Debian Jessie x64 installation on the PXE menu and let the magic begin ! No installation prompt anymore, installation screens follow the others without a single prompt waiting for you and everything run smoothly, that’s perfect ! Wait few minutes, your virtual machine will reboot and will be ready.

In the end, your /srv/tftp tree should look like that :

.
├── config
│   ├── debian
│   │   ├── debian-jessie-preseed.cfg
│   │   └── menu.cfg
│   └── menu.cfg
├── installers
│   └── debian
│       └── jessie
│           └── amd64
│               ├── initrd.gz
│               └── linux
├── pxelinux.0
├── pxelinux.cfg
│   └── default
└── vesamenu.c32

This tree is maybe not the state of the art but it is modular and simple, it allows us to organize efficiently our images. You can of course find a better way to organize these folders.

You can face an issue where PXE will tell you that ldlinux.c32, libcom32.c32 or libutil.c32 cannot be load, you can fix it with this kind of command.

cp /usr/lib/syslinux/modules/bios/ldlinux.c32 /srv/tftp/
cp /usr/lib/syslinux/modules/bios/libcom32.c32 /srv/tftp/
cp /usr/lib/syslinux/modules/bios/libutil.c32 /srv/tftp/

These commands require syslinux, you can easily install it with :

apt-get install syslinux

Virsh command example

Here is some virsh command I have dumped for you, you can use them to interact with the virtual machine.

You can create a new storage with these commands:

virsh vol-create-as default /var/lib/libvirt/images/debian8_01.qcow2 10GiB --prealloc-metadata --format qcow2
qemu-img create -f qcow2 -o preallocation=metadata /var/lib/libvirt/images/debian8_01.qcow2 10G

Then get a Debian minimal ISO:

curl -L http://cdimage.debian.org/debian-cd/8.6.0/amd64/iso-cd/debian-8.6.0-amd64-netinst.iso -o /var/lib/libvirt/images/debian-8.6.0-amd64-netinst.iso

And create a virtual machine with this one:

virt-install --connect qemu:///system --name debian8_01 --network=bridge:br0 --pxe --ram=2048 --vcpus=2 --check-cpu --os-type=linux --os-variant=generic --disk path=/var/lib/libvirt/images/debian8_01.qcow2 -c /var/lib/libvirt/images/debian-8.6.0-amd64-netinst.iso

Or you could create a new virtual machine using XML file, extracting the XML from an existing VM:

virsh dumpxml <vm_name> > vm.xml
virsh create vm.xml

Don’t forget edit the XML to change the UUID and MAC address field.

If you want to install the virtual machine manually and you don’t want to use to virt-manager GUI, you can still use spice server:

spicec -h 127.0.0.1 -p 5900

Shift + f12 to go out of spice.

Or to connect remotely with SSH, you could use this .ssh/config file.

Host vm
  Hostname 10.250.12.100
  User root
  Port 1535
  ForwardAgent yes
  LocalForward 5900 localhost:5900

If you use an ISO CD, you can eject it after the installation with this:

virsh change-media vm hda --eject

Conclusion

This is the first conclusion of my first article on my first blog. What an achievement. Here is a summary of what you have learn:

  • Install a complete KVM environement
  • Setup a PXE boot
  • Create a preseed file

What’s next you will ask. Check out those few references I put just below, these are very interesting things if you want to go understand better how KVM & PXE works, maybe find another point of view or others explanations that are more clear for you. I based my article on these references and my actual knowledge, I hope these will help you too. See you next time.

References

https://debian-administration.org/article/394/Automating_new_Debian_installations_with_preseeding

http://moss.csc.ncsu.edu/~mueller/cluster/arc/kvm_vm_creation.html

https://debian-administration.org/article/478/Setting_up_a_server_for_PXE_network_booting

https://wiki.debian.org/PXEBootInstall