Installing XenServer and Virtual Hosts in a Routed Network
A few days ago, we had to set up a XenServer Host, running on one of Hetzners dedicated servers. There are plenty of howto’s around, but since we happened to run into a few problems with routed networking and Domain Name Resolution during installation, that weren’t documented in Hetzners Wiki, i decided to provide a blog post for self-reference and as a way to contribute a (hopefully helpful) basic guide on how to install XenServer on a dedicated server – in our Case a Hetzner Root Server.
References:
- When i talk about the host system or simply the host, i am referring to the dedicated server itself.
- When i talk about the guest system, the guest or simply the VM, i refer to the virtual machines running on the Host system.
IP-Addresses and Networks
By default, Hetzners servers come with a single IP-address. Make sure to write down your network configuration from your originally installed base system (or boot up your dedicated server into a rescue system). You need that information during the installation of your base system! Also, when you plan on running externally accessible guest systems on your XenServer, you need to order additional IPs (you can either order a single IP or order a whole subnet with Hetzner). Keep in mind that Hetzners network setup delivers traffic only to the mac-address of the physical interface, so we have to do some trickery to set up the network for our VMs later on.
Downloading the installation image
First of all, download the current XenServer installation image: http://xenserver.org/open-source-virtualization-download.html. We decided to install our host system, using LARA (Hetzners remote console, providing KVM-over-IP). To mount the ISO installation image in LARA, click on the “Interfaces” tab in LARAs login page, then go to the “Virtual Media” link. Here you can enter the details of how you want to mount your image:
Mounting the ISO file and installing the base system
To access the virtual media interface please click on the “Interfaces” tab and then go to the “Virtual Media” link. Here you can enter the details of the location of the ISO file: You can mount it either via the SAMBA/CIFS networking protocol from your backup space or from your own server (as long as it is reachable via SAMBA/CIFS), or mount it directly in your LARA (although this isn’t officially supported by Hetzner).
Mount your ISO via SAMBA/CIFS:
Under Interfaces > Virtual Media > Choose “Image on Windows Share”:
Share Host/IP: <username>.your-backup.de
Share Name: backup
Image File with Path: <file_name>
User Name: <username>
Password: <password>
<user name> is the username of your backup space, which you can found under “Backup” in Hetzners robot.your-server.de
<file_name> is the name of your ISO installation image
<password> is the password that you have setup in your backup space.
Mount your ISO directly in LARA:
In your LARA, click on the floppy symbol on the top right and navigate to your local ISO installation image.
You can verify under Interfaces > Virtual Media > “Virtual Media Active Image” if your ISO is correctly mounted.
After rebooting your server, you should see the installation wizard coming up. It’s pretty easy from here, just follow through the the installation process, no biggies here…
Checking your raid configuration
To monitor your LSI raid controller with megactl, install MegaCLI RPM (available from http://download.hetzner.de/tools/LSI/ and other sources) on your base system:
[root@XenServer-Host ~]# unzip 8.07.14_MegaCLI.zip [root@XenServer-Host ~]# rpm -ivh Linux/MegaCli-8.07.14-1.noarch.rpm ... [root@XenServer-Host ~]#
Verify the installation by checking your raid status:
[root@XenServer-Host ~]# /opt/MegaRAID/MegaCli/MegaCli LDInfo -Lall -Aall Adapter 0 -- Virtual Drive Information: Virtual Drive: 0 (Target Id: 0) Name : RAID Level : Primary-1, Secondary-0, RAID Level Qualifier-0 Size : 1.818 TB Sector Size : 512 Is VD emulated : No Mirror Data : 1.818 TB State : Optimal Strip Size : 256 KB Number Of Drives : 2 Span Depth : 1 Default Cache Policy: WriteBack, ReadAhead, Cached, Write Cache OK if Bad BBU Current Cache Policy: WriteBack, ReadAhead, Cached, Write Cache OK if Bad BBU Default Access Policy: Read/Write Current Access Policy: Read/Write Disk Cache Policy : Disk's Default Encryption Type : None Bad Blocks Exist: No Is VD Cached: No Exit Code: 0x00 [root@XenServer-Host ~]#
Updating the base system
Updates are available from http://xenserver.org/open-source-virtualization-download.html. Citrix provides an official YUM repo, but for some reason it’s empty, so that leaves us with installing updates manually. Make sure to install all updates in the right order and reboot your host system when necessary!
For example: [root@XenServer-Host ~]# mkdir XS62ESP1014 [root@XenServer-Host ~]# cd XS62ESP1014 [root@XenServer-Host ~]# wget http://downloadns.citrix.com.edgesuite.net/akdlm/9708/XS62ESP1014.zip [root@XenServer-Host ~]# unzip XS62ESP1014.zip [root@XenServer-Hoste ~]# xe patch-upload file-name=XS62ESP1014.xsupdate [root@XenServer-Host ~]# xe patch-apply uuid=4fc82e62-b938-407d-a2c6-68c8922f3ec2 host-uuid=099a87af-0c80-4ea2-a406-f8a1bdf1e2a1 [root@XenServer-Host ~]# cd .. [root@XenServer-Host ~]# rm -Rf XS62ESP1014 [root@XenServer-Host ~]# reboot
Configuring a network interface for your guest systems
We ordered a dedicated Subnet for our guest systems, and added the first IP as an alias device to xenbr0:
[root@XenServer-Host ~]# cat /etc/sysconfig/network-scripts/ifcfg-xenbr0:1 DEVICE=xenbr0:1 ONBOOT=yes BOOTPROTO=none NETMASK=<netmask> IPADDR=<first_subnet_ip> [root@XenServer-Host ~]# ifup xenbr0:1 [root@XenServer-Host ~]# ifconfig eth0 Link encap:Ethernet HWaddr <hwaddress> UP BROADCAST RUNNING PROMISC MTU:1500 Metric:1 RX packets:38850 errors:0 dropped:0 overruns:0 frame:0 TX packets:26088 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:4238740 (4.0 MiB) TX bytes:3740798 (3.5 MiB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:2083 errors:0 dropped:0 overruns:0 frame:0 TX packets:2083 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:1065579 (1.0 MiB) TX bytes:1065579 (1.0 MiB) xenbr0 Link encap:Ethernet HWaddr <hwaddress> inet addr:<host_ip_address> Bcast:<broadcast> Mask:<netmask> UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 RX packets:45603 errors:0 dropped:0 overruns:0 frame:0 TX packets:31564 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:5305346 (5.0 MiB) TX bytes:4642674 (4.4 MiB) xenbr0:1 Link encap:Ethernet HWaddr <hwaddress> inet addr:<first_subnet_ip> Bcast:<broadcast> Mask:<netmask> UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 [root@XenServer-Host ~]#
Activate package forwarding
With routed setups (like with Hetzner, as mentioned earlier in this post), we need to set up package forwarding!
[root@XenServer-Host ~]# cat /etc/sysctl.conf # Kernel sysctl configuration file for Red Hat Linux # # For binary values, 0 is disabled, 1 is enabled. See sysctl(8) and # sysctl.conf(5) for more details. # Controls IP packet forwarding net.ipv4.ip_forward = 1 #net.ipv6.conf.all.forwarding=1 # Controls proxy arp net.ipv4.conf.default.proxy_arp = 0 # Turn off redirects net.ipv4.conf.all.send_redirects = 0 net.ipv4.conf.default.send_redirects = 0 net.ipv4.conf.lo.send_redirects = 0 net.ipv4.conf.xenbr0.send_redirects = 0 # Controls source route verification net.ipv4.conf.default.rp_filter = 0 # Enable broadcast echo Protection net.ipv4.icmp_echo_ignore_broadcasts = 1 # Do not accept source routing net.ipv4.conf.default.accept_source_route = 0 # Disable ICMP Redirect Acceptance net.ipv4.conf.all.accept_redirects = 0 # Don.t send Redirect Messages net.ipv4.conf.all.send_redirects = 0 # ARP replies only on single interface with corresponding IP address net.ipv4.conf.all.arp_filter = 1 # Disable *tables rules for bridge traffic to increase performance net.bridge.bridge-nf-call-iptables = 0 net.bridge.bridge-nf-call-ip6tables = 0 net.bridge.bridge-nf-call-arptables = 0 # Controls the System Request debugging functionality of the kernel kernel.sysrq = 1 # Controls whether core dumps will append the PID to the core filename # Useful for debugging multi-threaded applications kernel.core_uses_pid = 1 # Controls the use of TCP syncookies net.ipv4.tcp_syncookies = 1 # Controls the maximum size of a message, in bytes kernel.msgmnb = 65536 # Controls the default maxmimum size of a mesage queue kernel.msgmax = 65536 # Controls the maximum shared segment size, in bytes kernel.shmmax = 4294967295 # Controls the maximum number of shared memory segments, in pages kernel.shmall = 268435456 # Controls the percentage of system memory reserved for dirty pages vm.dirty_ratio = 5 # Mimimum priority of kernel message to log kernel.printk = 4 4 1 4 # Maximum io_events: 2048 tapdisks, 32k headroom fs.aio-max-nr = 1048576 # Panic on soft lockups. kernel.softlockup_panic = 1 [root@XenServer-Host ~]# sysctl -p net.ipv4.ip_forward = 1 net.ipv4.conf.default.proxy_arp = 0 net.ipv4.conf.all.send_redirects = 0 net.ipv4.conf.default.send_redirects = 0 net.ipv4.conf.lo.send_redirects = 0 net.ipv4.conf.xenbr0.send_redirects = 0 net.ipv4.conf.default.rp_filter = 0 net.ipv4.icmp_echo_ignore_broadcasts = 1 net.ipv4.conf.default.accept_source_route = 0 net.ipv4.conf.all.accept_redirects = 0 net.ipv4.conf.all.send_redirects = 0 net.ipv4.conf.all.arp_filter = 1 error: "net.bridge.bridge-nf-call-iptables" is an unknown key error: "net.bridge.bridge-nf-call-ip6tables" is an unknown key error: "net.bridge.bridge-nf-call-arptables" is an unknown key kernel.sysrq = 1 kernel.core_uses_pid = 1 net.ipv4.tcp_syncookies = 1 kernel.msgmnb = 65536 kernel.msgmax = 65536 kernel.shmmax = 4294967295 kernel.shmall = 268435456 vm.dirty_ratio = 5 kernel.printk = 4 4 1 4 fs.aio-max-nr = 1048576 kernel.softlockup_panic = 1 [root@XenServer-Host ~]#
Configuring the firewall
To allow our subnet to communicate outside the host system, we need to update some firewall settings. We learned during the installation of our guest system, that we failed with the netinstall routine: when it came to choosing a debian mirror, the installation wizard failed to resolve all available mirrors, which was quite strange since we had network connectivity. Basically, we could connect to and from the base installation with ssh to any external ip – proving that our routing setup worked perfectly fine. However, we could not resolve any domain names. All we got when trying to ping or resolve a domain name was a not-so-helpfull error message stating “bad domain name”. After some troubleshooting, we found out that XenServer firewalls network communication to and from the guest systems: Per default, only communication on ports 22, 80 and 443 (ssh, http, https) are allowed. Any other communication, including DNS (Port 53) is blocked, therefore causing Debian netinstall to fail.
To fix this, we added a few iptables rules:
[root@XenServer-Host ~]# iptables -I RH-Firewall-1-INPUT 3 -s <subnet>/28 -j ACCEPT [root@XenServer-Host ~]# [root@XenServer-Host ~]# iptables -nvL --line-numbers Chain INPUT (policy ACCEPT 0 packets, 0 bytes) num pkts bytes target prot opt in out source destination 1 19091 2967K RH-Firewall-1-INPUT all -- * * 0.0.0.0/0 0.0.0.0/0 Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) num pkts bytes target prot opt in out source destination 1 11971 1722K RH-Firewall-1-INPUT all -- * * 0.0.0.0/0 0.0.0.0/0 Chain OUTPUT (policy ACCEPT 21439 packets, 3530K bytes) num pkts bytes target prot opt in out source destination Chain RH-Firewall-1-INPUT (2 references) num pkts bytes target prot opt in out source destination 1 2073 1065K ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0 2 163 10568 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 icmp type 255 3 6079 898K ACCEPT all -- * * <subnet>/28 0.0.0.0/0 4 0 0 ACCEPT esp -- * * 0.0.0.0/0 0.0.0.0/0 5 0 0 ACCEPT ah -- * * 0.0.0.0/0 0.0.0.0/0 6 0 0 ACCEPT udp -- * * 0.0.0.0/0 224.0.0.251 udp dpt:5353 7 0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:631 8 1 40 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:631 9 18922 2496K ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 10 0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW udp dpt:694 11 1090 54556 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:22 12 123 6188 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:80 13 57 2796 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:443 14 2554 156K REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited [root@XenServer-Host ~]#
Providing ISO image files for guest system installation
Finally, we want to install a guest system on our XenServer host. Because we can’t simply mount an installation image within XenCenter, we need to create a ISO repository first:
[root@XenServer-Host ~]# mkdir -p /var/opt/xen/iso_import [root@XenServer-Host ~]# xe sr-create name-label="MyISORepository" type=iso device-config:location=/var/opt/xen/iso_import/ device-config:legacy_mode=true content-type=is
And, for instance, provide a Debian netinstall image here:
[root@XenServer-Host ~]# cd /var/opt/xen/iso_import [root@XenServer-Host ~]# wget http://cdimage.debian.org/debian-cd/7.7.0/amd64/iso-cd/debian-7.7.0-amd64-netinst.iso
Keep in mind that your root file system only has 4 GB available, therefore try to provide only one installation image at a time.
That’s it!
You should now have a running, properly configured XenServer with the capability of installing and running multiple virtual machines in it. Even though this is how we got it up and running on Hetzner hardware, i don’t claim that this is the only (or best) way to do it. If you have any questions, concerns or better guides on this topic, feel free to share it in the comments!
Additional Links & Sources
http://support.citrix.com/article/CTX121169
http://techblog.conglomer.net/adding-different-subnet-ips-to-xenserver/
http://www.jansipke.nl/using-xenserver-in-a-routed-ip-network/
http://wiki.hetzner.de/index.php/LSI_RAID_Controller
http://wiki.hetzner.de/index.php/XenServer_5.6_Installation_per_LARA
cy
In addition to the links Mike posted for saving up on public IP addresses, theres another possibility to add connectivity to multiple VMs:
instead of NATting the VMs on the Host you could also install a router guest, then use an private IP on your XenServer Host and the public IP (with NAT for Host and other guests) on the router.
Could be any OS of choice that supports NAT, but you might also use a specific router/security OS like vyatta or pfsense.
i would prefer this setup to the ones in Mike's links, as it prevents the XenHost from being accessible globally on the Internet. To secure it even more you could run a vpn endpoint on your router guest to access the XenHost.
Citrix in general is actually really slow on some updates. both hearthbleed and shellshock took weeks to be "hot"fixed in 2014, so i would rather add that extra layer of security.
If your router VM is down you can still restart it from the Hetzner KVM-over-IP interface. (and if everything goes down you can add your public ip back to the xenserver.)
Mike Kesler
Hi JJ,
In this setup, you would need an additional subnet. If you only have one public IP address available, you would need to set up a local network, like in these examples:
http://stackoverflow.com/questions/21500233/how-to-use-the-same-ip-for-dom0-and-one-domu-in-xen
http://www.x83.net/setting-a-local-network-on-xenserver/
Hope that helps :)
JJ
Hello,
very nice tutorial.
I have only one doubt, I need obligatorily to order an additional subnet? or I can work using the only one single ip address and route traffic to guests os?
Thanks in advance.
Best regards