Running Proxmox VE 3.3 on a software RAID

During installation of Proxmox VE 3.3 (available here), you are only given the choice of what disk you want to use for the installation process, but not what disk layout you want to use. Now, there are several ways of installing Proxmox on a software RAID. One of them is to run through the standard installation process and add a software RAID afterwards, using mdadm.
This blog post is just to show how setting up a software RAID on a fresh Proxmox installation worked for me! Feel free to try the following steps on a clean installation – however, I recommend not to perform these actions on a live production environment! I do not take responsibility for any data loss that might occur! Also, keep in mind that running Proxmox VE on a software RAID is neither supported nor recommended by Proxmox!
Now, that we got the disclaimer out of the way, let’s get started:
Installing Proxmox
Follow through the installation, using the provided ISO installer. Keep track of what disk you use as the installation disk – in my case, this would be /dev/sda. Once you complete the installation, run the following steps to get an up-to-date Proxmox VE Version to work with.

root@[server]:/# apt-get update
root@[server]:/# apt-get dist-upgrade
root@[server]:/# apt-get upgrade

Next step, if you haven’t done that already, is to install mdadm

root@[server]:/# apt-get install mdadm

Setting up the software RAID mirror
After finishing the installation, this is the partitioning scheme I had to work with:

root@[server]:/# parted
GNU Parted 2.3
Using /dev/sda
Welcome to GNU Parted! Type ‘help’ to view a list of commands.
(parted) print
Model: ATA HGST HUS724020AL (scsi)
Disk /dev/sda: 2000GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Number Start End Size File system Name Flags
1 1049kB 2097kB 1049kB primary bios_grub
2 2097kB 537MB 535MB ext3 primary boot
3 537MB 2000GB 2000GB primary lvm

Partitions 2 and 3 are the ones that need to be mirrored. Let’s prepare the second disk by copying the partition table from sda to sdb and changing the partition types to RAID.

root@[server]:/# sgdisk -R=/dev/sdb /dev/sda
root@[server]:/# sgdisk -t 2:fd00 /dev/sdb
root@[server]:/# sgdisk -t 3:fd00 /dev/sdb

Checking the partition table on sdb, it should now look like this:

root@[server]:/# parted
GNU Parted 2.3
Using /dev/sda
Welcome to GNU Parted! Type ‘help’ to view a list of commands.
(parted) select /dev/sdb
Using /dev/sdb
(parted) print
Model: ATA HGST HUS724020AL (scsi)
Disk /dev/sdb: 2000GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Number Start End Size File system Name Flags
1 1049kB 2097kB 1049kB primary bios_grub
2 2097kB 537MB 535MB primary raid
3 537MB 2000GB 2000GB primary raid

We now need to create the needed RAID arrays. Because we booted from a non-RAID partition, we are going to add a missing disk into the RAID for now. This missing disk will later be replaced by sda.

root@[server]:/# mdadm –create /dev/md0 –level=1 –raid-disks=2 missing /dev/sdb2
root@[server]:/# mdadm –create /dev/md1 –level=1 –raid-disks=2 missing /dev/sdb3

You might run into an error, saying “mdadm: cannot open /dev/sdb2: Device or resource busy”. In this case, make sure you don’t have running RAID sets already, which could happen if your disks on this machine have been used in a RAID array before. If this is the case, you want to wipe out the first 512 bytes of your disk to clear any existing RAID metadata – this was a big headache for me, took me a while to find out what the problem was.

root@[server]:/# cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sdb2[1]
523968 blocks super 1.2 [2/1] [_U]
[…]
unused devices:
root@[server]:/# mdadm –stop /dev/md0
mdadm: stopped /dev/md0
root@[server]:/# mdadm –stop /dev/md1
mdadm: stopped /dev/md1
root@[server]:/# dd if=/dev/zero of=/dev/sda bs=512 count=1
root@[server]:/# dd if=/dev/zero of=/dev/sdb bs=512 count=1

We need to format md0, copy grub onto it and set it up as our /boot mountpoint.

root@[server]:/# mkfs.ext3 /dev/md0
root@[server]:/# mkdir /mnt/tmp
root@[server]:/# mount /dev/md0 /mnt/tmp
root@[server]:/# cp -ax /boot/* /mnt/tmp
root@[server]:/# umount /mnt/tmp
root@[server]:/# rmdir /mnt/tmp

Let’s not forget to edit the fstab, so it will mount /boot from md0 from now on. I commented out the previous /boot entry, just in case…

/dev/pve/root / ext3 errors=remount-ro 0 1
/dev/pve/data /var/lib/vz ext3 defaults 0 1
#UUID=1e2982a5-75b8-4e62-a5dc-7a49226d842f /boot ext3 defaults 0 1
/dev/md0 /boot ext3 defaults 0 1
/dev/pve/swap none swap sw 0 0
proc /proc proc defaults 0 0

Also, you need to edit /etc/default/mdadm

-AUTOSTART=false
+AUTOSTART=true
-INITRDSTART=’none’
+INITRDSTART=’all’

Now we need to change a few grub entries and finally reinstall grub.

root@[server]:/# echo ‘GRUB_DISABLE_LINUX_UUID=true’ >> /etc/default/grub
root@[server]:/# echo ‘GRUB_PRELOAD_MODULES=”raid dmraid”‘ >> /etc/default/grub
root@[server]:/# echo raid1 >> /etc/modules
root@[server]:/# echo raid1 >> /etc/initramfs-tools/modules
root@[server]:/# grub-install /dev/sda
root@[server]:/# grub-install /dev/sdb
root@[server]:/# update-grub
root@[server]:/# update-initramfs -u

Once you rebooted the server, you can check if everything went as expected

root@[server]:/# mount | grep boot
/dev/md0 on /boot type ext3 (rw,relatime,errors=continue,user_xattr,acl,barrier=0,data=ordered)

Next, let’s change /dev/sda2 to a RAID partition and add it into the array.

root@[server]:/# sgdisk -t 2:fd00 /dev/sda
root@[server]:/# mdadm –add /dev/md0 /dev/sda2

The root partition for Proxmox is an LVM partition, so we need to create a logical volume on /dev/md1 and move the LVM partition over.

root@[server]:/# pvcreate /dev/md1
root@[server]:/# vgextend pve /dev/md1
root@[server]:/# pvmove /dev/sda3 /dev/md1

Note that this will take a looooooooong time – it took several hours for me! After you sucessfully moved the partition, you can remove /dev/sda3 from the volume and add /dev/sda3 to the RAID array.

root@[server]:/# vgreduce pve /dev/sda3
root@[server]:/# pvremove /dev/sda3
root@[server]:/# sgdisk -t 3:fd00 /dev/sda
root@[server]:/# mdadm –add /dev/md1 /dev/sda3

Finally, you can check your RAID and should see the 2 partitions healing. This is it, you are now running your Proxmox VE on a software RAID!