Hi, I'm Giacomo Arru and I'm the System Administrator at BETA Technologies. Managing datacenters requires constant studying and every experience is important and precious. Today I'll share some useful tips and considerations about the Virtualization Storage under Citrix XenServer and FreeNAS.
Here are 5 quick tips you should keep in mind while designing ZFS Storage for Virtualization use with Citrix XenServer (but it should be fine also with VmWare).
1 - The RAID geometry
The RAID geometry is the most performance factor you should always be designing correctly or you'll afterwards regret.
You must configure the ZFS pools with 2 disks at once with a RAID 1 geometry. Why? You could think you have the RLOG and the L2ARC for performance and disks are so expensive... but in the future you'll suffer a lot if you choose to deploy virtual machines under RAIDZ1/RAIDZ2. That's because of the ZFS nature: the fragmentation keeps growing because of the copy-on-write architecture and the performance has a drastic falldown when your zpool reaches 50% of free space.
So, always remember: use RAID1 for virtualization.
What if I already have a RAIDZ2 virtual machine pool? Export the data, destroy the ZPOOL and configure it correctly again as RAID1.
XenServer should allow you to perform a live migration of the virtual machines to another storage pool.
2 - The ISCSI zvol size
If your ZPOOL is full-flash and you have tons of RAM, you can safely use ZFS deduplication feature. Remember you'll benefit from this feature if you have lots of similar virtual machines. For instance a lot of Windows Servers for remote desktop small deployments. But what if you need to restore an ISCSI volume snapshot and plug it to the XenServer pool?
How many virtual machine disks do you keep in one single storage repository?
XenServer won't let you plug the storage repository because it has the same serial UUID.
One trick is to keep a spare server to plug the snapshot storage pools and power on the damaged virtual machines.
So, a good practice is to configure one iSCSI LUN for every virtual machine. This will make you faster in recovering snapshots because you just have to unplug the SR in XenServer and restore the snapshot in freenas. If you keep more than one virtual machine for SR, you can't do this but you have to mount the SR in another XenServer pool, export the disk, then re-import the disk in the correct XenServer SR. So, it could be take a long time and a lot of work to recover from a data corruption. If you deploy Microsoft SQL Server, you know what I'm talking about.
For this reason you should create small iSCSI targets. If you need 100G, create a 500G sparse, compressed, deduplicated ISCSI-disk and you'll have no problems recovering snapshots.
UPDATE: new XenServer 7.2 lets you create scheduled snapshots, so you could keep one or two snapshots to rollback in case of need.
3 - The ZFS snapshots and the ZFS replication
You should always, if you value your data, configure a correct snapshot planning along with replication to a second FreeNAS Server. Some zvol could require 1 hour snapshots, others 5 minutes. It depends on the importance level your servers have.
Always use a dedicated VLAN with dedicated NIC. Set a transfer limit or your FreeNAS will suffer from the big transfer data rate. In this case, you could set fast encryption algorithm to save CPU clocks. Benchmark the replication so you're sure it'll work correctly. Please let the first synchronization to finish to see if it keeps up to date the data on the second FreeNAS Server.
In case of a disaster with the first FreeNAS Server, you could mount the snapshots and export them with iSCSI from the second FreeNAS Server and keep your servers working in minutes.
Please, ensure you have the right hardware on both servers.
Also consider FreeNAS HA if you have the budget. Same thing for network switches and routers.
What would happen if the event of a FreeNAS main server crash? Could you power-up the virtualm machines from the backup snapshots? Think about it.
4 - The network connections between XenServer and FreeNAS
If your servers aren't equipped with 10Gbe NICs, you can use multiple network interfaces with multipathing.
Please be careful if you're planning to use NIC bonding.
It is very problematic and FreeNAS not always works very well with laggs.
Multipathing on multiple VLANS can give you the performance you're looking for, if tuned correctly.
Also, always use high-end network interfaces from Intel if the budgets lets you. A nice 10G fiber is a good starting point.
Don't use jumbo frames because iSCSI works very ugly with this configuration. While XenServer has no problems with jumbo frames, FreeNAS shows a very weird behavior so you're warned. The reason because jumbo frames are not enabled by default in the switches is that not every application works good with it on.
Monitor the switchs because a faulty switch could hurt your performace and your data!
5 - Scrubs and SMART tests.
If your storage is all-flash, disable SMART tests because they are meaningless. You can rely on scrubs for the data integrity. Also, remember you already configured replication so if a disk fails, you can replace it. SSDs are very quick with resilvering so you should be safe.
If you're deploying hard drives, you should carefully plan SMART short and long tests. Please be advised that SMART long tests are awfully blocking and you'll never, never let the FreeNAS Server to run them along a scrub. It will be a very bad scenario.
UPDATE: configure boot volume scrubbing every 7 days (if USB devices).
Last but not less important, you should expertise by studying everyday and determining ever-well configuration paramenters and tuning. You experience is priceless.
Always be practicing and studying and you'll be happy with your datacenter.