My experience with XCP-ng

As a Software Engineer and homelabber, I always run VMs for testing and daily use. Choosing a suitable and rock-solid hypervisor is always a topic for me. I have tried Proxmox (Proxmox Virtual Environment), VMWare ESXi (with vCenter) and Hyper-V Server over the years, and I now use Proxmox most of the time. But This year, I came across XCP-ng, which I had never heard of. XCP-ng is a fork from XenServer, a community-driven hypervisor, a closer solution to the ESXi.

Impression of the XCP-ng stack

The XCP-ng hypervisor doesn’t come with a WebGUI for management, so I need to set up a centralized management platform called Xen Orchestra, which is developed by the same team as XCP-ng.

When it comes to Xen Orchestra, I really like its versatile backup options. My favourite one is Rolling Snapshots. With this feature, I can roll back VMs immediately if I accidentally do something wrong, without the need for creating snapshots manually (which I usually forget), or even create a new VM directly from a snapshot. It provides a much lower RTO and RPO than delta backup and full backup.

Also, unlike Proxmox, I can manage multiple XCP-ng servers without requiring a cluster. Currently, I run two Proxmox clusters independently. I have to manage them through two separate WebGUI, which is inconvenient and error-prone.

Choosing a hypervisor

As a personal user, the way of choosing a solution is different from businesses. Paying for commercial products and support is not a big deal for companies – they get what they pay for. As a free user, no matter which one I choose, I have to deal with most issues with my own knowledge and skills and rely heavily on the community.

Some highlights of Proxmox

Proxmox VE is kind of an all-in-one system solution. It comes with fully unlocked capabilities in terms of computing, storage, backup, and management center. I can do most of the configurations through its WebUI. Since it has a decentralized architectural design (or, say, a hyper-converged infrastructure), I don’t need to worry about a single point of failure.

Despite its benefit, however, there is often some hacking work through SSH to make it run smoothly. But that introduces another highlight. Since Proxmox is built on top of Debian, I can apply all my professional skills to deal with problems and even learn something during the process. It’s easier to find solutions from both Proxmox and Linux communities.

XCP-ng stack is great, but with some issues

When I try to use XCP-ng and Xen Orchestra, I have to put some effort to get it on and running.

For example, to start my very first VM – Xen Orchestra, I have to manually create and start a VM through CLI, compiling it from source code, only to unlock the paid-only backup feature. It becomes tricky when this management VM doesn’t start (it did happen several times). I have to fix it through CLI or with a little help from XCP-ng Center, a management client only on Windows. Control plane high availability is quite crucial to an enterprise environment. However, after some research, I find that Xen Orchestra doesn’t provide a HA solution like vCenter. Maybe I could set up HA on the VM level, but I haven’t tested it yet.

Also, the Advanced Live Telemetry feature is somewhat unhandy. XCP-ng provides a monitoring solution based on Netdata, but I can’t register the host to Netdata Cloud, probably the reason of its old version (v1.19.0, released on Nov 28, 2019). I can neither upgrade it, nor manually install the official version as it conflicts with the XCP-ng version.

I won’t complain too much, as they are all paid features. There are one-click solutions and full support available for paid users. However, both of the problems can be solved easily on Proxmox. Speaking of observability, I can utilize the official supported metric options (Graphite or InfluxDB), or install whatever I want. Prometheus node exporter and Netdata agent can run flawlessly on Proxmox.

Conclusion

I didn’t talk much about the capabilities of XCP-ng itself. Well, I did install some Ubuntu and Windows VMs on it, and I won’t doubt its reliability and stability. Since the Xen project has been adopted by many tech giants for long, such as AWS and Alibaba, it’s definitely production-ready and enterprise-ready.

However, as a personal user, as well as a Software Engineer, I have to admit that the XCP-ng stack cannot provide the turnkey solution I need. So I decided to stick to Proxmox as my daily driver. I won’t say Proxmox is perfect, and neither is XCP-ng. They both have their problems. I believe most of my problems can be solved with effort, but I don’t want to spend too much time on it.

Even so, the future of XCP-ng is still promising. It’s unfair to compare both projects without considering how long they have been developed (XCP-ng started in 2018, while Proxmox started in 2008). The XCP-ng team is still actively developing new features. There will be a new management platform called XO-Lite with a redesigned UI/UX, which runs directly on XCP-ng for easier management. And I’m really impressed by the CEO of Vates – Olivier Lambert, the creator of XCP-ng and Xen Orchestra. His name always pops up while I’m searching in forums. He is basically an active member of every XCP-ng forum and every thread about XCP-ng. I can feel his passion for his projects. The tech world is changing fast, I’m sure things will be different next time I look back, and I will definitely give it another shot once it fits my need.