If you’re looking for a Rust based Vmm alternative to Firecracker, I’d check out Cloud Hypervisor. Cloud Hypervisor ended up hitting that sweet spot for me in many places where Firecracker is chasing different outcomes.
Both projects share so many things to love:
- Entirely Rust based, sharing many quality crates from the rust-vmm project.
- Built on top of the Linux KVM hypervisor
- Good virtio device support (disks, network interfaces, etc.)
- Simple Vmm Http API
In contrast to Firecracker, Cloud Hypervisor directly targets long-lived, stateful VMs (but can also slim down to ‘microvm’ territory), and thus supports other features that Firecracker will most likely never add.
UEFI boot support
Unlike Firecracker, which only supports direct Linux boot, Cloud Hypervisor can boot a minimal UEFI firmware. This gives your guest VM direct control of the kernel upgrade lifecycle from within the guest. It also allows running OS types other than Linux. If you’re running long-lived stateful VMs that expect to manage their own kernel version upgrades over time, then this is what you want. I’m no fan of the complexities of UEFI, but the realities are that if you want wide-spread guest boot support (including Windows, etc.), having at least basic UEFI support is a good and useful thing.
Firecracker requires using Linux Direct Boot. This limits your guest VMs to Linux only, and requires that the Vmm itself manage the kernel upgrade lifecycle of the guest VMs. Firecracker is optimizing for something different than Cloud Hypervisor: Lightweight, short-lived VMs that are biased towards serverless function workloads (It was built for Amazon’s Lambda after all). Cloud Hypervisor also supports Linux direct boot in addition to UEFI support.
Qcow2 disk images
Firecracker only ships with Raw disk image support. The biggest drawback I hit with raw disk images is that it requires pre-allocating the entire guest disk, consuming valuable host disk capacity (or implementing other complex setups like device-mapper for overlay). These pre-allocations also lead to extra byte copying when shuffling block devices around a host cluster.
Instead, Cloud Hypervisor has direct Qcow2 support. This gives us baked in support for thin-allocated disk images (saving over-allocation), as well as backing-file support. You can have read-only overlay semantics for shared disk image lineage, snapshots, etc. No external device mapper required.
For generating base images, it’s much easier to generate Qcow2 images with base a base OS install, and then at VM provisioning time, issue a quick metadata operation to the Qcow2 file to resize the image for each provisioned guest.
Beyond Qcow2 disk images, I previously wrote an experimental implementation of network attached block devices for Firecracker. While this quick and dirty experiment worked, Firecracker’s focus on short-lived VMs for serverless functions made me doubt whether network-attached storage code patches would be welcome upstream. I’d love to push contributions back to Cloud Hypervisor that enable arbitrary block-device backend support.
Live migration
I haven’t gotten to plumb in live-migration yet, but this is something Firecracker is also unlikely to ever add, due to its focus on short-lived VMs. I’m looking forward to implementing this one on my cluster control plane!
Not Amazon
Fundamentally, I’m just not comfortable making deep investments in a project that’s governed by Amazon. Amazon’s values don’t align well with mine as of late, and I’d rather invest in tooling that doesn’t put more compute power in Jeff Bezos’ pocket.
To be sure: there are other hyperscalers involved in Cloud Hypervisor, but the diversity gives me more confidence. The last major release saw contributions from technologists at Microsoft, Google, Meta, IBM, Tencent, and several more.
I’m much choosier about licenses and governance of open source projects I contribute to these days. Too many rug-pulls for comfort.
Give it a try
I was able to add Cloud Hypervisor support to my internal cloud control plane in about a day or so. It shares many of the broad feature sets that Firecracker has, and then goes further.
If you’ve ever been intrigued by Firecracker, and are looking for an alternative whose mission isn’t so pigeon-holed into serverless, I’d give Cloud Hypervisor a look!