Virtualized Linux on macOS Internals
So you want to run a Linux virtual machine on macOS. This is something I worked on for a while at Google — the more general problem of running Kubernetes clusters for development (i.e., minikube) on systems that didn’t natively support containers (i.e., macOS and Windows). To do so, you need to become somewhat of an expert in booting, provisioning, and managing ephemeral virtual machines (quickly).
Native Apple Hypervisors. Originally, there was the Hypervisor.framework APIs included in OS X. It was extremely low-level, poorly documented, and overall pretty difficult to work with. There was a port of the bhyve (BSD Hypervisor) virtual machine manager created to use the Apple-native version called xhyve. This is what we originally used to run virtual machines on macOS in minikube. Docker forked this into Hyperkit, which powered Docker for Mac. QEMU eventually supported the Hypervisor.framework as an accelerator. Later, in macOS Big Sur, a higher-level API called the Virtualization.framework was released (which simply builds on the Hypervisor.framework).
File sharing. Sharing files with the virtual machine is part of most workflows. Ideally, you want folders and files mounted inside the VM (so that changes are reflected back on the host). There’s really no amazing solution, but I’ll cover all of the ones I know. FUSE (Filesystem in Userspace) is one of the go-to solutions for remote filesystems, and most of them use some implementation. FUSE is great because it (1) runs completely in userspace, (2) doesn’t touch the networking stack.
Virtiofs. Not a network file system but a layer that takes advantage of the locality between the host and a virtual machine and exploits Direct Access (DAX). It uses FUSE as a protocol but goes far beyond it.
Plan 9 (9p) Filesystem. A simple filesystem API that survived from the Plan 9 distributed operating system (designed at Bell Labs in the 1980s). We used an implementation of this early in minikube.
gRPC Fuse, sshfs, osxfs, and other FUSE implementations. There are plenty of other ways to pass messages via FUSE over the network. Many of these don’t take full advantage of the fact that the hypervisor and host are sharing file access on the same machine but work generally in many scenarios.
Operating System. I covered some of the ways that you could build embeddable Linux distributions (e.g., Buildroot, Yocto, and more) here. In minikube, we built our own using Buildroot. Depending on the use case, you can either choose to use a minimal embeddable one or a longer-lived general purpose one like Ubuntu.