Virtual Machines (KubeVirt)

In addition to container pods, Berkelium supports running full hardware-virtualized Linux Virtual Machines (VMs) alongside your Kubernetes workloads using KubeVirt.

1-Click KubeVirt VMs via Coder

Want an interactive Linux VM with Docker Engine (docker + docker compose), persistent Ceph RBD root storage, and one-command coder ssh access without writing YAML manifests? Launch the KubeVirt Linux VM (kubevirt) template directly in the Coder Portal (coder.bk.lbl.gov)! See the Coder Workspaces Documentation for details.

When to Use a Virtual Machine vs. a Pod

Use Case	Recommended Workload	Why
PyTorch, Ray, MPI, Batch Jobs, Web Apps	Kubernetes Pod / Job	Lowest overhead, fastest startup, direct access to cluster GPUs and shared CephFS (ReadWriteMany).
Docker-in-Docker CI/CD Runners, Custom Systemd Daemons, Full OS Environments	KubeVirt VirtualMachine	Full guest Linux kernel isolation, dedicated systemd init, unprivileged nested container engines (dockerd), and persistent block root disks.

---

1. Prerequisites

Install virtctl

The virtctl CLI (or kubectl virt plugin) lets you start, stop, restart, open serial consoles, and forward SSH/TCP ports to your VMs:

# macOS (Homebrew)
brew install virtctl

# Or via Krew (macOS / Linux)
kubectl krew install virt

Upload Your SSH Public Key as a Secret

KubeVirt can automatically inject your SSH public key into the ubuntu user inside the guest OS using a Kubernetes Secret:

kubectl create secret generic my-ssh-keys \
  --from-file=key1=$HOME/.ssh/id_ed25519.pub \
  -n <your-namespace>

---

2. Launching a Generic Ubuntu 24.04 VM

Save the following manifest as my-vm.yaml. It provisions a persistent 20 GiB Ceph RBD root disk cloned automatically from the official Ubuntu 24.04 cloud image, an optional 100 GiB local NVMe ephemeral disk (/dev/vdb), and configures Berkelium’s native IPv6 + NAT64 networking:

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: ubuntu-vm
  namespace: <your-namespace>
spec:
  runStrategy: Always
  dataVolumeTemplates:
    - metadata:
        name: ubuntu-vm-root
      spec:
        pvc:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 20Gi
          storageClassName: ceph-rbd
        source:
          http:
            url: https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-amd64.img
  template:
    spec:
      architecture: amd64
      accessCredentials:
        - sshPublicKey:
            propagationMethod:
              noCloud: {}
            source:
              secret:
                secretName: my-ssh-keys
      networks:
        - name: default
          pod: {}
      domain:
        cpu:
          cores: 4
        resources:
          requests:
            cpu: "4"
            memory: 16Gi
          limits:
            cpu: "4"
            memory: 16Gi
        machine:
          type: q35
        devices:
          autoattachGraphicsDevice: true
          autoattachSerialConsole: true
          interfaces:
            - name: default
              masquerade: {}
          disks:
            - name: rootdisk
              bootOrder: 1
              disk:
                bus: virtio
            - name: scratchdisk
              disk:
                bus: virtio
            - name: cloudinit
              disk:
                bus: virtio
      volumes:
        - name: rootdisk
          dataVolume:
            name: ubuntu-vm-root
        - name: scratchdisk
          emptyDisk:
            capacity: 100Gi
        - name: cloudinit
          cloudInitNoCloud:
            networkData: |
              network:
                version: 2
                ethernets:
                  enp1s0:
                    dhcp4: true
                    addresses:
                      - fd10:0:2::2/120
                    routes:
                      - to: default
                        via: fd10:0:2::1
            userData: |
              #cloud-config
              package_update: true
              packages:
                - curl
                - git
                - htop

Apply the manifest and wait for the root disk import and VM boot to complete:

# Create the VM and its root DataVolume
kubectl apply -f my-vm.yaml

# Watch DataVolume import progress and VM status
kubectl get dv,vm,vmi -n <your-namespace>

# Wait until the VM reports Ready=True
kubectl wait --for=condition=Ready vm/ubuntu-vm -n <your-namespace> --timeout=180s

How KubeVirt Masquerade & NAT64 Work on Berkelium

Berkelium is an IPv6-native cluster with DNS64/NAT64 (2620:83:8000:64::/96) and RFC 4193 ULA Pod IPv6 addresses (fdf5:542c:f102:92cf:1000::/100). Configuring KubeVirt’s default masquerade CIDR (fd10:0:2::2/120 via fd10:0:2::1) in networkData with CoreDNS (172.24.0.10) ensures the virt-launcher masquerade router rewrites outbound packets to the pod’s IPv6 address and routes both IPv6 and IPv4 Internet destinations (including GitHub, Docker Hub, Ubuntu repositories, and PyPI) transparently through the cluster’s NAT64 gateway.

---

3. Connecting to Your VM

Direct SSH via virtctl ssh

You can SSH directly into your VM from your workstation without exposing a public IP address by running virtctl ssh:

virtctl ssh ubuntu@vm/ubuntu-vm -n <your-namespace>

If your SSH private key is not in the default ~/.ssh/id_rsa location or loaded into ssh-agent, pass -i:

virtctl ssh -i ~/.ssh/id_ed25519 ubuntu@vm/ubuntu-vm -n <your-namespace>

Optional: ~/.ssh/config for ssh, scp, rsync, and VS Code Remote-SSH

To use standard ssh ubuntu-vm, scp, rsync, or VS Code Remote-SSH directly, add an entry to your local ~/.ssh/config:

Host ubuntu-vm
  User ubuntu
  IdentityFile ~/.ssh/id_ed25519
  ProxyCommand virtctl port-forward -n <your-namespace> --stdio=true vmi/ubuntu-vm 22
  StrictHostKeyChecking no
  UserKnownHostsFile /dev/null

Then simply run:

ssh ubuntu-vm
scp dataset.tar.gz ubuntu-vm:/tmp/

Forwarding Local Ports (Web UIs & Services)

To forward a port (such as port 8080) from the VM to localhost:8080 on your workstation:

virtctl port-forward vmi/ubuntu-vm 8080:8080 -n <your-namespace>

Serial Console Access

If you need to debug boot messages or log in when SSH is unavailable, attach to the serial console:

virtctl console ubuntu-vm -n <your-namespace>

(Press Ctrl+] to detach from the serial console.)

---

4. Customizing Startup with Cloud-Init & Secrets

For larger provisioning scripts, store your #cloud-config in a separate file (e.g., cloud-config.yaml), upload it as a Kubernetes Secret with a userdata key, and reference it in cloudInitNoCloud.secretRef:

kubectl create secret generic ubuntu-vm-userdata \
  --from-file=userdata=cloud-config.yaml \
  -n <your-namespace>

Then reference secretRef in your VirtualMachine volume specification:

        - name: cloudinit
          cloudInitNoCloud:
            secretRef:
              name: ubuntu-vm-userdata
            networkData: |
              # ... (same networkData as above)

Formatting & Mounting the Ephemeral Disk (/dev/vdb) Across Reboots

When using an emptyDisk volume (scratchdisk mapped to /dev/vdb), the disk is preserved across warm guest reboots (sudo reboot) but starts unformatted whenever the VM pod is recreated (virtctl restart). Use an idempotent blkid check before formatting so both warm and cold restarts succeed cleanly:

#cloud-config
write_files:
  - path: /usr/local/sbin/mount-scratch.sh
    permissions: '0755'
    content: |
      #!/bin/bash
      set -euo pipefail
      if ! /usr/sbin/blkid /dev/vdb >/dev/null 2>&1; then
        /usr/sbin/mkfs.xfs -f /dev/vdb
      fi
      mkdir -p /scratch
      if ! mountpoint -q /scratch; then
        mount /dev/vdb /scratch
        chmod 1777 /scratch
      fi
runcmd:
  - /usr/local/sbin/mount-scratch.sh

---

5. Running Docker Inside a KubeVirt VM

If you install Docker inside your VM (for example, to build images or run CI/CD jobs), configure /etc/docker/daemon.json and an ip6tables SNAT rule so containers inside the VM inherit MTU 1430, query the cluster DNS64 resolver (172.24.0.10), and prefer NAT64 IPv6 routes automatically:

#cloud-config
write_files:
  - path: /etc/sysctl.d/99-docker-ipv6.conf
    content: |
      net.ipv6.conf.all.forwarding = 1
      net.ipv4.ip_forward = 1
  - path: /etc/docker/daemon.json
    content: |
      {
        "ipv6": true,
        "fixed-cidr-v6": "2001:db8:1::/80",
        "ip6tables": true,
        "mtu": 1430,
        "dns": ["172.24.0.10"]
      }
  - path: /etc/systemd/system/docker.service.d/override.conf
    content: |
      [Service]
      ExecStartPost=/bin/sh -c '/usr/sbin/ip6tables -t nat -D POSTROUTING -o enp1s0 -j SNAT --to-source fd10:0:2::2 2>/dev/null || true; /usr/sbin/ip6tables -t nat -I POSTROUTING 1 -o enp1s0 -j SNAT --to-source fd10:0:2::2'

---

6. VM Lifecycle & Operations

Start, Stop, and Restart

# Gracefully reboot the VM (recreates the virt-launcher pod while keeping persistent PVCs)
virtctl restart ubuntu-vm -n <your-namespace>

# Stop the VM (frees CPU and RAM quota while preserving the root PVC on Ceph RBD)
virtctl stop ubuntu-vm -n <your-namespace>

# Start a stopped VM
virtctl start ubuntu-vm -n <your-namespace>

# Delete the VM and its owned DataVolume / PVC
kubectl delete vm ubuntu-vm -n <your-namespace>

Troubleshooting Cloud-Init & Networking

Once connected via SSH, you can monitor or inspect the initial Cloud-Init provisioning run:

# Wait for cloud-init to finish and show status
sudo cloud-init status --wait

# View full provisioning output logs
sudo tail -n 100 /var/log/cloud-init-output.log

# Verify outbound NAT64 connectivity (to an IPv4-only service such as GitHub)
curl -I https://github.com