Nested virtualization is now available on non-metal instance types (patch_vmx workaround may be avoidable)

[felipedbene]

Summary

documentation/AWS.md states that "for full functionality on AWS, a 'metal' flavor is required as only metal flavors allow the use of nested virtualization", and scripts/00-patch_vmx.sh exists to bypass the VMX flag check so that some reference platforms run on non-metal flavors ("limits the list of nodes that actually work quite significantly and is not supported").

EC2 now offers real hardware nested virtualization on non-metal instance types, so on those flavors the VMX bypass should no longer be needed and full node support should be available without a metal instance.

Enabling it

The instance has to opt in explicitly via CpuOptions.NestedVirtualization. The Terraform AWS provider already supports this on both aws_instance and aws_launch_template (#46533):

resource "aws_instance" "cml_controller" {
  # ...
  cpu_options {
    nested_virtualization = "enabled"
  }
}

Sketch of the config surface, mirroring how other AWS options are handled today:

aws:
  flavor: m8i.xlarge          # non-metal
  nested_virtualization: true # -> cpu_options { nested_virtualization = "enabled" }

00-patch_vmx.sh (shipped in modules/deploy/data/, enabled through the app.customize list in config.yml) would then not be needed on such flavors.

Which instance types

Worth querying rather than hard-coding, because the published lists disagree. The Terraform provider documentation currently says "8th generation Intel-based instance types (C8i, M8i, R8i, and their flex variants) only", while the EC2 API in us-east-1 also reports 7th-generation types:

$ aws ec2 describe-instance-types --region us-east-1 \
    --filters Name=processor-info.supported-features,Values=nested-virtualization \
    --query 'InstanceTypes[].InstanceType' --output text | tr '\t' '\n' | \
    sed 's/\..*//' | sort -u | tr '\n' ' '
c7i c7i-flex c8i c8i-flex c8id i7i i7ie m7i m7i-flex m8i m8i-flex m8id r7i r7iz r8i r8i-flex r8id x8i

Availability also varies by region, so a preflight check in the module (or just a documented command) is probably better than a static list.

Two constraints I ran into that may be worth documenting alongside it:

• A recent AWS CLI is required for anything done outside Terraform: CLI 2.27 rejects --cpu-options NestedVirtualization=enabled as an unknown field; 2.36 accepts it.
• Minimum 4 vCPUs, since CML's setup aborts below that ("This system does not have the minimum number of CPUs required (4)"), which rules out the .large sizes.

What I actually verified, and what I did not

Verified: an m8i.xlarge launched with CpuOptions.NestedVirtualization=enabled runs a KVM-backed CML node with hardware acceleration and no VMX bypass involved. A node whose definition uses libvirt_domain_driver: kvm (alpine, CML 2.10.0) reaches BOOTED; the host has /dev/kvm with kvm_intel loaded and in use, and the node is a real accelerated QEMU domain:

$ sudo virsh list
 Id   Name                                   State
------------------------------------------------------
 1    08962779-70d6-4f9f-b0f0-0476cc6ce0fa   running
$ ps -eo args | grep qemu-system | grep -o -e '-accel kvm' -e '-cpu host,migratable=on'
-accel kvm
-cpu host,migratable=on

IOL/IOL-L2 nodes also work (a five-node MPLS L3VPN lab runs at ~1.25% host CPU and ~2 GB RAM), but IOL runs as a native process (libvirt_domain_driver: iol), so it does not exercise KVM and is not evidence of nested virtualization by itself.

Not verified: cloud-cml itself. I tested with CML-Free, which has no Smart Licensing token and ships as an ISO rather than a .pkg, so I could not exercise this repository's deployment path end to end. (The ISO/manual path is being documented separately in CiscoDevNet/cml-community#85; it is a different audience, not an alternative to this module.) I therefore have not confirmed that the .pkg install, the refplat copy, or licensing behave identically on a non-metal flavor — only that the underlying capability is now there.

Happy to submit a PR for the module change and the documentation wording if that is useful, but I did not want to send Terraform I cannot test against a licensed deployment without asking first.