wip: Direct Routed (L3) guest networks: route public IPv4/IPv6 to VM from Host - #13758
Open
wido wants to merge 8 commits into
Open
wip: Direct Routed (L3) guest networks: route public IPv4/IPv6 to VM from Host#13758wido wants to merge 8 commits into
wido wants to merge 8 commits into
Conversation
boring-cyborg
Bot
added
component:api
component:integration-test
component:marvin
component:security-group
component:storage
component:UI
Python
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #13758 +/- ##
==========================================
Coverage 19.77% 19.78%
- Complexity 19989 20035 +46
==========================================
Files 6371 6373 +2
Lines 575899 576315 +416
Branches 70495 70598 +103
==========================================
+ Hits 113912 114051 +139
- Misses 449563 449814 +251
- Partials 12424 12450 +26
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces Direct Routed (L3) guest networks where KVM hosts perform L3 routing directly to instances (no Virtual Router, no NAT, no DHCP), using ConfigDrive-only addressing and a shared link-local gateway (169.254.0.1 / fe80::1) on per-network host bridges.
Changes:
- Adds
GuestType.L3with server-side offering/network validation and a newDirectRoutedNetworkGuruthat forces NICs into host-route form (/32, /128) with link-local gateways. - Updates ConfigDrive generation to always emit
network_data.jsonfor direct-routed NICs even when DHCP/DNS services are not present. - Extends the KVM agent to create/manage per-network bridges (
brdr-<networkId>), program host routes/neighbour entries, and adapt security-group rule plumbing for routed traffic (including secondary IP handling).
Reviewed changes
Copilot reviewed 31 out of 31 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| utils/src/main/java/com/cloud/utils/net/NetUtils.java | Centralizes link-local gateway/CIDR constants and adds IPv6 link-local gateway getter. |
| ui/src/views/offering/AddNetworkOffering.vue | Adds UI support for L3 offerings and enforces L3-specific service/provider composition. |
| ui/src/views/network/CreateNetwork.vue | Adds an L3 network creation tab. |
| ui/src/views/network/CreateL3NetworkForm.vue | New UI form for creating L3 (Direct Routed) networks. |
| ui/public/locales/en.json | Adds UI strings for L3 offering/network creation messaging. |
| tools/marvin/marvin/config/test_data.py | Adds Marvin test data for L3 offerings and L3 networks. |
| test/integration/smoke/test_l3_networks.py | Adds smoke/integration tests covering L3 network lifecycle and validations. |
| server/src/test/java/com/cloud/network/guru/DirectRoutedNetworkGuruTest.java | Unit tests for the new guru selection/design and NIC host-route forcing. |
| server/src/test/java/com/cloud/configuration/ConfigurationManagerImplTest.java | Unit tests for L3 offering validation rules. |
| server/src/main/resources/META-INF/cloudstack/server-network/spring-server-network-context.xml | Registers the new DirectRoutedNetworkGuru bean. |
| server/src/main/java/com/cloud/network/security/SecurityGroupManagerImpl.java | Ensures agent commands are sent for secondary IPs on L3 networks even when SG rules aren’t applied. |
| server/src/main/java/com/cloud/network/NetworkServiceImpl.java | Extends network creation/IPv6 checks/VLAN handling and secondary-IP orchestration for L3. |
| server/src/main/java/com/cloud/network/guru/DirectRoutedNetworkGuru.java | New guru for L3 direct routed guest networks; forces /32 + /128 form and link-local gateways. |
| server/src/main/java/com/cloud/configuration/ConfigurationManagerImpl.java | Adds L3 offering validation + zone-wide IPv4 overlap checks for L3 ranges. |
| scripts/vm/network/vnet/modifymacip.sh | Adds targeted delete support for specific -4/-6 addresses (secondary IP remove). |
| scripts/vm/network/vnet/modifybrdr.sh | New script to create/delete per-network brdr-* bridges with sysctls + gateway addresses. |
| scripts/vm/network/tests/test_security_group.py | New unit tests for classic-vs-L3 security_group.py rule streams and --directrouted plumbing. |
| scripts/vm/network/tests/golden_default_network_rules.txt | Golden output for classic rule generation (used to ensure no regression). |
| scripts/vm/network/tests/golden_add_fw_framework.txt | Golden output for classic FW framework creation (ensures byte-identical behavior). |
| scripts/vm/network/security_group.py | Adds routed-aware framework/hooks and rule generation, plus --directrouted CLI plumbing. |
| plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtSecurityGroupRulesCommandWrapper.java | Passes direct-routed flag when applying SG rules for the VM’s first NIC. |
| plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtNetworkRulesVmSecondaryIpCommandWrapper.java | Threads direct-routed + apply-SG booleans into secondary-IP handling. |
| plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java | Adds --directrouted plumbing and direct-routed secondary-IP route/neigh programming via modifymacip.sh. |
| plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/BridgeVifDriver.java | Creates per-network bridges for direct-routed NICs and ensures MAC/IP script runs for those NICs. |
| engine/storage/configdrive/src/test/java/org/apache/cloudstack/storage/configdrive/ConfigDriveBuilderTest.java | Adds tests for direct-routed NIC detection and network_data.json generation behavior. |
| engine/storage/configdrive/src/main/java/org/apache/cloudstack/storage/configdrive/ConfigDriveBuilder.java | Always generates network_data.json for direct-routed NICs; adds NIC signature detection. |
| docs/design/direct-routed-networks.md | Adds comprehensive design document for Direct Routed networks. |
| core/src/main/java/com/cloud/agent/api/NetworkRulesVmSecondaryIpCommand.java | Adds directRouted and applySecurityGroupRules flags for secondary-IP commands. |
| api/src/main/java/org/apache/cloudstack/api/command/user/vm/RemoveIpFromVmNicCmd.java | Sends agent update for secondary IP removal on L3 even when zone SG is disabled. |
| api/src/main/java/org/apache/cloudstack/api/command/user/network/CreateNetworkCmd.java | Allows specifying physical network ID for L3 networks. |
| api/src/main/java/com/cloud/network/Network.java | Adds GuestType.L3 and parsing support. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+318
to
+323
| // Host-wide property (EVPN use case) runs the MAC/IP script for every NIC; a Direct | ||
| // Routed NIC needs it regardless, since the host route and static neighbour entry are | ||
| // what deliver its traffic. | ||
| if (_macIpStaticEnabled || isDirectRoutedNic(nic)) { | ||
| executeMacIpScript(intf.getBrName(), nic.getMac(), nic.getIp(), nic.getIp6Address(), nic.getNicSecIps()); | ||
| } |
Comment on lines
+1454
to
+1461
| Network network = _networkModel.getNetwork(nic.getNetworkId()); | ||
|
|
||
| // On a Direct Routed network the host needs a route and a static neighbour entry for the | ||
| // secondary IP before it is reachable at all. That is independent of security groups, | ||
| // which are optional there and which the Instance may not be using, so the agent is told | ||
| // either way - otherwise the address would stay dark until the Instance was restarted. | ||
| boolean directRouted = Network.GuestType.L3.equals(network.getGuestType()); | ||
|
|
Member
|
@blueorangutan package |
|
@kiranchavala a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 18739 |
Contributor
|
@wido this is still marked as wip. what is the status? |
Contributor
Author
The status is that I have real hardware in a datacenter now where I’m going to deploy this code for further testing. Please keep it WIP for now as I want to validate it there. |
… Instances Adds a new guest network type in which the hypervisor performs L3 routing for the Instance: no Virtual Router, no NAT and no DHCP. Each Instance receives a public IPv4 address as a /32 and/or an IPv6 address as a /128, with a shared, host-independent gateway (169.254.0.1 and fe80::1) that every hypervisor carries on the network's bridge. All addressing reaches the Instance exclusively via ConfigDrive/cloud-init; a routing daemon on the host (FRR, BIRD, ...) advertises the addresses to the fabric and is deliberately out of scope for CloudStack. Management server: - GuestType.L3; the guest_type column is char(32), so no schema change. - Offering validation: UserData via ConfigDrive is mandatory, Dns optional but ConfigDrive-only, SecurityGroup permitted (now allowed for L3 alongside Shared), Dhcp rejected as not supported and not needed. Network mode, specifyVlan and VPC use are rejected. - DirectRoutedNetworkGuru subclasses DirectNetworkGuru, inheriting the Shared-network address lifecycle. canHandle() selects on the offering's guest type alone; design() produces a Native broadcast domain with no isolation id. After allocation the NicProfile is forced into host-route form, which is also the signature by which the agent and ConfigDrive recognise these NICs. - createNetwork treats L3 like Shared for the subnet: explicit IP range mandatory, vlan/IP-range row created at network creation, IPv6 accepted without the /64 restriction, aclType Account. - Zone-wide IPv4 overlap validation for L3 ranges: all L3 subnets share one host routing table and one fabric, so an overlap is an address conflict. The IPv6 vlan check was already zone-wide. ConfigDrive: - Network data is always generated for a direct routed NIC; the historical gate (Dhcp or Dns supported) held while ConfigDrive supplemented a VR but would leave these NICs with no addressing at all. Route generation itself is unchanged: cloud-init detects an IPv4 gateway inside 169.254.0.0/16 and sets on-link on the rendered route by itself. KVM agent: - One uplink-less bridge per network, brdr-<network id>, created and removed by the new modifybrdr.sh (flock'd, idempotent, refuses to remove a bridge still in use). The bridge carries the gateway addresses, forwarding and strict rp_filter; separate bridges make isolation between networks topological rather than a filtering concern. - BridgeVifDriver plugs direct routed NICs into their brdr bridge and runs the existing modifymacip.sh hook per NIC to install the static neighbour entry and host route, regardless of the host-wide EVPN property, whose meaning is unchanged. The design document, including the decision log and the verification notes behind each choice, is added under docs/design/.
…adcast domains
Revises the isolation model of Direct Routed (L3) networks. A network is
now direct routed when its offering's guest type is L3 AND it lives on a
dedicated physical network carrying the new isolation method ROUTED --
the network operator's explicit, zone-level opt-in. Every network gets a
broadcast domain of the new type routed://<id>, and that id names the
per-network bridge on the hypervisors: routed://5828 -> brdr-5828. The
id is a label, never an encapsulation; nothing appears on the wire.
The id lifecycle reuses the Shared network's VLAN mechanics verbatim:
with specifyVlan the operator passes the id at network creation through
the existing vlan parameter (bridge names become plannable, so host
routing policy per brdr-<id> can exist before the network does); without
it the id is allocated from the ROUTED physical network's vnet range at
creation and released on deletion. Zone-wide URI overlap checks apply.
Management server:
- BroadcastDomainType.Routed ("routed", Long).
- DirectRoutedNetworkGuru registers IsolationMethod("ROUTED") and follows
the standard guru contract (isMyIsolationMethod); design() stamps the
Routed broadcast domain and carries the routed:// URI.
- encodeVlanIdIntoBroadcastUri() maps a ROUTED physical network to
routed://<id>; the network's broadcast domain type is derived from the
URI scheme instead of hard-coded Vlan.
- The Shared-without-specifyVlan vnet allocation/release paths extend to
L3; L3 offerings may carry specifyVlan either way.
- An L3 IPv6 subnet longer than /64 is rejected at network creation:
addresses are computed with EUI-64 from subnet + MAC (the existing
setNicIp6Address()/updateNicIpv6() behaviour -- no address pool exists,
only the allocated result on the NIC), and EUI-64 needs 64 bits.
SystemVMs (CPVM/SSVM), dual-stack from the start:
- PublicNetworkGuru stamps NICs of a routed public range (created with
vlan=routed://<id>) in host-route form: /32 + 169.254.0.1, /128 +
fe80::1, Routed broadcast domain. IPv6 comes from the same EUI-64
computation as everywhere else.
- The CPVM and SSVM boot-arg builders emit eth<N>ip6, eth<N>ip6prelen
and ip6gateway, as the VR builder already did.
- common.sh marks the v4 default route onlink for link-local gateways,
and installs a static v6 default from the previously parsed-but-unused
IP6GW -- on a direct routed bridge no RA ever arrives.
KVM agent:
- BridgeVifDriver recognises direct routed NICs by broadcast type
instead of inferring from the address form, for guest and public
traffic alike, and derives the bridge from the NIC's broadcast URI.
modifybrdr.sh mechanics are unchanged; -n is now the routed id.
UI: ROUTED joins the isolation method choices (zone wizard, physical
network form). The design document records the revision and rationale;
no schema change and no data migration.
wido
force-pushed
the
direct-routed-network
branch
from
September 4, 2026 11:59
bf3d8a3 to
526934c
Compare
Reviewing every path the overhaul touches surfaced four fixes: - SystemVM public IPv6 is now computed directly in PublicNetworkGuru (EUI-64 from the range's ip6_cidr and the NIC MAC, then /128 + fe80::1). The previously planned ipv6Service.updateNicIpv6() path is unusable here: it is gated on the public network offering's internet protocol (never set on the system public offering) and reserves through one placeholder NIC per network -- on the shared Public network, every SystemVM would have received the same address. - Guest networks and public ranges share one routed-id space (a routed id names a bridge on every host), so collisions between them would merge L2 domains. Both directions are now rejected: a guest network cannot take an id a public range carries, and a public range cannot take an id a guest network holds. - Creating an L3 network on a physical network without the ROUTED isolation method now fails early with a clear message, instead of an opaque no-guru error deep in setupNetwork(). - A direct routed guest NIC's isolation URI now mirrors its routed:// broadcast URI instead of the misleading vlan://<tag> inherited from the Shared allocation path. GuestType.L3 itself was re-verified as still required: some thirty management-server branches (offering validation, subnet handling, zone-wide overlap checks, secondary-IP host routes, security groups) key on the guest type in places where no physical network -- and thus no isolation method -- is in scope. The guest type says what a network is; the isolation method says where it may live.
…D message CreateNetworkCmd.getPhysicalNetworkId() now accepts guest type L3 and its error message reads 'Shared, L2 or L3 only'; the test still asserted the old 'Shared or L2 only' text and failed.
…maining UI form Add a default network offering for Direct Routed (L3) guest networks, DefaultL3NetworkOffering: UserData and DNS via ConfigDrive, Security Groups enabled - the only service set validateL3NetworkOffering() permits. It is created in NetworkOrchestrator.configure() like the other default offerings, so it appears on fresh installs and on upgrades alike; until now the operator had to create it by hand. The routed id is allocated from the ROUTED physical network's range (no specifyVlan). UI: the ROUTED isolation method was selectable in the zone wizard and in the infrastructure listing's add-physical-network form, but missing from the form on a zone's Physical Networks tab - add it there. Give the VLAN, VXLAN and ROUTED options hover descriptions explaining what each isolation method does, so operators understand what they are opting into; ROUTED's spells out the Direct Routed model (no VLANs on the wire, no Virtual Router, no DHCP, addressing via ConfigDrive, L3 offerings only, KVM only).
Creating an L3 network died in NetworkOrchestrator.createGuestNetwork(): after encodeVlanIdIntoBroadcastUri() produced routed://<id>, the unconditional PVLAN overlap check handed that URI to listByPhysicalNetworkPvlan(), which rejects every URI scheme other than vlan:// and vxlan://: 'Requested URI routed://189 is not in the expected format. Expected URI Scheme as vlan://VID or vxlan://VID.' The check is meaningless for a routed id: no PVLAN network can exist on a ROUTED physical network (only L3 networks can), and the id's real overlap checks - zone-wide URI, public ranges, the vnet range - have already run earlier in the method. Skip it when the URI scheme is routed. Also reject the isolatedpvlan parameter for L3 networks explicitly, so the pvlan branch (which would build a pvlan:// URI around the routed id) is unreachable for this guest type.
CreateL3NetworkForm only supported offerings that allocate the routed id automatically; an operator-chosen id (the API's vlan parameter, like a Shared network's VLAN tag) could be given through the API alone. Add a 'Routed ID' field to the form, shown to root admins when the selected L3 offering carries specifyVlan - where the API requires the id - and hidden otherwise, since the API then rejects the parameter and allocates the id from the ROUTED physical network's range. The field's description explains what the id is: a label naming the per-network bridge (brdr-<id>) on every KVM host, unique per zone, nothing on the wire, and useful to choose in advance so host routing policy can exist before the network does.
…t range check Creating an L3 network whose offering does not carry specifyVlan failed in ConfigurationManagerImpl.createVlanAndPublicIpRange(): 'The VLAN tag 573 is already being used for dynamic vlan allocation for the guest network in zone AMS02' even though the id showed as Free. The check calls findVnet(), which matches any op_dc_vnet_alloc row with that number - taken or free - so it really asks 'does this tag lie inside a dynamic allocation range', not 'is it in use'. An auto-allocated routed id always comes from the ROUTED physical network's vnet range (commitNetwork() takes it via allocateVnet() moments earlier), so the check rejected every such network; only bypassvlanoverlapcheck=true got past it, and the rollback released the id again, which is why it kept showing as Free. Shared networks with specifyVlan=false have long been exempt from this check for exactly this reason (isSharedNetworkWithoutSpecifyVlan forces bypassVlanOverlapCheck). Give L3 networks the same exemption: expose the orchestrator's isL3NetworkWithoutSpecifyVlan() helper on NetworkOrchestrationService and OR it into the bypass condition. The operator-specified path is unchanged: with specifyVlan the given routed id still must not collide with a vnet range, as for Shared.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This Pull Request adds a new guest network type in which the hypervisor performs L3 routing for the Instance: no Virtual Router, no NAT and no DHCP. Each Instance receives a public IPv4 address as a /32 and/or an IPv6 address as a /128, with a shared, host-independent gateway (169.254.0.1 and fe80::1) that every hypervisor carries on the network's bridge. All addressing reaches the Instance exclusively via ConfigDrive/cloud-init; a routing daemon on the host (FRR, BIRD, ...) advertises the addresses to the fabric and is deliberately out of scope for CloudStack.
Management server
ConfigDrive
KVM agent
The design document, including the decision log and the verification notes behind each choice, is added under docs/design/.
This implements issue #12210
Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
Bug Severity
Screenshots (if appropriate):
How Has This Been Tested?
Testing is still ongoing on real hardware and this PR currently (July 2026) exists to gain initial feedback.