OpenShift Virtualization lets you run VMs as first-class citizens in OpenShift. By default, VMs communicate over the cluster overlay (the OVN-Kubernetes pod network). That’s fine for east–west traffic inside the cluster, but often you want VMs to join an existing VLAN, talk to physical appliances, or get directly reachable IP addresses. Enter the ClusterUserDefinedNetwork (CUDN).
Why CUDN?
The ClusterUserDefinedNetwork (CUDN) is just an abstraction of the NetworkAttachmentDefinition (NAD), but has a features that differentiate it from the UserDefinedNetwork (UDN). The UDN is used for namespace-scopes network overlays, providing users the ability to create and manage additional L2 and L3 networks as software defined overlays on top of the OVN-K network. The UDN abstraction doesn’t provide for connecting to existing underlay networks, such as an existing VLAN connected into the cluster using a NodeNetworkConfigurationPolicy (NNCP).
The CUDN is cluster-scoped (not per-namespace) because the underlying bridge mappings and physical NIC plumbing have to be consistent across all nodes. CUDN lets you define a “localnet” topology that references physical bridges (via NMState + OVN bridge-mappings). You can set VLAN mode (access or trunk) and any namespace you select in the CUDN can then create workloads that join this underlay.
Starting from the Bottom
Let’s start from the bottom of the stack and provide some examples in terms of managing the physical connections at the node level. Below is an example of a NodeNetworkConfigurationPolicy that joins two ethernet connections together in an LACP bond.
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
name: ocp2-eno1-eno2-bond1
spec:
nodeSelector:
kubernetes.io/hostname: ocp2
desiredState:
interfaces:
- name: eno1
type: ethernet
state: up
mac-address: 02:00:00:00:00:01
- name: eno2
type: ethernet
state: up
mac-address: 02:00:00:00:00:02
- name: bond1
type: bond
state: up
link-aggregation:
mode: 802.3ad
port:
- eno1
- eno2
options:
miimon: '100'
lacp_rate: fast
xmit_hash_policy: layer3+4This type of NNCP is for post-install configuration on OCP worker nodes. Notice the nodeSelector being for a single machine so you would need to duplicate this configuration for every machine and replace the mac-address for the ethernet cards.
You also will notice no references to anything regarding VLAN. With this type of setup, we are going to assume that eno1 and eno2 are both connected to trunk ports, thus allowing all tagged traffic to come in. We will deal with the tags later.
Building Bridges
The next step is to build an OVS bridge. An OVS bridge is a virtual switch that lives inside your Linux host, powered by Open vSwitch (OVS). Think of it like a software version of the top-of-rack switch in your data center—but it’s running in the kernel, fast, programmable, and tightly integrated with SDN systems like OVN-Kubernetes. It was originally built to bring “real switch” features into virtualization platforms. Unlike the old linux-bridge, OVS understands VLAN tags, tunnels (VXLAN, Geneve, GRE), flow tables, and has a rich control API.
In OVN topology, localnet is a special logical port type that says:
“This network is not an OVN overlay. Instead, it connects directly to a local bridge on each node.”
When you declare a ClusterUserDefinedNetwork with topology: Localnet, OVN wires that logical network to a real OVS bridge on the host. Packets from pods/VMs don’t get encapsulated—they’re dumped raw onto that bridge. If you tag it with a VLAN ID, OVN will insert/remove the VLAN tags as it goes through.
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
name: ovs-bridge-trunk
spec:
nodeSelector:
node-role.kubernetes.io/worker: ""
desiredState:
interfaces:
- name: ovs-bridge-trunk
type: ovs-bridge
state: up
bridge:
allow-extra-patch-ports: true
options:
stp: false
port:
- name: bond1
ovn:
bridge-mappings:
- localnet: localnet-bridge-trunk # This is referenced below
bridge: ovs-bridge-trunk
state: presentNotes: The allow-extra-patch-ports lets OVN add its patch ports without fighting OVS and the stp: false avoids unexpected blocking for OVS when the upstream switch handles loops
Now, OVN doesn’t magically know which OVS bridge corresponds to which underlay. You have to tell it. That’s the bridge-mapping. The ovn bridge mapping above says:
- There is a physical network called localnet-bridge-trunk.
- On this node, it is backed by the OVS bridge named ovs-bridge-trunk.
- OVN should wire all logical localnet-bridge-trunk ports into that OVS bridge.
So when a VM attaches to a CUDN that references localnet-bridge-trunk, OVN will patch its vNIC straight into that OVS bridge, which in turn is bound to your NIC/bond/trunk. From there, packets are just Ethernet frames on VLAN 4, VLAN 31, etc.
Defining the ClusterUserDefinedNetwork
Now that the plumbing is in place, we can define a CUDN and connect it back into the bridge. Our first example is for connecting to a specific underlay VLAN and using the built-in IPAM for managing IP addresses for VMs connected to the network.
apiVersion: k8s.ovn.org/v1
kind: ClusterUserDefinedNetwork
metadata:
name: vlan4
spec:
namespaceSelector:
matchExpressions:
- key: kubernetes.io/metadata.name
operator: In
values: ["project1", "project2"]
network:
topology: Localnet
localnet:
role: Secondary
physicalNetworkName: localnet-bridge-trunk # this is referenced above
vlan:
mode: Access
access:
id: 4
subnets:
- "10.4.0.0/24"
excludeSubnets:
- "10.4.0.0/31" # excludes 10.4.0.0 – 10.4.0.1
- "10.4.0.255/32" # excludes 10.4.0.255
ipam:
mode: Enabled # DEFAULT!
lifecycle: PersistentFor this VLAN, I have it configured in my router for the entire /24 subnet. But the IPAM is dumb and unless you tell it to ignore the network, gateway and broadcast addresses, it’ll try to assign those. The IPAM is also all or nothing. If you have a VM connecting to a CUDN where the ipam.mode is enabled, then those vNICs are going to get assigned an IP from the network, regardless of any local configuration attempts.
If you want to use DHCP or something like cloud-init to setup the IPs for the vNIC, then you need to disable IPAM completely.
apiVersion: k8s.ovn.org/v1
kind: ClusterUserDefinedNetwork
metadata:
name: vlan37
spec:
namespaceSelector:
matchExpressions:
- key: kubernetes.io/metadata.name
operator: In
values: ["project1", "project2"]
network:
topology: Localnet
localnet:
role: Secondary
physicalNetworkName: localnet-bridge-trunk
vlan:
mode: Access
access:
id: 37
ipam:
mode: DisabledConfiguring a Virtual Machine
Once the CUDN is in place, the network will be able to be utilized by the VMs. Here’s an example of a VM using static IP configurations rather than allowing the CUDN ipam.
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
annotations:
vm.kubevirt.io/validations: |
[
{
"name": "minimal-required-memory",
"path": "jsonpath::.spec.domain.memory.guest",
"rule": "integer",
"message": "This VM requires more memory.",
"min": 1610612736
}
]
labels:
app: rhel9-cudn-example
kubevirt.io/dynamic-credentials-support: 'true'
vm.kubevirt.io/template: rhel9-server-small
vm.kubevirt.io/template.namespace: openshift
vm.kubevirt.io/template.revision: '1'
vm.kubevirt.io/template.version: v0.34.0
name: rhel9-cudn-example
namespace: project1
spec:
dataVolumeTemplates:
- apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
creationTimestamp: null
name: rhel9-cudn-example
spec:
sourceRef:
kind: DataSource
name: rhel9
namespace: openshift-virtualization-os-images
storage:
resources:
requests:
storage: 30Gi
runStrategy: Halted
template:
metadata:
annotations:
vm.kubevirt.io/flavor: small
vm.kubevirt.io/os: rhel9
vm.kubevirt.io/workload: server
creationTimestamp: null
labels:
kubevirt.io/domain: rhel9-cudn-example
kubevirt.io/size: small
spec:
architecture: amd64
domain:
cpu:
cores: 1
sockets: 1
threads: 1
devices:
disks:
- disk:
bus: virtio
name: rootdisk
- disk:
bus: virtio
name: cloudinitdisk
interfaces:
- macAddress: '02:85:38:00:00:0a'
masquerade: {}
model: virtio
name: default
- bridge: {}
macAddress: '02:85:38:00:00:0b'
model: virtio
name: nic-vlan37
state: up
rng: {}
features:
acpi: {}
smm:
enabled: true
firmware:
bootloader:
efi: {}
machine:
type: pc-q35-rhel9.6.0
memory:
guest: 2Gi
resources: {}
networks:
- name: default
pod: {}
- multus:
networkName: vlan37
name: nic-vlan37
terminationGracePeriodSeconds: 180
volumes:
- dataVolume:
name: rhel9-cudn-example
name: rootdisk
- cloudInitNoCloud:
networkData: |
version: 2
ethernets:
eth1:
dhcp4: no
addresses:
- 10.37.0.50/24
gateway4: 10.37.0.1
nameservers:
addresses:
- 10.3.0.3
- 9.9.9.9
dhcp6: no
accept-ra: false
userData: |
#cloud-config
user: cloud-user
password: Pass123!
chpasswd:
expire: false
name: cloudinitdisk
References
https://ovn-kubernetes.io/okeps/okep-5193-user-defined-networks
https://ovn-kubernetes.io/api-reference/userdefinednetwork-api-spec
https://guifreelife.com/blog/2025/01/02/OpenShift-Virtualization-VLAN-Guest-Tagging