Network Virtualization: VXLAN, Overlays, and the Modern Data Center

The data center outgrew the VLAN

For thirty years the VLAN was how we segmented networks, and for thirty years it was enough. Then virtualization and multi-tenancy broke it in two specific ways. The first is a hard ceiling: the 802.1Q VLAN ID is a 12-bit field, capping a network at 4,094 usable segments — a number a single large cloud or multi-tenant data center exhausts easily. The second is subtler and worse: VLANs are tied to physical topology and spanning tree, so stretching a segment where workloads actually live means stretching a fragile Layer 2 domain, blocking redundant links, and accepting that a loop or broadcast storm ripples across everything.

Network virtualization solves both by borrowing the idea that made server virtualization work: decouple the logical from the physical. Build a stable, boring, routed physical network once, then run flexible virtual networks on top of it as software constructs that can be created, moved, and torn down without recabling anything. In the modern data center that overlay is almost always VXLAN, and understanding it is understanding how nearly every current data-center and cloud fabric is built.

Overlay and underlay: two networks, one job

The core concept is the split between two layers that most operators eventually learn to think about separately:

• The underlay is the physical network — the switches, the cabling, the IP routing between them. Its only job is to move packets reliably between physical devices. It is designed to be simple, fast, redundant, and rarely changed.
• The overlay is the set of virtual networks that ride on top, carried inside tunnels across the underlay. This is where tenants, segments, and application networks live. It changes constantly and is defined in software.

The payoff is independence. The underlay does not care how many virtual networks exist or how often they change; the overlay does not care about the physical topology, only that the underlay can carry a packet from one endpoint to another. Add a tenant, stretch a segment, or spin up a new application network entirely in the overlay, and the physical network never moves. This is the same abstraction that let virtual machines stop caring which physical host they ran on.

How VXLAN encapsulation works

VXLAN — Virtual Extensible LAN — is the encapsulation that makes the overlay real. Mechanically, it wraps an original Ethernet frame inside a new packet so it can be carried across a routed IP underlay:

• The original Layer 2 frame is placed inside a VXLAN header, then inside UDP (destination port 4789), then inside outer IP and Ethernet headers. This is MAC-in-UDP encapsulation — the tenant's frame becomes the payload of an ordinary IP packet the underlay already knows how to route.
• The VXLAN header carries a 24-bit segment identifier, the VNI (VXLAN Network Identifier), which raises the segment ceiling from ~4,000 to roughly 16 million. That single field change is what makes large multi-tenant fabrics possible.
• The devices that perform encapsulation and decapsulation are VTEPs (VXLAN Tunnel Endpoints), implemented in the switches (hardware VTEPs) or in hypervisor software. An endpoint's traffic enters a VTEP, gets wrapped with the right VNI, crosses the IP underlay as a normal routed packet, and is unwrapped by the VTEP at the far end and delivered as if the two endpoints shared a wire.

Because the transport is routed IP, the underlay can use equal-cost multipath and every link, giving you the multipath, loop-free scaling that spanning tree never allowed — while the overlay still presents a clean Layer 2 (or Layer 3) segment to the workloads.

EVPN: the control plane that makes it sane

Encapsulation alone is not a network. Early VXLAN relied on multicast flood-and-learn to discover where MAC addresses lived — every VTEP flooding unknown traffic to every other, learning by observation. It worked and it did not scale, and it dragged the old broadcast-heavy behavior into the new overlay. The modern deployment pairs VXLAN with a real control plane: EVPN, Ethernet VPN, carried in BGP (specifically MP-BGP).

EVPN's job is to distribute reachability so VTEPs know rather than flood:

• VTEPs advertise the MAC and IP addresses they host to each other through BGP, so forwarding is based on learned control-plane state instead of data-plane flooding.
• ARP/ND suppression lets a VTEP answer address-resolution requests locally from what EVPN already told it, cutting broadcast traffic sharply across the fabric.
• Multihoming lets a host or switch connect to two VTEPs actively for redundancy without the loops that Layer 2 redundancy used to require.
• Integrated routing and bridging means the same fabric can bridge within a segment and route between segments efficiently, with a distributed gateway on every leaf.

EVPN-VXLAN — EVPN control plane, VXLAN data plane — is the combination that defines the modern spine-and-leaf data center. When people say a fabric is "EVPN-VXLAN," this is the pairing they mean.

The underlay still has to be right

The seductive part of overlays is that they hide the physical network, and the dangerous part is the same. An overlay is only as healthy as the underlay beneath it, and overlay abstraction makes underlay faults harder to see, because a physical problem now surfaces as a confusing overlay symptom two layers up. A few disciplines keep it sane:

• Keep the underlay simple and stable. A routed spine-and-leaf (Clos) fabric with a single routing protocol and equal-cost multipath is the standard because it is predictable and non-blocking. Resist the urge to make the underlay clever.
• Mind the MTU. Encapsulation adds ~50 bytes of headers per packet. If the underlay MTU is not raised (jumbo frames) to absorb it, you get fragmentation or silent drops that manifest as baffling application-layer failures. This is the most common VXLAN deployment bug.
• Monitor both layers together. You must be able to correlate an overlay segment's behavior with the physical path it currently rides, which requires infrastructure monitoring that sees underlay and overlay as one system rather than two disconnected views.
• Manage it as software. The value of virtualization is lost if every overlay change is still a manual CLI ritual. A network portal that models segments, tenants, and policy as intent — and pushes it consistently — is what turns the overlay's flexibility into operational reality instead of a new source of drift.

Where to start

Network virtualization is not a rip-and-replace; it is an architecture you grow into. Begin by getting the underlay right, because every overlay decision inherits its quality: a clean routed spine-and-leaf design with correct MTU and multipath is the foundation. Then introduce EVPN-VXLAN for the segments that most need mobility and scale — new multi-tenant workloads, environments straining against the VLAN ceiling, or fabrics that must stretch cleanly between sites. Instrument both layers from day one, and manage the overlay as software rather than as a pile of CLI sessions. Done deliberately, the result is a data center where logical networks bend to the business while the physical network stays boringly stable underneath. intSignal designs and operates EVPN-VXLAN fabrics end to end, underlay through overlay — talk to our team to modernize your data center network without inheriting a decade of accidental complexity.