Data Center Interconnect: Stitching Sites Into One Fabric

Two data centers, one hard problem

The moment a business runs more than one data center — for capacity, for disaster recovery, or because a cloud region now counts as a site — it inherits a deceptively hard problem: making those locations behave like one. Data Center Interconnect is the set of technologies that carry traffic between sites, and the reason it is hard is that the requirements pull against each other. Applications want to move freely between sites as if on one LAN. Physics imposes latency proportional to distance. Failure isolation demands that a problem in one site not propagate to the other. And the transport in between is someone else's fiber, crossing public ground.

Get DCI wrong and you get the worst of both worlds: two data centers that share failures like one and share bandwidth like two. Get it right and you get a fabric that survives the loss of a whole site while keeping the applications that depend on each other close enough to perform. The decisions that determine which outcome you get are made early, and they are mostly about how much Layer 2 you allow to cross the link.

Layer 2 or Layer 3: the decision that shapes everything

The central DCI question is whether to stretch Layer 2 between sites — extending the same broadcast domain and subnet across the link so a server keeps its IP when it moves — or to keep each site a distinct Layer 3 domain and route between them.

• Layer 2 extension is what legacy applications and live workload mobility often demand: clustering that assumes a shared subnet, virtual-machine migration that preserves IP addressing, and appliances that expect their neighbors on the same segment. The cost is steep. A stretched broadcast domain also stretches its failure modes — a broadcast storm, a spanning-tree loop, or a flood of unknown-unicast traffic now takes down both sites at once. Layer 2 turns two failure domains into one.
• Layer 3 interconnect keeps each site independent and routes between subnets. It contains faults, scales cleanly, and is the healthier long-term architecture. The cost is that anything requiring a shared subnet across sites needs redesigning.

The modern answer is not to choose ideologically but to minimize the Layer 2 you extend and to use a control plane that contains its blast radius. Stretch a segment only where an application genuinely requires it, route everything else, and treat every stretched VLAN as a liability to be justified rather than a convenience to be granted.

Transport: fiber, waves, and the internet

Underneath the logical design sits physical transport, and the options trade cost against control:

• Dark fiber. You lease unlit fiber pairs and light them yourself. Maximum control, lowest latency, highest capacity — and the highest cost and operational burden. It suits campus and metro distances where the fiber is available.
• DWDM / optical transport. Dense Wavelength Division Multiplexing carries many independent channels (lambdas) over one fiber pair, each a full-rate circuit. It is how a single fiber run becomes dozens of 100G+ links, and it is the workhorse of metro and regional DCI.
• Carrier Ethernet / MPLS. A provider delivers point-to-point or any-to-any Ethernet services with a contracted rate and class of service. Less control than dark fiber, far less operational overhead, and predictable performance.
• IP / internet with overlays. Over ordinary internet transport you build the interconnect logically. This is where EVPN-VXLAN has become the default: VXLAN tunnels carry Layer 2 and Layer 3 over any IP underlay, and EVPN (BGP-based) provides a proper control plane that advertises MAC and IP reachability, suppresses ARP flooding, and supports multihoming. It gives you the workload mobility of a stretched fabric without the raw broadcast-domain behavior of old Layer 2 DCI — the control plane learns and distributes addresses instead of flooding to discover them.

For most enterprises the practical stack is optical or carrier transport between metro sites and an EVPN-VXLAN overlay providing the logical fabric on top, regardless of the underlay beneath.

Latency is the constraint you cannot buy your way out of

Bandwidth is purchasable; latency is bounded by the speed of light in glass, roughly 5 microseconds per kilometer one way, and no amount of money changes it. That single fact governs DCI design more than any other:

• Synchronous replication — where a write is not acknowledged until both sites confirm it — is exquisitely sensitive to distance. Every kilometer adds round-trip delay to every write, so synchronous storage replication is a metro-distance technique. Beyond that, you move to asynchronous replication and accept a non-zero recovery point.
• Chatty applications that assume LAN latency degrade sharply when a "single" system is split across sites, because every request now pays the inter-site round trip many times over. Keep tightly coupled tiers co-located.
• Time synchronization matters for distributed systems, financial and industrial workloads, and correlated logging. Where microsecond accuracy is required, PTP (IEEE 1588) distributes it far more precisely than NTP — but PTP itself assumes well-engineered, symmetric paths to hold that accuracy.

Design DCI by first classifying which workloads must be synchronous and near, which tolerate distance, and which should simply live in one site and fail over rather than stretch.

Secure the link between the buildings

Traffic between data centers crosses ground you do not control — leased fiber through manholes, carrier equipment, shared conduits. It must be treated as untrusted transport and encrypted end to end. Two mechanisms dominate:

• MACsec (IEEE 802.1AE) encrypts at Layer 2, hop by hop, and is the natural fit for point-to-point optical and Ethernet DCI links where you want line-rate encryption with minimal overhead.
• IPsec encrypts at Layer 3 and is the fit for interconnects that traverse routed or internet paths, including overlay tunnels.

Encryption is not optional at data-center scale; a tapped inter-site fiber exposes replication traffic, backups, and management planes in bulk. Building and operating DCI as part of coherent global networks — with diverse physical paths, a contained control plane, and encryption on every segment — is what separates a resilient multi-site fabric from two data centers tied together with a single fragile string.

Where to start

Start from the workloads, not the wires. Inventory what actually needs to cross between sites and sort it into three buckets: must share a subnet (justify each one), must replicate synchronously (bounds your distance), and can route and fail over (the healthy majority). That inventory drives every other decision.

1. Choose transport to fit distance and control needs — optical for metro, carrier or IP overlay beyond.
2. Minimize stretched Layer 2 and put an EVPN-VXLAN control plane around whatever remains.
3. Provision diverse physical paths so one fiber cut does not sever the fabric.
4. Encrypt every segment with MACsec or IPsec as the transport dictates.
5. Test a full-site failover, not just a link failover, before you depend on it.

Data Center Interconnect done well is invisible: sites fail over, workloads move, and users never learn which building served them. intSignal designs and operates multi-site fabrics that hold under real failure and real distance — talk to our team to interconnect your sites without inheriting each other's outages.