VXLAN EVPN Multi-Site – Part 1 – Underlay Setup

Published by Nick Carlton on

In this series of guides, we will be expanding on the previous guides by making a multi-site VXLAN EVPN topology. In this part, we will look at the topology and setup the underlay network using OSPF.

Here is the topology we will be working with:

Image

The switches are all Nexus C9500v running NX-OS version 10.3(3).

Firstly we should enable all the features we need for the configuration on all switches:

feature ospf
feature bgp
feature pim
feature fabric forwarding
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay

Each of the switches have a lookback0 interface configured with IP addresses:

DC1-CORE1: 10.0.0.1/32
DC1-CORE2: 10.0.0.2/32
DC2-CORE1: 10.0.0.3/32
DC2-CORE2: 10.0.0.4/32

DC1-SPINE1: 10.0.0.5/32
DC1-SPINE2: 10.0.0.6/32
DC2-SPINE1: 10.0.0.7/32
DC2-SPINE2: 10.0.0.8/32

DC1-LEAF1: 10.0.0.9/32
DC2-LEAF1: 10.0.0.10/32
DC1-LEAF2: 10.0.0.11/32
DC2-LEAF2: 10.0.0.12/32

Routing Configuration

All switches have a basic OSPF configuration:

router ospf UNDERLAY
  log-adjacency-changes

Then each of the loopback0 interfaces are configured for OSPF in area 0.0.0.0:

interface Loopback0
  ip router ospf UNDERLAY area 0.0.0.0

Interface Configuration

The Spine interfaces towards the the leaves are configured as layer 3 ports using the loopback interface:

interface Ethernet1/3-4
  no switchport
  mtu 9216
  medium p2p
  ip unnumbered loopback0
  ip router ospf UNDERLAY area 0.0.0.0
  ip ospf network point-to-point
  no shutdown

The Leaf interfaces towards the the spines are also configured as layer 3 ports using the loopback interface:

interface Ethernet1/1-2
  no switchport
  mtu 9216
  medium p2p
  ip unnumbered loopback0
  ip router ospf UNDERLAY area 0.0.0.0
  ip ospf network point-to-point
  no shutdown

The interfaces between the Core switches and Spine switches also need configuring as Layer3 interfaces with the IP addressing on the diagram:

interface Ethernet1/X
  no switchport
  mtu 9216
  ip address ............
  ip ospf network point-to-point
  medium p2p
  ip router ospf UNDERLAY area 0.0.0.0
  no shutdown

These interfaces also need to be configured in the OSPF network. The above is a general template with the IP addressing required.

There are also interfaces between the Core switches within each DC, these should be configured as per the below:

interface Ethernet1/4
  no switchport
  mtu 9216
  ip address ............
  ip ospf network point-to-point
  ip router ospf UNDERLAY area 0.0.0.0
  no shutdown

Finally, the DCI links between the two DCs need configuring, these are not in the OSPF domain and are just normal Layer3 interfaces:

interface Ethernet1/1
  no switchport
  mtu 9216
  ip address ..............
  medium p2p
  no shutdown

We should now have basic connectivity within the topology, lets check the routing table on DC1-SPINE1 for the OSPF routes:

DC1-SPINE1# show ip route ospf-UNDERLAY 

10.0.0.1/32, ubest/mbest: 1/0
    *via 10.65.0.1, Eth1/1, [110/41], 00:09:46, ospf-UNDERLAY, intra
10.0.0.2/32, ubest/mbest: 1/0
    *via 10.65.1.1, Eth1/2, [110/41], 00:09:42, ospf-UNDERLAY, intra
10.0.0.6/32, ubest/mbest: 4/0
    *via 10.0.0.9, Eth1/3, [110/81], 00:20:59, ospf-UNDERLAY, intra
    *via 10.0.0.11, Eth1/4, [110/81], 00:20:54, ospf-UNDERLAY, intra
    *via 10.65.0.1, Eth1/1, [110/81], 00:09:45, ospf-UNDERLAY, intra
    *via 10.65.1.1, Eth1/2, [110/81], 00:09:42, ospf-UNDERLAY, intra
10.0.0.9/32, ubest/mbest: 1/0
    *via 10.0.0.9, Eth1/3, [110/41], 00:21:00, ospf-UNDERLAY, intra
10.0.0.11/32, ubest/mbest: 1/0
    *via 10.0.0.11, Eth1/4, [110/41], 00:20:54, ospf-UNDERLAY, intra
10.65.2.0/24, ubest/mbest: 1/0
    *via 10.65.0.1, Eth1/1, [110/80], 00:09:46, ospf-UNDERLAY, intra
10.65.3.0/24, ubest/mbest: 1/0
    *via 10.65.1.1, Eth1/2, [110/80], 00:09:42, ospf-UNDERLAY, intra
10.65.10.0/24, ubest/mbest: 2/0
    *via 10.65.0.1, Eth1/1, [110/80], 00:01:47, ospf-UNDERLAY, intra
    *via 10.65.1.1, Eth1/2, [110/80], 00:01:36, ospf-UNDERLAY, intra

Lets also look in the other DC at DC2-SPINE1:

DC2-SPINE1# show ip route ospf-UNDERLAY 

10.0.0.3/32, ubest/mbest: 1/0
    *via 10.66.0.1, Eth1/1, [110/41], 00:10:29, ospf-UNDERLAY, intra
10.0.0.4/32, ubest/mbest: 1/0
    *via 10.66.1.1, Eth1/2, [110/41], 00:07:00, ospf-UNDERLAY, intra
10.0.0.8/32, ubest/mbest: 4/0
    *via 10.0.0.10, Eth1/3, [110/81], 00:21:43, ospf-UNDERLAY, intra
    *via 10.0.0.12, Eth1/4, [110/81], 00:21:35, ospf-UNDERLAY, intra
    *via 10.66.0.1, Eth1/1, [110/81], 00:10:29, ospf-UNDERLAY, intra
    *via 10.66.1.1, Eth1/2, [110/81], 00:06:55, ospf-UNDERLAY, intra
10.0.0.10/32, ubest/mbest: 1/0
    *via 10.0.0.10, Eth1/3, [110/41], 00:21:43, ospf-UNDERLAY, intra
10.0.0.12/32, ubest/mbest: 1/0
    *via 10.0.0.12, Eth1/4, [110/41], 00:21:35, ospf-UNDERLAY, intra
10.66.2.0/24, ubest/mbest: 1/0
    *via 10.66.0.1, Eth1/1, [110/80], 00:10:29, ospf-UNDERLAY, intra
10.66.3.0/24, ubest/mbest: 1/0
    *via 10.66.1.1, Eth1/2, [110/80], 00:07:00, ospf-UNDERLAY, intra
10.66.10.0/24, ubest/mbest: 2/0
    *via 10.66.0.1, Eth1/1, [110/80], 00:02:14, ospf-UNDERLAY, intra
    *via 10.66.1.1, Eth1/2, [110/80], 00:01:08, ospf-UNDERLAY, intra

The above shows that we are seeing all the correct routes in the underlay. Note that we do not see inter DC routes because the DCI links cut the underlay routing protocol in half because they don't participate. Now we can move onto the Multicast setup.

Multicast Configuration

PIM is configured for the VXLAN Flood and Learn mechanism and in this topology, we will need to configure the spines to be RPs.

The configuration on the DC1 spines should be:

ip pim rp-address 10.0.0.98 group-list 224.0.0.0/4
ip pim ssm range 232.0.0.0/8
ip pim anycast-rp 10.0.0.98 10.0.0.5
ip pim anycast-rp 10.0.0.98 10.0.0.6

interface loopback1
  ip address 10.0.0.98/32
  ip router ospf UNDERLAY area 0.0.0.0
  ip pim sparse-mode

interface loopback0
  ip pim sparse-mode

int Ethernet1/1-4
  ip pim sparse-mode

The configuration on the DC2 spines should be:

ip pim rp-address 10.0.0.99 group-list 224.0.0.0/4
ip pim ssm range 232.0.0.0/8
ip pim anycast-rp 10.0.0.99 10.0.0.7
ip pim anycast-rp 10.0.0.99 10.0.0.8

interface loopback1
  ip address 10.0.0.99/32
  ip router ospf UNDERLAY area 0.0.0.0
  ip pim sparse-mode

interface loopback0
  ip pim sparse-mode

int Ethernet1/1-4
  ip pim sparse-mode

The configuration on the DC1 leaves is a little less involved:

ip pim rp-address 10.0.0.98 group-list 224.0.0.0/4
ip pim ssm range 232.0.0.0/8

interface loopback0
  ip pim sparse-mode

interface Ethernet1/1-2
  ip pim sparse-mode

And the DC2 leaves:

ip pim rp-address 10.0.0.99 group-list 224.0.0.0/4
ip pim ssm range 232.0.0.0/8

interface loopback0
  ip pim sparse-mode

interface Ethernet1/1-2
  ip pim sparse-mode

Finally, we need to put a Multicast configuration on the Core switches in both DC's too:

ip pim ssm range 232.0.0.0/8

interface loopback0
  ip pim sparse-mode

interface Ethernet1/2-4
  ip pim sparse-mode

This configuration makes the spines in each DC RPs with the assigned address and then configures the leaves to point towards their local RP. Making the setup and topology redundant.

We can validate the Multicast setup with the following command on the spines:

DC1-SPINE1# show ip pim rp 
PIM RP Status Information for VRF "default"
BSR disabled
Auto-RP disabled
BSR RP Candidate policy: None
BSR RP policy: None
Auto-RP Announce policy: None
Auto-RP Discovery policy: None

Anycast-RP 10.0.0.98 members:
  10.0.0.5*  10.0.0.6  

RP: 10.0.0.98*, (0), 
 uptime: 00:04:03   priority: 255, 
 RP-source: (local),  
 group ranges:
 224.0.0.0/4  
DC2-SPINE1# show ip pim rp 
PIM RP Status Information for VRF "default"
BSR disabled
Auto-RP disabled
BSR RP Candidate policy: None
BSR RP policy: None
Auto-RP Announce policy: None
Auto-RP Discovery policy: None

Anycast-RP 10.0.0.99 members:
  10.0.0.7*  10.0.0.8  

RP: 10.0.0.99*, (0), 
 uptime: 00:04:01   priority: 255, 
 RP-source: (local),  
 group ranges:
 224.0.0.0/4

We can see from the above output that the multicast setup is working in both DCs.

Now we can look at the final part of the underlay setup with the NVE setup.

NVE Configuration

NVE (Network Virtual Endpoint) is a logical interface where the encapsulation and de-encapsulation happens. This configuration is for the leaves and core switches only as they are the ones doing the encapsulation and de-encapsulation. It is also called the VTEP which stands for Virtual Tunnel Endpoint.

This is the base configuration for the nve1 interface:

interface nve1
  no shutdown
  host-reachability protocol bgp
  source-interface loopback0

We can check the status of the nve interface:

DC1-LEAF1# show interface nve 1
nve1 is up
admin state is up,  Hardware: NVE
  MTU 9216 bytes
  Encapsulation VXLAN
  Auto-mdix is turned off
  RX
    ucast: 0 pkts, 0 bytes - mcast: 0 pkts, 0 bytes
  TX
    ucast: 0 pkts, 0 bytes - mcast: 0 pkts, 0 bytes

The above output shows the interface is in an UP state.

We are done with the underlay setup now, and we will move onto the overlay configuration in the next part.

Categories: CiscoExpert TutorialVXLAN Multi-Site

0 Comments

Leave a Reply

Avatar placeholder

Your email address will not be published. Required fields are marked *

Related Posts

Cisco

VXLAN EVPN Leaf and Spine – YouTube Video

Here is a YouTube video I created showing the process of setting up the same Fabric as the guide I created on here: This video goes through all the parts I explained on here but Read more…

Cisco

VXLAN EVPN Multi-Site – Part 9 – External Access outside Fabric

In the last part, we got route leaking in tenant VRFs working, in this part, we will look at giving the VRFs a way out of the Fabric! We have added some new devices to Read more…

Cisco

VXLAN EVPN Multi-Site – Part 8 – L3VNI Configuration – Multi Tenant Leaking

In the last part, we got a multiple tenant VRFs working, in this part, we will look at joining the tenants up via route leaking. Lets remind ourselves of the topology: So, currently we have Read more…