In this series of guides, we will be expanding on the previous guides by making a multi-site VXLAN EVPN topology. In this part, we will look at the topology and setup the underlay network using OSPF.
Here is the topology we will be working with:
The switches are all Nexus C9500v running NX-OS version 10.3(3).
Firstly we should enable all the features we need for the configuration on all switches:
feature ospf
feature bgp
feature pim
feature fabric forwarding
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
Each of the switches have a lookback0 interface configured with IP addresses:
DC1-CORE1: 10.0.0.1/32
DC1-CORE2: 10.0.0.2/32
DC2-CORE1: 10.0.0.3/32
DC2-CORE2: 10.0.0.4/32
DC1-SPINE1: 10.0.0.5/32
DC1-SPINE2: 10.0.0.6/32
DC2-SPINE1: 10.0.0.7/32
DC2-SPINE2: 10.0.0.8/32
DC1-LEAF1: 10.0.0.9/32
DC2-LEAF1: 10.0.0.10/32
DC1-LEAF2: 10.0.0.11/32
DC2-LEAF2: 10.0.0.12/32
Routing Configuration
All switches have a basic OSPF configuration:
router ospf UNDERLAY
log-adjacency-changes
Then each of the loopback0 interfaces are configured for OSPF in area 0.0.0.0:
interface Loopback0
ip router ospf UNDERLAY area 0.0.0.0
Interface Configuration
The Spine interfaces towards the the leaves are configured as layer 3 ports using the loopback interface:
interface Ethernet1/3-4
no switchport
mtu 9216
medium p2p
ip unnumbered loopback0
ip router ospf UNDERLAY area 0.0.0.0
ip ospf network point-to-point
no shutdown
The Leaf interfaces towards the the spines are also configured as layer 3 ports using the loopback interface:
interface Ethernet1/1-2
no switchport
mtu 9216
medium p2p
ip unnumbered loopback0
ip router ospf UNDERLAY area 0.0.0.0
ip ospf network point-to-point
no shutdown
The interfaces between the Core switches and Spine switches also need configuring as Layer3 interfaces with the IP addressing on the diagram:
interface Ethernet1/X
no switchport
mtu 9216
ip address ............
ip ospf network point-to-point
medium p2p
ip router ospf UNDERLAY area 0.0.0.0
no shutdown
These interfaces also need to be configured in the OSPF network. The above is a general template with the IP addressing required.
There are also interfaces between the Core switches within each DC, these should be configured as per the below:
interface Ethernet1/4
no switchport
mtu 9216
ip address ............
ip ospf network point-to-point
ip router ospf UNDERLAY area 0.0.0.0
no shutdown
Finally, the DCI links between the two DCs need configuring, these are not in the OSPF domain and are just normal Layer3 interfaces:
interface Ethernet1/1
no switchport
mtu 9216
ip address ..............
medium p2p
no shutdown
We should now have basic connectivity within the topology, lets check the routing table on DC1-SPINE1
for the OSPF routes:
DC1-SPINE1# show ip route ospf-UNDERLAY
10.0.0.1/32, ubest/mbest: 1/0
*via 10.65.0.1, Eth1/1, [110/41], 00:09:46, ospf-UNDERLAY, intra
10.0.0.2/32, ubest/mbest: 1/0
*via 10.65.1.1, Eth1/2, [110/41], 00:09:42, ospf-UNDERLAY, intra
10.0.0.6/32, ubest/mbest: 4/0
*via 10.0.0.9, Eth1/3, [110/81], 00:20:59, ospf-UNDERLAY, intra
*via 10.0.0.11, Eth1/4, [110/81], 00:20:54, ospf-UNDERLAY, intra
*via 10.65.0.1, Eth1/1, [110/81], 00:09:45, ospf-UNDERLAY, intra
*via 10.65.1.1, Eth1/2, [110/81], 00:09:42, ospf-UNDERLAY, intra
10.0.0.9/32, ubest/mbest: 1/0
*via 10.0.0.9, Eth1/3, [110/41], 00:21:00, ospf-UNDERLAY, intra
10.0.0.11/32, ubest/mbest: 1/0
*via 10.0.0.11, Eth1/4, [110/41], 00:20:54, ospf-UNDERLAY, intra
10.65.2.0/24, ubest/mbest: 1/0
*via 10.65.0.1, Eth1/1, [110/80], 00:09:46, ospf-UNDERLAY, intra
10.65.3.0/24, ubest/mbest: 1/0
*via 10.65.1.1, Eth1/2, [110/80], 00:09:42, ospf-UNDERLAY, intra
10.65.10.0/24, ubest/mbest: 2/0
*via 10.65.0.1, Eth1/1, [110/80], 00:01:47, ospf-UNDERLAY, intra
*via 10.65.1.1, Eth1/2, [110/80], 00:01:36, ospf-UNDERLAY, intra
Lets also look in the other DC at DC2-SPINE1
:
DC2-SPINE1# show ip route ospf-UNDERLAY
10.0.0.3/32, ubest/mbest: 1/0
*via 10.66.0.1, Eth1/1, [110/41], 00:10:29, ospf-UNDERLAY, intra
10.0.0.4/32, ubest/mbest: 1/0
*via 10.66.1.1, Eth1/2, [110/41], 00:07:00, ospf-UNDERLAY, intra
10.0.0.8/32, ubest/mbest: 4/0
*via 10.0.0.10, Eth1/3, [110/81], 00:21:43, ospf-UNDERLAY, intra
*via 10.0.0.12, Eth1/4, [110/81], 00:21:35, ospf-UNDERLAY, intra
*via 10.66.0.1, Eth1/1, [110/81], 00:10:29, ospf-UNDERLAY, intra
*via 10.66.1.1, Eth1/2, [110/81], 00:06:55, ospf-UNDERLAY, intra
10.0.0.10/32, ubest/mbest: 1/0
*via 10.0.0.10, Eth1/3, [110/41], 00:21:43, ospf-UNDERLAY, intra
10.0.0.12/32, ubest/mbest: 1/0
*via 10.0.0.12, Eth1/4, [110/41], 00:21:35, ospf-UNDERLAY, intra
10.66.2.0/24, ubest/mbest: 1/0
*via 10.66.0.1, Eth1/1, [110/80], 00:10:29, ospf-UNDERLAY, intra
10.66.3.0/24, ubest/mbest: 1/0
*via 10.66.1.1, Eth1/2, [110/80], 00:07:00, ospf-UNDERLAY, intra
10.66.10.0/24, ubest/mbest: 2/0
*via 10.66.0.1, Eth1/1, [110/80], 00:02:14, ospf-UNDERLAY, intra
*via 10.66.1.1, Eth1/2, [110/80], 00:01:08, ospf-UNDERLAY, intra
The above shows that we are seeing all the correct routes in the underlay. Note that we do not see inter DC routes because the DCI links cut the underlay routing protocol in half because they don't participate. Now we can move onto the Multicast setup.
Multicast Configuration
PIM is configured for the VXLAN Flood and Learn mechanism and in this topology, we will need to configure the spines to be RPs.
The configuration on the DC1 spines should be:
ip pim rp-address 10.0.0.98 group-list 224.0.0.0/4
ip pim ssm range 232.0.0.0/8
ip pim anycast-rp 10.0.0.98 10.0.0.5
ip pim anycast-rp 10.0.0.98 10.0.0.6
interface loopback1
ip address 10.0.0.98/32
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
interface loopback0
ip pim sparse-mode
int Ethernet1/1-4
ip pim sparse-mode
The configuration on the DC2 spines should be:
ip pim rp-address 10.0.0.99 group-list 224.0.0.0/4
ip pim ssm range 232.0.0.0/8
ip pim anycast-rp 10.0.0.99 10.0.0.7
ip pim anycast-rp 10.0.0.99 10.0.0.8
interface loopback1
ip address 10.0.0.99/32
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
interface loopback0
ip pim sparse-mode
int Ethernet1/1-4
ip pim sparse-mode
The configuration on the DC1 leaves is a little less involved:
ip pim rp-address 10.0.0.98 group-list 224.0.0.0/4
ip pim ssm range 232.0.0.0/8
interface loopback0
ip pim sparse-mode
interface Ethernet1/1-2
ip pim sparse-mode
And the DC2 leaves:
ip pim rp-address 10.0.0.99 group-list 224.0.0.0/4
ip pim ssm range 232.0.0.0/8
interface loopback0
ip pim sparse-mode
interface Ethernet1/1-2
ip pim sparse-mode
Finally, we need to put a Multicast configuration on the Core switches in both DC's too:
ip pim ssm range 232.0.0.0/8
interface loopback0
ip pim sparse-mode
interface Ethernet1/2-4
ip pim sparse-mode
This configuration makes the spines in each DC RPs with the assigned address and then configures the leaves to point towards their local RP. Making the setup and topology redundant.
We can validate the Multicast setup with the following command on the spines:
DC1-SPINE1# show ip pim rp
PIM RP Status Information for VRF "default"
BSR disabled
Auto-RP disabled
BSR RP Candidate policy: None
BSR RP policy: None
Auto-RP Announce policy: None
Auto-RP Discovery policy: None
Anycast-RP 10.0.0.98 members:
10.0.0.5* 10.0.0.6
RP: 10.0.0.98*, (0),
uptime: 00:04:03 priority: 255,
RP-source: (local),
group ranges:
224.0.0.0/4
DC2-SPINE1# show ip pim rp
PIM RP Status Information for VRF "default"
BSR disabled
Auto-RP disabled
BSR RP Candidate policy: None
BSR RP policy: None
Auto-RP Announce policy: None
Auto-RP Discovery policy: None
Anycast-RP 10.0.0.99 members:
10.0.0.7* 10.0.0.8
RP: 10.0.0.99*, (0),
uptime: 00:04:01 priority: 255,
RP-source: (local),
group ranges:
224.0.0.0/4
We can see from the above output that the multicast setup is working in both DCs.
Now we can look at the final part of the underlay setup with the NVE setup.
NVE Configuration
NVE (Network Virtual Endpoint) is a logical interface where the encapsulation and de-encapsulation happens. This configuration is for the leaves and core switches only as they are the ones doing the encapsulation and de-encapsulation. It is also called the VTEP which stands for Virtual Tunnel Endpoint.
This is the base configuration for the nve1 interface:
interface nve1
no shutdown
host-reachability protocol bgp
source-interface loopback0
We can check the status of the nve interface:
DC1-LEAF1# show interface nve 1
nve1 is up
admin state is up, Hardware: NVE
MTU 9216 bytes
Encapsulation VXLAN
Auto-mdix is turned off
RX
ucast: 0 pkts, 0 bytes - mcast: 0 pkts, 0 bytes
TX
ucast: 0 pkts, 0 bytes - mcast: 0 pkts, 0 bytes
The above output shows the interface is in an UP state.
We are done with the underlay setup now, and we will move onto the overlay configuration in the next part.
0 Comments