In this part we are going to look at the L2VNI setup of the topology. This is basically the VLAN setup of the topology where we can define the vlans in use.
Lets remind ourselves of the topology:
The L2VNI setup is isolated to the leaves as the spines are not involved and are not specifically configured with the underlying VNI setup.
L2VNI Configuration
In our topology, we have servers in vlan 1000 and vlan 1001. By the end of this part, the servers on the same vlan will be able to communicate at layer 2 despite being separated by layer 3 boundaries.
The L2VNI configuration is nice and simple and means it can be replicated for as many layer 2 domain as your needs require.
This is the config for the Leaves in both DCs:
vlan 1000
vn-segment 2001000
vlan 1001
vn-segment 2001001
interface nve1
member vni 2001000
suppress-arp
mcast-group 239.0.0.1
member vni 2001001
suppress-arp
mcast-group 239.0.0.1
evpn
vni 2001000 l2
rd auto
route-target import auto
route-target export auto
vni 2001001 l2
rd auto
route-target import auto
route-target export autoThe configuration above creates the vlan and assigns the vn-segment ID for the vlan, this should match across the topology for this vlan. Next, we add the vni into the nve interface and add the command to suppress arp, this is a very useful command which allows the leaves to cache mac addresses of nodes attached to prevent excessive ARP messages being sent across the fabric.
The last part of the configuration adds the vni to the evpn control plane so it can be pushed into the MP-BGP topology.
We also need to put a similar config on the Core switches too but this has some multisite specific commands in it:
vlan 1000
vn-segment 2001000
vlan 1001
vn-segment 2001001
interface nve1
member vni 2001000
multisite ingress-replication
ingress-replication protocol bgp
member vni 2001001
multisite ingress-replication
ingress-replication protocol bgp
evpn
vni 2001000 l2
rd auto
route-target import auto
route-target export auto
vni 2001001 l2
rd auto
route-target import auto
route-target export autoThis is all the config we need!
Verification
We should in theory now have layer 2 communication between the servers, lets test that fact between VL1000-PC1 and VL1000-PC2 which are in the same DC:
VPCS> show ip
NAME : VPCS[1]
IP/MASK : 10.10.1.10/24
GATEWAY : 10.10.1.254
DNS :
MAC : 00:50:79:66:68:01
LPORT : 20000
RHOST:PORT : 127.0.0.1:30000
MTU : 1500
VPCS> ping 10.10.1.11
84 bytes from 10.10.1.11 icmp_seq=1 ttl=64 time=12.952 ms
84 bytes from 10.10.1.11 icmp_seq=2 ttl=64 time=8.834 ms
84 bytes from 10.10.1.11 icmp_seq=3 ttl=64 time=11.593 ms
84 bytes from 10.10.1.11 icmp_seq=4 ttl=64 time=9.253 ms
84 bytes from 10.10.1.11 icmp_seq=5 ttl=64 time=8.356 msSuccess! we have layer 2 connectivity. Lets do a few commands to confirm whats happening.
We can check the arp cache on the leaves to show the flood and learn technique in action:
DC1-LEAF1# show ip arp suppression-cache vlan 1000
Flags: + - Adjacencies synced via CFSoE
L - Local Adjacency
R - Remote Adjacency
L2 - Learnt over L2 interface
PS - Added via L2RIB, Peer Sync
RO - Dervied from L2RIB Peer Sync Entry
Ip Address Age Mac Address Vlan Physical-ifindex Flags Remote Vtep Addrs
10.10.1.10 00:00:38 0050.7966.6801 1000 Ethernet1/3 L2
10.10.1.11 00:00:43 0050.7966.6814 1000 (null) R 10.0.0.11 We can see from DC1-LEAF1 we have 10.10.1.10 (VL1000-PC1) learnt locally and then 10.10.1.11 (VL1000-PC2) learnt remotely from the remote vtep of 10.0.0.11 which is DC1-LEAF2 and is where VL1000-PC2 is connected.
We can also check the BGP l2vpn evpn table for the address 10.10.1.11 on DC1-LEAF1:
DC1-LEAF1# show bgp l2vpn evpn 10.10.1.11
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 10.0.0.9:33767 (L2VNI 2001000)
BGP routing table entry for [2]:[0]:[0]:[48]:[0050.7966.6814]:[32]:[10.10.1.11]/248, version 8
Paths: (1 available, best #1)
Flags: (0x000212) (high32 00000000) on xmit-list, is in l2rib/evpn, is not in HW
Advertised path-id 1
Path type: internal, path is valid, is best path, no labeled nexthop, in rib
Imported from 10.0.0.11:33767:[2]:[0]:[0]:[48]:[0050.7966.6814]:[32]:[10.10.1.11]/248
AS-Path: NONE, path sourced internal to AS
10.0.0.11 (metric 81) from 10.0.0.5 (10.0.0.5)
Origin IGP, MED not set, localpref 100, weight 0
Received label 2001000
Extcommunity: RT:100:2001000 ENCAP:8
Originator: 10.0.0.11 Cluster list: 10.0.0.5
Path-id 1 not advertised to any peer
Route Distinguisher: 10.0.0.11:33767
BGP routing table entry for [2]:[0]:[0]:[48]:[0050.7966.6814]:[32]:[10.10.1.11]/248, version 7
Paths: (2 available, best #2)
Flags: (0x000202) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW
Path type: internal, path is valid, not best reason: Neighbor Address, no labeled nexthop
AS-Path: NONE, path sourced internal to AS
10.0.0.11 (metric 81) from 10.0.0.6 (10.0.0.6)
Origin IGP, MED not set, localpref 100, weight 0
Received label 2001000
Extcommunity: RT:100:2001000 ENCAP:8
Originator: 10.0.0.11 Cluster list: 10.0.0.6
Advertised path-id 1
Path type: internal, path is valid, is best path, no labeled nexthop
Imported to 1 destination(s)
Imported paths list: L2-2001000
AS-Path: NONE, path sourced internal to AS
10.0.0.11 (metric 81) from 10.0.0.5 (10.0.0.5)
Origin IGP, MED not set, localpref 100, weight 0
Received label 2001000
Extcommunity: RT:100:2001000 ENCAP:8
Originator: 10.0.0.11 Cluster list: 10.0.0.5
Path-id 1 not advertised to any peerNow, thats a lot of output. But, lets break it down. We can see the top route is attached to the RD 10.0.0.9:33767. We can see that it was imported from the RD 10.0.0.11:33767 which is DC1-LEAF2. This makes sense because the route originated from 10.0.0.11. The reason there is only a single route within 10.0.0.9:33767 is because only a single route from the DC1-LEAF2 RD was marked as Best and that was the one that was imported. We can verify that by looking at the Cluster list. This is the RID of the Route Reflector (a spine).
We can also see the MAC address and IP address of VL1000-PC2 in the entry to confirm its the correct mapping.
There will be an entry on DC1-LEAF2 for the IP-MAC mapping for VL1000-PC1 for the return traffic.
Note - The mac addresses are learnt as normal so a host has to communicate on the network for the leaf to grab its MAC address and advertise it. This is the role of arp suppression in flood and learn.
Now we have communication within the local DC, lets test over the DCI!
Lets ping from VL1000-PC1 to VL1000-PC3 in DC2:
VPCS> show ip
NAME : VPCS[1]
IP/MASK : 10.10.1.10/24
GATEWAY : 10.10.1.254
DNS :
MAC : 00:50:79:66:68:01
LPORT : 20000
RHOST:PORT : 127.0.0.1:30000
MTU : 1500
VPCS> ping 10.10.1.12
84 bytes from 10.10.1.12 icmp_seq=1 ttl=64 time=19.652 ms
84 bytes from 10.10.1.12 icmp_seq=2 ttl=64 time=18.857 ms
84 bytes from 10.10.1.12 icmp_seq=3 ttl=64 time=15.974 ms
84 bytes from 10.10.1.12 icmp_seq=4 ttl=64 time=15.716 ms
84 bytes from 10.10.1.12 icmp_seq=5 ttl=64 time=20.179 msAnd a quick look at the L2VPN BGP entry for 10.10.1.10 on a leaf in DC2:
DC2-LEAF1# show bgp l2vpn evpn 10.10.1.10
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 200:2001000
BGP routing table entry for [2]:[0]:[0]:[48]:[0050.7966.6801]:[32]:[10.10.1.10]/248, version 163
Paths: (2 available, best #1)
Flags: (0x000202) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW
Advertised path-id 1
Path type: internal, path is valid, is best path, no labeled nexthop
Imported to 1 destination(s)
Imported paths list: L2-2001000
AS-Path: 100 , path sourced external to AS
10.222.222.1 (metric 81) from 10.0.0.7 (10.0.0.7)
Origin IGP, MED 2000, localpref 100, weight 0
Received label 2001000
Extcommunity: RT:200:2001000 ENCAP:8
Originator: 10.0.0.3 Cluster list: 10.0.0.7
Path type: internal, path is valid, not best reason: Neighbor Address, no labeled nexthop
AS-Path: 100 , path sourced external to AS
10.222.222.1 (metric 81) from 10.0.0.8 (10.0.0.8)
Origin IGP, MED 2000, localpref 100, weight 0
Received label 2001000
Extcommunity: RT:200:2001000 ENCAP:8
Originator: 10.0.0.3 Cluster list: 10.0.0.8
Path-id 1 not advertised to any peer
Route Distinguisher: 10.0.0.10:33767 (L2VNI 2001000)
BGP routing table entry for [2]:[0]:[0]:[48]:[0050.7966.6801]:[32]:[10.10.1.10]/248, version 171
Paths: (1 available, best #1)
Flags: (0x000212) (high32 00000000) on xmit-list, is in l2rib/evpn, is not in HW
Advertised path-id 1
Path type: internal, path is valid, is best path, no labeled nexthop, in rib
Imported from 200:2001000:[2]:[0]:[0]:[48]:[0050.7966.6801]:[32]:[10.10.1.10]/248
AS-Path: 100 , path sourced external to AS
10.222.222.1 (metric 81) from 10.0.0.7 (10.0.0.7)
Origin IGP, MED 2000, localpref 100, weight 0
Received label 2001000
Extcommunity: RT:200:2001000 ENCAP:8
Originator: 10.0.0.3 Cluster list: 10.0.0.7
Path-id 1 not advertised to any peerWe can see that the route has come from AS100 which is DC1 and its been advertised by the Route Reflectors in DC2 (Spines). The IP address (10.222.222.1) in the BGP table is the loopback100 interface of DC2-CORE1 which is denoted as the multisite border-gateway loopback interface on the nve interface.
In the next part we will look at setting up anycast gateways for the L2VNIs so we can start the process of getting the hosts of their layer 2 network!
0 Comments