VXLAN EVPN Multi-Site – Part 4 – L2VNI Configuration

In this part we are going to look at the L2VNI setup of the topology. This is basically the VLAN setup of the topology where we can define the vlans in use.

Lets remind ourselves of the topology:

The L2VNI setup is isolated to the leaves as the spines are not involved and are not specifically configured with the underlying VNI setup.

L2VNI Configuration

In our topology, we have servers in vlan 1000 and vlan 1001. By the end of this part, the servers on the same vlan will be able to communicate at layer 2 despite being separated by layer 3 boundaries.

The L2VNI configuration is nice and simple and means it can be replicated for as many layer 2 domain as your needs require.

This is the config for the Leaves in both DCs:

vlan 1000
  vn-segment 2001000
vlan 1001
  vn-segment 2001001

interface nve1
  member vni 2001000
    suppress-arp
    mcast-group 239.0.0.1
  member vni 2001001
    suppress-arp
    mcast-group 239.0.0.1

evpn
  vni 2001000 l2
    rd auto
    route-target import auto
    route-target export auto
  vni 2001001 l2
    rd auto
    route-target import auto
    route-target export auto

The configuration above creates the vlan and assigns the vn-segment ID for the vlan, this should match across the topology for this vlan. Next, we add the vni into the nve interface and add the command to suppress arp, this is a very useful command which allows the leaves to cache mac addresses of nodes attached to prevent excessive ARP messages being sent across the fabric.

The last part of the configuration adds the vni to the evpn control plane so it can be pushed into the MP-BGP topology.

We also need to put a similar config on the Core switches too but this has some multisite specific commands in it:

vlan 1000
  vn-segment 2001000
vlan 1001
  vn-segment 2001001

interface nve1
  member vni 2001000
    multisite ingress-replication
    ingress-replication protocol bgp
  member vni 2001001
    multisite ingress-replication
    ingress-replication protocol bgp

evpn
  vni 2001000 l2
    rd auto
    route-target import auto
    route-target export auto
  vni 2001001 l2
    rd auto
    route-target import auto
    route-target export auto

This is all the config we need!

Verification

We should in theory now have layer 2 communication between the servers, lets test that fact between VL1000-PC1 and VL1000-PC2 which are in the same DC:

VPCS> show ip        

NAME        : VPCS[1]
IP/MASK     : 10.10.1.10/24
GATEWAY     : 10.10.1.254
DNS         : 
MAC         : 00:50:79:66:68:01
LPORT       : 20000
RHOST:PORT  : 127.0.0.1:30000
MTU         : 1500

VPCS> ping 10.10.1.11

84 bytes from 10.10.1.11 icmp_seq=1 ttl=64 time=12.952 ms
84 bytes from 10.10.1.11 icmp_seq=2 ttl=64 time=8.834 ms
84 bytes from 10.10.1.11 icmp_seq=3 ttl=64 time=11.593 ms
84 bytes from 10.10.1.11 icmp_seq=4 ttl=64 time=9.253 ms
84 bytes from 10.10.1.11 icmp_seq=5 ttl=64 time=8.356 ms

Success! we have layer 2 connectivity. Lets do a few commands to confirm whats happening.

We can check the arp cache on the leaves to show the flood and learn technique in action:

DC1-LEAF1# show ip arp suppression-cache vlan 1000

Flags: + - Adjacencies synced via CFSoE
       L - Local Adjacency
       R - Remote Adjacency
       L2 - Learnt over L2 interface
       PS - Added via L2RIB, Peer Sync
       RO - Dervied from L2RIB Peer Sync Entry

Ip Address      Age      Mac Address    Vlan Physical-ifindex    Flags    Remote Vtep Addrs

10.10.1.10      00:00:38 0050.7966.6801 1000 Ethernet1/3         L2
10.10.1.11      00:00:43 0050.7966.6814 1000 (null)              R        10.0.0.11      

We can see from DC1-LEAF1 we have 10.10.1.10 (VL1000-PC1) learnt locally and then 10.10.1.11 (VL1000-PC2) learnt remotely from the remote vtep of 10.0.0.11 which is DC1-LEAF2 and is where VL1000-PC2 is connected.

We can also check the BGP l2vpn evpn table for the address 10.10.1.11 on DC1-LEAF1:

DC1-LEAF1# show bgp l2vpn evpn 10.10.1.11
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 10.0.0.9:33767    (L2VNI 2001000)
BGP routing table entry for [2]:[0]:[0]:[48]:[0050.7966.6814]:[32]:[10.10.1.11]/248, version 8
Paths: (1 available, best #1)
Flags: (0x000212) (high32 00000000) on xmit-list, is in l2rib/evpn, is not in HW

  Advertised path-id 1
  Path type: internal, path is valid, is best path, no labeled nexthop, in rib
             Imported from 10.0.0.11:33767:[2]:[0]:[0]:[48]:[0050.7966.6814]:[32]:[10.10.1.11]/248 
  AS-Path: NONE, path sourced internal to AS
    10.0.0.11 (metric 81) from 10.0.0.5 (10.0.0.5)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 2001000
      Extcommunity: RT:100:2001000 ENCAP:8
      Originator: 10.0.0.11 Cluster list: 10.0.0.5 

  Path-id 1 not advertised to any peer

Route Distinguisher: 10.0.0.11:33767
BGP routing table entry for [2]:[0]:[0]:[48]:[0050.7966.6814]:[32]:[10.10.1.11]/248, version 7
Paths: (2 available, best #2)
Flags: (0x000202) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW

  Path type: internal, path is valid, not best reason: Neighbor Address, no labeled nexthop
  AS-Path: NONE, path sourced internal to AS
    10.0.0.11 (metric 81) from 10.0.0.6 (10.0.0.6)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 2001000
      Extcommunity: RT:100:2001000 ENCAP:8
      Originator: 10.0.0.11 Cluster list: 10.0.0.6 

  Advertised path-id 1
  Path type: internal, path is valid, is best path, no labeled nexthop
             Imported to 1 destination(s)
             Imported paths list: L2-2001000
  AS-Path: NONE, path sourced internal to AS
    10.0.0.11 (metric 81) from 10.0.0.5 (10.0.0.5)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 2001000
      Extcommunity: RT:100:2001000 ENCAP:8
      Originator: 10.0.0.11 Cluster list: 10.0.0.5 

  Path-id 1 not advertised to any peer

Now, thats a lot of output. But, lets break it down. We can see the top route is attached to the RD 10.0.0.9:33767. We can see that it was imported from the RD 10.0.0.11:33767 which is DC1-LEAF2. This makes sense because the route originated from 10.0.0.11. The reason there is only a single route within 10.0.0.9:33767 is because only a single route from the DC1-LEAF2 RD was marked as Best and that was the one that was imported. We can verify that by looking at the Cluster list. This is the RID of the Route Reflector (a spine).

We can also see the MAC address and IP address of VL1000-PC2 in the entry to confirm its the correct mapping.

There will be an entry on DC1-LEAF2 for the IP-MAC mapping for VL1000-PC1 for the return traffic.

Note - The mac addresses are learnt as normal so a host has to communicate on the network for the leaf to grab its MAC address and advertise it. This is the role of arp suppression in flood and learn.

Now we have communication within the local DC, lets test over the DCI!

Lets ping from VL1000-PC1 to VL1000-PC3 in DC2:

VPCS> show ip        

NAME        : VPCS[1]
IP/MASK     : 10.10.1.10/24
GATEWAY     : 10.10.1.254
DNS         : 
MAC         : 00:50:79:66:68:01
LPORT       : 20000
RHOST:PORT  : 127.0.0.1:30000
MTU         : 1500

VPCS> ping 10.10.1.12

84 bytes from 10.10.1.12 icmp_seq=1 ttl=64 time=19.652 ms
84 bytes from 10.10.1.12 icmp_seq=2 ttl=64 time=18.857 ms
84 bytes from 10.10.1.12 icmp_seq=3 ttl=64 time=15.974 ms
84 bytes from 10.10.1.12 icmp_seq=4 ttl=64 time=15.716 ms
84 bytes from 10.10.1.12 icmp_seq=5 ttl=64 time=20.179 ms

And a quick look at the L2VPN BGP entry for 10.10.1.10 on a leaf in DC2:

DC2-LEAF1# show bgp l2vpn evpn 10.10.1.10
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 200:2001000
BGP routing table entry for [2]:[0]:[0]:[48]:[0050.7966.6801]:[32]:[10.10.1.10]/248, version 163
Paths: (2 available, best #1)
Flags: (0x000202) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW

  Advertised path-id 1
  Path type: internal, path is valid, is best path, no labeled nexthop
             Imported to 1 destination(s)
             Imported paths list: L2-2001000
  AS-Path: 100 , path sourced external to AS
    10.222.222.1 (metric 81) from 10.0.0.7 (10.0.0.7)
      Origin IGP, MED 2000, localpref 100, weight 0
      Received label 2001000
      Extcommunity: RT:200:2001000 ENCAP:8
      Originator: 10.0.0.3 Cluster list: 10.0.0.7 

  Path type: internal, path is valid, not best reason: Neighbor Address, no labeled nexthop
  AS-Path: 100 , path sourced external to AS
    10.222.222.1 (metric 81) from 10.0.0.8 (10.0.0.8)
      Origin IGP, MED 2000, localpref 100, weight 0
      Received label 2001000
      Extcommunity: RT:200:2001000 ENCAP:8
      Originator: 10.0.0.3 Cluster list: 10.0.0.8 

  Path-id 1 not advertised to any peer

Route Distinguisher: 10.0.0.10:33767    (L2VNI 2001000)
BGP routing table entry for [2]:[0]:[0]:[48]:[0050.7966.6801]:[32]:[10.10.1.10]/248, version 171
Paths: (1 available, best #1)
Flags: (0x000212) (high32 00000000) on xmit-list, is in l2rib/evpn, is not in HW

  Advertised path-id 1
  Path type: internal, path is valid, is best path, no labeled nexthop, in rib
             Imported from 200:2001000:[2]:[0]:[0]:[48]:[0050.7966.6801]:[32]:[10.10.1.10]/248 
  AS-Path: 100 , path sourced external to AS
    10.222.222.1 (metric 81) from 10.0.0.7 (10.0.0.7)
      Origin IGP, MED 2000, localpref 100, weight 0
      Received label 2001000
      Extcommunity: RT:200:2001000 ENCAP:8
      Originator: 10.0.0.3 Cluster list: 10.0.0.7 

  Path-id 1 not advertised to any peer

We can see that the route has come from AS100 which is DC1 and its been advertised by the Route Reflectors in DC2 (Spines). The IP address (10.222.222.1) in the BGP table is the loopback100 interface of DC2-CORE1 which is denoted as the multisite border-gateway loopback interface on the nve interface.

In the next part we will look at setting up anycast gateways for the L2VNIs so we can start the process of getting the hosts of their layer 2 network!