vPC VXLAN EVPN Leaf and Spine - Packet Walk with ARP Glean¶
While writing another blog post, I found that I had connectivity between two endpoints that I seemingly shouldn't have. This prompted me to look into why I was able to communicate and the results are pretty interesting.
In this post im going to complete an end-to-end packet walk showing the communications between two L2VNIs over an L3VNI. This is the topology:
Some background information on this topology:
Loopback0's (Router IDs):
spine-1: 10.0.0.1/32 spine-2: 10.0.0.2/32 leaf-1: 10.0.0.3/32 leaf-2: 10.0.0.4/32 leaf-3: 10.0.0.5/32 leaf-4: 10.0.0.6/32 leaf-5: 10.0.0.7/32 leaf-6: 10.0.0.8/32
Loopback1's (VTEPs):
leaf-1: 10.0.1.1/32 leaf-2: 10.0.1.2/32 vpc domain 100 anycast vtep: 10.0.1.101/32 leaf-3: 10.0.1.3/32 leaf-4: 10.0.1.4/32 vpc domain 200 anycast vtep: 10.0.1.102/32 leaf-5: 10.0.1.5/32 leaf-6: 10.0.1.6/32 vpc domain 300 anycast vtep: 10.0.1.103/32
The VNI mappings are as follows:
vlan 10 - vn-segment 100010 vlan 20 - vn-segment 100020
We are looking at connectivity between server-3-vl20 (10.2.1.3) and server-2-vl10 (10.1.1.2). As you can tell by their names, they are on different Vlans and therefore will use the L3VNI to route between. This topology also has Anycast Gateways. ARP Suppression is enabled on both L2VNIs too.
Route Tables Before¶
Lets check the route tables in the tenant VRF on the relevant leaves:
LEAF-5# show ip route vrf OVERLAY-TENANT1
IP Route Table for VRF "OVERLAY-TENANT1"
10.1.1.0/24, ubest/mbest: 1/0
*via 10.0.1.1%default, [200/0], 00:00:09, bgp-64500, internal, tag 64500, segid: 100999 tunnelid: 0xa000101 encap: VXLAN
10.2.1.0/24, ubest/mbest: 1/0, attached
*via 10.2.1.254, Vlan20, [0/0], 03:21:53, direct
10.2.1.254/32, ubest/mbest: 1/0, attached
*via 10.2.1.254, Vlan20, [0/0], 03:21:53, local
LEAF-6# show ip route vrf OVERLAY-TENANT1
IP Route Table for VRF "OVERLAY-TENANT1"
10.1.1.0/24, ubest/mbest: 1/0
*via 10.0.1.1%default, [200/0], 00:00:28, bgp-64500, internal, tag 64500, segid: 100999 tunnelid: 0xa000101 encap: VXLAN
10.2.1.0/24, ubest/mbest: 1/0, attached
*via 10.2.1.254, Vlan20, [0/0], 03:22:10, direct
10.2.1.254/32, ubest/mbest: 1/0, attached
*via 10.2.1.254, Vlan20, [0/0], 03:22:10, local
LEAF-3# show ip route vrf OVERLAY-TENANT1
IP Route Table for VRF "OVERLAY-TENANT1"
10.1.1.0/24, ubest/mbest: 1/0, attached
*via 10.1.1.254, Vlan10, [0/0], 03:22:39, direct
10.1.1.254/32, ubest/mbest: 1/0, attached
*via 10.1.1.254, Vlan10, [0/0], 03:22:39, local
10.2.1.0/24, ubest/mbest: 1/0
*via 10.0.1.5%default, [200/0], 00:00:49, bgp-64500, internal, tag 64500, segid: 100999 tunnelid: 0xa000105 encap: VXLAN
LEAF-4# show ip route vrf OVERLAY-TENANT1
IP Route Table for VRF "OVERLAY-TENANT1"
10.1.1.0/24, ubest/mbest: 1/0, attached
*via 10.1.1.254, Vlan10, [0/0], 03:23:01, direct
10.1.1.254/32, ubest/mbest: 1/0, attached
*via 10.1.1.254, Vlan10, [0/0], 03:23:01, local
10.2.1.0/24, ubest/mbest: 1/0
*via 10.0.1.5%default, [200/0], 00:01:13, bgp-64500, internal, tag 64500, segid: 100999 tunnelid: 0xa000105 encap: VXLAN
The route tables show that we have local and direct routes for the anycast gateway IPs and subnets, and then a remote route for the opposing anycast gateway and subnet. The interesting part here is that for the leaves that connect server-3-vl20, its only route to be able to get to server-2-vl10 is using 10.1.1.0/24 as there is no /32 host route for 10.1.1.2 yet as the client hasn't yet spoken on the network. It was my initial assumption because the route for 10.1.1.0/24 is being preferred via 10.0.1.1 (LEAF-1) that ths vl20 server wouldn't be able to communicate with the vl10 server connected to the middle two leaves, as the traffic would be sent to LEAF-1... right? Well, thats not the case, and there is a pretty cool reason why!
Lets follow the packets!
Packet Walk¶
The first thing server-3-vl20 does is ARP for its default gateway which is 10.2.1.254:
There is no VXLAN or anything here, this is just basic networking. The anycast gateway responds to the ARP request and gives the server the anycast mac address which I have set as aaaa.bbbb.cccc. This now populates the servers local ARP cache.
The server, now knowing the mac address of its default gateway, knows that the remote server is not on its own network, so it sends the ping to its default gateway, still not encapsulated in any VXLAN headers:
This first ping fails with no response as in the background things are still going on. This packet is seen on the link going from server-3-vl20 to LEAF-5. The outgoing packet is seen heading from LEAF-5 to the spines:
In this packet, we see more headers. The outer IP header (3rd down) shows the packet being sourced from the Anycast VTEP for the vPC domain and the destination is 10.0.1.1 which is LEAF-1. Its being sent there because the route lookup on LEAF-5 matches 10.1.1.0/24 which was shown in the route captures above. As we know, thats not where the remote host lives...
Further down in that packets headers we can see its being carried by the L3VNI as its routed traffic rather than bridged:
And the inner IP header we can see shows the two end hosts. Though, just above that, we see the inner ethernet header. The MAC ending 01:67 is a MAC on LEAF-5 and LEAF-6:
LEAF-5# show mac address-table address 0200.0a00.0167
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False, C - ControlPlane MAC, ~ - vsan,
(NA)- Not Applicable
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
* 999 0200.0a00.0167 static - F F Vlan999
G - 0200.0a00.0167 static - F F sup-eth1(R)
The destination MAC address is the NVE interface MAC of LEAF-1:
LEAF-1# show nve interface | inc MAC
Local Router MAC: 5003.0000.1b08
When this packet lands at LEAF-1, with the help of some debug commands, we can see what it does:
2024 Dec 20 03:29:29.882504 arp: arp_add_adj: ARP Entry Added: IP=10.1.1.2, MAC=0000.0000.0000, rcvdMAC=NULL, State=1, Preference=255, Interface=Vlan10, PhyInterface=Vlan10, Ismct=0, Flags=0x10, Rearp(Interval=2, Count=4), TTL=30s, UpdateSuppCache=FALSE SenttoAM=TRUE
2024 Dec 20 03:29:29.882851 arp: arp_send_request_common: Vinci: Set AnycastGlean: AnycastGlean=TRUE
2024 Dec 20 03:29:29.882922 arp: arp_send_request_common: Vinci: Sending ARP packet with SMAC to hosts: SMAC=aaaa.bbbb.cccc
2024 Dec 20 03:29:29.882992 arp: arp_send_request_common: Vinci: Sending ARP packet with SMAC to fabric: SMAC=5003.0000.1b08
2024 Dec 20 03:29:29.883372 arp: arp_process_mac_msg: ARP process MAC message Event=receivedSendRequest, Action=processSendRequest, PID=5682
2024 Dec 20 03:29:30.121418 arp: arp_l2rib_msg_cb: (Type=Route) Length=224, RTFlag=0, Sequence=0, Delete=0, Producer=5, PeerID=0, MAC=4e29.205e.baf4, RemotehostIP=10.1.1.2, AFI=2
2024 Dec 20 03:29:30.121498 arp: arp_l2rib_msg_cb: hostIP=10.1.1.2, RemoteVtepAddrCount=1
2024 Dec 20 03:29:30.121510 arp: arp_l2rib_msg_cb: RNHs=10.0.1.102
2024 Dec 20 03:29:30.122091 arp: arp_cache_create_cache_node: hostIP=10.1.1.2, RemoteVtepAddrCount=1
2024 Dec 20 03:29:30.122117 arp: arp_cache_create_cache_node: RNHs=10.0.1.102
2024 Dec 20 03:29:30.122208 arp: arp_cache_create_cache_node: Created new cache entry uuid=1290, VLAN=10, IP=10.1.1.2, MAC=4e29.205e.baf4, Local=FALSE, MACsCount=1
2024 Dec 20 03:29:31.882565 arp: arp_adj_timer_callback: Vinci: ARP callback, found route present in URIB for IP=10.1.1.2, host not local.
This is a very verbose output. However, reading through, it appears as though LEAF-1 completes an ARP Gleaning process. It gets the packet destined for 10.1.1.2 and ARPs for it:
This packet in its outer IP header is destined for 224.1.1.192 which is the mcast-group set on the VNI. In this topology we are using anycast RP so either SPINE-1 or SPINE-2 could get this packet. Looking at the multicast routing table on each spine, we should see the same outgoing interfaces. Which essentially means that this ARP packet is going to be flooded to all VTEPs.
Due to the fact we have arp suppression enabled, no ARP response is seen directly back to LEAF-1. However, the ARP request does make it to server-2-vl10:
This response from the server, triggers the normal advertisement for VXLAN and a type-2 route for the MAC and IP of server-2-vl10 is sent from the local leaves up to the Spines, which are then sent to all the other leaves.
We can see this being sent down to LEAF-5 where the ping originated:
This shows one per spine, if we take a deeper look into the packets, we can see the NLRI:
Showing that this is EVPN NLRI and is a type-2 route. The mac address is the mac address of the NIC on server-2-vl10. We also see the IPv4 address too. The next hop is the Anycast VTEP IP address of LEAF-3 and LEAF-4 to prove its origin.
This populates an entry in the route table on the leaves local to server-3-vl20. In server-3-vl20 talking on the network in the first place, the standard advertisement happens from LEAF-5 and LEAF-6 and we can see this being advertised to the spines just after we get the NLRI for server-2-vl10:
These are sourced from LEAF-5 to the spines and contain NLRI for 10.2.1.3 which is the IP address of server-3-vl20. This competes the learning and now the pings begin to succeed and are sent as Unicast packets across the fabric:
The above is the ping that got a response, we can see that its outer IP header is direct between the Anycast VTEP IPs and the inner ethernet header contains the "Virtual Router MAC" of LEAF-3 and LEAF-4:
LEAF-3# show nve interface nve 1 detail | inc "Virtual Router MAC"
Virtual Router MAC: 0200.0a00.0166
And that completes the packet walk. I have put together a diagram that shows the above in a single diagram with steps:
The cool thing here is that even though LEAF-1 received the initial packet, it completed an ARP gleaning process to aid the fabric in learning the endpoint and allowing the traffic to flow. For the record, 10.1.1.0/24 is being advertised by leaves 1-4. However, due to the BGP path selection algorithm, LEAF-1 was chosen as the best.