Interpreting VXLAN EVPN BGP Output – Type-2 Routes

Published by Nick Carlton on

The BGP output from a show bgp l2vpn evpn command is normally very long and sometimes confusing to understand. In this guide, I will walk through a sample output of a type-2 route and show why we get all the output and what it means.

Before reading this, it implies that you understand what route distinguishers and route targets are in relation to BGP and how they are used. Without background knowledge of these constructs, it may be hard(er) to follow.

A type-2 route is a mac address or mac-ip address based route, it holds at minimum a mac-address and does commonly have an IP address in there too. Though sometimes, you may see 0.0.0.0 which indicates that its just a mac entry, this happens when the VTEP sees traffic from an endpoint without an ARP request, which is somewhat uncommon.

We have a client 10.1.1.1 which lives on vlan-10, what vlan is mapped locally on the switch to vni-100010. This VNI is also part of an L3VNI which is vni-100999.

The topology has vPC switches and 2 spines acting as route reflectors.

This is the full output we get:

LEAF-5# show bgp l2vpn evpn 10.1.1.1
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 10.0.0.3:32777
BGP routing table entry for [2]:[0]:[0]:[48]:[a2b9.4605.948d]:[32]:[10.1.1.1]/272, version 424
Paths: (2 available, best #1)
Flags: (0x000202) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW

  Advertised path-id 1
  Path type: internal, path is valid, is best path, no labeled nexthop
             Imported to 3 destination(s)
             Imported paths list: OVERLAY-TENANT1 L3-100999 L2-100010
  AS-Path: NONE, path sourced internal to AS
    10.0.1.101 (metric 81) from 10.0.0.1 (10.0.0.1)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 100010 100999
      Extcommunity: RT:64500:100010 RT:64500:100999 SOO:10.0.1.101:0 ENCAP:8
          Router MAC:0200.0a00.0165
      Originator: 10.0.0.3 Cluster list: 10.0.0.1 

  Path type: internal, path is valid, not best reason: Neighbor Address, no labeled nexthop
  AS-Path: NONE, path sourced internal to AS
    10.0.1.101 (metric 81) from 10.0.0.2 (10.0.0.2)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 100010 100999
      Extcommunity: RT:64500:100010 RT:64500:100999 SOO:10.0.1.101:0 ENCAP:8
          Router MAC:0200.0a00.0165
      Originator: 10.0.0.3 Cluster list: 10.0.0.2 

  Path-id 1 not advertised to any peer

Route Distinguisher: 10.0.0.4:32777
BGP routing table entry for [2]:[0]:[0]:[48]:[a2b9.4605.948d]:[32]:[10.1.1.1]/272, version 433
Paths: (2 available, best #2)
Flags: (0x000202) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW

  Path type: internal, path is valid, not best reason: Neighbor Address, no labeled nexthop
  AS-Path: NONE, path sourced internal to AS
    10.0.1.101 (metric 81) from 10.0.0.2 (10.0.0.2)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 100010 100999
      Extcommunity: RT:64500:100010 RT:64500:100999 SOO:10.0.1.101:0 ENCAP:8
          Router MAC:0200.0a00.0165
      Originator: 10.0.0.4 Cluster list: 10.0.0.2 

  Advertised path-id 1
  Path type: internal, path is valid, is best path, no labeled nexthop
             Imported to 3 destination(s)
             Imported paths list: OVERLAY-TENANT1 L3-100999 L2-100010
  AS-Path: NONE, path sourced internal to AS
    10.0.1.101 (metric 81) from 10.0.0.1 (10.0.0.1)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 100010 100999
      Extcommunity: RT:64500:100010 RT:64500:100999 SOO:10.0.1.101:0 ENCAP:8
          Router MAC:0200.0a00.0165
      Originator: 10.0.0.4 Cluster list: 10.0.0.1 

  Path-id 1 not advertised to any peer

Route Distinguisher: 10.0.0.7:32777    (L2VNI 100010)
BGP routing table entry for [2]:[0]:[0]:[48]:[a2b9.4605.948d]:[32]:[10.1.1.1]/272, version 434
Paths: (2 available, best #2)
Flags: (0x000212) (high32 00000000) on xmit-list, is in l2rib/evpn, is not in HW

  Path type: internal, path is valid, not best reason: Router Id, no labeled nexthop
             Imported from 10.0.0.4:32777:[2]:[0]:[0]:[48]:[a2b9.4605.948d]:[32]:[10.1.1.1]/272 
  AS-Path: NONE, path sourced internal to AS
    10.0.1.101 (metric 81) from 10.0.0.1 (10.0.0.1)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 100010 100999
      Extcommunity: RT:64500:100010 RT:64500:100999 SOO:10.0.1.101:0 ENCAP:8
          Router MAC:0200.0a00.0165
      Originator: 10.0.0.4 Cluster list: 10.0.0.1 

  Advertised path-id 1
  Path type: internal, path is valid, is best path, no labeled nexthop, in rib
             Imported from 10.0.0.3:32777:[2]:[0]:[0]:[48]:[a2b9.4605.948d]:[32]:[10.1.1.1]/272 
  AS-Path: NONE, path sourced internal to AS
    10.0.1.101 (metric 81) from 10.0.0.1 (10.0.0.1)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 100010 100999
      Extcommunity: RT:64500:100010 RT:64500:100999 SOO:10.0.1.101:0 ENCAP:8
          Router MAC:0200.0a00.0165
      Originator: 10.0.0.3 Cluster list: 10.0.0.1 

  Path-id 1 not advertised to any peer

Route Distinguisher: 10.0.0.7:4    (L3VNI 100999)
BGP routing table entry for [2]:[0]:[0]:[48]:[a2b9.4605.948d]:[32]:[10.1.1.1]/272, version 435
Paths: (2 available, best #2)
Flags: (0x000202) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW

  Path type: internal, path is valid, not best reason: Router Id, no labeled nexthop
             Imported from 10.0.0.4:32777:[2]:[0]:[0]:[48]:[a2b9.4605.948d]:[32]:[10.1.1.1]/272 
  AS-Path: NONE, path sourced internal to AS
    10.0.1.101 (metric 81) from 10.0.0.1 (10.0.0.1)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 100010 100999
      Extcommunity: RT:64500:100010 RT:64500:100999 SOO:10.0.1.101:0 ENCAP:8
          Router MAC:0200.0a00.0165
      Originator: 10.0.0.4 Cluster list: 10.0.0.1 

  Advertised path-id 1
  Path type: internal, path is valid, is best path, no labeled nexthop
             Imported from 10.0.0.3:32777:[2]:[0]:[0]:[48]:[a2b9.4605.948d]:[32]:[10.1.1.1]/272 
  AS-Path: NONE, path sourced internal to AS
    10.0.1.101 (metric 81) from 10.0.0.1 (10.0.0.1)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 100010 100999
      Extcommunity: RT:64500:100010 RT:64500:100999 SOO:10.0.1.101:0 ENCAP:8
          Router MAC:0200.0a00.0165
      Originator: 10.0.0.3 Cluster list: 10.0.0.1 

  Path-id 1 not advertised to any peer

First off, the routes are grouped into Route Distinguishers. This does exactly what it says, its allows the routes to be distinguished. I am using rd auto in this topology so it picks the RD for me.

There is method to the madness behind the auto RD generation. The first part of the RD (before the : ) is the BGP router ID of the device it corresponds to. The other section (after the : ) is 32767 + the local VlanID. That is the VlanID on the switch that it originated from. So, take the first RD 10.0.0.3:32777, this came from a switch with a router-id of 10.0.0.3 with a local VlanID of 10 (32777-32767). This is common for all L2VNIs. We can prove this is being done by looking at the evi on that switch:

LEAF-1# show bgp evi 100010 | inc ID|RD
  L2VNI ID                     : 100010 (L2-100010)
  RD                           : 10.0.0.3:32777

So, 10.1.1.1 is connected to the switches 10.0.0.3 and 10.0.0.4. Those are the raw routes coming from those switches via the Route Reflectors.

The route itself is [2]:[0]:[0]:[48]:[a2b9.4605.948d]:[32]:[10.1.1.1]/272

  • [2] - Route is type-2
  • [48] - MAC Address to follow is 48 bytes in length
  • [a2b9.4605.948d] - MAC Address of the endpoint
  • [32] - IP Address to follow is a /32
  • [10.1.1.1] - IP Address of the endpoint
  • /272 - Length of the entry

Thats what a breakdown of the routing entry looks like.

Looking under 10.0.0.3:32777 and 10.0.0.4:32777, we see two routes, one from each RR, we can tell that by looking at the Cluster list on the routes. This is a marker that the Route Reflector adds to prevent it learning one of its own advertised routes. One of the routes under each is marked as best, we can see that the best route says its Imported to 3 destination(s). What happens when a route comes in, is the route-target (BGP extended community) is checked and depending on the RT, the route gets imported into various VRFs etc. Again here, I have used auto on the RT too, therefore it generates this as asn:vni, which in turn generates 64500:100010.

Checking the leaf we got the BGP output from, we can see that this RT is what its been set to import:

LEAF-5# show bgp evi 100010 | section RT
  Active Export RT list        : 
        64500:100010 
  Active Import RT list        : 
        64500:100010

Therefore, in the main BGP output we see Route Distinguisher: 10.0.0.7:32777. This is effectively the local routing definition for the route. Notice the routes in there say Imported from 10.0.0.X:32777. These are the best routes from the RDs of the switches this NLRI originated from. Again, only one of these can be the best path, we can see that is the path from 10.0.0.3. Which is the top route here:

Route Distinguisher: 10.0.0.7:32777    (L2VNI 100010)
BGP routing table entry for [2]:[0]:[0]:[48]:[a2b9.4605.948d]:[32]:[10.1.1.1]/272, version 434
Paths: (2 available, best #2)
Flags: (0x000212) (high32 00000000) on xmit-list, is in l2rib/evpn, is not in HW

  Path type: internal, path is valid, not best reason: Router Id, no labeled nexthop
             Imported from 10.0.0.4:32777:[2]:[0]:[0]:[48]:[a2b9.4605.948d]:[32]:[10.1.1.1]/272 
  AS-Path: NONE, path sourced internal to AS
    10.0.1.101 (metric 81) from 10.0.0.1 (10.0.0.1)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 100010 100999
      Extcommunity: RT:64500:100010 RT:64500:100999 SOO:10.0.1.101:0 ENCAP:8
          Router MAC:0200.0a00.0165
      Originator: 10.0.0.4 Cluster list: 10.0.0.1 

  Advertised path-id 1
  Path type: internal, path is valid, is best path, no labeled nexthop, in rib
             Imported from 10.0.0.3:32777:[2]:[0]:[0]:[48]:[a2b9.4605.948d]:[32]:[10.1.1.1]/272 
  AS-Path: NONE, path sourced internal to AS
    10.0.1.101 (metric 81) from 10.0.0.1 (10.0.0.1)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 100010 100999
      Extcommunity: RT:64500:100010 RT:64500:100999 SOO:10.0.1.101:0 ENCAP:8
          Router MAC:0200.0a00.0165
      Originator: 10.0.0.3 Cluster list: 10.0.0.1 

  Path-id 1 not advertised to any peer

That route has all the same attributes as it has just been imported. There is a lot of additional information attached to the route, mainly extended communities like RTs and SOO which is specific to a vPC implementation and is used to prevent a vPC peer learning routes from its peer through the fabric. The RMAC is the mac address of the NVE interface on the remote VTEP. Any traffic sent to the endpoint, will be destined for that MAC address so the receiving VTEP knows what to do with the traffic.

We do also see 100999 and 64500:100999 on this route too. That is the ID for the L3VNI in my environment, which is used for routing between L2VNIs. That has its own RD and we can get that using the same command:

LEAF-5# show bgp evi 100999 | inc ID|RD|VRF
  L3VNI ID                     : 100999 (L3-100999)
  RD                           : 10.0.0.7:4
  Associated IP-VRF            : OVERLAY-TENANT1

The 4 is the VRF ID which is local to the device. The route that came from the originating leaves has the RT 64500:100999 already attached to it, so this will be automatically imported into the L3VNI RD on this leaf too:

Route Distinguisher: 10.0.0.7:4    (L3VNI 100999)
BGP routing table entry for [2]:[0]:[0]:[48]:[a2b9.4605.948d]:[32]:[10.1.1.1]/272, version 435
Paths: (2 available, best #2)
Flags: (0x000202) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW

  Path type: internal, path is valid, not best reason: Router Id, no labeled nexthop
             Imported from 10.0.0.4:32777:[2]:[0]:[0]:[48]:[a2b9.4605.948d]:[32]:[10.1.1.1]/272 
  AS-Path: NONE, path sourced internal to AS
    10.0.1.101 (metric 81) from 10.0.0.1 (10.0.0.1)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 100010 100999
      Extcommunity: RT:64500:100010 RT:64500:100999 SOO:10.0.1.101:0 ENCAP:8
          Router MAC:0200.0a00.0165
      Originator: 10.0.0.4 Cluster list: 10.0.0.1 

  Advertised path-id 1
  Path type: internal, path is valid, is best path, no labeled nexthop
             Imported from 10.0.0.3:32777:[2]:[0]:[0]:[48]:[a2b9.4605.948d]:[32]:[10.1.1.1]/272 
  AS-Path: NONE, path sourced internal to AS
    10.0.1.101 (metric 81) from 10.0.0.1 (10.0.0.1)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 100010 100999
      Extcommunity: RT:64500:100010 RT:64500:100999 SOO:10.0.1.101:0 ENCAP:8
          Router MAC:0200.0a00.0165
      Originator: 10.0.0.3 Cluster list: 10.0.0.1 

  Path-id 1 not advertised to any peer

Again we see both routes here in the L3VNI are Imported from 10.0.0.X:32777. The reason they were imported? RT:64500:100999 in the Extcommunity section.

And because its in there, means its made it into the VRFs IP routing table as a VXLAN route:

LEAF-5# show ip route 10.1.1.1 vrf OVERLAY-TENANT1
IP Route Table for VRF "OVERLAY-TENANT1"

10.1.1.1/32, ubest/mbest: 1/0
    *via 10.0.1.101%default, [200/0], 00:36:34, bgp-64500, internal, tag 64500, segid: 100999 tunnelid: 0xa000165 encap: VXLAN

And with that, you should hopefully have a better understanding of type-2 routes and how to read the output from them and work out why there is so much of it. Lots of it is duplicated, but the most important sections are the ones with local RDs.

Categories: CiscoVXLAN

0 Comments

Leave a Reply

Avatar placeholder

Your email address will not be published. Required fields are marked *

Related Posts

Cisco

Using PIM BiDir Phantom RP in a VXLAN Fabric

In this post, I am going to show the process of using PIM BiDir Phantom RP in the VXLAN EVPN topology we build in Parts 1 to 8 here. PIM is an important part of Read more…

Cisco

VXLAN EVPN Leaf and Spine – YouTube Video

Here is a YouTube video I created showing the process of setting up the same Fabric as the guide I created on here: This video goes through all the parts I explained on here but Read more…

Cisco

VXLAN EVPN Leaf and Spine – Part 8 – External Access outside Fabric

In the last part, we got route leaking in tenant VRFs working, in this part, we will look at giving the VRFs a way out of the Fabric! We have added some new devices to Read more…