VXLAN EVPN Leaf and Spine – Part 8 – External Access outside Fabric

Published by Nick Carlton on

In the last part, we got route leaking in tenant VRFs working, in this part, we will look at giving the VRFs a way out of the Fabric!

We have added some new devices to the topology:

We can see the addition of the external router and server that are connected to the border leaf. We have to connect these devices to the border leaf because it has the default route pointing towards it for simplicity.

Our external-server has the IP address 10.99.99.1 and needs to be accessed by both vlan 10 and 20.

In order to achieve both tenants being able to access the shared resources, we will need another VRF for the shared routes. If you were breaking out local Tenant VRFs, then you would use that Tenant's VRF.

VRF Configuration

So, we only need to create the VRF and leak the routes on leaf-3:

vrf context external
  vni 100555
  rd auto
  address-family ipv4 unicast
    route-target both auto
    route-target both auto evpn
    route-target import 64500:100998
    route-target import 64500:100998 evpn
    route-target import 64500:100999
    route-target import 64500:100999 evpn

Before we continue, lets quickly go over what this config should do. It creates a new VRF called external and assigns a VNI (required later). Within the address family, we auto assign a RT, which will be 64500:100555 from the VNI we set. We then import the routes from the other Tenant VRFs so that we have routes back, note that we also have the EVPN keyword so that we bring the EVPN /32 routes in too, reverse routing will fail without the specific routes.

We also need to amend the VRF configuration for the existing ones on leaf-3:

vrf context OVERLAY-TENANT1
  address-family ipv4 unicast
    route-target import 64500:100555 

vrf context OVERLAY-TENANT2
  address-family ipv4 unicast
    route-target import 64500:100555

This config will import routes from the external VRF.

Lets do some quick verification:

leaf-3# show ip route vrf external 
IP Route Table for VRF external

0.0.0.0/0, ubest/mbest: 1/0
    *via vrf OVERLAY-TENANT2, Null0, [20/0], 00:11:10, bgp-64500, external, tag 64500
10.10.1.0/24, ubest/mbest: 1/0, attached
    *via 10.10.1.254%OVERLAY-TENANT1, Vlan10, [20/0], 00:11:10, bgp-64500, external, tag 64500
10.20.1.0/24, ubest/mbest: 1/0, attached
    *via 10.20.1.254%OVERLAY-TENANT2, Vlan20, [20/0], 00:11:10, bgp-64500, external, tag 64500
leaf-3# show ip route vrf OVERLAY-TENANT1
IP Route Table for VRF OVERLAY-TENANT1

0.0.0.0/0, ubest/mbest: 1/0
    *via Null0, [1/0], 00:21:30, static
10.10.1.0/24, ubest/mbest: 1/0, attached
    *via 10.10.1.254, Vlan10, [0/0], 00:21:29, direct
10.10.1.1/32, ubest/mbest: 1/0
    *via 10.0.0.3%default, [200/0], 00:06:27, bgp-64500, internal, tag 64500, segid: 100999 tunnelid: 0xa000003 encap: VXLAN

10.10.1.3/32, ubest/mbest: 1/0, attached
    *via 10.10.1.3, Vlan10, [190/0], 00:18:00, hmm
10.10.1.254/32, ubest/mbest: 1/0, attached
    *via 10.10.1.254, Vlan10, [0/0], 00:21:29, local
10.20.1.0/24, ubest/mbest: 1/0, attached
    *via 10.20.1.254%OVERLAY-TENANT2, Vlan20, [20/0], 00:19:56, bgp-64500, external, tag 64500
leaf-3# show ip route vrf OVERLAY-TENANT2
IP Route Table for VRF OVERLAY-TENANT2

0.0.0.0/0, ubest/mbest: 1/0
    *via Null0, [1/0], 00:21:50, static
10.10.1.0/24, ubest/mbest: 1/0, attached
    *via 10.10.1.254%OVERLAY-TENANT1, Vlan10, [20/0], 00:20:16, bgp-64500, external, tag 64500
10.10.1.1/32, ubest/mbest: 1/0
    *via 10.0.0.3%default, [200/0], 00:06:14, bgp-64500, internal, tag 64500, segid: 100999 (Asymmetric) tunnelid: 0xa000003 encap: VXLAN

10.20.1.0/24, ubest/mbest: 1/0, attached
    *via 10.20.1.254, Vlan20, [0/0], 00:21:49, direct
10.20.1.3/32, ubest/mbest: 1/0, attached
    *via 10.20.1.3, Vlan20, [190/0], 00:18:15, hmm
10.20.1.254/32, ubest/mbest: 1/0, attached
    *via 10.20.1.254, Vlan20, [0/0], 00:21:49, local

We can see that the external VRF contains the /24 routes from each Tenant VRF, however, it doesn't contain any of the /32 VXLAN routes. This is because we haven't completed the L3VNI setup for it yet.

L3VNI Setup

Now we have completed the VRF config, lets complete the L3VNI config:

vlan 555
  vn-segment 100555

interface Vlan555
  no shutdown
  vrf member external
  ip forward

interface nve1
  member vni 100555 associate-vrf

Now, we should see the VXLAN routes in the external VRF:

leaf-3# show ip route vrf external 
IP Route Table for VRF external

0.0.0.0/0, ubest/mbest: 1/0
    *via vrf OVERLAY-TENANT2, Null0, [20/0], 00:18:05, bgp-64500, external, tag 64500
10.10.1.0/24, ubest/mbest: 1/0, attached
    *via 10.10.1.254%OVERLAY-TENANT1, Vlan10, [20/0], 00:18:05, bgp-64500, external, tag 64500
<strong>10.10.1.1/32, ubest/mbest: 1/0
    *via 10.0.0.3%default, [200/0], 00:01:55, bgp-64500, internal, tag 64500, segid: 100999 (Asymmetric) tunnelid: 0xa000003 encap: VXLAN</strong>

10.20.1.0/24, ubest/mbest: 1/0, attached
    *via 10.20.1.254%OVERLAY-TENANT2, Vlan20, [20/0], 00:18:05, bgp-64500, external, tag 64500
<strong>10.20.1.1/32, ubest/mbest: 1/0
    *via 10.0.0.3%default, [200/0], 00:00:07, bgp-64500, internal, tag 64500, segid: 100998 (Asymmetric) tunnelid: 0xa000003 encap: VXLAN</strong>

<strong>10.20.1.2/32, ubest/mbest: 1/0
    *via 10.0.0.4%default, [200/0], 00:00:07, bgp-64500, internal, tag 64500, segid: 100998 (Asymmetric) tunnelid: 0xa000004 encap: VXLAN</strong>

Interface Config

Now the VRF is configured we need to configure the Eth1/5 on leaf-3 IP addressing to our external router:

interface Ethernet1/5
  no switchport
  vrf member external
  ip address 10.1.50.1/30
  no shutdown

We also need to configure the external router interface:

interface GigabitEthernet0/0
 ip address 10.1.50.2 255.255.255.252
 no shutdown
end

No VRF is required on the external router, only exiting the fabric on the Nexus side.

BGP Peerings

Now the interfaces are configured, we need to add a BGP peering on leaf-3:

router bgp 64500
  vrf external
  neighbor 10.1.50.2 remote-as 64520
    address-family ipv4 unicast 
  exit
exit

We also need to configure a peering on the external router to match:

router bgp 64520
 no bgp default ipv4-unicast
 neighbor 10.1.50.1 remote-as 64500
 address-family ipv4
  neighbor 10.1.50.1 activate
 exit-address-family

We should see the neighborships come up. However, we will realise that the default route presented from the border leaf is being advertised to the external router, this isnt great:

external#show bgp ipv4 unicast | beg Network
     Network          Next Hop            Metric LocPrf Weight Path
 <strong>*>   0.0.0.0          10.1.50.1                              0 64500 i</strong>
 *>   10.10.1.0/24     10.1.50.1                              0 64500 i
 *>   10.10.1.1/32     10.1.50.1                              0 64500 i
 *>   10.20.1.0/24     10.1.50.1                              0 64500 i
 *>   10.20.1.1/32     10.1.50.1                              0 64500 i
 *>   10.20.1.2/32     10.1.50.1                              0 64500 i

Lets filter this from being sent by leaf-3:

ip prefix-list default-route permit 0.0.0.0/0
route-map deny-default deny 10
 match ip address prefix-list default-route
route-map deny-default permit 20
exit

router bgp 64500
 vrf external
  neighbor 10.1.50.2
  address-family ipv4 unicast 
   route-map deny-default out

Now we have lost the default route:

external# show bgp ipv4 unicast | beg Network
     Network          Next Hop            Metric LocPrf Weight Path
 *>   10.10.1.0/24     10.1.50.1                              0 64500 i
 *>   10.10.1.1/32     10.1.50.1                              0 64500 i
 *>   10.20.1.0/24     10.1.50.1                              0 64500 i
 *>   10.20.1.1/32     10.1.50.1                              0 64500 i
 *>   10.20.1.2/32     10.1.50.1                              0 64500 i

You may want to keep the default route, but in lots of cases, external access would be providing internet services so it doesn't make sense for the external network to be learning a default route towards the fabric. The external router still gets the specific routes per tenant. Looking at the above, we see both the /24 networks and the /32 host routes. We don't need to clog up the external routers IP routing table with all these networks, so we can summarise them on leaf-3:

router bgp 64500
  vrf external
    address-family ipv4 unicast
      aggregate-address 10.10.1.0/24 summary-only
      aggregate-address 10.20.1.0/24 summary-only

After doing that, we can see only the /24 routes appear for each VRF:

external# show bgp ipv4 unicast | beg Network
     Network          Next Hop            Metric LocPrf Weight Path
 *>   10.10.1.0/24     10.1.50.1                              0 64500 i
 *>   10.20.1.0/24     10.1.50.1                              0 64500 i

Whereas before we had to have the specific /32 routes for the return traffic to work, this is still happening on the border node as the VRF still contains all the /32 routes. However, the external router doesn't need to know about all of them specifically as they would all have the same next-hop anyway.

Route Advertisement

Now we need to advertise the routes for the shared external network back into the Fabric, we can do this one of two ways:

  • Redistribute connected networks
  • Network Command

In this case, we also have a loopback we want to advertise, so will use redistribution on the external router:

router bgp 64520
 address-family ipv4
  redistribute connected
 exit-address-family

Verification

We are all done with our configuration now, lets check the VRF routing tables on leaf-3 for the external routes:

leaf-3# show ip route vrf OVERLAY-TENANT1
IP Route Table for VRF OVERLAY-TENANT1

0.0.0.0/0, ubest/mbest: 1/0
    *via Null0, [1/0], 00:44:13, static
<strong>10.1.50.0/30, ubest/mbest: 1/0
    *via 10.1.50.2%external, [20/0], 00:02:41, bgp-64500, external, tag 64520</strong>
10.10.1.0/24, ubest/mbest: 1/0, attached
    *via 10.10.1.254, Vlan10, [0/0], 00:44:12, direct
10.10.1.1/32, ubest/mbest: 1/0
    *via 10.0.0.3%default, [200/0], 00:29:10, bgp-64500, internal, tag 64500, segid: 100999 tunnelid: 0xa000003 encap: VXLAN

10.10.1.3/32, ubest/mbest: 1/0, attached
    *via 10.10.1.3, Vlan10, [190/0], 00:40:43, hmm
10.10.1.254/32, ubest/mbest: 1/0, attached
    *via 10.10.1.254, Vlan10, [0/0], 00:44:12, local
10.20.1.0/24, ubest/mbest: 1/0, attached
    *via 10.20.1.254%OVERLAY-TENANT2, Vlan20, [20/0], 00:42:39, bgp-64500, external, tag 64500
10.20.1.1/32, ubest/mbest: 1/0
    *via 10.0.0.3%default, [200/0], 00:16:24, bgp-64500, internal, tag 64500, segid: 100998 (Asymmetric) tunnelid: 0xa000003 encap: VXLAN

10.20.1.2/32, ubest/mbest: 1/0
    *via 10.0.0.4%default, [200/0], 00:16:24, bgp-64500, internal, tag 64500, segid: 100998 (Asymmetric) tunnelid: 0xa000004 encap: VXLAN

<strong>10.99.99.0/24, ubest/mbest: 1/0
    *via 10.1.50.2%external, [20/0], 00:02:41, bgp-64500, external, tag 64520</strong>
<strong>10.100.100.1/32, ubest/mbest: 1/0
    *via 10.1.50.2%external, [20/0], 00:02:41, bgp-64500, external, tag 64520</strong>

The same information is present in the OVERLAY-TENANT2 vrf also.

We should still only see default routes on the other leaf switches:

leaf-2# show ip route vrf OVERLAY-TENANT1
IP Route Table for VRF OVERLAY-TENANT1

0.0.0.0/0, ubest/mbest: 1/0
    *via 10.0.0.5%default, [200/0], 00:44:03, bgp-64500, internal, tag 64500, segid: 100999 tunnelid: 0xa000005 encap: VXLAN

10.10.1.0/24, ubest/mbest: 1/0, attached
    *via 10.10.1.254, Vlan10, [0/0], 00:46:17, direct
10.10.1.1/32, ubest/mbest: 1/0
    *via 10.0.0.3%default, [200/0], 00:42:23, bgp-64500, internal, tag 64500, segid: 100999 tunnelid: 0xa000003 encap: VXLAN

10.10.1.3/32, ubest/mbest: 1/0
    *via 10.0.0.5%default, [200/0], 00:42:23, bgp-64500, internal, tag 64500, segid: 100999 tunnelid: 0xa000005 encap: VXLAN

10.10.1.254/32, ubest/mbest: 1/0, attached
    *via 10.10.1.254, Vlan10, [0/0], 00:46:17, local
leaf-2# show ip route vrf OVERLAY-TENANT2
IP Route Table for VRF OVERLAY-TENANT2

0.0.0.0/0, ubest/mbest: 1/0
    *via 10.0.0.5%default, [200/0], 00:44:05, bgp-64500, internal, tag 64500, segid: 100998 tunnelid: 0xa000005 encap: VXLAN

10.20.1.0/24, ubest/mbest: 1/0, attached
    *via 10.20.1.254, Vlan20, [0/0], 00:46:18, direct
10.20.1.1/32, ubest/mbest: 1/0
    *via 10.0.0.3%default, [200/0], 00:18:06, bgp-64500, internal, tag 64500, segid: 100998 tunnelid: 0xa000003 encap: VXLAN

10.20.1.2/32, ubest/mbest: 1/0, attached
    *via 10.20.1.2, Vlan20, [190/0], 00:18:06, hmm
10.20.1.3/32, ubest/mbest: 1/0
    *via 10.0.0.5%default, [200/0], 00:42:21, bgp-64500, internal, tag 64500, segid: 100998 tunnelid: 0xa000005 encap: VXLAN

10.20.1.254/32, ubest/mbest: 1/0, attached
    *via 10.20.1.254, Vlan20, [0/0], 00:46:18, local

Note that there is no trace of 10.99.99.0/24 or 10.100.100.1/32 on the other leaves, they will instead forward the traffic over to leaf-3 which is our border node.

Finally, lets test connectivity between the PCs in the fabric and the external-server:

server-0-vl10:~$ ping 10.99.99.1
PING 10.99.99.1 (10.99.99.1): 56 data bytes
64 bytes from 10.99.99.1: seq=0 ttl=42 time=12.170 ms
64 bytes from 10.99.99.1: seq=1 ttl=42 time=9.152 ms
64 bytes from 10.99.99.1: seq=2 ttl=42 time=13.861 ms
64 bytes from 10.99.99.1: seq=3 ttl=42 time=13.179 ms
64 bytes from 10.99.99.1: seq=4 ttl=42 time=9.046 ms
^C
--- 10.99.99.1 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 9.046/11.481/13.861 ms
server-0-vl10:~$ traceroute 10.99.99.1
traceroute to 10.99.99.1 (10.99.99.1), 30 hops max, 46 byte packets
 1  10.10.1.254 (10.10.1.254)  1.641 ms  2.257 ms  1.135 ms
 2  10.10.1.254 (10.10.1.254)  6.348 ms  7.729 ms  5.839 ms
 3  10.1.50.2 (10.1.50.2)  8.250 ms  9.711 ms  8.163 ms
 4  10.99.99.1 (10.99.99.1)  8.496 ms  9.624 ms  8.388 ms
server-3-vl20:~$ ping 10.99.99.1
PING 10.99.99.1 (10.99.99.1): 56 data bytes
64 bytes from 10.99.99.1: seq=0 ttl=42 time=29.208 ms
64 bytes from 10.99.99.1: seq=1 ttl=42 time=15.948 ms
64 bytes from 10.99.99.1: seq=2 ttl=42 time=13.777 ms
64 bytes from 10.99.99.1: seq=3 ttl=42 time=9.348 ms
64 bytes from 10.99.99.1: seq=4 ttl=42 time=10.978 ms
^C
--- 10.99.99.1 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 9.348/15.851/29.208 ms
server-3-vl20:~$ traceroute 10.99.99.1
traceroute to 10.99.99.1 (10.99.99.1), 30 hops max, 46 byte packets
 1  10.20.1.254 (10.20.1.254)  2.366 ms  1.591 ms  1.364 ms
 2  10.20.1.254 (10.20.1.254)  11.394 ms  8.706 ms  5.643 ms
 3  10.1.50.2 (10.1.50.2)  12.406 ms  9.476 ms  12.254 ms
 4  10.99.99.1 (10.99.99.1)  8.691 ms  16.457 ms  11.293 ms

This proves that both vlan 10 and 20 are able to communicate with the external networks. Lets also double check they can still communicate between each other as we were changing the VRF leaking configuration:

server-0-vl10:~$ ping 10.20.1.3
PING 10.20.1.3 (10.20.1.3): 56 data bytes
64 bytes from 10.20.1.3: seq=0 ttl=42 time=9.277 ms
64 bytes from 10.20.1.3: seq=1 ttl=42 time=8.746 ms
64 bytes from 10.20.1.3: seq=2 ttl=42 time=13.971 ms
64 bytes from 10.20.1.3: seq=3 ttl=42 time=9.133 ms
64 bytes from 10.20.1.3: seq=4 ttl=42 time=8.624 ms
^C
--- 10.20.1.3 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 8.624/9.950/13.971 ms

Perfect! Full connectivity.

That is us done with the configuration guides for this VXLAN topology! I hope you found it helpful.

The full configuration files for the topology can be downloaded here: https://github.com/avidpontoon/VXLAN-EVPN-Leaf-and-Spine

Categories: CiscoExpert TutorialVXLAN

2 Comments

Ilya · 8th October 2023 at 6:59 pm

Hi Nick, can you show all configuration on leaf and spine?

Can you create Part 9 – Implementing Firewall to your lab.
Fabric – Border Leaf – Firewall – Border Leaf – ISP. For example. Single Firewall or Act/Stby.

This theme very good for many IT ingeenera, because to low info in Internet about it.

Reply

    Nick Carlton · 17th November 2023 at 9:06 pm

    Hi Ilya, sure, I will dig these out and put them into a GitHub repo and publish them on the site 🙂

    With the firewall part, are you meaning a firewall that connects directly to the internet and then allows you to break out of the Fabric to the internet?

    It would normally go: Fabric – Border Leaf(s) – Firewall(s) – ISP/Internet

    Thanks
    Nick

    Reply

Leave a Reply

Avatar placeholder

Your email address will not be published. Required fields are marked *

Related Posts

Cisco

VXLAN EVPN Leaf and Spine – YouTube Video

Here is a YouTube video I created showing the process of setting up the same Fabric as the guide I created on here: This video goes through all the parts I explained on here but Read more…

Cisco

VXLAN EVPN Multi-Site – Part 9 – External Access outside Fabric

In the last part, we got route leaking in tenant VRFs working, in this part, we will look at giving the VRFs a way out of the Fabric! We have added some new devices to Read more…

Cisco

VXLAN EVPN Multi-Site – Part 8 – L3VNI Configuration – Multi Tenant Leaking

In the last part, we got a multiple tenant VRFs working, in this part, we will look at joining the tenants up via route leaking. Lets remind ourselves of the topology: So, currently we have Read more…