Traditional data centers an campus networks have used layer 2 links to span racks, rooms floors and buildings. This would create large L2 domains that are not ideal due to unnecessarily large broadcast domains, slow convergence and reliance on spanning tree. The flexibility of these large L2 domains is also it’s challenge, VXLAN reduces the size of the L2 domain yet provides the flexibility of stretching L2 from rack to rack, row data center or building location.
VXLAN tunnels a L2 frame or L3 packet and routes it to another VXLAN Tunnel Endpoint (VTEP), functionally extending one L2 domain across a L3 networks. VXLAN tunnels are traditionally created between leaf switches using the multicast control plane. This is not a big deal in small datacenters but is not scalable when expanding. By using the Ethernet VPN (EVPN) control plane and the Multi-Protocol BGP (MP-BGP) address family we can address the scalability issue with every node (switch) in the fabric becomes a part of the EVPN overlay.
EVPN can be especially helpful in designs where a hierarchy is created utilizing spine-leaf pods interconnected by super-spines creating a modular fabric, or in the case of stretching L2 over a routed WAN link.
Functionally, EVPN is a vitrual, logical overlay network running on top of a physical underlay network that need not be aware of the overlay. However, the overlay fabric is dependant upon the the underlay.
In the context of VXLAN, EVPN is used as a Network Virtualization Overlay (NVO). To share Network Layer Reachability Informaiton (NLRI) between VTEPs. This can be either L2 VLANs for MAC addresses (MAC-VRF) or L3 IP prefix information (IP-VRF). Within the overlay network we can utilize VRF instances for multi-tenancy providing security and traffic isolation.
Type-2 MAC-VRF routes utilize a VXLAN Network Identifier (VNI) that is mapped to a VLAN to advertise an end host’s MAC and IP address. Any leaf confiured with the VNI will be able to share end-host MAC addresses to provide layer 2 reachability. As a switch learns locally attached MAC addresses they are then advertised to the EVPN enabling other leaf VTEPs to install the MAC in their CAM.
IP Prefixes are advertised as Type-5 routes. A VNI is mapped to a Virtual Routing & Forwarding (VRF) context that identifies the customer/tenant/segment within the fabric. This uniqely allows route tables to coexist in each VRF. This allows end-to-end segmentation in the overlay with the underlay being unaware of the segmetation itself. For example, only the leaf switches need to possess the VRFs that the endpoints are attached to, the spine switches simply provide the underlay transit betweent he leafs.
Functionally, this means that a VLAN can exist on multiple swithes that do not have layer 2 connectivity betwene them. As and example, in our case, a production can have the same IP network in multiple building locations.
*Spine*: A layer 3 switch that is unaware of any VXLAN overlay networks. This can be thought of as a Core or physical aggrregation layer to which leafs connect. *Super-Spine*: A layer 3 switch that is unaware of VXLAN overlay networks. This can be thought of as a higher level core that aggregates spine layer switches. *Leaf*: A layer 3 switch that participates in the underlay and overlay networks. This is where VTEPs exist. This can function as an access layer switch providing end-point connectivity as well as a L2 distribution switch aggregating L2 connectivity to downstream access switches.
The underlay provides basic connectivity and routing between the physical network devices and requires IP addressing and routing to provide this. As outlined in the diagram there are layer 3 point to point links between the leafs and spines. For our lab purposes we are foxusing on the leaf configureation, the spines are already configured.
Leaf01 ! interface Ethernet1 description UPLINK spine1:Et1 logging event link-status no switchport ip address 10.245.2.49/31 ! interface Ethernet2 description UPLINK spine2:Et1 logging event link-status no switchport ip address 10.245.3.49/31
Leaf02 ! interface Ethernet1 description UPLINK spine1:Et2 logging event link-status no switchport ip address 10.245.2.51/31 ! interface Ethernet2 description UPLINK spine2:Et2 logging event link-status no switchport ip address 10.245.3.51/31
Leaf03 ! interface Ethernet1 description UPLINK spine1:Et3 logging event link-status no switchport ip address 10.245.2.57/31 ! interface Ethernet2 description UPLINK spine2:Et3 logging event link-status no switchport ip address 10.245.3.57/31
Leaf01 ! interface Loopback0 description Underlay eBGP ip address 10.245.0.25/32
Leaf02 ! interface Loopback0 description Underlay eBGP ip address 10.245.0.26/32
Leaf03 ! interface Loopback0 description Underlay eBGP ip address 10.245.0.29/32
At this point you should be able to ping across each of the links to the directly connected neighbor
We will assign a BGP process for each pair, spines and leafs. these will be eBGP relationships with each leaf (or leaf pair) or spine (spine pair) having it’s own AS number. The leafs will later use this for the overlay as well. The basic is similar on all nodes whether leaf or spine. This is what will advertise the loopback addresses and make them reachable from other devices.
First there’s some basic, generic configuraiton.
Leaf01 ! router bgp 64535 router-id 10.245.0.25 no bgp default ipv4-unicast distance bgp 20 200 200 graceful-restart restart-time 300 maximum-paths 2 ecmp 2
Leaf02 ! router bgp 64536 router-id 10.245.0.26 no bgp default ipv4-unicast distance bgp 20 200 200 graceful-restart restart-time 300 maximum-paths 2 ecmp 2
Leaf03 ! router bgp 64539 router-id 10.245.0.29 no bgp default ipv4-unicast distance bgp 20 200 200 graceful-restart restart-time 300 maximum-paths 2 ecmp 2
On the leafs we use a peer-group called EBGP-UNDERLAY-IPV4 for repeat configuration parameters that will be applied to both spine adjecencies. On the spines we use a similary named peer group that the dynamically learned neighbors are added to. Each spine will peer with each leaf over the L3 p-t-p interface. By using the listen command on the spines we can dynamically create the BGP peering on that side. We set a range of addresses to listen for and assign them to a peer group which then allows us to assign similar configuration parameters to all of the dynamically learned neighbors as long as they match a filter of known leaf ASNs. Note for the spine configuration below there is no neighbor statement.
On the leafs we use a peer-group called EBGP-UNDERLAY-IPV4 for repeat configuration parameters that will be applied to both spine adjecencies. On the spines we use a similary named peer group that the dynamically learned neighbors are added to.
In addition to the basics we create peering across the point-to-pint links and activate the ipv4 address family. Most of the configuration is the same on all leafs, with the only differences being the BGP ASN, router-id and peer address to match with the point-to-point links.
Leaf01
!
router bgp 64535
router-id 10.245.0.25
no bgp default ipv4-unicast
distance bgp 20 200 200
graceful-restart restart-time 300
maximum-paths 2 ecmp 2
neighbor EBGP-UNDERLAY-IPV4 peer group
neighbor EBGP-UNDERLAY-IPV4 remote-as 64512
neighbor EBGP-UNDERLAY-IPV4 graceful-restart
neighbor EBGP-UNDERLAY-IPV4 password 7 YJzOEZJg+RZNENCoPvARl52sBjFfAn6Q
neighbor EBGP-UNDERLAY-IPV4 send-community
neighbor EBGP-UNDERLAY-IPV4 maximum-routes 100000 warning-only
neighbor 10.245.2.48 peer group EBGP-UNDERLAY-IPV4
neighbor 10.245.3.48 peer group EBGP-UNDERLAY-IPV4
redistribute connected route-map REDIST-CONN-IPV4
!
address-family ipv4
neighbor EBGP-UNDERLAY-IPV4 activate
Leaf02
!
router bgp 64536
router-id 10.245.0.26
no bgp default ipv4-unicast
distance bgp 20 200 200
graceful-restart restart-time 300
maximum-paths 2 ecmp 2
neighbor EBGP-UNDERLAY-IPV4 peer group
neighbor EBGP-UNDERLAY-IPV4 remote-as 64512
neighbor EBGP-UNDERLAY-IPV4 graceful-restart
neighbor EBGP-UNDERLAY-IPV4 password 7 YJzOEZJg+RZNENCoPvARl52sBjFfAn6Q
neighbor EBGP-UNDERLAY-IPV4 send-community
neighbor EBGP-UNDERLAY-IPV4 maximum-routes 100000 warning-only
neighbor 10.245.2.50 peer group EBGP-UNDERLAY-IPV4
neighbor 10.245.3.50 peer group EBGP-UNDERLAY-IPV4
redistribute connected route-map REDIST-CONN-IPV4
!
address-family ipv4
neighbor EBGP-UNDERLAY-IPV4 activate
Leaf03
!
router bgp 64539
router-id 10.245.0.29
no bgp default ipv4-unicast
distance bgp 20 200 200
graceful-restart restart-time 300
maximum-paths 2 ecmp 2
neighbor EBGP-UNDERLAY-IPV4 peer group
neighbor EBGP-UNDERLAY-IPV4 remote-as 64512
neighbor EBGP-UNDERLAY-IPV4 graceful-restart
neighbor EBGP-UNDERLAY-IPV4 password 7 YJzOEZJg+RZNENCoPvARl52sBjFfAn6Q
neighbor EBGP-UNDERLAY-IPV4 send-community
neighbor EBGP-UNDERLAY-IPV4 maximum-routes 100000 warning-only
neighbor 10.245.2.56 peer group EBGP-UNDERLAY-IPV4
neighbor 10.245.3.56 peer group EBGP-UNDERLAY-IPV4
redistribute connected route-map REDIST-CONN-IPV4
!
address-family ipv4
neighbor EBGP-UNDERLAY-IPV4 activate
ip prefix-list OVERLAY-LOOPBACKS-IPV4 seq 10 permit 10.245.1.0/24 eq 32 ip prefix-list UNDERLAY-LOOPBACKS-IPV4 seq 10 permit 10.245.0.0/24 eq 32 ip prefix-list UNDERLAY-P2P-IPV4 seq 10 permit 10.245.2.0/24 le 31 ip prefix-list UNDERLAY-P2P-IPV4 seq 20 permit 10.245.3.0/24 le 31 ! route-map REDIST-CONN-IPV4 permit 10 match ip address prefix-list OVERLAY-LOOPBACKS-IPV4 ! route-map REDIST-CONN-IPV4 permit 20 match ip address prefix-list UNDERLAY-LOOPBACKS-IPV4 ! route-map REDIST-CONN-IPV4 permit 30 match ip address prefix-list UNDERLAY-P2P-IPV4
- Each spine should have three BGP neighbors, one to each leaf. Each leaf should show each spine as a neighbor. For example:
spine1#sh bgp summ BGP summary information for VRF default Router identifier 10.245.0.5, local AS number 64512 Neighbor AS Session State AFI/SAFI AFI/SAFI State NLRI Rcd NLRI Acc ----------- ----------- ------------- ----------------------- -------------- ---------- ---------- 10.245.2.49 64535 Established IPv4 Unicast Negotiated 4 4 10.245.2.51 64536 Established IPv4 Unicast Negotiated 4 4 10.245.2.57 64539 Established IPv4 Unicast Negotiated 4 4 spine1#sh ip bgp summ BGP summary information for VRF default Router identifier 10.245.0.5, local AS number 64512 Neighbor Status Codes: m - Under maintenance Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc 10.245.2.49 4 64535 239 238 0 0 03:14:14 Estab 4 4 10.245.2.51 4 64536 95 92 0 0 01:11:06 Estab 4 4 10.245.2.57 4 64539 76 75 0 0 00:58:29 Estab 4 4 spine1#
leaf01#sh ip bgp summ BGP summary information for VRF default Router identifier 10.245.0.25, local AS number 64535 Neighbor Status Codes: m - Under maintenance Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc 10.245.2.48 4 64512 235 236 0 0 03:11:45 Estab 10 10 10.245.3.48 4 64512 236 239 0 0 03:11:45 Estab 10 10 leaf01# leaf01#sh bgp summ BGP summary information for VRF default Router identifier 10.245.0.25, local AS number 64535 Neighbor AS Session State AFI/SAFI AFI/SAFI State NLRI Rcd NLRI Acc ----------- ----------- ------------- ----------------------- -------------- ---------- ---------- 10.245.2.48 64512 Established IPv4 Unicast Negotiated 10 10 10.245.3.48 64512 Established IPv4 Unicast Negotiated 10 10 leaf01#
The command:
service routing protocols model multi-agent
Enables EVPN capability and will require a reboot of the switch to take effect. This would be a command to enter when first gettign started.
Some models of switch will require VXLAN routing to be enabled in the tcam profile:
hardware tcam profile vxlan-routing
Here we are creating a BGP peering between the Spine and Leaf
On each Leaf, configure a peer group with:
- Neighbor to the Loopback IP address of each Spine using the Loopback0 interface as the source
- Configure ebgp-multihop 3 to account for possibility of a Leaf needing to establish an EVPN BGP adjacency with a Spine through it’s peer link, this is in the case where there are two leafs with an MLAG between them
- The send-community extended command is required for attributes to be sent between EVPN peers
- Activate the evpn address-family for the appropriate peer-group
Leaf01
!
router bgp 64535
neighbor EVPN-OVERLAY-IPV4 peer group
neighbor EVPN-OVERLAY-IPV4 remote-as 64512
neighbor EVPN-OVERLAY-IPV4 update-source Loopback0
neighbor EVPN-OVERLAY-IPV4 ebgp-multihop 3
neighbor EVPN-OVERLAY-IPV4 graceful-restart
neighbor EVPN-OVERLAY-IPV4 password 7 5CCCJRiTkuVwWgNB+hJm51l2uVbPlPYj
neighbor EVPN-OVERLAY-IPV4 send-community
neighbor EVPN-OVERLAY-IPV4 maximum-routes 100000 warning-only
neighbor 10.245.0.5 peer group EVPN-OVERLAY-IPV4
neighbor 10.245.0.6 peer group EVPN-OVERLAY-IPV4
!
address-family evpn
neighbor EVPN-OVERLAY-IPV4 activate
Leaf02
!
router bgp 64536
neighbor EVPN-OVERLAY-IPV4 peer group
neighbor EVPN-OVERLAY-IPV4 remote-as 64512
neighbor EVPN-OVERLAY-IPV4 update-source Loopback0
neighbor EVPN-OVERLAY-IPV4 ebgp-multihop 3
neighbor EVPN-OVERLAY-IPV4 graceful-restart
neighbor EVPN-OVERLAY-IPV4 password 7 5CCCJRiTkuVwWgNB+hJm51l2uVbPlPYj
neighbor EVPN-OVERLAY-IPV4 send-community
neighbor EVPN-OVERLAY-IPV4 maximum-routes 100000 warning-only
neighbor 10.245.0.5 peer group EVPN-OVERLAY-IPV4
neighbor 10.245.0.6 peer group EVPN-OVERLAY-IPV4
!
address-family evpn
neighbor EVPN-OVERLAY-IPV4 activate
Leaf03
!
router bgp 64539
neighbor EVPN-OVERLAY-IPV4 peer group
neighbor EVPN-OVERLAY-IPV4 remote-as 64512
neighbor EVPN-OVERLAY-IPV4 update-source Loopback0
neighbor EVPN-OVERLAY-IPV4 ebgp-multihop 3
neighbor EVPN-OVERLAY-IPV4 graceful-restart
neighbor EVPN-OVERLAY-IPV4 password 7 5CCCJRiTkuVwWgNB+hJm51l2uVbPlPYj
neighbor EVPN-OVERLAY-IPV4 send-community
neighbor EVPN-OVERLAY-IPV4 maximum-routes 100000 warning-only
neighbor 10.245.0.5 peer group EVPN-OVERLAY-IPV4
neighbor 10.245.0.6 peer group EVPN-OVERLAY-IPV4
!
address-family evpn
neighbor EVPN-OVERLAY-IPV4 activate
At this point the EVP neighbors should be estabished between the Leafs and Spines, so we are now ready to transport VXLAN traffic.
From the perspective of the spines we should have EVPN BGP peering with ech of the leafs:
spine1#sh bgp summ BGP summary information for VRF default Router identifier 10.245.0.5, local AS number 64512 Neighbor AS Session State AFI/SAFI AFI/SAFI State NLRI Rcd NLRI Acc ----------- ----------- ------------- ----------------------- -------------- ---------- ---------- 10.245.0.25 64535 Established L2VPN EVPN Negotiated 1 1 10.245.0.26 64536 Established L2VPN EVPN Negotiated 1 1 10.245.0.29 64539 Established L2VPN EVPN Negotiated 1 1 10.245.2.49 64535 Established IPv4 Unicast Negotiated 4 4 10.245.2.51 64536 Established IPv4 Unicast Negotiated 4 4 10.245.2.57 64539 Established IPv4 Unicast Negotiated 4 4 spine1# spine1#sh bgp evpn summ BGP summary information for VRF default Router identifier 10.245.0.5, local AS number 64512 Neighbor Status Codes: m - Under maintenance Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc 10.245.0.25 4 64535 260 257 0 0 03:23:30 Estab 1 1 10.245.0.26 4 64536 106 105 0 0 01:20:19 Estab 1 1 10.245.0.29 4 64539 85 85 0 0 01:07:08 Estab 1 1 spine1#
From the perspective of the leaf, there should be peering with each spine:
leaf01#sh bgp summ BGP summary information for VRF default Router identifier 10.245.0.25, local AS number 64535 Neighbor AS Session State AFI/SAFI AFI/SAFI State NLRI Rcd NLRI Acc ----------- ----------- ------------- ----------------------- -------------- ---------- ---------- 10.245.0.5 64512 Established L2VPN EVPN Negotiated 2 2 10.245.0.6 64512 Established L2VPN EVPN Negotiated 2 2 10.245.2.48 64512 Established IPv4 Unicast Negotiated 10 10 10.245.3.48 64512 Established IPv4 Unicast Negotiated 10 10 leaf01#sh bgp evpn summ BGP summary information for VRF default Router identifier 10.245.0.25, local AS number 64535 Neighbor Status Codes: m - Under maintenance Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc 10.245.0.5 4 64512 259 263 0 0 03:25:27 Estab 2 2 10.245.0.6 4 64512 256 265 0 0 03:25:27 Estab 2 2 leaf01#
The VTEP is the tunnel interface that encapsulate/decapsulate and deliver L2 traffic over the L3 network between leafs in the EVPN fabric.
- Configure a loopback interfce and IP that will be shared among the VTEP leaf pairs
- Advertise the Loopback into BGP
- Configure the VTEP interface
Leaf01 ! interface Loopback1 description VTEP Source ip address 10.245.1.25/32
Leaf02 ! interface Loopback1 description VTEP Source ip address 10.245.1.26/32
Leaf03 ! interface Loopback1 description VTEP Source ip address 10.245.1.29/32
- This uses the prefix-list and route-map we saw earlier
ip prefix-list OVERLAY-LOOPBACKS-IPV4 seq 10 permit 10.245.1.0/24 eq 32 ip prefix-list UNDERLAY-LOOPBACKS-IPV4 seq 10 permit 10.245.0.0/24 eq 32 ip prefix-list UNDERLAY-P2P-IPV4 seq 10 permit 10.245.2.0/24 le 31 ip prefix-list UNDERLAY-P2P-IPV4 seq 20 permit 10.245.3.0/24 le 31 ! route-map REDIST-CONN-IPV4 permit 10 match ip address prefix-list OVERLAY-LOOPBACKS-IPV4 ! route-map REDIST-CONN-IPV4 permit 20 match ip address prefix-list UNDERLAY-LOOPBACKS-IPV4 ! route-map REDIST-CONN-IPV4 permit 30 match ip address prefix-list UNDERLAY-P2P-IPV4
- We use the just created Loopback 1 as the source and define the port. At this point we have not yet tied any traffic to use the tunnel.
interface Vxlan1 vxlan source-interface Loopback1 vxlan udp-port 4789
To pass traffic we need to map our L2 VLAN to a VXLAN Network Identifier (VNI) which is uniquely used on each leaf we want the same vlan to appear.
Leaf01
!
vlan 40
name test-l2-vlan
!
interface Vxlan1
vxlan vlan 40 vni 100040
!
router bgp 64535
vlan 40
rd 10.245.0.25:40
route-target both 40:40
redistribute learned
!
interface Ethernet4
switchport access vlan 40
Leaf02
!
vlan 40
name test-l2-vlan
!
interface Vxlan1
vxlan vlan 40 vni 100040
!
router bgp 64536
vlan 40
rd 10.245.0.26:40
route-target both 40:40
redistribute learned
!
interface Ethernet4
switchport access vlan 40
Leaf03
!
vlan 40
name test-l2-vlan
!
interface Vxlan1
vxlan vlan 40 vni 100040
!
router bgp 64539
vlan 40
rd 10.245.0.29:40
route-target both 40:40
redistribute learned
!
interface Ethernet4
switchport access vlan 40
- “show interface vxlan1” for a quick glance at the VTEP
- “show vxlan vtep” will show remote vteps
- “show vxlan address-table” will show MACs learned via VXLAN
- “show bgp evpn route-type mac-ip” will show the Type-2 EVPN routes, which are the MAC addresses transported over the IP fabric as L2 VXLAN packets
We can add some IP addresses to the end hosts to show that we have L2 connectivity.
VPC401 ip 40.40.40.10 255.255.255.0 40.40.40.1
VPC402 ip 40.40.40.20 255.255.255.0 40.40.40.1
VPC403 ip 40.40.40.30 255.255.255.0 40.40.40.1
But first let’s look at the switches and see the mac addresses because that’s really what we are enabling.
leaf01#sh int vxlan1
Vxlan1 is up, line protocol is up (connected)
Hardware is Vxlan
Source interface is Loopback1 and is active with 10.245.1.25
Replication/Flood Mode is headend with Flood List Source: EVPN
Remote MAC learning via EVPN
VNI mapping to VLANs
Static VLAN to VNI mapping is
[40, 100040]
Note: All Dynamic VLANs used by VCS are internal VLANs.
Use 'show vxlan vni' for details.
Static VRF to VNI mapping is not configured
Headend replication flood vtep list is:
40 10.245.1.26 10.245.1.29
Shared Router MAC is 0000.0000.0000
leaf01#sh vxlan vtep
Remote VTEPS for Vxlan1:
VTEP Tunnel Type(s)
----------------- --------------
10.245.1.26 unicast, flood
10.245.1.29 unicast, flood
Total number of remote VTEPS: 2
leaf01#sh vxlan address-table
Vxlan Mac Address Table
----------------------------------------------------------------------
VLAN Mac Address Type Prt VTEP Moves Last Move
---- ----------- ---- --- ---- ----- ---------
40 0050.7966.6808 EVPN Vx1 10.245.1.29 1 0:00:36 ago
40 0050.7966.680c EVPN Vx1 10.245.1.26 1 0:00:26 ago
Total Remote Mac Addresses for this criterion: 2
leaf01#sh bgp evpn route-type mac-ip
BGP routing table information for VRF default
Router identifier 10.245.0.25, local AS number 64535
Route status codes: * - valid, > - active, S - Stale, E - ECMP head, e - ECMP
c - Contributing to ECMP, % - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop
Network Next Hop Metric LocPref Weight Path
* > RD: 10.245.0.25:40 mac-ip 0050.7966.6806
- - - 0 i
* >Ec RD: 10.245.0.29:40 mac-ip 0050.7966.6808
10.245.1.29 - 100 0 64512 64539 i
* ec RD: 10.245.0.29:40 mac-ip 0050.7966.6808
10.245.1.29 - 100 0 64512 64539 i
* >Ec RD: 10.245.0.26:40 mac-ip 0050.7966.680c
10.245.1.26 - 100 0 64512 64536 i
* ec RD: 10.245.0.26:40 mac-ip 0050.7966.680c
10.245.1.26 - 100 0 64512 64536 i
leaf01#sh mac address-table
Mac Address Table
------------------------------------------------------------------
Vlan Mac Address Type Ports Moves Last Move
---- ----------- ---- ----- ----- ---------
40 0050.7966.6806 DYNAMIC Et4 1 0:00:44 ago
40 0050.7966.6808 DYNAMIC Vx1 1 0:00:57 ago
40 0050.7966.680c DYNAMIC Vx1 1 0:00:47 ago
Total Mac Addresses for this criterion: 3
Multicast Mac Address Table
------------------------------------------------------------------
Vlan Mac Address Type Ports
---- ----------- ---- -----
Total Mac Addresses for this criterion: 0
leaf01#
leaf02#sh int vxlan1
Vxlan1 is up, line protocol is up (connected)
Hardware is Vxlan
Source interface is Loopback1 and is active with 10.245.1.26
Replication/Flood Mode is headend with Flood List Source: EVPN
Remote MAC learning via EVPN
VNI mapping to VLANs
Static VLAN to VNI mapping is
[40, 100040]
Note: All Dynamic VLANs used by VCS are internal VLANs.
Use 'show vxlan vni' for details.
Static VRF to VNI mapping is not configured
Headend replication flood vtep list is:
40 10.245.1.25 10.245.1.29
Shared Router MAC is 0000.0000.0000
leaf02#sh vxlan vtep
Remote VTEPS for Vxlan1:
VTEP Tunnel Type(s)
----------------- --------------
10.245.1.25 unicast, flood
10.245.1.29 unicast, flood
Total number of remote VTEPS: 2
leaf02#sh vxlan address-table
Vxlan Mac Address Table
----------------------------------------------------------------------
VLAN Mac Address Type Prt VTEP Moves Last Move
---- ----------- ---- --- ---- ----- ---------
40 0050.7966.6806 EVPN Vx1 10.245.1.25 1 0:02:02 ago
40 0050.7966.6808 EVPN Vx1 10.245.1.29 1 0:02:16 ago
Total Remote Mac Addresses for this criterion: 2
leaf02#sh bgp evpn route-type mac-ip
BGP routing table information for VRF default
Router identifier 10.245.0.26, local AS number 64536
Route status codes: * - valid, > - active, S - Stale, E - ECMP head, e - ECMP
c - Contributing to ECMP, % - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop
Network Next Hop Metric LocPref Weight Path
* >Ec RD: 10.245.0.25:40 mac-ip 0050.7966.6806
10.245.1.25 - 100 0 64512 64535 i
* ec RD: 10.245.0.25:40 mac-ip 0050.7966.6806
10.245.1.25 - 100 0 64512 64535 i
* >Ec RD: 10.245.0.29:40 mac-ip 0050.7966.6808
10.245.1.29 - 100 0 64512 64539 i
* ec RD: 10.245.0.29:40 mac-ip 0050.7966.6808
10.245.1.29 - 100 0 64512 64539 i
* > RD: 10.245.0.26:40 mac-ip 0050.7966.680c
- - - 0 i
leaf02#sh mac address-table
Mac Address Table
------------------------------------------------------------------
Vlan Mac Address Type Ports Moves Last Move
---- ----------- ---- ----- ----- ---------
40 0050.7966.6806 DYNAMIC Vx1 1 0:02:14 ago
40 0050.7966.6808 DYNAMIC Vx1 1 0:02:28 ago
40 0050.7966.680c DYNAMIC Et4 1 0:02:18 ago
Total Mac Addresses for this criterion: 3
Multicast Mac Address Table
------------------------------------------------------------------
Vlan Mac Address Type Ports
---- ----------- ---- -----
Total Mac Addresses for this criterion: 0
leaf02#
leaf03#sh int vxlan1
Vxlan1 is up, line protocol is up (connected)
Hardware is Vxlan
Source interface is Loopback1 and is active with 10.245.1.29
Replication/Flood Mode is headend with Flood List Source: EVPN
Remote MAC learning via EVPN
VNI mapping to VLANs
Static VLAN to VNI mapping is
[40, 100040]
Note: All Dynamic VLANs used by VCS are internal VLANs.
Use 'show vxlan vni' for details.
Static VRF to VNI mapping is not configured
Headend replication flood vtep list is:
40 10.245.1.25 10.245.1.26
Shared Router MAC is 0000.0000.0000
leaf03#sh vxlan vtep
Remote VTEPS for Vxlan1:
VTEP Tunnel Type(s)
----------------- --------------
10.245.1.25 flood, unicast
10.245.1.26 flood, unicast
Total number of remote VTEPS: 2
leaf03#sh vxlan address-table
Vxlan Mac Address Table
----------------------------------------------------------------------
VLAN Mac Address Type Prt VTEP Moves Last Move
---- ----------- ---- --- ---- ----- ---------
40 0050.7966.6806 EVPN Vx1 10.245.1.25 1 0:03:22 ago
40 0050.7966.680c EVPN Vx1 10.245.1.26 1 0:03:25 ago
Total Remote Mac Addresses for this criterion: 2
leaf03#sh bgp evpn route-type mac-ip
BGP routing table information for VRF default
Router identifier 10.245.0.29, local AS number 64539
Route status codes: * - valid, > - active, S - Stale, E - ECMP head, e - ECMP
c - Contributing to ECMP, % - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop
Network Next Hop Metric LocPref Weight Path
* >Ec RD: 10.245.0.25:40 mac-ip 0050.7966.6806
10.245.1.25 - 100 0 64512 64535 i
* ec RD: 10.245.0.25:40 mac-ip 0050.7966.6806
10.245.1.25 - 100 0 64512 64535 i
* > RD: 10.245.0.29:40 mac-ip 0050.7966.6808
- - - 0 i
* >Ec RD: 10.245.0.26:40 mac-ip 0050.7966.680c
10.245.1.26 - 100 0 64512 64536 i
* ec RD: 10.245.0.26:40 mac-ip 0050.7966.680c
10.245.1.26 - 100 0 64512 64536 i
leaf03#sh mac address-table
Mac Address Table
------------------------------------------------------------------
Vlan Mac Address Type Ports Moves Last Move
---- ----------- ---- ----- ----- ---------
40 0050.7966.6806 DYNAMIC Vx1 1 0:03:33 ago
40 0050.7966.6808 DYNAMIC Et4 1 0:03:47 ago
40 0050.7966.680c DYNAMIC Vx1 1 0:03:37 ago
Total Mac Addresses for this criterion: 3
Multicast Mac Address Table
------------------------------------------------------------------
Vlan Mac Address Type Ports
---- ----------- ---- -----
Total Mac Addresses for this criterion: 0
leaf03#
In this example, we will isolate traffic into a VRF and transport that VRF over the EVPN network virtualization overlay using EVPN Type-5 routes. Rather than having VRFs peerings configured all over the place, we only need to configure the VRFs and the fabric will handle the isolation for us without countless numbers of BGP peerings. This funciton is transparent to the Spines and only configured on the Leafs we want the network to appear.
- Configure a VRF (I named it “gold_vrf”)
- Enable routing for the VRF
- Configure a VLAN SVI and add it to the vrf
- Map the VRF to a VNI (I used 100001)
- Configure the VRF under BGP
Leaf01
!
vrf instance gold_vrf
!
ip routing vrf gold_vrf
!
interface Vlan40
vrf gold_vrf
ip address virtual 40.40.40.1/24
!
interface vxlan1
vxlan vrf gold_vrf vni 100001
!
router bgp 64535
vrf gold_vrf
rd 10.245.0.25:40
route-target import evpn 40:40
route-target export evpn 40:40
redistribute connected
Leaf02
!
vrf instance gold_vrf
!
ip routing vrf gold_vrf
!
interface Vlan40
vrf gold_vrf
ip address virtual 40.40.40.1/24
!
interface vxlan1
vxlan vrf gold_vrf vni 100001
!
router bgp 64536
vrf gold_vrf
rd 10.245.0.26:40
route-target import evpn 40:40
route-target export evpn 40:40
redistribute connected
Leaf03
!
vrf instance gold_vrf
!
ip routing vrf gold_vrf
!
interface Vlan40
vrf gold_vrf
ip address virtual 40.40.40.1/24
!
interface vxlan1
vxlan vrf gold_vrf vni 100001
!
router bgp 64539
vrf gold_vrf
rd 10.245.0.29:40
route-target import evpn 40:40
route-target export evpn 40:40
redistribute connected
To make it interesting we will add a second, non-VXLAN interface to Leaf01
Leaf01 ! vlan 50 name test-l3-vlan ! interface Vlan50 vrf gold_vrf ip address 50.50.50.1/24 ! int e3 switchport access vlan 50
VPC50 ip 50.50.50.10 255.255.255.0 50.50.50.1
- “show vxlan vtep” will show remote vteps
- “show bgp evpn route-type ip-prefix ipv4” will show the Type-5 EVPN routes, which are the VRFs we’re transporting across the EVPN fabric
- “show ip route vrf gold_vrf” should show us the routes learned from the remote VTEP
- “show ip arp vrf gold_vrf”
- ping from end point to endpoint
leaf01#sh vxlan vtep
Remote VTEPS for Vxlan1:
VTEP Tunnel Type(s)
----------------- --------------
10.245.1.26 unicast, flood
10.245.1.29 unicast, flood
Total number of remote VTEPS: 2
leaf01#sh bgp evpn route-type ip-prefix ipv4
BGP routing table information for VRF default
Router identifier 10.245.0.25, local AS number 64535
Route status codes: * - valid, > - active, S - Stale, E - ECMP head, e - ECMP
c - Contributing to ECMP, % - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop
Network Next Hop Metric LocPref Weight Path
* > RD: 10.245.0.25:40 ip-prefix 40.40.40.0/24
- - - 0 i
* >Ec RD: 10.245.0.26:40 ip-prefix 40.40.40.0/24
10.245.1.26 - 100 0 64512 64536 i
* ec RD: 10.245.0.26:40 ip-prefix 40.40.40.0/24
10.245.1.26 - 100 0 64512 64536 i
* >Ec RD: 10.245.0.29:40 ip-prefix 40.40.40.0/24
10.245.1.29 - 100 0 64512 64539 i
* ec RD: 10.245.0.29:40 ip-prefix 40.40.40.0/24
10.245.1.29 - 100 0 64512 64539 i
* > RD: 10.245.0.25:40 ip-prefix 50.50.50.0/24
- - - 0 i
leaf01#sh ip route vrf gold_vrf
VRF: gold_vrf
Codes: C - connected, S - static, K - kernel,
O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
N2 - OSPF NSSA external type2, B - Other BGP Routes,
B I - iBGP, B E - eBGP, R - RIP, I L1 - IS-IS level 1,
I L2 - IS-IS level 2, O3 - OSPFv3, A B - BGP Aggregate,
A O - OSPF Summary, NG - Nexthop Group Static Route,
V - VXLAN Control Service, M - Martian,
DH - DHCP client installed default route,
DP - Dynamic Policy Route, L - VRF Leaked,
G - gRIBI, RC - Route Cache Route
Gateway of last resort is not set
B E 40.40.40.20/32 [20/0] via VTEP 10.245.1.26 VNI 100001 router-mac 50:00:00:cb:38:c2 local-interface Vxlan1
B E 40.40.40.30/32 [20/0] via VTEP 10.245.1.29 VNI 100001 router-mac 50:00:00:d5:5d:c0 local-interface Vxlan1
C 40.40.40.0/24 is directly connected, Vlan40
C 50.50.50.0/24 is directly connected, Vlan50
leaf01#
leaf02#sh vxlan vtep
Remote VTEPS for Vxlan1:
VTEP Tunnel Type(s)
----------------- --------------
10.245.1.25 unicast, flood
10.245.1.29 unicast, flood
Total number of remote VTEPS: 2
leaf02#sh bgp evpn route-type ip-prefix ipv4
BGP routing table information for VRF default
Router identifier 10.245.0.26, local AS number 64536
Route status codes: * - valid, > - active, S - Stale, E - ECMP head, e - ECMP
c - Contributing to ECMP, % - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop
Network Next Hop Metric LocPref Weight Path
* >Ec RD: 10.245.0.25:40 ip-prefix 40.40.40.0/24
10.245.1.25 - 100 0 64512 64535 i
* ec RD: 10.245.0.25:40 ip-prefix 40.40.40.0/24
10.245.1.25 - 100 0 64512 64535 i
* > RD: 10.245.0.26:40 ip-prefix 40.40.40.0/24
- - - 0 i
* >Ec RD: 10.245.0.29:40 ip-prefix 40.40.40.0/24
10.245.1.29 - 100 0 64512 64539 i
* ec RD: 10.245.0.29:40 ip-prefix 40.40.40.0/24
10.245.1.29 - 100 0 64512 64539 i
* >Ec RD: 10.245.0.25:40 ip-prefix 50.50.50.0/24
10.245.1.25 - 100 0 64512 64535 i
* ec RD: 10.245.0.25:40 ip-prefix 50.50.50.0/24
10.245.1.25 - 100 0 64512 64535 i
leaf02#sh ip route vrf gold_vrf
VRF: gold_vrf
Codes: C - connected, S - static, K - kernel,
O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
N2 - OSPF NSSA external type2, B - Other BGP Routes,
B I - iBGP, B E - eBGP, R - RIP, I L1 - IS-IS level 1,
I L2 - IS-IS level 2, O3 - OSPFv3, A B - BGP Aggregate,
A O - OSPF Summary, NG - Nexthop Group Static Route,
V - VXLAN Control Service, M - Martian,
DH - DHCP client installed default route,
DP - Dynamic Policy Route, L - VRF Leaked,
G - gRIBI, RC - Route Cache Route
Gateway of last resort is not set
C 40.40.40.0/24 is directly connected, Vlan40
B E 50.50.50.0/24 [20/0] via VTEP 10.245.1.25 VNI 100001 router-mac 50:00:00:d7:ee:0b local-interface Vxlan1
leaf02#
leaf03#sh vxlan vtep
Remote VTEPS for Vxlan1:
VTEP Tunnel Type(s)
----------------- --------------
10.245.1.25 flood, unicast
10.245.1.26 flood, unicast
Total number of remote VTEPS: 2
leaf03#sh bgp evpn route-type ip-prefix ipv4
BGP routing table information for VRF default
Router identifier 10.245.0.29, local AS number 64539
Route status codes: * - valid, > - active, S - Stale, E - ECMP head, e - ECMP
c - Contributing to ECMP, % - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop
Network Next Hop Metric LocPref Weight Path
* >Ec RD: 10.245.0.25:40 ip-prefix 40.40.40.0/24
10.245.1.25 - 100 0 64512 64535 i
* ec RD: 10.245.0.25:40 ip-prefix 40.40.40.0/24
10.245.1.25 - 100 0 64512 64535 i
* >Ec RD: 10.245.0.26:40 ip-prefix 40.40.40.0/24
10.245.1.26 - 100 0 64512 64536 i
* ec RD: 10.245.0.26:40 ip-prefix 40.40.40.0/24
10.245.1.26 - 100 0 64512 64536 i
* > RD: 10.245.0.29:40 ip-prefix 40.40.40.0/24
- - - 0 i
* >Ec RD: 10.245.0.25:40 ip-prefix 50.50.50.0/24
10.245.1.25 - 100 0 64512 64535 i
* ec RD: 10.245.0.25:40 ip-prefix 50.50.50.0/24
10.245.1.25 - 100 0 64512 64535 i
leaf03#sh ip route vrf gold_vrf
VRF: gold_vrf
Codes: C - connected, S - static, K - kernel,
O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
N2 - OSPF NSSA external type2, B - Other BGP Routes,
B I - iBGP, B E - eBGP, R - RIP, I L1 - IS-IS level 1,
I L2 - IS-IS level 2, O3 - OSPFv3, A B - BGP Aggregate,
A O - OSPF Summary, NG - Nexthop Group Static Route,
V - VXLAN Control Service, M - Martian,
DH - DHCP client installed default route,
DP - Dynamic Policy Route, L - VRF Leaked,
G - gRIBI, RC - Route Cache Route
Gateway of last resort is not set
C 40.40.40.0/24 is directly connected, Vlan40
B E 50.50.50.0/24 [20/0] via VTEP 10.245.1.25 VNI 100001 router-mac 50:00:00:d7:ee:0b local-interface Vxlan1
leaf03#