OTV

OTV:
L2 interconnect datacenter protocol.
What is OTV?
1) Overlay Transport Virtualization(OTV)
– Layer 2 VPN over IPv4
2) Specifically OTV is…
– IPv4/IPv6 over Ethernet…
– over MPLS..
– Over GRE..
-Over IPv4
– Over Ethernet (usually)
It have header overhead, it doesn’t support fragmentation. The ideas here, since we already have specification of ethernet over MPLS with feature like any transport over MPLS or feature like VPLS so don’t need any other technology. one of the advantage of OTV, we not stuck in certain condition where we must run MPLS or we must have these certain requirements for interconnect. Basically  we have any transport between our sites, could be dedicated link, could be over normal internet routing we are able to run OTV features..

Why Use OTV?
1) OTV is designed for layer 2 data center interconnect(DCI)
2) Layer 2 DCI is needed for Virtual Machine Workload mobility
e.g. VMware VMotion
3) OTV has built in enhancements to help scale L2 DCI

OTV vs Other DCIs
1) Many possible options for L2 DCI
– Dark Fiber(CWDM/DWDM)
– L2TPv3(it doesn’t needed MPLS)
– AToM(any trasport over MPLS)
– VPLS
AToM is point to point L2 MPLS VPN tunnel and VPLS is point to multipoint MPLS VPN tunnel. Disavantage of AToM and VPLS is to go over MPLS provider. One of the shortcoming of VPLS The service provider edge router i.e PE router in MPLS network side those must maintain the mac address for customer.
– Bridging over GRE(we could solve above problem using this technology)
With OTV we are using the GRE tunnel, there are some additional enhancement built with OTV specifically, it would stop the flooding like STP, ARP request and reply, ICMPv6 neighbour discovering request and reply and also failure domain which came from broadcast storm.
2) These options can be used for DCI, but OTC was made for DCI
– Optimizes ARP flooding over DCI
– Demarc of the STP domain
– Can overlay multiple VLANs without complicated design
– Allows multiple edge routers without complicated design

OTV Terminilogy
1) OTV edge device
– Edge router(s) running OTV
2) Authoritative Edge Device(AED)
AED used when we have multiple edge router for the purpose of redundancy and we make sure that we have load distribution between them. AED is active forwarder for particular VLAN.
– Active edge router for a particular VLANs
– Allows multiple redundant edge routes while preventing loops
There will be election to decide we will be the forwarder. Since we are not extending STP on DCI we dont have additional L2 loop prevention technique. So we can say only one router can actively forward the traffic one at a time.
3) Extend VLANs
– VLAN being bridged over OTV
It basically L2 segment that we are trying to tunnel over OTV.
4) Site VLAN
– internal VLAN used to elect AED.
5) Site Identifier
– Unique ID per DC site, shared between AEDs
This is basically loop preventative technique if packet would go out and come back in I am not going to receive my own packet since I am part of same site.
6) Overlay Interface:
– The logical OTV tunnel interface
This is equivalent to GRE tunnel interface. THis is the logical link used to tunneling. THen we have physical interface where tunnel is run top of this is we called it as OTV joined interface.
7) OTV join interface
– The physical link or port-channel that you use to route upstream towards the DCI
Join interface can not be SVI, so it either could be physical link or L3 port-channel
Then we have 2 OTV multicast group
1) OTV Control Group
– Multicast address used to discover the remote sites in the control plane
It is used to extending protocol like ARP, OSPF, eigrp over data center interconnect and this traffic is tunnel inside the multicast over the DCI network  This is then imply that by default your transit if you are using mpls vpn it does have to be multicast aware. It does support any source multicast(ASM) and SSM
2) OTV Data Group:
– Used when you’re tunneling multicast over OTV in the data plane.
if i have multicast data feed between the data center then it would get encapsulated inside multicast so that we can avoid head end replication. Basically taking multicast packet and not having multicast unicast copies of it inorder to forward to DCI.

OTV Control Plane
1) Uses IS-IS to advertises MAC addresses between AEDs.
– “MAC in IP” routing
In OTV its event driven behaviour that the edge router they are only going to know the mac address that are actively advertise with IS-IS. Now advantage of doing this is to give you more control as what specific types of traffic can sent over OTV. In case of MPLS l2 VPN we can apply L3 access list to the interface or some sort of vlan access list in order to stop the flow across the link. In case of OTV we can stop this in control plane by telling IS-IS not advertise specific MAC address. So i have host that I want to stay local and not goes over DCI. I can deny that from being advertise in IS-IS control plane. So it’s kind of distribute list that can apply to L2 mac address inside IS-IS advertisement. In OTV we can take L2 Ethernet and encapsulate inside of IP so again it’s Ethernet over MPLS, GRE and IP.
2) Encapsulated as Control group multicast
– IS-IS over ethernet over MPLS over GRE over IPv4 multicast
– Implied that DCI must support ASM multicast.
– Can be encapsulated as unicast with OTV adjacency server.
It’s needed to exchange IS-IS control plane. Specifically IS-IS does have it’s own L3 transport. It means if I have AED(edge device) on data center edge and I want them to as IS-IS adjacent I could have some problem considering that network between them is IPv4. Because IS-IS doesn not run over IP it run over CLNS or CLNP stack. So inorder to allows to do this the control plane of IS-IS is encapsulated inside ethernet over MPLS over GRE over IPv4 multicast. It means the core of data center it must Any source multicast aware. We can see there are some work around for this Can be encapsulated inside unicast with OTV adjacency server and that can remove the requirement that the core of the DCI network to be L3 multicast routing aware.

OTV Data Plane
1) Uses both unicast and multicast transport(IPv4).
2) Multicast control Group
– Multicast or broadcast control plane protocols like OSPF, EIGRP or PIM that can be tunnel over OTV
– ex. ARP, OSPF, EIGRP, etc.
3) Unicast data
– Normal Unicast is encapsulated as Unicast between AEDs
4) Multicast Data Group
– Multicast data flows are encapsulated as SSM multicast.
– Implies AEDs used IGMPv3 for (S,G) Joins.
5) OTV adjacency server can remove requirement for multicast completely
– Will result in “Head end replication” when ore than 2 DC’s connected over the DCI. hence when more thatn 2 DC connected we would not want to use this.

OTV DCI Optimization
1) Other DCI options bridge all traffic over DCI
– ex. STP, ARP, broadcast storms, etc
So without manual filtering like ARP, STP those are automatically going across the tunnel.
2) OTV reduces unnecessary flooding by…
– Proxy ARP/ICMPv6 ND cache on AED
– Terminating the STP domain on AED

N7k1-3           N7k1-4           N7k2-5            N7k2-6

+—-+           +—-+          +—–+           +——+
|    |           |    |          |     |           |      |
|    |E1/17      |    |          |     |       E1/9|      |
|    +———–+    +———-+     +———–+      |
|    |           |    |          |     |           |      |
E1/19+—-+E1/20      +—-+          +—–+           +——+—+
|    |                                             |          |
|    |                                             |          |
E1/3 |    |  E1/12                                      |          |
+——+–+    +–^—–+                              +—-^+         ++—+
|      |          |     |N7k1-2                 N7k2-7 |    |          |    |  N7k2-8
N7k1-1      |          |     |                              |    |          |    |
|      |          |     |                              |    |          |    |
|      |          |     |                              |    |          |    |
|      |          |     |                              |    |          |    |
+–+—+          +–+–+                              +–+-+          +-+–+
|                 |                                    |              |
|                 |                                    |              |
+—–+-+—————+-+—–+               +———-+-+————–+-+——-+
|                 |                                |                  |
|                 |                              +-+–+               |
+—++                |                              |    |            XXX+X
|    |           XXXXX+                              |    |          XXX   XX
|    |           X   XXX                             |    |          X      X
|    |           X     X                             |    |          X      X
|    |           X     X                             |    |          XXXXXXXX
+—-+           XX  XXX                             +-+–+
XXX R3                            Server 3            R2
Server 1
We need separate dedicated VDC to run OTV process. We need to separate OTV process from rest of the L3 process. Since there are some technical limitation.

N7K1-3 is AED.
N7K1-4 and N7K2-5 are L3 DCI. We can configure IGP and BGP
N7K2-6 is AED. L3 tunnel between the AED devices. This is our EoMPLS over GRE over IPV4 unicast and multicast.
So there is EoMPLS over GRE over IPV4. This is additinal encapsulation we need to take care into picture since the OTV AED device doesn’t support fragmentation. We will configure vlan 10 on both side and have same broadcast domain on both side. These devices thinks that they are connected on l2 WAN. The reason we are doing this way is virulization mobility application has this requirements. When we are doing vmware vmotion, you should have your vmware host on same subnet.

Part 2: OTV initial configuration

First we need to check between AED do we have basic IP connectivity.
N7K2-6: int e1/9
no shut
no switchport
ip address 150.1.56.6/24

N7k2-5: int e1/1
no shut
ip address 150.1.56.5/24
ping 150.1.56.6 — Success. We can do any routing here but goal is to have edge devices should ping each other over Data centre interface. We will going to configure eigrp routing here.
N7K2-6: feature eigrp
router eigrp 1
int lo0
ip address 1.1.1.76/32
ip router eigrp 1
int e1/9
ip router eigrp 1 // we are enabling eigrp under link level.

N7K2-5:
inte lo0
ip address 1.1.1.75/32
feature eigrp
router eigrp 1
int lo0
ip router eigrp 1
int e1/1
ip router eigrp 1
We have checked the eigrp is working fine. Now we will configure the connectivity between N7K2-5 and N7k1-4
N7K2-5: int e1/7
no shut
no switchport
ip add 150.1.45.5/24
ip router eigrp 1

N7k1-4: int e1/31
no shut
feature eigrp
router rigrp 1
int e1/31
ip address 150.1.45.4/24
ip router eigrp 1
int lo0
ip address 1.1.1.74/32
ip router eigrp 1
router eigrp 1
neighbour is up now. We will configure the connectivity between N7k1-4 and N7K1-3
N7K1-4: int e1/25
no shut
ip address 150.1.34.4/24
ip router eigrp 1

N7K1-3: int e1/17
no shut
no switchport
feature eigrp
int lo0
ip addres 1.1.1.73/24
ip router eigrp 1
router eigrp 1
int e1/17
ip address 150.1.34.3/24
ip router eigrp 1
show ip eigrp neig — we have the adjacency. show ip router we are route for other AED. hence we have route over DCI. Once we confirmed data centre can talk to each other. Now I need to make sure that from edge device I have L2 connectivity down to actual end host that I am tunneling over to OTV.  So N7K1-1 and N7K1-2 towards N7K1-3 should be L2 trunk. We can not do L3 routing here because goal is to make L2 domain end to end.
On N7K1-3:
int e1/19-20
switchport mode trunk
spanning-tree port type network
no shut
switchport
Those port can be VPC, but it doesn’t matter until we have L2 connectivity.
On N7K1-1
int e1/3
switchport
switchport mode trunk
spanning-tree port type network
no shut

On N7K1-2
int e1/12
switchport
switchport mode trunk
spanning-tree port type network
no shut

We have to configure 2 minimum vlan here. one of them going to be over-lay, it means one of them is forwarded over OTV tunnel and one of them going to be site vlan. Now site vlan is going to sync authoritative edge router(AED). When we have multiple AED, we can do load distribution.  Vlan 10 is for overlay and vlan 999 is site vlan.
N7K1-3:
Vlan 10,999
vlan 10
name OVERLAY_VLAN
vlan 999
name SITE_VLAN
Configure the same on both down stream router as well. (N7K1-1 and N7K1-2). We have to also do it on other side( N7K2-6, N7K2-7, N7K2-8 ) Site vlan is locally significant and it will not bridge over OTV. So it could be any random number. To test underlying l2 is working fine. We can configure IP address on Vlan 10 on AED router and try to ping it from R3.
N7K1-3:
feature interface-vlan
int vlan 10
ip add 10.0.0.73/24
no shut
ping 10.0.0.3 (R3 address)- Success
Ping 10.0.0.10 (server address) — Success
This means connectivity between individual sites are fine. We will do same configuration on other site of data centre.

N5K2: int e1/20, e1/22
switchport
switchport mode trunk
spanning-tree port type network
no shut
On remote site N7K2-8
int e2/27
switchport
switchport mode trunk
spanning-tree port type network
no shut

On N7k2-7
int e2/21
switchport
switchport mode trunk
spanning-tree port type network
no shut

N5K2: show spanning-tree vlan 10
We can see E1/20 and E1/22 are in forwarding state. Now we want L2 connectivity from N7K2-6 , N7K2-7, N7k2-8

N7K2-6–E1/11—————E1/27— N7k2-8
N7K2-6—-E1/20————E1/27——N7k2-8
On N7K2-6
int e1/11 – 12
switchport
switchport mode trunk
spanning-tree port type network
no shut
On other side we have 7 and 8
N7K2-7 and N7k2-8
int e1/20   and E1/27
switchport
switchport mode trunk
spanning-tree port type network
no shut

On N7K2-6
show spanning-tree vlan 10
we can see e1/11 and E1/20 are in fwd state.
feature interface-vlan
int vlan 10
ip address 10.0.0.76/24
no shut
If my site is working then I should able to ping R2 and server 3.
ping 10.0.0.20 — success
ping 10.0.0.30 — Success
and should not be able to ping R3 and Server 1, because we should have two disconnected L2 domains. We do have vlan 10 on both sides
ping 10.0.0.3— failed
Edge devices are doing to be multicast receivers and they are not going to be multicast router. It doesn’t necessary to mean they can not do multicast routing at the same time but you are not required to run PIM on the join interface. We certainly use PIM on outside and can use pim inside if we are doing our own multicast routing inside.
In this topology we do need to do multicast routing its gone be on N7K1-4 and N7K2-5
On N7K1-4
feature pim
int e1/25,e1/31
ip pim sparse
We will configure N7K1-4 is RP
int lo0
ip pim sparse
exit
ip pim rp-address 1.1.1.74
On N7K2-5
feature pim
int e1/1, e1/7
ip pim sparse
ip pim rp-address 1.1.1.74
show ip pim neig: we can see adjacency between them.
Now we will configure it on actual edge devices
On N71K3: We need to enable the feature otv
feature otv
On N7K2-6
feature otv
Now we need to define site parameter, such as site vlan, extended identifier, site-identifier.
On N71K3:
otv site-identifier 0.0.1 // if we you 2 AED(authoritative edfe device on same site then identifier should match but for other site identifier should be different since it’s loop prevention mechanism)
On N7K2-6:
otv site-identifier 0.0.2
otv site-vlan 999
On N71K3:
otv site-vlan 999
int e1/17
ip igmp ver 3
On N7K2-6:
int e1/9
ip igmp ver 3
interface overlay 1
otv control-group  225.6.7.8//RP to join share tree. This menas control-range should not be 232 group. 232 is SSM so anything outside of this range.
On N7K1-3
inte overlay 1
otv control-group 225.6.7.8
otv data-group 232.1.2.0/24 // this is for multicast in multicast. This we do want to be SSM group. When I have different data multicast stream that I have to send I can have multicast SSM group basically and multiple stram that I can join.
On N7K2-6:
otv data-group 232.1.2.0/24
otv extend-vlan add 10  // so what vlan going over the tunnel
On N7K1-3
otv extend-vlan add 10
otv join-interface e1/17 what the actual physical link we are joining on? the link which goes upto the DCI
On N7K2-6:
otv join-interface e1/9
The configure on both AED should be identical except the site-identifier, join interface.
We will brin up the interface up and if everything is working then we will form IS-IS adjacency
On N7K1-3
int o1
no shut
On N7K2-6:
int o1
no shut
On Core N7K1-4: show ip mroute
we can see multicast control group which has s: 225.6.7.8 for both AED’s which means it has IGMPv2 join using s,g to get anysouce mutlcast tree. so RP seeing both of the AED which it does.
We dont see source specific multicast yet since we are not tunneling any multicast traffic in yet. Once we go into end server and generate UDP filed then we should see 232 range in core.
On N7K2-6:
show otv isis adjacency
state is INIT: this state is not correct. What we should see since IS IS for OTV for fabric path for normal L3 routing it has host dynamic feature on by default so this means if the adjacency form completely one of the type val subfield is going to be router’s hostname . So it’s normal exchange between routers update. It means if I send my update to you and you send your update to me and we build spt fully. I should see your host name when I look at the show otv isis adjacency so it means there is some communication problem with IS IS. It could be possbility that We had that vlan should takenoff first.
on N7K2-6
show run int vlan 10
no int vlan 10  // we are not suppose to have SVI in OTV current VDC.
Lets shut the overlay interface down it would restart the process.
also enable debug on both AED debug otv isis adjacency
on both AED: sho otv isis adjacency// we didn’t see any output it means we are not receiving any packet from other side.
ON Core: N7K104: show ip mrout: we see outgoing interface list is NULL. It means we have receive the join but we didn’t actually build tree .. We may have misconfigure pim here.
show ip pim interf brief: we can see pim is propelry enable on interface.
show ip igmp group  // this will tell us did we receive igmp report from edge router since they are client/receiver of multicast.
N7K1-4: show ip igmp group// we didn’t see any report entry this is the issue we have. so we are not getting join from N7K1-3
On N7K1-3:
we can see we have igmp ver 3 enable on interface which is connected to the core N7K1-4. On core we have pim spare more enable. Where could be the problem? Lets clear the multicast routing table and refresh it on core: clear ip mroute *. We may have order of operation problem here since we have SVI enable on it. so we will remove the config and redo it.
ON N7K1-3:
inter e1/17
shut
no shut
On N7K2-6:
int e1/9
shut
no shut
We will shut down both the link and clar the multicast table on core and brin up the interface on AED. We are doing control-group ip address.
On N7K1-3: interface over 1
otv control-group 225.6.7.9
On N7K2-6
otv control-group 225.6.7.9
we have shut the physical link and overlay interface.
up the physical link first.
We are actually missing one command. The interface on core pointing towards AED should be igmp ver 3 aware.
On N7K1-4:
inte e1/25
ip igmp ver 3
On N7k2-5:
int e1/1
ip igmp ver 3
Documentation doesn’t show this since it’s outside of the otv configuration. Now bring overlay up.
On N7K1-3:
show otv isis adjacency:state is UP.
Now go to end device i.e. R3: ping 255.255.255.255
We see we got response from R2 i.e. all the across the overlay. To have same broadcast domain between two site, we should have same vlan tag id.
——————————————————
Video 3: OTV advanced configuration and verification:
On AED N7K1-3
show otv isis database // we can see edge devices
OTV tunnel is point to multi point tunnel if we have three or more site that are interconnect, we could have single otv tunnel representing all of them.
show otv isis database // this is were we will type len value specifically for OTV that advetising VLAN # and mac address of end host. On N7K2-6: It advertising mac address 000002, this is referring to R2 and another mac belong to Server 3.
From one server we are sending 229.9.9.9 stream and on other site another server is listening that stream. However, we are not seeing multicast is working. We will debug now. Here are may need to add SVI interfaces on Distribution layer switches (N7K1-1, N7K1-2)
N7K1-1
feature interface-vlan
interface vlan 10
ip add 10.0.0.71/24
feature hsrp
int vlan 10
hsrp 1
ip addres 10.0.0.254
no shut
On N7K1-2
feature interface-vlan
interface vlan 10
ip add 10.0.0.72/24
feature hsrp
int vlan 10
hsrp 1
ip addres 10.0.0.254
no shut
feature pim
int vla 10
ip pim sparse
N7K1-1
feature pim
int vlan 10
ip pim sparse
We need someone to be igmp querir to receive IGMP report from client and process then upstream. so we do need to turn on pim as above to actually this to work
show ip igmp snooping groups // we should see the receiver 239.9.9.9 which we do
show ip mroute // we should see forwarder from other side. which is not.
We will do same thing on other side distribution devices(N7K2-7 N7K2-8)
N7K2-7
feature interface-vlan
feature pim
feature hsrp
int vlan 10
ip add 10.0.0.77/24
hsrp 1
ip 10.0.0.254
int vlan 10
no shut
int vlan 10
ip pim sparse
end
N7K2-8
feature interface-vlan
feature pim
feature hsrp
int vlan 10
ip address 10.0.0.78/24
hsrp 1
ip 10.0.0.254
exit
ip pim sparse
On N7K1-1:
show ip pim neig // we actually formed control plane adjacency across data center interconnect which we would not want in this type of design. Typically we can keep your L3 control protocol local  and if we need L2 switching between two data center we can do that in OTV but there is no restriction right now by saying pim shouldn’t go to overlay or hsrp shouldn’t go to the overlay so when we look at all four box and execute
show hsrp // we can see all of them are part of same group and typically that you would not want. Because what this mean for L3 point of view. I could have devices in data center 1 that are using the default gateway that are in data center 2 that means all the L3 switch traffic would have to go over to the DCI in order to routed upto the core. So ideally we will keep our L3 traffic local and L2 traffic over OTV. Before get into solution for this we will see on server if they are able to send multicast. New group 230.1.1.1. We are not receving the multicast.
On N7K1-1:
show ip igmp snooping group // we can see there is receiver for 230.1.1.1 on e2/3. Then go to other side. N7K2-7: show ip igmp snooping groups // we see via E1/20 we got the igmp report . We will check on core router which is RP also, N7K1-4: show ip mroute, we can see there are SSM group which are coming in and going out. steam is using the multicast data group as we have configured, 232.1.2.1, show ip mroute summary software-forwarded // which show us the bit rate for the stream, it looks like the traffic is not forwarding. we turn multicast on and send basic multicast pim and see actually its getting encoded in multicast group on core.
On R2: debug ip mpacket
int g0/0
ip igmp join 224.11.22.33
On R3: ping 224.11.22.33 repeat 10000 time 0// We need to actually take the packet on data center interconnect core and see if these traffic is tunnel over multicast.
On N7K1-4: show ip mroute // we should see is a new SSM group its going to be soruce IP  150.1.34.3(N7K1-3 AED IP since source is connected that AED) dest ip: 232.1.2.0/32
show ip mroute summary software-forwarded  //we can see bunch of packets are going through from 150.1.34.3.
We are sending new traffic from server 231.1.1.1: I think there is some issues on receving server application. The best test is to check on core router and this multicast traffic should be passes through in data group rather than control-group. This is similar logic with l3 MPLS VPN where we basically creating multi point GRE tunnel that has multicast destination and diff between NDT data group and control group . Contol group name implies synch the control plane but once you want to send data feed you would not want to use same group over and over because if there are three of more data center site over DCI when one of them sent join you wouldn’t want all other receive that traffic thats the reason we want to use any source multicast which is gone synch the control plane and SSM so only particular site that want traffic is able to actually receive it.
OTV ping packet: ethernet header- ip header–GRE header–MPLS—Ethernet(original frame generated by end host )– IP —icmp

Design consideration:
We want L3 traffic on one site to be local. The way we can do that is to remove specific control plane protocol that we do not want to send over data centre interconnect. We need to configure filter on OTV edge device in order to stop the hsrp traffic , pim traffic or wahtever control plane traffic we want to drop.
Google search: OTV hsrp filter or otv hsrp mac list (to see the syntax)

If we want to add multiple AED, I can say lets say create L2 connection between N7K2-6 and N7K2-7, N7K2-8. N7K2-6 and N7K2-7 are AED’s. There is no connection between N7K2-6 and N7K2-7. N7K2-7 has L3 connection with core N7K2-5. On site vlan which runs over l2 link. There gone be election process to figure out who gone forward which vlans. OTV site-identifier does match between the both AED. There is no other additional configuration.
What configuration we want to changes if we do not want to use multicast at all.
N7K1-3: int ov1
shut
no data-group 232.1.2.0/24
no control-group 225.6.7.8
otv adjacency-server unicast-only
otv use-adjacency-server 150.1.34.3 unicast-only
no shut
N7K2-6: int ov1
shut
no data-group 232.1.2.0/24
no control-group 225.6.7.8
otv adjacency-server unicast-only
otv use-adjacency-server 150.1.34.3 unicast-only
no shut
sho otv isis adj: state is up.
now multicast and unicast both goes in unicast. Since there are only two site, it doesn’t make any difference if we are using the multicast or unicast here.
All server agree on what could be an adjacency server IP could be.