I want to set up a VXLAN tunnel between Linux Network Interfaces on different VMs. More precisely, the topology is as follows. I have two Ubuntu 16.04 Vagrant boxes with OvS 2.5.5 installed. On each VM, I created an OvS bridge named kos, and to each bridge I added a vxlan port/interface with the following command:
ovs-vsctl add-port kos vtep -- set interface vtep type=vxlan option:remote_ip=flow option:key=flow
Then I created two Linux Network Interfaces on each machine. The interfaces on the same machine have the same IP address but different MAC addresses. I want the four interfaces to be split into two separate VXLANs (IDs 1 and 2 respectively), and use OvS VXLAN tunneling to enable communication between interfaces on different VMs but in the same VXLAN. The commands used to create the interfaces and connect them to OvS are:
VM1:
ovs-vsctl add-port kos kos11 -- set interface kos11 type=internal mac="02\:00\:00\:00\:00\:11" && ip addr add 10.0.0.1/8 scope link dev kos11 && ip link set kos11 up
ovs-vsctl add-port kos kos21 -- set interface kos21 type=internal mac="02\:00\:00\:00\:00\:21" && ip addr add 10.0.0.1/8 scope link dev kos21 && ip link set kos21 up
VM2:
ovs-vsctl add-port kos kos12 -- set interface kos12 type=internal mac="02\:00\:00\:00\:00\:12" && ip addr add 10.0.0.2/8 scope link dev kos12 && ip link set kos12 up
ovs-vsctl add-port kos kos22 -- set interface kos22 type=internal mac="02\:00\:00\:00\:00\:22" && ip addr add 10.0.0.2/8 scope link dev kos22 && ip link set kos22 up
Each interface name is built by concatenating "kos" followed by two digits, the first one being the VNI of the interface, the second one the number of the VM it belongs to.
Finally, I retrieved the openflow ports of each interface (vtep on both VMs, kos11, kos21, kos12, kos22), and added the following OpenFlow flows:
VM1:
table=0,in_port=2,actions=set_field:1->tun_id,resubmit(,1)
table=0,in_port=3,actions=set_field:2->tun_id,resubmit(,1)
table=0,actions=resubmit(,1)
table=1,tun_id=1,arp,arp_tpa=10.0.0.1,actions=output:2
table=1,tun_id=1,dl_dst=02:00:00:00:00:11,actions=output:2
table=1,tun_id=2,arp,arp_tpa=10.0.0.1,actions=output:3
table=1,tun_id=2,dl_dst=02:00:00:00:00:21,actions=output:3
table=1,tun_id=1,arp,arp_tpa=10.0.0.2,actions=set_field:172.28.128.5->tun_dst,output:1
table=1,tun_id=1,dl_dst=02:00:00:00:00:12,actions=set_field:172.28.128.5->tun_dst,output:1
table=1,tun_id=2,arp,arp_tpa=10.0.0.2,actions=set_field:172.28.128.5->tun_dst,output:1
table=1,tun_id=2,dl_dst=02:00:00:00:00:22,actions=set_field:172.28.128.5->tun_dst,output:1
table=1,actions=drop
VM2:
table=0,in_port=2,actions=set_field:1->tun_id,resubmit(,1)
table=0,in_port=3,actions=set_field:2->tun_id,resubmit(,1)
table=0,actions=resubmit(,1)
table=1,tun_id=1,arp,arp_tpa=10.0.0.2,actions=output:2
table=1,tun_id=1,dl_dst=02:00:00:00:00:12,actions=output:2
table=1,tun_id=2,arp,arp_tpa=10.0.0.2,actions=output:3
table=1,tun_id=2,dl_dst=02:00:00:00:00:22,actions=output:3
table=1,tun_id=1,arp,arp_tpa=10.0.0.1,actions=set_field:172.28.128.4->tun_dst,output:1
table=1,tun_id=1,dl_dst=02:00:00:00:00:11,actions=set_field:172.28.128.4->tun_dst,output:1
table=1,tun_id=2,arp,arp_tpa=10.0.0.1,actions=set_field:172.28.128.4->tun_dst,output:1
table=1,tun_id=2,dl_dst=02:00:00:00:00:21,actions=set_field:172.28.128.4->tun_dst,output:1
table=1,actions=drop
1 is the openflow port associated with the interface vtep on both VMs.
On VM1 2 and 3 are the openflow ports associated with interfaces kos11 and kos21 respectively.
On VM2 2 and 3 are the openflow ports associated with interfaces kos12 and kos22 respectively.
1 is the VNI of the VXLAN kos11 and kos12 belong to, 2 is the VNI of the VXLAN kos21 and kos22 belong to.
172.28.128.4 and 172.28.128.5 are the IP addresses of the network interfaces through which the two VMs can ping each other (AKA the IPs of the tunnel endpoints).
After executing the aforementioned steps, kos11 can ping kos12 and vice versa (aka tunneling for the VXLAN with VNI 1 works), whereas pinging kos22 from kos21 or vice versa yields destination host unreachable. I ran tcpdump on kos22 while pinging it from kos21 and this is what I got:
ARP, Request who-has 10.0.0.2 tell 10.0.0.1, length 28
ARP, Request who-has 10.0.0.2 tell 10.0.0.1, length 28
ARP, Request who-has 10.0.0.2 tell 10.0.0.1, length 28
What am I doing wrong? How do I fix this?
The reason it was not working is that you cannot have multiple interfaces in the same subnet. The workaround I found is to put each interface in a separate Linux network namespace.