Search code examples
linuxnetwork-programminggatewayxenbridge

Bridged Xen domU with gateway in different subnet


I have a Xen dom0 running Debian Wheezy (7.8) and Xen 4.1, set up with bridged networking.

  • 199.XXX.161.64 is the dom0 gateway.
  • 199.XXX.161.65 is the dom0 address.
  • 192.XXX.13.128/28 is the subnet for the domU's.

Configuration dom0:

root@dom0:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet manual

auto xenbr0
iface xenbr0 inet static
  address 199.XXX.161.65
  netmask 255.255.255.254
  network 199.XXX.161.64
  broadcast 199.XXX.161.65
  gateway 199.XXX.161.64
  dns-nameservers 199.XXX.162.41 199.XXX.162.141
  bridge_ports eth0
  bridge_stp off      # disable Spanning Tree Protocol
  bridge_fd 0         # no forwarding delay
  bridge_maxwait 0    # no delay before a port becomes available

allow-hotplug xenbr0  # start interface on hotplug event


root@dom0:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr0 state UP qlen 1000
    link/ether 00:25:90:d5:06:1a brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 00:25:90:d5:06:1b brd ff:ff:ff:ff:ff:ff
4: xenbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    link/ether 00:25:90:d5:06:1a brd ff:ff:ff:ff:ff:ff
    inet 199.XXX.161.65/31 brd 199.XXX.161.65 scope global xenbr0
    inet6 fe80::XXXX:90ff:fed5:61a/64 scope link 
       valid_lft forever preferred_lft forever
8: vif1.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master xenbr0 state UP qlen 32
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fcff:ffff:feff:ffff/64 scope link 
       valid_lft forever preferred_lft forever


root@dom0:~# brctl show
bridge name bridge id         STP enabled interfaces
xenbr0      8000.002590d5061a no          eth0
                                          vif1.0


root@dom0:~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         199.XXX.161.64  0.0.0.0         UG    0      0        0 xenbr0
192.XXX.13.128  0.0.0.0         255.255.255.240 U     0      0        0 xenbr0
199.XXX.161.64  0.0.0.0         255.255.255.254 U     0      0        0 xenbr0

root@dom0:~# iptables -L -n
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-out vif1.0 --physdev-is-bridged
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-in vif1.0 --physdev-is-bridged udp spt:68 dpt:67
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-out vif1.0 --physdev-is-bridged
ACCEPT     all  --  192.XXX.13.129       0.0.0.0/0            PHYSDEV match --physdev-in vif1.0 --physdev-is-bridged

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

This host can reach its gateway and thus the internet.

root@dom0:~# ping -c 1 199.XXX.161.64
PING 199.XXX.161.64 (199.XXX.161.64) 56(84) bytes of data.
64 bytes from 199.XXX.161.64: icmp_req=1 ttl=64 time=0.459 ms

--- 199.XXX.161.64 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.459/0.459/0.459/0.000 ms

I also have a domU (with the same OS) which needs a primary IP address in a different subnet. There is no gateway on the network in this subnet. I want to keep my network setup bridged (no dom0 routing or NAT) so I added the dom0 gateway as the gateway for the domU as described in this blogpost.

Configuration domU:

root@domU:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:16:3e:b7:7e:cc brd ff:ff:ff:ff:ff:ff
    inet 192.XXX.13.129/28 brd 192.XXX.13.143 scope global eth0
    inet6 fe80::XXXX:3eff:feb7:7ecc/64 scope link 
       valid_lft forever preferred_lft forever


root@domU:~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         199.XXX.161.64  0.0.0.0         UG    0      0        0 eth0
192.XXX.13.128  0.0.0.0         255.255.255.240 U     0      0        0 eth0
199.XXX.161.64  0.0.0.0         255.255.255.255 UH    0      0        0 eth0

With this configuration the domU still has no network access. To test if the bridge was working I manually added a route to dom0.

root@domU:~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         199.XXX.161.64  0.0.0.0         UG    0      0        0 eth0
192.XXX.13.128  0.0.0.0         255.255.255.240 U     0      0        0 eth0
199.XXX.161.64  0.0.0.0         255.255.255.255 UH    0      0        0 eth0
199.XXX.161.65  0.0.0.0         255.255.255.255 UH    0      0        0 eth0

Now the dom0 and domU can communicate through the bridge.

root@domU:~# ping -c 1 199.XXX.161.65
PING 199.XXX.161.65 (199.XXX.161.65) 56(84) bytes of data.
64 bytes from 199.XXX.161.65: icmp_req=1 ttl=64 time=0.037 ms

--- 199.XXX.161.65 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.037/0.037/0.037/0.000 ms

root@dom0:~# ping -c 1 192.XXX.13.129
PING 192.184.13.129 (192.XXX.13.129) 56(84) bytes of data.
64 bytes from 192.XXX.13.129: icmp_req=1 ttl=64 time=0.100 ms

--- 192.XXX.13.129 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.100/0.100/0.100/0.000 ms

However, the domU still can't reach the gateway.

root@domU:~# ping -c 1 199.XXX.161.64
PING 199.XXX.161.64 (199.XXX.161.64) 56(84) bytes of data.
From 192.XXX.13.129 icmp_seq=1 Destination Host Unreachable

--- 199.XXX.161.64 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

I have attempted to log establish if the traffic is actually being sent through the bridge by inserting a -j LOG rule at the top of the INPUT, OUTPUT and FORWARD iptables chains. When the domU attempts to ping the gateway the dom0 does not log a single packet. I also tried manually adding an entry for the gateway in domU's ARP table but the results were the same. The domU can't reach the gateway and thus has no network access other than being able to communicate with dom0 via static route.


Solution

  • So if I am understanding this correctly, the following is the network configuration for your DomU:

    • 192.XXX.13.129/28 - DomU IP Address
    • 199.XXX.161.64 - DomU GW Address

    The problem is that your DomU does not have a route (layer 3) to allow it to talk to the GW address since the GW address is in a different subnet. So even though the router is on the same layer 2 network, the router (if it is processing your packets) does not know about your layer 3 network and is sending it's responses to it's default gateway.

    That you are able to ping the Dom0 from the DomU is odd and probably the result of both the Dom0 and the DomU using the same Linux Bridge (which it not a true ethernet switch, more like a dumb hub).

    The simple fix is to add an address from your DomU network to the LAN interface on your router.

    The better fix would be to use VLANs to segment the different networks via layer 2 and replace Linux Bridges with Open vSwitch. This would completely isolate the Dom0 and DomU traffic so that they would be required to communication via a router and possibly a firewall.