bridging lan segments across untrusted links

we’ve run out of the office space in one of the locations. in short term it was not possible to find a suitable and large enough place to rent so we had to split and relocate some of the staff to another building few kilometers away. it’s possible that we’ll shuffle people and servers between the two locations with a short notice so i decided to extend the L2 segment between both offices rather than set up separate IP subnets / use different ISPs. below an outline of how with help of OpenVPN, linux bridging and two fibers leased from different operators we’ve set up redundant, encrypted connection transporting multiple vlans.

servers:

  • st0 – a VM, could be as well a physical server, located in the first office. a single point of failure. eventually should be made into two vms on two physical servers; a termination point of both encrypted tunnels. eth0 connects it to vlan 10, eth1 – vlan11, eth5 – to leased line 1, eth6 – to leased line 2
  • st1, st2 – physical servers connected via two independent leased fibers to tun0.

vlan 10, vlan 11 – two vlans that are ‘transported’ over the encrypted connection. in reality i have more of those.

packages that had to be installed on all of the servers:

apt-get install openvpn vlan bridge-utils

st0

/etc/network/interfaces on st0, notice that on br0 – connected to vlan 10 – i’m adding an IP address used for management of this machine:

auto lo
iface lo inet loopback

# vmware's interface connecting to vlan 10
auto eth0
iface eth0 inet manual

# vmware's interface connecting to vlan 11
auto eth1
iface eth1 inet manual

# connecting to leased line 1
auto eth5
iface eth5 inet static
    address 192.168.100.1/29

# connecting to leased line 2
auto eth6
iface eth6 inet static
    address 192.168.100.9/29

# bridge handling vlan 10 + ip address for management
auto br0
iface br0 inet static
    address 10.0.0.15/22
    gateway 10.0.0.1
    bridge_ports tap0 tap10 eth0
    pre-up openvpn --mktun --dev tap0 ; openvpn --mktun --dev tap10 
    post-down openvpn --rmtun --dev tap0 ; openvpn --rmtun --dev tap10 

# bridge handling vlan 11
auto br1
iface br1 inet manual
    bridge_ports tap1 tap11 eth1
    pre-up openvpn --mktun --dev tap1 ; openvpn --mktun --dev tap11 
    post-down openvpn --rmtun --dev tap1 ; openvpn --rmtun --dev tap11

/etc/openvpn/line1_vlan10.conf:

dev tap0
secret static.key
port 1830
cipher aes-256-cbc

ping 1
ping-restart 20

ping-timer-rem
persist-tun
persist-key
daemon line1_vlan10

/etc/openvpn/line1_vlan11.conf:

dev tap1
secret static.key
port 1831
cipher aes-256-cbc

ping 1
ping-restart 20

ping-timer-rem
persist-tun
persist-key
daemon line1_vlan11

/etc/openvpn/line2_vlan10.conf:

dev tap10
secret static.key
port 1840
cipher aes-256-cbc

ping 1
ping-restart 20

ping-timer-rem
persist-tun
persist-key
daemon line2_vlan10

/etc/openvpn/line2_vlan11.conf:

dev tap11
secret static.key
port 1841
cipher aes-256-cbc

ping 1
ping-restart 20

ping-timer-rem
persist-tun
persist-key
daemon line2_vlan11

the static.key – which needs to be identical on the openvpn client/server – was generated with:

openvpn --genkey --secret /etc/openvpn/static.key

for improved security use own ca rather than static key.

vmware esxi 6.5, in its default configuration, would not pass to st0 ethernet frames that are not directed to st0’s MAC address. to change it i had to reconfigure the underlying vswitch and and port groups to allow promiscuous mode, mac address change, forged transmits.

st1

both st1 and st2 have similar physical setup – eth0 is connected directly to a leased line, eth1 – with trunked vlans – to the same switch.

/etc/network/interfaces:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
    address 192.168.100.2/29

auto eth1
iface eth1 inet manual

auto eth1.10
iface eth1.10 inet manual

auto eth1.11
iface eth1.11 inet manual

# ip address for management
auto br0
iface br0 inet static
    address 10.0.0.16/22
    gateway 10.0.0.1
    bridge_ports tap10 eth1.10
    pre-up openvpn --mktun --dev tap10
    post-down openvpn --rmtun --dev tap10

auto br1
iface br1 inet manual
    bridge_ports tap11 eth1.11
    pre-up openvpn --mktun --dev tap11
    post-down openvpn --rmtun --dev tap11

/etc/openvpn/vlan10.conf:

remote 192.168.100.1 1830
dev tap10
secret static.key
cipher aes-256-cbc

port 1830

ping 1
ping-restart 20

ping-timer-rem
persist-tun
persist-key
daemon vlan10

/etc/openvpn/vlan11.conf:

remote 192.168.100.1 1831
dev tap11
secret static.key
cipher aes-256-cbc

port 1831

ping 1
ping-restart 20

ping-timer-rem
persist-tun
persist-key
daemon vlan11

st2

since st1, st2 are connected with their trunk interfaces to the same switch – if on st2 i would use analogous configuration to one from st1 – i’d end up with a packet forwarding loop. to prevent it we’re using spanning tree protocol. it could be handled by a switch, but for better visibility it’s done on st2. i prefer that leased line 2 is used only when line 1 is down – hence choice of the port weights

/etc/network/interfaces:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
    address 192.168.100.10/29

auto eth1
iface eth1 inet manual

auto eth1.10
iface eth1.10 inet manual

auto eth1.11
iface eth1.11 inet manual

# ip address for management
auto br0
iface br0 inet static
    bridge_stp on
    bridge_pathcost eth1.10 10
    bridge_pathcost tap10 1
    bridge_hello 1
    bridge_maxage 6
    bridge_maxwait 1
    bridge_fd 2
    address 10.0.0.17/22
    gateway 10.0.0.1
    bridge_ports tap10 eth1.10
    pre-up openvpn --mktun --dev tap10
    post-down openvpn --rmtun --dev tap10

auto br1
iface br1 inet manual
    bridge_stp on
    bridge_pathcost eth1.11 10
    bridge_pathcost tap11 1
    bridge_hello 1
    bridge_maxage 6
    bridge_maxwait 1
    bridge_fd 2
    bridge_ports tap11 eth1.11
    pre-up openvpn --mktun --dev tap11
    post-down openvpn --rmtun --dev tap11

to minimize the time it takes to transition from blocking state on eth1.10, eth1.11 – which is expected behavior whenever leased line 1 works properly – to forwarding state whenever line 1 or st1 is down – i’ve played a bit with the bridge_* parameters. looks like the minimum allowed value for bridge_maxage is 6 seconds, although it’s not clearly documented.

openvpn setup is nearly identical to st1:

/etc/openvpn/vlan10.conf:

remote 192.168.100.9 1840
dev tap10
secret static.key
cipher aes-256-cbc

port 1840

ping 1
ping-restart 20

ping-timer-rem
persist-tun
persist-key
daemon vlan10

/etc/openvpn/vlan11.conf:

remote 192.168.100.9 1841
dev tap11
secret static.key
cipher aes-256-cbc

port 1841

ping 1
ping-restart 20

ping-timer-rem
persist-tun
persist-key
daemon vlan11

useful commands
monitoring state of a bridge

watch -n1 -d "brctl showstp br0"

making systemd aware of new vpn tunnels so they are started during the next reboot

systemctl daemon-reload

Leave a Reply

Your email address will not be published. Required fields are marked *

 

(Spamcheck Enabled)