we’ve run out of the office space in one of the locations. in short term it was not possible to find a suitable and large enough place to rent so we had to split and relocate some of the staff to another building few kilometers away. it’s possible that we’ll shuffle people and servers between the two locations with a short notice so i decided to extend the L2 segment between both offices rather than set up separate IP subnets / use different ISPs. below an outline of how with help of OpenVPN, linux bridging and two fibers leased from different operators we’ve set up redundant, encrypted connection transporting multiple vlans.
servers:
- st0 – a VM, could be as well a physical server, located in the first office. a single point of failure. eventually should be made into two vms on two physical servers; a termination point of both encrypted tunnels. eth0 connects it to vlan 10, eth1 – vlan11, eth5 – to leased line 1, eth6 – to leased line 2
- st1, st2 – physical servers connected via two independent leased fibers to tun0.
vlan 10, vlan 11 – two vlans that are ‘transported’ over the encrypted connection. in reality i have more of those.
packages that had to be installed on all of the servers:
apt-get install openvpn vlan bridge-utils
st0
/etc/network/interfaces on st0, notice that on br0 – connected to vlan 10 – i’m adding an IP address used for management of this machine:
auto lo
iface lo inet loopback
# vmware's interface connecting to vlan 10
auto eth0
iface eth0 inet manual
# vmware's interface connecting to vlan 11
auto eth1
iface eth1 inet manual
# connecting to leased line 1
auto eth5
iface eth5 inet static
address 192.168.100.1/29
# connecting to leased line 2
auto eth6
iface eth6 inet static
address 192.168.100.9/29
# bridge handling vlan 10 + ip address for management
auto br0
iface br0 inet static
address 10.0.0.15/22
gateway 10.0.0.1
bridge_ports tap0 tap10 eth0
pre-up openvpn --mktun --dev tap0 ; openvpn --mktun --dev tap10
post-down openvpn --rmtun --dev tap0 ; openvpn --rmtun --dev tap10
# bridge handling vlan 11
auto br1
iface br1 inet manual
bridge_ports tap1 tap11 eth1
pre-up openvpn --mktun --dev tap1 ; openvpn --mktun --dev tap11
post-down openvpn --rmtun --dev tap1 ; openvpn --rmtun --dev tap11
/etc/openvpn/line1_vlan10.conf:
dev tap0 secret static.key port 1830 cipher aes-256-cbc ping 1 ping-restart 20 ping-timer-rem persist-tun persist-key daemon line1_vlan10
/etc/openvpn/line1_vlan11.conf:
dev tap1 secret static.key port 1831 cipher aes-256-cbc ping 1 ping-restart 20 ping-timer-rem persist-tun persist-key daemon line1_vlan11
/etc/openvpn/line2_vlan10.conf:
dev tap10 secret static.key port 1840 cipher aes-256-cbc ping 1 ping-restart 20 ping-timer-rem persist-tun persist-key daemon line2_vlan10
/etc/openvpn/line2_vlan11.conf:
dev tap11 secret static.key port 1841 cipher aes-256-cbc ping 1 ping-restart 20 ping-timer-rem persist-tun persist-key daemon line2_vlan11
the static.key – which needs to be identical on the openvpn client/server – was generated with:
openvpn --genkey --secret /etc/openvpn/static.key
for improved security use own ca rather than static key.
vmware esxi 6.5, in its default configuration, would not pass to st0 ethernet frames that are not directed to st0’s MAC address. to change it i had to reconfigure the underlying vswitch and and port groups to allow promiscuous mode, mac address change, forged transmits.
st1
both st1 and st2 have similar physical setup – eth0 is connected directly to a leased line, eth1 – with trunked vlans – to the same switch.
/etc/network/interfaces:
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet static
address 192.168.100.2/29
auto eth1
iface eth1 inet manual
auto eth1.10
iface eth1.10 inet manual
auto eth1.11
iface eth1.11 inet manual
# ip address for management
auto br0
iface br0 inet static
address 10.0.0.16/22
gateway 10.0.0.1
bridge_ports tap10 eth1.10
pre-up openvpn --mktun --dev tap10
post-down openvpn --rmtun --dev tap10
auto br1
iface br1 inet manual
bridge_ports tap11 eth1.11
pre-up openvpn --mktun --dev tap11
post-down openvpn --rmtun --dev tap11
/etc/openvpn/vlan10.conf:
remote 192.168.100.1 1830 dev tap10 secret static.key cipher aes-256-cbc port 1830 ping 1 ping-restart 20 ping-timer-rem persist-tun persist-key daemon vlan10
/etc/openvpn/vlan11.conf:
remote 192.168.100.1 1831 dev tap11 secret static.key cipher aes-256-cbc port 1831 ping 1 ping-restart 20 ping-timer-rem persist-tun persist-key daemon vlan11
st2
since st1, st2 are connected with their trunk interfaces to the same switch – if on st2 i would use analogous configuration to one from st1 – i’d end up with a packet forwarding loop. to prevent it we’re using spanning tree protocol. it could be handled by a switch, but for better visibility it’s done on st2. i prefer that leased line 2 is used only when line 1 is down – hence choice of the port weights
/etc/network/interfaces:
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet static
address 192.168.100.10/29
auto eth1
iface eth1 inet manual
auto eth1.10
iface eth1.10 inet manual
auto eth1.11
iface eth1.11 inet manual
# ip address for management
auto br0
iface br0 inet static
bridge_stp on
bridge_pathcost eth1.10 10
bridge_pathcost tap10 1
bridge_hello 1
bridge_maxage 6
bridge_maxwait 1
bridge_fd 2
address 10.0.0.17/22
gateway 10.0.0.1
bridge_ports tap10 eth1.10
pre-up openvpn --mktun --dev tap10
post-down openvpn --rmtun --dev tap10
auto br1
iface br1 inet manual
bridge_stp on
bridge_pathcost eth1.11 10
bridge_pathcost tap11 1
bridge_hello 1
bridge_maxage 6
bridge_maxwait 1
bridge_fd 2
bridge_ports tap11 eth1.11
pre-up openvpn --mktun --dev tap11
post-down openvpn --rmtun --dev tap11
to minimize the time it takes to transition from blocking state on eth1.10, eth1.11 – which is expected behavior whenever leased line 1 works properly – to forwarding state whenever line 1 or st1 is down – i’ve played a bit with the bridge_* parameters. looks like the minimum allowed value for bridge_maxage is 6 seconds, although it’s not clearly documented.
openvpn setup is nearly identical to st1:
/etc/openvpn/vlan10.conf:
remote 192.168.100.9 1840 dev tap10 secret static.key cipher aes-256-cbc port 1840 ping 1 ping-restart 20 ping-timer-rem persist-tun persist-key daemon vlan10
/etc/openvpn/vlan11.conf:
remote 192.168.100.9 1841 dev tap11 secret static.key cipher aes-256-cbc port 1841 ping 1 ping-restart 20 ping-timer-rem persist-tun persist-key daemon vlan11
monitoring state of a bridge
watch -n1 -d "brctl showstp br0"
making systemd aware of new vpn tunnels so they are started during the next reboot
systemctl daemon-reload