{"id":2868,"date":"2018-05-27T10:03:42","date_gmt":"2018-05-27T09:03:42","guid":{"rendered":"https:\/\/kudzia.eu\/b\/?p=2868"},"modified":"2018-05-27T19:23:28","modified_gmt":"2018-05-27T18:23:28","slug":"bridging-lan-segments-across-untrusted-links","status":"publish","type":"post","link":"https:\/\/kudzia.eu\/b\/2018\/05\/bridging-lan-segments-across-untrusted-links\/","title":{"rendered":"bridging lan segments across untrusted links"},"content":{"rendered":"<p>we&#8217;ve run out of the office space in one of the locations. in short term it was not possible to find a suitable and large enough place to rent so we had to split and relocate some of the staff to another building few kilometers away. it&#8217;s possible that we&#8217;ll shuffle people and servers between the two locations with a short notice so i decided to extend the L2 segment between both offices rather than set up separate IP subnets \/ use different ISPs. below an outline of how with help of OpenVPN, linux bridging and two fibers leased from different operators we&#8217;ve set up redundant, encrypted connection transporting multiple vlans.<br \/>\n<!--more--><\/p>\n<p>servers:<\/p>\n<ul>\n<li>st0 &#8211; a VM, could be as well a physical server, located in the first office. a single point of failure. eventually should be made into two vms on two physical servers; a termination point of both encrypted tunnels. eth0 connects it to vlan 10, eth1 &#8211; vlan11, eth5 &#8211; to leased line 1, eth6 &#8211; to leased line 2<\/li>\n<li>st1, st2 &#8211; physical servers connected via two independent leased fibers to tun0.<\/li>\n<\/ul>\n<p>vlan 10, vlan 11 &#8211; two vlans that are &#8216;transported&#8217; over the encrypted connection. in reality i have more of those.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/kudzia.eu\/b\/wp-content\/uploads\/2018\/05\/vlansbridgeDrawing1.png\" alt=\"\" width=\"709\" height=\"451\" class=\"alignnone size-full wp-image-2882\" srcset=\"https:\/\/kudzia.eu\/b\/wp-content\/uploads\/2018\/05\/vlansbridgeDrawing1.png 709w, https:\/\/kudzia.eu\/b\/wp-content\/uploads\/2018\/05\/vlansbridgeDrawing1-300x191.png 300w\" sizes=\"auto, (max-width: 709px) 100vw, 709px\" \/><\/p>\n<p>packages that had to be installed on all of the servers:<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\napt-get install openvpn vlan bridge-utils\r\n<\/pre>\n<h3>st0<\/h3>\n<p>\/etc\/network\/interfaces on st0, notice that on br0 &#8211; connected to vlan 10 &#8211; i&#8217;m adding an IP address used for management of this machine:<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\nauto lo\r\niface lo inet loopback\r\n\r\n# vmware's interface connecting to vlan 10\r\nauto eth0\r\niface eth0 inet manual\r\n\r\n# vmware's interface connecting to vlan 11\r\nauto eth1\r\niface eth1 inet manual\r\n\r\n# connecting to leased line 1\r\nauto eth5\r\niface eth5 inet static\r\n    address 192.168.100.1\/29\r\n\r\n# connecting to leased line 2\r\nauto eth6\r\niface eth6 inet static\r\n    address 192.168.100.9\/29\r\n\r\n# bridge handling vlan 10 + ip address for management\r\nauto br0\r\niface br0 inet static\r\n    address 10.0.0.15\/22\r\n    gateway 10.0.0.1\r\n    bridge_ports tap0 tap10 eth0\r\n    pre-up openvpn --mktun --dev tap0 ; openvpn --mktun --dev tap10 \r\n    post-down openvpn --rmtun --dev tap0 ; openvpn --rmtun --dev tap10 \r\n\r\n# bridge handling vlan 11\r\nauto br1\r\niface br1 inet manual\r\n    bridge_ports tap1 tap11 eth1\r\n    pre-up openvpn --mktun --dev tap1 ; openvpn --mktun --dev tap11 \r\n    post-down openvpn --rmtun --dev tap1 ; openvpn --rmtun --dev tap11\r\n<\/pre>\n<p>\/etc\/openvpn\/line1_vlan10.conf:<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\ndev tap0\r\nsecret static.key\r\nport 1830\r\ncipher aes-256-cbc\r\n\r\nping 1\r\nping-restart 20\r\n\r\nping-timer-rem\r\npersist-tun\r\npersist-key\r\ndaemon line1_vlan10\r\n<\/pre>\n<p>\/etc\/openvpn\/line1_vlan11.conf:<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\ndev tap1\r\nsecret static.key\r\nport 1831\r\ncipher aes-256-cbc\r\n\r\nping 1\r\nping-restart 20\r\n\r\nping-timer-rem\r\npersist-tun\r\npersist-key\r\ndaemon line1_vlan11\r\n<\/pre>\n<p>\/etc\/openvpn\/line2_vlan10.conf:<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\ndev tap10\r\nsecret static.key\r\nport 1840\r\ncipher aes-256-cbc\r\n\r\nping 1\r\nping-restart 20\r\n\r\nping-timer-rem\r\npersist-tun\r\npersist-key\r\ndaemon line2_vlan10\r\n<\/pre>\n<p>\/etc\/openvpn\/line2_vlan11.conf:<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\ndev tap11\r\nsecret static.key\r\nport 1841\r\ncipher aes-256-cbc\r\n\r\nping 1\r\nping-restart 20\r\n\r\nping-timer-rem\r\npersist-tun\r\npersist-key\r\ndaemon line2_vlan11\r\n<\/pre>\n<p>the static.key &#8211; which needs to be identical on the openvpn client\/server &#8211; was generated with:<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\nopenvpn --genkey --secret \/etc\/openvpn\/static.key\r\n<\/pre>\n<p>for improved security use <a href=\"https:\/\/openvpn.net\/index.php\/open-source\/documentation\/howto.html#pki\">own ca<\/a> rather than static key. <\/p>\n<p>vmware esxi 6.5, in its default configuration, would not pass to st0 ethernet frames that are not directed to st0&#8217;s MAC address. to change it i had to reconfigure the underlying vswitch and and port groups to allow <i>promiscuous mode<\/i>, <i>mac address change<\/i>, <i>forged transmits<\/i>.<\/p>\n<h3>st1<\/h3>\n<p>both st1 and st2 have similar physical setup &#8211; eth0 is connected directly to a leased line, eth1 &#8211; with trunked vlans &#8211; to the same switch.<\/p>\n<p>\/etc\/network\/interfaces:<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\nauto lo\r\niface lo inet loopback\r\n\r\nauto eth0\r\niface eth0 inet static\r\n    address 192.168.100.2\/29\r\n\r\nauto eth1\r\niface eth1 inet manual\r\n\r\nauto eth1.10\r\niface eth1.10 inet manual\r\n\r\nauto eth1.11\r\niface eth1.11 inet manual\r\n\r\n# ip address for management\r\nauto br0\r\niface br0 inet static\r\n    address 10.0.0.16\/22\r\n    gateway 10.0.0.1\r\n    bridge_ports tap10 eth1.10\r\n    pre-up openvpn --mktun --dev tap10\r\n    post-down openvpn --rmtun --dev tap10\r\n\r\nauto br1\r\niface br1 inet manual\r\n    bridge_ports tap11 eth1.11\r\n    pre-up openvpn --mktun --dev tap11\r\n    post-down openvpn --rmtun --dev tap11\r\n<\/pre>\n<p>\/etc\/openvpn\/vlan10.conf:<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\nremote 192.168.100.1 1830\r\ndev tap10\r\nsecret static.key\r\ncipher aes-256-cbc\r\n\r\nport 1830\r\n\r\nping 1\r\nping-restart 20\r\n\r\nping-timer-rem\r\npersist-tun\r\npersist-key\r\ndaemon vlan10\r\n<\/pre>\n<p>\/etc\/openvpn\/vlan11.conf:<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\nremote 192.168.100.1 1831\r\ndev tap11\r\nsecret static.key\r\ncipher aes-256-cbc\r\n\r\nport 1831\r\n\r\nping 1\r\nping-restart 20\r\n\r\nping-timer-rem\r\npersist-tun\r\npersist-key\r\ndaemon vlan11\r\n<\/pre>\n<h3>st2<\/h3>\n<p>since st1, st2 are connected with their trunk interfaces to the same switch &#8211; if on st2 i would use analogous configuration to one from st1 &#8211; i&#8217;d end up with a packet forwarding loop. to prevent it we&#8217;re using <a href=\"https:\/\/en.wikipedia.org\/wiki\/Spanning_Tree_Protocol\">spanning tree protocol<\/a>. it could be handled by a switch, but for better visibility it&#8217;s done on st2. i prefer that leased line 2 is used only when line 1 is down &#8211; hence choice of the port weights<\/p>\n<p>\/etc\/network\/interfaces:<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\nauto lo\r\niface lo inet loopback\r\n\r\nauto eth0\r\niface eth0 inet static\r\n    address 192.168.100.10\/29\r\n\r\nauto eth1\r\niface eth1 inet manual\r\n\r\nauto eth1.10\r\niface eth1.10 inet manual\r\n\r\nauto eth1.11\r\niface eth1.11 inet manual\r\n\r\n# ip address for management\r\nauto br0\r\niface br0 inet static\r\n    bridge_stp on\r\n    bridge_pathcost eth1.10 10\r\n    bridge_pathcost tap10 1\r\n    bridge_hello 1\r\n    bridge_maxage 6\r\n    bridge_maxwait 1\r\n    bridge_fd 2\r\n    address 10.0.0.17\/22\r\n    gateway 10.0.0.1\r\n    bridge_ports tap10 eth1.10\r\n    pre-up openvpn --mktun --dev tap10\r\n    post-down openvpn --rmtun --dev tap10\r\n\r\nauto br1\r\niface br1 inet manual\r\n    bridge_stp on\r\n    bridge_pathcost eth1.11 10\r\n    bridge_pathcost tap11 1\r\n    bridge_hello 1\r\n    bridge_maxage 6\r\n    bridge_maxwait 1\r\n    bridge_fd 2\r\n    bridge_ports tap11 eth1.11\r\n    pre-up openvpn --mktun --dev tap11\r\n    post-down openvpn --rmtun --dev tap11\r\n<\/pre>\n<p>to minimize the time it takes to transition from blocking state on eth1.10, eth1.11 &#8211; which is expected behavior whenever leased line 1 works properly &#8211; to forwarding state whenever line 1 or st1 is down &#8211; i&#8217;ve played a bit with the bridge_* parameters. looks like the minimum allowed value for bridge_maxage is 6 seconds, although it&#8217;s not clearly documented.<\/p>\n<p>openvpn setup is nearly identical to st1:<\/p>\n<p>\/etc\/openvpn\/vlan10.conf:<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\nremote 192.168.100.9 1840\r\ndev tap10\r\nsecret static.key\r\ncipher aes-256-cbc\r\n\r\nport 1840\r\n\r\nping 1\r\nping-restart 20\r\n\r\nping-timer-rem\r\npersist-tun\r\npersist-key\r\ndaemon vlan10\r\n<\/pre>\n<p>\/etc\/openvpn\/vlan11.conf:<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\nremote 192.168.100.9 1841\r\ndev tap11\r\nsecret static.key\r\ncipher aes-256-cbc\r\n\r\nport 1841\r\n\r\nping 1\r\nping-restart 20\r\n\r\nping-timer-rem\r\npersist-tun\r\npersist-key\r\ndaemon vlan11\r\n<\/pre>\n<p><b3>useful commands<\/b3><br \/>\nmonitoring state of a bridge<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\nwatch -n1 -d &quot;brctl showstp br0&quot;\r\n<\/pre>\n<p>making systemd aware of new vpn tunnels so they are started during the next reboot<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\nsystemctl daemon-reload\r\n<\/pre>\n","protected":false},"excerpt":{"rendered":"<p>we&#8217;ve run out of the office space in one of the locations. in short term it was not possible to find a suitable and large enough place to rent so we had to split and relocate some of the staff to another building few kilometers away. it&#8217;s possible that we&#8217;ll shuffle people and servers between [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[17],"tags":[47,89],"class_list":["post-2868","post","type-post","status-publish","format-standard","hentry","category-tech","tag-linux-networking","tag-openvpn"],"_links":{"self":[{"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/posts\/2868","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/comments?post=2868"}],"version-history":[{"count":12,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/posts\/2868\/revisions"}],"predecessor-version":[{"id":2887,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/posts\/2868\/revisions\/2887"}],"wp:attachment":[{"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/media?parent=2868"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/categories?post=2868"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/tags?post=2868"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}