{"id":3140,"date":"2020-12-05T09:44:02","date_gmt":"2020-12-05T08:44:02","guid":{"rendered":"https:\/\/kudzia.eu\/b\/?p=3140"},"modified":"2022-01-26T08:58:53","modified_gmt":"2022-01-26T07:58:53","slug":"black-hole-somewhere-in-the-internet-swallowing-udp-packets","status":"publish","type":"post","link":"https:\/\/kudzia.eu\/b\/2020\/12\/black-hole-somewhere-in-the-internet-swallowing-udp-packets\/","title":{"rendered":"black hole, somewhere in the internet, swallowing UDP packets"},"content":{"rendered":"<p>UDP packets sent from specific source port, with public source IP address do not reach specific destination port of the public destination IP address. changing any of the parameters [ usually source port ] &#8211; fixes the issue.<\/p>\n<p>i&#8217;ve observed this phenomenon multiple times for long-running OpenVPN and Wireguard VPNs encapsulating encrypted traffic in UDP packets. i cannot put my finger on the actual case of it besides saying it&#8217;s not in our infrastructure, it&#8217;s somewhere in ISPs networks. i know it&#8217;s not on &#8216;my&#8217; side because i&#8217;ve sniffed packets on ISP-facing interfaces on both ends of the VPN tunnel. it does not sound like typical tcam corruption in a router &#8211; because that would affect only pair of source&lt;&gt;destination IP addresses. also &#8211; the &#8216;black hole&#8217; i&#8217;ve observed many times i always unidirectional.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-3146\" src=\"https:\/\/kudzia.eu\/b\/wp-content\/uploads\/2020\/12\/black-hole-UDP.png\" alt=\"\" width=\"710\" height=\"785\" srcset=\"https:\/\/kudzia.eu\/b\/wp-content\/uploads\/2020\/12\/black-hole-UDP.png 710w, https:\/\/kudzia.eu\/b\/wp-content\/uploads\/2020\/12\/black-hole-UDP-271x300.png 271w\" sizes=\"auto, (max-width: 710px) 100vw, 710px\" \/><\/p>\n<p>i&#8217;ve seen it happening for mix of different connection types [ different datacenters, different business and home ISPs ], in different countries. i&#8217;ve accepted it as a fact of life that needs to be work-around&#8217;ed rather than solved. since changing of source port is usually simplest and sufficient &#8211; i&#8217;m letting both OpenVPN and Wireguard to randomize that parameter and i&#8217;m just adding watchdogs wherever i have that type of tunnel. watchdog is just on the &#8216;remote&#8217; end &#8211; at machine &#8216;dialing up&#8217; to the VPN server.<\/p>\n<p>\/etc\/cron.d\/openvpnwatchdog has as many lines as VPN tunnels:<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">\r\n*\/5 * * * * root \/usr\/local\/bin\/openvpnTunnelWatchdog.sh 169.254.1.15 namveOfVPN0 &gt; \/dev\/null 2&gt;&amp;1\r\n*\/5 * * * * root \/usr\/local\/bin\/openvpnTunnelWatchdog.sh 169.254.1.71 namveOfVPN1 &gt; \/dev\/null 2&gt;&amp;1\r\n<\/pre>\n<p>1st argument &#8211; tells what IP is assigned to the remote end of the encrypted tunnel,<\/p>\n<p>where nameOfVPN1 is taken from the output of this command:<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">\r\nroot@vpnclient:~# systemctl list-units --type=service|grep openvpn\r\nopenvpn.service                                                                           loaded active exited  OpenVPN service\r\nopenvpn@namveOfVPN0.service                                                               loaded active running OpenVPN connection to implantis.vpn.kudzia.eu\r\nopenvpn@namveOfVPN1.service                                                               loaded active running OpenVPN connection to implantis.vpn.kudzia.eu\r\n<\/pre>\n<p>the \/usr\/local\/bin\/openvpnTunnelWatchdog.sh script itself:<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">\r\n#!\/bin\/bash\r\nif &#x5B; $# -lt 2 ] ; then\r\n        echo syntax:\r\n        echo &quot;openvpnTunnelWatchdog.sh ip tunel_name &#x5B; eg. namveOfVPN1 ]&quot;\r\n        exit 1\r\nfi\r\n\r\nip=$1\r\nservice=$2\r\n\r\n# &quot;( flock .. ; .. ) 200&gt; &quot; prevents piling up of multiple script instances running in parallel in case there's problem with service restart \/ pinging\r\n# based on http:\/\/stackoverflow.com\/questions\/7057234\/\r\n(\r\nflock -n -e 200 || exit 1\r\n\r\necho $ip\r\necho $service\r\n\r\nping -i 0.2 $ip -c 5\r\nif &#x5B; $? -eq 0 ]; then\r\n        echo &quot;everything is ok&quot;\r\nelse\r\n        echo &quot;restarting vpn: &quot; + $service\r\n        systemctl restart openvpn@$service.service\r\n\r\n        if &#x5B; $? -ne 0 ]; then\r\n                echo &quot;problems with restarting tunnel: &quot; + $service\r\n        fi\r\nfi\r\n) 200 &gt;\/tmp\/openvpnTunnelWatchdog-$service\r\n<\/pre>\n<p>equivalent for wireguard &#8211; \/etc\/cron.d\/wireguardwatchdog<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">\r\n*\/5 * * * * root \/usr\/local\/bin\/wireguardWatchdog.sh 2001:470:614d:ffff::1 \/etc\/wireguard\/wg1.conf &gt; \/tmp\/x 2&gt;&amp;1\r\n*\/5 * * * * root \/usr\/local\/bin\/wireguardWatchdog.sh 192.168.2.1 \/etc\/wireguard\/wg0.conf &gt; \/dev\/null 2&gt;&amp;1\r\n<\/pre>\n<p>\/usr\/local\/bin\/wireguardWatchdog.sh:<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">\r\n#!\/bin\/bash\r\n\r\nif &#x5B; $# -lt 2 ] ; then\r\n        echo syntax:\r\n        echo &quot;wireguardWatchdog.sh ipToPing \/etc\/wireguard\/wgX.conf&quot;\r\n        exit 1\r\nfi\r\n\r\nip=$1\r\nconfig=$2\r\n\r\n# &quot;( flock .. ; .. ) 200&gt; &quot; prevents piling up of multiple script instances running in parallel in case there's problem with service restart \/ pinging\r\n# based on http:\/\/stackoverflow.com\/questions\/7057234\/\r\n(\r\nflock -n -e 200 || exit 1\r\n\r\necho $ip\r\necho $service\r\n\r\nping -i 0.2 $ip -c 5\r\nif &#x5B; $? -eq 0 ]; then\r\n        echo &quot;everything is ok&quot;\r\nelse\r\n        echo &quot;restarting vpn: &quot; + $service\r\n        wg-quick down $config\r\n        wg-quick up $config\r\nfi\r\n) 200&gt;\/tmp\/wg-$ip\r\n<\/pre>\n","protected":false},"excerpt":{"rendered":"<p>UDP packets sent from specific source port, with public source IP address do not reach specific destination port of the public destination IP address. changing any of the parameters [ usually source port ] &#8211; fixes the issue. i&#8217;ve observed this phenomenon multiple times for long-running OpenVPN and Wireguard VPNs encapsulating encrypted traffic in UDP [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[51],"tags":[47,111,89,110],"class_list":["post-3140","post","type-post","status-publish","format-standard","hentry","category-unimportant","tag-linux-networking","tag-mysteries","tag-openvpn","tag-wireguard"],"_links":{"self":[{"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/posts\/3140","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/comments?post=3140"}],"version-history":[{"count":9,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/posts\/3140\/revisions"}],"predecessor-version":[{"id":3303,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/posts\/3140\/revisions\/3303"}],"wp:attachment":[{"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/media?parent=3140"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/categories?post=3140"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/tags?post=3140"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}