black hole, somewhere in the internet, swallowing UDP packets

UDP packets sent from specific source port, with public source IP address do not reach specific destination port of the public destination IP address. changing any of the parameters [ usually source port ] – fixes the issue.

i’ve observed this phenomenon multiple times for long-running OpenVPN and Wireguard VPNs encapsulating encrypted traffic in UDP packets. i cannot put my finger on the actual case of it besides saying it’s not in our infrastructure, it’s somewhere in ISPs networks. i know it’s not on ‘my’ side because i’ve sniffed packets on ISP-facing interfaces on both ends of the VPN tunnel. it does not sound like typical tcam corruption in a router – because that would affect only pair of source<>destination IP addresses. also – the ‘black hole’ i’ve observed many times i always unidirectional.

i’ve seen it happening for mix of different connection types [ different datacenters, different business and home ISPs ], in different countries. i’ve accepted it as a fact of life that needs to be work-around’ed rather than solved. since changing of source port is usually simplest and sufficient – i’m letting both OpenVPN and Wireguard to randomize that parameter and i’m just adding watchdogs wherever i have that type of tunnel. watchdog is just on the ‘remote’ end – at machine ‘dialing up’ to the VPN server.

/etc/cron.d/openvpnwatchdog has as many lines as VPN tunnels:

*/5 * * * * root /usr/local/bin/openvpnTunnelWatchdog.sh 169.254.1.15 namveOfVPN0 > /dev/null 2>&1
*/5 * * * * root /usr/local/bin/openvpnTunnelWatchdog.sh 169.254.1.71 namveOfVPN1 > /dev/null 2>&1

1st argument – tells what IP is assigned to the remote end of the encrypted tunnel,

where nameOfVPN1 is taken from the output of this command:

root@vpnclient:~# systemctl list-units --type=service|grep openvpn
openvpn.service                                                                           loaded active exited  OpenVPN service
openvpn@namveOfVPN0.service                                                               loaded active running OpenVPN connection to implantis.vpn.kudzia.eu
openvpn@namveOfVPN1.service                                                               loaded active running OpenVPN connection to implantis.vpn.kudzia.eu

the /usr/local/bin/openvpnTunnelWatchdog.sh script itself:

#!/bin/bash
if [ $# -lt 2 ] ; then
        echo syntax:
        echo "openvpnTunnelWatchdog.sh ip tunel_name [ eg. namveOfVPN1 ]"
        exit 1
fi

ip=$1
service=$2

# "( flock .. ; .. ) 200> " prevents piling up of multiple script instances running in parallel in case there's problem with service restart / pinging
# based on http://stackoverflow.com/questions/7057234/
(
flock -n -e 200 || exit 1

echo $ip
echo $service

ping -i 0.2 $ip -c 5
if [ $? -eq 0 ]; then
        echo "everything is ok"
else
        echo "restarting vpn: " + $service
        systemctl restart openvpn@$service.service

        if [ $? -ne 0 ]; then
                echo "problems with restarting tunnel: " + $service
        fi
fi
) 200 >/tmp/openvpnTunnelWatchdog-$service

equivalent for wireguard – /etc/cron.d/wireguardwatchdog

*/5 * * * * root /usr/local/bin/wireguardWatchdog.sh 2001:470:614d:ffff::1 /etc/wireguard/wg1.conf > /tmp/x 2>&1
*/5 * * * * root /usr/local/bin/wireguardWatchdog.sh 192.168.2.1 /etc/wireguard/wg0.conf > /dev/null 2>&1

/usr/local/bin/wireguardWatchdog.sh:

#!/bin/bash

if [ $# -lt 2 ] ; then
        echo syntax:
        echo "wireguardWatchdog.sh ipToPing /etc/wireguard/wgX.conf"
        exit 1
fi

ip=$1
config=$2

# "( flock .. ; .. ) 200> " prevents piling up of multiple script instances running in parallel in case there's problem with service restart / pinging
# based on http://stackoverflow.com/questions/7057234/
(
flock -n -e 200 || exit 1

echo $ip
echo $service

ping -i 0.2 $ip -c 5
if [ $? -eq 0 ]; then
        echo "everything is ok"
else
        echo "restarting vpn: " + $service
        wg-quick down $config
        wg-quick up $config
fi
) 200>/tmp/wg-$ip