we’re using LXC containers to host multiple workloads on the same physical servers. e.g. few instances of database servers running side-by-side.
once in a while we end up with strange situation where tcp connections between containers running on the same physical server get torn down abruptly. in our case – this manifested e.g. by those errors: Caused by: java.net.SocketException: Connection reset or The last packet successfully received from the server was 415 milliseconds ago. The last packet sent successfully to the server was 0 milliseconds ago.com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure.
it seems there’s a random issue where container’s network stack is not properly cleaned after container reboot or shutdown. mentions of similar cases: 1, 2.
in result we have multiple IP stacks having the same IP and the same MAC address. this manifests in 2 ARP responses sent to a single ARP who-has reply:
root@someserver:~# arping -c 1 somelxccontainer ARPING 172.16.1.21 42 bytes from 00:ff:1a:1e:ba:21 (172.16.1.21): index=0 time=7.881 usec 42 bytes from 00:ff:1a:1e:ba:21 (172.16.1.21): index=1 time=22.879 usec --- 172.16.17.21 statistics --- 1 packets transmitted, 2 packets received, 0% unanswered (1 extra) rtt min/avg/max/std-dev = 0.008/0.015/0.023/0.007 ms
once diagnosed remediation is simple: either reboot the whole lxc server or identify and delete no-longer-needed vethXXXXX interfaces.
brctl show
bridge name bridge id STP enabled interfaces
br0 8000.fe0023c26675 no veth1KCRWX
veth7G72IS
vethBH3Q2G
vethBMW0L7
vethEUGFKO
vethGRH1MH
vethJ7RPXB
vethJGTBAH
vethPE6HWW
vethPUKEU5
vethTCVY3G
vethVH9TQH
how to find out which interface is orphaned?
- within each container run cat /sys/class/net/eth0/iflink and write down the numerical interface id
- determine which veth are not referred from any of the containers – by running /sys/class/net/vethXXXXX/ifindex on the server hosting lxc guests for each of veths: for if in $(ls -1 /sys/devices/virtual/net/br0/brif); do echo -n “$if “; cat /sys/class/net/$if/ifindex ; done
- cross check both lists and find which veth does not belong to any lxc guest. then delete the no-longer needed interface by ip link del vethXXXXXX
how to avoid it? hopefully by explicitly naming network interfaces for guests created on the host – in /var/lib/lxc/guest/config:
lxc.net.1.veth.pair = veth0-nameOfOurGuest
resources that were helpful:
- https://superuser.com/a/1183520/1674
- https://lxc-users.linuxcontainers.narkive.com/EaQUUpe8/determine-which-veth-interface-belongs-to-which-container
- https://superuser.com/questions/1183454/finding-out-the-veth-interface-of-a-docker-container
- https://www.mail-archive.com/lxc-users@lists.sourceforge.net/msg05316.html
- https://serverfault.com/questions/765789/where-are-network-namespaces-in-lxc-lxd