Fixing Spontaneous 'Connection Reset by Peer' Errors in Kubernetes kube-proxy
Solution Summary
Kubernetes kube-proxy can experience spontaneous 'Connection Reset by Peer' errors when the Linux conntrack module allows INVALID packets through the KUBE-FORWARD chain. The fix injects an iptables rule to identify and silently drop packets with the INVALID connection state before they reach the TCP stack, preventing forceful termination of established connections.
The Problem
Resolve intermittent TCP connection resets in Kubernetes by configuring iptables to drop invalid conntrack packets. Improve stability for long-lived connections.
Why does this happen?
The issue occurs because the Linux kernel's conntrack module allows packets marked as 'INVALID' to pass through the KUBE-FORWARD chain. These invalid segments trigger the kernel's TCP state machine to issue a RST packet, forcefully terminating established connections.
Code Example
iptables -A KUBE-FORWARD -m conntrack --ctstate INVALID -j DROPStep-by-Step Fix
To resolve this, update your kube-proxy configuration to ensure explicit dropping of invalid conntrack packets. You can achieve this by ensuring your CNI or kube-proxy implementation includes an iptables rule that identifies and silently drops these packets before they reach the TCP stack. In managed environments, ensure your cluster version is updated to a release containing the 'INVALID' conntrack drop fix, or manually inject the rule into the KUBE-FORWARD chain.