Packet Loss on Linux: Counters First, Theories Second

ip -s link show eth0          # RX/TX dropped, errors, overruns
ethtool -S eth0 | grep -iE 'drop|miss|error|fifo' | grep -v ': 0'

rx missed/fifo growing under load = the NIC ring could not hand frames to the driver fast enough. Check and enlarge:

ethtool -g eth0                 # current vs maximum ring sizes
ethtool -G eth0 rx 4096 tx 4096

Ring increases are safe online and often end the incident on busy VMs whose virtual NIC defaults are small.

If rings are fine but ip -s link shows drops, the kernel's softirq processing may be saturated: one core at 100% si while others idle is the signature.

top            # watch %si per core
cat /proc/net/softnet_stat      # 2nd column nonzero = backlog drops/time-slices
ethtool -l eth0; ethtool -L eth0 combined 8   # spread queues across cores

Fixes in order: enable all RX queues (RSS), balance IRQs away from a busy core, and for sustained 10G+ consider RPS/GRO tuning or a bigger box — a single-queue vNIC on a noisy hypervisor is a classic generator of this pattern.

ethtool eth0 | grep -E 'Speed|Duplex|Link detected'
ethtool -S eth0 | grep -i crc            # NIC-side CRC counters
# switch port: errors, duplex, and whether autoneg disagreed

A half/full duplex mismatch produces collisions and late collisions plus symmetrical pain: retransmits everywhere, speed that collapses under load. Fix by hard-setting both ends identically or fixing autoneg — never "one side auto, one side forced". CRC errors are physical: cable, SFP, port. Swap one thing at a time.

Loss only for large payloads (scp dies, ping -s 64 works) is the MTU/DF signature: a path element with a lower MTU drops DF packets and the ICMP "need to frag" never comes back (filtered). Prove it:

ping -M do -s 1472 10.0.0.1      # fails
ping -M do -s 1400 10.0.0.1      # works  -> path MTU problem
tracepath 10.0.0.1               # reports the pmtu

Fix the mismatch (tunnel/VPN overheads are the usual authors: VXLAN/GRE/IPsec each eat bytes), not the symptom (tcp_mtu_probing hides it and costs throughput).

• Application-level "loss" with clean counters: socket buffers (netstat -s | grep -i overflow, ListenOverflows) — a slow listener drops, the network is innocent.
• Conntrack table full: dmesg | grep conntrack — NAT boxes drop new flows when the table fills.
• tc/qdisc drops: tc -s qdisc shows drops on shaped or pfifo_fast-overflowing queues.

When "the network" and "the server" blame each other, settle it with simultaneous captures at both ends plus switch port counters:

tcpdump -i eth0 -s 96 -w /tmp/loss.pcap host 10.0.0.5 and port 443
mtr -r -c 200 10.0.0.5        # per-hop loss over time

Count the same flows on both pcaps: loss between the captures is the network's; present on the wire but not delivered is the host's (rings/softirq/sockets). Two traps: capture with small snaplen and without GRO confusion — offloads make captured packets look retransmitted/merged oddly, so also record ethtool -k eth0 state; and never trust a single five-second sample — loss that appears only under load needs a capture window that spans the load.

TSO/GRO/LRO merge or segment packets below/above the wire view, producing "retransmissions" and odd sizes that never existed on the cable. When a capture looks impossible, record ethtool -k eth0, and for a clean dispute capture with offloads off on the test interface (ethtool -K eth0 tso off gro off) — briefly, on a non-production path — then restore. Also check the boring sysctls: net.core.rmem_max/socket buffers and nf_conntrack counts explain "loss" that is really the host shedding load.

nstat -az | grep -iE 'drop|overflow|retrans' | grep -v ' 0 '
ss -s                       # socket summary: TCPlastQueueRecv, ListenOverflows

ListenOverflows/ListenDrops prove accept-queue shedding; TCPBacklogDrop and PruneCalled name receive-path memory pressure; growing TCPRetransSegs with clean NIC counters points at the path, not the host. Snapshot these before and during load — the delta, not the absolute value, is the evidence.

• Graph NIC error/drop counters and switch port counters together; correlate ends during incidents.
• Standardise ring sizes and queue counts in the build image.
• Alert on softnet_stat backlog drops and on ListenOverflows — both fire before users do.
• Document path MTUs for every tunnel; VPN "slow" tickets are MTU tickets in disguise.