Trace & Fix Asymmetric Routing Loops on Multi-Homed Linux

If your multi-homed Linux root server drops API packets, times out on one NIC only, or returns half-open TCP sessions under load, you are almost certainly fighting asymmetric routing—not a flaky application.

This guide walks Australian site owners and sysadmins through a full terminal runbook: SSH to the box, inspect the routing matrix with ip route show, audit rp_filter, fix policy routes, and verify that request and reply paths match. When the matrix is too tangled for a safe DIY window, Fixwebnode Linux server support can take the session remotely and stabilise the host without marketplace middlemen.

Why asymmetric routing breaks server APIs

On a dual-homed or multi-ISP Linux server, packets often ingress on eth0 and egress on eth1 (or a secondary default route). Strict reverse-path filtering then drops the reply. Clients see RST, silent blackholes, or “broken” API payloads even though nginx/php-fpm are healthy. In Australia this pattern is common on colo and cloud boxes with a primary NBN/fibre path plus a backup link or floating IP.

How do I trace asymmetric routing loops and broken API packets on a Linux server in Australia?

SSH in as root, dump the full routing table and rules with ip route show table all and ip rule list, then compare each interface’s reverse-path filter via sysctl net.ipv4.conf.*.rp_filter. If ingress and egress interfaces disagree, tighten policy routing or set rp_filter to loose mode on the affected NIC, capture with tcpdump, and confirm symmetric replies before restoring strict filtering.

Common multi-homed issues that break API packets

1. Strict rp_filter dropping legitimate replies

Symptoms: client TCP handshake completes on paper, server application never sees the request, or kernel logs show martian / reverse-path failures. One uplink carries the request; the default route sends the reply out another.

2. Competing default routes without policy routing

Symptoms: ip route show lists two defaults with similar metrics; API calls randomly succeed or fail depending on which path the FIB chooses. Conntrack entries point at the wrong egress.

3. Broken source-based routing after a floating IP or failover

Symptoms: health checks pass on the management IP, but public API traffic sourced from the secondary address never returns; SSH still works on the primary.

4. Asymmetric path through a cloud security group or on-prem firewall

Symptoms: outbound API webhooks leave the server, return path is blocked or NAT’d differently; only one direction of the conversation appears in captures.

Fix 1 — Audit routes and rp_filter from an SSH root shell

Prerequisites: root or sudo, out-of-band console if you might lock yourself out, maintenance window for production APIs.

Step 1 — Connect and freeze a baseline

ssh root@YOUR_SERVER_IP
sudo -i
ip -br addr
ip route show table all
ip rule list
sysctl net.ipv4.conf.all.rp_filter net.ipv4.conf.default.rp_filter

Note every default route, table ID, and interface name. Australian dual-homed hosts often show default via … dev eth0 and a second default in table 100 or 200.

Step 2 — Inspect per-interface reverse-path filter

for i in /proc/sys/net/ipv4/conf/*/rp_filter; do echo -n "$i: "; cat $i; done
sysctl net.ipv4.conf.all.rp_filter
sysctl net.ipv4.conf.eth0.rp_filter
sysctl net.ipv4.conf.eth1.rp_filter

Value 1 is strict (drop if reply would not leave the ingress interface). Value 2 is loose (accept if any local interface could route it). Value 0 disables the check.

Step 3 — Prove asymmetry with a live client IP

ip route get CLIENT_PUBLIC_IP
ip route get CLIENT_PUBLIC_IP from SERVER_PUBLIC_IP iif eth0
tcpdump -ni any host CLIENT_PUBLIC_IP and port 443 -c 40

If the route lookup egresses on a different interface than the one that received the SYN, you have confirmed the loop/asymmetry that breaks API packets.

Step 4 — Temporary safe relief (test only)

sysctl -w net.ipv4.conf.all.rp_filter=2
sysctl -w net.ipv4.conf.default.rp_filter=2
sysctl -w net.ipv4.conf.eth0.rp_filter=2
sysctl -w net.ipv4.conf.eth1.rp_filter=2

Retest the API. If packets flow, the root cause is reverse-path policy, not the application. Do not leave production on loose mode without a documented exception.

Fix 2 — Install correct policy routing so replies exit the ingress path

Goal: traffic that arrives on a given public address must leave via the gateway that owns that address.

Step 1 — Create a dedicated table for the secondary uplink

echo "100 uplink1" >> /etc/iproute2/rt_tables
echo "200 uplink2" >> /etc/iproute2/rt_tables
cat /etc/iproute2/rt_tables | grep uplink

Step 2 — Add routes into each table

ip route add default via GW1_IP dev eth0 table uplink1
ip route add default via GW2_IP dev eth1 table uplink2
ip route add 10.0.0.0/8 dev eth0 table uplink1
ip route show table uplink1
ip route show table uplink2

Replace GW1_IP / GW2_IP and prefixes with your real gateways and local subnets.

Step 3 — Bind source addresses to tables

ip rule add from SERVER_IP_ON_ETH0 table uplink1 priority 100
ip rule add from SERVER_IP_ON_ETH1 table uplink2 priority 110
ip rule list

Step 4 — Persist across reboot (Debian/Ubuntu example)

cat > /etc/networkd-dispatcher/routable.d/50-multihome.sh <<'EOF'
#!/bin/bash
ip route replace default via GW1_IP dev eth0 table uplink1
ip route replace default via GW2_IP dev eth1 table uplink2
ip rule add from SERVER_IP_ON_ETH0 table uplink1 priority 100 2>/dev/null
ip rule add from SERVER_IP_ON_ETH1 table uplink2 priority 110 2>/dev/null
EOF
chmod +x /etc/networkd-dispatcher/routable.d/50-multihome.sh

On classic ifupdown, place equivalent up ip route … stanzas in /etc/network/interfaces. On RHEL-like systems use NetworkManager dispatcher scripts or nmcli routing rules.

Step 5 — Verify symmetry

ip route get CLIENT_IP from SERVER_IP_ON_ETH0 iif eth0
ip route get CLIENT_IP from SERVER_IP_ON_ETH1 iif eth1
curl -v --interface SERVER_IP_ON_ETH0 https://api.internal.health/ping
ss -ti state established '( sport = :443 )' | head

Both lookups must show the matching gateway and device. Established sockets should no longer flip interfaces mid-flow.

Fix 3 — Repair broken API packets and conntrack after the route fix

Asymmetric history leaves stale conntrack and half-closed API sessions.

Step 1 — Flush only the noisy entries (prefer targeted deletes)

conntrack -L | grep -E '443|8080|API_PORT' | head
conntrack -D -p tcp --dport 443
sysctl net.netfilter.nf_conntrack_tcp_timeout_established

Step 2 — Confirm reverse path accepts the flow

dmesg | tail -50 | grep -i -E 'martian|rp_filter|refused'
journalctl -k -n 80 --no-pager | grep -i martian

Step 3 — Application-layer sanity

systemctl status nginx php8.2-fpm
curl -sS -o /dev/null -w '%{http_code} %{time_total}\n' https://127.0.0.1/health
tail -n 100 /var/log/nginx/error.log

If HTTP is clean locally but remote clients still fail, re-run tcpdump on the public NIC only—routing is still wrong.

Step 4 — Lock rp_filter back to a deliberate policy

cat > /etc/sysctl.d/90-rp-filter.conf <<'EOF'
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.eth0.rp_filter = 1
net.ipv4.conf.eth1.rp_filter = 1
EOF
sysctl --system | grep rp_filter

Strict on each edge NIC plus loose on “all/default” is a common balanced stance once policy routing is correct. Document the choice.

Fix 4 — Catch residual asymmetric firewall / cloud path issues

Step 1 — Compare ingress ACL vs egress path

iptables -L -n -v | head -80
nft list ruleset 2>/dev/null | head -80
ip -4 route show table main

Step 2 — One-shot bidirectional capture

tcpdump -ni eth0 host CLIENT_IP -w /tmp/eth0.pcap &
tcpdump -ni eth1 host CLIENT_IP -w /tmp/eth1.pcap &
sleep 30
kill %1 %2
tcpdump -r /tmp/eth0.pcap -nn | head
tcpdump -r /tmp/eth1.pcap -nn | head

If SYNs land on eth0 and SYN-ACKs leave eth1 without a matching policy rule, extend Fix 2. If packets leave correctly but never return, the far-side firewall or provider route—not Linux—is at fault; open a ticket with the carrier while keeping the local matrix clean.

When DIY is enough vs when to book Fixwebnode

DIY is enough when you have console access, a single secondary default route, and can schedule a short API blip. Stop and book a specialist when any of the following apply: more than two routing tables with overlapping priorities, live payment or healthcare APIs with no maintenance window, repeated lockouts after sysctl changes, or hybrid cloud + on-prem paths you cannot capture fully.

Fixwebnode works as a direct remote Linux specialist for Australian businesses—not a freelance board. We step into the same SSH session, read your routing matrix, and leave you with persistent ip rule / sysctl config and a verification checklist. See where support is offered on the service areas page, including related on-site digital work such as Bunbury website repairs and maintenance and Melbourne software installation when the server issue ties into a broader stack rebuild.

Book a remote routing fix

If API packets are still asymmetric after the steps above, start a conversation with the team that handles Ubuntu and multi-homed Linux hosts day to day. Use the landing page to outline your interfaces, public IPs, and failure window: Fixwebnode Ubuntu Linux server support & bug fixing. Bring the output of ip route show table all, ip rule list, and your current rp_filter values—we will take it from there and restore symmetric, production-safe paths.