Catch ECC Errors, Fix Docker, Offline Bad CPUs & Failover
Mission-critical Linux recovery for Australian operators: catch ECC faults in dmesg, inspect unhealthy containers, offline a damaged CPU core, and run systemctl + ip route failover—step by step.
If your production host in Australia is throwing silent memory faults, flaky containers, or a core that keeps panicking after radiation or hardware stress, this runbook walks you through the exact commands to stabilise the box before the outage becomes a full mission failure.
You will catch ECC memory errors with dmesg, inspect unhealthy containers with docker ps, offline a radiation-damaged CPU core via /sys/devices/system/cpu/, and execute a systemctl + ip route failover so traffic lands on a healthy path. Fixwebnode provides direct remote specialist help for individuals, sole traders, and local operators across Australia—see website repair and recovery support when DIY stops being safe.
Assumptions: a modern Linux server (Ubuntu 22.04/24.04 or RHEL-family), root or sudo access, Docker Engine installed where containers run, and systemd managing services. Work from a console or SSH session you can keep open if the network path changes.
Why this recovery chain matters on Australian production hosts
ECC correctable errors often start as quiet kernel messages, then escalate to multi-bit faults, container thrashing, and sticky CPU soft-lockups. In remote or edge deployments—mining, research, coastal radio, or small business VPS stacks—waiting for a truck roll can cost the whole shift. A disciplined sequence (memory → containers → CPU isolation → service and route failover) keeps the mission alive while you schedule hardware replacement.
How do you catch ECC errors, fix unhealthy Docker containers, offline a bad CPU core, and fail over with systemctl and ip route in Australia?
Start by reading kernel ECC and MCE lines in dmesg and journal logs, mark or restart unhealthy containers from docker ps, write 0 to the damaged core’s online sysfs node, then stop the failing unit with systemctl and shift traffic with ip route (and optional floating address) onto a standby path. If errors keep climbing or the host is shared production, book Fixwebnode for remote triage rather than risking a hard crash mid-change.
| Symptom | Quick fix | When to call Fixwebnode |
|---|---|---|
| EDAC / MCE lines in dmesg | Log, rate-check, reduce load on bad DIMM channel | Uncorrectable errors or rising CE storm |
| Container status unhealthy | Inspect health logs, restart or replace task | Orchestrated stack you cannot safely restart |
| One CPU core soft-lockups | Offline core via sysfs; pin critical work elsewhere | Machine check storms or multi-core cascade |
Common issues you will see in the field
1. Correctable ECC errors climbing in the kernel log
Symptoms: intermittent slowdowns, EDAC or mce: lines, applications that segfault under load while free memory still looks fine. Root cause is often a degrading DIMM, bad slot, or marginal power—not “just software.”
2. Docker marks critical containers unhealthy while the host still pings
Symptoms: docker ps shows (unhealthy), healthcheck timeouts, dependent jobs queue up, reverse proxies return 502. Root cause may be memory pressure from ECC scrubbing, a wedged process inside the container, or a broken health endpoint—not always a full Docker daemon failure.
3. One logical CPU keeps soft-lockup or MCE after environmental stress
Symptoms: watchdog: BUG: soft lockup - CPU#N, repeated machine checks tied to the same core, latency spikes on pinned workloads. Treating the whole host as dead wastes uptime; offlining that core can buy hours.
4. Primary service and default route stay glued to the failing path
Symptoms: unit is active but broken, default route still points at a dead uplink or VIP, standby node is healthy but unused. Without a deliberate systemctl + ip route cutover, monitoring stays red and users stay offline.
Catch ECC memory errors with dmesg
Goal: confirm whether the kernel is reporting correctable (CE) or uncorrectable (UE) memory errors, capture evidence, and decide if you must shed load before hardware swap.
Step 1 — Pull recent kernel and MCE-related lines
sudo dmesg -T | egrep -i 'edac|ecc|mce|memory error|hardware error|Corrected error|Uncorrected'Timestamped output shows whether faults are historical or active right now. Note the CPU, bank, and DIMM labels if EDAC prints them.
Step 2 — Cross-check the persistent journal
sudo journalctl -k -b --no-pager | egrep -i 'edac|mce|Hardware Error|Memory read error'
sudo journalctl -b -p err..alert --no-pager | tail -n 80This catches messages that rotated out of the ring buffer and shows severity across the current boot.
Step 3 — Sample EDAC counters if the modules are loaded
ls /sys/devices/system/edac/mc 2>/dev/null
for mc in /sys/devices/system/edac/mc/mc*; do
echo "== $mc =="
grep -H . "$mc"/ce_count "$mc"/ue_count 2>/dev/null
grep -H . "$mc"/csrow*/ch*_ce_count 2>/dev/null
doneRising ce_count under steady load means schedule DIMM replacement; any non-zero ue_count is stop-the-line for critical data paths.
Step 4 — Freeze evidence and reduce risk
sudo sh -c 'dmesg -T > /root/ecc-dmesg-$(date +%F-%H%M).log'
sudo tar -czf /root/edac-sysfs-$(date +%F).tar.gz /sys/devices/system/edac 2>/dev/null
# Optionally cap runaway workers while you plan failover
sudo systemctl list-units --type=service --state=runningVerification: re-run the dmesg filter after 10–15 minutes. If new UE lines appear, skip straight to container quarantine and route failover; do not keep hammering memory-heavy jobs.
Inspect unhealthy containers with docker ps
Goal: identify which tasks are failing healthchecks, pull their logs, and restart or isolate them so the rest of the stack can fail over cleanly.
Step 1 — List container health and restart state
docker ps -a --format 'table {{.ID}}\t{{.Names}}\t{{.Status}}\t{{.Ports}}'
docker ps --filter health=unhealthy --format 'table {{.ID}}\t{{.Names}}\t{{.Status}}'Unhealthy does not always mean exited—read the STATUS column carefully for restart loops.
Step 2 — Inspect healthcheck configuration and last output
UNHEALTHY=$(docker ps --filter health=unhealthy -q)
for id in $UNHEALTHY; do
echo "==== $id ===="
docker inspect --format 'Name={{.Name}} Health={{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}} OOM={{.State.OOMKilled}} Exit={{.State.ExitCode}}' "$id"
docker inspect --format '{{json .State.Health}}' "$id" | sed 's/,/\n/g' | head -n 40
doneLook for failing probes, OOM kills, and exit codes that match the ECC-induced crash pattern.
Step 3 — Logs, then controlled restart or stop
for id in $UNHEALTHY; do
docker logs --tail 100 "$id" 2>&1 | tail -n 100
done
# Restart a single non-stateful worker after confirming it is safe
docker restart <container_name_or_id>
# If the container must not come back on this host yet:
docker update --restart=no <container_name_or_id>
docker stop <container_name_or_id>Step 4 — Verify daemon and resource pressure
systemctl is-active docker
sudo systemctl status docker --no-pager -l | head -n 40
free -h
uptimeVerification: docker ps --filter health=unhealthy should shrink. If every container on the node is unhealthy and ECC UE counts are rising, stop scheduling new work here and move to CPU isolation plus host failover.
Offline a radiation-damaged CPU core via /sys/devices/system/cpu/
Goal: remove a single logical CPU that is generating soft lockups or repeated MCEs so the rest of the machine can run the failover cleanly. Only offline secondary cores—never the boot CPU (usually CPU0) unless you fully understand the topology.
Step 1 — Identify the bad core and topology
nproc
lscpu | egrep 'CPU\(s\)|On-line|Off-line|Model name|NUMA'
grep -E 'soft lockup|mce:|Machine Check' /var/log/kern.log 2>/dev/null | tail -n 30
sudo dmesg -T | egrep -i 'soft lockup|cpu#[0-9]+|mce' | tail -n 40Confirm the CPU number from the lockup line (for example CPU#3).
Step 2 — Check that the core can be offlined
BAD=3
cat /sys/devices/system/cpu/cpu$BAD/online
ls /sys/devices/system/cpu/cpu$BAD/If online is missing, the architecture or hypervisor may not allow hot-unplug; plan full host failover instead.
Step 3 — Drain work and offline the core
BAD=3
# Optional: move IRQs / stop CPU-bound units first
sudo systemctl stop your-latency-sensitive.service 2>/dev/null || true
echo 0 | sudo tee /sys/devices/system/cpu/cpu$BAD/online
cat /sys/devices/system/cpu/cpu$BAD/online
grep -H . /sys/devices/system/cpu/online /sys/devices/system/cpu/offlineExpected: online reads 0 and the core appears in the offline list. Document the change in your incident notes; this does not repair silicon—it isolates it.
Step 4 — Verification and rollback note
nproc
# Confirm lockups stop targeting that CPU
sudo dmesg -T | tail -n 20
# Rollback only if the core was offlined by mistake and is known-good:
# echo 1 | sudo tee /sys/devices/system/cpu/cpu3/onlineIf offlining fails with EBUSY, processes or IRQs are pinned; stop those units, clear affinity, and retry. Multi-core MCE storms mean the DIMM/CPU package is unsafe—prioritise route failover and hardware replacement.
Execute systemctl + ip route failover to save the mission
Goal: take the failing role offline cleanly, point traffic at the standby path, and confirm users land on the healthy service. Adjust interface names, table IDs, and unit names to your host.
Step 1 — Snapshot service and routing state
systemctl list-units --failed --no-pager
systemctl status nginx.service docker.service your-app.service --no-pager -l | head -n 80
ip -br link
ip -br addr
ip route show table main
ip rule showSave outputs so you can reverse the cutover if the standby misbehaves.
Step 2 — Stop or isolate the failing local role
sudo systemctl stop your-app.service
sudo systemctl stop nginx.service
# Prevent surprise restarts during cutover
sudo systemctl mask your-app.service
# If Docker publishes the broken role:
docker stop <critical_container> 2>/dev/null || trueOnly mask units you intend to keep dark on this node until hardware is fixed.
Step 3 — Shift the default route or policy path to the standby uplink
# Example: primary uplink eth0 is degraded; standby is eth1 via 192.0.2.1
sudo ip route replace default via 192.0.2.1 dev eth1 metric 50
# Optional floating / VIP move (example address—use yours)
# sudo ip addr del 203.0.113.10/32 dev eth0
# sudo ip addr add 203.0.113.10/32 dev eth1
ip route get 1.1.1.1
ping -c 3 1.1.1.1Use replace rather than stacking conflicting defaults. If you use policy routing, update the correct table—not only main.
Step 4 — Bring the standby service path up and verify
# On the standby host (or local standby unit), start the healthy role
sudo systemctl unmask your-app.service 2>/dev/null || true
sudo systemctl start your-app.service
sudo systemctl start nginx.service
sudo systemctl is-active your-app.service nginx.service
curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1/healthz || true
ss -lntup | egrep ':80|:443|:8080' || trueStep 5 — Confirm external path and leave breadcrumbs
ip route show
sudo dmesg -T | tail -n 15
sudo systemctl --failed --no-pager
echo "Failover $(date -Is) default via standby; CPU offline + ECC log under /root" | sudo tee /root/FAILOVER-NOTE.txtVerification: health endpoint returns success from the standby path, systemctl --failed is clean for the roles you care about, and ECC logging continues in the background for the hardware ticket. Revert routes only after the primary is repaired and soak-tested.
Troubleshooting quick map
dmesg empty for ECC but apps crash: EDAC modules may be absent on a VM—check hypervisor health and guest OOM with journalctl -b -p err and free -h.
Cannot offline CPU: CPU0 or pinned IRQs; migrate services off-box instead of forcing sysfs writes.
Route change does not move users: DNS, anycast, or cloud LB still targets the old node—update the load balancer pool or floating IP in the same change window.
Docker unhealthy after restart: healthcheck may need a longer start period, or the image is OOMing because ECC scrubbing reduced effective bandwidth—scale out, do not tight-loop restarts.
When DIY is enough vs when to book Fixwebnode
DIY is reasonable when you have console access, CE-only ECC noise, a single unhealthy worker container, one offlinable core, and a documented standby route you can reverse. Stop and book a specialist when you see uncorrectable memory errors, multi-core MCE storms, clustered shared storage you cannot risk, unclear VIP ownership, or a regulated production window where a bad ip route could strand remote sites.
Fixwebnode is a direct remote provider for Australian individuals, sole traders, and local operators—not a freelance marketplace. If you need a guided cutover or a second pair of eyes on kernel and container evidence, start from the recovery landing page and the national coverage overview at service areas.
Talk through your failover before the next error storm
You now have a single coherent path: prove ECC faults with dmesg and EDAC counters, quarantine unhealthy containers via docker ps and inspect, offline the damaged core under /sys/devices/system/cpu/, then cut services and routes with systemctl and ip route so the mission continues on healthy silicon.
If you want a remote specialist to review your logs, validate the standby path, or sit with you during the cutover window in Australia, book a conversation through Fixwebnode website repair Australia. Bring the ECC log bundle and a note of which CPU you offlined—we will pick up from the evidence, not from guesswork.