Home
Explore
Contact
Sign in
Emergency Outage & Crash Recovery

Catch ECC Errors, Fix Docker, Offline Bad CPUs & Failover

Mission-critical Linux recovery for Australian operators: catch ECC faults in dmesg, inspect unhealthy containers, offline a damaged CPU core, and run systemctl + ip route failover—step by step.

Fixwebnode Support
Fixwebnode Support
11 min read 7 views
Catch ECC Errors, Fix Docker, Offline Bad CPUs & Failover

If your production host in Australia is throwing silent memory faults, flaky containers, or a core that keeps panicking after radiation or hardware stress, this runbook walks you through the exact commands to stabilise the box before the outage becomes a full mission failure.

You will catch ECC memory errors with dmesg, inspect unhealthy containers with docker ps, offline a radiation-damaged CPU core via /sys/devices/system/cpu/, and execute a systemctl + ip route failover so traffic lands on a healthy path. Fixwebnode provides direct remote specialist help for individuals, sole traders, and local operators across Australia—see website repair and recovery support when DIY stops being safe.

Assumptions: a modern Linux server (Ubuntu 22.04/24.04 or RHEL-family), root or sudo access, Docker Engine installed where containers run, and systemd managing services. Work from a console or SSH session you can keep open if the network path changes.

Why this recovery chain matters on Australian production hosts

ECC correctable errors often start as quiet kernel messages, then escalate to multi-bit faults, container thrashing, and sticky CPU soft-lockups. In remote or edge deployments—mining, research, coastal radio, or small business VPS stacks—waiting for a truck roll can cost the whole shift. A disciplined sequence (memory → containers → CPU isolation → service and route failover) keeps the mission alive while you schedule hardware replacement.

How do you catch ECC errors, fix unhealthy Docker containers, offline a bad CPU core, and fail over with systemctl and ip route in Australia?

Start by reading kernel ECC and MCE lines in dmesg and journal logs, mark or restart unhealthy containers from docker ps, write 0 to the damaged core’s online sysfs node, then stop the failing unit with systemctl and shift traffic with ip route (and optional floating address) onto a standby path. If errors keep climbing or the host is shared production, book Fixwebnode for remote triage rather than risking a hard crash mid-change.

SymptomQuick fixWhen to call Fixwebnode
EDAC / MCE lines in dmesgLog, rate-check, reduce load on bad DIMM channelUncorrectable errors or rising CE storm
Container status unhealthyInspect health logs, restart or replace taskOrchestrated stack you cannot safely restart
One CPU core soft-lockupsOffline core via sysfs; pin critical work elsewhereMachine check storms or multi-core cascade

Common issues you will see in the field

1. Correctable ECC errors climbing in the kernel log

Symptoms: intermittent slowdowns, EDAC or mce: lines, applications that segfault under load while free memory still looks fine. Root cause is often a degrading DIMM, bad slot, or marginal power—not “just software.”

2. Docker marks critical containers unhealthy while the host still pings

Symptoms: docker ps shows (unhealthy), healthcheck timeouts, dependent jobs queue up, reverse proxies return 502. Root cause may be memory pressure from ECC scrubbing, a wedged process inside the container, or a broken health endpoint—not always a full Docker daemon failure.

3. One logical CPU keeps soft-lockup or MCE after environmental stress

Symptoms: watchdog: BUG: soft lockup - CPU#N, repeated machine checks tied to the same core, latency spikes on pinned workloads. Treating the whole host as dead wastes uptime; offlining that core can buy hours.

4. Primary service and default route stay glued to the failing path

Symptoms: unit is active but broken, default route still points at a dead uplink or VIP, standby node is healthy but unused. Without a deliberate systemctl + ip route cutover, monitoring stays red and users stay offline.

Catch ECC memory errors with dmesg

Goal: confirm whether the kernel is reporting correctable (CE) or uncorrectable (UE) memory errors, capture evidence, and decide if you must shed load before hardware swap.

Step 1 — Pull recent kernel and MCE-related lines

sudo dmesg -T | egrep -i 'edac|ecc|mce|memory error|hardware error|Corrected error|Uncorrected'

Timestamped output shows whether faults are historical or active right now. Note the CPU, bank, and DIMM labels if EDAC prints them.

Step 2 — Cross-check the persistent journal

sudo journalctl -k -b --no-pager | egrep -i 'edac|mce|Hardware Error|Memory read error'
sudo journalctl -b -p err..alert --no-pager | tail -n 80

This catches messages that rotated out of the ring buffer and shows severity across the current boot.

Step 3 — Sample EDAC counters if the modules are loaded

ls /sys/devices/system/edac/mc 2>/dev/null
for mc in /sys/devices/system/edac/mc/mc*; do
 echo "== $mc =="
 grep -H . "$mc"/ce_count "$mc"/ue_count 2>/dev/null
 grep -H . "$mc"/csrow*/ch*_ce_count 2>/dev/null
done

Rising ce_count under steady load means schedule DIMM replacement; any non-zero ue_count is stop-the-line for critical data paths.

Step 4 — Freeze evidence and reduce risk

sudo sh -c 'dmesg -T > /root/ecc-dmesg-$(date +%F-%H%M).log'
sudo tar -czf /root/edac-sysfs-$(date +%F).tar.gz /sys/devices/system/edac 2>/dev/null
# Optionally cap runaway workers while you plan failover
sudo systemctl list-units --type=service --state=running

Verification: re-run the dmesg filter after 10–15 minutes. If new UE lines appear, skip straight to container quarantine and route failover; do not keep hammering memory-heavy jobs.

Inspect unhealthy containers with docker ps

Goal: identify which tasks are failing healthchecks, pull their logs, and restart or isolate them so the rest of the stack can fail over cleanly.

Step 1 — List container health and restart state

docker ps -a --format 'table {{.ID}}\t{{.Names}}\t{{.Status}}\t{{.Ports}}'
docker ps --filter health=unhealthy --format 'table {{.ID}}\t{{.Names}}\t{{.Status}}'

Unhealthy does not always mean exited—read the STATUS column carefully for restart loops.

Step 2 — Inspect healthcheck configuration and last output

UNHEALTHY=$(docker ps --filter health=unhealthy -q)
for id in $UNHEALTHY; do
 echo "==== $id ===="
 docker inspect --format 'Name={{.Name}} Health={{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}} OOM={{.State.OOMKilled}} Exit={{.State.ExitCode}}' "$id"
 docker inspect --format '{{json .State.Health}}' "$id" | sed 's/,/\n/g' | head -n 40
done

Look for failing probes, OOM kills, and exit codes that match the ECC-induced crash pattern.

Step 3 — Logs, then controlled restart or stop

for id in $UNHEALTHY; do
 docker logs --tail 100 "$id" 2>&1 | tail -n 100
done
# Restart a single non-stateful worker after confirming it is safe
docker restart <container_name_or_id>
# If the container must not come back on this host yet:
docker update --restart=no <container_name_or_id>
docker stop <container_name_or_id>

Step 4 — Verify daemon and resource pressure

systemctl is-active docker
sudo systemctl status docker --no-pager -l | head -n 40
free -h
uptime

Verification: docker ps --filter health=unhealthy should shrink. If every container on the node is unhealthy and ECC UE counts are rising, stop scheduling new work here and move to CPU isolation plus host failover.

Offline a radiation-damaged CPU core via /sys/devices/system/cpu/

Goal: remove a single logical CPU that is generating soft lockups or repeated MCEs so the rest of the machine can run the failover cleanly. Only offline secondary cores—never the boot CPU (usually CPU0) unless you fully understand the topology.

Step 1 — Identify the bad core and topology

nproc
lscpu | egrep 'CPU\(s\)|On-line|Off-line|Model name|NUMA'
grep -E 'soft lockup|mce:|Machine Check' /var/log/kern.log 2>/dev/null | tail -n 30
sudo dmesg -T | egrep -i 'soft lockup|cpu#[0-9]+|mce' | tail -n 40

Confirm the CPU number from the lockup line (for example CPU#3).

Step 2 — Check that the core can be offlined

BAD=3
cat /sys/devices/system/cpu/cpu$BAD/online
ls /sys/devices/system/cpu/cpu$BAD/

If online is missing, the architecture or hypervisor may not allow hot-unplug; plan full host failover instead.

Step 3 — Drain work and offline the core

BAD=3
# Optional: move IRQs / stop CPU-bound units first
sudo systemctl stop your-latency-sensitive.service 2>/dev/null || true
echo 0 | sudo tee /sys/devices/system/cpu/cpu$BAD/online
cat /sys/devices/system/cpu/cpu$BAD/online
grep -H . /sys/devices/system/cpu/online /sys/devices/system/cpu/offline

Expected: online reads 0 and the core appears in the offline list. Document the change in your incident notes; this does not repair silicon—it isolates it.

Step 4 — Verification and rollback note

nproc
# Confirm lockups stop targeting that CPU
sudo dmesg -T | tail -n 20
# Rollback only if the core was offlined by mistake and is known-good:
# echo 1 | sudo tee /sys/devices/system/cpu/cpu3/online

If offlining fails with EBUSY, processes or IRQs are pinned; stop those units, clear affinity, and retry. Multi-core MCE storms mean the DIMM/CPU package is unsafe—prioritise route failover and hardware replacement.

Execute systemctl + ip route failover to save the mission

Goal: take the failing role offline cleanly, point traffic at the standby path, and confirm users land on the healthy service. Adjust interface names, table IDs, and unit names to your host.

Step 1 — Snapshot service and routing state

systemctl list-units --failed --no-pager
systemctl status nginx.service docker.service your-app.service --no-pager -l | head -n 80
ip -br link
ip -br addr
ip route show table main
ip rule show

Save outputs so you can reverse the cutover if the standby misbehaves.

Step 2 — Stop or isolate the failing local role

sudo systemctl stop your-app.service
sudo systemctl stop nginx.service
# Prevent surprise restarts during cutover
sudo systemctl mask your-app.service
# If Docker publishes the broken role:
docker stop <critical_container> 2>/dev/null || true

Only mask units you intend to keep dark on this node until hardware is fixed.

Step 3 — Shift the default route or policy path to the standby uplink

# Example: primary uplink eth0 is degraded; standby is eth1 via 192.0.2.1
sudo ip route replace default via 192.0.2.1 dev eth1 metric 50
# Optional floating / VIP move (example address—use yours)
# sudo ip addr del 203.0.113.10/32 dev eth0
# sudo ip addr add 203.0.113.10/32 dev eth1
ip route get 1.1.1.1
ping -c 3 1.1.1.1

Use replace rather than stacking conflicting defaults. If you use policy routing, update the correct table—not only main.

Step 4 — Bring the standby service path up and verify

# On the standby host (or local standby unit), start the healthy role
sudo systemctl unmask your-app.service 2>/dev/null || true
sudo systemctl start your-app.service
sudo systemctl start nginx.service
sudo systemctl is-active your-app.service nginx.service
curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1/healthz || true
ss -lntup | egrep ':80|:443|:8080' || true

Step 5 — Confirm external path and leave breadcrumbs

ip route show
sudo dmesg -T | tail -n 15
sudo systemctl --failed --no-pager
echo "Failover $(date -Is) default via standby; CPU offline + ECC log under /root" | sudo tee /root/FAILOVER-NOTE.txt

Verification: health endpoint returns success from the standby path, systemctl --failed is clean for the roles you care about, and ECC logging continues in the background for the hardware ticket. Revert routes only after the primary is repaired and soak-tested.

Troubleshooting quick map

dmesg empty for ECC but apps crash: EDAC modules may be absent on a VM—check hypervisor health and guest OOM with journalctl -b -p err and free -h.

Cannot offline CPU: CPU0 or pinned IRQs; migrate services off-box instead of forcing sysfs writes.

Route change does not move users: DNS, anycast, or cloud LB still targets the old node—update the load balancer pool or floating IP in the same change window.

Docker unhealthy after restart: healthcheck may need a longer start period, or the image is OOMing because ECC scrubbing reduced effective bandwidth—scale out, do not tight-loop restarts.

When DIY is enough vs when to book Fixwebnode

DIY is reasonable when you have console access, CE-only ECC noise, a single unhealthy worker container, one offlinable core, and a documented standby route you can reverse. Stop and book a specialist when you see uncorrectable memory errors, multi-core MCE storms, clustered shared storage you cannot risk, unclear VIP ownership, or a regulated production window where a bad ip route could strand remote sites.

Fixwebnode is a direct remote provider for Australian individuals, sole traders, and local operators—not a freelance marketplace. If you need a guided cutover or a second pair of eyes on kernel and container evidence, start from the recovery landing page and the national coverage overview at service areas.

Talk through your failover before the next error storm

You now have a single coherent path: prove ECC faults with dmesg and EDAC counters, quarantine unhealthy containers via docker ps and inspect, offline the damaged core under /sys/devices/system/cpu/, then cut services and routes with systemctl and ip route so the mission continues on healthy silicon.

If you want a remote specialist to review your logs, validate the standby path, or sit with you during the cutover window in Australia, book a conversation through Fixwebnode website repair Australia. Bring the ECC log bundle and a note of which CPU you offlined—we will pick up from the evidence, not from guesswork.

Share this article
Fixwebnode Support
Fixwebnode Support

Hey there!
I am your assistant for Fixwebnode. Ask about our services, quotes, packages, orders, or how to get support.
While you wait
What’s your name and best email? We’ll reply even if you leave.