Trace and Fix systemd Service Restart Loops on Linux

If a background daemon on your Linux box keeps dying and coming back every few seconds, you are already in a restart death loop—and CPU, disk, and logs will pay for it until you intervene.

This guide is for sole traders, small businesses, and operators in Australia who run queue workers, scrapers, mailers, or custom automation under systemd. You will SSH in, read the unit journal, identify why the process exits, add native restart delays and burst caps, and bring the pipeline back under control. When the failure is deeper than a unit file tweak, Fixwebnode can take over the same remote diagnostics on your host.

Why systemd restart loops matter under pressure

systemd is designed to keep services alive. That is useful until a broken script, bad environment variable, missing socket, or dependency race causes the main process to exit immediately. With Restart=always and no burst limit, the manager respawns the unit as fast as it can. On production VPS and bare-metal hosts used across Australia, that pattern shows up as load spikes, full journals, failed job queues, and cascading timeouts for the apps that depend on the worker.

The workflow below stays on one problem: trace the failing unit, stop the thrash, fix the root cause, then restore clean restart policy so automation runs again without hammering the box.

How do I stop a systemd service stuck in an infinite restart loop in Australia?

SSH to the host, inspect the unit with sudo journalctl -u queue-worker.service -xe, then temporarily stop or mask the unit if it is thrashing. Edit the unit (or a drop-in) to set StartLimitIntervalSec, StartLimitBurst, and RestartSec, reload systemd, fix the underlying script or path error, and start the service once. If the process still exits on every start or the unit file is tangled with custom targets, book remote help rather than looping restarts overnight.

Common issues with background daemon crash loops

1. Infinite restart death loop on a queue worker

Symptoms: systemctl status queue-worker.service shows repeated starts; load average climbs; journalctl repeats the same exit code within seconds. Often caused by Restart=always with no delay when the worker binary or PHP/Python entrypoint fails instantly.

2. Start-limit lockout after a burst of crashes

Symptoms: unit sits in failed with “start request repeated too quickly”; automation is dead until someone runs reset-failed. This is systemd protecting the host after an uncontrolled burst—useful, but the underlying crash is still unfixed.

3. ExecStart path, user, or environment mismatch

Symptoms: status shows status=203/EXEC or status=217/USER; the script works interactively under your shell but not as a service. Missing WorkingDirectory=, wrong User=, or secrets only present in interactive profiles are typical.

4. Dependency and socket races at boot

Symptoms: service fails only on reboot; manual start later works. Network, database, or Redis is not ready when the worker starts, so the process exits and systemd restarts into a loop until the dependency appears—or until start limits trip.

Issue 1 — Trace and break an infinite restart death loop

Goal: confirm the unit is thrashing, stop the damage, then apply native delay and burst caps.

Step 1 — SSH in and inspect the unit journal

From your workstation, connect as a user with sudo and read the service log. Replace the unit name if yours differs.

ssh you@your-server.example
sudo systemctl status queue-worker.service --no-pager
sudo journalctl -u queue-worker.service -xe --no-pager
sudo journalctl -u queue-worker.service -n 100 --no-pager

Look for repeating exit codes, stack traces, “No such file or directory”, permission errors, or connection refused to Redis/Postgres. Note the exact ExecStart= path from the status block.

Step 2 — Stop the thrash immediately

sudo systemctl stop queue-worker.service
sudo systemctl reset-failed queue-worker.service
sudo systemctl status queue-worker.service --no-pager

If it respawns against your will (rare with a normal simple service, more common with path/socket activation), mask temporarily:

sudo systemctl mask queue-worker.service
sudo systemctl stop queue-worker.service

Step 3 — Add restart delay and burst caps via a drop-in

Prefer a drop-in so package updates do not wipe your policy.

sudo systemctl edit queue-worker.service

In the editor, use:

[Service]
Restart=on-failure
RestartSec=5s
StartLimitIntervalSec=300
StartLimitBurst=5

[Unit]
StartLimitIntervalSec=300
StartLimitBurst=5

On newer systemd, start-limit keys on [Unit] are the clear place; keep RestartSec under [Service]. Avoid bare Restart=always until the process can stay up.

Step 4 — Reload and verify policy without starting yet

sudo systemctl daemon-reload
sudo systemctl cat queue-worker.service
sudo systemctl show queue-worker.service -p Restart -p RestartUSec -p StartLimitIntervalUSec -p StartLimitBurst --no-pager

Step 5 — Unmask (if masked), fix the obvious crash, then one clean start

sudo systemctl unmask queue-worker.service
sudo systemctl start queue-worker.service
sudo systemctl status queue-worker.service --no-pager
sudo journalctl -u queue-worker.service -f

Watch the follow log for at least one full job cycle. If it exits again, stay on the journal—do not raise StartLimitBurst to hide the bug.

When to call a pro: the unit still cycles after path and dependency checks, or multiple workers share the same broken runtime. Remote session via Fixwebnode website repair in Australia is appropriate when you cannot risk further thrashing on a live queue.

Issue 2 — Clear start-limit lockout and restore controlled restarts

Goal: understand why systemd refused new starts, reset state, and keep limits that protect the host.

Step 1 — Confirm lockout messaging

sudo systemctl status queue-worker.service --no-pager
sudo journalctl -u queue-worker.service -b --no-pager | tail -n 80

Phrases like “start request repeated too quickly” or “Start operation timed out” mean the limiter fired.

Step 2 — Reset failed state only after you know the cause

sudo systemctl reset-failed queue-worker.service
sudo systemctl start queue-worker.service

Resetting without fixing the exit reason only buys another burst.

Step 3 — Tune limits for a background pipeline

Example drop-in for a worker that may need a few retries after a DB blip, but must not spin forever:

sudo mkdir -p /etc/systemd/system/queue-worker.service.d
sudo tee /etc/systemd/system/queue-worker.service.d/restart-limits.conf <<'EOF'
[Unit]
StartLimitIntervalSec=600
StartLimitBurst=3

[Service]
Restart=on-failure
RestartSec=10s
TimeoutStartSec=60
EOF
sudo systemctl daemon-reload
sudo systemctl restart queue-worker.service
sudo systemctl status queue-worker.service --no-pager

Step 4 — Verify the limiter works

After three deliberate quick failures (only on a staging host), status should show failed and refuse a fourth start until the interval passes. On production, rely on journal history rather than forcing crashes.

When to call a pro: lockout returns every night at the same time (cron/batch interaction), or timers and services fight each other. That pattern needs coordinated unit design, not endless reset-failed.

Issue 3 — Fix ExecStart, user, and environment mismatches

Goal: make the service runtime match the environment where the script already works by hand.

Step 1 — Compare interactive vs service execution

sudo systemctl cat queue-worker.service
which node || true
which python3 || true
ls -la /var/www/app/workers/queue-worker.js
id deploy || id www-data || true

Step 2 — Run the command as the service user

sudo -u deploy -H bash -lc 'cd /var/www/app && /usr/bin/node workers/queue-worker.js'

Fix any error you see here first (modules, .env, permissions on log directories).

Step 3 — Harden the unit with absolute paths

sudo systemctl edit queue-worker.service

[Service]
User=deploy
Group=deploy
WorkingDirectory=/var/www/app
EnvironmentFile=-/var/www/app/.env
ExecStart=/usr/bin/node /var/www/app/workers/queue-worker.js
Restart=on-failure
RestartSec=5s

Then:

sudo systemctl daemon-reload
sudo systemctl restart queue-worker.service
sudo journalctl -u queue-worker.service -n 50 --no-pager

Step 4 — Confirm process identity

ps aux | grep queue-worker | grep -v grep
sudo systemctl show queue-worker.service -p User -p FragmentPath -p ActiveState -p SubState --no-pager

When to call a pro: secrets managers, SELinux/AppArmor denials, or multi-stage paths differ between staging and production and the journal only shows 203/EXEC with no clear file miss.

Issue 4 — Kill boot-time dependency races

Goal: start the worker only after network and data stores are actually ready.

Step 1 — Map dependencies

sudo systemctl list-dependencies queue-worker.service --no-pager
sudo systemctl status redis-server.service postgresql.service network-online.target --no-pager

Step 2 — Order and want the prerequisites

sudo tee /etc/systemd/system/queue-worker.service.d/deps.conf <<'EOF'
[Unit]
After=network-online.target redis-server.service postgresql.service
Wants=network-online.target
Requires=redis-server.service
EOF
sudo systemctl daemon-reload

Adjust service names to what your distro uses (redis.service vs redis-server.service).

Step 3 — Add an ExecStartPre readiness check (optional but practical)

sudo tee /etc/systemd/system/queue-worker.service.d/precheck.conf <<'EOF'
[Service]
ExecStartPre=/usr/bin/bash -c 'until /usr/bin/redis-cli ping | grep -q PONG; do sleep 1; done'
EOF
sudo systemctl daemon-reload
sudo systemctl restart queue-worker.service

Step 4 — Test under reboot conditions when you can

sudo systemctl reboot

After reconnect:

sudo systemctl status queue-worker.service --no-pager
sudo journalctl -u queue-worker.service -b --no-pager

When to call a pro: the worker is part of a larger mesh (containers plus host units, or NFS-mounted code) and boot order still flakes after After= wiring.

End-to-end verification checklist

Run this sequence once the unit stays green:

sudo systemctl is-active queue-worker.service
sudo systemctl is-failed queue-worker.service
sudo journalctl -u queue-worker.service --since "1 hour ago" --no-pager
sudo systemd-analyze blame | head
ps -o pid,user,pcpu,pmem,etime,cmd -C node

Confirm queue depth or job throughput with whatever tool your app already uses (Horizon, custom admin, Redis LLEN, etc.). A quiet journal plus steady process uptime beats a green status alone.

When DIY is enough vs when to book Fixwebnode

DIY is enough when you can stop the unit, read a clear error in journalctl, correct ExecStart/env, apply RestartSec and start-limit caps, and the worker stays up through a full job cycle. Stay hands-on if you own root, have a recent backup or snapshot, and can tolerate a short pause in background automation.

Book a specialist when crashes span several units, the host is already load-saturated, start limits trip every night, or you cannot safely reboot to test dependency order. Fixwebnode works as a direct remote provider for individuals, sole traders, and local operators across Australia—not a freelance marketplace. Geography and coverage notes live on the Fixwebnode service areas page; the practical next step for this class of failure is a focused remote session on the failing host.

Restore the pipeline, then get help if it still loops

Infinite systemd restarts are a configuration and crash-handling problem, not a mystery: read sudo journalctl -u queue-worker.service, stop the thrash, add clean native delays and burst caps, fix ExecStart and dependencies, and only then return the worker to automatic start. If you want a second pair of hands on the same runbook—SSH-level diagnosis, unit drop-ins, and verification—start a conversation through https://fixwebnode.com.au/website-repair-australia and outline the unit name, distro, and what the journal already shows.