Script Linux Logs to Real-Time Webhook Alerts

If your Linux servers only “tell you what went wrong” after you SSH in and scroll auth.log or nginx error logs, you are flying without autopilot. This guide shows Australian sole traders, small businesses, and site operators how to convert static log files into real-time webhook alerts—architecture, copy-paste commands, and a working alerter—so failed logins, 5xx spikes, and disk pressure hit Slack, Discord, or a custom endpoint within seconds.

Fixwebnode works remote-first across Australia on exactly this class of problem: live log parsing, webhook delivery, and hardening the pipeline so noise does not drown signal. Start with the DIY runbook below, or book a specialist conversation via Fixwebnode website repair Australia when the stack is production-critical.

Why live log parsing matters more than another cron email

Batch greps and daily mail digests arrive too late for brute-force storms, certificate expiry noise, or PHP-FPM fatals that take a store offline at peak. A live pipeline watches the file (or journal), matches patterns as lines appear, rate-limits repeats, and POSTs a compact JSON payload to a webhook. That is the “autopilot”: the machine notices; you decide.

Assumptions for this guide: a Debian/Ubuntu-style host (or similar systemd Linux), shell access with sudo, and a webhook URL you control (Slack Incoming Webhook, Discord, Mattermost, or any HTTPS endpoint that accepts JSON). Remote delivery mode means every step is CLI-verifiable without a desk visit.

Why is my Linux log webhook alerter silent in Australia production hosts?

Most silent alerters fail for three boring reasons: the watcher is not attached to the real log path (rotated file or journal only), the match regex never hits the line format your distro writes, or outbound HTTPS to the webhook is blocked or mis-encoded. Fix the path and a single test curl first; only then tune patterns.

Architecture of live log parsing

Keep the design small and observable:

• Source — file under /var/log (nginx, auth, app) or the systemd journal.
• Watcher — tail -F (follows rotation) or journalctl -f.
• Filter — grep/awk or a few lines of bash/Python for severity and fields.
• Emitter — curl JSON POST to the webhook, with timeouts and exit checks.
• Supervisor — systemd service so the pipe restarts after reboot or crash.
• Guardrails — rate limit, allowlist of patterns, and a dead-letter local log of failed POSTs.

Do not parse multi-gigabyte historical dumps in the hot path. Live mode only reads new lines; backfill is a separate batch job.

Step-by-step: building an automated log alerter

These steps produce a minimal production-shaped alerter on a typical Ubuntu server. Adjust user and paths to match your host.

Prerequisites

Confirm tools and create a secrets file that is not world-readable.

sudo apt-get update
sudo apt-get install -y curl jq coreutils
sudo mkdir -p /etc/log-alerter /var/lib/log-alerter /var/log/log-alerter
sudo install -m 600 /dev/null /etc/log-alerter/webhook.env
sudo nano /etc/log-alerter/webhook.env

Put a single line in that file (no spaces around =):

WEBHOOK_URL=https://hooks.example.com/services/XXX/YYY

Step 1 — Prove the webhook accepts a test payload

set -a
. /etc/log-alerter/webhook.env
set +a
curl -sS -X POST -H 'Content-Type: application/json' \
 --max-time 10 \
 -d '{"text":"log-alerter probe from '"$(hostname)"'"}' \
 "$WEBHOOK_URL"
echo

You should see a success body from the provider (often ok) and a message in the channel. If this fails, stop—fix DNS, TLS, proxy, or the URL before any tail pipeline.

Step 2 — Identify the live log path (not a rotated stale file)

sudo ls -l /var/log/auth.log /var/log/nginx/error.log 2>/dev/null
sudo journalctl -u nginx -n 20 --no-pager
sudo tail -n 5 /var/log/auth.log

Prefer the active file that grows. On journal-only units, you will watch journalctl instead of a path under /var/log.

Step 3 — Install the parser script

sudo tee /usr/local/bin/log-alerter.sh >/dev/null <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
ENV_FILE=/etc/log-alerter/webhook.env
STATE_DIR=/var/lib/log-alerter
FAIL_LOG=/var/log/log-alerter/post-failures.log
COOLDOWN_SECONDS=${COOLDOWN_SECONDS:-120}
LOG_PATH=${LOG_PATH:-/var/log/auth.log}
PATTERN=${PATTERN:-'Failed password|Invalid user|authentication failure'}

# shellcheck disable=SC1090
set -a; source "$ENV_FILE"; set +a
: "${WEBHOOK_URL:?WEBHOOK_URL missing}"
mkdir -p "$STATE_DIR"
touch "$FAIL_LOG"

post_alert() {
 local line=$1
 local key sum now last
 key=$(printf '%s' "$line" | tr -cd '[:alnum:] _./:-' | cut -c1-120)
 sum=$(printf '%s' "$key" | sha256sum | awk '{print $1}')
 now=$(date +%s)
 last=0
 if [[ -f "$STATE_DIR/$sum.ts" ]]; then
 last=$(cat "$STATE_DIR/$sum.ts" || echo 0)
 fi
 if (( now - last < COOLDOWN_SECONDS )); then
 return 0
 fi
 local payload host ts
 host=$(hostname -f 2>/dev/null || hostname)
 ts=$(date --iso-8601=seconds)
 payload=$(jq -nc --arg text "[$host] $ts :: $line" '{text:$text}')
 if curl -sS -X POST -H 'Content-Type: application/json' \
 --max-time 10 -d "$payload" "$WEBHOOK_URL" >/dev/null; then
 echo "$now" >"$STATE_DIR/$sum.ts"
 else
 printf '%s POST_FAIL %s\n' "$ts" "$line" >>"$FAIL_LOG"
 fi
}

# -F follows the name across logrotate renames
tail -F -n 0 "$LOG_PATH" | while IFS= read -r line; do
 if printf '%s\n' "$line" | grep -E "$PATTERN" >/dev/null; then
 post_alert "$line"
 fi
done
EOF
sudo chmod 755 /usr/local/bin/log-alerter.sh

The script follows rotation with tail -F, matches an ERE pattern, de-duplicates identical lines for a cooldown window, and records failed POSTs locally.

Step 4 — Optional journal-based watcher (same webhook)

When the app only logs to systemd:

sudo tee /usr/local/bin/log-alerter-journal.sh >/dev/null <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
set -a; source /etc/log-alerter/webhook.env; set +a
UNIT=${UNIT:-nginx.service}
PATTERN=${PATTERN:-'emerg|alert|crit|error'}
journalctl -u "$UNIT" -f -o cat -n 0 | while IFS= read -r line; do
 if printf '%s\n' "$line" | grep -Eiq "$PATTERN"; then
 payload=$(jq -nc --arg text "$(hostname) journal: $line" '{text:$text}')
 curl -sS -X POST -H 'Content-Type: application/json' \
 --max-time 10 -d "$payload" "$WEBHOOK_URL" || true
 fi
done
EOF
sudo chmod 755 /usr/local/bin/log-alerter-journal.sh

Step 5 — systemd unit so the autopilot survives reboot

sudo tee /etc/systemd/system/log-alerter.service >/dev/null <<'EOF'
[Unit]
Description=Live log to webhook alerter
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
Environment=LOG_PATH=/var/log/auth.log
Environment=PATTERN=Failed password|Invalid user|authentication failure
Environment=COOLDOWN_SECONDS=120
ExecStart=/usr/local/bin/log-alerter.sh
Restart=always
RestartSec=3
User=root
# Prefer a dedicated user + ACL on the log file in locked-down environments

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now log-alerter.service
sudo systemctl status log-alerter.service --no-pager

Step 6 — Verification

# Inject a synthetic auth-style line if you use a dedicated test file,
# or temporarily widen PATTERN on a non-prod host.
sudo logger -t sshd 'Failed password for invalid user probe from 203.0.113.10'
# For file-based auth.log, append under controlled test:
echo "$(date -u +'%b %e %H:%M:%S') $(hostname) sshd[0]: Failed password for invalid user probe from 203.0.113.10" | sudo tee -a /var/log/auth.log

sudo journalctl -u log-alerter.service -n 50 --no-pager
tail -n 20 /var/log/log-alerter/post-failures.log

Confirm the channel received the alert and the service stayed active (running). Tighten PATTERN again after the probe.

Step 7 — nginx 5xx example configuration

Override environment for an error-log watcher (second unit or drop-in):

sudo systemctl edit log-alerter.service

[Service]
Environment=
Environment=LOG_PATH=/var/log/nginx/error.log
Environment=PATTERN=\[error\]|\[crit\]|upstream timed out|connect\(\) failed
Environment=COOLDOWN_SECONDS=60

sudo systemctl daemon-reload
sudo systemctl restart log-alerter.service

Common issues when scripting the autopilot

These are distinct failure modes Australian operators hit on VPS and small business Linux hosts—not generic “check the internet” tips.

Issue 1 — Alerts stop after logrotate (watcher stuck on inode)

Symptom: Alerter worked for days, then went quiet overnight; ls shows auth.log.1 grew and a new empty auth.log appeared; process still “running”.

DIY resolution:

1. Confirm rotation:

   sudo ls -l /var/log/auth.log*

2. Ensure the script uses tail -F (capital F), not tail -f.
3. Restart the unit after fixing the script:

   sudo systemctl restart log-alerter.service

4. Optional: add a logrotate postrotate hook that restarts the unit if you must use a fragile follower.
5. Verify new lines flow:

   sudo journalctl -u log-alerter.service -f

When to call Fixwebnode: Multiple rotated app logs, containers with JSON drivers, or hosts where rotation is custom and units flap.

Issue 2 — Webhook returns 400/403 or empty channel (payload shape / auth)

Symptom: curl exits non-zero; provider rejects body; Discord wants content, Slack wants text or blocks; failures pile into post-failures.log.

DIY resolution:

1. Capture the HTTP body:

   curl -sS -D- -X POST -H 'Content-Type: application/json' \
    -d '{"text":"shape test"}' "$WEBHOOK_URL"

2. Match field names to the provider (adjust jq in the script accordingly).
3. Escape control characters via jq -nc only—never hand-concatenate raw log lines into JSON.
4. Check clock skew and TLS:

   date -u; curl -vI https://hooks.example.com 2>&1 | head

5. Confirm the secret file mode stays 600 and the unit can read it.

When to call Fixwebnode: You need multi-destination fan-out, signed payloads, or corporate proxy MITM certificates on the egress path.

Issue 3 — Alert floods lock the channel (no de-duplication)

Symptom: One bad crawler fills Slack; identical “Failed password” lines repeat every second; people mute the channel and real outages are ignored.

DIY resolution:

1. Raise COOLDOWN_SECONDS in the unit environment and restart.
2. Narrow PATTERN to high-signal phrases; drop pure info lines.
3. Clear stale state if testing:

   sudo rm -f /var/lib/log-alerter/*.ts

4. Add a simple global token bucket if needed (max N posts per minute) before production cutover.
5. Keep fail2ban or firewall bans for the root cause; webhooks notify—they do not replace blocking.

When to call Fixwebnode: You want severity routing, per-pattern budgets, or correlation across nginx, PHP-FPM, and database logs without drowning ops.

Issue 4 — Service dies on logout or after apt upgrade

Symptom: Manual ./log-alerter.sh & worked until SSH closed; or upgrade restarted rsyslog and the background job vanished.

DIY resolution:

1. Never rely on nohup alone for production watchers—use the systemd unit above.
2. Check enablement:

   systemctl is-enabled log-alerter.service; systemctl is-active log-alerter.service

3. Inspect crash loops:

   sudo journalctl -u log-alerter.service -b --no-pager

4. Fix set -e failures (missing jq, bad source of env) so Restart=always has a clean ExecStart.
5. After major log daemon changes, re-validate LOG_PATH still exists.

When to call Fixwebnode: Fleet-wide units, non-root least privilege with ACLs, or SELinux/AppArmor denials you cannot decode quickly.

When DIY is enough vs when to book Fixwebnode

DIY is enough when you have one or two hosts, a single webhook, clear log paths, and you can run the probe curl plus a controlled test line during a maintenance window. The commands in this article are enough to stand up a durable tail-to-webhook autopilot and to fix rotation, payload, and flood issues yourself.

Book Fixwebnode when logs are fragmented across containers and bare metal, when silent failures already cost sales hours, when you need hardened secrets and least-privilege units, or when patterns must be tuned against real Australian traffic without false calm. Fixwebnode is a direct specialist provider for remote Linux and site recovery work—not a freelance marketplace. Service coverage is listed on the Fixwebnode service areas page; delivery for this topic is remote and digital.

Talk to Fixwebnode about your log autopilot

If static logs still hide outages until a customer complains, bring your log paths, sample lines, and webhook target to a focused remote session. We will help you validate parsing, systemd supervision, and alert hygiene end to end.

Book a conversation through the landing page for individuals, sole traders, and local operators: https://fixwebnode.com.au/website-repair-australia. Keep the DIY pipeline above as your baseline; escalate when production needs a specialist hand on the architecture of live log parsing and automated webhook alerters.