The PyTorch Deadlock: Fixing Frozen GPU Runs in Australia

If your PyTorch job sits at 100% GPU utilisation with zero step progress, you are not waiting on a slow epoch—you are stuck in a deadlock while the meter runs. This guide is for Australian teams, sole traders, and small labs running training on cloud or on-prem GPUs who need a practical way to unstick frozen runs, verify the root cause, and decide when remote specialist help is warranted.

Fixwebnode works as a direct remote specialist for this class of failure—not a freelance marketplace. If you need hands-on diagnosis of hung distributed jobs, DataLoader stalls, or CUDA sync freezes, start via the Fixwebnode website repair and technical recovery page for Australia. Coverage and remote support geography are summarised on our service areas page.

Why a PyTorch deadlock burns budget faster than a slow model

A slow training loop still advances loss and checkpoints. A deadlock does neither. The process is alive, NCCL or CUDA may still hold devices, cloud instances stay allocated, and monitoring dashboards often look “busy” because kernels or collectives never complete. In Australia, after-hours cloud spend on stuck A100/L4/T4 jobs is a common quiet cost leak for startups and agencies who only notice when the invoice arrives.

Below you will get concrete diagnostics you can run over SSH, three distinct deadlock patterns with DIY fixes, and a clear line for when to book Fixwebnode instead of burning another night on guesswork.

Why does my PyTorch GPU training freeze with no error in Australia?

Most “frozen GPU” PyTorch runs are true deadlocks or hard stalls: processes wait forever on NCCL collectives, DataLoader worker queues, or host–device synchronisation—not on slow compute. Check whether step counters, loss logs, and checkpoints have stopped advancing while nvidia-smi still shows the process attached; if they have, treat it as a hang and isolate the wait point before restarting blindly.

Common PyTorch deadlock issues (and what they look like)

1. Distributed NCCL collective hang (multi-GPU / multi-node)

Symptoms: Training starts, prints a few ranks, then never logs another step. nvidia-smi shows processes on every GPU; CPU is mostly idle; no Python traceback. Often appears after changing world size, enabling gradient accumulation incorrectly across ranks, or a straggler rank that never enters all_reduce.

2. DataLoader worker deadlock or queue stall

Symptoms: Single-GPU or multi-GPU job freezes at the start of an epoch or after a few batches. GPU utilisation drops to near zero. Killing the job leaves orphaned worker processes. Common with num_workers > 0, persistent_workers=True, NFS/home-directory datasets, or custom collate_fn that blocks.

3. Host–device sync / autograd stall (silent CUDA hang)

Symptoms: Freeze around loss.backward(), optimizer.step(), torch.cuda.synchronize(), or logging that calls .item() / .cpu() on tensors still tied to an incomplete graph. Sometimes only fails with AMP, cudnn benchmark, or a third-party CUDA op. Unlike NCCL hangs, this can affect a single process with one GPU.

Issue 1 — Fix a distributed NCCL deadlock

Goal: prove whether ranks are waiting on a collective, capture where they blocked, and restore a runnable configuration.

Step 1 — Confirm the job is hung, not slow

watch -n 2 nvidia-smi
ps -ef | grep -E 'torchrun|python' | grep -v grep
ls -lt ./runs/*/events* 2>/dev/null | head

If GPU memory is held, step logs have not grown for many minutes, and event files are stale, treat it as a hang.

Step 2 — Enable NCCL and distributed debug on the next run

export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=COLL
export TORCH_DISTRIBUTED_DEBUG=DETAIL
export TORCH_SHOW_CPP_STACKTRACES=1
export CUDA_DEVICE_MAX_CONNECTIONS=1

torchrun --nproc_per_node=4 train.py 2>&1 | tee nccl-hang.log

Look for ranks that never enter the same collective, bootstrap timeouts, or mismatched group sizes.

Step 3 — Dump Python stacks from a live hang (do this before kill -9)

# replace with your training PIDs
for p in $(pgrep -f 'train.py'); do
 echo "===== PID $p ====="
 sudo py-spy dump --pid "$p" 2>/dev/null || sudo kill -USR1 "$p"
done

If every rank sits inside all_reduce, barrier, or broadcast, you have a collective mismatch (different code paths per rank, uneven DDP graph, or a rank that skipped a step).

Step 4 — Minimal safe recovery pattern

1. Ensure every rank executes the same sequence of collectives every step (no rank-only early return).
2. Guard logging and checkpointing so only rank 0 writes, but all ranks still hit the same barriers if you insert them.
3. Temporarily set find_unused_parameters=False only if you are sure all parameters participate; if unused params exist, DDP can hang or error—fix the model path instead of guessing.
4. For multi-node, verify clocks, free disk for scratch, and that all nodes see the same job script revision.

# sanity: single-process baseline before scaling out
python train.py --device cuda:0

# then two ranks on one node
NCCL_DEBUG=WARN torchrun --nproc_per_node=2 train.py

Step 5 — Verify

grep -E 'AllReduce|Barrier|Timeout|WARN|ERROR' nccl-hang.log | tail -n 50
# confirm step/loss lines advance for several minutes

When to call Fixwebnode: ranks still diverge after a single-node baseline works, you use custom process groups, InfiniBand/EFA fabric, or the hang only appears at full world size. Remote session work is available for operators across Australia via the landing link above.

Issue 2 — Fix DataLoader worker freezes

Goal: separate dataset I/O and multiprocessing stalls from true CUDA deadlocks.

Step 1 — Bisect workers immediately

# inside dataset / DataLoader construction for a debug run
# num_workers=0, pin_memory=False, persistent_workers=False

python - <<'PY'
import torch
from torch.utils.data import DataLoader, TensorDataset
ds = TensorDataset(torch.randn(256, 3, 224, 224), torch.randint(0, 10, (256,)))
dl = DataLoader(ds, batch_size=16, num_workers=0, pin_memory=False)
for i, (x, y) in enumerate(dl):
 print('batch', i, x.shape)
 if i >= 5: break
print('ok')
PY

If num_workers=0 runs and num_workers=4 hangs, the deadlock is in workers, shared storage, or collate—not in your optimizer.

Step 2 — Check open files, mounts, and zombie workers

df -h .
mount | grep -E 'nfs|smb|fuse'
ps -ef | grep -E 'pt_data_worker|torch.utils.data' | grep -v grep
ls -l /dev/shm
free -h

Full /dev/shm, flaky NFS home directories, and antivirus locks on shared datasets are frequent freeze triggers on cloud VMs used from Australia-based teams.

Step 3 — Apply a known-good DataLoader profile

1. Start with num_workers=2 (not 16) on a small VM.
2. Keep pin_memory=True only when the host has spare RAM and the sample tensors are CUDA-bound next.
3. Set persistent_workers=False until the pipeline is stable.
4. Wrap collate_fn in try/except logging so a single bad sample cannot block the queue silently.
5. Move datasets off network home dirs onto local SSD or object storage caches when possible.

export OMP_NUM_THREADS=1
export MKL_NUM_THREADS=1
python train.py --num-workers 2 --no-persistent-workers

Step 4 — Verify

# during training: GPU should show non-zero util between steps
nvidia-smi dmon -s u -c 10
# workers should recycle cleanly on exit
ps -ef | grep pt_data_worker | grep -v grep || echo 'no orphan workers'

When to call Fixwebnode: freezes only on production storage, iterable datasets, or multi-node shared caches; or worker orphans keep accumulating and nodes need a hardened training image.

Issue 3 — Fix CUDA synchronisation and autograd stalls

Goal: turn a silent device hang into a visible failing call stack, then remove the blocking pattern.

Step 1 — Force synchronous CUDA to surface the real op

export CUDA_LAUNCH_BLOCKING=1
export TORCH_USE_CUDA_DSA=1
python train.py 2>&1 | tee cuda-sync.log

With blocking launches, the next hang or assert usually points at the offending kernel or extension instead of a later innocent .item().

Step 2 — Audit host syncs inside the training step

1. Search training code for .item(), .cpu(), .numpy(), torch.cuda.synchronize(), and print-debugging of CUDA tensors every iteration.
2. Move metric pulls behind a schedule (for example every N steps) and only on rank 0.
3. Ensure AMP scalers and custom autograd functions do not branch differently across ranks.
4. Temporarily disable third-party CUDA ops (deform conv, fused losses, home-built extensions) to bisect.

rg -n "\.item\(|\.cpu\(|cuda\.synchronize|numpy\(" train.py models/ || true

Step 3 — Optional device-side stack (when the process is hard-stuck)

nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv
sudo cuda-gdb -p <PID> # only if installed; capture backtrace then detach
# or generate a core for offline analysis
sudo gcore <PID>

Step 4 — Clean recovery after a freeze

# last resort after stacks are captured
pkill -f train.py
sudo fuser -v /dev/nvidia* 2>/dev/null
# if the driver is wedged after repeated hangs (rare but real):
# sudo nvidia-smi --gpu-reset -i 0

Only reset GPUs when no other tenants share the device and you understand the instance will interrupt all CUDA work.

Step 5 — Verify

python - <<'PY'
import torch
print(torch.__version__, torch.version.cuda)
x = torch.randn(1024, 1024, device='cuda')
y = x @ x
torch.cuda.synchronize()
print('matmul ok', y[0,0].item())
PY

Then rerun a short training smoke test with logging every step for 50–100 iterations before restoring full batch size.

When to call Fixwebnode: hangs only under AMP + checkpointing, custom CUDA extensions, or after driver/toolkit mismatches you cannot safely change on a shared machine.

Quick environment checklist before the next overnight run

python -c 'import torch; print(torch.__version__, torch.cuda.is_available(), torch.cuda.device_count())'
nvidia-smi
nvcc --version 2>/dev/null || true
env | grep -E 'NCCL|CUDA|TORCH' | sort

• Match PyTorch build CUDA version to the installed driver capabilities.
• Pin a known-good container tag; do not silently float latest on training nodes.
• Write heartbeats (step, loss, timestamp) to disk every step during triage so freezes are obvious in logs.
• Keep one reproducible smoke config: 1 GPU, 2 workers, tiny batch, fixed seed.

When DIY is enough vs when to book Fixwebnode

DIY is enough when a single-GPU run works, num_workers=0 unblocks you, NCCL debug shows an obvious rank code-path mismatch you can patch, or removing a host sync stops the hang. Document the failing commit, keep the logs, and add a short smoke test to CI so the deadlock cannot return unnoticed.

Book Fixwebnode when the freeze is intermittent, multi-node only, tied to shared storage or fabric, or blocking a production training schedule for a business in Australia. Fixwebnode is a direct specialist provider for remote diagnosis and recovery workflows—not a place to post jobs or collect bids. If you need a guided session on hung GPU jobs, start from the Australia technical recovery landing page. For where remote support is offered, see all service areas.

Talk through your frozen GPU run

Your AI training loop is not merely slow when step counters stop and devices stay claimed—it is deadlocked, and the cloud budget keeps running until someone proves the wait point. Use the diagnostics above to capture NCCL, DataLoader, or CUDA sync evidence, apply the numbered fixes, and verify with a short smoke run before scaling out again.

If you want a specialist to review logs, stacks, and launch scripts with you remotely, open a conversation through https://fixwebnode.com.au/website-repair-australia and outline your framework version, GPU type, single- vs multi-node setup, and what last changed before the freeze.