Amazon EKS + NVRx: Fault-Tolerant Distributed Training at 99%+ Efficiency

NVRx stands for NVIDIA Resiliency Extension, a toolkit that plugs into distributed training frameworks like PyTorch FSDP to improve fault tolerance through async checkpointing, in-process restart, and fast recovery from GPU faults.

Traditional recovery restarts the whole job from the last checkpoint on any failure, wasting cluster time. NVRx restarts only the affected process within the same job in seconds, while checkpoint writes happen in the background during training.

Teams training large language or vision models on rented or cloud GPU clusters should evaluate it. Cutting downtime lowers hourly costs and improves cluster utilization, which matters for budget-constrained or deadline-driven organizations.

You need PyTorch FSDP running on EKS with NVRx integrated, ft_launcher for in-job restart management, and checkpoint storage configured for high-throughput async writes. Full details are in the official AWS blog post.