最新版PyTorch支持自动混合精度训练

Typical Mixed Precision Training

Working with Unscaled Gradients

Gradient clipping

Working with Scaled Gradients

Gradient penalty

Working with Multiple Models, Losses, and Optimizers

Working with Multiple GPUs

DataParallel in a single process

DistributedDataParallel, one GPU per process

DistributedDataParallel, multiple GPUs per process

Autocast and Custom Autograd Functions

Functions with multiple inputs or autocastable ops

Functions that need a particular dtype