Typical Mixed Precision Training Working with Unscaled Gradients Gradient clipping Working with Scaled Gradients Gradient penalty Working with Multiple Models, Losses, and Optimizers Working with Multiple GPUs DataParallel in a single process DistributedDataParallel, one GPU per process DistributedDataParallel, multiple GPUs per process Autocast and Custom Autograd Functions Functions with multiple inputs or autocastable ops Functions that need a particular dtype