torch.cuda.amp > apex.amp

For a while now my main focus has been moving mixed precision functionality into Pytorch core. It was merged about a month ago: https://pytorch.org/docs/master/amp.html https://pytorch.org/docs/master/notes/amp_examples.html and is now usable via master or nightly pip/conda packages. (Full features did not make the 1.5 release, unfortunately.) `torch.cuda.amp` is more flexible and intuitive, and the native integration brings more future optimizations into scope. Also, `torch.cuda.amp` fixes many of `apex.amp`'s known pain points. Some things native amp can handle that apex amp can't: - Guaranteed Pytorch version compatibility, because it's part of Pytorch - No need to build extensions - Windows support - Bitwise accurate [saving](https://pytorch.org/docs/master/amp.html#torch.cuda.amp.GradScaler.state_dict)/[restoring](https://pytorch.org/docs/master/amp.html#torch.cuda.amp.GradScaler.load_state_dict) - [DataParallel](https://pytorch.org/docs/master/notes/amp_examples.html#dataparallel-in-a-single-process) and intra-process model parallelism (although we still recommend [torch.nn.DistributedDataParallel](https://pytorch.org/docs/master/notes/amp_examples.html#distributeddataparallel-one-gpu-per-process) with one GPU per process as the most performant approach) - [Gradient penalty](https://pytorch.org/docs/master/notes/amp_examples.html#gradient-penalty) (double backward) - `torch.cuda.amp.autocast()` has no effect outside regions where it's enabled, so it should serve [c

Posted by mcarilli on Apr 30, 2020, 4:32 PM

Tags

• support (Other)

Related issues

• some bike vendor told me that the biggest problem with regular to ebike conversion was that regular bike frames were not strong enough to...
• I think of this as the kiosk economy.
• What a goofy comment.
• There's lots of evidence for positive influences of Caffeine on the body (some negative as well), especially the brain.
• Compose support in WSLc
• [RFC] Roadmap to v2.0