Code

Enable torch.autocast with ZeRO#6993

Code

Open

tohtana wants to merge 63 commits intomasterdeepspeedai/DeepSpeed:masterfrom tohtana/support_autocastdeepspeedai/DeepSpeed:tohtana/support_autocastCopy head branch name to clipboard

+704-245

Commits

Commits on Feb 28, 2025

• Use ds-specific module id to avoid conflicts (#6847)

  Show description for a4fbc3a
  

  2 peopleauthored and

  tohtana

  committed
  a4fbc3a

  Copy full SHA for a4fbc3a

• add autocast support and ds_config item

  Show description for 9d48dc6
  tohtana

  committed
  9d48dc6

  Copy full SHA for 9d48dc6

• prepare ipg buckets for multiple dtypes

  Show description for 8957009
  tohtana

  committed
  8957009

  Copy full SHA for 8957009

• switch communication data type

  Show description for c390c76
  tohtana

  committed
  c390c76

  Copy full SHA for c390c76

• add gradscaler

  Show description for 458797d
  tohtana

  committed
  458797d

  Copy full SHA for 458797d

• Update A6000 workflows to use newer docker container - 24.09 vs 24.03 (#6967)

  Show description for 2984415
  

  loadams

  authored and

  tohtana

  committed
  2984415

  Copy full SHA for 2984415

• fix import and formatting

  Show description for 6b6a600
  tohtana

  committed
  6b6a600

  Copy full SHA for 6b6a600

• convert comm type for z3

  Show description for 2817c02
  tohtana

  committed
  2817c02

  Copy full SHA for 2817c02

• Allow NVIDIA Blackwell (#6991)

  Show description for 790c43a
  

  fabiendupont

  authored and

  tohtana

  committed
  790c43a

  Copy full SHA for 790c43a

• Update GH org references (#6998)

  Show description for fcdeda3
  

  2 peopleauthored and

  tohtana

  committed
  fcdeda3

  Copy full SHA for fcdeda3

• Update CNAME

  Show description for b476d07
  

  loadams

  authored and

  tohtana

  committed
  b476d07

  Copy full SHA for b476d07

• Update CNAME

  Show description for 72f9687
  

  loadams

  authored and

  tohtana

  committed
  72f9687

  Copy full SHA for 72f9687

• [XPU] max1100 workflow update for docker and softwares (#7003)

  Show description for a2425da
  

  Liangliang-Ma

  authored and

  tohtana

  committed
  a2425da

  Copy full SHA for a2425da

• autotp training(fix dco) (#7004)

  Show description for a896693
  

  inkcherry

  authored and

  tohtana

  committed
  a896693

  Copy full SHA for a896693

• cast dtype for allgather

  Show description for 1c1d43a
  tohtana

  committed
  1c1d43a

  Copy full SHA for 1c1d43a

• import triton files when triton is supported and installed (#6989)

  Show description for a265cad
  

  oelayan7

  authored and

  tohtana

  committed
  a265cad

  Copy full SHA for a265cad

• Update A6000 tests transformers version (#7016)

  Show description for 359c85d
  

  2 peopleauthored and

  tohtana

  committed
  359c85d

  Copy full SHA for 359c85d

• Fix ds-chat CI regression (#7015)

  Show description for 735fc2c
  

  tjruwase

  authored and

  tohtana

  committed
  735fc2c

  Copy full SHA for 735fc2c

• [Ulysses tutorial] typos (#7024)

  Show description for 1e7888c
  

  stas00

  authored and

  tohtana

  committed
  1e7888c

  Copy full SHA for 1e7888c

• fix hostname -I for macOS #6497 (#6990)

  Show description for 7d15f26
  

  6 peopleauthored and

  tohtana

  committed
  7d15f26

  Copy full SHA for 7d15f26

• Update workflows to cuda 12.4 (#7000)

  Show description for ef5a2a4
  

  6 peopleauthored and

  tohtana

  committed
  ef5a2a4

  Copy full SHA for ef5a2a4

• [ROCm] Enable fp_quantizer on ROCm (#7027)

  Show description for f1aea5d
  

  2 peopleauthored and

  tohtana

  committed
  f1aea5d

  Copy full SHA for f1aea5d

• add gds chinese blog (#7034)

  Show description for 8152824
  

  2 peopleauthored and

  tohtana

  committed
  8152824

  Copy full SHA for 8152824

• Add chinese blog for deepspeed windows, and fix format (#7035)

  Show description for c898ac5
  

  2 peopleauthored and

  tohtana

  committed
  c898ac5

  Copy full SHA for c898ac5

• AIO on ROCM (#7023)

  Show description for e3ea926
  

  3 peopleauthored and

  tohtana

  committed
  e3ea926

  Copy full SHA for e3ea926

• Control trace cache warnings (#7039)

  Show description for e946615
  

  tjruwase

  authored and

  tohtana

  committed
  e946615

  Copy full SHA for e946615

• Update CUDA compute capability to support Blackwell (#7047)

  Show description for 38e9bf3
  

  hwchen2017

  authored and

  tohtana

  committed
  38e9bf3

  Copy full SHA for 38e9bf3

• Update setup.py handling of ROCm cupy (#7051)

  Show description for acc6a1e
  

  loadams

  authored and

  tohtana

  committed
  acc6a1e

  Copy full SHA for acc6a1e

• nv-ds-chat breaks with latest transformers (#7052)

  Show description for bda1430
  

  loadams

  authored and

  tohtana

  committed
  bda1430

  Copy full SHA for bda1430

• Rename aio_thread_count to intra_op_parallelism (#7056)

  Show description for c184b16
  

  tjruwase

  authored and

  tohtana

  committed
  c184b16

  Copy full SHA for c184b16

• add autoTP training zero2 tests (#7049)

  Show description for a2b8219
  

  2 peopleauthored and

  tohtana

  committed
  a2b8219

  Copy full SHA for a2b8219

• Fix, bf16 optimizer remove dup loop (#7054)

  Show description for 9f50cde
  

  3 peopleauthored and

  tohtana

  committed
  9f50cde

  Copy full SHA for 9f50cde

• Update version.txt after 0.16.4 release (#7063)

  Show description for 71bd64e
  

  loadams

  authored and

  tohtana

  committed
  71bd64e

  Copy full SHA for 71bd64e

• fix an outdated doc wrt CUDA_VISIBLE_DEVICES (#7058)

  Show description for 5c9fd4b
  

  stas00

  authored and

  tohtana

  committed
  5c9fd4b

  Copy full SHA for 5c9fd4b

• Tecorigin sdaa accelerator (#6903)

  Show description for 20a46b7
  

  4 peopleauthored and

  tohtana

  committed
  20a46b7

  Copy full SHA for 20a46b7

• Handle special case of libuv for Windows (#7064)

  Show description for 5f68587
  

  loadams

  authored and

  tohtana

  committed
  5f68587

  Copy full SHA for 5f68587

• Update README with info on newest accelerator (#7065)

  Show description for 3f5cd1a
  

  loadams

  authored and

  tohtana

  committed
  3f5cd1a

  Copy full SHA for 3f5cd1a

• Bug Fix for offload_states API (#7050)

  Show description for b80d2d4
  

  U-rara

  and

  tohtana

  committed
  b80d2d4

  Copy full SHA for b80d2d4

• Fix TOCTOU issues, switch to fstat (#7067)

  Show description for 877c30e
  

  loadams

  authored and

  tohtana

  committed
  877c30e

  Copy full SHA for 877c30e

• config torch to avoid graph breaks caused by logger (#6999)

  Show description for 59fe7f6
  

  2 peopleauthored and

  tohtana

  committed
  59fe7f6

  Copy full SHA for 59fe7f6

• Fix meta load tensor imcompatible issue (#7073)

  Show description for 060aa5a
  

  2 peopleauthored and

  tohtana

  committed
  060aa5a

  Copy full SHA for 060aa5a

• Replace calls to python setup.py sdist with python -m build --sdist (#7069)

  Show description for 817b31d
  

  loadams

  authored and

  tohtana

  committed
  817b31d

  Copy full SHA for 817b31d

• Revert "Handle special case of libuv for Windows (#7064)" (#7076)

  Show description for f99605b
  

  loadams

  authored and

  tohtana

  committed
  f99605b

  Copy full SHA for f99605b

• Add DeepseekV3 AutoTP. (#7045)

  Show description for 57805b2
  

  Yejing-Lai

  authored and

  tohtana

  committed
  57805b2

  Copy full SHA for 57805b2

• Improve inference tutorial docs (#7083)

  Show description for 697050e
  

  loadams

  authored and

  tohtana

  committed
  697050e

  Copy full SHA for 697050e

• Pin transformers version on tests that use latest. (#7085)

  Show description for 83c9461
  

  loadams

  authored and

  tohtana

  committed
  83c9461

  Copy full SHA for 83c9461

• Update README.md with ICS '23 MoE paper link (#7087)

  Show description for c6bf7fb
  

  2 peopleauthored and

  tohtana

  committed
  c6bf7fb

  Copy full SHA for c6bf7fb

• Update parallelism for nv-torch-latest/nightly tests due to more GPUs/runner (#7086)

  Show description for 2c36865
  

  2 peopleauthored and

  tohtana

  committed
  2c36865

  Copy full SHA for 2c36865

• Remove workflows for very old torch versions (#7090)

  Show description for f2b89ec
  

  loadams

  authored and

  tohtana

  committed
  f2b89ec

  Copy full SHA for f2b89ec

• Merge branch 'master' into tohtana/support_autocast

  tohtana

  authored
  965cb2b

  Copy full SHA for 965cb2b

Commits on Mar 1, 2025

• clear reduce buffer

  Show description for 981e8e2
  tohtana

  committed
  981e8e2

  Copy full SHA for 981e8e2

• add config to set lower precision modules

  Show description for 37f77ae
  tohtana

  committed
  37f77ae

  Copy full SHA for 37f77ae

Commits on Mar 3, 2025

• fix to use comm dtype in config when autocast is disabled

  Show description for 3083d94
  tohtana

  committed
  3083d94

  Copy full SHA for 3083d94

• Merge branch 'master' into tohtana/support_autocast

  tohtana

  authored
  d688b75

  Copy full SHA for d688b75

Commits on Mar 4, 2025

• add tests

  Show description for aa60eb3
  tohtana

  committed
  aa60eb3

  Copy full SHA for aa60eb3

Commits on Mar 5, 2025

• sort dtypes

  Show description for 9529830
  tohtana

  committed
  9529830

  Copy full SHA for 9529830

• Merge branch 'master' into tohtana/support_autocast

  tohtana

  authored
  c8056a8

  Copy full SHA for c8056a8

Commits on Mar 6, 2025

• fix for cases where param and param.ds_tensor have different dtypes

  Show description for c56339c
  tohtana

  committed
  c56339c

  Copy full SHA for c56339c

Commits on Mar 7, 2025

• Merge branch 'master' into tohtana/support_autocast

  loadams

  authored
  1d6ed6e

  Copy full SHA for 1d6ed6e

Commits on Mar 8, 2025

• fix moe tests

  Show description for aa10e11
  tohtana

  committed
  aa10e11

  Copy full SHA for aa10e11

• fix tests for opt state offloading

  Show description for 26e62e4
  tohtana

  committed
  26e62e4

  Copy full SHA for 26e62e4

• fix var name

  Show description for a74fa1e
  tohtana

  committed
  a74fa1e

  Copy full SHA for a74fa1e

Commits on Mar 10, 2025

• Merge branch 'master' into tohtana/support_autocast

  tohtana

  authored
  2016995

  Copy full SHA for 2016995