Code

Add SophiaG optimizer#844

Code

Open

PeanutButterRat wants to merge 18 commits intofacebookresearch:mainfacebookresearch/fairseq2:mainfrom PeanutButterRat:sophia-optimizerPeanutButterRat/fairseq2:sophia-optimizerCopy head branch name to clipboard

+339-1

Commits

Commits on Oct 28, 2024

• Create sophia.py

  PeanutButterRat

  committed
  d7feed2

  Copy full SHA for d7feed2

• Register SophiaG as a usable optimizer

  PeanutButterRat

  committed
  72e4672

  Copy full SHA for 72e4672

• Add differentiable default

  PeanutButterRat

  committed
  daa4e61

  Copy full SHA for daa4e61

Commits on Oct 30, 2024

• Add finetuning recipe for SophiaG

  PeanutButterRat

  committed
  6205ed3

  Copy full SHA for 6205ed3

• Add optimizer field to SophiaG recipe

  PeanutButterRat

  committed
  5ed973c

  Copy full SHA for 5ed973c

• Move SophiaG recipe to instruction_finetune.py

  PeanutButterRat

  committed
  f8d1eaa

  Copy full SHA for f8d1eaa

• Format and add type hints

  PeanutButterRat

  committed
  3f28592

  Copy full SHA for 3f28592

Commits on Nov 1, 2024

• Join sophiag and _single_tensor_sophiag

  PeanutButterRat

  committed
  f006e3b

  Copy full SHA for f006e3b

• Add license header

  PeanutButterRat

  committed
  c7b5ce2

  Copy full SHA for c7b5ce2

• Add docstrings to SophiaG and SophiaGConfig

  PeanutButterRat

  committed
  e7f3338

  Copy full SHA for e7f3338

Commits on Nov 6, 2024

• Add hessian update and k hyperparameter

  PeanutButterRat

  committed
  e5cdff4

  Copy full SHA for e5cdff4

• Fix usage of k hyperparameter

  PeanutButterRat

  committed
  e3c81cd

  Copy full SHA for e3c81cd

Commits on Nov 9, 2024

• Add type hints and format SophiaG

  PeanutButterRat

  committed
  d1e6d57

  Copy full SHA for d1e6d57

• Remove unused imports

  PeanutButterRat

  committed
  50fd1fb

  Copy full SHA for 50fd1fb

Commits on Nov 11, 2024

• Explicitly cast step to an int

  PeanutButterRat

  committed
  8e13174

  Copy full SHA for 8e13174

• Create test_sophiag.py

  PeanutButterRat

  committed
  5f30548

  Copy full SHA for 5f30548

Commits on Nov 14, 2024

• Add SophiaG unit test

  PeanutButterRat

  committed
  8ae5403

  Copy full SHA for 8ae5403

• Change SophiaG test to check weights

  PeanutButterRat

  committed
  148f647

  Copy full SHA for 148f647