Code

Change GeGLU and add MQA.#239

Code

Open

GeorgiosSmyrnis wants to merge 7 commits intomainmlfoundations/open_lm:mainfrom gsmyrnis/geglu_testmlfoundations/open_lm:gsmyrnis/geglu_testCopy head branch name to clipboard

+76-10

Commits

Commits on Mar 21, 2024

• Geglu like griffin.

  GeorgiosSmyrnis

  committed
  b2417f2

  Copy full SHA for b2417f2

Commits on Mar 26, 2024

• Revert to original sizes and add MQA.

  GeorgiosSmyrnis

  committed
  c5b6e65

  Copy full SHA for c5b6e65

• Remove debug msg.

  GeorgiosSmyrnis

  committed
  65e7c10

  Copy full SHA for 65e7c10

• Update geglu configs.

  GeorgiosSmyrnis

  committed
  f78ae81

  Copy full SHA for f78ae81

Commits on Mar 28, 2024

• MQA compatible with qk norm.

  GeorgiosSmyrnis

  committed
  c978f10

  Copy full SHA for c978f10

Commits on Apr 1, 2024

• Add torch_attn_mqa.

  GeorgiosSmyrnis

  committed
  1df7858

  Copy full SHA for 1df7858

Commits on Apr 9, 2024

• Update some configs.

  GeorgiosSmyrnis

  committed
  da91d29

  Copy full SHA for da91d29