Which global batch size?

[cerisara]

Gradient accumulation is an important parameter to tune to find the best compromise between throughput and convergence speed:

• small batch: bad throughput because of internodes latency, but "faster" SGD
• large batch: high throughput but slower SGD

Llama2 uses 4M tokens for pretraining == 4096 x 1000
GPT3 uses 250k tokens
My script uses initially 1024x256 = 260k tokens, or 2048x256 = 500k tokens

See figure 1 from https://arxiv.org/pdf/1812.06162.pdf

See tuning playbook from Google: https://github.com/google-research/tuning_playbook
==> batch size shall be tuned wrt speed, not performance, so they recommend using the largest possible batch size on the hardware! ==> this conclusion is true for the raw batch size, but when using gradient accumulation, it's more about finding the best compromise between convergence speed and throughput.

Global batchsize may be "ramp up", from 192 to 2048 (+16 every 9_765_625 samples): https://github.com/huggingface/llm_training_handbook/tree/main/hparams