Support gradient accumulation

[btbujiangjun]

Due to limited memory size, We have to set a small batch-size, gradient accumulation maybe a rigid demand

[awni]

that's something you can definitely do on a case-by-case basis, but we likely wont add it in the core library

we also have plans for gradient checkpointing and a few other levers we are exploring to reduce memory pressure.