Batch size beyond 16 is throwing an error

[venkycreator]

Earlier I used to do inference using the past code till batch size of 64 , but don't know now it is not even running till batch size of 16 what things should i need to change in order to run as like earlier

[regisss]

Please give more information:

• what code are you talking about?
• the commands you ran?
• what are the versions of the library you used?

[venkycreator]

Earlier there is a code (for text generation run_generation.py) which doesn't have flash attention and also there is a argument called attn_softmax_bf16 , i exactly dont know what is the reason that i was unable to run batch size beyond 16

The commands that were there in the readme were only executed

And below attached file is the error that i got while running
error.txt