How to limit KV cache size?

[lostmsu]

I am trying to load a very large model (70B in 4 bit) into 2x24GB GPUs. The model loads fine (takes about 20GB on each), but then the KV cache is failing to allocate enough memory (it tries to get 5GB).

How do I limit its size?

[martindevans]

Setting ModelParams.ContextSize should do the job I think.