[Bug]: Achieved compression rate with (Long)LLMLingua overshot

[cornzz]

I was evaluating how well the (Long)LLMLingua is able to achieve the requested compression rate (focusing on the rate parameter, not target_tokens) and came to these conclusions:

• For smaller prompts (< 150 tokens) barely any compression can be achieved, if any at all
• Requested compression rate is best achieved for prompts around 2000 tokens
• For longer prompts (>5000 tokens) the requested rate is overshot (or undershot)

More detailed results are below.
My question is, am doing something wrong when invoking LLMLingua, or is this behaviour normal?
I adhered to the usage examples in README.md:

Code snippet

compressor = PromptCompressor(
    model_name="NousResearch/Llama-2-7b-hf",  # or "openai-community/gpt2"
    device_map="balanced"
)
...
def compress(prompt, rate, question=""):
    if longllmlingua:
        res = compressor.compress_prompt(
            [prompt],
            question=question,
            rate=rate,
            condition_in_question="after_condition",
            reorder_context="sort",
            dynamic_context_compression_ratio=0.3,
            condition_compare=True,
            rank_method="longllmlingua",
        )
    else:
        res = compressor.compress_prompt(prompt, rate=rate)
    return res

I tested with the default Llama 2 7b as well as with GPT-2. It seems that with the smaller model the deviation overall is smaller than with the bigger model.

(Prompt lengths measured using the GPT-3.5 tokenizer)

LLMLingua with Llama 2

LLMLingua with GPT-2

LongLLMLingua with Llama 2

LongLLMLingua with GPT-2

In contrast, LLMLingua-2 adheres to the requested compression rate quite well, only slightly overshooting the requested rate:

LLMLingua-2

The prompts I used are truncated from the longest prompt in the LongBench GovReport task (link).

[cornzz]

-- Moved to separate issue: #196 --

[cornzz]

Reopening as I cannot figure out how to correctly use LLMLingua without overshooting the target compression rate.
No matter how I set iterative_size, large prompts (2K+) are overcompressed.