如何使vllm下测评结果更接近hf？

[miaoyuxun]

vllm中的测评结果在各个模型上都比hf差一或两个点
评测代码逻辑大致如下：

hf

import json

with open('diff_inputs.json','r',encoding='utf-8') as file:
    inputs = json.load(file)

from transformers import AutoModelForCausalLM,AutoTokenizer
import torch
model = AutoModelForCausalLM.from_pretrained(
    ".../llama-2-7B"
    ).bfloat16()
tokenizer = AutoTokenizer.from_pretrained(".../llama-2-7B")
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

def prepare_inputs(input_text):
    input_ids = tokenizer(input_text, return_tensors="pt").input_ids
    max_new_tokens = 128
    context_length = 4096 - max_new_tokens

    # truncate from the middle
    if input_ids.shape[1] > context_length:
        half = context_length // 2
        input_ids = torch.cat(
            [input_ids[:, :half], input_ids[:, -half:]], dim=-1
        )

    inputs = dict()
    inputs["input_ids"] = input_ids.to(model.device)
    inputs["max_new_tokens"] = max_new_tokens
    inputs["num_beams"] = 1
    inputs["do_sample"] = False

    inputs["eos_token_id"] = [
        tokenizer.eos_token_id,
        tokenizer.encode("\n", add_special_tokens=False)[-1],
        tokenizer.encode(".\n", add_special_tokens=False)[-1],
    ]

    return inputs

for input in inputs:
    input = prepare_inputs(input)
    input_len = input["input_ids"].shape[1]
    outputs = model.generate(**input)[:, input_len:]
    output_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
    print(output_text)

vllm

import json
import vllm

with open('diff_inputs.json','r',encoding='utf-8') as file:
    inputs = json.load(file)

from transformers import AutoModelForCausalLM,AutoTokenizer
import torch
llm = vllm.LLM(".../llama-2-7B", max_seq_len_to_capture=4096, gpu_memory_utilization=0.8, dtype='bfloat16')

tokenizer = AutoTokenizer.from_pretrained(".../llama-2-7B")
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

def truncate_inputs(input_texts):
    inputs = []
    for input_text in input_texts:
        input_ids = tokenizer(input_text, return_tensors="pt").input_ids
        max_new_tokens = 128
        context_length = 4096 - max_new_tokens

        # truncate from the middle
        if input_ids.shape[1] > context_length:
            half = context_length // 2
            input_ids = torch.cat(
                [input_ids[:, :half], input_ids[:, -half:]], dim=-1
            )

        inputs.append(tokenizer.decode(input_ids[0], skip_special_tokens=True))

    return inputs

stop_token_ids = [
                tokenizer.eos_token_id,
                tokenizer.encode("\n", add_special_tokens=False)[-1],
                tokenizer.encode(".\n", add_special_tokens=False)[-1],
            ]
sampling_params = vllm.SamplingParams(
            max_tokens=128,
            temperature=0.0,
            stop_token_ids=stop_token_ids,
            #repetition_penalty=1.2,
        ) 

inputs = truncate_inputs(inputs)
outputs = llm.generate(inputs, sampling_params=sampling_params)
output_texts = list(map(lambda x: list(map(lambda y: y.text, x.outputs)), outputs))
print(output_texts)

结果不同部分如下（narrativeqa），output1为hf，output2为vllm：
output1: He is living in a boarding house.
output2: He is living in a hotel.
output1: In a cave on the side of a mountain.
output2: In a cave on Atlas' mountain.
output1: 10 years
output2:
output1: A thief.
output2: A slave.
output1: The teens faces were frozen in a look of terror.
output2: They were pale and their eyes were wide open.
output1: Soames is upset with Beerbohm because he is not included in the list of writers who are to be included in the book.
output2: Soames is upset with Beerbohm because Beerbohm has written a book about the literature of the eighteen-nineties and has not included Soames in it.
output1: Falder was worried about Ruth because he was afraid that she might tell the truth about him to the police.
output2: Falder was worried about Ruth because he was afraid that she would tell the police about his criminal past.
output1: The painting was of the Statue of Liberty.
output2: The Ghost of Slimer
output1: The teens faces were frozen in a look of terror.
output2: They were pale and their eyes were wide open.
output1: The Witch creates a Witch.
output2: She creates a Witch.
output1: Reiko discovers that the tape is a copy of the original tape.
output2: Reiko discovers that the tape is a copy of the original.
output1: Holmes had been sitting in the window, and the housekeeper had been standing outside.
output2: Holmes had been sitting on the floor, and his feet were wet.
output1: He is living in a boarding house.
output2: He is living in a hotel.
output1: The video that was shown to them.
output2: The teenagers were discussing the rumor that if you watch a video called "Ring" and then you die within a week, you will come back to life.
output1: Alabama was a prostitute.
output2: She was a prostitute.
output1: Jezzie was a Vietnamese girl who was raped by a group of American soldiers.
output2: Jezzie was a Vietnamese girl who was raped by a soldier.
output1: Nupton thought that Soames was a figment of my brain.
output2: He thought that Soames was a figment of the author's imagination.
output1: He discovered that she had been unfaithful to him.
output2: He discovered that she had been having an affair with a man named Gorenflot.
output1: The message is that Reiko is the next victim of the curse.
output2: The message is that Reiko will die in one week.
output1: The American Revolutionary War.
output2: In the Second War of Independence.
output1: 1927
output2: 2419 A.D.
output1: He believes that artists laugh at his work.
output2:
output1: Because he wants to marry the Baron's daughter.
output2: Baron Henry attacks Castle Drachenhausen because he wants to marry the Baron's daughter.
output1: The Methodist minister, Reverend Simms.
output2: The Mayor, Joe Clark.
output1: Soames's presence in the future affected others by making them aware of the possibility of time travel.
output2: Soames's presence in the future affected others in a variety of ways.
output1: In a cave on the top of Atlas' mountain.
output2: In a cave on the side of a mountain.
output1: Holmes had been sitting in the window, and the housekeeper had been standing outside.
output2: Holmes had been sitting on the floor, and his feet were wet.
output1: When she sees the videotape of her father.
output2: When she sees the videotape.

其他数据集分数如下：
hf
Dataset Metrics
+-----------------+---------+
| Dataset | Score |
|-----------------+---------|
| narrativeqa | 20.6 |
| qasper | 25.36 |
| multifieldqa_en | 32.27 |
| hotpotqa | 30.52 |
| 2wikimqa | 27.07 |
| musique | 10.32 |
| gov_report | 26.41 |
| qmsum | 20.72 |
| multi_news | 3.31 |
| trec | 65.5 |
| triviaqa | 87.12 |
| samsum | 34.32 |
| lcc | 66.68 |
| repobench-p | 59.4 |
| � |
| Avg. | 36.4 |
+-----------------+---------+
vllm
Dataset Metrics
+-----------------+---------+
| Dataset | Score |
|-----------------+---------|
| narrativeqa | 20.6 |
| qasper | 25.72 |
| multifieldqa_en | 31.69 |
| hotpotqa | 30.72 |
| 2wikimqa | 26.57 |
| musique | 11.01 |
| gov_report | 18.77 |
| qmsum | 8.55 |
| multi_news | 1.97 |
| trec | 65 |
| triviaqa | 88.51 |
| samsum | 25.5 |
| lcc | 69.12 |
| repobench-p | 62.57 |
| � |
| Avg. | 34.7357 |
+-----------------+---------+

[bys0318]

你好，看起来你在hf和vllm的inference中均用了greedy decoding，请问你用hf和vllm中打多次输出一样吗？

[miaoyuxun]

你好，看起来你在hf和vllm的inference中均用了greedy decoding，请问你用hf和vllm中打多次输出一样吗？

我用上面贴的代码测了五次，hf是一样的，vllm出现了2种不同，具体如下：（前面贴的结果不是用这两份代码跑的，所以稍有不同）
[[' He is living in a hotel.'], [" In a cave on Atlas' mountain."], [''], [' A slave.'], [' They were pale and their eyes were wide open.'], [' Soames is upset with Beerbohm because he is not included in the list of writers who are to be included in the book.'], [' Falder was worried about Ruth because he was afraid that she would tell the police about his criminal past.'], [' The Ghost of Slimer.'], [' They were pale and their eyes were wide open.'], [' She creates a Witch.'], [' Reiko discovers that the tape is a copy of the original.'], [" Holmes had been sitting on the floor and had gotten up to go to the window. He had forgotten to wipe his feet and had tracked mud into the housekeeper's room."], [' He is living in a hotel.'], [' The teenagers were discussing the rumor that if you watch a video called "Ring" and then you die within a week, you will come back to life.'], [' She was a prostitute.'], [' Jezzie was a Vietnamese girl who was raped by a group of American soldiers.'], [" He thought that Soames was a figment of the author's imagination."], [' He discovered that she had been having an affair with a man named Gorenflot.'], [' The message is that Reiko will die in one week.'], [' In the Second War of Independence.'], [' 2419 A.D.'], [''], [" Baron Henry attacks Castle Drachenhausen because he wants to marry the Baron's daughter."], [' The Mayor, Joe Clark.'], [" Soames's presence in the future affected others in a variety of ways. "], [' In a cave on the side of a mountain.'], [" Holmes had been sitting on the floor and had gotten up to go to the window. He had forgotten to wipe his feet and had tracked mud into the housekeeper's room."], [' When she sees the videotape of her father.']]（vllm）
[[' He is living in a hotel.'], [" In a cave on Atlas' mountain."], [''], [' A slave.'], [' They were pale and their eyes were wide open.'], [' Soames is upset with Beerbohm because he is not included in the list of writers who are to be included in the book.'], [' Falder was worried about Ruth because he was afraid that she would tell the police about his criminal past.'], [' The painting was of the Statue of Liberty.'], [' They were pale and their eyes were wide open.'], [' She creates a Witch.'], [' Reiko discovers that the tape is a copy of the original.'], [' Holmes had been sitting on the floor, and the housekeeper had been standing over him.'], [' He is living in a hotel.'], [' The teenagers were discussing the rumor that if you watch the video, you will die within a week.'], [' She was a prostitute.'], [' Jezzie was a Vietnamese girl who was raped by a soldier.'], [" He thought that Soames was a figment of the author's imagination."], [' He discovered that she had been having an affair with a man named Gorenflot.'], [' The message is that Reiko is the next victim of the curse.'], [' In the Second War of Independence.'], [' 2419 A.D.'], [''], [" Baron Henry attacks Castle Drachenhausen because he wants to marry the Baron's daughter."], [' The Mayor, Joe Clark.'], [" Soames's presence in the future affected others in a variety of ways. "], [" In a cave on Atlas' mountain."], [' Holmes had been sitting on the floor, and the housekeeper had been standing over him.'], [' When she sees the videotape of her father.']]（相同的vllm）
['He is living in a hotel.\n', 'On the top of a mountain.\n', '\n', 'A thief.\n', 'The teens faces were frozen in a look of terror.\n', 'Soames is upset with Beerbohm because he is not included in the list of writers who are to be included in the book.\n', 'Falder was worried about Ruth because he was afraid that she would tell the police about his criminal past.\n', 'The Ghost of Slimer\n', 'The teens faces were frozen in a look of terror.\n', 'The Witch creates a Witch.\n', 'Reiko discovers that the tape is a copy of the original tape.\n', "Holmes had been sitting on the floor and had gotten up to go to the window. He had forgotten to wipe his feet and had tracked mud into the housekeeper's room.\n", 'He is living in a hotel.\n', 'The video that was shown to them.\n', 'She was a prostitute.\n', 'Jezzie was a Vietnamese girl who was raped by a soldier.\n', 'Nupton thought that Soames was a figment of my brain.\n', 'He discovered that she had been having an affair with a man named Gorenflot.\n', 'The message is that Reiko is the next victim of the curse.\n', 'The American Revolutionary War.\n', '1927\n', 'He believes that artists laugh at his work.\n', "Because he wants to marry the Baron's daughter.\n", 'The Methodist minister, Reverend Simms.\n', "Soames's presence in the future affected others in a variety of ways. \n", 'On the top of Atlas Mountain.\n', "Holmes had been sitting on the floor and had gotten up to go to the window. He had forgotten to wipe his feet and had tracked mud into the housekeeper's room.\n", 'When she sees the videotape of her father.\n']（hf）