How to evaluate commonsense locality?

[lliutianc]

Hi,

Thanks for maintaining the repo!

After reading through the codes and your paper: Editing Large Language Models: Problems, Methods, and Opportunities, I am not sure how to evaluate the locality results shown in Table 4 in the paper. The dataset looks like "locality" but I didn't find an example of using it properly. Can you share a minimal example?

[littlefive5]

Hi there, you can find it in Appendix B.3.3. For the computation, we combine the question and choice as the input, compute the loss between different choices, and select the one with the minimum loss as the answer.

[zxlzr]

Hi buddy, do you have any further questions?

[lliutianc]

Thanks for your answer. By loss do you mean using PPL? BTW, were distracting neighbor and other attribution computed in the same logic?

[littlefive5]

Yes, PPL.
But, distracting neighbor and other attribution are computed by the token-level exact metric, this can be calculated by our evaluation code.
This means you can directly use our code to get results of distracting neighbor and other attribution but you need to evaluate the reasoning task by your own.