When LLMs Play the Telephone Game: Cumulative Changes and Attractors in Iterated Cultural Transmissions

This repo contains code for the paper "When LLMs Play the Telephone Game: Cumulative Changes and Attractors in Iterated Cultural Transmissions"

The paper is also accompanied by a website that features a Data Explorer tool, allowing to look at the simulated data used in the paper.

How to use

conda env create -f telephone_llm.yml
conda activate telephone_llm

Set-up environment variables

• To use OpenAI models with the OpenAI API, set the OPENAI_API_KEY env variable (e.g. in your .bashrc):

export OPENAI_API_KEY="<your_key>"

• To use OpenAI model with the Azure API, set the variables for each model, for example:

export AZURE_OPENAI_ENDPOINT_gpt_35_turbo_0125="<your_endpoint>"
export AZURE_OPENAI_KEY_gpt_35_turbo_0125="<your_key>"
export AZURE_OPENAI_API_VERSION_gpt_35_turbo_0125="<your_version>"

• To use huggingface models, set the HF_HOME env variable to define your cache directory:

export HF_HOME="$HOME/.cache/huggingface"

• To use huggingface gated models, set the HF_TOKEN env variable

export HF_TOKEN="<your_token>"

Reproducing the simulations comparing different models and prompts

Run the simulations

The script slurm_script.sh takes in argument a model name and a prompt name. The model name is the name of LLM that you want to use (e.g. meta-llama/Meta-Llama-3-8B). Prompt names are either "rephrase", "inspiration" or "continue".

• Regular machine:

for i in {0..99}; do SLURM_ARRAY_TASK_ID=$i bash slurm_script.sh <model_name> <prompt_name> ; done

This will sequentially launch 100 chains (20 initial texts * 5 seeds) for the specified <model_name> and <prompt_name>

• Slurm-based machine:

sbatch slurm_script.sh <model_name> <prompt_name>

Generated texts will be stored in Results/<model_name>/<prompt_name>

Run the analyses

• Perform the analyses:

  After running simulations for several models, you can compare them by running:

  python3 run_all_analyses.py -m <_model_name1_> <_model_name2_> <_model_name3_> <...> -sn data-for-plotting
  

  This will create a file 'data-for-plotting/all_data.pkl' that can be used for generating figures and running the statistical tests

Generate the figures

cd plots
python3 plot_all_figures.py <_model_name1_> <_model_name2_> <_model_name3_> <...>

Figures will be stored in Figures/figures_<current_date>

• Run the statistical models

  python3 stat_models.py <_model_name1_> <_model_name2_> <_model_name3_> <...>

Figures and tables will be stored in Results/StatModels<current_date>

Reproducing the experiments on effect of temperature

Run the simulations

• Regular machine:

for i in {0..99}; do SLURM_ARRAY_TASK_ID=$i bash slurm_script_temperature.sh <model_name> <prompt_name> ; done

This will sequentially launch 100 chains (20 initial texts * 5 seeds) for the specified <model_name> and <prompt_name>, with temperature taking values in [0, 0.4, 1.2, 1.6]

• Slurm-based machine:

sbatch slurm_script_temperature.sh <model_name> <prompt_name>

Generated texts will be stored in Results/<model_name>_<temperature_value>/<prompt_name>

Run the analyses

 python3 run_all_analyses.py -m <_model_name1_>_<_temperature_value1_> <_model_name1_>_<_temperature_value2_> <...> -sn data-for-plotting-temperature

This will create a file 'data-for-plotting-temperature/all_data.pkl' that can be used for generating figures

Generate the figures

 python3 plot_effect_of_temperature.py

Figures will be stored in Figures_temperature/figures_<current_date>

Reproducing the experiments on effect of fine-tuning

Run the simulations

To compare Base and Instruct models, follow to same steps as when comparing differnt models, but pass as <model_name> arguments both the Base and Instruct version. For instance, with "Meta-Llama-3-70B-Base" and "Meta-Llama-3-70B-Instruct"

• Regular machine:

for i in {0..99}; do SLURM_ARRAY_TASK_ID=$i bash slurm_script_temperature.sh <model_name> <prompt_name> ; done

This will sequentially launch 100 chains (20 initial texts * 5 seeds) for the specified <model_name> and <prompt_name>, with temperature taking values in [0, 0.4, 1.2, 1.6]

• Slurm-based machine:

sbatch slurm_script_temperature.sh <model_name> <prompt_name>

Generated texts will be stored in Results/<model_name>/<prompt_name>

Run the analyses

 python3 run_all_analyses.py -m <_model_name1_> <_model_name2_> <...> -sn data-for-plotting-finetuning

where <model_name1> is the Base version and <model_name2> is the Instruct version.

This will create a file 'data-for-plotting-finetuning/all_data.pkl' that can be used for generating figures

Generate the figures

 python3 plot_effect_of_finetuning.py

Figures will be stored in Figures_finetuning/figures_<current_date>