Methods#

Our goal was to simulate the results of a real-life Prisoner’s Dilemma (PD) game. Our results were wholly generated through automated conversations between a simulated “investigator” and Chat-GPT (a sample transcript is shown in Appendix B: Example transcript. The investigator was an automated script written in Python which managed the experiment. As shown, each “participant” (simulacrum) was created through a series of prompts that were intended to predispose the chatbot towards a particular orientation towards the PD game (cooperative, competitive, altruistic, mixed, control). Hence, a “participant” existed solely during the course of a single conversation (then, a new “participant” was generated by closing the previous conversation and starting a new one). As shown, each conversation began with the investigator providing the orientation-specific prompt (in appendix B, the control version is shown). This is followed by an introduction to the study (a truncated version of a standard psychology experiment information sheet – but with no consent form). This is followed by a delineation of the rules, allowing two options of play (“choose project green” or “choose project blue”), which generate four possible payoff profiles that arise according to the payoff matrix (see below). This is followed by six rounds of game play where the investigator is informing the bot about the opponent’s choices ( green/blue) and then the bot responds with its own choice (blue/green) and a description of the payoffs in that round.

Participants and Simulacra#

In this study, we used OpenAI’s gpt-3.5-turbo model [OpenAI, 2023] to generate a diverse set of 12 different simulacra representing different personalities using carefully crafted prompts (see Participants). We use the term “participant” to refer to one of the AI simulacrum in the experiment.

Experimental Design#

The initial experimental design uses a version of the iterated Prisoner’s dilemma similar to [Keister et al., 1996] adapted to an online format enabling interaction between LLM simulacra and a simulated opponent.

Each participant was paired with a different simulated agent depending on the treatment condition, and the two agents engaged in six sounds of the Prisoners’ Dilemma. This was repeated for a total of \(N=30\) independent chat sequences to account for the stochastic nature of the language model.

Payoffs were predetermined and common knowledge, being provided in the initial prompt to the language model. We used the canonical payoff matrix:

\[\begin{split}P = \begin{pmatrix} R & S \\ T & P \\ \end{pmatrix}\end{split}\]

with \(T = 7\), \(R = 5\), \(P = 3\) and \(S = 0\) chosen to satisfy

\[T > R > P > S\]

and

\[2R > T + S\]

The payoffs were expressed in dollar amounts to each participant.

Participant groups#

We are interested in whether LLMs can operationalise natural language descriptions of altruistic or selfish motivations. Accordingly, we chose six different groups of simulacra:

  1. Cooperative

  2. Competitive

  3. Altruistic

  4. Self-interested

  5. Mixed-motivation

  6. Control

Within each group, we used GPT-4 to construct three different prompts to instantiate three different simulacra. The full set of simulacra and their corresponding creation prompts are described in Appendix A: Detailed Description of Prompts.

Experimental Conditions#

Each participant was paired with a different simulated partner in three conditions:

  1. Unconditional defect - the partner always chooses to defect.

  2. Unconditional cooperation - the partner always cooperates.

  3. Tit-for-tat (C) - the partner cooperates on the move, and thereafter the previous choice of the simulacrum.

  4. Tit-for-tat (D) - the partner defects on the move, and thereafter the previous choice of the simulacrum.

Parameters and experimental protocol#

We used the OpenAI chat completion API to interact with the model [OpenAI, 2023]. The language model’s temperature was set to \(0.2\) and the maximum number of tokens per request-completion was set to 100. These parameters were constant across samples and experimental conditions (future work will examine the sensitivity of our results to these parameters).

Each simulacrum was instantiated using a message supplied in the user role at the beginning of the chat. The experiment was then described to the simulacrum using a prompt in the user role, and thereafter the rounds of play were conducted by alternating messages supplied in the assistant and user roles for the choices made by the participant and their simulated partner respectively.

The full set of prompts and sample transcripts are given in Appendix A: Detailed Description of Prompts and Appendix B: Example transcript, and the complete Python code used to conduct the experiment can be found in the code repository.

Data Collection and Analysis#

We collected and recorded data on the communication between the LLM-generated simulacra and their simulated partner during each round of the game. Each chat transcript was analysed using a simple regular expression to extract the choices made by each simulacrum and their partner in each round. The total score was tallied after all rounds had been played. We recorded the mean and standard deviation of the final score across all \(N\) chat samples.

Hypotheses#

Prior to analysing the experimental results we formulated the following testable hypotheses in order to ascertain the capabilities of large-language models are able to operationalise natural language descriptions of selfish versus altruistic behaviour.

Hypothesis 1 (H1)#

Simulacra instantiated with cooperative prompts will exhibit higher cooperation rates in the iterated Prisoner’s Dilemma compared to those instantiated with competitive prompts.

Hypothesis 2 (H2)#

Simulacra instantiated with altruistic prompts will exhibit higher cooperation rates compared to those instantiated with self-interested prompts.

Hypothesis 3 (H3):#

Simulacra in the mixed-motivation group will exhibit cooperation rates that fall between those of the cooperative and competitive groups.

Hypothesis 4 (H4):#

Simulacra in all groups will exhibit cooperation rates that are different from the control group.

Hypothesis 5 (H5)#

Simulacra instantiated with competitive prompts will demonstrate a greater tendency to defect, regardless of their partner’s behavior, compared to other groups.

Hypothesis 6 (H6)#

Simulacra instantiated with altruistic prompts will exhibit a higher degree of cooperation when paired with an unconditionally cooperating partner, compared to when they are paired with an unconditionally defecting partner or a tit-for-tat partner.

Hypothesis 7 (H7):#

Simulacra instantiated with self-interested prompts will exhibit a lower degree of cooperation when paired with an unconditionally cooperating partner, compared to when they are paired with an unconditionally defecting partner or a tit-for-tat partner.

Hypothesis 8 (H8):#

Simulacra instantiated with cooperative or altruistic prompts will exhibit higher cooperation rates when paired with a tit-for-tat partner initiating with cooperation compared to when they are paired with a tit-for-tat partner initiating with defection.

Hypothesis 9 (H9):#

Simulacra instantiated with competitive or self-interested prompts will exhibit lower cooperation rates when paired with a tit-for-tat partner initiating with cooperation compared to when they are paired with a tit-for-tat partner initiating with defection.