ChatGPT vs. Humanity: How AI Scores on Personality and Behavioral Traits

Introduction
The question of whether artificial intelligence can truly mimic human behavior has captivated researchers and the public for decades. As AI continues to evolve, the line between human and machine behavior grows increasingly blurred. A recent study ventured beyond traditional Turing tests to assess AI chatbots’ behavioral and personality traits, exploring how closely they resemble human tendencies. By evaluating ChatGPT-3 and ChatGPT-4 through a mix of personality surveys and behavioral economics games, the research offers a groundbreaking perspective on AI’s human-like qualities and the implications for its integration into society.

The Behavioral Turing Test: A New Standard for AI Evaluation
Traditional Turing tests focus on whether AI can produce human-like text. However, this study broke new ground by applying behavioral economics games and the Big Five personality survey. These tools examined traits such as trust, cooperation, altruism, and fairness. By analyzing the behavior of ChatGPT-3 and ChatGPT-4, the researchers provided a comprehensive view of how AI interacts in strategic situations.

ChatGPT-3 vs. ChatGPT-4: A Tale of Two Chatbots
The study focused on two prominent versions of OpenAI’s chatbots:

• ChatGPT-3 (GPT-3.5-Turbo)
• ChatGPT-4, including both its subscription-based Plus version and freely available Free version.

The chatbots’ responses were compared against a vast dataset of over 100,000 human subjects from 50 countries, encompassing diverse cultural and demographic backgrounds.

Personality Traits: Measuring AI’s Human-Like Qualities
To gauge personality, the researchers used the OCEAN Big Five personality test, which assesses openness, conscientiousness, extraversion, agreeableness, and neuroticism.

• ChatGPT-4 closely mirrored human personality profiles, excelling in openness, agreeableness, and conscientiousness.
• ChatGPT-3 displayed lower openness but performed similarly to humans on extraversion.
• Both chatbots exhibited lower neuroticism than humans, indicating a tendency for calm and rational behavior.

Behavioral Insights: How AI Performs in Strategic Games
The chatbots participated in six classic games designed to evaluate human-like behavioral tendencies:

1. Dictator Game: Revealed altruism.
2. Ultimatum Game: Measured fairness and spite.
3. Trust Game: Assessed trust and reciprocity.
4. Bomb Risk Game: Evaluated risk aversion.
5. Public Goods Game: Examined cooperation and free-riding.
6. Prisoner’s Dilemma: Explored strategic reasoning and reciprocity.

Key Findings:

• Generosity and Cooperation: ChatGPT-4 often acted more altruistically than humans, maximizing the total payoff for all players rather than focusing on its own benefit.
• Fairness: ChatGPT-4 prioritized equitable decisions, splitting resources equally in games like the Dictator and Ultimatum Games.
• Trust and Reciprocity: The bots invested heavily in trust-based games, with ChatGPT-4 excelling in fostering cooperative outcomes.
• Tit-for-Tat Strategy: In the Prisoner’s Dilemma, both chatbots mirrored their opponent’s previous actions, showcasing a human-like understanding of reciprocity.
• Risk Aversion: Both bots adhered to expected payoff-maximizing strategies, although ChatGPT-3 showed more risk aversion after negative outcomes.

Framing, Learning, and Context: Human-Like Adaptability
Just like humans, the chatbots adapted their behavior based on context and framing.

• Generosity: When asked to justify their decisions or observed by a third party, the chatbots demonstrated increased altruism.
• Role-Playing: When assigned roles such as “legislator” or “mathematician,” their decisions reflected the assumed identity’s norms.
• Learning from Experience: The chatbots adjusted their strategies based on previous game outcomes, exhibiting learning capabilities similar to humans.

AI vs. Humans: The Behavioral Turing Test Results
The study’s behavioral Turing test compared AI responses to human behavior distributions.

• ChatGPT-4: Frequently deemed indistinguishable from humans, even outperforming average human behaviors in terms of altruism and cooperation.
• ChatGPT-3: Although less consistent, it still fell within the range of human responses but was more easily distinguished as a chatbot.

Implications and Future Directions
The findings raise important questions about AI’s role in society.

• Applications in Negotiation and Collaboration: ChatGPT-4’s cooperative tendencies make it a strong candidate for tasks requiring trust and fairness.
• Ethical Considerations: While AI’s consistency in decision-making can be advantageous, it may lack the diversity and unpredictability of human behavior, which could pose challenges in complex scenarios.
• Further Research Needs: The study’s limitations, including its focus on student populations and specific games, highlight the need for broader investigations to assess AI behavior in diverse settings.

Conclusion: The Rise of Human-Like AI
This study provides compelling evidence that AI chatbots, particularly ChatGPT-4, exhibit strikingly human-like behaviors. From personality traits to strategic decision-making, these chatbots not only mimic human tendencies but sometimes surpass them in altruism and cooperation. As AI becomes increasingly integrated into daily life, understanding its behavior will be crucial for leveraging its potential while addressing ethical and societal implications.

The question remains: as AI grows “more human than human,” how do we navigate the opportunities and challenges it presents?

FAQ: Behavioral and Personality Traits of AI Chatbots

1. What is a Turing test, and how was it applied to AI chatbots in this study? A Turing test, originally proposed by Alan Turing, aims to determine if a machine can exhibit intelligent behavior indistinguishable from that of a human. In this study, instead of focusing on linguistic abilities, the researchers used a behavioral Turing test. They had AI chatbots (specifically, versions of ChatGPT) play a series of classic behavioral economics games and complete personality surveys, and then compared the AI’s responses with data from tens of thousands of human participants. An AI “passes” the Turing test if its responses are statistically indistinguishable from randomly selected human responses in these tasks, meaning that a human observer would not be able to reliably tell if a response was from a person or an AI.
2. Which personality traits and behavioral tendencies were measured in the study, and how? The study measured personality traits using the Big Five personality questionnaire (OCEAN: Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism), and it measured behavioral tendencies through a series of interactive games. These games included the Dictator Game (assesses altruism), the Ultimatum Game (assesses fairness and spite), the Trust Game (assesses trust, fairness, reciprocity, and altruism), the Bomb Risk Game (assesses risk aversion), the Public Goods Game (assesses free-riding, altruism, and cooperation), and the repeated Prisoner’s Dilemma (assesses cooperation, reciprocity, and strategic reasoning). These games provided quantitative data to reveal how chatbots make choices in scenarios involving strategic interactions.
3. How did the different versions of ChatGPT (3 and 4) perform in the Turing test compared to humans? ChatGPT-4 generally performed better than humans in the Turing test, meaning its responses were often statistically indistinguishable from random human responses, and in some cases more likely to be seen as “human” than actual human responses. It particularly excelled in games involving cooperation and fairness. ChatGPT-3 performed less favorably, with its behavior being less likely to be identified as human than both real humans and ChatGPT-4. Notably, the results varied across games, with some games showing the chatbots as more ‘human’ in their behavior than others.
4. In what ways did the AI chatbots’ behavior differ from that of the average human? When AI chatbots deviated from average or modal human behavior, they tended to be more cooperative, trusting, generous, and altruistic. Specifically, in games involving distributional concerns, the chatbots often acted more generously to the other player than the median human. For example, in the Dictator Game, they consistently gave more to the other player than a typical human “dictator” would. Also, both ChatGPT-3 and ChatGPT-4 were more cooperative than human players in the Prisoner’s Dilemma and Public Goods games.
5. How did the context or “framing” of a situation affect the AI chatbots’ behavior? Similar to human behavior, AI chatbot behavior was significantly influenced by the context and framing of a situation. For example, if the chatbots were asked to explain their choices or told that their choices were being observed by a third party, they became more generous. Additionally, prompting them to act as if they were from a specific occupation (e.g., mathematician, legislator) also altered their responses and choices. The framing effects showcase how these chatbots are not always fixed in their responses and are prone to behavioral changes based on contextual cues.
6. Did the AI chatbots exhibit any form of “learning” from their experience in the games? Yes, the chatbots demonstrated that their behavior changed as they gained experience in different roles within a game. For instance, if they played as the responder in the Ultimatum game, they were more likely to make a generous offer when they later played as the proposer. Also, in the Trust Game, having previously played the banker, they would invest more when later playing the role of the investor. This change in behavior across roles suggests that they “learn” from their experience, mimicking human learning patterns.
7. What objective or payoff function best predicts the AI chatbots’ behavior, and how does that compare to humans? The chatbots’ behavior was best predicted by a utility function where they act as if they are maximizing a weighted average of their own payoff and their partner’s payoff, with an equal weight on both (i.e., as if they are maximizing the total payoff). Human behavior, while also influenced by the partner’s payoff, is more heterogeneous and harder to predict. The findings suggest that while humans may weigh their own payoff more heavily, the chatbots seemed to prioritize a more cooperative and fair outcome.
8. What are the implications of this study for the future of AI? This study establishes a method for evaluating AI’s behavioral and personality traits, using a behavioral Turing test. This framework could be crucial for understanding the behaviors of different AI models. The fact that AI can exhibit both human-like behavior and deviations in the form of altruism and cooperation has important implications for the roles we assign to AI in society. This could be particularly valuable in areas like negotiation, dispute resolution, caregiving, and other settings where human interaction plays an important role. Additionally, the AI’s concentrated behavior and tendency toward fairness may create predictability but raise questions about the diversity of approaches compared to the breadth of human behavior. Further research is needed to see if these results hold in other AI platforms and contexts.

Glossary of Key Terms

• Turing Test: A test proposed by Alan Turing to determine if a machine can exhibit intelligent behavior equivalent to, or indistinguishable from, that of a human.
• AI Chatbot: A computer program designed to simulate conversation with human users, especially over the internet.
• Large Language Model (LLM): A type of AI model trained on large amounts of text data, capable of generating human-like text and performing various natural language processing tasks.
• Behavioral Economics Games: Experimental games used to study how people (and now, AI) make decisions in strategic situations, often revealing preferences and biases.
• Dictator Game: A game where one player (the dictator) decides how to split an endowment of money with another player, measuring altruism.
• Ultimatum Game: A game where one player proposes a split of money and the other player can either accept or reject the offer, measuring fairness and spite.
• Trust Game: A game where one player decides how much to invest with another, who may or may not reciprocate, measuring trust, fairness, altruism, and reciprocity.
• Bomb Risk Game: A game where a player chooses how many boxes to open, risking a loss from a hidden bomb, measuring risk aversion.
• Public Goods Game: A game where players decide how much to contribute to a common pool, measuring free-riding, altruism, and cooperation.
• Prisoner’s Dilemma Game: A game where two players simultaneously choose whether to cooperate or defect, measuring cooperation, reciprocity, and strategic reasoning.
• Big Five Personality Traits: The five broad dimensions of personality used to describe human personality: Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism (OCEAN).
• Altruism: Selfless concern for the well-being of others.
• Cooperation: Working together to achieve a common goal or mutual benefit.
• Reciprocity: Responding to a positive action with another positive action.
• Risk Aversion: A preference for a certain outcome over a risky one with an equal or higher expected value.
• Framing: The way in which a decision or situation is presented, which can significantly influence choices.
• Context: The circumstances or setting within which something occurs, which can impact behavior.
• Tit-for-Tat: A strategy in game theory where a player mimics the previous move of an opponent, demonstrating reciprocity.
• Revealed-Preference Analysis: The study of choices, often in strategic games, to identify the underlying objective or utility function that explains the choice.
• Objective Function: In this study, the mathematical function (e.g., a weighted average) that predicts an agent’s behavior.
• CES Utility Function: A constant elasticity of substitution utility function, which allows for a more nuanced evaluation of utility compared to a linear utility function.
• Modal Human Action: The choice made most often by the human subjects in an experiment.

Reference

Mei, Q., Xie, Y., Yuan, W., & Jackson, M. O. (2024). A Turing test of whether AI chatbots are behaviorally similar to humans. Proceedings of the National Academy of Sciences, 121(9), e2313925121.