LLM Psychology (or Machine Psychology): cognitive biases, human-like system-1-failures, cognitive errors and content effects, anchoring and framing effects, theory of mind, creativity, cooperation and coordination behaviour, altrusitic behaviour, moral reasoning and beliefs, classical psychology experiments, social biases
LLM Psychology (Binz & Schulz, 2023; Hagendorff, 2023) studies language models using behavioral experiments, computational analyses and all of the other techniques that psychologists have used to study human cognition and behavior
In a similar vein, Jones and Steinhardt (2022) investigated anchoring and framing effects in LLMs.
Various papers investigated artificial theory of mind capabilities in LLMs (Sap et al. 2022; Trott et al. 2022; Dou 2023; Ullman 2023; Bubeck et al. 2023; Holterman and van Deemter 2023; Strachan et al. 2024; Street et al. 2024). Building on this, Hagendorff (2024a) studied emergent deception abilities of LLMs by examining their understanding of how to induce false beliefs in other agents.
Moreover, several papers evaluated the personality of LLMs (Miotto et al. 2022; Karra et al. 2022; Li et al. 2022; Pellert et al. 2022).
Fischer et al. (2023) used a psychological value theory to test value biases in ChatGPT. Sicilia et al. (2023) analyzed demographic traits of GPT by using methods from psycho-linguistics. Horton (2023) administered a range of behavioral economic experiments to LLMs. Han et al. (2023) studied their ability for conducting inductive reasoning. Webb et al. (2022) applied a battery of intelligence tests to GPT-3. Stevenson et al. (2022) compared GPT-3’s abilities for creativity and out-of-the-box thinking with humans. Akata et al. (2023) let LLMs play repeated games with each other to study their cooperation and coordination behavior. Prystawski et al. (2022) investigated metaphor understanding in GPT-3 by using prompts based on psychological models of metaphor comprehension. Li et al. (2023) studied the impact of incorporating emotional stimuli into prompts on the behavior of LLMs. Johnson and Obradovich (2023) investigated altruistic as well as self-interested machine behavior in GPT-3. Jin et al. (2022) as well as Hagendorff and Danks (2023) analyzed LLMs from the perspective of moral psychology, for instance by applying moral disengagement questionnaires. Similarly, Scherrer et al. (2023) examined the moral beliefs embedded in LLMs with the help of a large-scale survey that included various moral scenarios. Huang and Chang (2022) and Qiao et al. (2022) did conceptual analyses of the meaning of reasoning as an emergent ability in LLMs. Park et al. (2023b) compared human responses to psychology experiments with outputs from GPT-3. Moreover, Aher et al. (2022) used GPT-3 to simulate humans in classical psychology experiments (Ultimatum Game, Milgram Experiment, wisdom of crowds experiments, etc.), framing LLMs as implicit computational models of humans. Such studies could be deemed reverse machine psychology.
Contrary to existing benchmarks (-> focus on mastering individual tasks and technical capabilitites, not on how LLMs solve a task)
Binz and Schulz (2023), Dasgupta et al. (2022), Hagendorff et al. (2023), Nye et al. (2021), Talboy et al. (2023), as well as Chen et al. (2023b) applied a set of canonical experiments from the psychology of judgment and decision-making to LLMs (Linda problem, Wason selection task, Cab problem, cognitive reflection tests, semantic illusions, etc.) to test for cognitive biases and other human-like system-1 failures in the models.
Decision-making, information search, deliberation and causal reasoning
As the models expand in size and linguistic proficiency they increasingly display human-like intuitive system 1 thinking and associated cognitive errors. This pattern shifts notably with the introduction of ChatGPT models, which tend to respond correctly, avoiding the traps embedded in the tasks. Both ChatGPT-3.5 and 4 utilize the input–output context window to engage in chain-of-thought reasoning, reminiscent of how people use notepads to support their system 2 thinking. Yet, they remain accurate even when prevented from engaging in chain-of-thought reasoning, indicating that their system-1-like next-word generation processes are more accurate than those of older models. (Hagendorff et al., 2023)
Human-like problem-solving abilities in large language models using ChatGPT (insight problems) (Orru et al., 2023)