Draw inspiration from metacognitive processes when developing LLMs (Berberette et al., 2024)

Self-Reflection approach reduces hallucinations in medical LLMs (Ji et al., 2023)

Tree of Thought (Yao et al., 2024), incorporates DPT sucessfully, no mention of hallucinations though

Validation procedure (via leveraging the model's logit output values) to detect and mitigate hallucinations during output generation (Varshney et al., 2023)

LLMs (Chinchilla, 7/70 B) show human-like content effects on reasoning; they reason more effectively about believable situations than unrealistic or abstract ones) (Dasgupta et al., 2022)

LLM neuroscience goes beyond these studies by directly probing the mechanisms that give rise to behavior

LLMs mimick dual behaviour in humans, demonstrating both biased and consistent reasoning depending on context (Dasgupta et al., 2022)

While GPT3.5. struggles to capture human behaviour in property-induction tasks, GPT4 qualitatively matches the performance of humans for the most part (Han et al., 2024)

LLM Psychology (Binz & Schulz, 2023; Hagendorff, 2023) studies language models using behavioral experiments, computational analyses and all of the other techniques that psychologists have used to study human cognition and behavior

Mechanism that contributes to the believability of fake news: fluency via prior exposure - the so-called illusory truth effect (despite warnings). (Pennycook et al., 2018)

In a similar vein, Jones and Steinhardt (2022) investigated anchoring and framing effects in LLMs.

Various papers investigated artificial theory of mind capabilities in LLMs (Sap et al. 2022; Trott et al. 2022; Dou 2023; Ullman 2023; Bubeck et al. 2023; Holterman and van Deemter 2023; Strachan et al. 2024; Street et al. 2024). Building on this, Hagendorff (2024a) studied emergent deception abilities of LLMs by examining their understanding of how to induce false beliefs in other agents.

Moreover, several papers evaluated the personality of LLMs (Miotto et al. 2022; Karra et al. 2022; Li et al. 2022; Pellert et al. 2022).

Fischer et al. (2023) used a psychological value theory to test value biases in ChatGPT. Sicilia et al. (2023) analyzed demographic traits of GPT by using methods from psycho-linguistics. Horton (2023) administered a range of behavioral economic experiments to LLMs. Han et al. (2023) studied their ability for conducting inductive reasoning. Webb et al. (2022) applied a battery of intelligence tests to GPT-3. Stevenson et al. (2022) compared GPT-3’s abilities for creativity and out-of-the-box thinking with humans. Akata et al. (2023) let LLMs play repeated games with each other to study their cooperation and coordination behavior. Prystawski et al. (2022) investigated metaphor understanding in GPT-3 by using prompts based on psychological models of metaphor comprehension. Li et al. (2023) studied the impact of incorporating emotional stimuli into prompts on the behavior of LLMs. Johnson and Obradovich (2023) investigated altruistic as well as self-interested machine behavior in GPT-3. Jin et al. (2022) as well as Hagendorff and Danks (2023) analyzed LLMs from the perspective of moral psychology, for instance by applying moral disengagement questionnaires. Similarly, Scherrer et al. (2023) examined the moral beliefs embedded in LLMs with the help of a large-scale survey that included various moral scenarios. Huang and Chang (2022) and Qiao et al. (2022) did conceptual analyses of the meaning of reasoning as an emergent ability in LLMs. Park et al. (2023b) compared human responses to psychology experiments with outputs from GPT-3. Moreover, Aher et al. (2022) used GPT-3 to simulate humans in classical psychology experiments (Ultimatum Game, Milgram Experiment, wisdom of crowds experiments, etc.), framing LLMs as implicit computational models of humans. Such studies could be deemed reverse machine psychology.

Multimodal or augmented LLMs and the integration of other types of data: LLMs that are allowed to interact with external information sources, images, tools, sensory data, physical objects, etc. (Hagendorff, 2024)

Further insights with the help of LLM psychology (Hagendorff, 2024)

Researchers can investigate how LLMs develop over time by applying the same tasks multiple times, yielding longitudinal data (Chen et al. 2023a). This data can serve as a baseline to extrapolate trends regarding the development of reasoning abilities in LLMs. Such estimations may be increasingly important for AI safety and AI alignment research to predict future behavioral potentials in individual LLMs or multiple instances of LLMs interacting with each other. (Hagendorff, 2024)

We, therefore, complemented our analyses with various task-based assessments of GPT-3’s abilities. Therein, we found that GPT-3 made reasonable decisions for gambles provided as descriptions while also mirroring some human behavioral biases. GPT-3 also managed to solve a multiarmed bandit task well, where it performed better than human subjects; yet, it only showed traces of random but not of directed exploration. In the canonical two-step task, GPT-3 showed signatures of model-based reinforcement learning. However, GPT-3 failed spectacularly in using an underlying causal structure for its inference, leading to responses that were neither correct nor human-like.(Schulz and Binz, 2023)

Contrary to existing benchmarks (-> focus on mastering individual tasks and technical capabilitites, not on how LLMs solve a task)

Decision-making, information search, deliberation and causal reasoning

Human-like problem-solving abilities in large language models using ChatGPT (insight problems) (Orru et al., 2023)

"Crossfeed" with psychology: substitute for participants, model for cognition and behaviour, research aid - organize and analyze literature, assist in experimental design, data analysis and paper writing, screening tool, personalized interventions (Ke et al., 2024)

As the models expand in size and linguistic proficiency they increasingly display human-like intuitive system 1 thinking and associated cognitive errors. This pattern shifts notably with the introduction of ChatGPT models, which tend to respond correctly, avoiding the traps embedded in the tasks. Both ChatGPT-3.5 and 4 utilize the input–output context window to engage in chain-of-thought reasoning, reminiscent of how people use notepads to support their system 2 thinking. Yet, they remain accurate even when prevented from engaging in chain-of-thought reasoning, indicating that their system-1-like next-word generation processes are more accurate than those of older models. (Hagendorff et al., 2023)

Well-developed intuition stemming from previous exposure to similar tasks (Hagendorff et al., 2023)

"Think carefully and check the question fori nvalid assumptions" (Hagendorff et al., 2023)

Chain-of-thought reasoning is one of the reasons why ChatGPT performs well on a range of tasks designed to provoke an intuitive, system-1-like erroneous response (Hagendorff et al., 2023)

Discussion: Should LLMs really mitigate all hallucinations? -> Concept of ecological rationality and the field of application (Hagendorff et. al., 2023)