LLMs Are Not (Consistently) Bayesian: Quantifying Internal (In)consistencies of LLMs’ Probabilistic Beliefs

LLMs Are Not (Consistently) Bayesian: Quantifying Internal (In)consistencies of LLMs’ Probabilistic Beliefs

Introduction

Large Language Models (LLMs) have become the backbone of many modern AI applications, from drafting legal briefs to suggesting medical diagnoses. Their impressive ability to generate fluent, context‑aware text masks a deeper question: can these models maintain coherent probabilistic beliefs when faced with new evidence? In domains such as medicine, science, and law, decisions are rarely black‑and‑white; they hinge on updating uncertain beliefs in a rational, Bayesian manner. The recent study “LLMs Are Not (Consistently) Bayesian” introduces a novel technique for treating LLMs as information‑processing rules, allowing researchers to probe how well these systems adhere to the principles of Bayesian inference. This article summarizes the methodology, key findings, and broader implications of that work.

What Happened

Researchers designed a battery of controlled experiments that presented LLMs with classic Bayesian reasoning tasks. Each task began by asking the model to assign a probability to a hypothesis (the prior) based on a textual description of the world. Then, new evidence was introduced, and the model was asked to update its belief (the posterior). By comparing the model’s reported posterior to the mathematically correct Bayesian update, the team could quantify the degree of (in)consistency.

The experiments covered a range of scenarios: simple coin‑flip likelihoods, medical diagnostic tests with known sensitivity and specificity, and more complex causal chains involving multiple pieces of evidence. Across these tasks, the models displayed a spectrum of behavior. Some, like smaller GPT‑2 variants, often over‑reacted to new evidence, inflating posterior probabilities far beyond Bayesian predictions. Others, such as GPT‑4, sometimes under‑reacted, leaving the posterior too close to the prior. Crucially, no model consistently matched the Bayesian benchmark across all tasks.

These findings were not isolated incidents. By repeating each task with varied phrasings, different random seeds, and across multiple model sizes, the researchers demonstrated that the inconsistencies are systematic rather than anecdotal. The variability suggests that LLMs encode probabilistic information in a way that is highly sensitive to surface form and context, rather than adhering to a stable internal belief state.

Key Details

The core technique—treating an LLM as an information‑processing rule—relies on extracting the model’s implicit probability distribution from its token logits. By prompting the model with “What is the probability that …?” and interpreting the softmax output over a calibrated set of answer tokens, the researchers obtained a numeric belief that could be directly compared to Bayesian calculations. Calibration steps, such as temperature scaling and logit adjustment, were applied to mitigate known over‑confidence issues in raw logits.

One striking pattern emerged: models tended to respect the qualitative direction of evidence (i.e., they moved beliefs in the right direction) but frequently mis‑estimated the magnitude of the shift. For example, in a classic medical test scenario where a disease has a 1 % prevalence and a test is 99 % accurate, the correct posterior after a positive result is about 50 %. GPT‑3.5 often reported a posterior near 80 %, while GPT‑4 hovered around 30 %. Both are qualitatively correct (the belief increases) but quantitatively far from the Bayesian ideal.

Another specificity involved “order effects.” When evidence was presented sequentially, the order in which facts were introduced altered the final posterior, even though Bayesian theory dictates that the order should be irrelevant. This reveals that LLMs do not maintain a stable, order‑invariant belief state; instead, each prompt reshapes the internal representation anew, leading to path‑dependent updates.

Background

Bayesian inference has long been the normative framework for rational belief updating. It provides a mathematically rigorous rule for combining prior knowledge with new data, ensuring that beliefs remain coherent across multiple pieces of evidence. In artificial intelligence, Bayesian methods underpin many probabilistic models, from hidden Markov models to modern deep generative networks. However, LLMs are trained via next‑token prediction on massive text corpora, not through explicit probabilistic reasoning objectives. Their “knowledge” emerges implicitly from patterns in language, raising doubts about whether they can be expected to follow Bayesian logic without additional constraints.

Recent work has begun to bridge this gap by fine‑tuning LLMs on tasks that require explicit probability estimation, or by augmenting them with external reasoning modules. The study in question, however, takes a diagnostic stance: rather than modifying the models, it measures how far their native behavior deviates from Bayesian expectations. This approach provides a baseline for future improvements and highlights the inherent limitations of treating raw LLM outputs as reliable probabilistic judgments.

Why It Matters

In high‑stakes domains, decisions are often made by aggregating uncertain evidence. If an AI system systematically over‑estimates the impact of new data, it may trigger unnecessary interventions (e.g., overtreatment in medicine). Conversely, under‑estimation can lead to missed diagnoses or legal oversights. Understanding that LLMs are not consistently Bayesian alerts practitioners to the need for calibration, post‑processing, or complementary decision‑support mechanisms before deploying these models in critical settings.

Beyond safety, the findings have theoretical implications for AI alignment. A core tenet of aligned AI is that systems should act in accordance with human values and reasoning standards. Bayesian rationality is a cornerstone of human statistical reasoning; deviations from it suggest a misalignment between the model’s internal “beliefs” and the normative standards we expect. Quantifying these gaps provides a concrete metric for alignment research and a target for future model design.

What Happens Next

Future research will likely pursue two complementary avenues. First, methodological refinements—such as more sophisticated prompting strategies, chain‑of‑thought elicitation, or explicit belief‑state extraction—could reduce surface‑form sensitivity and bring LLM updates closer to Bayesian norms. Second, architectural innovations may embed probabilistic reasoning directly into the training objective, for instance by jointly optimizing next‑token prediction and likelihood‑based belief updates.

In parallel, practitioners are expected to adopt safety layers: calibrating model outputs with external Bayesian calculators, employing ensemble methods to average out order effects, or integrating LLMs with symbolic reasoning engines that enforce consistency. As the community iterates on these solutions, the ultimate goal is to produce AI systems that can both generate fluent language and uphold the rigorous standards of rational belief updating required in real‑world decision making.

Conclusion

The investigation into LLMs’ Bayesian consistency reveals a nuanced picture: while these models excel at linguistic generation, their internal probabilistic beliefs are neither stable nor reliably Bayesian. The novel technique of treating LLMs as information‑processing rules uncovers systematic over‑ and under‑updates, order dependencies, and sensitivity to prompt phrasing. These inconsistencies matter profoundly for applications where uncertainty must be managed carefully. By quantifying the gap between LLM behavior and Bayesian rationality, researchers have laid the groundwork for both diagnostic tools and future model improvements, steering the field toward AI systems that are not only articulate but also epistemically sound.