Connection between Temperature and Emergent Behavior in Large Language Models

Abstract

This paper investigates the effect of the Sampling Temperature Parameter on behavioral changes, known as Emergent Behavior, in Large Language Models (LLM). Emergent Behavior was previously attributed to a direct result of an LLM’s scale. This study explores whether the Sampling Temperature Parameter has any effect on Emergent Behavior with three LLMs of various sizes. These LLMs were tested using 100 questions that belong to 5 categories, with controlled experiments at 11 Sampling Temperature settings. The findings show that LLMs tend to ignore user prompts, changing their behavior with higher temperatures. These findings suggest that temperature changes strongly affect the LLM’s behavior and induce abrupt changes, which can be attributed to emergent-like behavior. Even smaller language models exhibit emergent-like behaviors, suggesting that emergence is not solely a function of model size. This study presents a critical, underexplored area of research that is capable of triggering nonlinear performance changes. This paper broadens the theoretical understanding of LLMs’ dynamics and offers practical value in controlling their behavior for safer and more reliable use.