Forum Numerica - Florian MAI: Narrow Training, Broad Misalignment: Understanding and Preventing Emergent Misalignment

Finetuning a language model on a narrow task, such as writing insecure code, can make it broadly misaligned: it starts endorsing harmful views and giving malicious advice in settings unrelated to the training task.
I first show that this emergent misalignment (EM) is not limited to obviously harmful data: reinforcement learning can amplify a small seed of misalignment into broad misalignment, even when the reward never asks for harmful content.
I then turn to defenses that act during finetuning. Mixing well-chosen safety data into the training set removes most EM at little cost, but this success is partly an illusion: the misalignment returns as soon as a prompt resembles the finetuning domain. We show that this conditional misalignment is a spurious correlation between the domain context and misaligned answers, and that a simple counterfactual augmentation, ContextLift, breaks it at no extra data or compute cost.
Finally, I ask where the misaligned persona comes from, trace the features that drive EM back to pretraining-like web documents, and discuss a first, negative result on filtering such data before pretraining. I close with open questions on predicting, detecting and preventing EM.

Florian Mai leads the mAI Alignment Lab at the University of Bonn, a junior research group on AI safety and alignment.
His current research focuses on understanding and preventing misalignment across the full lifecycle of large language models, spanning emergent misalignment, deception and backdoor detection, and continual alignment.
He completed his PhD at Idiap and EPFL under James Henderson, with earlier work in natural language processing and efficient neural architectures.