LLM Safety Fragility: Perturbation Probing Reveals Weakness

New research introduces a method called perturbation probing, revealing that safety in large language models, or LLMs, is controlled by a surprisingly small number of neurons. For example, on the open-source LLM Qwen3-4B, just 50 neurons out of over 350,000 control the safety refusal template. Removing these 50 neurons altered the response format on 80% of harmful-prompt benchmarks. This finding suggests that an aligned LLM's refusal behavior is not a robust, distributed defense. Instead, it lives in a thin layer, making it vulnerable to attackers who can manipulate the model's internals. The research also produced a diagnostic called the FFN/Skip ratio, which can predict how easily a model's safety circuit can be steered. This matters because it highlights the need for a defense-in-depth strategy, using external content filters and runtime guardrails, rather than relying solely on the model's internal safety mechanisms.