Preference Alignment Can Be Done at Inference Time! 256-16384 Control Parameters Achieve DPO-Level Performance

The conventional path for preference alignment is methods like RLHF and DPO that optimize model weights through feedback. But this study shifts attention to another link: the computation supporting the target may already exist; the question is whether it stably dominates generation. The paper proposes Direct Hidden-State Alignment (DHSA), which treats the inference-time hidden states carrying preference computation as the direct adaptation space, while base weights remain frozen.

This shift means the object of alignment changes from "model parameters" to "states during inference." The paper's rationale is that external preferences describe final behavior, while the model internally runs layer-by-layer computation; merely finding components with "large influence" on generation is not enough to determine whether they are promoting the target or counteracting it.

To answer "how preferences form during inference," the team proposes the Residual Competition Map (RCM), which maps externally specified preferences back to native inference-time computation. Its concrete operation, using RCM-Zero as an example: for the same input, compare subsequent computation after a component's normal write versus zeroed write, and let the model continue generating. If the target behavior is more dominant relative to alternative behaviors under the normal write, this native effect supports the target; otherwise it counteracts the target. The difference between the two gives the direction and strength of the effect.

Using RCM, the team observed across different preference tasks: when a single prediction forms, effects supporting the target and counteracting the target can coexist; the paper calls this residual competition, and its prevalence varies by task. Some components on average support the target, some on average counteract it; the same component may also change its direction of effect depending on the input. This competition also appears in samples where the model has already given the target answer.

The paper also gives a knowledge conflict case: the model originally insisted on its own knowledge, and merely zeroing the output of a single attention head made it switch to answering according to the given material. Neither the input nor the base weights changed; a single local state change altered the generated answer. This case is an author self-report, used to illustrate the impact of local state changes on final behavior.

An easily overlooked finding is: DPO training significantly reorganizes internal effects and can weaken some counteracting effects, but these effects may still remain. The paper does not provide a clear quantification of the degree of retention.

This constitutes a qualification of the default assumption that "alignment equals weight rewriting": output conforming more to preferences does not mean internal effects are already consistent. Conversely, if counteracting effects may still remain after DPO, then optimization at the weight level alone may not fully eliminate internal competition—this is an inference from the paper's observations, not a direct conclusion of the paper.

Causal Activation State Transition (CAST) is the paper's concrete implementation. RCM selects a small number of preference-related interfaces; CAST learns local adjustments through constant vectors or low-rank mappings, altering states before components write back to the residual stream, letting the model continue generating from there.

In terms of training, this paper trains CAST with a DPO-style paired preference objective, updating only the controller, not the base weights. The controller is decoupled from the base weights, can be enabled or removed at inference time, and can also be stacked onto a DPO model. The intervention timing can be chosen to cover the entire prompt processing and generation process, or to intervene only during the prompt processing (prefill) stage, influencing subsequent generation by changing prompt computation and cached states.

The paper claims that some settings require intervention only during the prefill stage, but does not specify which settings. The paper also claims that transferring a controller learned on SFT to a DPO model, with only strength calibration, can further improve performance, but does not give the specific improvement magnitude. Both points are author self-reports.

Experiments are conducted on ConFiQA, IMDb, and TL;DR, respectively simulating RAG knowledge conflict, examining sentiment control, and open-ended summarization. Models include GPT-2-large (0.774B), GPT-J (6B), Qwen2 (7B), Llama-3 and its LION version (8B); RCM localization experiments also cover Qwen2.5 (14B).

The paper claims that across the three task types, CAST achieves alignment levels competitive with DPO at deployment using only 256–16,384 control parameters. Selection-position comparisons are used to support RCM's guiding role for intervention.

It must be made clear that this performance conclusion is an author self-report and lacks independent third-party verification. The paper also does not give a unified quantitative conclusion across tasks; the prevalence of residual competition varies by task. Therefore "256 parameters reaching DPO level" should be understood as the paper's claim, not an independently reproduced fact.

The paper claims DHSA's scope of application can be further extended to other state transformations and learning objectives, and proposes that in the future, reinforcement learning methods such as PPO and GRPO, as well as distillation methods such as MOPD, could be used for hidden-state controller training, but these remain to be explored and verified.

From the disclosed information, the positioning of this method is: to provide a preference adaptation approach for a frozen base that is independent of weights and can be enabled on demand, and to bring mechanistic analysis into practical alignment. Whether it can stably reproduce the paper's self-reported level on larger models and more tasks currently has no independent evidence.