Tsinghua AIR Proposes High-Order Action Supervision: Focusing on First-Order Temporal Information, Significant Gains with 10k Small Data

Mainstream imitation learning and reinforcement learning usually supervise only zero-order actions—given state s, what action a should be output. But continuous control scenarios differ: robot joint torques, vehicle steering angles, and drone thrust all evolve continuously over time, and even if adjacent actions are each reasonable, the jumps between them can still cause trajectory deviation, mechanical wear, and extra energy consumption. The paper gives an intuitive example with maze navigation: ordinary behavior cloning fits only action labels, the first-order action direction swings frequently, and the agent ultimately takes a suboptimal path; after adding higher-order action supervision, both zero-order and first-order actions become smoother, and the agent reaches the goal along a more stable path.

The first-order action here is the rate of action change, approximated in discrete data by the difference between adjacent actions. The paper's core claim is: expand the training objective from fitting only single-point action labels to simultaneously fitting the first-order change of actions over time. This change does not require reconstructing the policy network, only adding an extra first-order action constraint term to the original loss.

One simple and direct idea is to have the policy network output both the action a and the first-order action ȧ, but the paper argues this is not reasonable: the two live in different action spaces, and the added output term would not only enlarge the optimization problem but could also introduce multi-task gradient conflicts. The paper chooses another path—adding an extra first-order action constraint term without changing the original policy network framework.

Specifically, for consecutive samples (s,a,s′,a′) in offline trajectories, the authors adopt a unit time interval and write the state change and action change as ṡ=s'-s and ȧ=a'-a, respectively. After differentiating the policy π(s) along physical time, the chain rule gives the core formula: the derivative of the policy along the state trajectory direction corresponds to the first-order action. During training, one only needs to make this directional derivative close to the action difference in the data; for the most basic deterministic behavior cloning, the added term is the squared error between the policy directional derivative and the true action difference.