The essay argues that rational agents, including AIs, should not be goal-directed. Instead, it proposes a virtue-ethical framework where actions are aligned with practices and action-dispositions, challenging current AI alignment approaches.
SynthePulse Insight · AI deep reading
Version 1 · 1 source
This article explores a non-goal-directed form of rationality—eudaimonic rationality—arguing it may be key to aligning AI with human flourishing, and that the current dominant goal-optimization paradigm suffers from a fundamental 'type mismatch.'
The article opens with a counterintuitive thesis: rational people have no goals, and rational AI should not have goals either. The rationality of human behavior does not lie in aiming at final goals, but in aligning actions with 'practices'—networks of actions, action dispositions, evaluative standards, and resources that self-structure, clarify, and develop. If AI is to truly support, collaborate with, or obey human agency, its deliberation must share this practice-based logic.
This thesis concerns not only aligning AI with grand ethical ideals like human flourishing, but also with core safety properties such as transparency, beneficence, harmlessness, and corrigibility. For agents that interpret safety properties as goals or rules, these concepts are fragile, unstable, and arbitrary; but for agents that understand them as dynamics of a practice network, they are natural.
The article defines eudaimonic rationality as a form of rational activity in which there is no strict separation between means and ends, nor between instrumental and intrinsic value. Rational action is an element of a practice, like a note in a melody, a time step in a computation, or a cell's life in the self-maintenance and development of an organism.
The core formula is 'promote x x-ingly.' For example, art is promoting art in an artistic way, romance is promoting romance in a romantic way, and caring about kindness is promoting kindness in a kind way. This structure captures the essence of meaningful human life activities and genuine morality.
The article uses mathematician Terry Tao's description of mathematical research to illustrate how eudaimonic rationality operates. The mathematician's goal is 'mathematical excellence,' i.e., promoting mathematical excellence through mathematical excellence. This structure explains why 'mathematical excellence' can itself be a concept.
For eudaimonic agents (e.g., mathematicians pursuing excellence), the distinction between instrumental and intrinsic value is largely unnatural. The way they reflect on value differs from effective altruism-style optimization: eudaimonic agents seek organic causal coherence within a value network, rather than reducing value to a minimal intrinsic value basis.
The article defines eudaimonic practice as a network of actions, action dispositions, evaluative standards, and resources, where high-scoring actions reliably (but defeasibly) causally promote future high-scoring actions. Eudaimonic rationality is then a class of reflective equilibrium and deliberation processes that assume an underlying eudaimonic practice and optimize aggregate action scores through high-scoring actions.
This definition makes 'naturalness' a key concept: naturalness involves stability, coherence, relative non-contingency, ease of learning, low algorithmic complexity, cross-cultural convergence, and potential convergent evolution between humans and neural-network AI.
The article points out that many paradoxes and difficulties in AI alignment stem from the assumption that mature AI agents will be effective altruism-style goal optimizers. There is a 'type mismatch' between such optimization and eudaimonic rationality, making it nearly impossible to translate human interests (practitioners of eudaimonic rationality) into a utility function readable by an optimizer.
But this does not mean human values themselves are fragile or arbitrary: while goal optimizers may be a natural agent type, eudaimonic agents (whether biological or AI) are equally highly natural. The article argues that if the goal is for AI to support human flourishing, and eudaimonic rationality is natural and effective, then many classic AI safety considerations and paradoxes support endowing AI with eudaimonic rationality.
The article argues that eudaimonic rationality has advantages in stability, safety, and value alignment. For goal-optimizing agents, concepts like 'harmlessness' or 'corrigibility' are fragile, but for eudaimonic agents, these concepts can be naturally embedded in the practice network.
The article emphasizes that the problem is not whether human values can be formalized, but whether we have chosen the right form of rationality. Eudaimonic rationality offers an alternative that may make AI alignment more robust and natural.
This article is based on an academic blog post; the author presents original arguments but lacks empirical data or experimental validation. The arguments are theoretical and constructive, not established facts.
AI alignment should not focus solely on goal optimization but should draw on eudaimonic rationality from virtue ethics, designing AI agents as actors embedded in practice networks to achieve more stable and natural value alignment.
Primary report
Primary source