The latest Import AI newsletter discusses concerns that AI alignment efforts are not progressing adequately, along with mentions of FrontierCode and synthetic research interns. The brief content query 'Where are your agents right now?' hints at challenges in monitoring AI agents.
SynthePulse Insight · AI deep reading
Version 1 · 1 source
Researchers from the UK AI Safety Institute's alignment team and Timaeus have jointly founded the nonprofit organization Sequent, which plans to raise $100–150 million in initial funding to explore alignment theory routes different from those of mainstream AI labs, aiming to achieve higher-confidence safety guarantees before recursive self-improvement occurs.
Researchers from the UK AI Safety Institute (UK AISI) alignment team and the theoretical alignment startup Timaeus have jointly established a new nonprofit research organization called Sequent. In its announcement, the organization states bluntly: 'Artificial superintelligence (ASI) could be developed within the next few years. It is unclear whether alignment research will be ready on the same timeline.'
Sequent argues that the empirical programs of mainstream AI labs 'are unlikely to provide a priori confidence, before training an ASI, that things will go well.' Ideally, one would first develop a method for building superintelligence while providing a theoretical proof of its safety, and only then actually build it. But the reality is that 'we may have to fall far short of this ideal.'
Sequent plans to take a different path in alignment research from major AI labs. Its goal is to find 'principled reasons to be confident that alignment observed in controllable situations (e.g., during training or in evaluations under selected conditions) can generalize to hard-to-control situations (e.g., large scale, long duration, tasks performed in the world).'
Sequent criticizes frontier AI labs' methods as 'inherently reactive, leading to approaches that are workable but provide no principled insight into when they will fail.'
Sequent is interested in multiple alignment theory areas, including scalable oversight, learning theory, heuristic arguments, game theory, and personhood. It plans to build an internal research portfolio and collaborate with sister organizations.
Sequent believes there may be promising interactions between different research directions: for example, attainable equilibria can 'tell us what equilibrium a scalable oversight method will converge to'; by combining insights from learning theory and personhood to understand adjustable variables during training, then using scalable oversight to determine the magnitude of adjustments.
Sequent notes that current AI systems are aligned to some extent, but also have sharp edges that manifest as unexpected failures. As AI systems become smarter, humans will delegate more core research tasks to these systems, and AI systems may also begin to autonomously perform recursive self-improvement (RSI).
Sequent believes that before RSI occurs, better alignment techniques must be developed, otherwise we will 'roll very scary dice.' The organization maintains independence and can 'shout loudly' when necessary.
This article's information primarily comes from Sequent's public statement and Import AI's report. Sequent's statement is a primary source, and Import AI's report is a reliable retelling. All goals, plans, and research directions of Sequent are its own claims and have not been independently verified.
The founding of Sequent reflects the alignment research community's concern about the inadequacy of mainstream AI labs' safety methods and an urgent attempt to build higher safety confidence before superintelligence arrives through a theory-driven differentiated research portfolio.
Primary report
Primary source