Researchers from the UK AI Safety Institute argue that using AI systems to oversee AI training processes (automated alignment research) is not a panacea. 'Errors in automated alignment research may be harder to identify than human baselines' for reasons including: AI research is optimized for human approval, AI errors are unintuitive to humans, research content is highly correlated, research volume is enormous, and alignment solutions may rely on arguments that humans cannot evaluate.
They suggest improving the situation through measurement (e.g., testing AI's ability to continue research by logging successful projects), generalization experiments (e.g., testing different training agents on post-knowledge-cutoff data), and scalable oversight (e.g., compressing research paper corpora).