LegoFlow: Letting Agents Autonomously Run the Full Code Data Construction, Training, and Evaluation Pipeline

Researchers from Huawei, the Chinese University of Hong Kong, and the Hong Kong University of Science and Technology jointly launched the open-source code data engineering framework LegoFlow, packaging repository and PR collection, task verification, trajectory reasoning, training, and evaluation into plugin skills for coding agents such as Claude Code and Codex to call directly. The team organizes the process into five blocks—Root, Curator, Tracer, Trainer, and Evaluator—responsible for goal transformation and coordination, task construction, trajectory generation, fine-tuning, and benchmark evaluation, respectively.

The starting point for this design is the long-standing bottleneck in code data production: high manual annotation costs, uncontrollable LLM generation, and difficult automatic verification. LegoFlow breaks the complex engineering workflow into clearly bounded stages, with each block acting like an engineer, managed by a coding agent, and carrying the repositories, scripts, dependencies, and environment needed to fulfill its responsibilities. Curator turns repositories and pull requests into verified tasks, Tracer generates training trajectories, Trainer performs fine-tuning, and Evaluator returns benchmark evaluation results.

It should be noted that the above framework description and all subsequent results come from the team's own account, and there is currently no third-party independent reproduction or verification.

The construction of LegoFlow-SWE begins with collecting over 700,000 GitHub repositories and their associated 12 million candidate PRs. After filtering by conditions such as valid diffs, patch size, tests, and issue evidence, only 600,000 PRs remain; further removal of simple or unverifiable tasks through an LLM reviewer and execution verification ultimately leaves 5,000 verified tasks, accounting for only 0.0417% of the original data pool.

These 5,000 tasks cover 8 programming languages and 20 task labels. On the trajectory side, GLM-5.2 generated 9,767 runs through the OpenHands SDK and OpenCode, of which 2,780 passed verification, forming the verified successful trajectories in the dataset.

Each level of the filtering funnel relies on specific judgment criteria, but the LLM reviewer and execution verification on which task filtering depends have their judgment criteria and misjudgment rates left unstated in the text. This means the 0.0417% filter rate is itself a product of the team's process, and its reproducibility depends on these undisclosed judgment details.