Beihang, CUHK, and NUS Teams Propose OmniHarness, Enabling Visual AI Experience Transfer to Video Generation

Visual creation is often a multi-step chain: redraw a doodle into a realistic red flower, then use its style to generate a flower field, where the output of the previous step is the input of the next. If any link in the middle deviates, the error propagates down the entire pipeline. So the question becomes: after a visual agent completes all its planning and trial-and-error, can the correct experience be retained and still be usable with a different image or a different theme?

The answer given by OmniHarness, jointly completed by teams from Beihang University, CUHK, and NUS, is: keep parameters unchanged, collect correct workflows into a method library, and let the next creation have experience to draw on. Through symbolic strategy learning, it distills validated practices into methods reusable for a class of tasks.

A strategy here is described as a workflow with usage instructions: it retains shared steps, applicable conditions, and dependent resources, while removing the specific inputs of the current task. When a new task arrives, the system checks the conditions, fills in the new inputs, and then adjusts or combines existing strategies. In other words, the red flower and the input sketch are the details of this particular problem, while "first redraw, then use the result to guide generation" is the transferable part.

The system does not passively wait for tasks; it practices first. Repeatedly practicing familiar problems yields limited gains, while problems far beyond current capability easily lead to inefficient attempts, so task selection must balance exploratory value with support from existing methods. By default, 10 candidate tasks are proposed per round, and tasks are selected by the product of novelty and capability-boundary scores: novelty looks at where practice has been scarce, and the capability-boundary score looks at whether the difficulty suits the current capability.

How is reliability calculated? The system estimates it from the number of successful calls and verifications, takes the lower bound of the 95% Wilson confidence interval, and selects the most reliable applicable and non-disabled workflow for each capability. There is a design worth noting here: task capability estimation is determined by the weakest capability, so if a key step lacks a reliable method, it will drag down the final result. For multi-step tasks, the weakest link may determine the overall outcome.

In the execution phase, the system first plans steps and dependencies, then converts code into ComfyUI workflows, and verifies process executability, intermediate results, and final output. Intermediate verification is especially critical—if the red flower has already deviated from the requirements, continuing to use it to generate a flower field may propagate the deviation down the pipeline. After discovering a problem, it locates and repairs the failed step, retains the verified parts, and when necessary calls sub-agents to re-verify within a limited number of retries.

Successful workflows are abstracted into strategies, while failures leave behind evidence, causes, and correction methods; the system continuously updates reliability and merges equivalent workflows, letting new experience participate in the next round of task selection and execution. Succeeding once does not mean it will be reliable from then on: the successes and failures of subsequent calls will continue to affect a strategy's credibility and reuse priority.

On ComfyBench creative tasks, OmniHarness reaches a 95% solve rate, 27.5 percentage points higher than the strongest baseline SymbOmni; under the Codex GPT-4o configuration, the overall solve rate reaches 92.5%. These numbers come from a single source submitted by the paper team, count as source claims, and have no third-party independent replication yet.

What is more worth examining is how many tasks are actually completed after the pipeline runs successfully. The workflow run pass rate is 100% for both configurations, and ComfyMind and SymbOmni have also reached this level, so the pass rate itself does not constitute differentiation. When both use GPT-4o, OmniHarness has an overall solve rate of 89.5%, higher than ComfyMind's 83%; on creative tasks it is 95%, versus 57.5% for the baseline.

Switching to the Codex planning configuration, the overall solve rate rises from 89.5% to 92.5%, and complex tasks rise from 76.7% to 83.3%, reflecting the impact of the planning configuration on multi-step pipeline performance. But complex tasks still have room for improvement: the Codex configuration is 83.3%, slightly lower than ComfyMind's 85%.

The question that must be retained is this: for the comparison between the 95% on creative tasks and the strongest baseline SymbOmni's 57.5%, the source does not state whether the baseline configuration is completely identical to OmniHarness. In the absence of this premise, the 27.5 percentage point gap should be regarded as a result reported by the source, not as a confirmed performance gap.

The strategy library obtained from autonomous practice remains frozen and is adapted through a knowledge interface for use by other agents, preserving the host's control flow and without fine-tuning the model. This strategy library contains both reusable workflow templates and failure evidence and correction methods, for other agents to invoke through their own retrieval, planning, and execution processes.

The changes in overall solve rate after connection are: ComfyAgent from 32.5% to 57.0%, ComfyMind from 83.0% to 88.0%, and SymbOmni from 86.0% to 89.0%; the creative task solve rates of the three systems increase by 27.5, 17.5, and 10 percentage points respectively. The increases on creative tasks for all three systems are higher than their respective overall increases, which suggests that when facing new creative requirements, invocable methods and correction experience deserve particular attention.

But the differences in gain magnitude are themselves worth noting: ComfyAgent improves by 24.5 percentage points, while SymbOmni improves by only 3 percentage points. This difference suggests that transfer effectiveness may be highly correlated with the host system's original capabilities, and its stability and applicable boundaries remain unclear.

The frozen strategy library obtained from practicing only on image tasks can also participate in video generation. The overall video solve rate reaches 85.9%, 5.1 percentage points higher than the best baseline in the table; video-to-video tasks reach 80.0%, 13.3 percentage points higher than that baseline.

According to the source's description, image strategies support operations such as content construction and reference preservation, while video-specific components handle temporal ordering, and the two complete tasks through adaptation and combination. However, the effect of transferring image strategies to video generation is reported only by this source, and the details of the adaptive combination of video-specific components and image strategies are not fully elaborated, so the robustness of this cross-modal conclusion awaits more evidence.

This approach is not without cost. Autonomous practice and multiple rounds of checking still incur computational overhead, and retrieval, verification, and inference budgets need to be optimized going forward. Repair of failed steps and sub-agent calls are limited to a finite number of retries, which is both an engineering constraint and an implication that the repair capability for complex tasks has an upper limit.

The more fundamental limitation lies in the evidence itself: all benchmark numbers come from a single source submitted by the paper team, lacking third-party independent replication; although the full paper, project homepage, and code links have been provided, the source does not provide verifiable third-party evaluation or replication results. Therefore, numbers such as 95%, 92.5%, 89.5%, and 85.9% in this article should be understood as source claims, not as confirmed conclusions.