Einsia AI Releases PPTBench to Evaluate Coding Agents on Visual Coding

PPTBench's task setup is straightforward: given a flowchart, the Agent uses code to reconstruct it into a one-page PowerPoint. The output must be a genuinely code-generated, editable PPTX; it cannot simply paste the original image into the slide.

This constraint pushes the task from "looks similar" to "expresses it correctly." The Agent needs to re-express what it sees using native PowerPoint objects such as text, shapes, and connectors, rather than compressing the visual information into a bitmap.

The task materials come from flowcharts and architecture diagrams in real scientific papers, building 500 tasks in total, covering 13 domains including systems and software, artificial intelligence, computer vision, language and speech, quantum and fundamental physics, astronomy, robotics, and biomedicine. Layout types include multi-panel diagrams, hierarchical structures, repeated grids, complex connections, and text-dense varieties.

The same visual element can be expressed through multiple object representations; a tic-tac-toe grid can be composed of four squares, or of one square plus two straight lines. Therefore PPTBench designed an Agentic Judge, which does not require the Agent to reconstruct according to a fixed object tree, but instead judges layer by layer from the final delivered PPT whether the task was truly completed.

The first gate is the Artifact Gate: it checks whether the PPTX can be parsed and rendered normally, whether a single-page slide was generated, whether it contains enough native editable objects, and whether there is direct image pasting, among other things. If the artifact is invalid, it is directly scored 0.

After passing the Gate, it enters three-stage evaluation. Stage 1 judges whether the flowchart semantics are correct; as long as the flowchart itself changes, it does not count as correct even if it is visually very close; Stage 2 judges whether it has basic readability, excluding severe rendering problems or layout errors; Stage 3 only then compares the overall layout and local visual differences, and structurally records the problems.

The key to this design is separating "understanding" from "getting it right": Stage 1 handles semantics, Stage 2 handles usability, Stage 3 handles similarity. The order of the three cannot be reversed, because if the semantics are wrong, the latter two being beautiful is meaningless.

The research team tested 10 models and 36 configurations, with each configuration completing the same set of 500 tasks, and each result undergoing three rounds of review, accumulating 54,000 review records.

GPT-6 Astra High ranks first with 77.34 points, generating valid artifacts for all 500 tasks, with 80.8% passing both semantic and rendering checks. Among the other evaluated models, Kimi K3 High performed best with a score of 67.80; GPT-5.6 Sol Max and Qwen 3.8 Max XHigh scored 49.28 and 47.69 respectively.

Even for the optimal configuration, nearly one-fifth of tasks still failed to pass both checks simultaneously. This number itself shows that from "able to reconstruct" to "stably reconstruct correctly," there is still a clear distance.

Of all 18,000 results, 394 (2.19%) failed the artifact check; subsequently 11,526 (64.03%) were rejected for flowchart semantic errors. Of the remaining 6,080, another 365 failed the rendering/text gate, and ultimately 5,715 (31.75%) entered detailed scoring.

In other words, the first wall that truly trips up the models is not whether the code can run, but whether they understood what the diagram is saying. More than sixty percent of results were stopped at the semantic stage.

Among the artifacts that entered detailed scoring, text and layout issues contributed 53.0% of the detail deductions, local graphics accounted for 34.3%, and layout accounted for 12.8%; the single item with the most deductions was unexpected text line breaks.

These problems are precisely what the generation script itself cannot intuitively reveal: the script will not tell the model whether the rendered text will break lines where it should not, or whether it will burst the text box. Only models that truly perform visual review and iteration can discover and resolve them.

The gains from increasing reasoning effort are mainly reflected in the gate pass rate. For GPT-6 Astra from Low to High, the average score rose from 59.42 to 77.34, and the proportion passing all gates rose from 63.0% to 80.8%, an increase of 17.8 percentage points; while the conditional detail score after passing the gates rose only from 94.32 to 95.72.

But more thinking does not guarantee a higher score. Astra's five levels' average scores are, in order, Low 59.42, Medium 68.16, High 77.34, XHigh 72.60, Max 68.76. From High to Max, the gate pass rate dropped from 80.8% to 71.0%; although the conditional detail score rose to 96.85, the total score still declined.

This shows that the main benefit brought by depth of thinking is allowing more artifacts to retain the correct flowchart, while improvement in detail precision is relatively limited. Astra reaching its best performance at High also suggests that setting effort to the highest level may not be a reliable default choice.

The process by which models build PPTs differs from humans: humans assemble components one by one, while models write a complete PPTX generation script in one go, then adjust based on visual feedback.

After analyzing the models' invocation traces, it was found that models that visually inspect their own artifacts more often tend to score higher. Among the 27 configurations whose invocation traces were parsed, the Pearson correlation coefficient between the number of model review rounds and score was 0.881; by contrast, the correlation coefficient between the number of modification rounds and score was only 0.255.

It should be noted that this is correlation rather than causal evidence: the source does not state whether the review behavior itself directly brings score improvements, and there may also be confounding in which "stronger models both like reviewing more and are more likely to score well." But at least from the shape of the data, the association between visual review and high scores is far stronger than that of blind modification.

PPTBench's experiments show that frontier Agents can already complete fairly complex visual reconstruction, but semantic understanding is the main bottleneck; once semantics are basically correct, text, layout, and local details become new challenges.

This conclusion currently appears only in the team's own release and a Machine Heart report, lacking third-party independent reproduction. Model names and scores such as GPT-6 Astra, Kimi K3, GPT-5.6 Sol, and Qwen 3.8 all come from the benchmark's own evaluation, and no external validation is shown in the source.

The source also does not explain the consistency of the Agentic Judge's judgments at each stage, the proportion of manual review, or the scoring error range; it does not give each model's per-domain performance across the 13 domains, so it cannot be determined whether capability differences are domain-related; it does not explain the specific distribution and difficulty stratification of the 500 tasks across the 13 domains; nor does it explain the specific versions, invocation parameters, and runtime environment of the models used in the evaluation, so cross-configuration comparability is unknown.

Therefore, caution is needed when reading this leaderboard as a ranking of "who is stronger"; its more reliable value is that it raises a question that can be pursued further: when an Agent sees a visual target, can it truly understand the structure within it and reliably write that understanding into code.