Zhejiang University Proposes ProVisE: Let Generative Models 'Draw' Spatial Cognition Instead of Outputting Coordinates

Spatial judgment originally happens in continuous visual scenes, yet existing spatial cognition evaluations almost all require models to answer with coordinates, options, or text. In the paper "Show, Don't Tell: Evaluating Spatial Cognition in Generative Pixels Rather Than LLM Text," the OmniAI team at Zhejiang University calls this requirement "counterintuitive": it is no longer natural for text VLMs, while image generation models that naturally work in pixel space and could directly "draw" their answers are shut out of the evaluation.

This is the starting point of the research, and it is also a source-side claim rather than a neutral fact. The question it truly raises is not "can generative models do spatial problems," but whether the answering interface of evaluation itself filters out a certain class of models. Around this point, the team proposes the ProVisE framework, letting image generation models answer spatial questions directly in images and restoring visual answers into structured predictions that the original benchmark can score.

ProVisE's process has three steps: the Protocol Router selects a visual protocol according to the task; the image generation model produces a parseable visual answer following the Guidance Prompt; and the Protocol Parser then converts the pixel result into the labels, points, masks, states, or trajectories required by the original benchmark. The visual protocol is the basic unit of this framework, and each protocol contains a guidance prompt and a corresponding parser.

Specific forms of the protocols include marking targets, filling in direction grids, generating depth maps, painting out target regions, or drawing robot motion trajectories. The source gives examples: the model can directly point out the target bus or draw the robotic arm's movement trajectory on the original image; when judging which of points A and B is closer, it generates a depth map; and for a question like "can one more person still sit on the cable car," it adds a second passenger onto the seat, intuitively showing whether there is enough space.

To handle spatial benchmarks of varying forms, ProVisE incorporates the Agentic Protocol Construction framework. The Agentic Builder scans the data, input form, answer structure, and scoring rules task by task, unifies them into a task specification, and then chooses among three protocol construction paths: Reuse, Build, or Fallback. Reuse directly reuses a compatible existing protocol; Build assembles a new protocol using registered parsing components; and Fallback, when deterministic components cannot cover the case, has an auxiliary VLM parse the generated image and recover the structured prediction.

One easily overlooked constraint is that after protocol construction is completed, it must be validated to ensure the visual answer can be generated, parsed, and connected to the original metric normally; only after passing is it frozen and used for evaluating all image generation models. This means the comparability of the evaluation depends on the protocol being fixed first, rather than each model using its own parsing method.