The short answer

The Astra Blender demos made me wonder how well the same kind of spatial ability carries over to real places. Could a general model watch a room tour and understand how the space fits together?

I tested GPT-6-Astra on 10,866 questions across VSI-Bench and VSTI-Bench: distances, layouts, navigation, and camera motion. I’ve spent years working on 3D. Seeing a general model perform at this level still caught me off guard.

VSI-Bench73.055,130 questions · 8-task macro
VSTI-Bench70.335,736 questions · 5-family macroSame answers: 78.84 when all 9 subtypes count equally. Here, each of the 5 task families counts equally.

Scores are out of 100. VSTI uses corrected labels and five-family aggregation; its historical paper scores are not directly comparable.

VSI’s 73.05 places Astra alongside strong spatial specialists in the selected reports. Its 87.63 on route planning is the highest available task score in that comparison. VSTI’s 70.33 is numerically the highest in the selected table, although the label revision prevents treating it as a same-protocol win.

The task breakdown is more revealing than either total: route planning and object relationships stand out, while room scale and camera displacement remain clear weaknesses.

What do these benchmarks actually test?

VSI-Bench asks a model to reason about real indoor spaces from video. Its eight task families cover object counting, object and room size, absolute and relative distance, relative direction, route planning, and appearance order.[1]

Think of questions like: how far apart are two objects, how large is the room, or which route connects two locations? These examples illustrate the skills; they are not verbatim benchmark prompts. Answering them requires bringing together evidence from different views, including things that may no longer be on screen.

VSTI-Bench adds the moving observer. It tests camera-to-object distance, camera displacement, camera movement direction, relative object position, and relative camera-to-object distance. The scenes are static; the camera moves through them.[2]

That makes the pair useful: one emphasizes a scene’s layout, while the other makes the changing viewpoint explicit. These are pieces of the spatial understanding needed for navigation, robotics, and AR—not a complete test of any of those systems.

Look at the actual questions

Explore 38 examples: two for each of VSI’s 10 subcategories and VSTI’s 9 subcategories. This keeps direction difficulties and relative-position and distance variants separate. Each pair includes a full-credit answer and a miss or partial-credit estimate. Select a subcategory, scrub through the frames, and compare Astra’s answer with the benchmark answer. These examples illustrate the tasks; they are not a separate performance estimate.

38 examples · 19 subcategories · 2 each

1 / 38

VSI-Bench / Object counting

Counting objects in a room

Full credit

How many bed(s) are in this room?

VSI sample 4587, frame 17 / 33, 0:16.0↗
Frame 17 / 330:16.0
Astra’s answer2
Benchmark answer2
Score for this question100 / 100 MRA

Numerical answers receive partial credit under MRA’s relative-error thresholds. This score is not a confidence estimate.

View saved model response
2

VSI-Bench · sample 4587 · ScanNet / scene0221_01

All sampled input frames are retained in their original order; images are compressed for display. Questions, options, and saved answers are shown verbatim.

Why this is a meaningful test

A video offers a sequence of views. A spatial answer often needs a more stable account of what lies behind them: which objects are the same across frames, where they sit, what is hidden, and how the camera has moved.

Cambrian-S describes part of this challenge as “inferring the world behind pixels.” Its experiments motivate going beyond semantic video descriptions toward persistent spatial understanding. Cambrian-P studies camera pose as a training signal that ties changing views to a shared three-dimensional space.[3][4]

The original VSI-Bench study found that familiar language-reasoning techniques, including chain-of-thought, did not improve spatial performance for the models and settings it tested. That result is not a permanent ceiling, but it explains why a strong general-model result is worth taking seriously.[1]

The evaluation here covers VSI-Bench and VSTI-Bench. It does not measure the separate, longer-horizon VSI-Super tasks introduced by Cambrian-S.

A general model in a specialist field

The comparison below keeps general-purpose models and models with spatial or embodied training or architecture changes in separate groups. Other models’ values are the authors’ reported results; we did not rerun them. Click a model name to inspect its source.

This evaluation alongside selected paper reports. Inputs, training, and inference settings vary by source.

VSI-Bench: selected reported overall scores
Model / sourceScore / 100
General-purpose models
GPT-6-AstraThis evaluation
73.05
Qwen3.5-122B-A10BScore [14] · Model [28]
66.6
Qwen3.5-9BScore [16] · Model [29]
57.9
InternVL3.5-8BScore [17] · Model [30]
56.1
Gemini-3-ProScore [15] · Model [26]
56.0
GPT-5.4Score [14] · Model [25]
49.2
Claude Sonnet 4.6Score [14] · Model [27]
44.4
Spatial / embodied specialists
ConsiSpace UC-SSRLScore [18]
76.6
RynnBrain 1.1-122B-A10BScore [14]
75.0
Molmo2-ER-5BScore [14] · Model [31]
74.5
Cambrian-PScore [4]
73.7
SpatioLM (SenseNova-SI-8B)Score [17]
71.6
LLaVA-OneVision-2-8BScore [16] · Model [32]
70.9
SpaceMindScore [15]
70.2
Cambrian-S-7BScore [3]
67.5
VLM-3R-7BScore [2]
60.9
Spatial-MLLM-4BScore [19]
47.0
Figure 1. Selected reports, grouped by model type and sorted by reported score within each group. Model names open score sources; numbered citations jump to the references. “Model” identifies a separate original paper or official model page. No unified rank is assigned across protocols.

On VSI, Astra is close to Cambrian-P’s reported 73.7, with several specialists still higher overall. The route-planning result stands out much more than the aggregate. That is a useful distinction: a high overall score can conceal very different capabilities.

VSTI needs one additional piece of context. On July 13, 2026, its authors corrected a camera-position bug affecting distance and displacement labels. Astra was evaluated on the corrected 5,736-question revision; the specialist and general-model literature rows here retain their historical reports. The authors explicitly state that pre-fix and corrected results are not comparable.[5]

Route planning excels. Room scale and camera displacement lag.

The task scores help separate the different demands of spatial reasoning. Astra can reason remarkably well about routes and object relationships, yet its estimates become less reliable when it has to recover the scale of an entire room or the camera’s change in position.

Click a legend item to highlight a model. Hover or tap an axis or marker to inspect exact scores.

Choose models 7 / 7

Select 1–7 models. At the limit, uncheck a model to add another.

Radar models
VSI-Bench · 8-axis spatial ability radarEvery axis runs from 0 to 100. Select an axis or marker for exact values. Missing task scores are not plotted as zero or connected across gaps.GPT-6-Astra · Object counting: 61.82 MRAGPT-6-Astra · Absolute distance: 61.53 MRAGPT-6-Astra · Object size: 78.43 MRAGPT-6-Astra · Room size: 56.49 MRAGPT-6-Astra · Relative distance: 76.06 AccuracyGPT-6-Astra · Relative direction: 82.48 AccuracyGPT-6-Astra · Route planning: 87.63 AccuracyGPT-6-Astra · Appearance order: 79.94 AccuracyCambrian-S-7B · Object counting: 73.2 MRACambrian-S-7B · Absolute distance: 50.5 MRACambrian-S-7B · Object size: 74.9 MRACambrian-S-7B · Room size: 72.2 MRACambrian-S-7B · Relative distance: 71.1 AccuracyCambrian-S-7B · Relative direction: 76.2 AccuracyCambrian-S-7B · Route planning: 41.8 AccuracyCambrian-S-7B · Appearance order: 80.1 AccuracyGemini-3-Pro · Object counting: 49.0 MRAGemini-3-Pro · Absolute distance: 42.8 MRAGemini-3-Pro · Object size: 71.5 MRAGemini-3-Pro · Room size: 41.8 MRAGemini-3-Pro · Relative distance: 56.6 AccuracyGemini-3-Pro · Relative direction: 57.5 AccuracyGemini-3-Pro · Route planning: 61.9 AccuracyGemini-3-Pro · Appearance order: 60.0 AccuracyConsiSpace UC-SSRL · Object counting: 77.9 MRAConsiSpace UC-SSRL · Absolute distance: 78.9 MRAConsiSpace UC-SSRL · Object size: 82.2 MRAConsiSpace UC-SSRL · Room size: 76.3 MRAConsiSpace UC-SSRL · Relative distance: 72.1 AccuracyConsiSpace UC-SSRL · Relative direction: 90.1 AccuracyConsiSpace UC-SSRL · Route planning: 54.7 AccuracyConsiSpace UC-SSRL · Appearance order: 79.9 AccuracyCambrian-P · Object counting: 74.9 MRACambrian-P · Absolute distance: 60.1 MRACambrian-P · Object size: 76.0 MRACambrian-P · Room size: 76.9 MRACambrian-P · Relative distance: 74.8 AccuracyCambrian-P · Relative direction: 89.5 AccuracyCambrian-P · Route planning: 52.6 AccuracyCambrian-P · Appearance order: 85.0 AccuracySpatioLM (SenseNova-SI-8B) · Object counting: 81.9 MRASpatioLM (SenseNova-SI-8B) · Absolute distance: 55.7 MRASpatioLM (SenseNova-SI-8B) · Object size: 75.2 MRASpatioLM (SenseNova-SI-8B) · Room size: 71.5 MRASpatioLM (SenseNova-SI-8B) · Relative distance: 78.2 AccuracySpatioLM (SenseNova-SI-8B) · Relative direction: 74.4 AccuracySpatioLM (SenseNova-SI-8B) · Route planning: 47.9 AccuracySpatioLM (SenseNova-SI-8B) · Appearance order: 87.8 AccuracyQwen3.5-9B · Object counting: 62.0 MRAQwen3.5-9B · Absolute distance: 44.0 MRAQwen3.5-9B · Object size: 73.6 MRAQwen3.5-9B · Room size: 61.9 MRAQwen3.5-9B · Relative distance: 69.7 AccuracyQwen3.5-9B · Relative direction: 79.9 AccuracyQwen3.5-9B · Route planning: 45.4 AccuracyQwen3.5-9B · Appearance order: 26.7 AccuracyObject countingMRAAbsolute distanceMRAObject sizeMRARoom sizeMRARelative distanceAcc.Relative directionAcc.Route planningAcc.Appearance orderAcc.
GPT-6-Astra
61.82
Cambrian-S-7B
73.2
Gemini-3-Pro
49.0
ConsiSpace UC-SSRL
77.9
Cambrian-P
74.9
SpatioLM (SenseNova-SI-8B)
81.9
Qwen3.5-9B
62.0

This evaluation alongside selected paper reports; input, training, and inference settings differ.

Figure 2. The dashboard’s seven-model default comparison. Every axis is fixed at 0–100; MRA and accuracy are different metrics. Missing values are not replaced by zero, and polygon area is not the overall score.

VSI: following a route and measuring a room draw on different strengths. Astra’s 87.63 accuracy on route planning is the highest reported value in this comparison, well above Cambrian-P’s 52.6. On room-size estimation, the ordering reverses: Astra scores 56.49 MRA, compared with Cambrian-P’s 76.9. The overall scores hide a substantial difference in what these models do well.

A route question asks which turns connect landmarks from a specified starting orientation. Getting those relationships right can be enough even with an approximate sense of scale. Room sizing asks for the area in square meters, requiring a judgment of the space’s full extent across partial views.[1] Astra’s much stronger 78.43 MRA on object size also matters: its weakness is not shared by all numerical tasks. My reading is that reasoning about a layout transfers more reliably than estimating the geometry of the whole room. Familiar objects may provide useful size cues that are harder to extend to an entire space.

VSTI: strong object distance and relationships, weaker camera displacement. The standout results are 94.59 accuracy on relative object position and 86.81 on relative camera–object distance. Absolute camera–object distance reaches 68.83 MRA, but camera displacement falls to 42.52 MRA. The latter pair uses the same metric, making it a useful contrast within this run. Camera movement direction is also weaker at 58.93 accuracy.

The distinction is between estimating where an object is relative to the observer and recovering how the observer’s own position changes. A displacement question asks for the distance between camera positions at two frames.[2] Solving it requires geometry that stays consistent across views: a large change in the image can come from turning the camera, translating it, or both. My hypothesis is that this separation of rotation, translation, and scale is a harder step for Astra than judging object relationships. Strong camera–object distance estimates alone do not guarantee an accurate estimate of camera displacement.

GPT-6-AstraCambrian-P
Object countingMRA
GPT-6-Astra: 61.82
Cambrian-P: 74.9
Absolute distanceMRA
GPT-6-Astra: 61.53
Cambrian-P: 60.1
Object sizeMRA
GPT-6-Astra: 78.43
Cambrian-P: 76.0
Room sizeMRA
GPT-6-Astra: 56.49
Cambrian-P: 76.9
Relative distanceAccuracy
GPT-6-Astra: 76.06
Cambrian-P: 74.8
Relative directionAccuracy
GPT-6-Astra: 82.48
Cambrian-P: 89.5
Route planningAccuracy
GPT-6-Astra: 87.63
Cambrian-P: 52.6
Appearance orderAccuracy
GPT-6-Astra: 79.94
Cambrian-P: 85.0

All bars use a 0–100 scale. Numerical tasks use MRA; multiple-choice tasks use accuracy. These are different measures.

Figure 3. Look beneath the overall score. Select a comparison model to inspect its task profile. Only models with reported task scores are offered.

Astra’s strong route planning and object relationships suggest it can form a useful picture of how a scene fits together. Its stronger object-size and camera–object-distance scores point to an ability to estimate scale, too. The larger gaps emerge when the question requires the full extent of a room or the camera’s change in position across views. It can choose the right turns while still struggling to estimate how large the space is or how far the camera moved through it.

How I ran it

SettingVSI-BenchVSTI-Bench
Questions5,1305,736
Video frames1 FPS, full video, no frame cap32 uniform frames, original frame references
Model / reasoninggpt-6-astra / high
ImagesNative resolution, JPEG quality 85, high detail
Article score8-task macro; direction difficulties averaged firstQuestions pooled within a family; 5 families averaged equally

The inputs contain the sampled video frames and benchmark questions, without ground-truth answers, 3D annotations, or answers to previous questions. Dataset revisions are pinned, and the evaluation uses the benchmarks’ answer parsing and scoring conventions.[6][7]

Multiple-choice questions use accuracy. Numerical answers use mean relative accuracy (MRA), which averages whether a prediction falls within a series of relative-error tolerances. The overall score combines these task metrics; it is not simply the percentage of questions answered correctly.[1]

Why does VSTI also have a score of 78.84?

Both scores use the same 5,736 answers. The difference is how much weight each kind of task receives.

The public evaluator gives each of nine subtypes equal weight, producing 78.84. Astra’s two strongest families—relative object position and relative camera–object distance—each contain three subtypes. Together, they account for six of nine weights, or two-thirds of the score.

The headline 70.33 first pools questions within each of five task families, then averages those five scores equally. Each family gets 20% of the weight, so the two relationship families together account for 40%. This gives the weaker camera-motion tasks more influence and yields a lower overall score. We use this view to give each of the five spatial skills equal importance.

Five-family aggregation makes the task breakdown easier to read alongside paper tables. It does not fix the difference between historical and corrected labels.

These are full-dataset results from this evaluation, not peer-reviewed claims or a uniform rerun of every comparison model. The downloadable record includes exact task values, data revisions, and generation settings.

From CAD skills to real-world spatial reasoning

The strong route-planning and relationship scores bring me back to the Blender demos that motivated this evaluation. My hypothesis is that RL on CAD or other 3D-tool tasks could produce spatial skills that transfer to understanding real-world video. This would connect the ability to construct a coherent 3D scene with the ability to infer one from incomplete views. Astra’s specific CAD/RL training history remains unconfirmed.

The public examples make that connection worth considering. Sharif Shameem showed Astra recreating the Palace of Fine Arts in Blender; Yunfan Ye reported turning listing photos into a 3D house, with some details still wrong.[8][9] Thomas Ricouard documented an editable house developed through repeated rendering, inspection, and geometry corrections.[10] In SolidWorks, MecAgent reported a 41-part turbojet and an 11-part robot arm with editable feature trees and moving assemblies, though their constraints still needed work before they could be manufactured.[11][12]

What might transfer is a way of reasoning about geometry. Placing objects, rotating a component, changing a camera, or satisfying an assembly constraint all require relationships to remain consistent as something changes. RL on such tasks could reward tracking relative positions, reasoning across coordinate frames, and anticipating how a scene will look from another viewpoint. Those same operations could help answer which object is closer, what lies to the left from a given orientation, or which turns connect two landmarks in a real room.

There is a sim-to-real flavor to this idea. A familiar robotics example trains a control policy in simulation and deploys it on a physical robot.[13] Here, the proposed transfer crosses both a domain boundary and a task boundary: from constructing or manipulating synthetic geometry to interpreting recorded views of real places. The potentially reusable skill is spatial reasoning, and the target task is video understanding.

This framing also gives the uneven task scores a more coherent interpretation. A grasp of relative geometry could support route planning and object relationships across those settings. Estimating an entire room’s area or the camera’s displacement adds a further challenge: recovering consistent scale and position from partial, changing images, without access to the scene’s underlying coordinates. The weaker results on those tasks could mark where such transfer becomes less reliable, even as object-size and camera–object-distance estimates remain stronger.

The possibility is that learning to construct and manipulate 3D spaces also develops skills for understanding real spaces that the model can only observe.

Data & references

Every literature score in both benchmarks is covered below, with its reporting paper, version, and model names. When a score comes from another paper’s comparison table, the original model paper or official model page is also listed separately. Astra’s scores come from this evaluation. Benchmark, dataset, and 3D-demo references are included too.

Download results and sources (JSON) ↓

Download case questions, answers, and frame details (JSON) ↓

  1. Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces
  2. VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction
    VSI-Bench · 2505.20279v5 · 2026-04-21Reported scores: VLM-3R-7B
    VSTI-Bench · arXiv:2505.20279Reported scores: VLM-3R-7B
  3. Cambrian-S: Towards Spatial Supersensing in Video
    VSI-Bench · 2511.04670v1 · 2025-11-06Reported scores: Cambrian-S-7B
  4. Cambrian-P: Pose-Grounded Video Understanding
    VSI-Bench · 2605.22819v2 · 2026-07-23Reported scores: Cambrian-P
    VSTI-Bench · v1 · 2026-05-21（v2 Table 2 数值相同）Reported scores: Cambrian-P
  5. VSTI-Bench label correction · July 13, 2026
  6. VSI-Bench · pinned dataset revision
  7. VSTI-Bench · pinned corrected dataset revision
  8. Sharif Shameem · Astra recreates the Palace of Fine Arts in Blender · X, September 3, 2026
  9. Yunfan Ye · Listing photos to a 3D house and video · X, September 3, 2026
  10. Thomas Ricouard · Architectural visualization with Astra · OpenAI Developers
  11. Alexandre Senet / MecAgent · Astra turbojet assembly in SolidWorks
  12. Alexandre Senet / MecAgent · Astra robot-arm assembly in SolidWorks
  13. Peng et al. · Sim-to-Real Transfer of Robotic Control with Dynamics Randomization · ICRA 2018
  14. RynnBrain 1.1: Towards More Capable and Generalizable Embodied Foundation Model
    VSI-Bench · 2607.17977v2 · 2026-07-31Reported scores: GPT-5.4, Claude Sonnet 4.6, Qwen3.5-122B-A10B, RynnBrain 1.1-122B-A10B
    VSI-Bench · 2607.17977v2 · 2026-07-31Reported scores: Molmo2-ER-5B
  15. SpaceMind: Camera-Guided Modality Fusion for Spatial Reasoning in Vision-Language Models
    VSI-Bench · CVPR 2026 正式论文集 · pp. 16811–16822Reported scores: Gemini-3-Pro, SpaceMind
  16. Annotations as Rollouts: Efficient and Scalable Reinforcement Learning for Video MLLMs
    VSI-Bench · 2608.20492v1 · 2026-08-20Reported scores: Qwen3.5-9B, LLaVA-OneVision-2-8B
  17. SpatioLM: Towards General Physical Spatial Intelligence in Vision-Language Models
    VSI-Bench · 2608.01899v1 · 2026-08-03Reported scores: InternVL3.5-8B, SpatioLM (SenseNova-SI-8B)
  18. ConsiSpace: Learning Geometric Consistency Matters for Video Spatial Reasoning
    VSI-Bench · 2607.17599v1 · 2026-07-20Reported scores: ConsiSpace UC-SSRL
  19. Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence
    VSI-Bench · 2505.23747v2 · 2026-05-19Reported scores: Spatial-MLLM-4B
  20. Vision-Language Memory for Spatial Reasoning
    VSTI-Bench · v2 · 2026-07-09Reported scores: GPT-5, Gemini-3-Pro, Qwen3-VL-8B, VLM²-7B
  21. Q-GeoMem: Question-Guided Geometric Memory for Video Spatial Reasoning
    VSTI-Bench · v2 · 2026-07-06Reported scores: Q-GeoMem-7B
  22. Cog3DMap: Multi-View Vision-Language Reasoning with 3D Cognitive Maps
    VSTI-Bench · v1 · 2026-03-24Reported scores: Cog3DMap-8B
  23. Thinking with Geometry: Active Geometry Integration for Spatial Reasoning
    VSTI-Bench · v1 · 2026-02-05Reported scores: GeoThinker
  24. 4D-RGPT: Toward Region-level 4D Understanding via Perceptual Distillation
    VSTI-Bench · v4 · 2026-06-02Reported scores: 4D-RGPT-8B
  25. OpenAI · Introducing GPT-5.4
    Model paper or official documentation: GPT-5.4
  26. Google · Gemini 3 Pro: the frontier of vision AI
    Model paper or official documentation: Gemini-3-Pro
  27. Anthropic · Introducing Claude Sonnet 4.6
    Model paper or official documentation: Claude Sonnet 4.6
  28. Qwen3.5-122B-A10B · official model card
    Model paper or official documentation: Qwen3.5-122B-A10B
  29. Qwen3.5-9B · official model card
    Model paper or official documentation: Qwen3.5-9B
  30. InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
    Model paper or official documentation: InternVL3.5-8B
  31. MolmoAct2: Action Reasoning Models for Real-World Deployment (Molmo2-ER)
    Model paper or official documentation: Molmo2-ER-5B
  32. LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligence
    Model paper or official documentation: LLaVA-OneVision-2-8B
  33. OpenAI · Introducing GPT-5
    Model paper or official documentation: GPT-5
  34. Qwen3-VL Technical Report
    Model paper or official documentation: Qwen3-VL-8B

Citation

BibTeX
@misc{wang2026astraspatial,
  title        = {How well does {Astra} understand real-world space?},
  author       = {Wang, Yipeng},
  year         = {2026},
  howpublished = {Blog post},
  url          = {https://www.yipeng.dev/blog/astra-spatial-intelligence}
}
Back to top ↑