PinpointQA: A Dataset and Benchmark for Small Object-Centric Spatial Understanding in Indoor Videos
A new benchmark reveals that today's best AI vision models still struggle to pinpoint small objects in indoor video — a gap that matters for robotics and assistive devices.
The Thesis
PinpointQA is a new dataset and benchmark designed to test whether multimodal AI models — systems that process both video and language — can reliably locate small objects in indoor scenes and describe their positions precisely enough to be useful. The practical stakes are real: robots that fetch items, smart glasses that help visually impaired users find keys or medication, and warehouse automation all depend on this capability. The paper finds that current leading models fail badly on the hardest tasks, particularly when asked to produce structured spatial descriptions like 'the bottle is on the second shelf to the left of the red container.' Fine-tuning models on this new dataset produces meaningful improvements, suggesting the gap is partly a data problem rather than a fundamental architectural one. The catch is that this is a benchmark paper, not a new model — it diagnoses a problem without fully solving it.
Catalyst
Multimodal large language models capable of processing video have only become widely available and evaluable in the last two years, making a rigorous benchmark timely. The source datasets — ScanNet++ and ScanNet200 — provide the high-resolution 3D-annotated indoor scans needed to generate precise spatial ground truth automatically, infrastructure that only recently reached sufficient scale and quality. Assistive robotics and indoor navigation are simultaneously moving from research labs toward commercial deployment, creating urgent demand for exactly this kind of capability measurement.
What's New
Prior spatial reasoning benchmarks for video — such as those evaluating embodied navigation or general scene understanding — tested broad scene comprehension rather than precise localization of small, specific objects. Those benchmarks would accept an answer like 'the cup is on the table' as correct; PinpointQA requires the model to specify which table, relative to which nearby objects, and at what position. This paper introduces four progressively harder task levels, from simple presence detection up to structured spatial prediction with enough precision for a downstream system to act on the answer.
The Counter
PinpointQA is a benchmark, not a solution — and the history of AI benchmarks is littered with datasets that measured something real but didn't move the needle on actual deployment. The 10,094 QA pairs are automatically generated from existing scan data and then quality-controlled, a process that can introduce systematic blind spots or artifacts that trained models exploit without learning genuine spatial reasoning. ScanNet scenes are also relatively clean, structured indoor environments; real-world deployment involves clutter, poor lighting, and object classes far outside the training distribution. Most critically, the paper shows fine-tuning helps, but does not show that a fine-tuned model actually improves performance in a real robot or assistive device — the gap between benchmark accuracy and deployed usefulness is exactly the hard problem this paper leaves open. Finally, competing labs are actively building spatial reasoning into next-generation multimodal models, so the capability gap documented here may close from the model side rather than through specialized datasets.
Longs
- MSFT — Azure AI and HoloLens spatial computing stack
- IRBT (iRobot, now part of Amazon ecosystem) — home robotics needing object localization
- BOTZ (robotics ETF) — broad exposure to indoor robot platforms that need this capability
- NVDA — inference hardware for embodied AI and edge vision models
- GOOG/Alphabet — Project Astra and Android-based assistive vision products
Shorts
- Companies selling general-purpose vision-language APIs (e.g., OpenAI GPT-4o vision, Google Gemini) — this benchmark publicly documents their failures on precise spatial tasks, giving customers reason to seek specialized alternatives
- Early-stage robotics startups that assumed off-the-shelf MLLMs were 'good enough' for object localization without domain-specific fine-tuning
Enablers (Picks & Shovels)
- ScanNet and ScanNet++ (open 3D indoor scan datasets) — the raw material for PinpointQA's annotations
- Hugging Face Transformers — fine-tuning infrastructure used to validate the benchmark
- Open-source MLLMs such as LLaVA and InternVL — the baseline models evaluated in the paper
- ARKit and similar depth-sensing SDKs — needed to capture the kind of data PinpointQA evaluates
Private Watchlist
- Figure AI — humanoid robotics requiring precise indoor object manipulation
- Physical Intelligence (pi) — general-purpose robot learning using vision-language models
- Enchanted Tools — assistive robot development
- Orion (Meta AR glasses division) — wearable assistive vision applications
Resources
The Paper
Small object-centric spatial understanding in indoor videos remains a significant challenge for multimodal large language models (MLLMs), despite its practical value for object search and assistive applications. Although existing benchmarks have advanced video spatial intelligence, embodied reasoning, and diagnostic perception, no existing benchmark directly evaluates whether a model can localize a target object in video and express its position with sufficient precision for downstream use. In this work, we introduce PinpointQA, the first dataset and benchmark for small object-centric spatial understanding in indoor videos. Built from ScanNet++ and ScanNet200, PinpointQA comprises 1,024 scenes and 10,094 QA pairs organized into four progressively challenging tasks: Target Presence Verification (TPV), Nearest Reference Identification (NRI), Fine-Grained Spatial Description (FSD), and Structured Spatial Prediction (SSP). The dataset is built from intermediate spatial representations, with QA pairs generated automatically and further refined through quality control. Experiments on representative MLLMs reveal a consistent capability gap along the progressive chain, with SSP remaining particularly difficult. Supervised fine-tuning on PinpointQA yields substantial gains, especially on the harder tasks, demonstrating that PinpointQA serves as both a diagnostic benchmark and an effective training dataset. The dataset and project page are available at https://rainchowz.github.io/PinpointQA.