Spatial Floorplan Lab

Research prototype for converting indoor handheld RGB video into rough 2D floor plan drafts through segmentation, monocular depth estimation, and camera pose analysis.

This project is intentionally experimental. The current goal is not a production-grade measured floor plan, but a plausible ratio-based layout draft that can preserve the rough spatial relationships of walls, openings, free space, furniture, and obstacles.

Goals

• Estimate indoor spatial structure from regular smartphone video or photos.
• Avoid LiDAR, TrueDepth, AR-only sensors, or device-specific hardware requirements.
• Build a post-processing MVP first, with real-time capture assistance left as a future extension.
• Represent uncertain structure as candidates with confidence, not as guaranteed geometry.
• Keep a record of experiments, failed paths, and design decisions so the technical direction stays honest.

Current Direction

The current working direction is a hybrid RGB-only pipeline:

1. Extract frames from a handheld video.
2. Run semantic segmentation to identify floor, wall, opening, and object regions.
3. Estimate relative monocular depth for each frame.
4. Use COLMAP camera poses as the shared coordinate system.
5. Back-project segmentation/depth evidence into a common top-down space.
6. Generate a rough floor plan candidate from accumulated free-space, occupied-space, and wall-boundary evidence.

Early experiments showed that segmentation-only and sparse-point-only approaches are not reliable enough for cluttered commercial interiors. The most promising direction so far is:

COLMAP camera pose + semantic segmentation + relative depth

Scope

Target spaces include ordinary rooms as well as small commercial interiors such as cafes and restaurants. The first test video is an approximately 20-pyeong cafe interior.

The pipeline assumes that people are intentionally kept out of the capture video. This is a capture guideline, not a model guarantee.

Repository Layout

• src/: pipeline scripts and layout generation utilities
• docs/: experiment reports, decisions, and technical notes
• examples/: sample output schema
• experiments/: lightweight experiment notes
• data/README.md: local data folder guide

Large local files are intentionally ignored by git:

• data/input/: original videos and photos
• data/output/: generated frames, masks, depth maps, COLMAP outputs, and visualizations
• .venv/: local Python virtual environment

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Some experiments also require a local COLMAP installation.

Example Workflow

Extract frames:

python -m src.floorplan.extract_frames data/input/IMG_0548.mov data/output/IMG_0548/frames_3fps --fps 3

Run semantic segmentation:

python -m src.floorplan.segment_frames data/output/IMG_0548/frames_3fps data/output/IMG_0548/segmentation_3fps

Run monocular depth estimation:

python -m src.floorplan.run_depth_estimation data/output/IMG_0548/frames_3fps data/output/IMG_0548/depth_3fps

The later COLMAP fusion steps expect a COLMAP sparse reconstruction exported as text.

Experiment Summary

The first test video is IMG_0548.mov, an approximately 35.5-second handheld cafe video. Frames were sampled at 1fps, 3fps, and 5fps. The project settled on 3fps for the next experiments because it produced 107 frames, gave much denser evidence than 1fps, and did not show a clear quality gain from 5fps at the segmentation-only stage.

What worked:

• SegFormer produced usable semantic masks for floor/free-space, commercial objects, and openings. This made it possible to generate the first layout JSON/SVG drafts.
• COLMAP successfully registered all 107 frames in the 3fps set. The best sparse reconstruction contained about 12.5K sparse points with a mean reprojection error of about 0.72px.
• Combining COLMAP sparse observations with SegFormer labels separated some free-space, occupied-space, and opening evidence better than segmentation-only ratio maps.
• Depth Anything V2 Small produced relative depth for all sampled frames, and 104 frames passed the alignment/filtering step for dense back-projection.
• The densest hybrid experiment produced a rough layout candidate with one room polygon, five occupied-space clusters, and two opening candidates.

What did not work well:

• A segmentation-only ratio map collapsed furniture and obstacles into large blobs because frames were accumulated without a real shared camera coordinate system.
• Splitting the video into view segments helped organize the evidence, but the composed result was closer to an unfolded sweep map than a true top-down floor plan.
• COLMAP sparse points alone were not enough to infer walls. They captured camera motion well, but mixed walls, furniture, counters, texture, and objects in the same sparse point cloud.
• Point-cloud support boundaries generated meaningful outer-edge hints, but they could not reliably distinguish actual walls from counters, furniture fronts, and other structures.
• Direct semantic wall back-projection produced many wall-labeled points, but the result was still noisy because visible wall regions do not always correspond to the true floor-level wall boundary.
• Per-frame wall/floor boundary mapping was cleaner than full wall-surface back-projection, but in the cafe video many true wall-floor contact lines were hidden by furniture and counters.
• "Build a 3D model with COLMAP and rotate it into a top view" did not work in the current local setup. COLMAP SfM worked, but dense stereo could not run because the installed COLMAP 4.1.0 build had no CUDA support, and the sparse Delaunay mesh top view was not strong enough for floor plan extraction.

Current conclusion:

The project has not yet reached a reliable floor plan generator. The strongest evidence so far is that camera pose estimation is usable, but wall inference remains the main blocker. The next useful direction is still a hybrid approach: keep COLMAP poses as the coordinate backbone, use semantic/depth evidence for floor/object/opening regions, and treat wall lines as hypotheses that need repeated support across many frames instead of single-frame certainty.

Attribution And Third-Party Notices

This repository does not vendor third-party model weights, datasets, or COLMAP binaries. The following projects and models were used or referenced during experimentation:

• COLMAP was used for structure-from-motion and camera pose estimation. COLMAP is licensed under the new BSD license according to its official license page.
• SegFormer is the semantic segmentation architecture used through the Hugging Face model nvidia/segformer-b0-finetuned-ade-512-512. The original NVIDIA SegFormer repository states that SegFormer may be used non-commercially for research or evaluation purposes only, so commercial use needs a separate license review.
• Depth Anything V2 Small was used for relative monocular depth estimation through Hugging Face Transformers. The Hugging Face model card lists the Small checkpoint as Apache-2.0.
• Structured3D was used only as a conceptual reference for structured indoor layout representation. Its GitHub README states that the code is MIT licensed and the dataset is released under the Structured3D Terms of Use. No Structured3D data is included in this repository.
• Hugging Face Transformers, PyTorch, and OpenCV are used as runtime dependencies through requirements.txt.

Please check each upstream project's current license before redistribution, commercial use, or publishing derived model outputs. This README is a project note, not legal advice.

Project License

No project-level license file has been added yet. Until a license is chosen, all rights are reserved by default.