🛰️ GeoSeg-OV: Bridging Geospatial Gaps with Structural Guidance for Open-Vocabulary Remote Sensing Segmentation

Ruizhong Liu¹, Tingzhang Luo², Zaiyan Zhang³, Jundong Chen⁴, Hongruixuan Chen⁵, Shaoguang Huang^1,*, Hongyan Zhang¹

¹China University of Geosciences   ²City University of Hong Kong   ³Wuhan University   ⁴Shiga University   ⁵RIKEN AIP   ^*Corresponding author

Overview  |  Benchmark  |  Quick Start  |  Model Zoo  |  Results  |  Citation

GeoSeg-OV is a structure-guided cost aggregation framework for open-vocabulary remote sensing semantic segmentation. Instead of using auxiliary vision foundation model features as additional visual-text matching evidence, GeoSeg-OV keeps the CLIP matching space intact and uses frozen VFM features as structural priors to guide where cost tokens exchange information.

📰 News

• [2026.08] Paper released on arXiv.
• [2026.06] Repository created.

🌍 Overview

GeoSeg-OV addresses open-vocabulary semantic segmentation for remote sensing imagery, where models must recognize user-specified land-cover categories across heterogeneous geographic regions, sensors, and spatial resolutions. The project provides the training and evaluation code for GeoSeg-OV, together with dataset registration files, class vocabularies, and scripts for cross-dataset evaluation on the HRLC benchmark.

GeoSeg-OV follows the cost aggregation paradigm and introduces structural priors from frozen vision foundation models to improve cross-dataset generalization while preserving CLIP-based open-vocabulary recognition.

🗺️ HRLC Benchmark

The HRLC benchmark is designed for cross-dataset open-vocabulary land cover segmentation. Models are trained on one source dataset and evaluated on all remaining datasets without fine-tuning.

Dataset	Split in this repo	Role	Link
FLAIR	FLAIR_train_sem_seg, FLAIR_val_sem_seg, FLAIR_all_sem_seg	Train / Eval	split / all
OpenEarthMap	OEM_train_sem_seg, OEM_val_sem_seg, OEM_all_sem_seg	Train / Eval	split / all
LoveDA	LoveDA_all_sem_seg	Eval	LoveDA.zip
EarthMiss	EarthMiss_all_sem_seg	Eval	EarthMiss.zip
DeepGlobe	DeepGlobe_all_sem_seg	Eval	DeepGlobe.zip
Potsdam	Potsdam_all_sem_seg	Eval	Potsdam.zip
Vaihingen	Vaihingen_all_sem_seg	Eval	Vaihingen.zip

Category vocabularies are stored in datasets/. Evaluation uses the original category names from each dataset as text prompts.

⚡ Quick Start

Environment

This codebase is developed with PyTorch, Detectron2, CLIP, and Depth Anything V2.

conda create -n geoseg python=3.9.25 -y
conda activate geoseg

pip install torch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 --index-url https://download.pytorch.org/whl/cu121
pip install -r requirements.txt

Note: PyTorch, TorchVision, and Detectron2 versions need to be compatible. See pytorch.org for PyTorch/CUDA matching and the Detectron2 installation docs for supported combinations.

Reference environment used during development:

Package	Version
Python	3.9
PyTorch	2.1.2 + CUDA 12.1
TorchVision	0.16.2
Detectron2	installed from source
GPU	NVIDIA RTX 4090

VFM Setup

This project uses Depth Anything V2 as a VFM encoder. Our framework is flexible and can work with different VFMs, such as DINO or SAM; we chose Depth Anything V2 for its achieving better performance in our experiments. Download the pretrained weights:

• depth_anything_v2_vitb.pth

Place them in ./pretrained/ directory, and modify the following paths in cat_seg/cat_seg_model.py:

# Line 19: Depth Anything V2 source code path
sys.path.insert(0, '/data1/ruizhong_data/GeoSeg/Depth-Anything-V2')  # <-- modify this

# Line 80: ViT-B depth weights path
depth_path = '/data1/ruizhong_data/GeoSeg/pretrained/depth_anything_v2_vitb.pth'  # <-- modify this

Place Depth Anything V2 encoder checkpoints under pretrained/.

Encoder	File name	Link
Depth Anything V2 ViT-B	depth_anything_v2_vitb.pth	TODO

Then update the checkpoint paths in cat_seg/cat_seg_model.py if your local layout differs.

Dataset Setup

The current dataset registration files assume one root directory containing all datasets:

GHRLandCover/
+-- FLAIR_split/
|   +-- train/
|   |   +-- images/
|   |   +-- labels/
|   +-- val/
|       +-- images/
|       +-- labels/
+-- FLAIR/
|   +-- images/
|   +-- labels/
+-- OEM_split/
|   +-- train/
|   |   +-- images/
|   |   +-- labels/
|   +-- val/
|       +-- images/
|       +-- labels/
+-- OEM/
|   +-- images/
|   +-- labels/
+-- LoveDA/
|   +-- images/
|   +-- labels/
+-- EarthMiss/
|   +-- images/
|   +-- labels/
+-- DeepGlobe/
|   +-- images/
|   +-- labels/
+-- Potsdam/
|   +-- images/
|   +-- labels/
+-- Vaihingen/
    +-- images/
    +-- labels/

Update the root variable in the registration files under cat_seg/data/datasets/ to your local dataset path:

root = "TODO_LOCAL_PATH_TO/GHRLandCover"

🏆 Model Zoo

Pretrained GeoSeg-OV checkpoints and reproduction logs are provided below.
mIoU Avg. is the mean mIoU over the six held-out datasets in the HRLC cross-dataset protocol (excluding the training source).

Training set	Backbone	Auxiliary VFM	mIoU Avg.	Checkpoint	Log
FLAIR	CLIP ViT-B/16	Depth Anything V2 ViT-B	44.64	output_flair1.zip	eval_flair_log.txt
OpenEarthMap	CLIP ViT-B/16	Depth Anything V2 ViT-B	42.90	output_oem1.zip	eval_oem_log.txt

Download a checkpoint zip, unzip it in the repo root, then evaluate with the corresponding script (weights are output_flair1/model_final.pth or output_oem1/model_final.pth):

# FLAIR-trained model
wget -O output_flair1.zip "https://huggingface.co/datasets/rzliu1026/GHRLandCover/resolve/main/output_flair1.zip?download=true"
unzip output_flair1.zip
sh eval_flair.sh configs/vitb_384_flair.yaml 1 output_flair1

# OpenEarthMap-trained model
wget -O output_oem1.zip "https://huggingface.co/datasets/rzliu1026/GHRLandCover/resolve/main/output_oem1.zip?download=true"
unzip output_oem1.zip
sh eval_oem.sh configs/vitb_384_oem.yaml 1 output_oem1

Training

Train GeoSeg-OV on FLAIR with CLIP ViT-B/16:

sh run_flair.sh configs/vitb_384_flair.yaml 1 output_flair

Train GeoSeg-OV on OpenEarthMap with CLIP ViT-B/16:

sh run_oem.sh configs/vitb_384_oem.yaml 1 output_oem

The training scripts automatically launch the corresponding cross-dataset evaluation script after training finishes.

Evaluation

Evaluate a FLAIR-trained model:

sh eval_flair.sh configs/vitb_384_flair.yaml 1 output_flair

Evaluate an OpenEarthMap-trained model:

sh eval_oem.sh configs/vitb_384_oem.yaml 1 output_oem

For direct Detectron2-style evaluation on a single target dataset:

Note: FLAIR dataset does not use sliding window due to its large number of images.

python train_net.py \
  --config configs/vitb_384_flair.yaml \
  --num-gpus 1 \
  --eval-only \
  OUTPUT_DIR output/eval_single \
  MODEL.WEIGHTS output_flair1/model_final.pth \
  MODEL.SEM_SEG_HEAD.TEST_CLASS_JSON datasets/Potsdam.json \
  DATASETS.TEST '("Potsdam_all_sem_seg",)' \
  TEST.SLIDING_WINDOW True \
  MODEL.SEM_SEG_HEAD.POOLING_SIZES "[1,1]"

📈 Results

Main cross-dataset results and qualitative comparisons will be updated after release.

📁 Repository Structure

GeoSeg/
+-- cat_seg/                 # GeoSeg-OV model, CAT-Seg components, dataset registration
+-- configs/                 # ViT-B training configs
+-- datasets/                # Category vocabularies for text prompts
+-- demo/                    # Image / video visualization scripts
+-- Depth-Anything-V2/       # Auxiliary structural encoder source
+-- paper/                   # LaTeX paper source
+-- pretrained/              # Put downloaded VFM checkpoints here
+-- train_net.py             # Detectron2 training and evaluation entry
+-- run_flair.sh             # Train on FLAIR, then evaluate
+-- run_oem.sh               # Train on OpenEarthMap, then evaluate
+-- eval_flair.sh            # Cross-dataset evaluation for FLAIR-trained models
+-- eval_oem.sh              # Cross-dataset evaluation for OpenEarthMap-trained models

📖 Citation

If GeoSeg-OV is useful for your research, please consider citing our work.

@article{liu2026geoseg,
  title={GeoSeg-OV: Bridging Geospatial Gaps with Structural Guidance for Open-Vocabulary Remote Sensing Segmentation},
  author={Liu, Ruizhong and Luo, Tingzhang and Zhang, Zaiyan and Chen, Jundong and Chen, Hongruixuan and Huang, Shaoguang and Zhang, Hongyan},
  journal={arXiv preprint arXiv:2608.10426},
  year={2026}
}

🙏 Acknowledgements

This project is built upon excellent open-source work including CAT-Seg, Detectron2, CLIP, and Depth Anything V2. We sincerely thank the authors and contributors for making their code and models publicly available.

📬 Contact

For questions about the code, data, or paper, please open an issue or contact:

• Ruizhong Liu: rzliu1026@gmail.com
• TODO additional contact