A project by: Kjetil Indrehus
Document Visual Question Answering (DocVQA) requires vision–language models to reason not only about what information in a document is relevant to a question, but also where the answer is grounded on the page. Despite strong predictive performance, existing DocVQA systems entangle these two aspects and operate largely as black boxes, offering limited means to verify how predictions depend on visual evidence. We propose CoExVQA, a self-explainable DocVQA framework that enforces a grounded reasoning process through a chain-of-explanation design. The model first identifies question-relevant evidence, then explicitly localizes the answer region, and finally decodes the answer exclusively from the grounded region. By making both evidence selection and spatial grounding intrinsic to prediction, CoExVQA enables direct inspection and verification of the reasoning process across modalities. Empirical results show that restricting decoding to grounded evidence yields competitive performance while providing transparent and verifiable predictions.
Note
This work is part of the following submissions:
- Submitted to NeurIPS 2026
- MSc. thesis project of Kjetil Indrehus for Informatics: Programming and Systems Architecture, University of Oslo (2024-2026).
This project uses an augmented version of the DocVQA dataset with additional annotations for question priors and answer locations. See Dataset Setup for more details.
For detailed setup instructions, see Environment Setup.
Quick start with conda:
conda env create -f environment.yml
conda activate coexvqaThe trained model is available on Hugging Face: indrehus/CoExVQA
To load the model:
from src.model.model import CoExVQA
model = CoExVQA.from_hf(repo_id="indrehus/CoExVQA").to("cuda")
model.eval()To use the model:
import torch
from PIL import Image
image = Image.open("path/to/document.png").convert("RGB")
question = "What is the total amount?"
with torch.no_grad():
# Forward pass
outputs = model(images=[image], questions=[question])
# Get text answer from outputs
preds = model.generate_preds_from_output(outputs)
# Predicted text answer, e.g. "Predicted answer: $42.00"
print("Predicted answer:", preds[0])
# Predicted bounding box in normalized cxcywh format, e.g. "Predicted bbox (cxcywh): torch.Size([1, 4]) [0.52, 0.78, 0.14, 0.03]"
print("Predicted bbox (cxcywh):", outputs["box"].shape, outputs["box"][0].tolist())
# Question attention mask over image patches, e.g. "Question heatmap (q_mask): torch.Size([1, 765, 1])"
print("Question heatmap (q_mask):", outputs["q_mask"].shape)Note
The training and evaluation scripts have default paths for model checkpoints and datasets that must be set for your environment before running (e.g. PROJECT_AREA_PATH, MODEL_SAVE_DIR_PATH).
Train the dual projector model:
python -m src.scripts.training.train \
--experiment_name RUN_A1 \
--d_model 768 \
--epochs 40 \
--ds_name indrehus/docvqa-single-page-questions-answer-ocr-colSmol500M-q-priors \
--config base \
--iou_loss_type giou \
--lambda_iou 1.0 \
--lambda_center 1.5 \
--lambda_edges 0.1 \
--lambda_area 0.1 \
--lambda_q_prior 1.0 \
--lambda_dec 0.5 \
--lr 3e-5 --backbone_lr 1e-5 --weight_decay 1e-2 --batch_size 24 --optimizer adamw --scheduler reduceonplateau \
--augmentation artifacts \
--project_name="End-to-End (8B)" \
--use_decoder \
--decoder_loss_warmup_epochs 15Training metrics are logged with Weights & Biases. See Experiment Logging for setup and details.
Evaluate a trained model by using our pretrained model found on Huggingface:
python -m src.scripts.evaluation.eval --model_checkpoint_dir indrehus/CoExVQA This project includes a test suite for development and debugging purposes. The tests verify that individual components (loss functions, model blocks, projectors, etc.) work correctly, but do not test model prediction accuracy or final performance.
Read more about the testing tool Pytest here: https://docs.pytest.org/en/stable/
To run all tests:
pytestTo run tests with coverage report:
pytest --cov=src --cov-report=term-missing.
├── src/
│ ├── datasets/ # Dataset loaders and preprocessing
│ ├── evaluation/ # Evaluation metrics and utilities
│ ├── model/ # Model architectures and components
│ ├── pipelines/ # Prior generation
│ ├── priors/ # Prior handling
│ ├── scripts/
│ │ ├── training/ # Training scripts
│ │ ├── evaluation/ # Evaluation scripts
│ │ └── generation/ # Generation scripts
│ └── utils/ # Utility functions and helpers
├── tests/ # Test suite
├── slurm_scripts/ # SLURM job submission scripts
├── assets/ # Static assets
├── requirements.txt # Python dependencies (pip)
└── environment.yml # Conda environment specification
Note: The repository includes experimental code and alternative implementations. The key files for the main model are the training script at src/scripts/training/train.py and the evaluation script at src/scripts/evaluation/eval.py.
This project is licensed under the CC BY 4.0 License. If you found the work useful, please give proper citation for both the original dataset used and our work.
If you use this code or dataset in your research, please cite:
@misc{indrehus2026scoexvqa,
title={Towards Self-Explainable Document Visual Question Answering with Chain-of-Explanation Predictions},
author={Kjetil Indrehus and Adrian Duric and Changkyu Choi and Ali Ramezani-Kebrya},
year={2026},
eprint={2605.06058},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2605.06058},
}This work uses the DocVQA dataset:
@misc{mathew2021docvqadatasetvqadocument,
title={DocVQA: A Dataset for VQA on Document Images},
author={Minesh Mathew and Dimosthenis Karatzas and C. V. Jawahar},
year={2021},
eprint={2007.00398},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2007.00398},
}