📄 doc2md — Offline Document ETL for LLM Context Optimization

Convert PDFs, DOCX files, and images into clean, structured, and LLM-ready Markdown. 100% offline, private, and air-gapped.

---

💡 Why doc2md?

Most LLM orchestrators and RAG pipelines waste precious context window space and compute processing raw, unformatted documents.

doc2md solves this by pre-converting files into highly structured Markdown with standardized metadata headers. Your local or cloud models (e.g., Ollama, OpenClaw, Llama 3, Claude, GPT-4) read pure semantic signal rather than noise, resulting in faster execution, lower costs, and more accurate retrieval.

---

✨ Features

• 🔍 Recursive Scanning: Automatically crawls directories for .pdf, .docx, .jpg, .png, .tiff, and more.
• 🧠 Smart Routing: Attempts digital parsing first, with automatic OCR fallback if the file is a scanned copy/image.
• 🛡️ 100% Offline & Private: Zero external API calls. Your documents never leave your machine.
• ⚡ Incremental Caching: Skips files that have already been converted to speed up subsequent runs.
• 📊 Data Lineage: Every generated .md file starts with a YAML frontmatter tracking the source file name, relative path, extraction method, and timestamp.

---

🛠️ Technology Stack

Layer	Component	Description
Primary Parser	Microsoft MarkItDown	Standard layout-preserving document converter.
PDF Extraction	PyMuPDF4LLM	RAG-optimized text extraction from PDF files.
OCR Engine	Tesseract OCR	Offline optical character recognition engine.
OCR Wrapper	pytesseract + pdf2image	Python wrappers for OCR pipeline.
Target Runtime	Python 3.10+	Validated on macOS (Apple Silicon arm64).

---

🚀 Quick Start

1. Install System Dependencies

Install Tesseract OCR (with languages) and Poppler (for PDF rendering):

# macOS (using Homebrew)
brew install tesseract tesseract-lang poppler

2. Set Up Python Environment

Clone the repository, initialize your virtual environment, and install dependencies:

# Create and activate virtual environment
python3 -m venv .venv
source .venv/bin/activate

# Install requirements
pip install -r requirements.txt

3. Run the ETL Pipeline

Run the converter on your folder of documents:

python etl_to_markdown.py "/path/to/your/documents"

For detailed system setup, custom language configurations, and troubleshooting, refer to the RUNBOOK.md.

---

📋 Command Line Interface Options

usage: etl_to_markdown.py [-h] [--lang LANG] [--debug] input_dir

ETL: Converte PDFs/DOCX/Imagens para Markdown LLM-ready.

positional arguments:
  input_dir    Root directory containing source documents

options:
  -h, --help   show this help message and exit
  --lang LANG  Tesseract language(s) (default: por+eng)
  --debug      Verbose logging (level DEBUG)

---

📁 Output Example

Every output document generates a Markdown file with a YAML frontmatter header detailing its origin and extraction metadata:

---
source_file: "contract_2024.pdf"
source_relative_path: "docs/contract_2024.pdf"
source_absolute_path: "/Users/usuario/.../workspace_docs/docs/contract_2024.pdf"
extraction_method: "tesseract-ocr-pdf"
converted_at_utc: "2026-06-10T02:15:32+00:00"
---

# Service Agreement — January 2024

**Company:** Acme Corporation S.A.
...

---

🏗️ Project Structure

doc2md/
├── etl_to_markdown.py   # ETL pipeline implementation (10 SRP functions)
├── requirements.txt     # Python package requirements
├── RUNBOOK.md           # Installation, configuration, and troubleshooting guide
└── README.md            # Project overview and introduction

---

📐 Design Principles

• 📴 Offline-First: Engineered specifically for air-gapped, high-privacy environments.
• ⚙️ Zero-AI Rule: Document conversion is strictly algorithmic and deterministic, avoiding generative hallucinations.
• 📐 Single Responsibility Principle (SRP): The codebase is modular, readable, and easy to maintain.
• 🌐 LLM-Agnostic: Outputs work flawlessly with any chat framework, agent engine, or vector database.

---

📄 License

This project is licensed under the MIT License.