RAGMind

AI-Powered Website Intelligence & Dataset Builder

RAGMind is a web crawling and dataset generation platform designed for Retrieval-Augmented Generation (RAG), AI training pipelines, semantic search systems, and knowledge base construction.

The system crawls websites, extracts meaningful content, removes duplicate information, classifies pages, chunks content into AI-friendly segments, and exports structured datasets ready for LLM workflows.

📸 Live Dashboard Preview

Crawling Intelligence Dashboard

Active Crawl Process

Dataset Output

---

Features

Intelligent Website Crawling

• Internal link discovery
• Sitemap seeding
• Robots.txt compliance
• URL normalization
• Priority-based crawling
• Depth control
• Anti-bot page detection

Smart Content Extraction

• Trafilatura-powered extraction
• Boilerplate removal
• XML support
• Dynamic page support via Playwright
• Automatic weak-content detection

Data Quality Layer

• Exact duplicate detection
• Near-duplicate detection
• Content validation
• Block-page filtering

AI Dataset Processing

• Page classification
• Content chunking
• Importance scoring
• Metadata enrichment

Export Formats

• JSONL (LLM / RAG Ready)
• CSV

Interactive Dashboard

• Live crawl monitoring
• Dataset statistics
• Export controls
• Responsive UI
• LangChain-inspired intelligence dashboard

---

Architecture

Website
   │
   ▼
Crawler
   │
   ▼
Content Extraction
   │
   ▼
Quality Filtering
   │
   ▼
Deduplication
   │
   ▼
Classification
   │
   ▼
Chunking
   │
   ▼
Dataset Builder
   │
   ▼
JSONL / CSV Export

---

Project Structure

RAGMind/

├── crawler/
│   ├── crawl.py
│   └── playwright_fetcher.py
│
├── extractor/
│   └── clean.py   
│   └── xml_parser.py 
├── processor/
│   ├── chunk.py
│   ├── dataset_builder.py
│   └── exporter.py
│
├── classifier/
│   └── page_type.py
│
├── utils/
│   ├── normalize.py
│   ├── sitemap.py
│   ├── robots.py
│   ├── dedupe.py
│   ├── page_quality.py
│   └── content_quality.py
│
├── frontend/
│   ├── index.html
│   ├── style.css
│   └── app.js
│
├── output/
│
├── main.py
└── README.md

---

Installation

playwright install

Clone the repository:

git clone https://github.com/sharondroid/RAGMind.git

cd RAGMind

Create virtual environment:

python -m venv venv

Activate:

Windows

venv\Scripts\activate

Mac/Linux

source venv/bin/activate

Install dependencies:

pip install -r requirements.txt

---

Running the API

Start FastAPI:

python -m uvicorn main:app --reload

API:

http://127.0.0.1:8000

Dashboard:

http://127.0.0.1:8000/ui

---

Example Crawl

Request:

POST /crawl?url=https://docs.stripe.com&max_pages=20

Response:

{
  "pages_crawled": 20,
  "dataset_records": 143,
  "exported": true
}

---

Output Example

JSONL

{
  "id": 11,
  "content": "AWS Cloud Practitioner Tutorial ... (full page text)",
  "content_stats": {
    "length": 1011,
    "total_page_length": 1011
  },
  "source": {
    "url": "https://www.w3schools.com/aws/index.php",
    "title": "AWS Cloud Practitioner Tutorial",
    "page_type": "pricing"
  },
  "chunk": {
    "index": 0
  },
  "crawl": {
    "used_playwright": false,
    "priority_score": null,
    "depth": null,
    "is_duplicate": false
  },
  "enrichment": {
    "keywords": ["cloud", "tutorial", "practitioner", "learn", "skills", "time", "overview", "services", "large", "content"],
    "importance_score": 8
  }
}

---

Use Cases

• RAG Dataset Creation
• Enterprise Knowledge Bases
• AI Chatbots
• Internal Search Engines
• Semantic Search Systems
• Documentation Mining
• LLM Fine Tuning Pipelines
• Content Intelligence Platforms

---

Tech Stack

• Python
• FastAPI
• BeautifulSoup
• Trafilatura
• Playwright
• HTTPX
• JavaScript
• HTML
• CSS

---

Future Roadmap

• Vector Embeddings
• Semantic Search Layer
• LangChain Integration
• LlamaIndex Integration
• Async Concurrent Crawling
• Distributed Crawling
• PostgreSQL Storage
• FAISS Vector Store
• OpenAI Embedding Support
• Crawl Scheduling

---

Author

Built by Sharon Attafi

AI Systems Developer

Product Manager/ BizOps

Ghana