Firecrawl Agents — Use Cases

A collection of practical examples using Firecrawl's agent-based extraction to pull structured data from websites into clean CSVs — no scraping logic required.

What This Does

Instead of writing custom scrapers for each website, this notebook defines what data you want using Pydantic schemas, and lets Firecrawl's AI agent figure out how to extract it. Each use case follows the same pattern:

1. Define a Pydantic schema describing the data structure
2. Pass a URL (or let the agent search) + a natural language prompt
3. Get back a validated DataFrame, auto-saved to Google Drive

Setup

Requirements:

• Google Colab (uses google.colab for secrets and Drive access)
• Firecrawl API key (stored in Colab's userdata as Firecrawl_API)
• Google Drive mounted`

Install:

pip install -q firecrawl

Use Cases

#	Use Case	Source	Output
1	Serie A football clubs	Agent search (no URL)	Club names, URLs, managers
2	Exa product offerings	exa.ai	Products, pricing, descriptions
3	Snowflake vs Databricks	Both company sites	Side-by-side product comparison
4	Ramp vs Brex	Both company sites	Fintech product comparison
5	YC startup industries	ycombinator.com	Industry/sub-industry breakdown with company counts
6	Vector databases	Agent search (no URL)	12+ vector DBs with features, pricing, integrations
7	EPL club ownership	Agent search (no URL)	20 clubs with owner details, acquisition info, multi-club structures
8	Football player stats	Wikipedia pages	Season-by-season career stats for Mbappé, Messi, Ronaldo, R9
9	MotherDuck integrations	MotherDuck docs	Data ingestion tools compatible with MotherDuck
10	Claude blog posts	claude.com/blog	Recent posts with titles, URLs, summaries, dates

How It Works

The core helper function abstracts away the repetitive parts:

def extract_and_save(schema_class, urls, prompt, filename):
    result = app.agent(schema=schema_class, urls=urls, prompt=prompt, model="spark-1-mini")
    # Validates response → builds DataFrame → saves CSV to Google Drive
    return df

Each use case only needs to define a Pydantic schema and call extract_and_save. The agent handles navigation, content discovery, and structured extraction.

Key Patterns Demonstrated

• URL-based extraction — point the agent at specific pages (Use Cases 2, 3, 4, 9, 10)
• URL-free search — let the agent find sources on its own (Use Cases 1, 6, 7)
• Multi-URL comparison — extract from competing products side by side (Use Cases 3, 4)
• Wikipedia table extraction — pull structured stats from wiki pages (Use Case 8)
• Nested schema design — typed inner models with Optional fields for graceful handling of missing data

Tech Stack

• Firecrawl — AI-powered web agent and extraction
• Pydantic — Schema definition and validation
• pandas — DataFrame creation and CSV export
• Google Colab — Runtime environment and secrets management
• Google Drive — Output storage