A production-quality universal shopping search engine that federates product discovery across multiple retailers with AI-assisted natural-language query planning and deterministic ranking.
Browser Query
→ [WebLLM] creates SearchPlan JSON (optional, progressive enhancement)
→ browser sends query + candidate plan to backend
→ backend validates plan
→ backend searches eligible retail providers concurrently
→ backend normalizes and entity-resolves results
→ backend deterministically filters and ranks products/offers
→ backend returns structured results
→ browser renders ranked products with offer comparisons
The search engine works without WebLLM. WebLLM is progressive enhancement, not a hard dependency.
- User types a natural-language query (e.g., "best waterproof trail running shoes under $150 that ship to 11756; prioritize trail grip and wide sizes")
- Browser: If WebLLM is available, it generates a
SearchPlanwith parsed constraints, search queries, and ranking criteria - Browser: Sends
POST /api/searchwith the query, preferences, and candidate plan - Server: Validates the plan (or falls back to deterministic planning)
- Server: Determines eligible providers based on plan + capabilities
- Server: Searches all eligible providers concurrently with timeout/retry
- Server: Normalizes results into common
ProviderProductCandidateformat - Server: Entity-resolves equivalent products across retailers (by title similarity)
- Server: Applies deterministic hard filters (price, condition, availability, brands, features)
- Server: Ranks products and offers using plan-specified criteria weights
- Server: Returns
SearchApiResponsewith ranked products, metadata, and source coverage - Browser: Renders ranked product cards with offer comparisons, reasons, tradeoffs, and ranking transparency
- Framework: Next.js 16 (App Router)
- Language: TypeScript end-to-end
- Styling: Tailwind CSS v4
- Icons: Lucide React
- Validation: Zod
- Testing: Vitest
- API Integration: Walmart, Best Buy, Target, eBay, Costco, Shopify Global Catalog MCP
- AI: WebLLM architecture (progressive enhancement, currently mock fallback)
All retailer API credentials are server-side only. The browser never receives API keys, tokens, provider URLs, or raw provider responses. This is a hard security boundary:
- Browser: Only receives structured, sanitized
SearchApiResponseJSON - Server: The only layer allowed to access retailer APIs, Shopify MCP, and credentials
- WebLLM: Runs locally in browser; never transmits prompts or user search text to any server
Each retailer has a dedicated adapter implementing RetailerSearchProvider:
| Provider | Capabilities |
|---|---|
| Walmart | Keyword search, shipping data, availability |
| Best Buy | Keyword search, brand/price filters, rich category/manufacturer data |
| Target | Keyword search, store-level pricing, pickup availability |
| eBay | Keyword search, GTIN lookup, condition/price filters, marketplace seller data |
| Costco | Keyword search, availability (requires session cookies) |
| Shopify | Full keyword search, category/brand/price/destination filters, variants, seller data (Global Catalog MCP) |
Unsupported filters are explicitly reported in metadata.unsupportedFilters. Adapters never pretend an unsupported filter was applied.
Cross-retailer product matching uses title similarity (Jaccard-like token overlap). In priority order:
- Shopify UPID — within Shopify, same UPID = same product
- Title similarity > 90% → medium confidence merge
- Title similarity > 75% → low confidence merge
- Below 75% → kept separate (avoids false merges)
The system never merges products with different model numbers, generations, variants, or conditions. Match confidence is preserved for diagnostic transparency.
Ranking is deterministic server-side code — the LLM does not determine final rank.
Each plan specifies up to 8 ranking criteria with weights summing to 1.0:
price,featureMatch,availability,shipping,pickupseller,retailer,brandPreference,variantFit,condition,preferenceFit
Rules:
- Rank canonical product fit first, then offers within each product
- Compare prices only when currency, condition, and variant are comparable
- Penalize missing data (never award favorable assumptions)
- Prefer new condition by default
- Use stable tie-breakers for deterministic results
- Never claim "best everywhere" or "lowest price online"
Architecture for browser-local AI planning using @mlc-ai/web-llm:
- Default model: Qwen3-1.7B-q4f16_1-MLC
- Low-memory fallback: Qwen3-4B-q4f16_1-MLC
- Enhanced model: Qwen3-8B-q4f16_1-MLC
- Role: Parse natural-language queries into
SearchPlanJSON - Not used for: Factual product data, pricing, filtering, ranking
Currently using a mock adapter (always falls back to deterministic planning). To enable actual WebLLM, replace the adapter in lib/webllm/client.ts.
WebGPU compatibility is detected cleanly. If unavailable, the app degrades gracefully to "fast search mode" with the deterministic fallback planner.
# Walmart API
WALMART_CONSUMER_ID=
WALMART_PRIVATE_KEY_BASE64=
WALMART_KEY_VERSION=1
# Best Buy API
BESTBUY_API_KEY=
# Target API
TARGET_STORE_ID=
TARGET_ZIP=
# eBay API
EBAY_CLIENT_ID=
EBAY_CLIENT_SECRET=
EBAY_SANDBOX=false # set true when using eBay sandbox keys
# Costco
COSTCO_COOKIES=
COSTCO_API_KEY=
COSTCO_REFRESH_SECRET=
# Cron auth (when set, /api/cron/refresh-costco requires this as Bearer token)
CRON_SECRET=
# Shopify
SHOPIFY_CLIENT_ID=
SHOPIFY_CLIENT_SECRET=
SHOPIFY_AGENT_PROFILE=
# Optional buyer-linked token (JWT) for personalized catalog + checkout auto-discounts
SHOPIFY_BUYER_TOKEN=
# WebLLM (optional, progressive enhancement)
NEXT_PUBLIC_WEBLLM_ENABLED=true
NEXT_PUBLIC_WEBLLM_DEFAULT_MODEL=Qwen3-1.7B-q4f16_1-MLC
NEXT_PUBLIC_WEBLLM_LOW_MEMORY_MODEL=Qwen3-4B-q4f16_1-MLC
NEXT_PUBLIC_WEBLLM_ENHANCED_MODEL=Qwen3-8B-q4f16_1-MLC
NEXT_PUBLIC_BASE_URL= # base URL for cron self-calls (optional; VERCEL_URL is auto-set on Vercel)
# Search configuration
SEARCH_MAX_PROVIDER_QUERIES=5
SEARCH_MAX_CONCURRENCY=4
SEARCH_MAX_ENRICHMENT_PRODUCTS=15
SEARCH_TIMEOUT_MS=20000
PROVIDER_TOOL_TIMEOUT_MS=8000
PLANNER_CLARIFICATION_THRESHOLD=0.55
npm install
npm run dev # Start dev server at http://localhost:3000
npm run build # Production build
npm run test # Run tests
npm run test:watch # Watch modeVitest with 107 tests across 13 test files:
npm testTest categories:
- Schema validation: SearchPlan parsing, rejection of invalid plans, weight validation
- Ranking engine: Price/availability/condition scoring, determinism, breakdown generation
- Entity resolution: Title matching, dissimilarity detection, empty input handling
- Provider selection: Capability-based provider eligibility and query fan-out
- Planning & filtering: Size/category extraction, adjacent-category exclusions
- Interaction learning: Deterministic preference derivation from local signals
- Shopify UCP: Cart and Checkout MCP lifecycle (mocked fetch)
Tests do not require WebGPU, browser model download, live retailer APIs, or Shopify MCP access.
- All retailer credentials are server-side only
- Raw provider responses and product images are not cached or persisted beyond request lifecycle
- Anonymous request IDs and aggregate telemetry are acceptable
- Never retain: API credentials, OAuth tokens, MCP authorization headers, full raw catalog data, identifiable user prompts, browser GPU fingerprints
- WebLLM prompts remain local to the browser; never transmitted to analytics
- Incomplete fields: Not all providers supply price, shipping, availability, brand, or condition data
- Variant comparability: Cross-retailer variant matching is title-based; exact option matching limited
- Unsupported filters: Many providers do not support brand/price/destination filters — these are labeled as unsupported
- Source coverage: Results reflect only the searched sources, not the entire web
- No guaranteed data: Shipping estimates, return policies, and availability claims are only as good as provider data
- Entity resolution: Currently title-similarity based; GTIN/UPC/MPN matching requires those fields from providers
To use a self-hosted or server-hosted OpenAI-compatible planner instead of browser-local WebLLM:
- Keep the
SearchPlanschema (it's provider-neutral) - Create a server-side planner that calls your LLM API
- Validate the output with the same Zod schemas
- Return the validated plan to the orchestrator
- Optionally remove the
lib/webllm/directory
The search orchestrator accepts any valid SearchPlan regardless of source.
emporika/
├── app/
│ ├── api/
│ │ ├── search/route.ts # GET (simple) + POST (intelligent) search API
│ │ ├── trending/route.ts # Trending products feed
│ │ ├── target/nearest-store/route.ts # Nearest Target store lookup
│ │ ├── shopify/cart/route.ts # Shopify cart operations
│ │ ├── costco/
│ │ │ ├── set-cookie/route.ts # Store Costco session cookie
│ │ │ └── refresh-cookie/route.ts # Refresh Costco session cookie
│ │ └── cron/refresh-costco/route.ts # Cron entry point (Vercel Cron in vercel.json)
│ ├── images/ # Retailer favicons
│ ├── offline/page.tsx # PWA offline page
│ ├── layout.tsx # Root layout
│ └── page.tsx # Main search UI
├── components/ # UI: SearchBar, ProductCard, ProductResultCard,
│ # SearchStatus, WebLLMStatus, ClarificationPrompt,
│ # SearchFilters, Sidebar, Pagination, SortSelect,
│ # RetailerToggle, CartDrawer, TrendingFeed, PWAInstallPrompt, ...
├── context/
│ └── CartContext.tsx # Client-side cart state
├── hooks/
│ ├── useIntelligentSearch.ts # Intelligent search flow
│ ├── useMerchantCheckout.ts # Merchant checkout handling
│ └── useTargetStore.ts # Target store selection
├── search/
│ ├── types.ts # All domain types
│ ├── schemas.ts # Zod schemas (SearchPlan, request)
│ ├── orchestrator.ts # Main search pipeline
│ ├── planner.ts # Deterministic fallback planner
│ ├── normalize.ts # Candidate normalization
│ ├── entity-resolution.ts # Cross-retailer ER
│ ├── offer-normalize.ts # Offer comparability
│ ├── filter.ts # Deterministic hard filters
│ ├── ranker.ts # Deterministic ranking engine
│ ├── errors.ts # Typed error classes
│ ├── telemetry.ts # Structured logging
│ └── providers/
│ ├── capabilities.ts # Provider capability definitions
│ └── adapter.ts # Provider adapters (wraps existing API clients)
├── lib/
│ ├── webllm/
│ │ ├── types.ts # WebLLM adapter interface + model defaults
│ │ ├── client.ts # WebLLM client (singleton)
│ │ ├── mock-adapter.ts # Mock adapter (always degrades gracefully)
│ │ ├── real-adapter.ts # Real WebLLM adapter
│ │ ├── worker.ts # WebLLM worker wiring
│ │ └── prompts.ts # Versioned prompts
│ ├── bestbuy.ts / ebay.ts / target.ts / walmart.ts / costco.ts / shopify.ts # Retailer API clients
│ ├── costco-cookie-cache.ts # In-memory Costco cookie cache
│ ├── costco-cookie-fetcher.ts # Costco cookie fetcher
│ └── retailer.ts # Retailer registry
├── tests/
│ └── search/
│ ├── schemas.test.ts # SearchPlan validation tests
│ ├── ranker.test.ts # Ranking engine tests
│ ├── entity-resolution.test.ts # Entity resolution tests
│ └── provider-selection.test.ts # Provider eligibility tests
├── types/ # Retailer-specific type definitions
├── scripts/
│ └── diagnose-ebay.mjs # eBay integration diagnostics
├── public/ # Static assets, PWA icons, sw.js (generated)
├── netlify.toml # Netlify config (headers, Node 20)
├── vercel.json # Vercel config (Costco cookie refresh cron)
├── next.config.ts # Next.js + PWA config, image domains
└── vitest.config.ts # Vitest (node env, @ → root alias)