Skip to content

chore: improve dedicated scraping - #15

Merged
hazedav merged 1 commit into
masterfrom
dedicated-scrapers
Feb 1, 2026
Merged

hazedav merged 1 commit into
masterfrom
dedicated-scrapers

Conversation

@hazedav

@hazedav hazedav commented Feb 1, 2026

Copy link
Copy Markdown
Owner

Summary

This PR makes manufacturer filtering in FilamentProfilesScraper programmatic by introducing a manufacturer property on the base FilamentScraper class. Dedicated scrapers now self-identify, eliminating the need for hardcoded manufacturer lists.

Key Changes

  • Base class enhancement: Added optional manufacturer property to FilamentScraper that returns None by default
  • Dedicated scraper updates: PolymakerScraper, SunluScraper, and OvertureScraper now return their manufacturer name
  • Dynamic filtering: FilamentProfilesScraper uses get_dedicated_manufacturers() instead of a hardcoded tuple
  • New helper function: get_dedicated_manufacturers() in __init__.py introspects all scraper classes to build the exclusion set

Benefits

  • Extensibility: Adding a new dedicated scraper (e.g., HatchboxScraper with manufacturer = "Hatchbox") automatically filters that manufacturer from the aggregate scraper
  • Single source of truth: SCRAPER_CLASSES registry eliminates duplicate maintenance
  • Type safety: The manufacturer property is properly typed as str | None

Files Changed

File Change
filament_sites/base.py Added manufacturer property with docstring
filament_sites/__init__.py Added SCRAPER_CLASSES registry and get_dedicated_manufacturers()
filament_sites/filamentprofiles.py Replaced hardcoded tuple with dynamic lookup
filament_sites/polymaker.py Added manufacturer property returning "Polymaker"
filament_sites/sunlu.py Added manufacturer property returning "SUNLU"
filament_sites/overture.py Added manufacturer property returning "Overture"
filament_sites/test_filamentprofiles.py Added TestDynamicManufacturerFiltering test class

Test Plan

  • pytest teamtone/fetch/filament_sites/ passes (115 tests)
  • New tests verify get_dedicated_manufacturers() returns lowercase set
  • New tests confirm known scrapers (polymaker, sunlu, overture) are in the set
  • New tests ensure aggregate scraper doesn't pollute the set

🤖 Generated with Claude Code

@hazedav
hazedav merged commit e185445 into master Feb 1, 2026
1 check passed
@hazedav
hazedav deleted the dedicated-scrapers branch February 1, 2026 22:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant