fix(carousell_sg): add PROXY_PROVIDER switching and forward proxy env (BUY-63593)

[BuyWhere]

What

Adds a --proxy-provider {brightdata,scraperapi} flag to the Carousell SG scraper and routes the scheduler subprocess env to forward the relevant proxy credentials. Closes the actionable code half of BUY-63593.

Why

The Carousell SG daemon was producing 0 products with 407 Invalid Auth. Two root causes in this workspace:

1. No fallback provider. scrapers/carousell_sg.py had no path to use ScraperAPI, so when the BrightData customer hl_3ab737be is suspended, the scraper has no other option.
2. Scheduler strips proxy env. scripts/scraper_scheduler.py created a fresh env dict with only {SCRAPERAPI_KEY, BUYWHERE_API_KEY} for the subprocess. All BRIGHTDATA_* credentials were dropped, so proxy_config.proxy_url(Zone.RESIDENTIAL_PROXY1) built proxy URLs with empty passwords, causing 407 even when the upstream zone was healthy.

The BrightData suspension is still owned by BUY-56686 (human-action-router, Rich). ScraperAPI credits are still at 0. With this fix, switching to ScraperAPI is a one-flag flip once credits are topped up, and the BrightData path will also work as soon as the zone is reactivated.

Changes

• scrapers/carousell_sg.py
  • New PROXY_PROVIDER env default (brightdata) and --proxy-provider CLI arg with {brightdata, scraperapi} choices.
  • CarousellSGScraper.__init__ accepts proxy_provider; __aenter__ skips the httpx tunnel proxy when provider is scraperapi.
  • Split _fetch_page into _fetch_page_brightdata (3 retries, current behavior) and _fetch_page_scraperapi (single attempt that surfaces the upstream 401/403 body immediately so exhausted credits are not masked by retry delays).
  • run() summary records proxy_provider; __aenter__ logs it on startup.
• scripts/scraper_scheduler.py
  • New --proxy-provider CLI flag forwarded to the subprocess.
  • New _build_proxy_env() forwards SCRAPERAPI_KEY, BUYWHERE_API_KEY, PROXY_PROVIDER, BRIGHTDATA_RESIDENTIAL_*, BRIGHTDATA_DATACENTER_*, and the legacy BRIGHTDATA_* vars from the scheduler process.
  • Continuous and single-run paths use _build_proxy_env(). Only require SCRAPERAPI_KEY when provider is scraperapi.

Verification

• Syntax + import OK on Python 3 + httpx 0.25.2.
• Both providers instantiate cleanly; proxy provider: <name> is logged at __aenter__.
• Dry-run fetch on https://www.carousell.sg/categories/electronics-phones/:
  • brightdata: 3x 407 Auth failed then None (matches current behavior; expected until zone reactivated).
  • scraperapi with dummy key: 401 Unauthorized request, please make sure your API key is valid. in a single attempt.
• Diff stat: 89 insertions / 18 deletions across 2 files. No other files touched.

Operational notes

• Deploy via serviceInstanceDeployV2(commitSha="161ad33d8") on fix/BUY-63593-carousell-proxy-provider (not main).
• Restart the scheduler after deploy and relaunch with the existing crontab.
• A fresh data/carousell-sg/products_*.jsonl with nonzero rows will only appear once the upstream proxy accounts are healthy: BUY-56686 (BrightData reactivation) and/or ScraperAPI credit top-up.

Co-Authored-By: Paperclip noreply@paperclip.ing