Is this "50% off" real? This pipeline collects product prices from Brazilian marketplaces, keeps the full history, and judges every price against the product's own recent past. It separates real deals from fake discounts, where the store inflates the "was" price while the actual price never moved.
$ tracker collect synthetic && tracker transform && tracker deals
title source observed_date price usual_price pct_below_usual store_claimed_was
Lumen Coffee maker S (2026 model) synthetic 2026-10-31 723.16 1107.15 34.7 1105.12
Kappa Headphones Pro synthetic 2026-10-31 343.71 522.17 34.2 508.33
Nimbus Monitor 27in Max synthetic 2026-10-31 1070.61 1570.74 31.8 1576.76
...
flowchart LR
ML[Mercado Livre<br/>listing pages · Playwright] --> P[Source parsers<br/>→ Observation]
SH[Shopee Affiliate API<br/>GraphQL · SHA256-signed] --> P
SY[Synthetic generator<br/>with ground truth] --> P
P --> RAW[(raw.price_observations<br/>append-only, PK source+product+time)]
RAW --> STG[stg_price_observations]
STG --> DIM[dim_product_history<br/>SCD type 2]
STG --> DAY[fct_daily_prices<br/>trailing 30-day baseline]
DAY --> SIG[fct_deal_signals<br/>REAL_DEAL · FAKE_DISCOUNT · PRICE_HIKE]
SIG --> RPT[rpt_current_deals] --> AL[tracker deals<br/>stdout / webhook]
| Layer | Highlights |
|---|---|
| Sources | Each source returns the same Observation record. The Mercado Livre parser is a pure function over HTML, tested against a saved fixture (tracking redirects, lazy-loaded images, sponsored cards without price). The Shopee client signs requests as the official API requires, and the signature is unit-tested. |
| Landing | Append-only DuckDB table with primary key (source, product_id, observed_at) and INSERT OR IGNORE, so re-running a collection never duplicates rows. |
dim_product_history |
Type-2 SCD of title/seller, derived from history with window functions instead of dbt snapshot, so a backfill rebuilds the same dimension every time |
fct_daily_prices |
Daily min/max/close plus a trailing 30-day median and minimum that exclude the current day, so a sale can't hide inside its own baseline |
fct_deal_signals |
REAL_DEAL: price ≤ 85% of the usual price and at a 30-day low. FAKE_DISCOUNT: the store claims ≥ 30% off but the price is not below normal. PRICE_HIKE: often the set-up for a fake sale. Thresholds are dbt vars. |
The synthetic source simulates four behaviours (stable, honest sale, fake discount, and
bait-and-switch, where the price is raised before a "sale") and stores the true
behaviour in the raw payload. A dbt test (assert_signals_match_synthetic_ground_truth)
fails the build if any fake discount is ever flagged as a real deal, or if any honest
sale is missed.
| True behaviour | REAL_DEAL | FAKE_DISCOUNT | PRICE_HIKE | NORMAL |
|---|---|---|---|---|
| honest sale | 25 | 0 | 0 | 1137 |
| fake discount | 0 | 152 | 0 | 678 |
| bait-and-switch | 0 | 0 | 50 | 365 |
| stable | 0 | 0 | 0 | 2573 |
(product-days, 60 products × 90 days)
There are zero false positives: a bait-and-switch "sale" brings the price back to its usual level, so it is correctly not a deal.
pip install -e ".[dev]"
tracker collect synthetic # 60 products x 90 days, deterministic
tracker transform # dbt build: models + 13 tests
tracker deals # or: tracker deals --webhook https://hooks.slack.com/...
# real sources
pip install -e ".[scrape]" && playwright install chromium
tracker collect mercadolivre --url "https://lista.mercadolivre.com.br/air-fryer?" --pages 2
SHOPEE_APP_ID=... SHOPEE_SECRET=... tracker collect shopee --pages 2Scheduling is just tracker collect ... && tracker transform && tracker deals in cron
or any orchestrator. Scrape responsibly and within each site's terms of use.
- pytest: BRL price parsing, the HTML parser on a fixture, Shopee signing and parsing, deterministic generation, idempotent loads
- dbt: grain uniqueness, SCD2 validity (one current version per product,
valid_to > valid_from), price consistency, accepted signals, ground-truth evaluation - CI runs both on every push, plus the full collect → transform → deals flow
MIT