Implements CLAUDE.md Build Order Phase 2 — same scraping/extraction
logic as the legacy app.py, reorganized into the three-layer structure
and upgraded per §10's explicit instructions:
- core/scraper.py — playwright-stealth replaces the manual
add_init_script stealth block; smart proxy (Webshare→Bright Data)
wired via core/proxy_service.py with block-detection + auto-escalation;
check_robots_txt() gate added per §30
- core/extractor.py — OpenRouter model cycling, merging the legacy
JSON-repair fallback chain with the spec's env-driven free/paid
model list and Sentry breadcrumb on paid-fallback use
- core/ai_fetch.py — AI web fetch fallback, ported unchanged
- industries/adventure_travel/{fields,prompt}.py ported unchanged;
schema.py added as the future V2 custom-fields attachment point
- config/client.py — gspread Sheet write-back, ported unchanged
- services/analysis_service.py now sources CACHE_STALENESS_HOURS from
core/scraper.py, its canonical location per §7
Dropped the legacy's commented-out Groq/Gemini alternates — OpenRouter
is the sole active provider and the dead code added nothing.
requirements.txt: relaxed playwright to >=1.55.0 and pinned
playwright-stealth==1.0.6 + setuptools<81 — Python 3.14 has no
prebuilt wheels for the originally-listed pins, and playwright-stealth
2.x replaced the stealth_async(page) API §10 documents verbatim with a
class-based one.
192 tests, 94% coverage. Also fixes a real bug surfaced by this phase:
several test helpers mutated core.extractor / industries.adventure_travel.prompt
via raw setattr instead of monkeypatch.setattr, silently corrupting
shared module state across tests once those modules became real.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Implements CLAUDE.md Build Order Phase 1 end to end:
- PocketBase migration covering all 15 MVP-scope collections
- Repository layer (11 repos) with real PocketBase-backed + in-memory
mock implementations for every collection
- Service layer (licence, trip-finder, battlecard, embedding, brief,
alert, analysis, onboarding, billing, auth) — Phase 2 scraping/
extraction dependencies are lazy-imported so this layer is fully
testable ahead of the app.py refactor
- Flask API (api.py) covering every MVP route from the spec, with
X-API-Key auth, Flask-Limiter rate limits, and structured JSON errors
- RQ job queue (jobs.py) with per-competitor failure isolation and
duration-based Gotify alerting
- 142 tests, 95% coverage, 100% on licence validation / seat
management / Stripe webhook dispatch per CLAUDE.md's testing floor
/sheet/push and /sheet/tabs intentionally return structured 501s until
the Apps Script Web App deployment exists (Phase 4).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
JasonFraser