Phase 5 — Automation:
- Daily scrape cron, daily digest email, and Monday brief n8n workflows
- price_history_repository.py + price diffing/is_new detection wired
into analysis_service's refresh path
- Numeric unread-count badge on the topbar bell (was a plain dot)
Change detection architecture correction:
- CLAUDE.md's Firecrawl Monitor design doesn't match the real
self-hosted product: no /v1/monitor in v1, the real /v2/monitor is
cron-scheduled and credit-metered (Supabase-gated), and self-hosted
Firecrawl is a 3-service stack (api/worker/playwright-service), not
the single container originally spec'd
- Replaced with content-hash comparison computed during the scrape
itself (core/scraper.py's compute_content_hash/normalise_html,
analysis_service's detect_change()) — no external dependency, and
the refresh path now skips LLM extraction entirely when a scrape
finds no content diff
- Fixed a real bug in the first pass of this migration: change_detected
was set and then immediately reset within the same function call,
making it unobservable to the CompetitorsPage badge, the Apps Script
sidebar, and the n8n cron filter. It now stays visible until the
next scrape confirms nothing further changed.
- Added a composite index on scrape_runs(competitor_id, content_hash,
started_at) backing the per-competitor recency lookup
- Docker Compose's firecrawl service replaced with the real
firecrawl-api/firecrawl-worker/firecrawl-playwright stack, gated
behind an opt-in profile
- Added CORS (absent from the original spec entirely) so the
browser-based dashboard can reach the API cross-origin
Phase 6 — Semantic trip matching:
- client_trips_repository.py — this collection existed in schema but
nothing ever populated it; comparable-trip matching could never
produce a real match without it
- embedding_service.save_client_trip() upserts a client_trips record
from every /research submission's trip-intent description, with
stale-embedding invalidation when destination/duration change
- /internal/embed-products now embeds client_trips as well as
competitor products; /internal/match-comparable reads both sides
from PocketBase instead of expecting client_trips in the request
body, so it's callable unattended by n8n
- comparable_matches upserts on (client_product, competitor_product)
instead of always creating — first_matched/last_matched only make
sense if repeat weekly matches update in place, and a PM's dismissed
match now survives re-matching
- embedding_and_matching_cron.json — Sunday night n8n workflow
249 tests passing, 96% coverage.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Implements CLAUDE.md Build Order Phase 2 — same scraping/extraction
logic as the legacy app.py, reorganized into the three-layer structure
and upgraded per §10's explicit instructions:
- core/scraper.py — playwright-stealth replaces the manual
add_init_script stealth block; smart proxy (Webshare→Bright Data)
wired via core/proxy_service.py with block-detection + auto-escalation;
check_robots_txt() gate added per §30
- core/extractor.py — OpenRouter model cycling, merging the legacy
JSON-repair fallback chain with the spec's env-driven free/paid
model list and Sentry breadcrumb on paid-fallback use
- core/ai_fetch.py — AI web fetch fallback, ported unchanged
- industries/adventure_travel/{fields,prompt}.py ported unchanged;
schema.py added as the future V2 custom-fields attachment point
- config/client.py — gspread Sheet write-back, ported unchanged
- services/analysis_service.py now sources CACHE_STALENESS_HOURS from
core/scraper.py, its canonical location per §7
Dropped the legacy's commented-out Groq/Gemini alternates — OpenRouter
is the sole active provider and the dead code added nothing.
requirements.txt: relaxed playwright to >=1.55.0 and pinned
playwright-stealth==1.0.6 + setuptools<81 — Python 3.14 has no
prebuilt wheels for the originally-listed pins, and playwright-stealth
2.x replaced the stealth_async(page) API §10 documents verbatim with a
class-based one.
192 tests, 94% coverage. Also fixes a real bug surfaced by this phase:
several test helpers mutated core.extractor / industries.adventure_travel.prompt
via raw setattr instead of monkeypatch.setattr, silently corrupting
shared module state across tests once those modules became real.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Implements CLAUDE.md Build Order Phase 1 end to end:
- PocketBase migration covering all 15 MVP-scope collections
- Repository layer (11 repos) with real PocketBase-backed + in-memory
mock implementations for every collection
- Service layer (licence, trip-finder, battlecard, embedding, brief,
alert, analysis, onboarding, billing, auth) — Phase 2 scraping/
extraction dependencies are lazy-imported so this layer is fully
testable ahead of the app.py refactor
- Flask API (api.py) covering every MVP route from the spec, with
X-API-Key auth, Flask-Limiter rate limits, and structured JSON errors
- RQ job queue (jobs.py) with per-competitor failure isolation and
duration-based Gotify alerting
- 142 tests, 95% coverage, 100% on licence validation / seat
management / Stripe webhook dispatch per CLAUDE.md's testing floor
/sheet/push and /sheet/tabs intentionally return structured 501s until
the Apps Script Web App deployment exists (Phase 4).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
JasonFraser