The highest-impact fix: run_research_job() only ever read entry['id'] from
confirmed competitors, silently discarding the Firecrawl-matched or
manually-swapped URL — every scrape fell back to the competitor's bare
root domain regardless of what was confirmed. This explains most of the
"unpredictable/wrong data" behavior seen against the old app.py. Threaded
confirmed_url through jobs.py -> analysis_service.py so the actually
confirmed page is what gets scraped.
Other fixes bundled in this pass:
- Proxy layer: Webshare rotating-gateway country code lowercased (was
silently failing auth), Bright Data port corrected to 33335 (current
cert), dedicated-IP candidate rotation added before falling back to the
shared gateway/Bright Data escalation.
- Trip-intent matching: score_and_rank_urls() now hard-gates on
destination keyword and excludes bare root-domain candidates, instead
of letting duration/category signals alone push a wrong-country or
homepage URL over the confidence threshold.
- force_refresh cache-bypass toggle added (opt-in, collapsed "Advanced
options"), plus a "Skip this competitor" affordance for unmatched URLs.
- De-hardcoded "G Adventures" (the hackathon reference client) out of the
extraction prompt — now uses the real tenant's company name.
- RunHistory/Apps-Script "latest results" queries filtered to
triggered_by = "manual" so per-competitor content-hash bookkeeping rows
no longer drown out real batch runs.
- Dashboard fixes: stuck-state bug on revisiting the Analyse page,
AccountPage/BattlecardsPage restacked to single-column layout, Alerts
moved to a header bell dropdown, invisible Export/Run Analysis icons
(fill -> stroke), tour popup width mismatch, horizontal-scroll bug on
cards (missing min-width: 0 in flex layout).
- New PocketBase migrations for scrape_runs (content_hash fields,
tab_type, created_by_email) and products (departures, start_days).
335 backend tests passing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds CLAUDE.md (authoritative spec), the legacy app.py scraper and
approved dashboard mockup as reference material under z_starting_files/,
and the phased dev_plan.md for Phase 1+ implementation.
JasonFraser