From cc81038ba58eb16b42d92c2056583a05180e7e1e Mon Sep 17 00:00:00 2001 From: JasonFraser Date: Tue, 30 Jun 2026 16:07:46 -0400 Subject: [PATCH] Initial commit: project spec, reference files, and dev plan Adds CLAUDE.md (authoritative spec), the legacy app.py scraper and approved dashboard mockup as reference material under z_starting_files/, and the phased dev_plan.md for Phase 1+ implementation. --- .gitignore | 33 + CLAUDE.md | 5827 +++++++++++++++++++ z_starting_files/app.py | 764 +++ z_starting_files/bettersight-dashboard.html | 1148 ++++ z_starting_files/dev_plan.md | 580 ++ 5 files changed, 8352 insertions(+) create mode 100644 .gitignore create mode 100644 CLAUDE.md create mode 100644 z_starting_files/app.py create mode 100644 z_starting_files/bettersight-dashboard.html create mode 100644 z_starting_files/dev_plan.md diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..3f0bc9e --- /dev/null +++ b/.gitignore @@ -0,0 +1,33 @@ +# Environment / secrets +.env +.env.* +!.env.example + +# Python +__pycache__/ +*.py[cod] +*.egg-info/ +.pytest_cache/ +.coverage +htmlcov/ +venv/ +.venv/ +*.log + +# Node / frontend +node_modules/ +dist/ +.vite/ +npm-debug.log* + +# PocketBase data +pb_data/ +pb_migrations/*.local.js + +# OS / editor +.DS_Store +.vscode/ +.idea/ + +# Google service account credentials +*service-account*.json diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..fcd9c5d --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,5827 @@ +# CLAUDE.md — Bettersight Competitive Intelligence SaaS + +> This file is the authoritative specification for Claude Code building Bettersight. +> Read every section before writing any code. Never skip a section. +> When in doubt, re-read this file. + +--- + +## 1. Project Overview + +Bettersight is a vertical competitive intelligence SaaS built specifically for adventure +travel operators. It automates competitor price tracking, trip discovery, and market +analysis — replacing hours of manual research with automated intelligence delivered +to tools PMs already use. + +**Tagline:** Better sight, better moves. +**Secondary tagline:** Better sight, better decisions. +**Domain:** bettersight.io +**Brand position:** The only competitive intelligence platform that understands +adventure travel — not just that a page changed, but what trip was added, what +price moved, and what it means for your catalogue. + +**The anchor feature** is an on-demand competitor analysis Google Sheet add-on that +runs structured AI extraction against competitor trip pages and writes results directly +into the client's spreadsheet. Everything else supports and extends this core. + +**Primary reference client:** G Adventures (hackathon origin) +**Target market:** Adventure travel operators — product managers, revenue managers, +strategy leads + +--- + +## 2. Business Context + +### Pricing Tiers + +| Tier | Price | Seats | Trial | +|---|---|---|---| +| Analyse | $349/month | 5 seats | 14 days — no credit card required | +| Discover | $499/month | 8 seats | 21 days — no credit card required | +| Intelligence | $999/month | 10 seats | 30 days — no credit card required | +| Enterprise | $1,499/month | Unlimited | No trial — custom demo and pilot | + +Additional seats: $45/seat/month on any tier. +Enterprise tier is not publicly listed — available on request only. +The $199 Monitor tier has been dropped entirely. + +### Tier Feature Summary + +**Analyse — $349/month (MVP launch tier)** +- Dashboard as the primary interface — analysis runs, results viewed, push to Sheet +- Trip-intent matching — PM enters destination + duration + style, system finds + the right competitor trip URL automatically via Firecrawl /map (on demand) +- URL confirmation screen before scrape runs — PM can swap any auto-matched URL +- On-demand competitor analysis with side-by-side dashboard results view +- Push to Google Sheet — one button exports results to PM's chosen workbook tab +- Auto-generated battlecards updated after each scrape +- Comparable trip discovery (semantic similarity matching) +- In-dashboard notifications — price changes, new products, alerts in activity feed +- Weekly Monday HTML email brief via Resend +- Fast path (PocketBase cache) / refresh path (live scrape) on-demand analysis +- Firecrawl /monitor for real-time change detection +- Firecrawl /map on demand — for trip-intent URL matching per analysis job only + +**Discover — $499/month** +- Everything in Analyse +- Firecrawl /map scheduled — weekly full catalogue crawl per competitor + → Proactive surveillance, not reactive search + → Finds new trips the PM never knew to look for + → New URLs auto-queued for scraping Sunday night without PM involvement +- New trip detection — flags trips added since last crawl +- Catalogue completeness — how many trips per competitor per destination +- Destination gap analysis — destinations competitors cover that you don't +- Sunday night discovery workflow (n8n) +- Monday brief includes all auto-discovered new products +- External notification integrations (PM connects one or more): + → Slack webhook — real-time alerts into any channel + → Microsoft Teams webhook — for Microsoft 365 organisations + → WhatsApp via Twilio — alerts to PM's phone number + +**Intelligence — $999/month** +- Everything in Discover +- Full OSINT signal pipeline (15+ sources — see Section 23) +- Competitor Trajectory Score (weekly) +- Wayback Machine positioning history tracking +- Network mapping (leadership, board, investor connections) +- Monthly narrative intelligence report with predicted moves and confidence ratings +- Early warning alerts on significant signal clusters + +**Enterprise — $1,499/month (unlisted)** +- Everything in Intelligence +- Dedicated onboarding and account management +- Custom signal sources on request +- Quarterly strategy briefing call +- SLA on scrape freshness and alert delivery +- Custom PDF branded reports + +### Tier Pitch Distinction + +**Analyse** — "Tell me about this specific trip across these competitors" +PM drives every research job. They describe their product, system finds +the matching competitor trips, PM confirms and runs. Reactive intelligence. + +**Discover** — "Tell me everything that changed this week that I didn't know to ask about" +System runs autonomously. PM wakes Monday to a brief that includes trips +they never knew existed. Catches unknown unknowns. Proactive intelligence. + +**Intelligence** — "Tell me what my competitors are about to do" +System aggregates signals across 15+ public sources and forecasts competitor +moves 3-6 months ahead. Predictive intelligence. + +### MVP Scope (build this first) + +Single tier at $349/month (Analyse). Five features only: + +1. Dashboard — primary interface for running analysis and viewing results +2. Trip-intent matching — Firecrawl /map on demand finds competitor trip URLs +3. On-demand competitor analysis with dashboard results view + push to Sheet +4. Auto-generated battlecards +5. In-dashboard notifications (activity feed) + Monday HTML email brief via Resend + +Comparable trip discovery (semantic similarity) is a Phase 2 addition +within the Analyse tier — build after core analysis flow is working. + +Do not build Discover, Intelligence, or Enterprise tier features until +Analyse tier is live with paying clients. + +--- + +## 3. Tech Stack — Authoritative + +**Never deviate from this stack without explicit instruction.** + +| Layer | Tool | Notes | +|---|---|---| +| Infrastructure | Hetzner VPS | Existing | +| Orchestration | Dokploy | Existing | +| Reverse proxy | Pangolin | Existing | +| Networking | Docker Compose | pangolin_default network | +| Backend API | Flask | Python only. No FastAPI. | +| Scraping engine | app.py (Flask) | Existing — do not rewrite | +| Job queue | RQ (Redis Queue) | Async job processing | +| Queue broker | Redis | Shared with Firecrawl | +| Monitoring | Firecrawl self-hosted | /monitor — real-time change detection (all tiers) | +| Trip-intent matching | Firecrawl self-hosted | /map on demand — finds competitor trip URLs (Analyse+) | +| Discovery | Firecrawl self-hosted | /map scheduled — weekly catalogue crawl (Discover+ only) | +| Change detection | Firecrawl Monitor + PocketBase flags | Event-driven — fires only on detected change | +| Proxy — primary | Webshare | ~$3/month, free tier available, used by default | +| Proxy — fallback | Bright Data | Pay-as-you-go, auto-escalated per domain when Webshare blocked | +| AI extraction | OpenRouter | Free tier model cycling with claude-haiku-4-5 paid fallback | +| Embeddings | text-embedding-3-small | Via OpenRouter | +| Database | PocketBase | Multi-tenant + auth | +| DB replication | Litestream | Real-time SQLite WAL streaming to Hetzner S3 — near-zero RPO | +| Auth | PocketBase native | Email/password (default) + Google OAuth | +| API rate limiting | Flask-Limiter | Protects against queue abuse and runaway scripts | +| Workflow automation | n8n | Self-hosted | +| Frontend | Vite + Vue 3 | Minimal dashboard only — no Nuxt runtime | +| UI component library | Nuxt UI v4 (free, MIT) | Same library as Ledra/AccountFlow | +| Base template | nuxt-ui-templates/dashboard-vue | Clone as starting point | +| Routing | Vue Router v4 | Client-side SPA routing | +| State management | Pinia | Lightweight store | +| Date handling | date-fns | Formatting and computation | +| Validation | Zod | Schema validation | +| Sheet add-on | Google Apps Script | Export integration only — push results from dashboard to Sheet | +| User notifications | In-dashboard activity feed | MVP — price changes, new products, alerts | +| User notifications (future) | Slack / Teams / WhatsApp | Discover tier — webhook integrations | +| Owner monitoring alerts | Gotify + iGotify | Self-hosted — owner phone only, never shown to clients | +| Weekly brief | Resend HTML email | Replaces WeasyPrint — no PDF, no Docker dependency | +| Payments | Stripe | Subscription management + customer portal | +| Support tickets | Freescout (self-hosted) | Email-to-ticket — support@bettersight.io → Freescout | + +--- + +## 4. Architecture + +### Service Map + +``` +Bettersight Dashboard — PRIMARY INTERFACE (app.bettersight.io) + → PM enters: destination + duration + travel style + selected competitors + → Flask POST /research/find-urls + → Firecrawl /map per competitor (on demand, <5s each) + → URL scoring returns best match per competitor + → Confirmation screen — PM confirms or swaps any URL + → PM confirms → Flask POST /research → RQ job queued → job_id returned + → Dashboard polls /research/status/{job_id} every 3 seconds + → On-demand check flow per competitor: + Path 1 FAST — last_scraped < 24hrs AND change_detected = false + → PocketBase structured data → LLM analysis narrative (~seconds) + Path 2 REFRESH — last_scraped > 24hrs OR change_detected = true + → RQ job → app.py Playwright scrape → PocketBase → LLM (~2 mins) + → Results displayed in dashboard side-by-side comparison view + → PM clicks "Push to Sheet" → Apps Script writes to selected workbook tab + +Apps Script (standalone — export integration only) + → getWorkbookTabs() — returns PM's workbook tab list for dashboard dropdown + → pushToSheet(results, tabName) — writes dashboard results to Sheet on demand + → validateLicence() — confirms PM is authorised + → Three functions only. No analysis UI in the script. + +Firecrawl Monitor (event-driven, runs continuously) + → Watches all active competitor URLs + → Detects content change (JS-rendered pages, React/Vue SPAs supported) + → Fires webhook → Flask /internal/monitor-webhook + → Sets competitor.change_detected = true in PocketBase + +n8n (scheduled workflows) + → Daily cron (6am) — scrapes changed/stale competitors + → Resets change_detected = false after each scrape + → PocketBase webhook on price_history insert → creates alert record in PocketBase + → PocketBase webhook on products insert (is_new=true) → creates alert record in PocketBase + → n8n reads new alert records → sends Resend email notification (daily digest) + → Monday 6am → brief_service.py → Resend HTML email + → [Discover tier] Sunday night → Firecrawl /map scheduled → full catalogue crawl + +Vue 3 dashboard auth + account + → PocketBase auth (Google OAuth or email/password) + → Stripe (payment, subscription status) + → On competitor add → Flask registers with Firecrawl Monitor automatically +``` + +### Change Detection — Firecrawl Monitor + +Firecrawl Monitor is the change detection layer. It replaces both +changedetection.io (previously removed) and the plain HTTP hash approach +(rejected due to JS-heavy page limitations). + +**Why Firecrawl Monitor over alternatives:** +- Renders JS fully before hashing — works on React/Vue SPAs +- Event-driven — fires only on change, not on a polling schedule +- Lower server load than a 2-4hr Playwright cron +- Single service handles both monitoring (/monitor) and discovery (/map for Intelligence tier) +- Already in stack — no new infrastructure + +**PocketBase competitor record flags:** +``` +change_detected bool — set true by monitor webhook, reset after scrape +change_detected_at date — timestamp of last detected change +last_scraped date — timestamp of last completed scrape +``` + +**Monitor registration — on competitor add:** +```python +def register_competitor_monitor(competitor_id: str, url: str): + """ + Registers a competitor URL with Firecrawl Monitor on first add. + Stores the monitor ID in PocketBase for future deregistration. + + Data flow: + competitor url → Firecrawl POST /v1/monitor → + monitor_id returned → stored in competitors.firecrawl_monitor_id + """ + response = requests.post( + "http://firecrawl:3002/v1/monitor", + json={ + "url": url, + "webhook": "https://api.bettersight.io/internal/monitor-webhook", + "changeTypes": ["content"] + } + ) + monitor_id = response.json().get("id") + competitor_repository.update(competitor_id, { + "firecrawl_monitor_id": monitor_id + }) +``` + +**Monitor webhook handler:** +```python +@app.route('/internal/monitor-webhook', methods=['POST']) +def monitor_webhook(): + """ + Receives Firecrawl Monitor webhook on competitor page change. + Sets change_detected flag on the competitor record. + + Data flow: + Firecrawl webhook payload → extract competitor URL → + PocketBase competitor lookup by URL → + change_detected = true, change_detected_at = now + """ + data = request.json + url = data.get('url') + competitor = competitor_repository.get_by_url(url) + if competitor: + competitor_repository.update(competitor['id'], { + 'change_detected': True, + 'change_detected_at': datetime.now().isoformat() + }) + return jsonify({'status': 'ok'}) +``` + +### Docker Network + +All services on `pangolin_default` network. +Internal service calls use container names, never localhost or host IPs. + +``` +flask-api → http://flask-api:5000 +bettersight-worker → connects to redis://bettersight-redis:6379 +bettersight-redis → redis://bettersight-redis:6379 +firecrawl → http://firecrawl:3002 +pocketbase → http://pocketbase:8090 +``` + +### Compose Services + +```yaml +services: + flask-api: + build: ./backend + container_name: flask-api + command: gunicorn app:app -w 4 -b 0.0.0.0:5000 + env_file: .env + networks: + - pangolin_default + restart: unless-stopped + depends_on: + - bettersight-redis + + bettersight-worker: + build: ./backend + container_name: bettersight-worker + command: rq worker high normal low -u redis://bettersight-redis:6379 + env_file: .env + networks: + - pangolin_default + restart: unless-stopped + deploy: + replicas: 3 + depends_on: + - bettersight-redis + + bettersight-redis: + image: redis:alpine + container_name: bettersight-redis + networks: + - pangolin_default + restart: unless-stopped + + firecrawl: + image: ghcr.io/mendableai/firecrawl:latest + container_name: firecrawl + environment: + - REDIS_URL=redis://bettersight-redis:6379 + - USE_DB_AUTHENTICATION=false + - PORT=3002 + networks: + - pangolin_default + restart: unless-stopped + depends_on: + - bettersight-redis + + freescout: + image: freescout/freescout:latest + container_name: freescout + environment: + - APP_URL=https://support.bettersight.io + - MAIL_FROM=support@bettersight.io + networks: + - pangolin_default + restart: unless-stopped + # Pangolin routes support.bettersight.io → freescout:80 + + litestream: + image: litestream/litestream:latest + container_name: litestream + command: replicate + volumes: + - pocketbase_data:/pb_data # same volume as PocketBase + environment: + - LITESTREAM_ACCESS_KEY_ID=${HETZNER_S3_ACCESS_KEY} + - LITESTREAM_SECRET_ACCESS_KEY=${HETZNER_S3_SECRET_KEY} + configs: + - source: litestream_config + target: /etc/litestream.yml + networks: + - pangolin_default + restart: unless-stopped + depends_on: + - pocketbase + # litestream.yml config: + # dbs: + # - path: /pb_data/data.db + # replicas: + # - url: s3://bettersight-backups/litestream + # endpoint: https://fsn1.your-objectstorage.com + +networks: + pangolin_default: + external: true + +volumes: + pocketbase_data: +``` + +**Litestream recovery procedure:** +```bash +# Restore PocketBase database from Litestream replica +docker run --rm \ + -e LITESTREAM_ACCESS_KEY_ID=${HETZNER_S3_ACCESS_KEY} \ + -e LITESTREAM_SECRET_ACCESS_KEY=${HETZNER_S3_SECRET_KEY} \ + -v ./pb_data:/pb_data \ + litestream/litestream restore \ + -o /pb_data/data.db \ + s3://bettersight-backups/litestream +``` + +Recovery point objective (RPO): seconds — not 24 hours. +Recovery time objective (RTO): under 5 minutes from backup restore to running. + +--- + +## 5. Project Structure + +``` +bettersight/ +├── backend/ +│ ├── core/ +│ │ ├── scraper.py # Playwright stealth browser — never touches industry logic +│ │ ├── extractor.py # OpenRouter model cycling, JSON parsing, fallback chain +│ │ ├── ai_fetch.py # AI web fetch fallback +│ │ └── proxy_service.py # Proxy provider — swap here to change from Bright Data +│ ├── industries/ +│ │ └── adventure_travel/ +│ │ ├── fields.py # STANDARD_FIELDS, NGS_FIELDS column maps +│ │ ├── prompt.py # build_prompt(), inject_page_content() +│ │ └── schema.py # JSON extraction schema for this vertical +│ ├── config/ +│ │ └── client.py # SHEET_ID, tab names, column assignments +│ ├── services/ +│ │ ├── licence_service.py # Domain + seat validation (no sheet_id) +│ │ ├── trip_finder_service.py # find_competitor_trip_url(), score_and_rank_urls() +│ │ ├── battlecard_service.py +│ │ ├── embedding_service.py +│ │ ├── brief_service.py # HTML email brief generation +│ │ └── alert_service.py +│ ├── repositories/ +│ │ ├── tenant_repository.py +│ │ ├── product_repository.py +│ │ ├── seat_repository.py +│ │ ├── scrape_repository.py +│ │ └── proxy_repository.py +│ ├── templates/ +│ │ └── emails/ +│ │ ├── welcome.html +│ │ ├── trial_reminder.html +│ │ ├── brief.html +│ │ ├── price_alert.html +│ │ └── new_product.html +│ ├── api.py # Flask routes +│ ├── jobs.py # RQ job functions +│ ├── app.py # Entry point — imports and wires everything +│ ├── requirements.txt +│ └── Dockerfile +├── frontend/ # Vite + Vue 3 + Nuxt UI v4 dashboard +│ ├── src/ +│ │ ├── pages/ +│ │ │ ├── LoginPage.vue # Auth — email/password + Google OAuth + forgot password +│ │ │ ├── ResetPasswordPage.vue # /reset-password?token=xxx — PocketBase token confirm +│ │ │ ├── AccountPage.vue # Subscription, seats, upgrade prompt, template download +│ │ │ ├── CompetitorsPage.vue # Add/edit competitors with status badges + CSV upload +│ │ │ ├── AnalysePage.vue # CORE PAGE — full analysis workflow +│ │ │ ├── ActivityPage.vue # Market movement feed + alerts +│ │ │ └── BattlecardsPage.vue # Battlecard view per competitor +│ │ ├── components/ +│ │ │ ├── DotGrid.vue # Reusable dot grid SVG — signature element +│ │ │ ├── analyse/ +│ │ │ │ ├── TripIntentForm.vue # Destination + duration + style + competitor checkboxes +│ │ │ │ ├── UrlConfirmation.vue # Confirm or swap matched URLs before run +│ │ │ │ ├── JobProgress.vue # Per-competitor real-time status during job +│ │ │ │ ├── ResultsTable.vue # Side-by-side competitor comparison +│ │ │ │ ├── ConfidenceBar.vue # Per-competitor confidence indicator +│ │ │ │ ├── PushToSheet.vue # Tab selector + push button +│ │ │ │ └── RunHistory.vue # Last 5 runs with restore option +│ │ │ ├── activity/ +│ │ │ │ ├── FeedItem.vue # Single activity feed row with dot grid +│ │ │ │ └── AlertCard.vue # Alert card with dot grid +│ │ │ ├── shared/ +│ │ │ │ ├── StatCard.vue # Stat card with accent bar + dot grid +│ │ │ │ ├── SeatManager.vue # Add/remove seats table +│ │ │ │ ├── CompetitorForm.vue # Add/edit competitor modal form +│ │ │ │ ├── BattlecardCard.vue # Single battlecard display +│ │ │ │ ├── SubscriptionCard.vue +│ │ │ │ ├── CsvUpload.vue # File picker + PapaParse + row validation +│ │ │ │ └── ImportPreviewModal.vue # Valid/invalid preview before bulk submit +│ │ ├── composables/ +│ │ │ ├── useAuth.js +│ │ │ ├── useTenant.js +│ │ │ ├── useAnalysis.js # Analysis job lifecycle management +│ │ │ ├── useCompetitors.js +│ │ │ └── useOnboardingTour.js # Shepherd.js guided tour — 5 steps +│ │ ├── assets/ +│ │ │ └── tour.css # Custom Shepherd.js styles matching design system +│ │ ├── services/ +│ │ │ └── api_service.js # All Flask API calls +│ │ ├── repositories/ +│ │ │ ├── tenant_repository.js +│ │ │ ├── seat_repository.js +│ │ │ ├── competitor_repository.js +│ │ │ └── analysis_repository.js # Research jobs, history, results +│ │ ├── stores/ +│ │ │ ├── auth_store.js # Pinia — JWT, email, tier +│ │ │ └── analysis_store.js # Pinia — active job, results, history +│ │ ├── router/ +│ │ │ └── index.js +│ │ └── main.js +│ ├── package.json +│ ├── vite.config.js +│ └── index.html +├── appscript/ # Export integration only — 3 files +│ ├── Code.gs # onOpen(), install entry point +│ ├── Validation.gs # validateLicence() — email only +│ ├── Export.gs # pushToSheet(), getWorkbookTabs(), detectColumns() +│ └── Sidebar.html # Minimal UI — tab selector + push button only +├── n8n/ +│ └── workflows/ # n8n workflow JSON exports +├── docker-compose.yml +├── .env.example +└── CLAUDE.md +``` + +--- + +## 5b. Frontend Setup — Nuxt UI v4 + +### Bootstrap + +```bash +# Clone the dashboard-vue base template +git clone https://github.com/nuxt-ui-templates/dashboard-vue frontend +cd frontend +npm install + +# Additional dependencies +npm install pinia date-fns zod axios +``` + +### Key dependencies (package.json) + +```json +{ + "dependencies": { + "@nuxt/ui": "^4.0.0", + "vue": "^3.4.0", + "vue-router": "^4.0.0", + "pinia": "^2.0.0", + "date-fns": "^3.0.0", + "zod": "^3.0.0", + "axios": "^1.6.0" + }, + "devDependencies": { + "vite": "^5.0.0", + "@vitejs/plugin-vue": "^5.0.0" + } +} +``` + +### Nuxt UI v4 component usage + +Use Nuxt UI components throughout — never build custom UI primitives. +Key components for this dashboard: + +``` +UButton — all buttons +UInput — all text inputs +UCard — content containers +UTable — seat management, competitor list +UBadge — subscription status, tier labels +UModal — add competitor, add seat dialogs +UForm — all forms with Zod validation +UAlert — error and success messages +UAvatar — user display +UDropdown — action menus +UDivider — section separators +UContainer — page layout wrapper +USkeleton — loading states +UNotification — toast notifications +``` + +### Design tokens — align with Ledra/AccountFlow + +```javascript +// app.config.ts or equivalent +export default { + ui: { + primary: 'indigo', + gray: 'cool' + } +} +``` + +### Separation of concerns — frontend + +``` +Pages (src/pages/) + → orchestrate composables and repositories + → NO direct API calls + → NO business logic + +Composables (src/composables/) + → reusable stateful logic + → calls repositories + → NO direct API calls + +Repositories (src/repositories/) + → all API calls via api_service.js + → returns plain data objects + → NO component or store references + +Stores (src/stores/) + → global state only (auth, tenant) + → NO API calls directly + → calls repositories for data + +Services (src/services/) + → api_service.js is the single Axios instance + → all endpoints configured here + → NO business logic +``` + +### Six pages — full dashboard + +**LoginPage.vue** — email/password default + Google OAuth option + +**AccountPage.vue** +- Subscription status card (tier, renewal date, seat count, trial countdown) +- **Upgrade prompt card** — shown below subscription for Analyse and Discover tier clients: + ``` + ┌────────────────────────────────────────────────┐ + │ 🚀 Upgrade to Discover │ + │ │ + │ + Weekly catalogue auto-discovery │ + │ + Spot new competitor trips automatically │ + │ + Slack, Teams and WhatsApp alerts │ + │ │ + │ $499/month · 3 more seats · 21-day trial │ + │ │ + │ [Upgrade now →] │ + └────────────────────────────────────────────────┘ + ``` + → [Upgrade now] creates a Stripe checkout session for the next tier + → On successful upgrade: PocketBase tier updated via Stripe webhook + → Upgrade card hidden for Intelligence and Enterprise clients +- Download Sheet template button +- Seat management table (add/remove seats) +- Stripe customer portal link (manage billing, cancel) +- Onboarding checklist (tracked in PocketBase) +- "↺ Take the guided tour again" link + +**CompetitorsPage.vue** +- Table of active competitors with status badges (✓ Current, ⚠ Changed, ○ Stale) +- Primary market flag per competitor +- Add competitor modal — name, website, catalogue URL, primary market (required) +- On add → auto-registers with Firecrawl Monitor +- Edit/deactivate actions +- [Download CSV template] button — pre-formatted with correct headers +- [Upload CSV] button — bulk import with preview and validation + +**AnalysePage.vue — THE CORE PAGE** +- TripIntentForm — destination, duration, travel style, competitor checkboxes +- [Find & Analyse] → POST /research/find-urls → UrlConfirmation screen +- UrlConfirmation — matched URL per competitor, swap any URL, manual entry for no-match +- [Confirm & Run] → POST /research → job_id → polling begins +- JobProgress — per-competitor real-time status (queued / scraping / done / failed) +- ResultsTable — side-by-side competitor comparison on completion + Multi-currency display rule: + → Show competitor's primary market currency as hero price + (GB competitor = GBP hero, US competitor = USD hero) + → Always show USD alongside in secondary muted text + → Format: "£2,190 $2,780 USD" + → Never currency-convert — use extracted values only + → If primary market currency not extracted, fall back to USD only +- ConfidenceBar — per-competitor AI confidence with low-confidence warning +- RunHistory — last 5 runs, restore any run to current results view +- PushToSheet — tab selector dropdown + push button (calls /sheet/push) + +**ActivityPage.vue** +- Stat cards (price changes, drops, new products, competitors tracked) +- Market movement feed — all competitor events this week with dot grid rows +- Alerts panel — unread alerts with badge count in sidebar nav +- Price position progress bars +- Mark as read / dismiss per alert +- Bell icon in topbar shows unread count + +**BattlecardsPage.vue** +- Battlecard per competitor — positioning summary, strengths, weaknesses, pricing observation +- Last updated timestamp +- Topic pills (destinations, warnings, signals) +- Comparable trips section (semantic matches) + + + +### tenants +``` +id auto +name text +email text (admin email from signup) +domain text (e.g. gadventures.com — used for seat validation) +tier select [analyse, discover, intelligence, enterprise] +status select [trial, active, inactive] +trial_ends_at date (set on signup based on tier trial period) +max_seats integer (set from TIER_SEATS on signup) +timezone text (IANA timezone e.g. Europe/London — captured on signup) +stripe_customer_id text +stripe_subscription_id text +onboarding_checklist json (tracks completion of onboarding steps) +data_region text (default: 'global' — for future GDPR regional routing) +created autodate +``` +Note: `sheet_id` binding removed — standalone Apps Script model, +validation is email-only. `monitor` tier removed — dropped entirely. + +**Timezone capture on signup:** +```javascript +// Captured automatically from browser on account creation +// Never ask the PM to select a timezone manually +const timezone = Intl.DateTimeFormat().resolvedOptions().timeZone +// e.g. "Europe/London", "America/New_York", "Australia/Sydney" +// Stored in tenants.timezone on first signup POST +``` + +### tenant_seats +``` +id auto +tenant_id relation → tenants +email text +active bool +last_accessed date +added_by text +created autodate +``` + +### competitors +``` +id auto +tenant_id relation → tenants +name text +website url +catalogue_url url +primary_market text (ISO country code e.g. GB, US, AU — required on add) +firecrawl_monitor_id text (returned by Firecrawl on monitor registration) +change_detected bool (set true by monitor webhook, reset after scrape) +change_detected_at date (timestamp of last detected change) +active bool +last_scraped date +created autodate +``` + +### products +``` +id auto +tenant_id relation → tenants +competitor_id relation → competitors +trip_name text +url url +destination text +duration_days integer +majority_price_usd number +price_low_usd number +price_high_usd number +aud_price number (nullable) +cad_price number (nullable) +eur_price number (nullable) +gbp_price number (nullable) +date_used text +seasonality text +group_size integer +meals integer +service_level text +activities text +start_location text +end_location text +target_audience text +hotels text +comments text +relevancy integer +trip_code text +is_new bool +first_seen autodate +last_seen date +is_active bool +embedding json (vector array for similarity matching) +``` + +### products_ngs_meta +``` +id auto +product_id relation → products +exclusive_access text +group_leader text +sustainability text +``` + +### price_history +``` +id auto +tenant_id relation → tenants +product_id relation → products +majority_price_usd number +price_low_usd number +price_high_usd number +aud_price number (nullable) +cad_price number (nullable) +eur_price number (nullable) +gbp_price number (nullable) +date_used text +change_field text (which price field changed) +change_amount number +change_percent number +scraped_at autodate +``` + +### scrape_runs +``` +id auto (also used as RQ job_id) +tenant_id relation → tenants +competitor_id relation → competitors (nullable for bulk runs) +triggered_by select [cron, webhook, manual, sheet] +status select [pending, running, complete, failed] +competitors_total integer +competitors_done integer +results json +error_log text +started_at autodate +completed_at date +``` + +### alerts +``` +id auto +tenant_id relation → tenants +competitor_id relation → competitors +product_id relation → products (nullable) +alert_type select [price_change, new_product, product_removed, new_comparable] +message text +change_amount number (nullable) +change_percent number (nullable) +delivered_dashboard bool (always true — alert always appears in dashboard) +delivered_email bool (set true when included in daily digest email) +delivered_slack bool (Discover tier — set true when Slack webhook fires) +delivered_teams bool (Discover tier — set true when Teams webhook fires) +delivered_whatsapp bool (Discover tier — set true when WhatsApp fires) +read bool (PM marks as read in dashboard) +created autodate +``` + +### battlecards +``` +id auto +tenant_id relation → tenants +competitor_id relation → competitors +content text (AI generated battlecard text) +content_json json (structured: top_routes, avg_price, positioning, strengths, weaknesses) +generated_at autodate +``` + +### comparable_matches +``` +id auto +tenant_id relation → tenants +client_product text (client trip name — not a PocketBase relation) +competitor_product relation → products +similarity_score number +price_difference number +duration_difference integer +first_matched autodate +last_matched date +dismissed bool +``` + +### tenant_notifications +``` +id auto +tenant_id relation → tenants +slack_webhook_url text (nullable — Discover tier only) +teams_webhook_url text (nullable — Discover tier only) +whatsapp_number text (nullable — Discover tier only) +email_digest bool (default true — daily digest email) +email_digest_time text (default "18:00" — local time in tenant timezone) +created autodate +``` + +**Timezone-aware digest scheduling in n8n:** +``` +Daily digest n8n workflow: + → Query all active tenants with email_digest = true + → For each tenant: + Convert 18:00 tenant local time → UTC using tenants.timezone + If current UTC time matches → send digest for this tenant + → Each tenant receives their digest at 6pm their local time +``` + +n8n Code node timezone conversion: +```javascript +// n8n Code node — per tenant digest time calculation +const { DateTime } = require('luxon') +const localTime = '18:00' +const tenantTz = tenant.timezone || 'UTC' +const sendAt = DateTime.fromFormat(localTime, 'HH:mm', { zone: tenantTz }) + .toUTC() +// Compare sendAt.hour and sendAt.minute against current UTC time +``` + +### digest_logs +``` +id auto +tenant_id relation → tenants +digest_type select [weekly, monthly] +pdf_path text +sent_at autodate +recipient_email text +status select [sent, failed] +``` + +### proxy_overrides +``` +id auto +domain text (e.g. intrepidtravel.com) +provider select [webshare, brightdata] +blocked_count integer +last_blocked date +created autodate +``` + +### client_trips +``` +id auto +tenant_id relation → tenants +trip_name text +destination text +duration_days integer +price_usd number +activities text +embedding json +created autodate +``` + +--- + +## 7. On-Demand Analysis — Check Flow + +This is the core data flow for every on-demand analysis request from the Sheet. +It determines whether to serve from PocketBase (fast) or trigger a live scrape (slow). +The PM always gets fresh, relevant data. The system decides the path invisibly. + +``` +PM triggers analysis in Sheet sidebar + │ + ▼ +For each selected competitor: + │ + ├── Check PocketBase competitor record + │ last_scraped < 24hrs? AND change_detected = false? + │ │ + │ ├── YES → FAST PATH (~seconds) + │ │ Pull structured data from PocketBase products table + │ │ Pass JSON to LLM for analysis narrative only (no extraction) + │ │ Write results to Sheet + │ │ Show: ✅ Current (updated {time}) + │ │ + │ └── NO → REFRESH PATH (~2 mins) + │ last_scraped > 24hrs OR change_detected = true + │ Trigger RQ job → Playwright scrape → AI extraction + │ Update PocketBase (products, price_history) + │ Reset change_detected = false + │ Update last_scraped = now + │ Pass fresh data to LLM for analysis narrative + │ Write results to Sheet + │ Show: 🔄 Refreshed just now + │ + ▼ +Sheet sidebar status indicators: + ✅ Current (updated today 6:02am) — fast path served + ⚠️ Change detected (2hrs ago) — triggers refresh automatically + 🔄 Refreshing... — slow path in progress + ✓ Complete — results written to Sheet +``` + +### Fast path — LLM role + +In the fast path the LLM does NOT extract data — it analyses it. +PocketBase already has structured JSON. The LLM receives that JSON +and generates the analysis narrative, comments, and battlecard content only. + +```python +def analyse_from_cache(tenant_id: str, competitor_id: str, context: dict) -> dict: + """ + Fast path analysis — uses cached PocketBase data instead of live scrape. + LLM receives structured JSON and generates narrative analysis only. + + Data flow: + competitor_id → PocketBase products (latest) → + structured JSON + client trip context → + LLM prompt (analysis only, no extraction) → + narrative + comments + relevancy score returned + """ +``` + +### Staleness threshold + +24 hours is the default staleness threshold. +This is a configurable constant — adjust if client feedback indicates +they need fresher data or if server load requires a longer window: + +```python +CACHE_STALENESS_HOURS = 24 # in core/scraper.py — adjust here only +``` + +--- + +## 8. Trip-Intent Matching + +### Overview + +The PM never manually enters a competitor trip URL. They describe their +own product — destination, duration, travel style — and the system finds +the best matching trip page on each competitor site automatically via +Firecrawl /map with scoring. + +This feature is available in the **Analyse tier** (on demand, per job). +The **Discover tier** extends this with a scheduled weekly crawl. + +### Dashboard Flow + +``` +PM fills in the Analyse form: + Product name: Inca Trail Classic + Destination: Peru + Duration: 15 days + Travel style: Classic + Competitors: ☑ Intrepid ☑ Exodus ☑ Flash Pack ☑ Kimkim + +PM clicks [Find & Analyse] + → Flask POST /research/find-urls + → Firecrawl /map runs per competitor simultaneously (async) + → URL scoring selects best match per competitor + +Confirmation screen shown in dashboard: + Intrepid Travel → intrepidtravel.com/peru-classic-15-days [✓ Use] [↗ Swap] + Exodus Travels → exodustravels.com/machu-picchu-trek [✓ Use] [↗ Swap] + Flash Pack → flashpack.com/peru-adventure-15d [✓ Use] [↗ Swap] + Kimkim → No match found [Enter URL manually] + +PM confirms (or swaps any URL) + → Flask POST /research with confirmed URLs + → RQ job queued → analysis runs + → Results displayed in dashboard +``` + +### Flask Route + +```python +@app.route('/research/find-urls', methods=['POST']) +def find_urls(): + """ + Finds the best matching competitor trip URL for each selected + competitor given the PM's research intent (destination, duration, + travel style). Runs Firecrawl /map per competitor concurrently. + + Body: { + destination: str, + duration: int, + travel_style: str, + competitor_ids: list[str] + } + + Returns: { + matches: [ + { competitor_id, competitor_name, url, confidence, found } + ] + } + + Data flow: + competitor_ids → PocketBase (get root URLs per competitor) → + Firecrawl /map per competitor (concurrent, ThreadPoolExecutor) → + score_and_rank_urls() per competitor → + return best match per competitor with confidence score + """ +``` + +### URL Finding Service — `services/trip_finder_service.py` + +```python +def find_competitor_trip_url(root_url: str, destination: str, + duration: int, travel_style: str) -> dict: + """ + Uses Firecrawl /map with search terms to find the most likely + trip page URL on a competitor site matching the PM's research intent. + + Data flow: + root_url + search terms → Firecrawl /map → + filtered URL list → score_and_rank_urls() → + { url, confidence, found } returned + """ + response = requests.post( + "http://firecrawl:3002/v1/map", + json={ + "url": root_url, + "search": f"{destination} {duration} days {travel_style} tour" + }, + timeout=15 + ) + urls = response.json().get("links", []) + + if not urls: + return {"url": None, "confidence": 0, "found": False} + + return score_and_rank_urls(urls, destination, duration) + + +def score_and_rank_urls(urls: list, destination: str, + duration: int) -> dict: + """ + Scores candidate URLs by keyword match against destination + and duration. Returns highest scoring URL with confidence score. + + Scoring rules: + +1 per destination keyword found in URL + +2 if duration (as integer) appears in URL + +1 if travel category words appear (classic, trek, trail, etc.) + + Confidence = top_score / max_possible_score, capped at 1.0 + If confidence < 0.3, returns found=False — prompt manual entry. + """ + dest_keywords = destination.lower().split() + category_keywords = ['classic', 'trek', 'trail', 'safari', + 'adventure', 'explorer', 'journey'] + scores = [] + + for url in urls: + url_lower = url.lower() + score = 0 + score += sum(1 for kw in dest_keywords if kw in url_lower) + if str(duration) in url_lower: + score += 2 + score += sum(1 for kw in category_keywords if kw in url_lower) + scores.append((score, url)) + + scores.sort(reverse=True) + if not scores or scores[0][0] == 0: + return {"url": None, "confidence": 0, "found": False} + + top_score, top_url = scores[0] + max_possible = len(dest_keywords) + 2 + 1 + confidence = min(top_score / max_possible, 1.0) + + return { + "url": top_url, + "confidence": round(confidence, 2), + "found": confidence >= 0.3 + } +``` + +### Concurrent Execution + +Run /map calls for all competitors simultaneously — not sequentially: + +```python +from concurrent.futures import ThreadPoolExecutor, as_completed + +def find_all_competitor_urls(competitors: list, destination: str, + duration: int, travel_style: str) -> list: + """ + Runs trip URL finding for all competitors concurrently. + Max 5 concurrent Firecrawl /map calls — respects server limits. + + Data flow: + competitors list → ThreadPoolExecutor (max 5) → + find_competitor_trip_url() per competitor in parallel → + collect results → return matches list + """ + results = [] + with ThreadPoolExecutor(max_workers=5) as executor: + futures = { + executor.submit( + find_competitor_trip_url, + c['website'], destination, duration, travel_style + ): c for c in competitors + } + for future in as_completed(futures): + competitor = futures[future] + match = future.result() + results.append({ + "competitor_id": competitor['id'], + "competitor_name": competitor['name'], + **match + }) + return results +``` + +--- + +## 9. API Routes + +All routes require `X-API-Key` header matching `API_SECRET_KEY` env var +except `/health` and `/stripe/webhook`. + +### Licence and Auth +``` +POST /validate-email + Body: { email } + Returns: { valid, tier, reason } + Logic: domain check → seat count → seat registration + Note: sheet_id binding removed — standalone script model + +POST /stripe/webhook + Stripe signature verified via STRIPE_WEBHOOK_SECRET + Creates/updates tenant on payment events + Sets status=inactive on cancellation +``` + +### Trip-Intent Matching + Research +``` +POST /research/find-urls + Body: { destination, duration, travel_style, competitor_ids } + Returns: { matches: [ { competitor_id, name, url, confidence, found } ] } + Runs Firecrawl /map concurrently per competitor + +POST /research + Body: { tenant_id, destination, duration, travel_style, tab_type, + competitors: [ { id, name, url (confirmed) } ], + prompt_template? } + Returns: { job_id, status: "queued" } + Enqueues to RQ — never runs synchronously + +GET /research/status/ + Returns: { status, progress, results, error } + +GET /research/history + Query: email, limit (default 5) + Returns: last N runs for tenant + +GET /research/history/ + Returns: full stored results for a specific run + +POST /preview-prompt + Returns: { prompt } — dry run, no scraping +``` + +### Dashboard Data +``` +GET /competitors + Header: X-User-Email + Returns: [ { id, name, url, primary_market, last_scraped, + change_detected, status } ] + +POST /battlecards + Body: { email } + Returns: [ { competitor_name, content, content_json, generated_at } ] + +POST /comparable-trips + Body: { email } + Returns: [ { client_trip, competitor_trip, competitor, + similarity_score, price_difference, duration_difference } ] +``` + +### Apps Script Export Integration +``` +POST /sheet/push + Body: { email, job_id, tab_name } + Triggers Apps Script pushToSheet() with stored results for job_id + Returns: { success, rows_written } + +GET /sheet/tabs + Header: X-User-Email + Returns list of available tabs in PM's registered workbook +``` + +### Dashboard Account +``` +GET /account/ +POST /account/seats +DELETE /account/seats/ +POST /account/competitors — add single competitor (requires: name, website, + catalogue_url, primary_market) +POST /account/competitors/bulk — bulk import from CSV (validates, previews, imports) +GET /account/competitors/template — download blank CSV template with correct headers +PUT /account/competitors/ +DELETE /account/competitors/ — deactivates + deregisters Firecrawl Monitor +GET /account/template +PATCH /account/onboarding — update onboarding_checklist fields +GET /billing/portal — Stripe customer portal session URL +POST /billing/upgrade — creates Stripe checkout for tier upgrade + Body: { tenant_id, target_tier } + Returns: { checkout_url } +POST /auth/request-password-reset — Body: { email } → triggers PocketBase reset email +POST /auth/confirm-password-reset — Body: { token, password, passwordConfirm } +``` + +### Internal (n8n calls) +``` +POST /internal/scrape/ +POST /internal/battlecard/ +POST /internal/embed-products/ +POST /internal/match-comparable/ +POST /internal/weekly-brief/ +POST /internal/monitor-webhook +POST /internal/monitor/register/ +POST /internal/monitor/deregister/ +POST /internal/discover/ — Discover tier only +``` + +### System +``` +GET /health — returns { status: "ok", version } +``` + +--- + +## 10. Concurrency — RQ Job Queue + +The `/research` endpoint NEVER runs synchronously. +Every research job is enqueued to RQ and processed by background workers. + +### Queue Priority +```python +# Three priority queues — Intelligence tier gets processed first +high_q = Queue('high', connection=redis_conn) # Intelligence tier +normal_q = Queue('normal', connection=redis_conn) # Discover + Analyse +low_q = Queue('low', connection=redis_conn) # background tasks + +# Worker consumes in priority order +# command: rq worker high normal low -u redis://bettersight-redis:6379 +``` + +### Job Function (jobs.py) +The job function calls scrape and extract functions from `core/scraper.py` +and `core/extractor.py`. It adds PocketBase status updates and the +fast/slow path check around the existing logic. + +When running as an RQ job, `jobs.py` imports from `core/scraper.py` +and `core/extractor.py`. Never bypass the layer structure. + +--- + +## 9. Licence Validation Logic + +### Three layers — all must pass + +**Layer 1 — Domain check** +Extract domain from email. Look up active tenant with matching domain in PocketBase. +If no match → reject with "Domain not registered". + +**Layer 2 — Seat count** +Check if email already has an active seat for this tenant. +If yes → update last_accessed, allow. +If no → count active seats. If at limit → reject with "Seat limit reached". +If under limit → register new seat, allow. + +**Layer 3 — Sheet ID binding** +On first validation for a tenant, store the sheet_id in the tenants record. +On subsequent validations, compare sheet_id. If mismatch → reject with +"Licence bound to a different Sheet. Contact support." + +### Cache +Apps Script caches validation result for 6 hours via CacheService. +Cache key: `BETTERSIGHT_VALID` and `BETTERSIGHT_TIER`. +On cache hit — skip API call entirely. + +--- + +## 10. Scraping Architecture + +### Refactor app.py Before New Development + +The existing `app.py` must be refactored into the modular three-layer +structure before any new development begins. Logic stays identical — +it is reorganised into correct layers, not rewritten. + +**Core IP preserved in `core/`:** +- Playwright stealth browser with cookie consent handling and lazy load scrolling +- BeautifulSoup HTML cleaning +- AI web fetch fallback via OpenRouter for bot-protected sites +- OpenRouter model cycling with fallback chain +- Multi-currency extraction + +**Industry config moved to `industries/adventure_travel/`:** +- NGS vs Standard field differentiation +- Prompt builder and injection +- JSON extraction schema + +**Client config moved to `config/client.py`:** +- Google Sheet write-back via gspread +- Sheet ID, tab names, column assignments + +### Modular Scraping Architecture + +app.py must be refactored into a three-layer modular structure before +any new development begins. This separates infrastructure from industry +config from client config — making the codebase maintainable, testable, +and extensible to other industries without touching core scraping logic. + +**Target folder structure:** + +``` +backend/ + core/ + scraper.py — Playwright browser setup, stealth, waiting, HTML cleaning + extractor.py — OpenRouter model cycling, JSON parsing, fallback chain + ai_fetch.py — AI web fetch fallback (existing fetch_with_ai()) + industries/ + adventure_travel/ + fields.py — STANDARD_FIELDS, NGS_FIELDS column maps + prompt.py — build_prompt(), inject_page_content() + schema.py — JSON extraction schema for this vertical + # future verticals added here — never touch core/ + config/ + client.py — SHEET_ID, tab names, column assignments per client + jobs.py — RQ job functions — imports from core/ and industries/ + api.py — Flask routes + app.py — Entry point only — imports and wires everything together +``` + +**The three layers:** + +Layer 1 — `core/` — never changes between industries or clients +- Playwright stealth browser config +- Smart waiting and scroll logic +- Cookie consent handling +- HTML cleaning via BeautifulSoup +- OpenRouter model cycling and JSON parsing +- AI web fetch fallback + +Layer 2 — `industries/{vertical}/` — swapped per vertical +- Field maps (column positions in the Sheet) +- Prompt template and context ("you are a travel industry analyst") +- JSON extraction schema (what fields to extract) +- Tab names + +Layer 3 — `config/client.py` — swapped per client within a vertical +- Sheet ID +- Column assignments per competitor +- Competitor DB tab structure + +**To pivot to a new industry:** +Create a new folder under `industries/` with three files. +Core scraping logic is never touched. +Sheet template changes are driven entirely by the field map. + +### playwright-stealth Integration + +Replace the manual `add_init_script` block in `core/scraper.py` with +`playwright-stealth` — a comprehensive stealth library applying 20+ +evasion techniques automatically. + +``` +pip install playwright-stealth +``` + +In `core/scraper.py` — replace the existing `add_init_script` call: + +```python +from playwright_stealth import stealth_async + +async def render_page(url: str) -> dict: + async with async_playwright() as p: + browser = await p.chromium.launch(...) + context = await browser.new_context(...) + page = await context.new_page() + + # Replaces the manual add_init_script block entirely + await stealth_async(page) + + # Rest of existing code unchanged + await page.goto(url, ...) +``` + +`stealth_async` handles: navigator.webdriver spoofing, plugins array, +languages, chrome runtime, canvas fingerprint, WebGL masking, audio +context randomisation, and ~15 additional evasion techniques that the +manual script did not cover. + +### Smart Proxy Service — `core/proxy_service.py` + +The proxy layer is a standalone service in `core/proxy_service.py`. +It is the ONLY place in the codebase that knows about any proxy provider. +Switching providers or adjusting the switching logic requires changing only this file. + +**Two providers — smart auto-switching:** +- **Primary:** Webshare (~$3/month flat, free tier available) +- **Fallback:** Bright Data pay-as-you-go (~$4-8/GB, only used when Webshare is blocked) + +At MVP scraping scale (6 competitors × 5 clients = 30 pages/day) total proxy +cost is under $5/month. Bright Data fallback costs pennies at this volume. + +**PocketBase collection — `proxy_overrides`** +``` +id auto +tenant_id relation → tenants (nullable — global overrides have no tenant) +domain text (e.g. intrepidtravel.com) +provider select [webshare, brightdata] +blocked_count integer +last_blocked date +created autodate +``` + +**ProxyService class:** +```python +class ProxyService: + """ + Smart proxy provider — uses Webshare by default with country targeting, + auto-escalates to Bright Data for domains that block Webshare. + + Country targeting uses competitors.primary_market (ISO code e.g. GB, US, AU) + passed through from the competitor record — no hardcoded country map in code. + + Data flow (first request to a domain): + domain + country → proxy_overrides lookup → no override found → + Webshare config with country targeting returned → + scraper attempts request → if blocked → + escalate_to_brightdata(domain) called → + Bright Data config returned → scraper retries → + override stored in PocketBase for this domain + + Data flow (subsequent requests to a known blocked domain): + domain + country → proxy_overrides lookup → brightdata override found → + Bright Data config returned immediately — no Webshare attempt + """ + + def get_proxy_for_domain(self, domain: str, country: str = None) -> dict: + """ + Returns requests proxy config for a domain with optional country targeting. + Routes to Bright Data if domain has been previously blocked on Webshare. + """ + override = proxy_repository.get_by_domain(domain) + if override and override['provider'] == 'brightdata': + return self._brightdata_requests() + return self._webshare_requests(country=country) + + def get_playwright_proxy_for_domain(self, domain: str, country: str = None) -> dict: + """ + Returns Playwright proxy config for a domain with optional country targeting. + Routes to Bright Data if domain has been previously blocked on Webshare. + """ + override = proxy_repository.get_by_domain(domain) + if override and override['provider'] == 'brightdata': + return self._brightdata_playwright() + return self._webshare_playwright(country=country) + + def escalate_to_brightdata(self, domain: str): + """ + Records that Webshare was blocked on this domain. + Upserts a proxy_override record so future requests skip Webshare entirely. + """ + existing = proxy_repository.get_by_domain(domain) + if existing: + proxy_repository.update(existing['id'], { + 'provider': 'brightdata', + 'blocked_count': existing['blocked_count'] + 1, + 'last_blocked': datetime.now().isoformat() + }) + else: + proxy_repository.create({ + 'domain': domain, + 'provider': 'brightdata', + 'blocked_count': 1, + 'last_blocked': datetime.now().isoformat() + }) + + def _webshare_requests(self, country: str = None) -> dict: + """ + Webshare requests proxy config with optional country targeting. + Format per Webshare docs: username-{COUNTRY}-rotate + No country = general rotation across full pool. + """ + suffix = f"-{country.upper()}-rotate" if country else "-rotate" + username = f"{WEBSHARE_USER}{suffix}" + return { + "http": f"http://{username}:{WEBSHARE_PASS}@p.webshare.io:80", + "https": f"http://{username}:{WEBSHARE_PASS}@p.webshare.io:80" + } + + def _webshare_playwright(self, country: str = None) -> dict: + """Webshare Playwright proxy config with optional country targeting.""" + suffix = f"-{country.upper()}-rotate" if country else "-rotate" + return { + "server": "http://p.webshare.io:80", + "username": f"{WEBSHARE_USER}{suffix}", + "password": WEBSHARE_PASS + } + + def _brightdata_requests(self) -> dict: + return { + "http": f"http://{BRIGHTDATA_USER}:{BRIGHTDATA_PASS}@brd.superproxy.io:22225", + "https": f"http://{BRIGHTDATA_USER}:{BRIGHTDATA_PASS}@brd.superproxy.io:22225" + } + + def _brightdata_playwright(self) -> dict: + return { + "server": "http://brd.superproxy.io:22225", + "username": BRIGHTDATA_USER, + "password": BRIGHTDATA_PASS + } +``` + +**Block detection and auto-switch in `core/scraper.py`:** +```python +async def render_page(url: str, country: str = None) -> dict: + """ + Renders a page using the smart proxy service with country targeting. + country comes from competitors.primary_market in PocketBase. + Detects blocks and auto-escalates to Bright Data if Webshare fails. + + Data flow: + url + country → domain extracted → + proxy_service.get_playwright_proxy_for_domain(domain, country) → + Playwright render → if blocked → escalate_to_brightdata(domain) → + retry with Bright Data (no country targeting at fallback level) + """ + domain = extract_domain(url) + proxy = proxy_service.get_playwright_proxy_for_domain(domain, country) + result = await _render_with_proxy(url, proxy) + + if _is_blocked(result): + logger.warning(f'Webshare blocked on {domain} — escalating to Bright Data') + proxy_service.escalate_to_brightdata(domain) + result = await _render_with_proxy(url, proxy_service._brightdata_playwright()) + + return result + + +def _is_blocked(result: dict) -> bool: + """ + Detects common block signals from scrape result. + Returns True if the response looks like a bot wall rather than real content. + Signals: low char count, HTTP 403/429, Cloudflare challenge, CAPTCHA text. + """ + if result.get('char_count', 0) < 500: + return True + if result.get('status_code') in [403, 429]: + return True + block_signals = [ + 'cf-browser-verification', 'captcha', + 'access denied', 'blocked', 'robot' + ] + return any(s in result.get('text', '').lower() for s in block_signals) +``` + +**In `jobs.py` — primary_market passed through automatically:** +```python +def run_research_job(job_id: str, data: dict): + """ + Pulls primary_market from each competitor record and passes it + to scrape() for automatic country-targeted proxy routing. + No manual country configuration needed per job. + """ + for i, competitor in enumerate(data['competitors']): + competitor_record = competitor_repository.get_by_id(competitor['id']) + page_data = scrape( + url=competitor['url'], + country=competitor_record.get('primary_market') # e.g. "GB", "US", "AU" + ) + # rest of job unchanged +``` + +**Cost summary:** +``` +Webshare free tier — 10 proxies shared, enough for MVP testing +Webshare paid — ~$3/month, handles the vast majority of requests +Bright Data fallback — pay-as-you-go ~$4-8/GB, <$2/month at MVP scale +Total proxy cost MVP — under $5/month +``` + +### OpenRouter Model Cycling — `core/extractor.py` + +Cycles through free tier models. Falls back to a paid model if all +free models fail or return rate limit errors. Never fails silently. + +```python +FREE_MODELS = os.getenv('OPENROUTER_FREE_MODELS', '').split(',') +FALLBACK_MODEL = os.getenv('OPENROUTER_FALLBACK_MODEL', + 'anthropic/claude-haiku-4-5') + +def extract_with_openrouter(prompt: str, content: str) -> dict: + """ + Cycles through free OpenRouter models in sequence. + Falls back to the paid fallback model if all free models fail. + Never returns None — raises ExtractorError only after all models exhausted. + + Data flow: + prompt + content → for each FREE_MODELS: + POST openrouter.ai/api/v1/chat/completions → + if success: return parsed JSON → + if rate limit or error: try next model → + if all free models fail → try FALLBACK_MODEL (paid) → + if fallback fails → raise ExtractorError, log to Sentry + """ + models_to_try = FREE_MODELS + [FALLBACK_MODEL] + + for model in models_to_try: + try: + response = _call_openrouter(model, prompt, content) + if response: + if model == FALLBACK_MODEL: + logger.warning( + f'Free models exhausted — used paid fallback: {model}' + ) + sentry_sdk.capture_message( + f'OpenRouter paid fallback used', + level='warning' + ) + return response + except RateLimitError: + logger.warning(f'Rate limit on {model} — trying next') + continue + except Exception as e: + logger.error(f'Model {model} failed: {str(e)}') + continue + + raise ExtractorError('All OpenRouter models exhausted including paid fallback') +``` + +Firecrawl is in the MVP stack for `/monitor` only. +`/map` discovery endpoint is Intelligence tier — do not build in MVP. + +--- + +## 11. Battlecard Generation + +After each scrape run completes, n8n calls `/internal/battlecard/`. + +```python +def generate_battlecard(competitor_id: str, tenant_id: str) -> dict: + """ + Pulls latest product data for a competitor from PocketBase. + Passes structured data to Claude Haiku via OpenRouter. + Saves generated battlecard text and JSON back to PocketBase. + Returns battlecard dict for immediate use. + + Data flow: + competitor_id → PocketBase products (latest 10) → + structured prompt → OpenRouter (claude-haiku) → + battlecard text + JSON → PocketBase battlecards table + """ +``` + +Prompt template: +``` +You are a competitive intelligence analyst for an adventure travel operator. +Based on this competitor data, write a concise battlecard. + +Competitor: {name} +Top trips: {top_trips} +Price range: {price_low} - {price_high} USD +Avg duration: {avg_duration} days +Positioning: {positioning_summary} +Recent changes: {recent_changes} + +Write: +1. One line positioning summary (max 15 words) +2. Three strengths (bullet points, max 8 words each) +3. Three weaknesses (bullet points, max 8 words each) +4. One pricing observation (max 20 words) + +Return ONLY valid JSON: +{ + "positioning": "...", + "strengths": ["...", "...", "..."], + "weaknesses": ["...", "...", "..."], + "pricing_observation": "..." +} +``` + +--- + +## 12. Semantic Trip Matching + +### Embedding generation + +After products are stored, embed each new trip using text-embedding-3-small. +Store vector as JSON in the products.embedding field. + +```python +def embed_trip(trip: dict) -> list: + """ + Generates a semantic embedding for a trip using OpenRouter. + Combines trip name, destination, duration, and activities into + a single text representation before embedding. + + Data flow: + trip dict → formatted text string → OpenRouter embeddings API → + vector list → stored in products.embedding + """ + text = f""" + {trip['trip_name']} + Destination: {trip['destination']} + Duration: {trip['duration_days']} days + Activities: {trip['activities']} + Start: {trip['start_location']} + """ + response = openai_client.embeddings.create( + model="openai/text-embedding-3-small", + input=text + ) + return response.data[0].embedding +``` + +### Similarity matching + +```python +def find_comparable_trips(client_trips, competitor_trips, threshold=0.75): + """ + Calculates cosine similarity between all client trips and all + competitor trips. Returns matches above the threshold sorted by + similarity score descending. + + Data flow: + client_trips + competitor_trips (each with .embedding) → + pairwise cosine similarity → filter by threshold → + sort by score → list of match dicts + """ +``` + +Threshold: 0.75. Matches stored in comparable_matches table. + +--- + +## 13. Weekly HTML Email Brief + +### No PDF — HTML email only + +WeasyPrint has been removed from the stack. The Monday brief is a +rich Jinja2-rendered HTML email sent via Resend. Same content, zero +Docker dependency, zero PDF complexity. + +Brief email includes: +- Bettersight branded header with tenant name +- Market summary (price changes, new products, competitors monitored) +- Price movements table (competitor, trip, previous, current, delta, percent) +- New products this week (formatted cards) +- Battlecard updates summary +- Link to app.bettersight.io +- Unsubscribe footer + +### n8n Workflow — Monday 6am + +``` +Schedule Trigger (Monday 06:00 AST) + → HTTP POST /internal/weekly-brief/{tenant_id} for each active tenant + → Flask renders brief.html Jinja2 template with weekly PocketBase data + → Resend sends HTML email to tenant admin + → Logged to digest_logs +``` + +### Brief service + +```python +def send_weekly_brief(tenant_id: str): + """ + Renders the weekly intelligence brief as branded HTML and + sends it via Resend to the tenant admin email. + + Data flow: + tenant_id → PocketBase (price_history, products, battlecards + for current week) → brief.html Jinja2 template rendered → + Resend API → digest_logs record created + """ + data = brief_repository.get_weekly_data(tenant_id) + html = render_template('emails/brief.html', **data) + resend.Emails.send({ + "from": "Bettersight ", + "to": data['admin_email'], + "subject": f"Bettersight Weekly Brief — Week of {data['week_of']}", + "html": html + }) + brief_repository.log_digest(tenant_id, 'weekly') +``` + +--- + +## 15. Google Apps Script + +### Role — Export Integration Only + +The Apps Script is no longer the primary interface. The dashboard is. +The script's only job is to push analysis results from the dashboard +into the PM's Google Sheet workbook on demand. + +**Three functions only:** + +``` +pushToSheet(results, tabName) — writes dashboard results to selected tab +getWorkbookTabs() — returns tab list for dashboard push dropdown +validateLicence() — confirms PM is authorised (email check only) +``` + +### Deployment Model — Standalone Script + +Deployed as a standalone script, not container-bound to any workbook. +PM installs once via a shareable link from app.bettersight.io/account. +Works in any Google Sheet workbook they open from that point forward. + +### File Structure + +``` +appscript/ + Code.gs — onOpen(), install entry point + Validation.gs — validateLicence() email-only + Export.gs — pushToSheet(), getWorkbookTabs() + Sidebar.html — minimal UI: tab selector + push button only +``` + +### Export Flow + +```javascript +/** + * Pushes analysis results from the dashboard to the PM's selected + * workbook tab. Called via the dashboard "Push to Sheet" button + * which hits Flask POST /sheet/push, which calls this function. + * + * Writes results using header-based column detection — never + * hardcoded column positions. Resilient to any sheet restructuring. + * + * Flow: + * job_id → Flask /research/history/{job_id} → stored results → + * getWorkbookTabs() shows tab selector in dashboard dropdown → + * PM selects tab → POST /sheet/push → + * detectColumns(sheet) → writeResults(results, columnMap, sheet) + */ +function pushToSheet(results, tabName) { ... } + +/** + * Returns list of all tab names in the PM's current workbook. + * Used by the dashboard to populate the "Push to Sheet" tab selector. + */ +function getWorkbookTabs() { + return SpreadsheetApp.getActiveSpreadsheet() + .getSheets() + .map(s => s.getName()); +} + +/** + * Detects column positions by matching header names in row 1. + * Returns { fieldName: columnIndex } map. + * Falls back to STANDARD_FIELDS positions if headers not found. + */ +function detectColumns(sheet) { ... } +``` + +### Sidebar UI (minimal) + +``` +┌─────────────────────────────┐ +│ 🔵 Bettersight │ +│ Export results to Sheet │ +│ ───────────────────────── │ +│ Push to tab: │ +│ [Bettersight Analysis ▾] │ +│ │ +│ [ Push last analysis ] │ +│ ───────────────────────── │ +│ Last push: Today 9:14am │ +│ 5 competitors · Peru │ +└─────────────────────────────┘ +``` + +The full analysis workflow — competitor selection, trip-intent matching, +job status, results view, history, confidence indicators — all live in +the dashboard. The sidebar is a push button only. + +The Apps Script is deployed as a **standalone script**, NOT container-bound +to a specific Google Sheet file. This is the most important architectural +decision for the add-on. + +**Why standalone:** +- Works in any Google Sheet the PM opens — including their existing + 20-sheet stage gate workbook +- PM installs once via a shareable install link from the Bettersight dashboard +- They import the Bettersight template tab into their existing workbook +- The sidebar runs analysis and writes to whichever tab they select +- Cross-sheet formulas in their stage gate sheet reference the Bettersight tab + natively — no copy-paste ever needed after setup + +**What the PM receives:** +1. A template Sheet tab — downloadable from app.bettersight.io/account + as a Google Sheets link. They copy this tab into their existing workbook. +2. A sidebar install link — one click installs the standalone script on + their Google account. Works in any workbook from that point forward. + +**Sheet ID binding update:** +Because the script is standalone, the licence validation no longer binds +to a single Sheet ID. Instead it binds to the PM's Google account email +(already implemented via domain + seat validation). Remove the +sheet_id binding layer from the licence validation — it is redundant +and would break the standalone model. + +--- + +### File Structure + +``` +appscript/ + Code.gs — onOpen(), menu setup, sidebar launch, install entry point + Validation.gs — validateLicence() + Research.gs — runResearch(), pollStatus(), writeResultsToSheet() + Competitors.gs — loadCompetitorsFromPocketBase(), refreshCompetitorList() + Settings.gs — saveSettings(), loadSettings(), getTargetSheet() + Battlecards.gs — refreshBattlecards(), writeBattlecardsTab() + ComparableTrips.gs — refreshComparableTrips(), writeComparableTab() + Sidebar.html — sidebar UI +``` + +--- + +### Validation Flow (Validation.gs) + +```javascript +/** + * Validates the current user's licence against the Bettersight API. + * Standalone script — validates by email only, no sheet_id binding. + * + * Flow: + * 1. Check 6-hour cache — return true immediately if cache valid + * 2. Get user email via Session.getActiveUser().getEmail() + * 3. POST to /validate-email with email only + * 4. On success — cache result and tier for 6 hours, return true + * 5. On failure — show alert with reason, return false + * + * Cache keys: BETTERSIGHT_VALID, BETTERSIGHT_TIER + * Cache duration: 21600 seconds (6 hours) + */ +function validateLicence() { ... } +``` + +--- + +### Settings — Tab Selection and Persistence (Settings.gs) + +The PM selects which tab in their workbook to write results to. +This setting persists across sessions via PropertiesService. + +```javascript +/** + * Returns the target sheet tab for writing analysis results. + * Reads from UserProperties — falls back to tab named + * 'Bettersight Analysis' if no preference is stored. + * + * Flow: + * UserProperties.getProperty('TARGET_TAB') → + * if found: return that tab → + * if not found: find 'Bettersight Analysis' tab → + * if not found: prompt PM to select tab via dropdown + */ +function getTargetSheet() { ... } + +/** + * Saves all sidebar settings to UserProperties so they persist + * across sessions. Called automatically after each successful run. + * + * Settings persisted: + * TARGET_TAB — which sheet tab to write to + * TRAVEL_STYLE — last used travel style + * PRODUCT_NAME — last used G Adventures product name + * DESTINATION — last used destination + * DURATION — last used duration + * TAB_TYPE — Standard or NGS + * SELECTED_COMPS — JSON array of last selected competitor IDs + */ +function saveSettings(settings) { ... } + +/** + * Loads all persisted settings and returns them to the sidebar + * so the PM's last run is pre-filled on open. + */ +function loadSettings() { ... } +``` + +--- + +### Competitor Auto-Population (Competitors.gs) + +Competitors are loaded from PocketBase on sidebar open. +The PM never manually types a URL again. + +```javascript +/** + * Fetches the PM's competitor list from the Bettersight API. + * Called on sidebar open. Returns array of competitor objects + * for rendering as checkboxes in the sidebar. + * + * Flow: + * validateLicence() → GET /competitors with email header → + * returns [ { id, name, url, primary_market, last_scraped, + * change_detected, status } ] → + * rendered as checkboxes in sidebar with status indicators + * + * Status indicators shown per competitor: + * ✓ Current (updated {time}) — last_scraped < 24hrs, no change + * ⚠ Changed ({time} ago) — change_detected = true + * ○ Stale — last_scraped > 24hrs + */ +function loadCompetitorsFromPocketBase() { ... } +``` + +--- + +### Column Auto-Detection (Research.gs) + +Column positions are detected by reading header names, not hardcoded numbers. +Makes the sheet resilient to any structural changes the PM makes. + +```javascript +/** + * Detects the column index for each field by matching header names + * in row 1 of the target sheet. Returns a field-to-column map. + * Falls back to STANDARD_FIELDS hardcoded positions if headers + * cannot be found — logs a warning when falling back. + * + * Flow: + * Read row 1 of target sheet → + * Build { fieldName: columnIndex } map by matching header text → + * Return map for use by writeResultsToSheet() + */ +function detectColumns(sheet) { ... } +``` + +--- + +### Research Flow with Per-Competitor Progress (Research.gs) + +```javascript +/** + * Submits a research job and polls for completion. + * Shows per-competitor status in the sidebar in real time. + * + * Flow: + * 1. validateLicence() — exit if invalid + * 2. getTargetSheet() — confirm which tab to write to + * 3. detectColumns(sheet) — map field names to column positions + * 4. loadSettings() — pre-fill job params from last run + * 5. POST to /research with selected competitor IDs and job params + * → receives job_id immediately + * 6. Poll GET /research/status/{job_id} every 3 seconds + * 7. On each poll update sidebar with per-competitor status: + * ✓ Intrepid Travel — Peru Classic $2,190 + * ✓ Exodus Travels — Machu Picchu $2,050 + * ⟳ Flash Pack — scraping... + * Kimkim — queued + * 8. On complete — writeResultsToSheet() using detected columns + * 9. saveSettings() — persist this run's settings for next time + * 10. Show confidence indicators per competitor in sidebar + * 11. Timeout after 75 attempts (5 minutes) + */ +function runResearch() { ... } + +/** + * Writes extracted competitor data to the target sheet. + * Uses column map from detectColumns() — never hardcoded positions. + * Only writes fields that exist in the detected column map. + * + * Flow: + * results array + columnMap + sheet → + * for each competitor result: + * for each field in columnMap: + * sheet.getRange(fieldRow, columnMap[field]).setValue(value) + */ +function writeResultsToSheet(results, columnMap, sheet) { ... } +``` + +--- + +### Single Competitor Re-Run (Research.gs) + +```javascript +/** + * Triggers a fresh scrape and extraction for a single competitor. + * Used when the PM wants to refresh one competitor without + * re-running the full job. Updates only that competitor's column. + * + * Flow: + * competitorId → POST /research with single competitor → + * poll status → write result to sheet (single column only) → + * update sidebar status for that competitor only + * + * Called from a "Refresh" button shown per competitor + * in the completed run status list in the sidebar. + */ +function refreshSingleCompetitor(competitorId) { ... } +``` + +--- + +### Confidence Indicators (Research.gs) + +```javascript +/** + * Renders a confidence bar per competitor in the sidebar + * after a run completes. Based on the relevancy score (1-5) + * returned by the AI extraction. + * + * Display format: + * Intrepid Travel ████████░░ 82% — Good match + * Flash Pack ██████░░░░ 61% — Review recommended + * Kimkim ████░░░░░░ 44% — Low confidence ⚠ + * + * Low confidence (< 60%) shows a warning prompt: + * "The AI had low confidence extracting Flash Pack data. + * The page may have changed structure. Consider a manual review." + */ +function renderConfidenceIndicators(results) { ... } +``` + +--- + +### Analysis History (Research.gs) + +```javascript +/** + * Fetches the last 5 analysis runs for the current tenant + * from PocketBase. Displayed in the sidebar as a history panel. + * PM can restore a previous run's data to the sheet for comparison. + * + * Flow: + * GET /research/history?email={email}&limit=5 → + * returns [ { job_id, run_at, competitors, summary } ] → + * rendered as list in sidebar history panel + * + * Each history item shows: + * Mon 9 Jun 9:14am · 5 competitors · Intrepid, Exodus, Flash Pack... + * [Restore to sheet] + * + * Restore writes that run's stored results back to the sheet + * using the same writeResultsToSheet() function. + */ +function loadAnalysisHistory() { ... } + +/** + * Restores a previous run's results to the current sheet. + * Fetches stored results from PocketBase by job_id. + * Warns PM that this will overwrite current sheet data. + */ +function restoreHistoryRun(jobId) { ... } +``` + +--- + +### Copy to Clipboard (Sidebar.html) + +A "Copy rows" button in the sidebar copies selected competitor rows +as tab-separated values — paste-ready into any Google Sheet, Excel, +or Notion table without formatting issues. + +```javascript +/** + * Copies the selected competitor result rows as tab-separated + * values to the clipboard. PM can paste directly into any sheet. + * No formatting, no formulas — clean values only. + * + * Called from "Copy selected rows" button in sidebar. + */ +function copyRowsToClipboard(selectedCompetitorIds) { ... } +``` + +--- + +### Sidebar UI Sections (Sidebar.html) + +``` +┌─────────────────────────────────┐ +│ 🔵 Bettersight │ +│ ───────────────────────────── │ +│ ANALYSIS SETTINGS │ +│ Travel style [Classic ▾] │ +│ Product name [____________] │ +│ Destination [____________] │ +│ Duration [____________] │ +│ Tab type [Standard ● NGS○]│ +│ Write to tab [Bettersight ▾]│ ← tab selector +│ ───────────────────────────── │ +│ COMPETITORS │ +│ ☑ Intrepid Travel ✓ Current │ ← from PocketBase +│ ☑ Exodus Travels ✓ Current │ +│ ☑ Flash Pack ⚠ Changed │ +│ ☐ Kimkim ○ Stale │ +│ ───────────────────────────── │ +│ [ Run Analysis ] │ +│ [ Copy selected rows ] │ +│ ───────────────────────────── │ +│ LAST RUN RESULTS │ +│ ✓ Intrepid ████████░░ 82% │ ← confidence +│ ✓ Exodus █████████░ 91% │ +│ ⚠ Flash Pk ██████░░░░ 61% │ ← low confidence warn +│ [↺ Refresh Flash Pack] │ ← single re-run +│ [ Copy selected rows ] │ +│ ───────────────────────────── │ +│ HISTORY │ +│ Mon 9 Jun · 5 competitors │ +│ [Restore] │ +│ Mon 2 Jun · 4 competitors │ +│ [Restore] │ +└─────────────────────────────────┘ +``` + +--- + +### New Flask Routes Supporting Apps Script Improvements + +``` +GET /competitors — returns competitor list for sidebar auto-population + includes: id, name, url, primary_market, + last_scraped, change_detected, status +GET /research/history — last 5 runs for tenant (by email header) +GET /research/history/ — full results for a specific run by job_id + used by restoreHistoryRun() +``` + +--- + +## 15. n8n Workflows + +### Daily scrape (6am every day) +``` +Schedule Trigger (06:00 AST) + → For each active tenant: + → Query PocketBase for competitors where: + change_detected = true OR last_scraped > 24hrs ago + → HTTP POST /internal/scrape/{tenant_id} with filtered competitor list + → RQ job queued → Playwright scrape → AI extraction + → PocketBase updated (products, price_history) + → change_detected reset to false + → last_scraped updated to now + → price_history diff calculated automatically + → is_new flag set on new products automatically +``` + +### Battlecard regeneration (after scrape) +``` +PocketBase webhook on scrape_runs status=complete + → HTTP POST /internal/battlecard/{competitor_id} + → Runs for each competitor in the completed scrape run +``` + +### Price change alert +``` +PocketBase webhook on price_history insert + → Code node: check change_percent threshold (>5% triggers alert) + → Creates alert record in PocketBase alerts table + { alert_type: "price_change", competitor_id, product_id, + change_amount, change_percent, delivered_dashboard: true } + → Alert appears in dashboard activity feed immediately + → n8n daily digest (6pm): collects unread alerts → Resend email summary +``` + +### New product alert +``` +PocketBase webhook on products insert where is_new=true + → Creates alert record in PocketBase alerts table + { alert_type: "new_product", competitor_id, product_id, + delivered_dashboard: true } + → Alert appears in dashboard activity feed immediately + → Included in daily digest email (6pm) and Monday brief +``` + +### [Discover tier] External notification integrations +``` +When PM connects Slack/Teams/WhatsApp in account settings: + → webhook_url stored in tenant_notifications table + → n8n alert workflow checks tenant_notifications on each alert + → If webhook configured: HTTP POST to Slack/Teams webhook URL + → If Twilio configured: POST to Twilio WhatsApp API +``` + +### Monday morning brief (6am) +``` +Schedule Trigger (Monday 06:00 AST) + → HTTP POST /internal/weekly-brief/{tenant_id} per active tenant + → Flask renders brief.html Jinja2 template with weekly data + → Resend sends HTML email to tenant admin email + → Logged to digest_logs +``` + +### Sunday night catalogue discovery (Intelligence tier — NOT in MVP) +``` +DEFERRED — do not build until Intelligence tier development begins. +Requires Firecrawl self-hosted which is not in the current stack. +``` + +--- + +## 16. Stripe Integration + +### Webhook events handled + +``` +checkout.session.completed + → Create tenant in PocketBase + → Set status=active + → Generate welcome email with Sheet template download link + +customer.subscription.updated + → Update tier if plan changed + → Update max_seats based on new tier + +customer.subscription.deleted + → Set tenant status=inactive + → All seats blocked at next cache expiry (max 6 hours) +``` + +### Tier → seat mapping + +```python +TIER_SEATS = { + "analyse": 5, + "discover": 8, + "intelligence": 10, + "enterprise": 999 # effectively unlimited +} + +# Additional seats available on any tier +ADDITIONAL_SEAT_PRICE = 45 # USD per seat per month +# Configured as a Stripe add-on price on each subscription + +# Enterprise tier is not in Stripe public product list +# Created manually per client as a custom Stripe price +``` + +--- + +## 17. Environment Variables + +```bash +# Flask +FLASK_ENV=production +API_SECRET_KEY= + +# PocketBase +POCKETBASE_URL=http://pocketbase:8090 +POCKETBASE_ADMIN_EMAIL= +POCKETBASE_ADMIN_PASSWORD= + +# OpenRouter +OPENROUTER_API_KEY= +OPENROUTER_FREE_MODELS=google/gemini-flash-1.5,meta-llama/llama-3.1-8b-instruct,mistralai/mistral-7b-instruct +OPENROUTER_FALLBACK_MODEL=anthropic/claude-haiku-4-5 # paid fallback ~$0.25/1M tokens — used when free tier unavailable + +# Proxy — Primary (Webshare) +WEBSHARE_USER= +WEBSHARE_PASS= + +# Proxy — Fallback (Bright Data — pay-as-you-go, auto-escalated per domain) +BRIGHTDATA_USER= +BRIGHTDATA_PASS= + +# Proxy settings +PROXY_ESCALATION_THRESHOLD=2 # blocked_count before treating domain as permanently blocked + +# Stripe +STRIPE_SECRET_KEY= +STRIPE_WEBHOOK_SECRET= +STRIPE_PRICE_ANALYSE= # trial_period_days: 14 +STRIPE_PRICE_DISCOVER= # trial_period_days: 21 +STRIPE_PRICE_INTELLIGENCE= # trial_period_days: 30 +# Enterprise: created manually per client — no trial, no env var + +# Transactional email +RESEND_API_KEY= + +# Error tracking +SENTRY_DSN= # Required in staging and production — never skip + +# Hetzner Object Storage (backups) +HETZNER_S3_ACCESS_KEY= +HETZNER_S3_SECRET_KEY= +HETZNER_S3_ENDPOINT=https://fsn1.your-objectstorage.com +HETZNER_S3_BUCKET=bettersight-backups + +# Gotify (owner monitoring — never client-facing) +GOTIFY_URL=https://gotify.bettertend.net +GOTIFY_APP_TOKEN=A78_l3r9y.ZzBr- + +# Google Sheets (existing from app.py) +GOOGLE_SERVICE_ACCOUNT_JSON= +SHEET_ID= +STANDARD_TAB= +NGS_TAB= +``` + +--- + +## 18. Enterprise Software Development Standards + +> These standards are absolute. They are never negotiated, never skipped, +> and never deferred to a later phase. + +### Languages and Conventions + +- Backend: Python only. No mixed languages in the same service. +- Frontend: JavaScript only. No TypeScript. No `.ts` files. +- `snake_case` for Python. `camelCase` for JavaScript. +- No abbreviations unless universally understood (id, url, pdf). + +### Separation of Concerns + +**Backend layer hierarchy — no layer skips another:** + +``` +Routes (api.py) + → validate input, call services, return response + → NO business logic. NO direct DB calls. NO external API calls. + +Services (services/) + → all business logic and orchestration + → NO HTTP request/response objects + → NO direct DB queries or external API calls + → calls repositories and other services only + +Repositories (repositories/) + → all data access (PocketBase reads/writes, external API calls) + → NO business logic + → NO service calls + → returns plain dicts only + +jobs.py + → imports and calls service functions only + → NO direct DB access + → NO business logic +``` + +### Security + +- Every authenticated route checks `X-API-Key` against `API_SECRET_KEY` env var +- Every DB query for non-admin operations is tenant-scoped + (`tenant_id` filter on every PocketBase query) +- All credentials from env vars — hardcoded credentials are a build failure +- Stripe webhook validated via signature before any processing +- No stack traces in production error responses +- Seat validation checks domain and seat count — both every time + +### Observability + +- Every route logs: method, path, tenant_id (if known), duration, status code +- Every RQ job logs: job_id, tenant_id, start, completion, error if any +- `/health` endpoint always returns `{ status: "ok", version }` — no auth required +- Sentry configured via `SENTRY_DSN` env var in production +- n8n monitors `/health` every 5 minutes — Gotify alert on failure + +### Resilience + +- RQ job failures are logged to scrape_runs.error_log +- Failed scrapes do not fail the entire batch — continue to next competitor +- Side-effect failures (Gotify, email) never fail a primary operation +- All external API calls (OpenRouter, Bright Data, Firecrawl, Stripe) have + try/except with logged errors and graceful fallback +- RQ retries failed jobs automatically up to 3 times with backoff + +### Code Commenting — Mandatory + +**All code must be adequately commented. This is not optional.** + +Every function, route, service method, and job function must have: +1. A docstring stating what the function does (one sentence) +2. The data flow — what comes in, what happens to it, what goes out +3. Inline comments on every meaningful step + +```python +def validate_email(email: str, sheet_id: str) -> dict: + """ + Validates a Google user's access to Polaris via three checks: + domain registration, seat count, and Sheet ID binding. + + Data flow: + email + sheet_id → + domain extraction → PocketBase tenant lookup (Layer 1) → + seat count check → seat registration if new (Layer 2) → + sheet_id binding on first use / comparison on subsequent (Layer 3) → + { valid: bool, tier: str, reason: str } + """ + # Extract domain from email for tenant lookup + domain = email.split("@")[1] + + # Layer 1: Check if domain is registered to an active or trial tenant + tenant = tenant_repository.get_by_domain(domain) + if not tenant: + return {"valid": False, "reason": "Domain not registered"} + if tenant["status"] not in VALID_STATUSES: + return {"valid": False, "reason": "Subscription inactive"} + + # Layer 2: Check seat availability or register existing seat + seat_result = seat_repository.validate_or_register(tenant, email) + if not seat_result["valid"]: + return seat_result + + return {"valid": True, "tier": tenant["tier"]} +``` + +### Testing + +- 80% overall test coverage enforced — cannot merge below this +- 100% coverage on: licence validation, Stripe webhook, seat management, rollback paths +- Every repository has a mock implementation alongside the real one +- Unit tests for all services +- Integration tests for all API routes +- Tests live in `backend/tests/` mirroring the service/repository structure + +### Environments and Deployment + +- Three environments: development, staging, production +- Nothing reaches production without passing staging +- Production deploys require a tagged version — no unversioned production deploys +- Dokploy handles zero-downtime deploys via rolling restart + +### Data and Backup + +**Two-layer backup strategy — belt and braces:** + +**Layer 1 — Litestream (real-time WAL streaming)** +- Streams every SQLite write to Hetzner Object Storage in real-time +- Recovery point objective: seconds +- Recovery time objective: under 5 minutes +- Runs as a sidecar container alongside PocketBase + +**Layer 2 — Daily zip backup (point-in-time snapshots)** +- Daily 2am via crontab on the VPS — survives container restarts +- 7-day retention on Hetzner Object Storage +- Recovery point objective: 24 hours (fallback if Litestream stream is corrupted) + +```bash +# /etc/cron.d/bettersight-backup +0 2 * * * root /usr/local/bin/backup-pocketbase.sh + +# backup-pocketbase.sh +#!/bin/bash +DATE=$(date +%Y%m%d) +docker exec pocketbase /pb/pocketbase backup \ + --dir /pb/backups --name backup_$DATE.zip +rclone copy /pb/backups/ \ + hetzner-s3:bettersight-backups/ \ + --max-age 7d +find /pb/backups -name "*.zip" -mtime +2 -delete +``` + +- Audit logs (alerts, digest_logs) are append-only — no update or delete permitted +- Multi-step operations (scrape → store → alert) documented with rollback strategy + +### Strict Rules — Never Violate + +- Every authenticated endpoint validates `X-API-Key` — no exceptions +- Every DB query for non-admin roles includes `tenant_id` filter — no exceptions +- All credentials from env vars — hardcoded values are a build failure +- No layer skips another in the separation of concerns hierarchy +- A side-effect failure never fails a primary operation +- Every multi-step operation has a documented rollback strategy +- Mock implementations ship alongside every real repository +- 80% test coverage enforced by CI — cannot merge below this +- 100% coverage on payment, auth, webhook, and rollback paths +- Nothing reaches production without passing staging +- Production deploys require a tagged version +- Audit logs are append-only — no update or delete ever +- Daily backup with off-server storage — no exceptions +- Sentry configured in staging AND production — silent failures are unacceptable +- `/health` endpoint exists and is monitored — automated alert on failure +- All code and functions are adequately commented — uncommented code is a build failure +- Every Flask error response is structured JSON — never plain text +- Every page has a defined empty state — no blank pages ever +- Rate limiting applied to every public endpoint via Flask-Limiter +- PII (email addresses) never sent to Sentry — scrubbed before send +- Data retention policy enforced by weekly n8n cleanup workflow +- GDPR deletion endpoint exists and tested before launch +- Session expiry handled gracefully — PM returned to exact page after re-login +- In-progress analysis jobs survive session expiry — resume on return + +--- + +## 19. Build Order + +Follow this sequence. Do not start a phase until the previous is complete and tested. + +``` +Phase 1 — Data foundation + 1. PocketBase schema (all collections) + 2. Repository layer (all repositories with mocks) + 3. Service layer (licence, scrape, battlecard, embedding, brief, alert, + trip_finder) + 4. Flask API routes (all endpoints including /research/find-urls) + 5. RQ job queue setup (jobs.py, worker config) + 6. Tests for all of the above (80% floor) + +Phase 2 — Refactor app.py into modular structure + 7. Create proxy_overrides PocketBase collection + 8. Create repositories/proxy_repository.py + 9. Create core/proxy_service.py — Webshare primary + Bright Data fallback + 10. Create core/scraper.py — Playwright + playwright-stealth + ProxyService + 11. Create core/extractor.py — OpenRouter cycling and JSON parsing + 12. Create core/ai_fetch.py — AI web fetch fallback + ProxyService + 13. Create industries/adventure_travel/fields.py + 14. Create industries/adventure_travel/prompt.py + 15. Create industries/adventure_travel/schema.py + 16. Create config/client.py + 17. Create services/trip_finder_service.py — find_competitor_trip_url(), + score_and_rank_urls(), find_all_competitor_urls() + 18. Update jobs.py to import from core/ and industries/ + 19. Verify all existing functionality works identically after refactor + +Phase 3 — Dashboard (PRIMARY — build before Apps Script) + 20. Clone nuxt-ui-templates/dashboard-vue as frontend base + 21. Configure Vue Router: + /login — auth page + /account — subscription, seats, template download + /competitors — add/edit competitor list + /analyse — trip-intent form + confirmation + results (CORE PAGE) + /battlecards — battlecard view + /activity — market movement feed + 22. Pinia auth store — tenant_id, email, tier, JWT + 23. api_service.js — Axios instance with X-API-Key header + 24. LoginPage.vue — email/password + Google OAuth + 25. AccountPage.vue — subscription card, seats table, template download + 26. CompetitorsPage.vue — competitor list with status badges, add/edit modal + 27. AnalysePage.vue — THE CORE PAGE: + a. Trip-intent form (destination, duration, style, competitor checkboxes) + b. [Find & Analyse] button → POST /research/find-urls + c. URL confirmation screen — PM confirms or swaps matches + d. [Confirm & Run] → POST /research → poll status + e. Per-competitor progress display during job + f. Results view — side-by-side comparison table + g. Confidence indicators per competitor + h. [Push to Sheet ▾] button with tab selector dropdown + i. Run history panel — last 5 runs with [Restore] option + 28. Stripe integration — checkout, webhook, portal link + 29. Resend email templates — welcome, trial reminders, daily digest, weekly brief + 30. Freescout deploy — container + support.bettersight.io Pangolin route + + support@bettersight.io MX record + support link in dashboard footer + 31. useOnboardingTour.js — Shepherd.js 5-step guided tour + 32. tour.css — custom Shepherd styles matching Bettersight design system + 33. data-tour anchor attributes on nav items and key components + 34. PATCH /account/onboarding route + 35. Tour trigger on ActivityPage.vue onMounted + 36. Tour replay link on AccountPage.vue + +Phase 4 — Apps Script (export integration — simple) + 31. Deploy as standalone Apps Script + 32. Validation.gs — validateLicence() email-only + 33. Export.gs — pushToSheet(), getWorkbookTabs(), detectColumns() + 34. Sidebar.html — minimal: tab selector + push button only + 35. Sheet template tab — importable into any existing workbook + 36. Flask POST /sheet/push + GET /sheet/tabs routes + +Phase 5 — Automation + 37. n8n daily scrape cron — changed/stale competitors only + 38. PocketBase price_history diff logic (change_amount, change_percent) + 39. PocketBase is_new flag logic (first_seen detection) + 40. Flask /internal/monitor-webhook + 41. Flask /internal/monitor/register + /deregister + 42. Auto-register Firecrawl Monitor on competitor add + 43. n8n PocketBase webhook → create price_change alert record in PocketBase + 44. n8n PocketBase webhook → create new_product alert record in PocketBase + 45. n8n daily digest email (6pm) — collects unread alerts → Resend email + 46. n8n battlecard regeneration workflow (fires on scrape_runs complete) + 47. n8n Monday brief workflow (6am — Jinja2 HTML → Resend email) + 48. Dashboard bell icon + unread count from alerts table + 49. Mark as read / dismiss on AlertCard.vue + +Phase 6 — Semantic matching + 50. embedding_service.py — embed_trip(), find_comparable_trips() + 51. n8n embedding and matching workflows (Sunday night) + 52. comparable_matches table populated and visible in dashboard + +Phase 7 — Ship + 53. Flask-Limiter configured on all public routes + 54. Sentry initialised in Flask and Vue — staging first, then production + 55. GDPR deletion endpoint POST /account/delete tested end to end + 56. n8n weekly data cleanup workflow (retention policy) + 57. Empty states implemented on all seven dashboard pages (incl. ResetPasswordPage) + 58. Session expiry handling — token refresh interceptor + return-to logic + 59. In-progress job survival across session expiry + 60. Password reset — "Forgot password?" on LoginPage + ResetPasswordPage.vue + 61. Timezone captured on signup → stored in tenants.timezone + 62. n8n daily digest uses timezone-aware send time per tenant (luxon) + 63. Upgrade prompt on AccountPage — next tier card with [Upgrade now →] button + 64. POST /billing/upgrade route — Stripe checkout for tier upgrade + 65. Multi-currency display in ResultsTable — primary market hero + USD secondary + 66. Litestream container added to docker-compose.yml — replicating PocketBase to Hetzner S3 + 67. Hetzner Object Storage backup configured — rclone cron daily 2am (belt + braces with Litestream) + 68. robots.txt check added to core/scraper.py — check_robots_txt() before each scrape + 69. OpenRouter paid fallback configured — OPENROUTER_FALLBACK_MODEL env var set + 70. Scraping legal clause added to bettersight.io/terms + 71. Privacy policy and terms pages live at bettersight.io/privacy and /terms + 72. Google OAuth consent screen approved (requires privacy policy URL) + 73. End to end test with real competitor data + 74. Staging deploy + 75. Tag version v1.0.0 + 76. Production deploy via Dokploy +``` + +--- + +## 20. Authentication — PocketBase Native + +### Default: Email + Password +Standard PocketBase auth. User registers with email and password. +PocketBase issues JWT on login. Stored in Pinia auth store. +All Flask API calls include `Authorization: Bearer {jwt}` header. + +### Option: Google OAuth +PocketBase handles the full OAuth flow natively — no custom auth code. + +**Setup in PocketBase admin:** +1. Settings → Auth Providers → Google → Enable +2. Add Google OAuth Client ID and Secret from Google Cloud Console +3. Set redirect URL to `https://app.bettersight.io/auth/callback` + +**Frontend flow:** +```javascript +// LoginPage.vue — Google OAuth button +const loginWithGoogle = async () => { + const authData = await pb.collection('users').authWithOAuth2({ + provider: 'google' + }) + // PocketBase returns verified email — same email used for licence validation + authStore.setUser(authData) + router.push('/account') +} +``` + +**Why this matters for licence validation:** +Google OAuth email is verified by Google — same email registered in tenant_seats. +Domain matching is perfectly reliable. No manual email entry. No typos. +`Session.getActiveUser().getEmail()` in Apps Script returns the exact same email. + +### Login page design +- Email/password form as default (UForm + UInput + UButton) +- "Continue with Google" button below the form (UButton with Google icon) +- Trial messaging above the form — reads `?plan=` query param to show correct length: + ```javascript + // LoginPage.vue + const plan = route.query.plan || 'analyse' + const TRIAL_COPY = { + analyse: '14-day free trial — no credit card required', + discover: '21-day free trial — no credit card required', + intelligence: '30-day free trial — no credit card required', + } + const trialCopy = TRIAL_COPY[plan] || TRIAL_COPY.analyse + ``` +- "Forgot password?" link below the email input field +- Link to privacy policy and terms below + +### Password Reset Flow + +PocketBase handles password reset natively via email. +The frontend needs two small additions only. + +**LoginPage.vue — "Forgot password?" link:** +```javascript +// Triggers PocketBase native password reset email +async function requestPasswordReset() { + await pb.collection('users').requestPasswordReset(email.value) + toast.success('Reset link sent — check your inbox') +} +``` + +**`/reset-password` route — Vue Router:** +```javascript +// Reads token from URL params — PocketBase reset link format: +// app.bettersight.io/reset-password?token=xxx +async function confirmPasswordReset() { + await pb.collection('users').confirmPasswordReset( + route.query.token, + newPassword.value, + newPasswordConfirm.value + ) + router.push('/login?reset=success') +} +``` + +**Add to Vue Router:** +```javascript +{ path: '/reset-password', component: ResetPasswordPage } +``` + +**Add `ResetPasswordPage.vue` to pages:** +- New password input + confirm password input +- Submit button calls `confirmPasswordReset()` +- Redirect to login with success toast on completion +- Error state if token is expired or invalid + +--- + +## 21. Professional Onboarding and Brand Touches + +### Transactional Emails via Resend + +All system emails sent via Resend from `alerts@bettersight.io`. +Never use raw SMTP for user-facing emails. + +```python +import resend + +resend.api_key = os.getenv('RESEND_API_KEY') + +def send_welcome_email(to_email: str, tenant_name: str, template_url: str): + """ + Sends branded welcome email on signup with Sheet template download link. + Data flow: tenant email + name + template URL → Resend API → inbox + """ + resend.Emails.send({ + "from": "Bettersight ", + "to": to_email, + "subject": "Welcome to Bettersight — you're all set", + "html": render_email_template('welcome', { + "name": tenant_name, + "template_url": template_url + }) + }) +``` + +**Email triggers and templates:** + +| Trigger | Template | From | +|---|---|---| +| Signup complete | welcome.html | alerts@bettersight.io | +| Trial day 3 | checkin.html | alerts@bettersight.io | +| Trial day 7 | feature_tip.html | alerts@bettersight.io | +| Weekly brief | brief.html | brief@bettersight.io | +| Price alert | alert.html | alerts@bettersight.io | +| New product alert | alert.html | alerts@bettersight.io | +| Stripe receipt | Stripe native | billing@bettersight.io | + +All HTML email templates in `backend/templates/emails/`. +Every template includes: Bettersight logo, branded header, consistent +typography, unsubscribe footer, and link to bettersight.io. + +### Trial Periods + +Trial periods are configured per Stripe price object: + +```python +TRIAL_PERIODS = { + "analyse": 14, # 14 days — low price point, one person decides, value visible in first run + "discover": 21, # 21 days — slight committee risk at $499, needs second internal review + "intelligence": 30, # 30 days — $999 needs finance approval, signal pipeline needs time to prove value + "enterprise": 0, # No trial — custom demo and pilot conversation only +} +``` + +No credit card required to start any trial — configure in Stripe +checkout settings (payment_method_collection: 'if_required'). + +PocketBase tenant status flow: +``` +signup → status = 'trial' +day 12 (Analyse) / day 19 (Discover) / day 28 (Intelligence) + → n8n fires trial_reminder email (2 days before expiry) +trial expires → Stripe charges card (if provided) → status = 'active' +trial expires → no card provided → status = 'inactive' → access blocked +first Stripe charge succeeds → status = 'active' +``` + +Apps Script and dashboard allow access for both `trial` and `active` status: +```python +VALID_STATUSES = ['active', 'trial'] +``` + +### Stripe Customer Portal + +Self-service billing — clients manage their own subscription, upgrade, cancel. +No support tickets for billing changes. + +```python +@app.route('/billing/portal', methods=['POST']) +def billing_portal(): + """ + Creates a Stripe customer portal session for self-service billing. + Data flow: tenant_id → PocketBase stripe_customer_id → + Stripe portal session → redirect URL returned to frontend + """ + tenant = tenant_repository.get_by_id(request.json['tenant_id']) + session = stripe.billing_portal.Session.create( + customer=tenant['stripe_customer_id'], + return_url='https://app.bettersight.io/account' + ) + return jsonify({'url': session.url}) +``` + +### Onboarding Checklist + +Displayed on AccountPage.vue after signup. Tracked in PocketBase tenant record. +Updated automatically as each action is completed. Collapses when all steps done. + +``` +onboarding_checklist (JSON field on tenants) + { + "add_competitor": false, // set true when first competitor saved + "run_analysis": false, // set true when first job completes + "push_to_sheet": false, // set true when first push to Sheet runs + "download_template": false, // set true when template downloaded + "tour_completed": false, // set true when tour reaches final step + "tour_skipped": false, // set true when PM clicks Skip + "tour_step": 0 // last step reached — for resume on return + } +``` + +Each step checked off automatically when the action is completed. +Full checklist completion triggers the day-3 check-in email sequence. + +--- + +### Guided Onboarding Tour + +A 5-step guided tour fires automatically on first login using Shepherd.js. +It is the primary first-run experience — more active than the checklist, +which is passive. + +**Install:** +```bash +npm install shepherd.js +``` + +**Tour triggers:** +- Fires automatically on first dashboard load when + `tour_completed = false AND tour_skipped = false` +- "Take the tour again" link on AccountPage for PMs who skipped +- Never fires again after `tour_completed = true` +- If PM closes the browser mid-tour, `tour_step` stores their last position + and the tour resumes from that step on next login + +**Tour steps:** + +``` +Step 1 of 5 — Welcome (no anchor element — modal overlay) + ┌─────────────────────────────────────────────────┐ + │ 👋 Welcome to Bettersight │ + │ │ + │ Let's get you to your first competitive │ + │ analysis in under 5 minutes. │ + │ │ + │ [Skip tour] [Let's go →] │ + └─────────────────────────────────────────────────┘ + +Step 2 of 5 — Add a competitor (anchors to Competitors sidebar nav item) + ┌─────────────────────────────────────────────────┐ + │ 🔍 Add your first competitor │ + │ │ + │ Click Competitors in the sidebar to add the │ + │ first company you want to track. │ + │ │ + │ You'll need their website URL and their │ + │ primary operating market (e.g. United Kingdom).│ + │ │ + │ [← Back] [Go to Competitors →] │ + └─────────────────────────────────────────────────┘ + → On [Go to Competitors →]: router.push('/competitors') + → Tour pauses. Resumes when PM clicks [+ Add Competitor] + +Step 3 of 5 — Primary market (anchors to primary_market dropdown in add form) + ┌─────────────────────────────────────────────────┐ + │ 🌍 Set the primary market │ + │ │ + │ This tells Bettersight which country to │ + │ use when scraping — so you see the right │ + │ pricing and currency for that market. │ + │ │ + │ [← Back] [Next →] │ + └─────────────────────────────────────────────────┘ + +Step 4 of 5 — Run analysis (anchors to Analyse sidebar nav item) + ┌─────────────────────────────────────────────────┐ + │ ▶ Run your first analysis │ + │ │ + │ Describe the trip you want to research — │ + │ destination, duration, and travel style. │ + │ │ + │ Bettersight finds the matching competitor │ + │ pages automatically. No URLs to copy-paste. │ + │ │ + │ [← Back] [Go to Analyse →] │ + └─────────────────────────────────────────────────┘ + → On [Go to Analyse →]: router.push('/analyse') + +Step 5 of 5 — Push to Sheet (anchors to PushToSheet component) + ┌─────────────────────────────────────────────────┐ + │ 📊 Push results to your Sheet │ + │ │ + │ Once your analysis is complete, push the │ + │ results directly into your Google Sheet │ + │ with one click. Select which tab to write to. │ + │ │ + │ [← Back] [Done — let's go ✓] │ + └─────────────────────────────────────────────────┘ + → On [Done]: set tour_completed = true in PocketBase + → Tour dismissed permanently +``` + +**Implementation — `composables/useOnboardingTour.js`:** + +```javascript +import Shepherd from 'shepherd.js' +import 'shepherd.js/dist/css/shepherd.css' +import { useRouter } from 'vue-router' +import { useTenantStore } from '@/stores/tenant_store' + +/** + * Provides the guided onboarding tour using Shepherd.js. + * Fires on first login. Resumes from last step if interrupted. + * Marks tour_completed in PocketBase on final step. + * + * Data flow: + * onboarding_checklist.tour_completed → false → + * Shepherd.Tour initialised and started → + * PM progresses through 5 steps → + * final step → PATCH /account/onboarding { tour_completed: true } → + * tour never fires again + */ +export function useOnboardingTour() { + const router = useRouter() + const tenantStore = useTenantStore() + + const tour = new Shepherd.Tour({ + useModalOverlay: true, + defaultStepOptions: { + cancelIcon: { enabled: true }, + classes: 'bs-tour-step', + scrollTo: { behavior: 'smooth', block: 'center' }, + when: { + cancel: () => markTourSkipped() + } + } + }) + + tour.addStep({ + id: 'welcome', + text: ` + Welcome to Bettersight 👋

+ Let's get you to your first competitive analysis + in under 5 minutes. + `, + buttons: [ + { text: 'Skip tour', action: tour.cancel, + classes: 'bs-btn-ghost' }, + { text: "Let's go →", action: tour.next, + classes: 'bs-btn-primary' } + ] + }) + + tour.addStep({ + id: 'add-competitor', + attachTo: { element: '[data-tour="nav-competitors"]', + on: 'right' }, + text: ` + Add your first competitor 🔍

+ Click Competitors to add the first company you want to track. + You'll need their website URL and primary operating market. + `, + buttons: [ + { text: '← Back', action: tour.back, + classes: 'bs-btn-ghost' }, + { text: 'Go to Competitors →', + action: () => { + router.push('/competitors') + saveTourStep(2) + tour.next() + }, + classes: 'bs-btn-primary' } + ] + }) + + tour.addStep({ + id: 'primary-market', + attachTo: { element: '[data-tour="primary-market-field"]', + on: 'bottom' }, + text: ` + Set the primary market 🌍

+ This tells Bettersight which country to use when scraping + — so you see the right pricing and currency for that market. + `, + buttons: [ + { text: '← Back', action: tour.back, + classes: 'bs-btn-ghost' }, + { text: 'Next →', action: () => { + saveTourStep(3) + tour.next() + }, + classes: 'bs-btn-primary' } + ] + }) + + tour.addStep({ + id: 'run-analysis', + attachTo: { element: '[data-tour="nav-analyse"]', + on: 'right' }, + text: ` + Run your first analysis ▶

+ Describe the trip you want to research — destination, duration, + and travel style. Bettersight finds matching competitor pages + automatically. No URLs needed. + `, + buttons: [ + { text: '← Back', action: tour.back, + classes: 'bs-btn-ghost' }, + { text: 'Go to Analyse →', + action: () => { + router.push('/analyse') + saveTourStep(4) + tour.next() + }, + classes: 'bs-btn-primary' } + ] + }) + + tour.addStep({ + id: 'push-to-sheet', + attachTo: { element: '[data-tour="push-to-sheet"]', + on: 'top' }, + text: ` + Push results to your Sheet 📊

+ Once your analysis completes, push results directly into + your Google Sheet with one click. Select which tab to write to. + `, + buttons: [ + { text: '← Back', action: tour.back, + classes: 'bs-btn-ghost' }, + { text: 'Done — let\'s go ✓', + action: () => { + markTourCompleted() + tour.complete() + }, + classes: 'bs-btn-primary' } + ] + }) + + async function saveTourStep(step) { + await tenantStore.updateOnboarding({ tour_step: step }) + } + + async function markTourCompleted() { + await tenantStore.updateOnboarding({ + tour_completed: true, + tour_step: 5 + }) + } + + async function markTourSkipped() { + await tenantStore.updateOnboarding({ tour_skipped: true }) + } + + function shouldShowTour(onboarding) { + return !onboarding.tour_completed && !onboarding.tour_skipped + } + + function getResumeStep(onboarding) { + return onboarding.tour_step || 0 + } + + return { tour, shouldShowTour, getResumeStep } +} +``` + +**Trigger in `ActivityPage.vue` (first page after login):** + +```javascript +// ActivityPage.vue —