# CLAUDE.md — Bettersight Competitive Intelligence SaaS > This file is the authoritative specification for Claude Code building Bettersight. > Read every section before writing any code. Never skip a section. > When in doubt, re-read this file. --- ## 1. Project Overview Bettersight is a vertical competitive intelligence SaaS built specifically for adventure travel operators. It automates competitor price tracking, trip discovery, and market analysis — replacing hours of manual research with automated intelligence delivered to tools PMs already use. **Tagline:** Better sight, better moves. **Secondary tagline:** Better sight, better decisions. **Domain:** bettersight.io **Brand position:** The only competitive intelligence platform that understands adventure travel — not just that a page changed, but what trip was added, what price moved, and what it means for your catalogue. **The anchor feature** is an on-demand competitor analysis Google Sheet add-on that runs structured AI extraction against competitor trip pages and writes results directly into the client's spreadsheet. Everything else supports and extends this core. **Primary reference client:** G Adventures (hackathon origin) **Target market:** Adventure travel operators — product managers, revenue managers, strategy leads --- ## 2. Business Context ### Pricing Tiers | Tier | Price | Seats | Trial | |---|---|---|---| | Analyse | $349/month | 5 seats | 14 days — no credit card required | | Discover | $499/month | 8 seats | 21 days — no credit card required | | Intelligence | $999/month | 10 seats | 30 days — no credit card required | | Enterprise | $1,499/month | Unlimited | No trial — custom demo and pilot | Additional seats: $45/seat/month on any tier. Enterprise tier is not publicly listed — available on request only. The $199 Monitor tier has been dropped entirely. ### Tier Feature Summary **Analyse — $349/month (MVP launch tier)** - Dashboard as the primary interface — analysis runs, results viewed, push to Sheet - Trip-intent matching — PM enters destination + duration + style, system finds the right competitor trip URL automatically via Firecrawl /map (on demand) - URL confirmation screen before scrape runs — PM can swap any auto-matched URL - On-demand competitor analysis with side-by-side dashboard results view - Push to Google Sheet — one button exports results to PM's chosen workbook tab - Auto-generated battlecards updated after each scrape - Comparable trip discovery (semantic similarity matching) - In-dashboard notifications — price changes, new products, alerts in activity feed - Weekly Monday HTML email brief via Resend - Fast path (PocketBase cache) / refresh path (live scrape) on-demand analysis - Firecrawl /monitor for real-time change detection - Firecrawl /map on demand — for trip-intent URL matching per analysis job only **Discover — $499/month** - Everything in Analyse - Firecrawl /map scheduled — weekly full catalogue crawl per competitor → Proactive surveillance, not reactive search → Finds new trips the PM never knew to look for → New URLs auto-queued for scraping Sunday night without PM involvement - New trip detection — flags trips added since last crawl - Catalogue completeness — how many trips per competitor per destination - Destination gap analysis — destinations competitors cover that you don't - Sunday night discovery workflow (n8n) - Monday brief includes all auto-discovered new products - External notification integrations (PM connects one or more): → Slack webhook — real-time alerts into any channel → Microsoft Teams webhook — for Microsoft 365 organisations → WhatsApp via Twilio — alerts to PM's phone number **Intelligence — $999/month** - Everything in Discover - Full OSINT signal pipeline (15+ sources — see Section 23) - Competitor Trajectory Score (weekly) - Wayback Machine positioning history tracking - Network mapping (leadership, board, investor connections) - Monthly narrative intelligence report with predicted moves and confidence ratings - Early warning alerts on significant signal clusters **Enterprise — $1,499/month (unlisted)** - Everything in Intelligence - Dedicated onboarding and account management - Custom signal sources on request - Quarterly strategy briefing call - SLA on scrape freshness and alert delivery - Custom PDF branded reports ### Tier Pitch Distinction **Analyse** — "Tell me about this specific trip across these competitors" PM drives every research job. They describe their product, system finds the matching competitor trips, PM confirms and runs. Reactive intelligence. **Discover** — "Tell me everything that changed this week that I didn't know to ask about" System runs autonomously. PM wakes Monday to a brief that includes trips they never knew existed. Catches unknown unknowns. Proactive intelligence. **Intelligence** — "Tell me what my competitors are about to do" System aggregates signals across 15+ public sources and forecasts competitor moves 3-6 months ahead. Predictive intelligence. ### MVP Scope (build this first) Single tier at $349/month (Analyse). Five features only: 1. Dashboard — primary interface for running analysis and viewing results 2. Trip-intent matching — Firecrawl /map on demand finds competitor trip URLs 3. On-demand competitor analysis with dashboard results view + push to Sheet 4. Auto-generated battlecards 5. In-dashboard notifications (activity feed) + Monday HTML email brief via Resend Comparable trip discovery (semantic similarity) is a Phase 2 addition within the Analyse tier — build after core analysis flow is working. Do not build Discover, Intelligence, or Enterprise tier features until Analyse tier is live with paying clients. --- ## 3. Tech Stack — Authoritative **Never deviate from this stack without explicit instruction.** | Layer | Tool | Notes | |---|---|---| | Infrastructure | Hetzner VPS | Existing | | Orchestration | Dokploy | Existing | | Reverse proxy | Pangolin | Existing | | Networking | Docker Compose | pangolin_default network | | Backend API | Flask | Python only. No FastAPI. | | Scraping engine | app.py (Flask) | Existing — do not rewrite | | Job queue | RQ (Redis Queue) | Async job processing | | Queue broker | Redis | Shared with Firecrawl | | Trip-intent matching | Firecrawl self-hosted | /v1/map on demand — finds competitor trip URLs (Analyse+) | | Discovery | Firecrawl self-hosted | /v1/map scheduled — weekly catalogue crawl (Discover+ only) | | Change detection | Content-hash diff in n8n cron | SHA-256 comparison per daily scrape — no external service | | Proxy — primary | Webshare | ~$3/month, free tier available, used by default | | Proxy — fallback | Bright Data | Pay-as-you-go, auto-escalated per domain when Webshare blocked | | AI extraction | OpenRouter | Free tier model cycling with claude-haiku-4-5 paid fallback | | Embeddings | text-embedding-3-small | Via OpenRouter | | Database | PocketBase | Multi-tenant + auth | | DB replication | Litestream | Real-time SQLite WAL streaming to Hetzner S3 — near-zero RPO | | Auth | PocketBase native | Email/password (default) + Google OAuth | | API rate limiting | Flask-Limiter | Protects against queue abuse and runaway scripts | | Workflow automation | n8n | Self-hosted | | Frontend | Vite + Vue 3 | Minimal dashboard only — no Nuxt runtime | | UI component library | Nuxt UI v4 (free, MIT) | Same library as Ledra/AccountFlow | | Base template | nuxt-ui-templates/dashboard-vue | Clone as starting point | | Routing | Vue Router v4 | Client-side SPA routing | | State management | Pinia | Lightweight store | | Date handling | date-fns | Formatting and computation | | Validation | Zod | Schema validation | | Sheet add-on | Google Apps Script | Export integration only — push results from dashboard to Sheet | | User notifications | In-dashboard activity feed | MVP — price changes, new products, alerts | | User notifications (future) | Slack / Teams / WhatsApp | Discover tier — webhook integrations | | Owner monitoring alerts | Gotify + iGotify | Self-hosted — owner phone only, never shown to clients | | Weekly brief | Resend HTML email | Replaces WeasyPrint — no PDF, no Docker dependency | | Payments | Stripe | Subscription management + customer portal | | Support tickets | Freescout (self-hosted) | Email-to-ticket — support@bettersight.io → Freescout | --- ## 4. Architecture ### Service Map ``` Bettersight Dashboard — PRIMARY INTERFACE (app.bettersight.io) → PM enters: destination + duration + travel style + selected competitors → Flask POST /research/find-urls → Firecrawl /map per competitor (on demand, <5s each) → URL scoring returns best match per competitor → Confirmation screen — PM confirms or swaps any URL → PM confirms → Flask POST /research → RQ job queued → job_id returned → Dashboard polls /research/status/{job_id} every 3 seconds → On-demand check flow per competitor: Path 1 FAST — last_scraped < 24hrs AND change_detected = false → PocketBase structured data → LLM analysis narrative (~seconds) Path 2 REFRESH — last_scraped > 24hrs OR change_detected = true → RQ job → app.py Playwright scrape → PocketBase → LLM (~2 mins) → Results displayed in dashboard side-by-side comparison view → PM copies rows to clipboard, or opens Sheet + Bettersight sidebar to pull Apps Script (standalone — Sheet-initiated pull integration) → getWorkbookTabs() — returns tab list for the sidebar's own selector → pullLatestResults(tabName) — Sheet-initiated fetch from Flask + write → validateLicence() — confirms PM is authorised → Three functions only. Flask never pushes to the Sheet. Template setup (one-time per PM, via AccountPage.vue) → PM opens hosted template Sheet from AccountPage → PM copies Bettersight Analysis tab into their existing workbook via Google Sheets "Copy to → Existing spreadsheet" → PM optionally renames the imported tab to anything they want → PM installs standalone Apps Script (binds to Google account, not workbook) → From that point: every workbook the PM opens has Extensions → Bettersight available, and pullLatestResults() writes into whichever workbook and whichever tab is selected in the sidebar dropdown Change detection (via daily cron content-hash comparison) → n8n daily scrape cron computes SHA-256 of normalised page content → Compares against previous scrape's content_hash in scrape_runs → If changed: sets competitor.change_detected = true, records new hash → If unchanged: skips LLM extraction (fast path knows to use cache) → Rationale: replaces Firecrawl Monitor which self-hosted requires credit accounting + Supabase and only runs on cron anyway. Daily granularity is appropriate for adventure travel pricing cycles. n8n (scheduled workflows) → Daily cron (6am) — scrapes changed/stale competitors → Resets change_detected = false after each scrape → PocketBase webhook on price_history insert → creates alert record in PocketBase → PocketBase webhook on products insert (is_new=true) → creates alert record in PocketBase → n8n reads new alert records → sends Resend email notification (daily digest) → Monday 6am → brief_service.py → Resend HTML email → [Discover tier] Sunday night → Firecrawl /map scheduled → full catalogue crawl Vue 3 dashboard auth + account → PocketBase auth (Google OAuth or email/password) → Stripe (payment, subscription status) ``` ### Change Detection — Content-Hash Comparison Change detection is computed in-process by the scrape service itself, during the daily scrape cron — not by an external monitoring service. This replaces both changedetection.io (previously removed) and Firecrawl Monitor (evaluated and rejected — see Section 4's Compose Services note and Section 27 "What NOT to Build"). **Why content-hash diff over an external monitor:** - No dependency on Firecrawl's credit-metered, cron-scheduled v2 Monitor feature, which requires Supabase auth even in self-hosted mode - Runs as a normal step of the scrape job already being performed daily - Renders JS fully before hashing (same Playwright render used for extraction) — works on React/Vue SPAs - Daily granularity matches adventure travel pricing update cadence **PocketBase competitor record flags:** ``` change_detected bool — set true when the scrape service detects a hash diff, reset after next scrape change_detected_at date — timestamp of last detected change last_scraped date — timestamp of last completed scrape ``` See Section 10 "Content-hash change detection" for `compute_content_hash()` and `detect_change()` — the functions that implement this during `jobs.py`'s scrape job. ### Docker Network All services on `pangolin_default` network. Internal service calls use container names, never localhost or host IPs. ``` flask-api → http://flask-api:5000 bettersight-worker → connects to redis://bettersight-redis:6379 bettersight-redis → redis://bettersight-redis:6379 firecrawl → http://firecrawl:3002 pocketbase → http://pocketbase:8090 ``` ### Compose Services ```yaml services: flask-api: build: ./backend container_name: flask-api command: gunicorn app:app -w 4 -b 0.0.0.0:5000 env_file: .env networks: - pangolin_default restart: unless-stopped depends_on: - bettersight-redis bettersight-worker: build: ./backend container_name: bettersight-worker command: rq worker high normal low -u redis://bettersight-redis:6379 env_file: .env networks: - pangolin_default restart: unless-stopped deploy: replicas: 3 depends_on: - bettersight-redis bettersight-redis: image: redis:alpine container_name: bettersight-redis networks: - pangolin_default restart: unless-stopped ``` ### Firecrawl — real self-hosted stack Firecrawl self-hosted is a multi-service application, not a single container. It needs api + playwright-service + a NUQ Postgres + a RabbitMQ, and the api service shares the existing `bettersight-redis` container for rate limiting. No Supabase — that's only required if you enable the credit/auth system, which Bettersight does not use (`USE_DB_AUTHENTICATION=false`). There is no separate worker container. Checked upstream's own `docker-compose.yaml` — it defines exactly one `api` service, command `node dist/src/harness.js --start-docker`. That harness process spins up the queue workers internally (`NUQ_WORKER_COUNT` etc — see `apps/api/src/harness.ts`) rather than as a separate process. An earlier version of this section documented a second `firecrawl-worker` container running `pnpm run workers`, based on an older split-process Firecrawl architecture — against the current codebase that container crash-loops immediately: the runtime image's final stage only copies `node_modules`/`dist`/`native` (see `apps/api/Dockerfile`'s runtime stage), not `package.json`, so pnpm has no manifest to resolve `workers` against. Do not reintroduce a second container here. Firecrawl's queue backend (NUQ) requires a real Postgres, not just Redis — this only surfaces once the api container actually starts: without `NUQ_DATABASE_URL` set, `harness.ts`'s `setupNuqPostgres()` tries to auto-provision a Postgres container by shelling out to `docker`/`podman` from *inside* the api container itself, which has neither installed (and isn't meant to — no Docker-in-Docker). Firecrawl ships a purpose-built Postgres image for this at `apps/nuq-postgres/` — `pg_cron` preloaded plus the NUQ schema (`nuq.sql`) baked in as an initdb script — build that rather than pointing at a vanilla postgres image, which would boot without the schema NUQ expects. Setting `POSTGRES_HOST` to that container's name (anything other than `localhost`) is what flips `setupNuqPostgres()` into its docker-compose branch, which just builds the connection URL directly and skips the Docker/Podman auto-provision path entirely — this is the code path Firecrawl's own authors intended for exactly this setup, not a workaround. `NUQ_RABBITMQ_URL` is required, not optional, despite most call sites in the API gracefully checking `if (config.NUQ_RABBITMQ_URL)` and falling back to polling mode when absent. The harness always spawns an `extract-worker` subprocess (`dist/src/services/extract-worker.js`) regardless of which routes are actually used, and that subprocess's `consumeExtractJobs()` throws immediately without it — no config check gates whether it starts at all. The harness treats any subprocess exit as fatal to the whole process, so without RabbitMQ the entire api container crash-loops every ~2s even though Bettersight never calls `/extract`. Upstream's own docker-compose.yaml confirms this — it lists `rabbitmq` as a hard `depends_on: condition: service_healthy` for the api service. Use the stock `rabbitmq:3-management` image — no build needed. Build context must be the app subdirectory itself (`apps/api`, `apps/playwright-service-ts`, `apps/nuq-postgres`), not the repo root — each Dockerfile's COPY instructions (e.g. `apps/api/Dockerfile`'s `COPY sharedLibs/go-html-to-md ...`) are written relative to that subdirectory, matching upstream firecrawl's own docker-compose.yaml (`build: apps/api`). Pointing context at the repo root and `dockerfile:` at the nested path resolves the Dockerfile correctly but still resolves every COPY inside it against the repo root instead — breaks on the first COPY that assumes otherwise. Confirmed against a real build attempt: `context: ` + `dockerfile: apps/api/Dockerfile` fails with `"/sharedLibs/go-html-to-md": not found`; `context: /apps/api` builds correctly. For a remote git context, Docker's syntax puts the subdirectory after a colon in the fragment (`#branch:subdirectory`), not as a separate `dockerfile:` field: ```yaml firecrawl-rabbitmq: image: rabbitmq:3-management container_name: firecrawl-rabbitmq networks: - pangolin_default restart: unless-stopped firecrawl-nuq-postgres: build: context: https://github.com/firecrawl/firecrawl.git#main:apps/nuq-postgres container_name: firecrawl-nuq-postgres environment: - POSTGRES_USER=postgres - POSTGRES_PASSWORD=postgres - POSTGRES_DB=postgres networks: - pangolin_default restart: unless-stopped firecrawl-api: build: context: https://github.com/firecrawl/firecrawl.git#main:apps/api container_name: firecrawl-api environment: - PORT=3002 - HOST=0.0.0.0 - REDIS_URL=redis://bettersight-redis:6379 - REDIS_RATE_LIMIT_URL=redis://bettersight-redis:6379 - PLAYWRIGHT_MICROSERVICE_URL=http://firecrawl-playwright:3000/scrape - USE_DB_AUTHENTICATION=false - NUM_WORKERS_PER_QUEUE=8 - POSTGRES_HOST=firecrawl-nuq-postgres - POSTGRES_PORT=5432 - POSTGRES_USER=postgres - POSTGRES_PASSWORD=postgres - POSTGRES_DB=postgres - NUQ_RABBITMQ_URL=amqp://firecrawl-rabbitmq:5672 depends_on: - bettersight-redis - firecrawl-playwright - firecrawl-nuq-postgres - firecrawl-rabbitmq networks: - pangolin_default restart: unless-stopped firecrawl-playwright: build: context: https://github.com/firecrawl/firecrawl.git#main:apps/playwright-service-ts container_name: firecrawl-playwright environment: - PORT=3000 networks: - pangolin_default restart: unless-stopped mem_limit: 4g ``` Resource budget: the playwright-service alone wants ~2-4GB RAM. The CX31 VPS (2 vCPU, 8GB RAM) sized in Section 31 handles this alongside PocketBase, Flask, and 3 RQ workers because Firecrawl is only used for /v1/map calls (trip-intent matching + weekly discovery), never continuously. Only the api service is called by Bettersight — at `http://firecrawl-api:3002/v1/map`. playwright-service, nuq-postgres, and rabbitmq are internal-only. ```yaml freescout: image: freescout/freescout:latest container_name: freescout environment: - APP_URL=https://support.bettersight.io - MAIL_FROM=support@bettersight.io networks: - pangolin_default restart: unless-stopped # Pangolin routes support.bettersight.io → freescout:80 litestream: image: litestream/litestream:latest container_name: litestream command: replicate volumes: - pocketbase_data:/pb_data # same volume as PocketBase environment: - LITESTREAM_ACCESS_KEY_ID=${HETZNER_S3_ACCESS_KEY} - LITESTREAM_SECRET_ACCESS_KEY=${HETZNER_S3_SECRET_KEY} configs: - source: litestream_config target: /etc/litestream.yml networks: - pangolin_default restart: unless-stopped depends_on: - pocketbase # litestream.yml config: # dbs: # - path: /pb_data/data.db # replicas: # - url: s3://bettersight-backups/litestream # endpoint: https://fsn1.your-objectstorage.com networks: pangolin_default: external: true volumes: pocketbase_data: ``` **Litestream recovery procedure:** ```bash # Restore PocketBase database from Litestream replica docker run --rm \ -e LITESTREAM_ACCESS_KEY_ID=${HETZNER_S3_ACCESS_KEY} \ -e LITESTREAM_SECRET_ACCESS_KEY=${HETZNER_S3_SECRET_KEY} \ -v ./pb_data:/pb_data \ litestream/litestream restore \ -o /pb_data/data.db \ s3://bettersight-backups/litestream ``` Recovery point objective (RPO): seconds — not 24 hours. Recovery time objective (RTO): under 5 minutes from backup restore to running. --- ## 5. Project Structure ``` bettersight/ ├── backend/ │ ├── core/ │ │ ├── scraper.py # Playwright stealth browser — never touches industry logic │ │ ├── extractor.py # OpenRouter model cycling, JSON parsing, fallback chain │ │ ├── ai_fetch.py # AI web fetch fallback │ │ └── proxy_service.py # Proxy provider — swap here to change from Bright Data │ ├── industries/ │ │ └── adventure_travel/ │ │ ├── fields.py # STANDARD_FIELDS, NGS_FIELDS column maps │ │ ├── prompt.py # build_prompt(), inject_page_content() │ │ └── schema.py # JSON extraction schema for this vertical │ ├── config/ │ │ └── client.py # SHEET_ID, tab names, column assignments │ ├── services/ │ │ ├── licence_service.py # Domain + seat validation (no sheet_id) │ │ ├── trip_finder_service.py # find_competitor_trip_url(), score_and_rank_urls() │ │ ├── battlecard_service.py │ │ ├── embedding_service.py │ │ ├── brief_service.py # HTML email brief generation │ │ └── alert_service.py │ ├── repositories/ │ │ ├── tenant_repository.py │ │ ├── product_repository.py │ │ ├── seat_repository.py │ │ ├── scrape_repository.py │ │ └── proxy_repository.py │ ├── templates/ │ │ └── emails/ │ │ ├── welcome.html │ │ ├── trial_reminder.html │ │ ├── brief.html │ │ ├── price_alert.html │ │ └── new_product.html │ ├── api.py # Flask routes │ ├── jobs.py # RQ job functions │ ├── app.py # Entry point — imports and wires everything │ ├── requirements.txt │ └── Dockerfile ├── frontend/ # Vite + Vue 3 + Nuxt UI v4 dashboard │ ├── src/ │ │ ├── pages/ │ │ │ ├── LoginPage.vue # Auth — email/password + Google OAuth + forgot password │ │ │ ├── ResetPasswordPage.vue # /reset-password?token=xxx — PocketBase token confirm │ │ │ ├── AccountPage.vue # Subscription, seats, upgrade prompt, template download │ │ │ ├── CompetitorsPage.vue # Add/edit competitors with status badges + CSV upload │ │ │ ├── AnalysePage.vue # CORE PAGE — full analysis workflow │ │ │ ├── ActivityPage.vue # Market movement feed + alerts │ │ │ └── BattlecardsPage.vue # Battlecard view per competitor │ │ ├── components/ │ │ │ ├── DotGrid.vue # Reusable dot grid SVG — signature element │ │ │ ├── analyse/ │ │ │ │ ├── TripIntentForm.vue # Destination + duration + style + competitor checkboxes │ │ │ │ ├── UrlConfirmation.vue # Confirm or swap matched URLs before run │ │ │ │ ├── JobProgress.vue # Per-competitor real-time status during job │ │ │ │ ├── ResultsTable.vue # Side-by-side competitor comparison │ │ │ │ ├── ConfidenceBar.vue # Per-competitor confidence indicator │ │ │ │ ├── SyncToSheet.vue # Copy rows button + pull instruction card │ │ │ │ └── RunHistory.vue # Last 5 runs with restore option │ │ │ ├── activity/ │ │ │ │ ├── FeedItem.vue # Single activity feed row with dot grid │ │ │ │ └── AlertCard.vue # Alert card with dot grid │ │ │ ├── shared/ │ │ │ │ ├── StatCard.vue # Stat card with accent bar + dot grid │ │ │ │ ├── SeatManager.vue # Add/remove seats table │ │ │ │ ├── CompetitorForm.vue # Add/edit competitor modal form │ │ │ │ ├── BattlecardCard.vue # Single battlecard display │ │ │ │ ├── SubscriptionCard.vue │ │ │ │ ├── CsvUpload.vue # File picker + PapaParse + row validation │ │ │ │ └── ImportPreviewModal.vue # Valid/invalid preview before bulk submit │ │ ├── composables/ │ │ │ ├── useAuth.js │ │ │ ├── useTenant.js │ │ │ ├── useAnalysis.js # Analysis job lifecycle management │ │ │ ├── useCompetitors.js │ │ │ └── useOnboardingTour.js # Shepherd.js guided tour — 5 steps │ │ ├── assets/ │ │ │ └── tour.css # Custom Shepherd.js styles matching design system │ │ ├── services/ │ │ │ └── api_service.js # All Flask API calls │ │ ├── repositories/ │ │ │ ├── tenant_repository.js │ │ │ ├── seat_repository.js │ │ │ ├── competitor_repository.js │ │ │ └── analysis_repository.js # Research jobs, history, results │ │ ├── stores/ │ │ │ ├── auth_store.js # Pinia — JWT, email, tier │ │ │ └── analysis_store.js # Pinia — active job, results, history │ │ ├── router/ │ │ │ └── index.js │ │ └── main.js │ ├── package.json │ ├── vite.config.js │ └── index.html ├── appscript/ # Sheet-initiated pull integration — 3 files │ ├── Code.gs # onOpen(), install entry point │ ├── Validation.gs # validateLicence() — email only │ ├── Sync.gs # pullLatestResults(), getWorkbookTabs(), detectColumns() │ └── Sidebar.html # Tab selector + pull button + last-pull status ├── n8n/ │ └── workflows/ # n8n workflow JSON exports ├── docker-compose.yml ├── .env.example └── CLAUDE.md ``` --- ## 5b. Frontend Setup — Nuxt UI v4 ### Bootstrap ```bash # Clone the dashboard-vue base template git clone https://github.com/nuxt-ui-templates/dashboard-vue frontend cd frontend npm install # Additional dependencies npm install pinia date-fns zod axios ``` ### Key dependencies (package.json) ```json { "dependencies": { "@nuxt/ui": "^4.0.0", "vue": "^3.4.0", "vue-router": "^4.0.0", "pinia": "^2.0.0", "date-fns": "^3.0.0", "zod": "^3.0.0", "axios": "^1.6.0" }, "devDependencies": { "vite": "^5.0.0", "@vitejs/plugin-vue": "^5.0.0" } } ``` ### Nuxt UI v4 component usage Use Nuxt UI components throughout — never build custom UI primitives. Key components for this dashboard: ``` UButton — all buttons UInput — all text inputs UCard — content containers UTable — seat management, competitor list UBadge — subscription status, tier labels UModal — add competitor, add seat dialogs UForm — all forms with Zod validation UAlert — error and success messages UAvatar — user display UDropdown — action menus UDivider — section separators UContainer — page layout wrapper USkeleton — loading states UNotification — toast notifications ``` ### Design tokens — align with Ledra/AccountFlow ```javascript // app.config.ts or equivalent export default { ui: { primary: 'indigo', gray: 'cool' } } ``` ### Separation of concerns — frontend ``` Pages (src/pages/) → orchestrate composables and repositories → NO direct API calls → NO business logic Composables (src/composables/) → reusable stateful logic → calls repositories → NO direct API calls Repositories (src/repositories/) → all API calls via api_service.js → returns plain data objects → NO component or store references Stores (src/stores/) → global state only (auth, tenant) → NO API calls directly → calls repositories for data Services (src/services/) → api_service.js is the single Axios instance → all endpoints configured here → NO business logic ``` ### Six pages — full dashboard **LoginPage.vue** — email/password default + Google OAuth option **AccountPage.vue** - Subscription status card (tier, renewal date, seat count, trial countdown) - **Upgrade prompt card** — shown below subscription for Analyse and Discover tier clients: ``` ┌────────────────────────────────────────────────┐ │ 🚀 Upgrade to Discover │ │ │ │ + Weekly catalogue auto-discovery │ │ + Spot new competitor trips automatically │ │ + Slack, Teams and WhatsApp alerts │ │ │ │ $499/month · 3 more seats · 21-day trial │ │ │ │ [Upgrade now →] │ └────────────────────────────────────────────────┘ ``` → [Upgrade now] creates a Stripe checkout session for the next tier → On successful upgrade: PocketBase tier updated via Stripe webhook → Upgrade card hidden for Intelligence and Enterprise clients - Template import section — three items: 1. [Open template ↗] button that opens the hosted template Sheet in a new tab (GET /account/template returns the Drive URL) 2. Two-step import instruction card: ``` Import the Bettersight tab into your workbook: 1. Right-click any tab in the template → Copy to → Existing spreadsheet → select your workbook 2. Rename the tab to anything you want (optional) 3. Install the Bettersight sidebar (one click): [Install sidebar ↗] ``` 3. Verification helper — after import, PM can paste their workbook URL into an "It worked" confirmation input which marks onboarding_checklist.template_imported = true and dismisses this section. Purely a completion signal — no data leaves the browser. - Seat management table (add/remove seats) - Stripe customer portal link (manage billing, cancel) - Onboarding checklist (tracked in PocketBase) - "↺ Take the guided tour again" link **CompetitorsPage.vue** - Table of active competitors with status badges (✓ Current, ⚠ Changed, ○ Stale) - Primary market flag per competitor - Add competitor modal — name, website, catalogue URL, primary market (required) - Edit/deactivate actions - [Download CSV template] button — pre-formatted with correct headers - [Upload CSV] button — bulk import with preview and validation **AnalysePage.vue — THE CORE PAGE** - TripIntentForm — destination, duration, travel style, competitor checkboxes - [Find & Analyse] → POST /research/find-urls → UrlConfirmation screen - UrlConfirmation — matched URL per competitor, swap any URL, manual entry for no-match - [Confirm & Run] → POST /research → job_id → polling begins - JobProgress — per-competitor real-time status (queued / scraping / done / failed) - ResultsTable — side-by-side competitor comparison on completion Multi-currency display rule: → Show competitor's primary market currency as hero price (GB competitor = GBP hero, US competitor = USD hero) → Always show USD alongside in secondary muted text → Format: "£2,190 $2,780 USD" → Never currency-convert — use extracted values only → If primary market currency not extracted, fall back to USD only - ConfidenceBar — per-competitor AI confidence with low-confidence warning - RunHistory — last 5 runs, restore any run to current results view - SyncToSheet — post-analysis affordance replacing the removed PushToSheet: → [Copy rows] button — puts tab-separated results on the clipboard → Instruction card below: "Your results are saved. To pull them into your Sheet: 1) Open your workbook 2) Extensions → Bettersight → Pull latest results 3) Select the tab to write to." → No tab selector on this page — tab selection lives inside the sidebar where the workbook is actually open **ActivityPage.vue** - Stat cards (price changes, drops, new products, competitors tracked) - Market movement feed — all competitor events this week with dot grid rows - Alerts panel — unread alerts with badge count in sidebar nav - Price position progress bars - Mark as read / dismiss per alert - Bell icon in topbar shows unread count **BattlecardsPage.vue** - Battlecard per competitor — positioning summary, strengths, weaknesses, pricing observation - Last updated timestamp - Topic pills (destinations, warnings, signals) - Comparable trips section (semantic matches) ### tenants ``` id auto name text email text (admin email from signup) domain text (e.g. gadventures.com — used for seat validation) tier select [analyse, discover, intelligence, enterprise] status select [trial, active, inactive] trial_ends_at date (set on signup based on tier trial period) max_seats integer (set from TIER_SEATS on signup) timezone text (IANA timezone e.g. Europe/London — captured on signup) stripe_customer_id text stripe_subscription_id text onboarding_checklist json (tracks completion of onboarding steps) data_region text (default: 'global' — for future GDPR regional routing) created autodate ``` Note: `sheet_id` binding removed — standalone Apps Script model, validation is email-only. `monitor` tier removed — dropped entirely. **Timezone capture on first login (not signup — see note):** There is no signup form in the dashboard's scope — account creation happens via the Stripe `checkout.session.completed` webhook or the manual onboarding runbook (§32), neither of which runs browser JS. First login is the earliest point `Intl.DateTimeFormat` can actually run, so that's where this happens instead: `auth_store.js`'s `fetchTenant()` checks `tenant.timezone` after every `GET /account/me`, and if it's still empty, captures the browser's timezone and sends it to `PATCH /account/timezone`. Never asks the PM to select one manually. `PATCH /account/timezone` never overwrites an already-captured value — first write wins. ```javascript // auth_store.js — fires once per tenant, on whichever login is first const timezone = Intl.DateTimeFormat().resolvedOptions().timeZone // e.g. "Europe/London", "America/New_York", "Australia/Sydney" ``` ### tenant_seats ``` id auto tenant_id relation → tenants email text active bool last_accessed date added_by text created autodate ``` ### competitors ``` id auto tenant_id relation → tenants name text website url catalogue_url url primary_market text (ISO country code e.g. GB, US, AU — required on add) change_detected bool (set true by scrape service on content-hash diff, reset after next scrape) change_detected_at date (timestamp of last detected change) active bool last_scraped date created autodate ``` ### products ``` id auto tenant_id relation → tenants competitor_id relation → competitors trip_name text url url destination text duration_days integer majority_price_usd number price_low_usd number price_high_usd number aud_price number (nullable) cad_price number (nullable) eur_price number (nullable) gbp_price number (nullable) date_used text seasonality text group_size integer meals integer service_level text activities text start_location text end_location text target_audience text hotels text comments text relevancy integer trip_code text is_new bool first_seen autodate last_seen date is_active bool embedding json (vector array for similarity matching) ``` ### products_ngs_meta ``` id auto product_id relation → products exclusive_access text group_leader text sustainability text ``` ### price_history ``` id auto tenant_id relation → tenants product_id relation → products majority_price_usd number price_low_usd number price_high_usd number aud_price number (nullable) cad_price number (nullable) eur_price number (nullable) gbp_price number (nullable) date_used text change_field text (which price field changed) change_amount number change_percent number scraped_at autodate ``` ### scrape_runs ``` id auto (also used as RQ job_id) tenant_id relation → tenants competitor_id relation → competitors (nullable for bulk runs) triggered_by select [cron, webhook, manual, sheet] status select [pending, running, complete, failed] competitors_total integer competitors_done integer results json error_log text created_by_email text (the seat/PM who submitted the run — set once at creation from the dashboard's auth session, api.py's research() route; not persisted for cron-triggered rows. Shown in RunHistory.vue's "last 5 runs" list.) started_at autodate completed_at date ``` ### scrape_runs (additions for content-hash detection) Add these fields to the existing scrape_runs collection: ``` content_hash text # SHA-256 of normalised page content content_length integer # raw byte length of normalised content hash_algorithm text # "sha256" — allows future migration ``` Indexed via a composite index — `(competitor_id, content_hash, started_at)` — rather than on content_hash alone, since every lookup is scoped to one competitor and sorted by recency (`get_last_hash_for_url()`). An index on content_hash by itself wouldn't serve that query. The `competitors.change_detected` and `competitors.change_detected_at` fields already exist — no schema change needed for them. Semantics change from "set by Firecrawl webhook" to "set by scrape service on hash diff." No repository or service signature changes. ### alerts ``` id auto tenant_id relation → tenants competitor_id relation → competitors product_id relation → products (nullable) alert_type select [price_change, new_product, product_removed, new_comparable] message text change_amount number (nullable) change_percent number (nullable) delivered_dashboard bool (always true — alert always appears in dashboard) delivered_email bool (set true when included in daily digest email) delivered_slack bool (Discover tier — set true when Slack webhook fires) delivered_teams bool (Discover tier — set true when Teams webhook fires) delivered_whatsapp bool (Discover tier — set true when WhatsApp fires) read bool (PM marks as read in dashboard) created autodate ``` ### battlecards ``` id auto tenant_id relation → tenants competitor_id relation → competitors content text (AI generated battlecard text) content_json json (structured: top_routes, avg_price, positioning, strengths, weaknesses) generated_at autodate ``` ### comparable_matches ``` id auto tenant_id relation → tenants client_product text (client trip name — not a PocketBase relation) competitor_product relation → products similarity_score number price_difference number duration_difference integer first_matched autodate last_matched date dismissed bool ``` ### tenant_notifications ``` id auto tenant_id relation → tenants slack_webhook_url text (nullable — Discover tier only) teams_webhook_url text (nullable — Discover tier only) whatsapp_number text (nullable — Discover tier only) email_digest bool (default true — daily digest email) email_digest_time text (default "18:00" — local time in tenant timezone) created autodate ``` **Timezone-aware digest scheduling in n8n:** ``` Daily digest n8n workflow: → Query all active tenants with email_digest = true → For each tenant: Convert 18:00 tenant local time → UTC using tenants.timezone If current UTC time matches → send digest for this tenant → Each tenant receives their digest at 6pm their local time ``` n8n Code node timezone conversion: ```javascript // n8n Code node — per tenant digest time calculation const { DateTime } = require('luxon') const localTime = '18:00' const tenantTz = tenant.timezone || 'UTC' const sendAt = DateTime.fromFormat(localTime, 'HH:mm', { zone: tenantTz }) .toUTC() // Compare sendAt.hour and sendAt.minute against current UTC time ``` Implemented in `n8n/workflows/daily_digest.json` — runs hourly (not once daily at a fixed UTC hour), comparing only `.hour` since `email_digest_time` is minute-granular but the schedule itself isn't. `email_digest`'s "default true" isn't a PocketBase-level field default (the migration never configured one) — it's handled in the Code node instead: a tenant with no `tenant_notifications` row at all is treated as enabled at the default 18:00, since nothing in this codebase creates that row yet. If something later creates a `tenant_notifications` row without explicitly setting `email_digest`, PocketBase's bool zero-value (`false`) would silently disable digests for that tenant — always set it explicitly on creation. ### digest_logs ``` id auto tenant_id relation → tenants digest_type select [weekly, monthly] pdf_path text sent_at autodate recipient_email text status select [sent, failed] ``` ### proxy_overrides ``` id auto domain text (e.g. intrepidtravel.com) provider select [webshare, brightdata] blocked_count integer last_blocked date created autodate ``` ### client_trips ``` id auto tenant_id relation → tenants trip_name text destination text duration_days integer price_usd number activities text embedding json created autodate ``` --- ## 7. On-Demand Analysis — Check Flow This is the core data flow for every on-demand analysis request from the Sheet. It determines whether to serve from PocketBase (fast) or trigger a live scrape (slow). The PM always gets fresh, relevant data. The system decides the path invisibly. ``` PM triggers analysis in Sheet sidebar │ ▼ For each selected competitor: │ ├── Check PocketBase competitor record │ last_scraped < 24hrs? AND change_detected = false? │ │ │ ├── YES → FAST PATH (~seconds) │ │ Pull structured data from PocketBase products table │ │ Pass JSON to LLM for analysis narrative only (no extraction) │ │ Write results to Sheet │ │ Show: ✅ Current (updated {time}) │ │ │ └── NO → REFRESH PATH (~2 mins) │ last_scraped > 24hrs OR change_detected = true │ Trigger RQ job → Playwright scrape → AI extraction │ Update PocketBase (products, price_history) │ Reset change_detected = false │ Update last_scraped = now │ Pass fresh data to LLM for analysis narrative │ Write results to Sheet │ Show: 🔄 Refreshed just now │ ▼ Sheet sidebar status indicators: ✅ Current (updated today 6:02am) — fast path served ⚠️ Change detected (2hrs ago) — triggers refresh automatically 🔄 Refreshing... — slow path in progress ✓ Complete — results written to Sheet ``` ### Fast path — LLM role In the fast path the LLM does NOT extract data — it analyses it. PocketBase already has structured JSON. The LLM receives that JSON and generates the analysis narrative, comments, and battlecard content only. The returned dict must still carry the cached trip/pricing fields (tripName, majorityPrice, duration, gbpPrice, etc.) alongside the fresh narrative — the LLM call itself is constrained to return only `{comments, relevancy}`, and those two get merged into the cached fields, not returned as the entire result. A real "complete" job with an empty-looking dashboard results table surfaced why this matters: an earlier version returned the LLM's raw output as the whole result, and since that prompt only ever asked for a narrative with no schema, the model invented its own ad-hoc JSON shape on every call (a bare `{"analysis": "..."}`, sometimes with extra invented keys) — never the `tripName`/`majorityPrice`/`gbpPrice`/`duration`/`comments` shape `ResultsTable.vue` actually renders. The fast path's whole point is to reuse already-extracted data, not have the LLM re-derive (or mis-derive) it. ```python def analyse_from_cache(tenant_id: str, competitor_id: str, context: dict) -> dict: """ Fast path analysis — uses cached PocketBase data instead of live scrape. LLM receives structured JSON and generates narrative analysis only. Data flow: competitor_id → PocketBase products (latest) → structured fields mapped from the cached record (trip_name → tripName, majority_price_usd → majorityPrice, etc.) → client trip context → LLM prompt constrained to {comments, relevancy} only → cached structured fields + fresh comments/relevancy merged and returned """ ``` ### Staleness threshold 24 hours is the default staleness threshold. This is a configurable constant — adjust if client feedback indicates they need fresher data or if server load requires a longer window: ```python CACHE_STALENESS_HOURS = 24 # in core/scraper.py — adjust here only ``` ### PM-initiated cache bypass — `force_refresh` A collapsed "Advanced options" disclosure on `TripIntentForm.vue` (closed by default, not a prominent field) exposes a single "Force fresh scrape" checkbox, unchecked by default. Unchecked, a run respects the normal fast/refresh-path decision (§7) — a competitor scraped recently is served from cache quickly. When checked, `force_refresh: true` rides through the `/research` POST body → `jobs.py`'s per-competitor `context` dict → `analysis_service.run_competitor_analysis()`, bypassing BOTH cache shortcuts for that run: the initial fast-path gate (`needs_refresh()`) and the post-scrape "content unchanged, reuse cached narrative" shortcut — so a forced run always ends in a real `extract_with_openrouter()` call, never `analyse_from_cache()`. Content-hash bookkeeping (`scrape_runs` row, `change_detected` clearing) still runs normally even when forced, so future non-forced runs keep an accurate hash history. Not persisted on the `scrape_runs` job record — it's a one-time execution-time override with no restore/display use case, unlike `tab_type`. This briefly defaulted to fully-fresh (checkbox inverted to an opt-in "prefer cache") while a real bug was being tracked down: the confirmed/ matched trip URL was silently discarded (see below), so cached data could be for the *wrong page entirely*, not just stale. Once that was fixed, the cache reflects the correct trip page, so the original opt-in design (fast by default, force-fresh as the explicit override) is trustworthy again and was reverted to. The daily cron (`/internal/scrape/`, `triggered_by='cron'`) builds its own request body with no `force_refresh` key and no trip-intent fields at all — `jobs.py`'s `data.get('force_refresh', False)` naturally stays `False` there regardless of the dashboard's default, so the cron's caching efficiency is untouched by this. ### Confirmed/matched URL actually reaches the scrape `context['confirmed_url']` — the page confirmed at `UrlConfirmation.vue` (a Firecrawl `/map` match or a manual swap), threaded per-competitor through `jobs.py`'s loop — is what `run_competitor_analysis()` actually scrapes, falling back to `competitor['website']` (the stored root domain) only when nothing was confirmed for this run (e.g. the cron path, which has no trip-intent context). This was a real regression fixed in this session: `jobs.py` previously only ever read `entry['id']` from each confirmed competitor, silently discarding `entry['url']` — every scrape fell back to the competitor's bare root domain regardless of what was matched or manually swapped, which is why trip-intent matching (§8) appeared to have no effect on real analysis runs. ### Generic-homepage detection — `is_generic_page` `services/url_utils.py`'s `is_generic_homepage(url)` flags a URL with no meaningful path (the bare root domain, with or without a trailing slash, query string, or `www` prefix) — used in two places: - `score_and_rank_urls()` (§8) excludes bare-root candidates before scoring, the same hard-gate treatment as the destination-keyword check — a competitor whose only Firecrawl `/map` candidate is its own homepage correctly falls through to `found: False` rather than a false positive. - `run_competitor_analysis()` computes `is_generic_page` from whichever URL was actually scraped this run (or the cached record's own URL, on the fast path) and returns it as a top-level sibling of `path`/`result` — not nested inside `result`, since it's a meta-fact about the scrape, not an extracted trip field. `ResultsTable.vue` renders a "⚠ Generic page" badge in the competitor's column header when set. `UrlConfirmation.vue` also gained a per-competitor "Skip this competitor" toggle, scoped to the `found: false` case — a PM can now exclude a competitor with no real match from the batch entirely, instead of being pressured into typing a low-value guess (like the competitor's own homepage) just to unblock "Confirm & Run." --- ## 8. Trip-Intent Matching ### Overview The PM never manually enters a competitor trip URL. They describe their own product — destination, duration, travel style — and the system finds the best matching trip page on each competitor site automatically via Firecrawl /map with scoring. This feature is available in the **Analyse tier** (on demand, per job). The **Discover tier** extends this with a scheduled weekly crawl. ### Dashboard Flow ``` PM fills in the Analyse form: Product name: Inca Trail Classic Destination: Peru Duration: 15 days Travel style: Classic Competitors: ☑ Intrepid ☑ Exodus ☑ Flash Pack ☑ Kimkim PM clicks [Find & Analyse] → Flask POST /research/find-urls → Firecrawl /map runs per competitor simultaneously (async) → URL scoring selects best match per competitor Confirmation screen shown in dashboard: Intrepid Travel → intrepidtravel.com/peru-classic-15-days [✓ Use] [↗ Swap] Exodus Travels → exodustravels.com/machu-picchu-trek [✓ Use] [↗ Swap] Flash Pack → flashpack.com/peru-adventure-15d [✓ Use] [↗ Swap] Kimkim → No match found [Enter URL manually] PM confirms (or swaps any URL) → Flask POST /research with confirmed URLs → RQ job queued → analysis runs → Results displayed in dashboard ``` ### Flask Route ```python @app.route('/research/find-urls', methods=['POST']) def find_urls(): """ Finds the best matching competitor trip URL for each selected competitor given the PM's research intent (destination, duration, travel style). Runs Firecrawl /map per competitor concurrently. Body: { destination: str, duration: int, travel_style: str, competitor_ids: list[str] } Returns: { matches: [ { competitor_id, competitor_name, url, confidence, found } ] } Data flow: competitor_ids → PocketBase (get root URLs per competitor) → Firecrawl /map per competitor (concurrent, ThreadPoolExecutor) → score_and_rank_urls() per competitor → return best match per competitor with confidence score """ ``` ### URL Finding Service — `services/trip_finder_service.py` ```python def find_competitor_trip_url(root_url: str, destination: str, duration: int, travel_style: str) -> dict: """ Uses Firecrawl /map with search terms to find the most likely trip page URL on a competitor site matching the PM's research intent. Data flow: root_url + search terms → Firecrawl /map → filtered URL list → score_and_rank_urls() → { url, confidence, found } returned """ response = requests.post( "http://firecrawl:3002/v1/map", json={ "url": root_url, "search": f"{destination} {duration} days {travel_style} tour" }, timeout=15 ) urls = response.json().get("links", []) if not urls: return {"url": None, "confidence": 0, "found": False} return score_and_rank_urls(urls, destination, duration) def score_and_rank_urls(urls: list, destination: str, duration: int) -> dict: """ Scores candidate URLs by keyword match against destination and duration. Returns highest scoring URL with confidence score. A URL must contain at least one destination keyword to be considered at all — this is a hard gate, not an additive bonus. Regression found in production: a URL for a completely different country could still cross the found threshold purely from a duration-digit match plus a generic category-keyword match (a Turkey trip page scored 0.75 confidence against a Peru search just because it happened to be a "9 day classic" tour) — category/duration signals only ever score URLs that pass the destination gate, they can never compensate for failing it. A bare root-domain URL (no path at all — see services/url_utils.py's is_generic_homepage()) is excluded the same way: Firecrawl's /map candidate list commonly includes the site's own root URL, and if the destination keyword happens to appear in the domain itself (e.g. a competitor literally named "peru-travel.com"), the root URL could otherwise score high enough to be returned as a confident "found" match — silently presenting a homepage as if it were a trip-specific page. Scoring rules (applied only to URLs that pass both gates): +1 per destination keyword found in URL +2 if duration (as integer) appears in URL +1 if travel category words appear (classic, trek, trail, etc.) Confidence = top_score / max_possible_score, capped at 1.0 If no URL passes both gates, or confidence < 0.3, returns found=False — prompt manual entry. """ dest_keywords = destination.lower().split() category_keywords = ['classic', 'trek', 'trail', 'safari', 'adventure', 'explorer', 'journey'] scores = [] for url in urls: if is_generic_homepage(url): continue # bare root domain — never a trip-specific candidate url_lower = url.lower() dest_matches = sum(1 for kw in dest_keywords if kw in url_lower) if dest_matches == 0: continue # zero destination relevance — never a candidate score = dest_matches if str(duration) in url_lower: score += 2 score += sum(1 for kw in category_keywords if kw in url_lower) scores.append((score, url)) scores.sort(reverse=True) if not scores: return {"url": None, "confidence": 0, "found": False} top_score, top_url = scores[0] max_possible = len(dest_keywords) + 2 + 1 confidence = min(top_score / max_possible, 1.0) return { "url": top_url, "confidence": round(confidence, 2), "found": confidence >= 0.3 } ``` ### Concurrent Execution Run /map calls for all competitors simultaneously — not sequentially: ```python from concurrent.futures import ThreadPoolExecutor, as_completed def find_all_competitor_urls(competitors: list, destination: str, duration: int, travel_style: str) -> list: """ Runs trip URL finding for all competitors concurrently. Max 5 concurrent Firecrawl /map calls — respects server limits. Data flow: competitors list → ThreadPoolExecutor (max 5) → find_competitor_trip_url() per competitor in parallel → collect results → return matches list """ results = [] with ThreadPoolExecutor(max_workers=5) as executor: futures = { executor.submit( find_competitor_trip_url, c['website'], destination, duration, travel_style ): c for c in competitors } for future in as_completed(futures): competitor = futures[future] match = future.result() results.append({ "competitor_id": competitor['id'], "competitor_name": competitor['name'], **match }) return results ``` --- ## 9. API Routes All routes require `X-API-Key` header matching `API_SECRET_KEY` env var except `/health` and `/stripe/webhook`. ### Licence and Auth ``` POST /validate-email Body: { email } Returns: { valid, tier, reason } Logic: domain check → seat count → seat registration Note: sheet_id binding removed — standalone script model POST /stripe/webhook Stripe signature verified via STRIPE_WEBHOOK_SECRET Creates/updates tenant on payment events Sets status=inactive on cancellation ``` ### Trip-Intent Matching + Research ``` POST /research/find-urls Body: { destination, duration, travel_style, competitor_ids } Returns: { matches: [ { competitor_id, name, url, confidence, found } ] } Runs Firecrawl /map concurrently per competitor POST /research Body: { tenant_id, destination, duration, travel_style, tab_type, competitors: [ { id, name, url (confirmed) } ], prompt_template? } Returns: { job_id, status: "queued" } Enqueues to RQ — never runs synchronously GET /research/status/ Returns: { status, progress, results, error } GET /research/history Query: email, limit (default 5) Returns: last N runs for tenant GET /research/history/latest Query: email Returns: full stored results for the PM's most recent completed job Used by Apps Script sidebar for [Pull latest results] GET /research/history/ Returns: full stored results for a specific run POST /preview-prompt Returns: { prompt } — dry run, no scraping ``` ### Dashboard Data ``` GET /competitors Header: X-User-Email Returns: [ { id, name, url, primary_market, last_scraped, change_detected, status } ] POST /battlecards Body: { email } Returns: [ { competitor_name, content, content_json, generated_at } ] POST /comparable-trips Body: { email } Returns: [ { client_trip, competitor_trip, competitor, similarity_score, price_difference, duration_difference } ] ``` ### Apps Script Sheet Sync No dedicated routes. The Sheet-initiated pull uses the existing research history endpoints: ``` GET /research/history/latest?email={email} Returns: full stored results for the PM's most recent completed job Called by the Apps Script sidebar's [Pull latest results] button See Section 15 — Google Apps Script for the full pull flow GET /research/history/ Returns: full stored results for a specific run (used by dashboard history panel — see Trip-Intent Matching + Research routes above) ``` Rationale: A stateless Flask backend has no way to reach into whichever Sheet the PM currently has open. Only the Apps Script itself, running inside that open Sheet, can write to it. So the Sheet pulls from Flask rather than Flask pushing to the Sheet. `getWorkbookTabs()` runs inside the script and needs no Flask counterpart. ### Dashboard Account ``` GET /account/ POST /account/seats DELETE /account/seats/ POST /account/competitors — add single competitor (requires: name, website, catalogue_url, primary_market) POST /account/competitors/bulk — bulk import from CSV (validates, previews, imports) GET /account/competitors/template — download blank CSV template with correct headers PUT /account/competitors/ DELETE /account/competitors/ — deactivates the competitor record GET /account/template PATCH /account/onboarding — update onboarding_checklist fields PATCH /account/timezone — captures browser timezone on first login, never overwrites an existing value GET /billing/portal — Stripe customer portal session URL POST /billing/upgrade — creates Stripe checkout for tier upgrade Body: { tenant_id, target_tier } Returns: { checkout_url } POST /auth/request-password-reset — Body: { email } → triggers PocketBase reset email POST /auth/confirm-password-reset — Body: { token, password, passwordConfirm } ``` ### Internal (n8n calls) ``` POST /internal/scrape/ POST /internal/battlecard/ POST /internal/embed-products/ POST /internal/match-comparable/ POST /internal/weekly-brief/ POST /internal/daily-digest/ POST /internal/welcome-email/ POST /internal/data-cleanup — global, not per-tenant (§22 DATA_RETENTION) POST /internal/discover/ — Discover tier only ``` ### System ``` GET /health — returns { status: "ok", version } ``` --- ## 10. Concurrency — RQ Job Queue The `/research` endpoint NEVER runs synchronously. Every research job is enqueued to RQ and processed by background workers. ### Queue Priority ```python # Three priority queues — Intelligence tier gets processed first high_q = Queue('high', connection=redis_conn) # Intelligence tier normal_q = Queue('normal', connection=redis_conn) # Discover + Analyse low_q = Queue('low', connection=redis_conn) # background tasks # Worker consumes in priority order # command: rq worker high normal low -u redis://bettersight-redis:6379 ``` ### Job Function (jobs.py) The job function calls scrape and extract functions from `core/scraper.py` and `core/extractor.py`. It adds PocketBase status updates and the fast/slow path check around the existing logic. When running as an RQ job, `jobs.py` imports from `core/scraper.py` and `core/extractor.py`. Never bypass the layer structure. ### Content-hash change detection Every scrape job computes a content hash and compares against the last recorded hash for the same competitor URL. This replaces Firecrawl Monitor's role in the original design. ```python import hashlib def compute_content_hash(page_html: str) -> str: """ Returns SHA-256 hash of normalised page content. Normalisation strips whitespace, script blocks, and known ephemeral elements (timestamps, session IDs) so cosmetic changes do not trigger false positives. """ normalised = normalise_html(page_html) return hashlib.sha256(normalised.encode('utf-8')).hexdigest() def detect_change(competitor_id: str, new_hash: str) -> bool: """ Compares new_hash against last recorded hash on scrape_runs. On change: sets competitor.change_detected = true, competitor.change_detected_at = now. Never clears the flag itself — detection and extraction happen in the same scrape now (unlike the old async Firecrawl Monitor webhook), so there's no "detected but not yet handled" state for this function to resolve. Clearing is the caller's responsibility — see below. """ last = scrape_repository.get_last_hash_for_url(competitor_id) if last is None: return True # first scrape — always process if last != new_hash: competitor_repository.update(competitor_id, { 'change_detected': True, 'change_detected_at': datetime.now().isoformat() }) return True return False ``` **`change_detected` stays visible until confirmed stable, not until "handled."** The scrape that finds a change also fully extracts and saves it — so there's no later "handled" event to reset the flag on. The caller (`analysis_service.run_competitor_analysis()`) does NOT reset it after a successful extraction; it leaves the flag as `detect_change()` set it, so the CompetitorsPage "⚠ Changed" badge, the Apps Script sidebar status, and the next day's n8n cron filter (`change_detected = true || stale`) can all still see it. It's only cleared on the *next* scrape that finds the content unchanged since — at most one extra hash-only confirmation scrape per genuine change, with no LLM cost since an unchanged result always skips extraction. An earlier version of this design reset `change_detected` to false immediately after every successful extraction, in the same function call that `detect_change()` set it — meaning nothing outside that one call could ever observe it as true. That silently broke the status badge and the cron filter's `change_detected = true` clause. Do not reintroduce that reset. Fields on `scrape_runs` collection gain: `content_hash`, `content_length`, `hash_algorithm` — see Section 6's "scrape_runs (additions for content-hash detection)" for the composite index that backs `get_last_hash_for_url()`'s competitor-scoped, recency-sorted lookup. --- ## 9. Licence Validation Logic ### Three layers — all must pass **Layer 1 — Domain check** Extract domain from email. Look up active tenant with matching domain in PocketBase. If no match → reject with "Domain not registered". **Layer 2 — Seat count** Check if email already has an active seat for this tenant. If yes → update last_accessed, allow. If no → count active seats. If at limit → reject with "Seat limit reached". If under limit → register new seat, allow. **Layer 3 — Sheet ID binding** On first validation for a tenant, store the sheet_id in the tenants record. On subsequent validations, compare sheet_id. If mismatch → reject with "Licence bound to a different Sheet. Contact support." ### Cache Apps Script caches validation result for 6 hours via CacheService. Cache key: `BETTERSIGHT_VALID` and `BETTERSIGHT_TIER`. On cache hit — skip API call entirely. --- ## 10. Scraping Architecture ### Refactor app.py Before New Development The existing `app.py` must be refactored into the modular three-layer structure before any new development begins. Logic stays identical — it is reorganised into correct layers, not rewritten. **Core IP preserved in `core/`:** - Playwright stealth browser with cookie consent handling and lazy load scrolling - BeautifulSoup HTML cleaning - AI web fetch fallback via OpenRouter for bot-protected sites - OpenRouter model cycling with fallback chain - Multi-currency extraction **Industry config moved to `industries/adventure_travel/`:** - NGS vs Standard field differentiation - Prompt builder and injection - JSON extraction schema **Client config moved to `config/client.py`:** - Google Sheet write-back via gspread - Sheet ID, tab names, column assignments ### Modular Scraping Architecture app.py must be refactored into a three-layer modular structure before any new development begins. This separates infrastructure from industry config from client config — making the codebase maintainable, testable, and extensible to other industries without touching core scraping logic. **Target folder structure:** ``` backend/ core/ scraper.py — Playwright browser setup, stealth, waiting, HTML cleaning extractor.py — OpenRouter model cycling, JSON parsing, fallback chain ai_fetch.py — AI web fetch fallback (existing fetch_with_ai()) industries/ adventure_travel/ fields.py — STANDARD_FIELDS, NGS_FIELDS column maps prompt.py — build_prompt(), inject_page_content() schema.py — JSON extraction schema for this vertical # future verticals added here — never touch core/ config/ client.py — SHEET_ID, tab names, column assignments per client jobs.py — RQ job functions — imports from core/ and industries/ api.py — Flask routes app.py — Entry point only — imports and wires everything together ``` **The three layers:** Layer 1 — `core/` — never changes between industries or clients - Playwright stealth browser config - Smart waiting and scroll logic - Cookie consent handling - HTML cleaning via BeautifulSoup - OpenRouter model cycling and JSON parsing - AI web fetch fallback Layer 2 — `industries/{vertical}/` — swapped per vertical - Field maps (column positions in the Sheet) - Prompt template and context ("you are a travel industry analyst") - JSON extraction schema (what fields to extract) - Tab names Layer 3 — `config/client.py` — swapped per client within a vertical - Sheet ID - Column assignments per competitor - Competitor DB tab structure **To pivot to a new industry:** Create a new folder under `industries/` with three files. Core scraping logic is never touched. Sheet template changes are driven entirely by the field map. ### playwright-stealth Integration Replace the manual `add_init_script` block in `core/scraper.py` with `playwright-stealth` — a comprehensive stealth library applying 20+ evasion techniques automatically. ``` pip install playwright-stealth ``` In `core/scraper.py` — replace the existing `add_init_script` call: ```python from playwright_stealth import stealth_async async def render_page(url: str) -> dict: async with async_playwright() as p: browser = await p.chromium.launch(...) context = await browser.new_context(...) page = await context.new_page() # Replaces the manual add_init_script block entirely await stealth_async(page) # Rest of existing code unchanged await page.goto(url, ...) ``` `stealth_async` handles: navigator.webdriver spoofing, plugins array, languages, chrome runtime, canvas fingerprint, WebGL masking, audio context randomisation, and ~15 additional evasion techniques that the manual script did not cover. ### Smart Proxy Service — `core/proxy_service.py` The proxy layer is a standalone service in `core/proxy_service.py`. It is the ONLY place in the codebase that knows about any proxy provider. Switching providers or adjusting the switching logic requires changing only this file. **Two providers — smart auto-switching:** - **Primary:** Webshare (~$3/month flat, free tier available) - **Fallback:** Bright Data pay-as-you-go (~$4-8/GB, only used when Webshare is blocked) At MVP scraping scale (6 competitors × 5 clients = 30 pages/day) total proxy cost is under $5/month. Bright Data fallback costs pennies at this volume. **PocketBase collection — `proxy_overrides`** ``` id auto tenant_id relation → tenants (nullable — global overrides have no tenant) domain text (e.g. intrepidtravel.com) provider select [webshare, brightdata] blocked_count integer last_blocked date created autodate ``` **ProxyService class:** ```python class ProxyService: """ Smart proxy provider — uses Webshare by default with country targeting, auto-escalates to Bright Data for domains that block Webshare. Country targeting uses competitors.primary_market (ISO code e.g. GB, US, AU) passed through from the competitor record — no hardcoded country map in code. Data flow (first request to a domain): domain + country → proxy_overrides lookup → no override found → Webshare config with country targeting returned → scraper attempts request → if blocked → escalate_to_brightdata(domain) called → Bright Data config returned → scraper retries → override stored in PocketBase for this domain Data flow (subsequent requests to a known blocked domain): domain + country → proxy_overrides lookup → brightdata override found → Bright Data config returned immediately — no Webshare attempt Data flow (specific-IP candidate rotation — before ever escalating to Bright Data): domain not already known-blocked → the account's own dedicated Webshare IPs for this country fetched (cached) via get_webshare_ip_candidates() → core/scraper.py's render_page() tries a handful of those specific IPs in turn → only once all of them are exhausted (or none were available) does it fall through to the single `-rotate` gateway attempt and then Bright Data, as before. """ def __init__(self): # In-memory cache of the account's dedicated Webshare IPs, keyed by # uppercased country code — {'expires_at': float, 'results': [...]}. # Same time.time()-based pattern as pocketbase_client.py's token cache. self._ip_cache = {} def is_domain_blocked_on_webshare(self, domain: str) -> bool: """ Returns True if this domain already has a brightdata override — lets render_page() skip the specific-IP candidate loop entirely for a domain already established as Bright-Data-only. """ override = proxy_repository.get_by_domain(domain) return bool(override and override.get('provider') == 'brightdata') def get_webshare_ip_candidates(self, country: str, limit: int = WEBSHARE_IP_CANDIDATE_LIMIT) -> list: """ Returns up to `limit` Playwright proxy configs for specific dedicated Webshare IPs assigned to this account, filtered to the target country. Returns [] if none are available (no country, API failure, or no results) — render_page() treats an empty list as "skip straight to the existing single-gateway attempt". Each candidate carries its OWN username/password/port from Webshare's proxy-list API response — different from _webshare_playwright()'s single account-level rotating credentials. """ if not country: return [] results = self._fetch_webshare_ip_list(country) if not results: return [] chosen = random.sample(results, min(limit, len(results))) return [ {"server": f"http://{ip['proxy_address']}:{ip['port']}", "username": ip['username'], "password": ip['password']} for ip in chosen ] def _fetch_webshare_ip_list(self, country: str) -> list: """ Fetches (and caches) the account's own dedicated Webshare IPs for a country via Webshare's proxy-list *management* API (proxy.webshare.io/api/v2/proxy/list/) — a different host and credential entirely from the proxy gateway (p.webshare.io + WEBSHARE_USER/PASS). Authenticated with a dashboard API token (WEBSHARE_API_KEY, "Authorization: Token "), and — per Webshare's own docs — wants the country code UPPERCASE (their example: "FR,US"). This is the opposite case convention from _webshare_playwright()'s gateway username suffix (lowercase) — two unrelated Webshare systems with inconsistent casing on their end, not a mistake here. Cached 30 min on success, 5 min on failure (a short negative TTL bounds a retry storm within one batch run without permanently caching what might be a transient failure). """ cache_key = country.upper() cached = self._ip_cache.get(cache_key) if cached and time.time() < cached['expires_at']: return cached['results'] results = [] try: url = WEBSHARE_PROXY_LIST_URL params = {'mode': 'direct', 'country_code__in': cache_key, 'page_size': 100} for _ in range(3): # defensive pagination cap response = requests.get( url, headers={'Authorization': f'Token {WEBSHARE_API_KEY}'}, params=params, timeout=10 ) response.raise_for_status() data = response.json() results.extend(r for r in data.get('results', []) if r.get('valid')) url = data.get('next') params = None if not url: break self._ip_cache[cache_key] = { 'expires_at': time.time() + WEBSHARE_IP_CACHE_TTL_SECONDS, 'results': results, } except (requests.RequestException, ValueError) as e: logger.warning(f'Webshare proxy list fetch failed for {country}: {str(e)[:200]}') self._ip_cache[cache_key] = { 'expires_at': time.time() + WEBSHARE_IP_CACHE_NEGATIVE_TTL_SECONDS, 'results': [], } return [] return results def get_proxy_for_domain(self, domain: str, country: str = None) -> dict: """ Returns requests proxy config for a domain with optional country targeting. Routes to Bright Data if domain has been previously blocked on Webshare. """ override = proxy_repository.get_by_domain(domain) if override and override['provider'] == 'brightdata': return self._brightdata_requests() return self._webshare_requests(country=country) def get_playwright_proxy_for_domain(self, domain: str, country: str = None) -> dict: """ Returns Playwright proxy config for a domain with optional country targeting. Routes to Bright Data if domain has been previously blocked on Webshare. """ override = proxy_repository.get_by_domain(domain) if override and override['provider'] == 'brightdata': return self._brightdata_playwright() return self._webshare_playwright(country=country) def escalate_to_brightdata(self, domain: str): """ Records that Webshare was blocked on this domain. Upserts a proxy_override record so future requests skip Webshare entirely. """ existing = proxy_repository.get_by_domain(domain) if existing: proxy_repository.update(existing['id'], { 'provider': 'brightdata', 'blocked_count': existing['blocked_count'] + 1, 'last_blocked': datetime.now().isoformat() }) else: proxy_repository.create({ 'domain': domain, 'provider': 'brightdata', 'blocked_count': 1, 'last_blocked': datetime.now().isoformat() }) def _webshare_requests(self, country: str = None) -> dict: """ Webshare requests proxy config with optional country targeting. Format per Webshare's own rotating-endpoint docs (help.webshare.io/en/articles/8596715): username-{country}-rotate, with the country code lowercase — their examples show `-us`, `-gb`, `-ca`, never uppercase. `country` arrives as an uppercase ISO code from competitors.primary_market (this schema's own convention — "GB", "US", "AU"), so this always lowercases it before building the username rather than assuming the caller already did. An earlier version of this function (and the real proxy_service.py it generated) used .upper() here — every proxied scrape failed Webshare auth silently, which fell through to the AI web-fetch fallback and then to an outright scrape failure, which in turn meant analysis_service never reached build_prompt() at all for any competitor requiring the refresh path. No country = general rotation across full pool. """ suffix = f"-{country.lower()}-rotate" if country else "-rotate" username = f"{WEBSHARE_USER}{suffix}" return { "http": f"http://{username}:{WEBSHARE_PASS}@p.webshare.io:80", "https": f"http://{username}:{WEBSHARE_PASS}@p.webshare.io:80" } def _webshare_playwright(self, country: str = None) -> dict: """Webshare Playwright proxy config with optional country targeting — see _webshare_requests() for why the country code is lowercased.""" suffix = f"-{country.lower()}-rotate" if country else "-rotate" return { "server": "http://p.webshare.io:80", "username": f"{WEBSHARE_USER}{suffix}", "password": WEBSHARE_PASS } def _brightdata_requests(self) -> dict: # Port 33335, not the commonly-documented 22225 — Bright Data's # currently valid SSL certificate requires 33335; 22225 is tied to # their deprecated cert (phased out ahead of its 2026 expiry). Using # 22225 is what produced a real net::ERR_CERT_AUTHORITY_INVALID on # a live scrape before this fix. return { "http": f"http://{BRIGHTDATA_USER}:{BRIGHTDATA_PASS}@brd.superproxy.io:33335", "https": f"http://{BRIGHTDATA_USER}:{BRIGHTDATA_PASS}@brd.superproxy.io:33335" } def _brightdata_playwright(self) -> dict: return { "server": "http://brd.superproxy.io:33335", "username": BRIGHTDATA_USER, "password": BRIGHTDATA_PASS } ``` **Block detection and auto-switch in `core/scraper.py`:** ```python async def render_page(url: str, country: str = None) -> dict: """ Renders a page using the smart proxy service with country targeting. country comes from competitors.primary_market in PocketBase. Detects blocks and auto-escalates to Bright Data if Webshare fails. Before the single `-rotate` gateway attempt, tries a handful of the account's own dedicated Webshare IPs for this country (get_webshare_ip_candidates()) — gives IP diversity a chance to resolve a one-off block without paying for a Bright Data request. Skipped entirely for a domain already known (via proxy_overrides) to block the whole Webshare account, and falls through gracefully to the existing gateway/Bright Data flow when no candidates are available (no country given, Webshare API unreachable, or every candidate still blocked). Data flow: url + country → domain extracted → is_domain_blocked_on_webshare(domain)? → no → get_webshare_ip_candidates(country) → try each in turn, return on first success → all exhausted or none available → proxy_service.get_playwright_proxy_for_domain(domain, country) → Playwright render → if blocked → escalate_to_brightdata(domain) → retry with Bright Data (no country targeting at fallback level) """ domain = extract_domain(url) if not proxy_service.is_domain_blocked_on_webshare(domain): for candidate_proxy in proxy_service.get_webshare_ip_candidates(country): result = await _render_with_proxy(url, candidate_proxy) if not _is_blocked(result): return result proxy = proxy_service.get_playwright_proxy_for_domain(domain, country) result = await _render_with_proxy(url, proxy) if _is_blocked(result): logger.warning(f'Webshare blocked on {domain} — escalating to Bright Data') proxy_service.escalate_to_brightdata(domain) result = await _render_with_proxy(url, proxy_service._brightdata_playwright()) return result def _is_blocked(result: dict) -> bool: """ Detects common block signals from scrape result. Returns True if the response looks like a bot wall rather than real content. Signals: low char count, HTTP 403/429, Cloudflare challenge, CAPTCHA text. """ if result.get('char_count', 0) < 500: return True if result.get('status_code') in [403, 429]: return True block_signals = [ 'cf-browser-verification', 'captcha', 'access denied', 'blocked', 'robot' ] return any(s in result.get('text', '').lower() for s in block_signals) ``` **In `jobs.py` — primary_market passed through automatically:** ```python def run_research_job(job_id: str, data: dict): """ Pulls primary_market from each competitor record and passes it to scrape() for automatic country-targeted proxy routing. No manual country configuration needed per job. """ for i, competitor in enumerate(data['competitors']): competitor_record = competitor_repository.get_by_id(competitor['id']) page_data = scrape( url=competitor['url'], country=competitor_record.get('primary_market') # e.g. "GB", "US", "AU" ) # rest of job unchanged ``` **Cost summary:** ``` Webshare free tier — 10 proxies shared, enough for MVP testing Webshare paid — ~$3/month, handles the vast majority of requests Bright Data fallback — pay-as-you-go ~$4-8/GB, <$2/month at MVP scale Total proxy cost MVP — under $5/month ``` ### OpenRouter Model Cycling — `core/extractor.py` Cycles through free tier models, retrying each one up to `RETRY_ATTEMPTS_PER_MODEL` (2) times with backoff before moving to the next — ported from the legacy `app.py`'s `extract_with_groq()`. An earlier version of this section (and the Phase 2 refactor built from it) moved to the next model on the very first error of any kind, including a transient rate limit — that meant one rate-limit burst could burn through the entire free-tier list in a single request with no retry. Falls back to a paid model if every free model is exhausted. Never fails silently. `OPENROUTER_FREE_MODELS` model IDs need the `:free` suffix — that's OpenRouter's actual naming convention for the free-tier variant of a model; without it the ID may 404 as "no endpoints found" (this is exactly what shipped in `.env.example` before — a stale, suffix-less model list, `google/gemini-flash-1.5,meta-llama/llama-3.1-8b-instruct, mistralai/mistral-7b-instruct`, that 404s on every real request and silently burns the paid fallback on every single extraction). Current working default: ``` OPENROUTER_FREE_MODELS=openai/gpt-oss-120b:free,nvidia/nemotron-3-super-120b-a12b:free,meta-llama/llama-3.3-70b-instruct:free ``` `minimax/minimax-m2.5:free` was in this list until a real test run against a live job showed it 404ing on every single call: OpenRouter deprecated the free-tier suffix for this model ("This model is unavailable for free. The paid version is available now - use this slug instead: minimax/minimax-m2.5"). Unlike a rate limit, this is not transient — every extraction call was guaranteed to burn both retry attempts on a model that can never succeed. Removed rather than left in to fail predictably; OpenRouter's free-tier lineup shifts over time, so revisit this list if extractions start failing at every free model again. **MVP launch runs paid-first, not free-tier-first.** The same test run that surfaced the dead `minimax` free-tier model also showed `openai/gpt-oss-120b:free` eating its full rate-limit backoff (5s + 10s) on nearly every extraction call — combined, a single research job took 5m47s, which blew past the dashboard's polling timeout (raised from 5 to 20 minutes in `analysis_store.js` as a separate fix, but the underlying slowness is still worth avoiding). Free-tier rotation is not reliable enough to depend on for launch, so `OPENROUTER_USE_FREE_MODELS_FIRST=false` skips `FREE_MODELS` entirely and calls `FALLBACK_MODEL` directly — set to `deepseek/deepseek-v4-flash` rather than `anthropic/claude-haiku-4-5`. `minimax/minimax-m2.5` (the paid slug) was tried here first, since it was already the paid fallback other fixes in this session had set up — it failed 100% of the time (10/10 calls: 2 retry attempts × 5 competitors in one real job) with `_call_openrouter`'s `_parse_json_response` crashing on `None` content. Root cause, confirmed directly against OpenRouter's API using the actual production prompt (real JSON schema + realistic page-content length, not a synthetic approximation): `minimax/minimax-m2.5` has *mandatory* reasoning that cannot be disabled (`reasoning: {enabled: false}` returns `400 Reasoning is mandatory for this endpoint and cannot be disabled`), and its hidden chain-of-thought token consumption scales unpredictably with how much the model has to reason through — real page content demands far more reasoning than a short, repetitive test string, so it's much more likely to consume the entire `max_tokens=4096` budget before ever writing the visible JSON answer, leaving `content: null`. Small/simple test prompts against this model succeed fine, which is why isolated smoke-testing didn't catch it — only a prompt shaped like the real extraction call did. `deepseek/deepseek-v4-flash` was tested against the identical real prompt (4/4 successful runs, `finish_reason: "stop"` every time, no `None` content) and is also roughly 3.5x cheaper per call. If a future model swap is considered, always test against the real `build_prompt()` output at realistic page-content length — a short synthetic prompt is not sufficient to catch a reasoning-budget truncation failure like this one. `_call_openrouter()` also sends `response_format={'type': 'json_object'}` on every call — a second, unrelated real failure surfaced after the `deepseek` switch: it would write a full prose comparative analysis (markdown headers, tables, "Key Differentiators" sections) instead of the requested JSON whenever a competitor's page content didn't clearly match the requested trip, prioritizing a "helpful" explanation of the mismatch over the format instruction. Not reproducible with a short synthetic prompt either — only the real, long production prompt triggered it. `response_format` constrains this at the API level rather than relying on prompt wording alone, and is a systemic fix for this whole class of "model deviates into prose" failure, not just the one reproduced case. Confirmed compatible with every model in `OPENROUTER_FREE_MODELS`/`OPENROUTER_FALLBACK_MODEL` before shipping — a model that rejects the parameter would 400 immediately, not the 429s (rate limiting, unrelated) seen during that compatibility check. `FREE_MODELS` is left populated and untouched in `.env`/`.env.example` specifically so re-enabling free-tier cycling later is a one-line flip (`OPENROUTER_USE_FREE_MODELS_FIRST=true`) — nothing else needs reconfiguring. ```python FREE_MODELS = os.getenv('OPENROUTER_FREE_MODELS', '').split(',') FALLBACK_MODEL = os.getenv('OPENROUTER_FALLBACK_MODEL', 'anthropic/claude-haiku-4-5') # MVP launch toggle — false skips FREE_MODELS entirely and calls # FALLBACK_MODEL directly. See the paragraph above for why. USE_FREE_MODELS_FIRST = os.getenv('OPENROUTER_USE_FREE_MODELS_FIRST', 'true').lower() == 'true' RETRY_ATTEMPTS_PER_MODEL = 2 def extract_with_openrouter(prompt: str, content: str) -> dict: """ Cycles through free OpenRouter models, retrying each one up to RETRY_ATTEMPTS_PER_MODEL times before moving to the next. Falls back to the paid fallback model if all free models fail. Never returns None — raises ExtractorError only after all models exhausted. Skips straight to FALLBACK_MODEL when USE_FREE_MODELS_FIRST is false. Data flow: prompt + content → for each model in FREE_MODELS + [FALLBACK_MODEL] (or just [FALLBACK_MODEL] when USE_FREE_MODELS_FIRST is false): up to RETRY_ATTEMPTS_PER_MODEL attempts: POST openrouter.ai/api/v1/chat/completions → if success: return parsed JSON → if rate limit: sleep 5*(attempt+1)s, retry same model → if credit/quota exhausted (403): move to next model immediately (quota won't recover mid-request — retrying wastes time) → if any other error: retry same model up to the attempt limit, then move to next model → if all free models fail → try FALLBACK_MODEL (paid) → if fallback fails → raise ExtractorError, log to Sentry """ models_to_try = FREE_MODELS + [FALLBACK_MODEL] if USE_FREE_MODELS_FIRST else [FALLBACK_MODEL] for model in models_to_try: for attempt in range(RETRY_ATTEMPTS_PER_MODEL): try: response = _call_openrouter(model, prompt, content) if response: if USE_FREE_MODELS_FIRST and model == FALLBACK_MODEL and FREE_MODELS: logger.warning( f'Free models exhausted — used paid fallback: {model}' ) sentry_sdk.capture_message( 'OpenRouter paid fallback used', level='warning' ) return response except RateLimitError: wait = 5 * (attempt + 1) logger.warning(f'{model} rate limited — waiting {wait}s...') time.sleep(wait) except CreditExhaustedError: logger.warning(f'{model} free tier exhausted — trying next model...') break except Exception as e: logger.warning(f'{model} failed: {str(e)[:200]}') raise ExtractorError('All OpenRouter models exhausted including paid fallback') ``` Firecrawl is in the MVP stack for `/monitor` only. `/map` discovery endpoint is Intelligence tier — do not build in MVP. --- ## 11. Battlecard Generation After each scrape run completes, n8n calls `/internal/battlecard/`. ```python def generate_battlecard(competitor_id: str, tenant_id: str) -> dict: """ Pulls latest product data for a competitor from PocketBase. Passes structured data to Claude Haiku via OpenRouter. Saves generated battlecard text and JSON back to PocketBase. Returns battlecard dict for immediate use. Data flow: competitor_id → PocketBase products (latest 10) → structured prompt → OpenRouter (claude-haiku) → battlecard text + JSON → PocketBase battlecards table """ ``` Prompt template: ``` You are a competitive intelligence analyst for an adventure travel operator. Based on this competitor data, write a concise battlecard. Competitor: {name} Top trips: {top_trips} Price range: {price_low} - {price_high} USD Avg duration: {avg_duration} days Positioning: {positioning_summary} Recent changes: {recent_changes} Write: 1. One line positioning summary (max 15 words) 2. Three strengths (bullet points, max 8 words each) 3. Three weaknesses (bullet points, max 8 words each) 4. One pricing observation (max 20 words) Return ONLY valid JSON: { "positioning": "...", "strengths": ["...", "...", "..."], "weaknesses": ["...", "...", "..."], "pricing_observation": "..." } ``` --- ## 12. Semantic Trip Matching ### Client trip population `client_trips` has no PM-facing form — it's populated automatically every time a PM submits `/research` (CLAUDE.md §8's TripIntentForm). The destination/duration/travel_style/product_name they describe IS a client trip, so `POST /research` upserts it via `embedding_service.save_client_trip()` before enqueueing the job — a side effect that never blocks the primary research job if it fails (§18 resilience rule). Upsert key: `(tenant_id, trip_name)`, where `trip_name` is the PM's product name if given, else a `{destination} {travel_style}` fallback. Re-running analysis for the same product updates the existing row instead of duplicating it. If `destination` or `duration_days` change on an update, the stored `embedding` is cleared (set to null) so the next embedding pass regenerates it — `/internal/embed-products` only embeds rows missing a vector, so a stale one would otherwise never get refreshed. ### Embedding generation After products are stored, embed each new trip using text-embedding-3-small. Store vector as JSON in the products.embedding field. ```python def embed_trip(trip: dict) -> list: """ Generates a semantic embedding for a trip using OpenRouter. Combines trip name, destination, duration, and activities into a single text representation before embedding. Data flow: trip dict → formatted text string → OpenRouter embeddings API → vector list → stored in products.embedding """ text = f""" {trip['trip_name']} Destination: {trip['destination']} Duration: {trip['duration_days']} days Activities: {trip['activities']} Start: {trip['start_location']} """ response = openai_client.embeddings.create( model="openai/text-embedding-3-small", input=text ) return response.data[0].embedding ``` ### Similarity matching ```python def find_comparable_trips(client_trips, competitor_trips, threshold=0.75): """ Calculates cosine similarity between all client trips and all competitor trips. Returns matches above the threshold sorted by similarity score descending. Data flow: client_trips + competitor_trips (each with .embedding) → pairwise cosine similarity → filter by threshold → sort by score → list of match dicts """ ``` Threshold: 0.75. Matches upserted into comparable_matches on `(tenant_id, client_product, competitor_product)` — a repeat weekly match updates `similarity_score`/`price_difference`/`duration_difference`/ `last_matched` in place rather than duplicating a row, which is the whole reason `first_matched`/`last_matched` exist as separate fields. A PM's `dismissed = true` on a match is left untouched by the upsert, so dismissing a match survives it matching again in a later run. `/internal/embed-products/` embeds both un-embedded competitor products AND un-embedded client_trips — both sides need a vector before `/internal/match-comparable/` can compare them, and both are read from PocketBase (via their repositories), not passed in the request body, so the Sunday-night n8n workflow can call either route unattended per tenant. --- ## 13. Weekly HTML Email Brief ### No PDF — HTML email only WeasyPrint has been removed from the stack. The Monday brief is a rich Jinja2-rendered HTML email sent via Resend. Same content, zero Docker dependency, zero PDF complexity. Brief email includes: - Bettersight branded header with tenant name - Market summary (price changes, new products, competitors monitored) - Price movements table (competitor, trip, previous, current, delta, percent) - New products this week (formatted cards) - Battlecard updates summary - Link to app.bettersight.io - Unsubscribe footer ### n8n Workflow — Monday 6am ``` Schedule Trigger (Monday 06:00 AST) → HTTP POST /internal/weekly-brief/{tenant_id} for each active tenant → Flask renders brief.html Jinja2 template with weekly PocketBase data → Resend sends HTML email to tenant admin → Logged to digest_logs ``` ### Brief service ```python def send_weekly_brief(tenant_id: str): """ Renders the weekly intelligence brief as branded HTML and sends it via Resend to the tenant admin email. Data flow: tenant_id → PocketBase (price_history, products, battlecards for current week) → brief.html Jinja2 template rendered → Resend API → digest_logs record created """ data = brief_repository.get_weekly_data(tenant_id) html = render_template('emails/brief.html', **data) resend.Emails.send({ "from": "Bettersight ", "to": data['admin_email'], "subject": f"Bettersight Weekly Brief — Week of {data['week_of']}", "html": html }) brief_repository.log_digest(tenant_id, 'weekly') ``` --- ## 15. Google Apps Script ### Role — Sheet-Initiated Pull Integration The Apps Script is not the primary interface. The dashboard is. The script's only job is to **pull** completed analysis results from Flask into the PM's currently open Sheet on demand. **Architectural rule that governs this section:** A standalone Apps Script cannot be invoked by an external server. Google's execution model requires the script to run in response to a user action inside the Sheet itself. Therefore Flask never pushes to the Sheet — the Sheet pulls from Flask. The dashboard's post-analysis affordance is a "Copy rows" button (tab-separated to clipboard) plus a short instruction reminding the PM to open their Sheet and click "Pull latest results" in the Bettersight sidebar when they want the data written directly. **Three functions only:** ``` pullLatestResults(tabName) — Sheet-initiated fetch + write for the most recent job getWorkbookTabs() — returns tab list for the sidebar's own tab selector validateLicence() — confirms PM is authorised (email check only) ``` ### Deployment Model — Standalone Script Deployed as a standalone script, not container-bound to any workbook. PM installs once via a shareable link from app.bettersight.io/account. Works in any Google Sheet workbook they open from that point forward. ### Template Import — The PM's One-Time Setup The PM never moves their existing stage-gate workbook. They import a single tab from our template into their workbook and continue using their workbook exactly as before. **Two-step import performed once:** 1. From app.bettersight.io/account, click [Open template ↗] to open the Bettersight template Sheet we host on Google Drive 2. In their existing stage-gate workbook: right-click any tab at the bottom of the page → "Copy to" → "Existing spreadsheet" → select their workbook 3. The Bettersight Analysis tab is added alongside their existing tabs **The PM can rename the imported tab to anything.** The sidebar's tab selector dropdown reads live tab names via getWorkbookTabs() every time it opens, and detectColumns() maps data to columns by header name — never by tab name or column position. Rename the tab "Comp Intel", "Peru Research 2026", or leave it as "Bettersight Analysis" — pullLatestResults() writes correctly either way as long as the PM selects the intended tab from the dropdown. **After import:** - Their existing stage-gate formulas can reference the new tab like any other tab: ='Bettersight Analysis'!B5 or ='Whatever They Renamed It'!B5 - The standalone Apps Script (installed once against their Google account) appears in Extensions → Bettersight in every workbook they open, including this one - When they click Pull latest results the script uses SpreadsheetApp.getActiveSpreadsheet() to write into whichever workbook is currently open — no configuration required per workbook **What we do not do:** - We do not ask the PM to migrate their stage-gate workbook into our template - We do not require them to duplicate their existing workbook - We do not bind the script to a specific spreadsheet ID - We do not depend on a specific tab name - We do not host the PM's stage-gate data — only the imported Bettersight tab holds any Bettersight data, and that tab lives inside their own workbook on their own Drive ### Template File Structure The template Sheet we host on Google Drive contains a single tab named Bettersight Analysis by default (PM can rename after import). **Column structure:** Row 1 is a bold header row with column names matching STANDARD_FIELDS in the exact order the extractor produces. Rows 2+ are empty and will be populated by pullLatestResults() based on header-name column detection. **Header resilience:** - If the PM renames a column header, detectColumns() still finds it as long as the header name is one of the recognised aliases in detectColumns()'s header alias map - If the PM inserts new columns, reorders columns, or deletes optional columns, detectColumns() adapts — never hardcoded column positions - If detectColumns() cannot find a required header, it logs a warning to the sidebar and falls back to appending the missing column at the end of the existing data range with the standard name **Template hosting:** The template is a single Google Sheets file on Drive at a stable share URL. The URL is stored as an env var TEMPLATE_SHEET_URL and served via GET /account/template. Any updates to the template columns are made in-place on the source file — no versioning needed since the PM's copy is a snapshot at their moment of import. ### File Structure ``` appscript/ Code.gs — onOpen(), install entry point Validation.gs — validateLicence() email-only Sync.gs — pullLatestResults(), getWorkbookTabs(), detectColumns() Sidebar.html — tab selector + pull button + last-pull status ``` ### Pull Flow ```javascript /** * Fetches the most recent completed analysis for the PM from Flask * and writes it to the selected tab of the currently open workbook. * Sheet-initiated only — cannot be triggered by any external server. * * Writes results using header-based column detection — never * hardcoded column positions. Resilient to any sheet restructuring. * * Flow: * PM clicks [Pull latest results] in sidebar → * validateLicence() confirms authorisation → * GET Flask /research/history/latest?email={sessionEmail} → * stored results returned → * detectColumns(sheet) builds { field: column } map → * writeResults(results, columnMap, sheet) writes into selected tab → * sidebar shows "Pulled: Today 9:14am · 5 competitors · Peru" */ function pullLatestResults(tabName) { ... } /** * Returns list of all tab names in the PM's currently open workbook. * Used by the sidebar (not the dashboard) to populate its own tab selector. */ function getWorkbookTabs() { return SpreadsheetApp.getActiveSpreadsheet() .getSheets() .map(s => s.getName()); } /** * Detects column positions by matching header names in row 1. * Returns { fieldName: columnIndex } map. * Falls back to STANDARD_FIELDS positions if headers not found. */ function detectColumns(sheet) { ... } ``` ### Sidebar UI ``` ┌─────────────────────────────┐ │ 🔵 Bettersight │ │ Pull results into Sheet │ │ ───────────────────────── │ │ Write to tab: │ │ [Bettersight Analysis ▾] │ │ │ │ [ Pull latest results ] │ │ ───────────────────────── │ │ Pulled: Today 9:14am │ │ 5 competitors · Peru │ └─────────────────────────────┘ ``` The full analysis workflow — competitor selection, trip-intent matching, job status, results view, history, confidence indicators — all live in the dashboard. The sidebar is a pull button only. The Apps Script is deployed as a **standalone script**, NOT container-bound to a specific Google Sheet file. This is the most important architectural decision for the add-on. **Why standalone:** - Works in any Google Sheet the PM opens — including their existing 20-sheet stage gate workbook - PM installs once via a shareable install link from the Bettersight dashboard - They import the Bettersight template tab into their existing workbook - The sidebar runs analysis and writes to whichever tab they select - Cross-sheet formulas in their stage gate sheet reference the Bettersight tab natively — no copy-paste ever needed after setup **What the PM receives:** 1. A template Sheet tab — downloadable from app.bettersight.io/account as a Google Sheets link. They copy this tab into their existing workbook. 2. A sidebar install link — one click installs the standalone script on their Google account. Works in any workbook from that point forward. **Sheet ID binding update:** Because the script is standalone, the licence validation no longer binds to a single Sheet ID. Instead it binds to the PM's Google account email (already implemented via domain + seat validation). Remove the sheet_id binding layer from the licence validation — it is redundant and would break the standalone model. --- ### File Structure ``` appscript/ Code.gs — onOpen(), menu setup, sidebar launch, install entry point Validation.gs — validateLicence() Research.gs — runResearch(), pollStatus(), writeResultsToSheet() Competitors.gs — loadCompetitorsFromPocketBase(), refreshCompetitorList() Settings.gs — saveSettings(), loadSettings(), getTargetSheet() Battlecards.gs — refreshBattlecards(), writeBattlecardsTab() ComparableTrips.gs — refreshComparableTrips(), writeComparableTab() Sidebar.html — sidebar UI ``` --- ### Validation Flow (Validation.gs) ```javascript /** * Validates the current user's licence against the Bettersight API. * Standalone script — validates by email only, no sheet_id binding. * * Flow: * 1. Check 6-hour cache — return true immediately if cache valid * 2. Get user email via Session.getActiveUser().getEmail() * 3. POST to /validate-email with email only * 4. On success — cache result and tier for 6 hours, return true * 5. On failure — show alert with reason, return false * * Cache keys: BETTERSIGHT_VALID, BETTERSIGHT_TIER * Cache duration: 21600 seconds (6 hours) */ function validateLicence() { ... } ``` --- ### Settings — Tab Selection and Persistence (Settings.gs) The PM selects which tab in their workbook to write results to. This setting persists across sessions via PropertiesService. ```javascript /** * Returns the target sheet tab for writing analysis results. * Reads from UserProperties — falls back to tab named * 'Bettersight Analysis' if no preference is stored. * * Flow: * UserProperties.getProperty('TARGET_TAB') → * if found: return that tab → * if not found: find 'Bettersight Analysis' tab → * if not found: prompt PM to select tab via dropdown */ function getTargetSheet() { ... } /** * Saves all sidebar settings to UserProperties so they persist * across sessions. Called automatically after each successful run. * * Settings persisted: * TARGET_TAB — which sheet tab to write to * TRAVEL_STYLE — last used travel style * PRODUCT_NAME — last used G Adventures product name * DESTINATION — last used destination * DURATION — last used duration * TAB_TYPE — Standard or NGS * SELECTED_COMPS — JSON array of last selected competitor IDs */ function saveSettings(settings) { ... } /** * Loads all persisted settings and returns them to the sidebar * so the PM's last run is pre-filled on open. */ function loadSettings() { ... } ``` --- ### Competitor Auto-Population (Competitors.gs) Competitors are loaded from PocketBase on sidebar open. The PM never manually types a URL again. ```javascript /** * Fetches the PM's competitor list from the Bettersight API. * Called on sidebar open. Returns array of competitor objects * for rendering as checkboxes in the sidebar. * * Flow: * validateLicence() → GET /competitors with email header → * returns [ { id, name, url, primary_market, last_scraped, * change_detected, status } ] → * rendered as checkboxes in sidebar with status indicators * * Status indicators shown per competitor: * ✓ Current (updated {time}) — last_scraped < 24hrs, no change * ⚠ Changed ({time} ago) — change_detected = true * ○ Stale — last_scraped > 24hrs */ function loadCompetitorsFromPocketBase() { ... } ``` --- ### Column Auto-Detection (Research.gs) Column positions are detected by reading header names, not hardcoded numbers. Makes the sheet resilient to any structural changes the PM makes. ```javascript /** * Detects the column index for each field by matching header names * in row 1 of the target sheet. Returns a field-to-column map. * Falls back to STANDARD_FIELDS hardcoded positions if headers * cannot be found — logs a warning when falling back. * * Flow: * Read row 1 of target sheet → * Build { fieldName: columnIndex } map by matching header text → * Return map for use by writeResultsToSheet() */ function detectColumns(sheet) { ... } ``` --- ### Research Flow with Per-Competitor Progress (Research.gs) ```javascript /** * Submits a research job and polls for completion. * Shows per-competitor status in the sidebar in real time. * * Flow: * 1. validateLicence() — exit if invalid * 2. getTargetSheet() — confirm which tab to write to * 3. detectColumns(sheet) — map field names to column positions * 4. loadSettings() — pre-fill job params from last run * 5. POST to /research with selected competitor IDs and job params * → receives job_id immediately * 6. Poll GET /research/status/{job_id} every 3 seconds * 7. On each poll update sidebar with per-competitor status: * ✓ Intrepid Travel — Peru Classic $2,190 * ✓ Exodus Travels — Machu Picchu $2,050 * ⟳ Flash Pack — scraping... * Kimkim — queued * 8. On complete — writeResultsToSheet() using detected columns * 9. saveSettings() — persist this run's settings for next time * 10. Show confidence indicators per competitor in sidebar * 11. Timeout after 75 attempts (5 minutes) */ function runResearch() { ... } /** * Writes extracted competitor data to the target sheet. * Uses column map from detectColumns() — never hardcoded positions. * Only writes fields that exist in the detected column map. * * Flow: * results array + columnMap + sheet → * for each competitor result: * for each field in columnMap: * sheet.getRange(fieldRow, columnMap[field]).setValue(value) */ function writeResultsToSheet(results, columnMap, sheet) { ... } ``` --- ### Single Competitor Re-Run (Research.gs) ```javascript /** * Triggers a fresh scrape and extraction for a single competitor. * Used when the PM wants to refresh one competitor without * re-running the full job. Updates only that competitor's column. * * Flow: * competitorId → POST /research with single competitor → * poll status → write result to sheet (single column only) → * update sidebar status for that competitor only * * Called from a "Refresh" button shown per competitor * in the completed run status list in the sidebar. */ function refreshSingleCompetitor(competitorId) { ... } ``` --- ### Confidence Indicators (Research.gs) ```javascript /** * Renders a confidence bar per competitor in the sidebar * after a run completes. Based on the relevancy score (1-5) * returned by the AI extraction. * * Display format: * Intrepid Travel ████████░░ 82% — Good match * Flash Pack ██████░░░░ 61% — Review recommended * Kimkim ████░░░░░░ 44% — Low confidence ⚠ * * Low confidence (< 60%) shows a warning prompt: * "The AI had low confidence extracting Flash Pack data. * The page may have changed structure. Consider a manual review." */ function renderConfidenceIndicators(results) { ... } ``` --- ### Analysis History (Research.gs) ```javascript /** * Fetches the last 5 analysis runs for the current tenant * from PocketBase. Displayed in the sidebar as a history panel. * PM can restore a previous run's data to the sheet for comparison. * * Flow: * GET /research/history?email={email}&limit=5 → * returns [ { job_id, run_at, competitors, summary } ] → * rendered as list in sidebar history panel * * Each history item shows: * Mon 9 Jun 9:14am · 5 competitors · Intrepid, Exodus, Flash Pack... * [Restore to sheet] * * Restore writes that run's stored results back to the sheet * using the same writeResultsToSheet() function. */ function loadAnalysisHistory() { ... } /** * Restores a previous run's results to the current sheet. * Fetches stored results from PocketBase by job_id. * Warns PM that this will overwrite current sheet data. */ function restoreHistoryRun(jobId) { ... } ``` --- ### Copy to Clipboard (Sidebar.html) A "Copy rows" button in the sidebar copies selected competitor rows as tab-separated values — paste-ready into any Google Sheet, Excel, or Notion table without formatting issues. ```javascript /** * Copies the selected competitor result rows as tab-separated * values to the clipboard. PM can paste directly into any sheet. * No formatting, no formulas — clean values only. * * Called from "Copy selected rows" button in sidebar. */ function copyRowsToClipboard(selectedCompetitorIds) { ... } ``` --- ### Sidebar UI Sections (Sidebar.html) ``` ┌─────────────────────────────────┐ │ 🔵 Bettersight │ │ ───────────────────────────── │ │ ANALYSIS SETTINGS │ │ Travel style [Classic ▾] │ │ Product name [____________] │ │ Destination [____________] │ │ Duration [____________] │ │ Tab type [Standard ● NGS○]│ │ Write to tab [Bettersight ▾]│ ← tab selector │ ───────────────────────────── │ │ COMPETITORS │ │ ☑ Intrepid Travel ✓ Current │ ← from PocketBase │ ☑ Exodus Travels ✓ Current │ │ ☑ Flash Pack ⚠ Changed │ │ ☐ Kimkim ○ Stale │ │ ───────────────────────────── │ │ [ Run Analysis ] │ │ [ Copy selected rows ] │ │ ───────────────────────────── │ │ LAST RUN RESULTS │ │ ✓ Intrepid ████████░░ 82% │ ← confidence │ ✓ Exodus █████████░ 91% │ │ ⚠ Flash Pk ██████░░░░ 61% │ ← low confidence warn │ [↺ Refresh Flash Pack] │ ← single re-run │ [ Copy selected rows ] │ │ ───────────────────────────── │ │ HISTORY │ │ Mon 9 Jun · 5 competitors │ │ [Restore] │ │ Mon 2 Jun · 4 competitors │ │ [Restore] │ └─────────────────────────────────┘ ``` --- ### New Flask Routes Supporting Apps Script Improvements ``` GET /competitors — returns competitor list for sidebar auto-population includes: id, name, url, primary_market, last_scraped, change_detected, status GET /research/history — last 5 runs for tenant (by email header) GET /research/history/ — full results for a specific run by job_id used by restoreHistoryRun() ``` --- ## 15. n8n Workflows ### Daily scrape (6am every day) ``` Schedule Trigger (06:00 AST) → For each active tenant: → Query PocketBase for competitors where: change_detected = true OR last_scraped > 24hrs ago → HTTP POST /internal/scrape/{tenant_id} with filtered competitor list → RQ job queued → Playwright scrape → AI extraction → PocketBase updated (products, price_history) → change_detected reset to false → last_scraped updated to now → price_history diff calculated automatically → is_new flag set on new products automatically ``` ### Battlecard regeneration (after scrape) ``` PocketBase webhook on scrape_runs status=complete → HTTP POST /internal/battlecard/{competitor_id} → Runs for each competitor in the completed scrape run ``` ### Price change alert ``` PocketBase webhook on price_history insert → Code node: check change_percent threshold (>5% triggers alert) → Creates alert record in PocketBase alerts table { alert_type: "price_change", competitor_id, product_id, change_amount, change_percent, delivered_dashboard: true } → Alert appears in dashboard activity feed immediately → n8n daily digest (6pm): collects unread alerts → Resend email summary ``` ### New product alert ``` PocketBase webhook on products insert where is_new=true → Creates alert record in PocketBase alerts table { alert_type: "new_product", competitor_id, product_id, delivered_dashboard: true } → Alert appears in dashboard activity feed immediately → Included in daily digest email (6pm) and Monday brief ``` ### [Discover tier] External notification integrations ``` When PM connects Slack/Teams/WhatsApp in account settings: → webhook_url stored in tenant_notifications table → n8n alert workflow checks tenant_notifications on each alert → If webhook configured: HTTP POST to Slack/Teams webhook URL → If Twilio configured: POST to Twilio WhatsApp API ``` ### Monday morning brief (6am) ``` Schedule Trigger (Monday 06:00 AST) → HTTP POST /internal/weekly-brief/{tenant_id} per active tenant → Flask renders brief.html Jinja2 template with weekly data → Resend sends HTML email to tenant admin email → Logged to digest_logs ``` ### Embedding + comparable matching (Sunday night — Analyse tier, in MVP) ``` Schedule Trigger (Sunday 23:00) → For each active tenant: → HTTP POST /internal/embed-products/{tenant_id} → Embeds any un-embedded competitor products AND client_trips → HTTP POST /internal/match-comparable/{tenant_id} → Semantic similarity matching, upserted into comparable_matches → client_trips itself is populated continuously by every /research submission (embedding_service.save_client_trip()) — this workflow only embeds and matches what's already there, it never creates client_trips records itself. ``` Not to be confused with the Intelligence-tier catalogue discovery workflow below, which is a different, deferred feature. ### Sunday night catalogue discovery (Intelligence tier — NOT in MVP) ``` DEFERRED — do not build until Intelligence tier development begins. Requires Firecrawl self-hosted which is not in the current stack. ``` --- ## 16. Stripe Integration ### Webhook events handled ``` checkout.session.completed → Create tenant in PocketBase → Set status=active → Generate welcome email with Sheet template download link customer.subscription.updated → Update tier if plan changed → Update max_seats based on new tier customer.subscription.deleted → Set tenant status=inactive → All seats blocked at next cache expiry (max 6 hours) ``` ### Tier → seat mapping ```python TIER_SEATS = { "analyse": 5, "discover": 8, "intelligence": 10, "enterprise": 999 # effectively unlimited } # Additional seats available on any tier ADDITIONAL_SEAT_PRICE = 45 # USD per seat per month # Configured as a Stripe add-on price on each subscription # Enterprise tier is not in Stripe public product list # Created manually per client as a custom Stripe price ``` --- ## 17. Environment Variables ```bash # Flask FLASK_ENV=production API_SECRET_KEY= # PocketBase POCKETBASE_URL=http://pocketbase:8090 POCKETBASE_ADMIN_EMAIL= POCKETBASE_ADMIN_PASSWORD= # OpenRouter OPENROUTER_API_KEY= # MVP launch default: paid-first — see §10 "OpenRouter Model Cycling" # for why. Set to "true" to re-enable free-tier cycling; FREE_MODELS # stays populated below so that's a one-line switch. OPENROUTER_USE_FREE_MODELS_FIRST=false OPENROUTER_FREE_MODELS=openai/gpt-oss-120b:free,nvidia/nemotron-3-super-120b-a12b:free,meta-llama/llama-3.3-70b-instruct:free OPENROUTER_FALLBACK_MODEL=deepseek/deepseek-v4-flash # paid — used directly at launch (USE_FREE_MODELS_FIRST=false), or as fallback once free tier is re-enabled. Not minimax/minimax-m2.5 — see §10, mandatory-reasoning model, 100% failure rate on the real extraction prompt # Proxy — Primary (Webshare) WEBSHARE_USER= WEBSHARE_PASS= WEBSHARE_API_KEY= # dashboard API token (Authorization: Token ) — different credential from WEBSHARE_USER/PASS above (those are proxy-gateway auth); used only to list the account's own dedicated IPs for candidate rotation. Generate from the Webshare dashboard's API/account section. # Proxy — Fallback (Bright Data — pay-as-you-go, auto-escalated per domain) BRIGHTDATA_USER= BRIGHTDATA_PASS= # Proxy settings PROXY_ESCALATION_THRESHOLD=2 # blocked_count before treating domain as permanently blocked # Stripe STRIPE_SECRET_KEY= STRIPE_WEBHOOK_SECRET= STRIPE_PRICE_ANALYSE= # trial_period_days: 14 STRIPE_PRICE_DISCOVER= # trial_period_days: 21 STRIPE_PRICE_INTELLIGENCE= # trial_period_days: 30 # Enterprise: created manually per client — no trial, no env var # Transactional email RESEND_API_KEY= # Error tracking SENTRY_DSN= # Required in staging and production — never skip # Hetzner Object Storage (backups) HETZNER_S3_ACCESS_KEY= HETZNER_S3_SECRET_KEY= HETZNER_S3_ENDPOINT=https://fsn1.your-objectstorage.com HETZNER_S3_BUCKET=bettersight-backups # Gotify (owner monitoring — never client-facing) GOTIFY_URL=https://gotify.bettertend.net GOTIFY_APP_TOKEN=A78_l3r9y.ZzBr- # Google Sheets (existing from app.py) GOOGLE_SERVICE_ACCOUNT_JSON= SHEET_ID= STANDARD_TAB= NGS_TAB= ``` --- ## 18. Enterprise Software Development Standards > These standards are absolute. They are never negotiated, never skipped, > and never deferred to a later phase. ### Languages and Conventions - Backend: Python only. No mixed languages in the same service. - Frontend: JavaScript only. No TypeScript. No `.ts` files. - `snake_case` for Python. `camelCase` for JavaScript. - No abbreviations unless universally understood (id, url, pdf). ### Separation of Concerns **Backend layer hierarchy — no layer skips another:** ``` Routes (api.py) → validate input, call services, return response → NO business logic. NO direct DB calls. NO external API calls. Services (services/) → all business logic and orchestration → NO HTTP request/response objects → NO direct DB queries or external API calls → calls repositories and other services only Repositories (repositories/) → all data access (PocketBase reads/writes, external API calls) → NO business logic → NO service calls → returns plain dicts only jobs.py → imports and calls service functions only → NO direct DB access → NO business logic ``` ### Security - Every authenticated route checks `X-API-Key` against `API_SECRET_KEY` env var - Every DB query for non-admin operations is tenant-scoped (`tenant_id` filter on every PocketBase query) - All credentials from env vars — hardcoded credentials are a build failure - Stripe webhook validated via signature before any processing - No stack traces in production error responses - Seat validation checks domain and seat count — both every time ### Observability - Every route logs: method, path, tenant_id (if known), duration, status code - Every RQ job logs: job_id, tenant_id, start, completion, error if any - `/health` endpoint always returns `{ status: "ok", version }` — no auth required - Sentry configured via `SENTRY_DSN` env var in production - n8n monitors `/health` every 5 minutes — Gotify alert on failure ### Resilience - RQ job failures are logged to scrape_runs.error_log - Failed scrapes do not fail the entire batch — continue to next competitor - Side-effect failures (Gotify, email) never fail a primary operation - All external API calls (OpenRouter, Bright Data, Firecrawl, Stripe) have try/except with logged errors and graceful fallback - RQ retries failed jobs automatically up to 3 times with backoff ### Code Commenting — Mandatory **All code must be adequately commented. This is not optional.** Every function, route, service method, and job function must have: 1. A docstring stating what the function does (one sentence) 2. The data flow — what comes in, what happens to it, what goes out 3. Inline comments on every meaningful step ```python def validate_email(email: str, sheet_id: str) -> dict: """ Validates a Google user's access to Polaris via three checks: domain registration, seat count, and Sheet ID binding. Data flow: email + sheet_id → domain extraction → PocketBase tenant lookup (Layer 1) → seat count check → seat registration if new (Layer 2) → sheet_id binding on first use / comparison on subsequent (Layer 3) → { valid: bool, tier: str, reason: str } """ # Extract domain from email for tenant lookup domain = email.split("@")[1] # Layer 1: Check if domain is registered to an active or trial tenant tenant = tenant_repository.get_by_domain(domain) if not tenant: return {"valid": False, "reason": "Domain not registered"} if tenant["status"] not in VALID_STATUSES: return {"valid": False, "reason": "Subscription inactive"} # Layer 2: Check seat availability or register existing seat seat_result = seat_repository.validate_or_register(tenant, email) if not seat_result["valid"]: return seat_result return {"valid": True, "tier": tenant["tier"]} ``` ### Testing - 80% overall test coverage enforced — cannot merge below this - 100% coverage on: licence validation, Stripe webhook, seat management, rollback paths - Every repository has a mock implementation alongside the real one - Unit tests for all services - Integration tests for all API routes - Tests live in `backend/tests/` mirroring the service/repository structure ### Environments and Deployment - Three environments: development, staging, production - Nothing reaches production without passing staging - Production deploys require a tagged version — no unversioned production deploys - Dokploy handles zero-downtime deploys via rolling restart ### Data and Backup **Two-layer backup strategy — belt and braces:** **Layer 1 — Litestream (real-time WAL streaming)** - Streams every SQLite write to Hetzner Object Storage in real-time - Recovery point objective: seconds - Recovery time objective: under 5 minutes - Runs as a sidecar container alongside PocketBase **Layer 2 — Daily zip backup (point-in-time snapshots)** - Daily 2am via crontab on the VPS — survives container restarts - 7-day retention on Hetzner Object Storage - Recovery point objective: 24 hours (fallback if Litestream stream is corrupted) Implemented in `scripts/backup-pocketbase.sh` and `scripts/bettersight-backup.cron` — see that file for exact install steps (`cp` to `/usr/local/bin/`, `chmod +x`, `cp` the crontab entry to `/etc/cron.d/bettersight-backup`). Requires an `rclone` remote named `hetzner-s3` configured on the VPS with the `HETZNER_S3_*` credentials. Cron output is redirected to `/var/log/bettersight-backup.log` (a small addition beyond the original spec below, for debuggability). ```bash # /etc/cron.d/bettersight-backup 0 2 * * * root /usr/local/bin/backup-pocketbase.sh # backup-pocketbase.sh #!/bin/bash DATE=$(date +%Y%m%d) docker exec pocketbase /pb/pocketbase backup \ --dir /pb/backups --name backup_$DATE.zip rclone copy /pb/backups/ \ hetzner-s3:bettersight-backups/ \ --max-age 7d find /pb/backups -name "*.zip" -mtime +2 -delete ``` - Audit logs (alerts, digest_logs) are append-only — no update or delete permitted - Multi-step operations (scrape → store → alert) documented with rollback strategy ### Strict Rules — Never Violate - Every authenticated endpoint validates `X-API-Key` — no exceptions - Every DB query for non-admin roles includes `tenant_id` filter — no exceptions - All credentials from env vars — hardcoded values are a build failure - No layer skips another in the separation of concerns hierarchy - A side-effect failure never fails a primary operation - Every multi-step operation has a documented rollback strategy - Mock implementations ship alongside every real repository - 80% test coverage enforced by CI — cannot merge below this - 100% coverage on payment, auth, webhook, and rollback paths - Nothing reaches production without passing staging - Production deploys require a tagged version - Audit logs are append-only — no update or delete ever - Daily backup with off-server storage — no exceptions - Sentry configured in staging AND production — silent failures are unacceptable - `/health` endpoint exists and is monitored — automated alert on failure - All code and functions are adequately commented — uncommented code is a build failure - Every Flask error response is structured JSON — never plain text - Every page has a defined empty state — no blank pages ever - Rate limiting applied to every public endpoint via Flask-Limiter - PII (email addresses) never sent to Sentry — scrubbed before send - Data retention policy enforced by weekly n8n cleanup workflow - GDPR deletion endpoint exists and tested before launch - Session expiry handled gracefully — PM returned to exact page after re-login - In-progress analysis jobs survive session expiry — resume on return --- ## 19. Build Order Follow this sequence. Do not start a phase until the previous is complete and tested. ``` Phase 1 — Data foundation 1. PocketBase schema (all collections) 2. Repository layer (all repositories with mocks) 3. Service layer (licence, scrape, battlecard, embedding, brief, alert, trip_finder) 4. Flask API routes (all endpoints including /research/find-urls) 5. RQ job queue setup (jobs.py, worker config) 6. Tests for all of the above (80% floor) Phase 2 — Refactor app.py into modular structure 7. Create proxy_overrides PocketBase collection 8. Create repositories/proxy_repository.py 9. Create core/proxy_service.py — Webshare primary + Bright Data fallback 10. Create core/scraper.py — Playwright + playwright-stealth + ProxyService 11. Create core/extractor.py — OpenRouter cycling and JSON parsing 12. Create core/ai_fetch.py — AI web fetch fallback + ProxyService 13. Create industries/adventure_travel/fields.py 14. Create industries/adventure_travel/prompt.py 15. Create industries/adventure_travel/schema.py 16. Create config/client.py 17. Create services/trip_finder_service.py — find_competitor_trip_url(), score_and_rank_urls(), find_all_competitor_urls() 18. Update jobs.py to import from core/ and industries/ 19. Verify all existing functionality works identically after refactor Phase 3 — Dashboard (PRIMARY — build before Apps Script) 20. Clone nuxt-ui-templates/dashboard-vue as frontend base 21. Configure Vue Router: /login — auth page /account — subscription, seats, template download /competitors — add/edit competitor list /analyse — trip-intent form + confirmation + results (CORE PAGE) /battlecards — battlecard view /activity — market movement feed 22. Pinia auth store — tenant_id, email, tier, JWT 23. api_service.js — Axios instance with X-API-Key header 24. LoginPage.vue — email/password + Google OAuth 25. AccountPage.vue — subscription card, seats table, template download 26. CompetitorsPage.vue — competitor list with status badges, add/edit modal 27. AnalysePage.vue — THE CORE PAGE: a. Trip-intent form (destination, duration, style, competitor checkboxes) b. [Find & Analyse] button → POST /research/find-urls c. URL confirmation screen — PM confirms or swaps matches d. [Confirm & Run] → POST /research → poll status e. Per-competitor progress display during job f. Results view — side-by-side comparison table g. Confidence indicators per competitor h. [Push to Sheet ▾] button with tab selector dropdown i. Run history panel — last 5 runs with [Restore] option 28. Stripe integration — checkout, webhook, portal link 29. Resend email templates — welcome, trial reminders, daily digest, weekly brief 30. Freescout deploy — container + support.bettersight.io Pangolin route + support@bettersight.io MX record + support link in dashboard footer 31. useOnboardingTour.js — Shepherd.js 5-step guided tour 32. tour.css — custom Shepherd styles matching Bettersight design system 33. data-tour anchor attributes on nav items and key components 34. PATCH /account/onboarding route 35. Tour trigger on ActivityPage.vue onMounted 36. Tour replay link on AccountPage.vue Phase 4 — Apps Script (Sheet-initiated pull — simple) 31. Deploy as standalone Apps Script — not container-bound to any workbook 32. Validation.gs — validateLicence() email-only 33. Sync.gs — pullLatestResults(), getWorkbookTabs(), detectColumns() 34. Sidebar.html — tab selector + [Pull latest results] + last-pull status 35. Sheet template tab — importable into any existing workbook 36. Flask GET /research/history/latest route (reuses history handler) Phase 5 — Automation 37. n8n daily scrape cron — changed/stale competitors only 38. PocketBase price_history diff logic (change_amount, change_percent) 39. PocketBase is_new flag logic (first_seen detection) 40. Content-hash logic in scrape service — compute_content_hash(), detect_change(), normalise_html() 41. Add content_hash, content_length, hash_algorithm fields to scrape_runs collection 42. Verify daily cron sets competitor.change_detected on hash diff and skips LLM extraction when hash unchanged + cache fresh 43. n8n PocketBase webhook → create price_change alert record in PocketBase 44. n8n PocketBase webhook → create new_product alert record in PocketBase 45. n8n daily digest email (6pm) — collects unread alerts → Resend email 46. n8n battlecard regeneration workflow (fires on scrape_runs complete) 47. n8n Monday brief workflow (6am — Jinja2 HTML → Resend email) 48. Dashboard bell icon + unread count from alerts table 49. Mark as read / dismiss on AlertCard.vue Phase 6 — Semantic matching 50. embedding_service.py — embed_trip(), find_comparable_trips() 51. n8n embedding and matching workflows (Sunday night) 52. comparable_matches table populated and visible in dashboard Phase 7 — Ship 53. Flask-Limiter configured on all public routes 54. Sentry initialised in Flask and Vue — staging first, then production 55. GDPR deletion endpoint POST /account/delete tested end to end 56. n8n weekly data cleanup workflow (retention policy) 57. Empty states implemented on all seven dashboard pages (incl. ResetPasswordPage) 58. Session expiry handling — token refresh interceptor + return-to logic 59. In-progress job survival across session expiry 60. Password reset — "Forgot password?" on LoginPage + ResetPasswordPage.vue 61. Timezone captured on first login (no signup form in scope) → PATCH /account/timezone → stored in tenants.timezone 62. n8n daily digest uses timezone-aware send time per tenant (luxon) 63. Upgrade prompt on AccountPage — next tier card with [Upgrade now →] button 64. POST /billing/upgrade route — Stripe checkout for tier upgrade 65. Multi-currency display in ResultsTable — primary market hero + USD secondary 66. Litestream container added to docker-compose.yml — replicating PocketBase to Hetzner S3 67. Hetzner Object Storage backup configured — rclone cron daily 2am (belt + braces with Litestream) 68. robots.txt check added to core/scraper.py — check_robots_txt() before each scrape 69. OpenRouter paid fallback configured — OPENROUTER_FALLBACK_MODEL env var set 70. Scraping legal clause added to bettersight.io/terms 71. Privacy policy and terms pages live at bettersight.io/privacy and /terms 72. Google OAuth consent screen approved (requires privacy policy URL) 73. End to end test with real competitor data 74. Staging deploy 75. Tag version v1.0.0 76. Production deploy via Dokploy ``` --- ## 20. Authentication — PocketBase Native ### Default: Email + Password Standard PocketBase auth. User registers with email and password. PocketBase issues JWT on login. Stored in Pinia auth store. All Flask API calls include `Authorization: Bearer {jwt}` header. ### Option: Google OAuth PocketBase handles the full OAuth flow natively — no custom auth code. **Setup in PocketBase admin:** 1. Settings → Auth Providers → Google → Enable 2. Add Google OAuth Client ID and Secret from Google Cloud Console 3. Set redirect URL to `https://app.bettersight.io/auth/callback` **Frontend flow:** ```javascript // LoginPage.vue — Google OAuth button const loginWithGoogle = async () => { const authData = await pb.collection('users').authWithOAuth2({ provider: 'google' }) // PocketBase returns verified email — same email used for licence validation authStore.setUser(authData) router.push('/account') } ``` **Why this matters for licence validation:** Google OAuth email is verified by Google — same email registered in tenant_seats. Domain matching is perfectly reliable. No manual email entry. No typos. `Session.getActiveUser().getEmail()` in Apps Script returns the exact same email. ### Login page design - Email/password form as default (UForm + UInput + UButton) - "Continue with Google" button below the form (UButton with Google icon) - Trial messaging above the form — reads `?plan=` query param to show correct length: ```javascript // LoginPage.vue const plan = route.query.plan || 'analyse' const TRIAL_COPY = { analyse: '14-day free trial — no credit card required', discover: '21-day free trial — no credit card required', intelligence: '30-day free trial — no credit card required', } const trialCopy = TRIAL_COPY[plan] || TRIAL_COPY.analyse ``` - "Forgot password?" link below the email input field - Link to privacy policy and terms below ### Password Reset Flow PocketBase handles password reset natively via email. The frontend needs two small additions only. **LoginPage.vue — "Forgot password?" link:** ```javascript // Triggers PocketBase native password reset email async function requestPasswordReset() { await pb.collection('users').requestPasswordReset(email.value) toast.success('Reset link sent — check your inbox') } ``` **`/reset-password` route — Vue Router:** ```javascript // Reads token from URL params — PocketBase reset link format: // app.bettersight.io/reset-password?token=xxx async function confirmPasswordReset() { await pb.collection('users').confirmPasswordReset( route.query.token, newPassword.value, newPasswordConfirm.value ) router.push('/login?reset=success') } ``` **Add to Vue Router:** ```javascript { path: '/reset-password', component: ResetPasswordPage } ``` **Add `ResetPasswordPage.vue` to pages:** - New password input + confirm password input - Submit button calls `confirmPasswordReset()` - Redirect to login with success toast on completion - Error state if token is expired or invalid --- ## 21. Professional Onboarding and Brand Touches ### Transactional Emails via Resend All system emails sent via Resend from `alerts@bettersight.io`. Never use raw SMTP for user-facing emails. ```python import resend resend.api_key = os.getenv('RESEND_API_KEY') def send_welcome_email(to_email: str, tenant_name: str, template_url: str): """ Sends branded welcome email on signup with Sheet template download link. Data flow: tenant email + name + template URL → Resend API → inbox """ resend.Emails.send({ "from": "Bettersight ", "to": to_email, "subject": "Welcome to Bettersight — you're all set", "html": render_email_template('welcome', { "name": tenant_name, "template_url": template_url }) }) ``` **Email triggers and templates:** | Trigger | Template | From | |---|---|---| | Signup complete | welcome.html | alerts@bettersight.io | | Trial day 3 | checkin.html | alerts@bettersight.io | | Trial day 7 | feature_tip.html | alerts@bettersight.io | | Weekly brief | brief.html | brief@bettersight.io | | Price alert | alert.html | alerts@bettersight.io | | New product alert | alert.html | alerts@bettersight.io | | Stripe receipt | Stripe native | billing@bettersight.io | All HTML email templates in `backend/templates/emails/`. Every template includes: Bettersight logo, branded header, consistent typography, unsubscribe footer, and link to bettersight.io. ### Trial Periods Trial periods are configured per Stripe price object: ```python TRIAL_PERIODS = { "analyse": 14, # 14 days — low price point, one person decides, value visible in first run "discover": 21, # 21 days — slight committee risk at $499, needs second internal review "intelligence": 30, # 30 days — $999 needs finance approval, signal pipeline needs time to prove value "enterprise": 0, # No trial — custom demo and pilot conversation only } ``` No credit card required to start any trial — configure in Stripe checkout settings (payment_method_collection: 'if_required'). PocketBase tenant status flow: ``` signup → status = 'trial' day 12 (Analyse) / day 19 (Discover) / day 28 (Intelligence) → n8n fires trial_reminder email (2 days before expiry) trial expires → Stripe charges card (if provided) → status = 'active' trial expires → no card provided → status = 'inactive' → access blocked first Stripe charge succeeds → status = 'active' ``` Apps Script and dashboard allow access for both `trial` and `active` status: ```python VALID_STATUSES = ['active', 'trial'] ``` ### Stripe Customer Portal Self-service billing — clients manage their own subscription, upgrade, cancel. No support tickets for billing changes. ```python @app.route('/billing/portal', methods=['POST']) def billing_portal(): """ Creates a Stripe customer portal session for self-service billing. Data flow: tenant_id → PocketBase stripe_customer_id → Stripe portal session → redirect URL returned to frontend """ tenant = tenant_repository.get_by_id(request.json['tenant_id']) session = stripe.billing_portal.Session.create( customer=tenant['stripe_customer_id'], return_url='https://app.bettersight.io/account' ) return jsonify({'url': session.url}) ``` ### Onboarding Checklist Displayed on AccountPage.vue after signup. Tracked in PocketBase tenant record. Updated automatically as each action is completed. Collapses when all steps done. ``` onboarding_checklist (JSON field on tenants) { "add_competitor": false, // set true when first competitor is created "run_analysis": false, // set true when first job completes "template_imported": false, // set true from AccountPage confirmation "sidebar_installed": false, // set true first time validateLicence() succeeds "sync_from_sheet": false, // set true first time pullLatestResults() succeeds "tour_completed": false, // set true when guided tour reaches final step "tour_skipped": false, // set true when PM clicks Skip "tour_step": 0 // last step reached — for resume on return } ``` Each step checked off automatically when the action is completed. Full checklist completion triggers the day-3 check-in email sequence. --- ### Guided Onboarding Tour A 5-step guided tour fires automatically on first login using Shepherd.js. It is the primary first-run experience — more active than the checklist, which is passive. **Install:** ```bash npm install shepherd.js ``` **Tour triggers:** - Fires automatically on first dashboard load when `tour_completed = false AND tour_skipped = false` - "Take the tour again" link on AccountPage for PMs who skipped - Never fires again after `tour_completed = true` - If PM closes the browser mid-tour, `tour_step` stores their last position and the tour resumes from that step on next login **Tour steps:** ``` Step 1 of 5 — Welcome (no anchor element — modal overlay) ┌─────────────────────────────────────────────────┐ │ 👋 Welcome to Bettersight │ │ │ │ Let's get you to your first competitive │ │ analysis in under 5 minutes. │ │ │ │ [Skip tour] [Let's go →] │ └─────────────────────────────────────────────────┘ Step 2 of 5 — Add a competitor (anchors to Competitors sidebar nav item) ┌─────────────────────────────────────────────────┐ │ 🔍 Add your first competitor │ │ │ │ Click Competitors in the sidebar to add the │ │ first company you want to track. │ │ │ │ You'll need their website URL and their │ │ primary operating market (e.g. United Kingdom).│ │ │ │ [← Back] [Go to Competitors →] │ └─────────────────────────────────────────────────┘ → On [Go to Competitors →]: router.push('/competitors') → Tour pauses. Resumes when PM clicks [+ Add Competitor] Step 3 of 5 — Primary market (anchors to primary_market dropdown in add form) ┌─────────────────────────────────────────────────┐ │ 🌍 Set the primary market │ │ │ │ This tells Bettersight which country to │ │ use when scraping — so you see the right │ │ pricing and currency for that market. │ │ │ │ [← Back] [Next →] │ └─────────────────────────────────────────────────┘ Step 4 of 5 — Run analysis (anchors to Analyse sidebar nav item) ┌─────────────────────────────────────────────────┐ │ ▶ Run your first analysis │ │ │ │ Describe the trip you want to research — │ │ destination, duration, and travel style. │ │ │ │ Bettersight finds the matching competitor │ │ pages automatically. No URLs to copy-paste. │ │ │ │ [← Back] [Go to Analyse →] │ └─────────────────────────────────────────────────┘ → On [Go to Analyse →]: router.push('/analyse') Step 5 of 5 — Sync to Sheet (anchors to SyncToSheet component) ┌─────────────────────────────────────────────────┐ │ 📊 Get results into your Sheet │ │ │ │ Once your analysis is complete, copy rows or │ │ open your workbook and click Bettersight → │ │ Pull latest results in the sidebar. │ │ │ │ [← Back] [Done — let's go ✓] │ └─────────────────────────────────────────────────┘ → On [Done]: set tour_completed = true in PocketBase → Tour dismissed permanently ``` **Implementation — `composables/useOnboardingTour.js`:** ```javascript import Shepherd from 'shepherd.js' import 'shepherd.js/dist/css/shepherd.css' import { useRouter } from 'vue-router' import { useTenantStore } from '@/stores/tenant_store' /** * Provides the guided onboarding tour using Shepherd.js. * Fires on first login. Resumes from last step if interrupted. * Marks tour_completed in PocketBase on final step. * * Data flow: * onboarding_checklist.tour_completed → false → * Shepherd.Tour initialised and started → * PM progresses through 5 steps → * final step → PATCH /account/onboarding { tour_completed: true } → * tour never fires again */ export function useOnboardingTour() { const router = useRouter() const tenantStore = useTenantStore() const tour = new Shepherd.Tour({ useModalOverlay: true, defaultStepOptions: { cancelIcon: { enabled: true }, classes: 'bs-tour-step', scrollTo: { behavior: 'smooth', block: 'center' }, when: { cancel: () => markTourSkipped() } } }) tour.addStep({ id: 'welcome', text: ` Welcome to Bettersight 👋

Let's get you to your first competitive analysis in under 5 minutes. `, buttons: [ { text: 'Skip tour', action: tour.cancel, classes: 'bs-btn-ghost' }, { text: "Let's go →", action: tour.next, classes: 'bs-btn-primary' } ] }) tour.addStep({ id: 'add-competitor', attachTo: { element: '[data-tour="nav-competitors"]', on: 'right' }, text: ` Add your first competitor 🔍

Click Competitors to add the first company you want to track. You'll need their website URL and primary operating market. `, buttons: [ { text: '← Back', action: tour.back, classes: 'bs-btn-ghost' }, { text: 'Go to Competitors →', action: () => { router.push('/competitors') saveTourStep(2) tour.next() }, classes: 'bs-btn-primary' } ] }) tour.addStep({ id: 'primary-market', attachTo: { element: '[data-tour="primary-market-field"]', on: 'bottom' }, text: ` Set the primary market 🌍

This tells Bettersight which country to use when scraping — so you see the right pricing and currency for that market. `, buttons: [ { text: '← Back', action: tour.back, classes: 'bs-btn-ghost' }, { text: 'Next →', action: () => { saveTourStep(3) tour.next() }, classes: 'bs-btn-primary' } ] }) tour.addStep({ id: 'run-analysis', attachTo: { element: '[data-tour="nav-analyse"]', on: 'right' }, text: ` Run your first analysis ▶

Describe the trip you want to research — destination, duration, and travel style. Bettersight finds matching competitor pages automatically. No URLs needed. `, buttons: [ { text: '← Back', action: tour.back, classes: 'bs-btn-ghost' }, { text: 'Go to Analyse →', action: () => { router.push('/analyse') saveTourStep(4) tour.next() }, classes: 'bs-btn-primary' } ] }) tour.addStep({ id: 'sync-to-sheet', attachTo: { element: '[data-tour="sync-to-sheet"]', on: 'top' }, text: ` Get results into your Sheet 📊

Copy rows to the clipboard from here, or open your workbook and click Bettersight → Pull latest results in the sidebar. `, buttons: [ { text: '← Back', action: tour.back, classes: 'bs-btn-ghost' }, { text: 'Done — let\'s go ✓', action: () => { markTourCompleted() tour.complete() }, classes: 'bs-btn-primary' } ] }) async function saveTourStep(step) { await tenantStore.updateOnboarding({ tour_step: step }) } async function markTourCompleted() { await tenantStore.updateOnboarding({ tour_completed: true, tour_step: 5 }) } async function markTourSkipped() { await tenantStore.updateOnboarding({ tour_skipped: true }) } function shouldShowTour(onboarding) { return !onboarding.tour_completed && !onboarding.tour_skipped } function getResumeStep(onboarding) { return onboarding.tour_step || 0 } return { tour, shouldShowTour, getResumeStep } } ``` **Trigger in `ActivityPage.vue` (first page after login):** ```javascript // ActivityPage.vue —