# CLAUDE.md — Bettersight Competitive Intelligence SaaS > This file is the authoritative specification for Claude Code building Bettersight. > Read every section before writing any code. Never skip a section. > When in doubt, re-read this file. --- ## 1. Project Overview Bettersight is a vertical competitive intelligence SaaS built specifically for adventure travel operators. It automates competitor price tracking, trip discovery, and market analysis — replacing hours of manual research with automated intelligence delivered to tools PMs already use. **Tagline:** Better sight, better moves. **Secondary tagline:** Better sight, better decisions. **Domain:** bettersight.io **Brand position:** The only competitive intelligence platform that understands adventure travel — not just that a page changed, but what trip was added, what price moved, and what it means for your catalogue. **The anchor feature** is an on-demand competitor analysis Google Sheet add-on that runs structured AI extraction against competitor trip pages and writes results directly into the client's spreadsheet. Everything else supports and extends this core. **Primary reference client:** G Adventures (hackathon origin) **Target market:** Adventure travel operators — product managers, revenue managers, strategy leads --- ## 2. Business Context ### Pricing Tiers | Tier | Price | Seats | Trial | |---|---|---|---| | Analyse | $349/month | 5 seats | 14 days — no credit card required | | Discover | $499/month | 8 seats | 21 days — no credit card required | | Intelligence | $999/month | 10 seats | 30 days — no credit card required | | Enterprise | $1,499/month | Unlimited | No trial — custom demo and pilot | Additional seats: $45/seat/month on any tier. Enterprise tier is not publicly listed — available on request only. The $199 Monitor tier has been dropped entirely. ### Tier Feature Summary **Analyse — $349/month (MVP launch tier)** - Dashboard as the primary interface — analysis runs, results viewed, push to Sheet - Trip-intent matching — PM enters destination + duration + style, system finds the right competitor trip URL automatically via Firecrawl /map (on demand) - URL confirmation screen before scrape runs — PM can swap any auto-matched URL - On-demand competitor analysis with side-by-side dashboard results view - Push to Google Sheet — one button exports results to PM's chosen workbook tab - Auto-generated battlecards updated after each scrape - Comparable trip discovery (semantic similarity matching) - In-dashboard notifications — price changes, new products, alerts in activity feed - Weekly Monday HTML email brief via Resend - Fast path (PocketBase cache) / refresh path (live scrape) on-demand analysis - Firecrawl /monitor for real-time change detection - Firecrawl /map on demand — for trip-intent URL matching per analysis job only **Discover — $499/month** - Everything in Analyse - Firecrawl /map scheduled — weekly full catalogue crawl per competitor → Proactive surveillance, not reactive search → Finds new trips the PM never knew to look for → New URLs auto-queued for scraping Sunday night without PM involvement - New trip detection — flags trips added since last crawl - Catalogue completeness — how many trips per competitor per destination - Destination gap analysis — destinations competitors cover that you don't - Sunday night discovery workflow (n8n) - Monday brief includes all auto-discovered new products - External notification integrations (PM connects one or more): → Slack webhook — real-time alerts into any channel → Microsoft Teams webhook — for Microsoft 365 organisations → WhatsApp via Twilio — alerts to PM's phone number **Intelligence — $999/month** - Everything in Discover - Full OSINT signal pipeline (15+ sources — see Section 23) - Competitor Trajectory Score (weekly) - Wayback Machine positioning history tracking - Network mapping (leadership, board, investor connections) - Monthly narrative intelligence report with predicted moves and confidence ratings - Early warning alerts on significant signal clusters **Enterprise — $1,499/month (unlisted)** - Everything in Intelligence - Dedicated onboarding and account management - Custom signal sources on request - Quarterly strategy briefing call - SLA on scrape freshness and alert delivery - Custom PDF branded reports ### Tier Pitch Distinction **Analyse** — "Tell me about this specific trip across these competitors" PM drives every research job. They describe their product, system finds the matching competitor trips, PM confirms and runs. Reactive intelligence. **Discover** — "Tell me everything that changed this week that I didn't know to ask about" System runs autonomously. PM wakes Monday to a brief that includes trips they never knew existed. Catches unknown unknowns. Proactive intelligence. **Intelligence** — "Tell me what my competitors are about to do" System aggregates signals across 15+ public sources and forecasts competitor moves 3-6 months ahead. Predictive intelligence. ### MVP Scope (build this first) Single tier at $349/month (Analyse). Five features only: 1. Dashboard — primary interface for running analysis and viewing results 2. Trip-intent matching — Firecrawl /map on demand finds competitor trip URLs 3. On-demand competitor analysis with dashboard results view + push to Sheet 4. Auto-generated battlecards 5. In-dashboard notifications (activity feed) + Monday HTML email brief via Resend Comparable trip discovery (semantic similarity) is a Phase 2 addition within the Analyse tier — build after core analysis flow is working. Do not build Discover, Intelligence, or Enterprise tier features until Analyse tier is live with paying clients. --- ## 3. Tech Stack — Authoritative **Never deviate from this stack without explicit instruction.** | Layer | Tool | Notes | |---|---|---| | Infrastructure | Hetzner VPS | Existing | | Orchestration | Dokploy | Existing | | Reverse proxy | Pangolin | Existing | | Networking | Docker Compose | pangolin_default network | | Backend API | Flask | Python only. No FastAPI. | | Scraping engine | app.py (Flask) | Existing — do not rewrite | | Job queue | RQ (Redis Queue) | Async job processing | | Queue broker | Redis | Shared with Firecrawl | | Monitoring | Firecrawl self-hosted | /monitor — real-time change detection (all tiers) | | Trip-intent matching | Firecrawl self-hosted | /map on demand — finds competitor trip URLs (Analyse+) | | Discovery | Firecrawl self-hosted | /map scheduled — weekly catalogue crawl (Discover+ only) | | Change detection | Firecrawl Monitor + PocketBase flags | Event-driven — fires only on detected change | | Proxy — primary | Webshare | ~$3/month, free tier available, used by default | | Proxy — fallback | Bright Data | Pay-as-you-go, auto-escalated per domain when Webshare blocked | | AI extraction | OpenRouter | Free tier model cycling with claude-haiku-4-5 paid fallback | | Embeddings | text-embedding-3-small | Via OpenRouter | | Database | PocketBase | Multi-tenant + auth | | DB replication | Litestream | Real-time SQLite WAL streaming to Hetzner S3 — near-zero RPO | | Auth | PocketBase native | Email/password (default) + Google OAuth | | API rate limiting | Flask-Limiter | Protects against queue abuse and runaway scripts | | Workflow automation | n8n | Self-hosted | | Frontend | Vite + Vue 3 | Minimal dashboard only — no Nuxt runtime | | UI component library | Nuxt UI v4 (free, MIT) | Same library as Ledra/AccountFlow | | Base template | nuxt-ui-templates/dashboard-vue | Clone as starting point | | Routing | Vue Router v4 | Client-side SPA routing | | State management | Pinia | Lightweight store | | Date handling | date-fns | Formatting and computation | | Validation | Zod | Schema validation | | Sheet add-on | Google Apps Script | Export integration only — push results from dashboard to Sheet | | User notifications | In-dashboard activity feed | MVP — price changes, new products, alerts | | User notifications (future) | Slack / Teams / WhatsApp | Discover tier — webhook integrations | | Owner monitoring alerts | Gotify + iGotify | Self-hosted — owner phone only, never shown to clients | | Weekly brief | Resend HTML email | Replaces WeasyPrint — no PDF, no Docker dependency | | Payments | Stripe | Subscription management + customer portal | | Support tickets | Freescout (self-hosted) | Email-to-ticket — support@bettersight.io → Freescout | --- ## 4. Architecture ### Service Map ``` Bettersight Dashboard — PRIMARY INTERFACE (app.bettersight.io) → PM enters: destination + duration + travel style + selected competitors → Flask POST /research/find-urls → Firecrawl /map per competitor (on demand, <5s each) → URL scoring returns best match per competitor → Confirmation screen — PM confirms or swaps any URL → PM confirms → Flask POST /research → RQ job queued → job_id returned → Dashboard polls /research/status/{job_id} every 3 seconds → On-demand check flow per competitor: Path 1 FAST — last_scraped < 24hrs AND change_detected = false → PocketBase structured data → LLM analysis narrative (~seconds) Path 2 REFRESH — last_scraped > 24hrs OR change_detected = true → RQ job → app.py Playwright scrape → PocketBase → LLM (~2 mins) → Results displayed in dashboard side-by-side comparison view → PM clicks "Push to Sheet" → Apps Script writes to selected workbook tab Apps Script (standalone — export integration only) → getWorkbookTabs() — returns PM's workbook tab list for dashboard dropdown → pushToSheet(results, tabName) — writes dashboard results to Sheet on demand → validateLicence() — confirms PM is authorised → Three functions only. No analysis UI in the script. Firecrawl Monitor (event-driven, runs continuously) → Watches all active competitor URLs → Detects content change (JS-rendered pages, React/Vue SPAs supported) → Fires webhook → Flask /internal/monitor-webhook → Sets competitor.change_detected = true in PocketBase n8n (scheduled workflows) → Daily cron (6am) — scrapes changed/stale competitors → Resets change_detected = false after each scrape → PocketBase webhook on price_history insert → creates alert record in PocketBase → PocketBase webhook on products insert (is_new=true) → creates alert record in PocketBase → n8n reads new alert records → sends Resend email notification (daily digest) → Monday 6am → brief_service.py → Resend HTML email → [Discover tier] Sunday night → Firecrawl /map scheduled → full catalogue crawl Vue 3 dashboard auth + account → PocketBase auth (Google OAuth or email/password) → Stripe (payment, subscription status) → On competitor add → Flask registers with Firecrawl Monitor automatically ``` ### Change Detection — Firecrawl Monitor Firecrawl Monitor is the change detection layer. It replaces both changedetection.io (previously removed) and the plain HTTP hash approach (rejected due to JS-heavy page limitations). **Why Firecrawl Monitor over alternatives:** - Renders JS fully before hashing — works on React/Vue SPAs - Event-driven — fires only on change, not on a polling schedule - Lower server load than a 2-4hr Playwright cron - Single service handles both monitoring (/monitor) and discovery (/map for Intelligence tier) - Already in stack — no new infrastructure **PocketBase competitor record flags:** ``` change_detected bool — set true by monitor webhook, reset after scrape change_detected_at date — timestamp of last detected change last_scraped date — timestamp of last completed scrape ``` **Monitor registration — on competitor add:** ```python def register_competitor_monitor(competitor_id: str, url: str): """ Registers a competitor URL with Firecrawl Monitor on first add. Stores the monitor ID in PocketBase for future deregistration. Data flow: competitor url → Firecrawl POST /v1/monitor → monitor_id returned → stored in competitors.firecrawl_monitor_id """ response = requests.post( "http://firecrawl:3002/v1/monitor", json={ "url": url, "webhook": "https://api.bettersight.io/internal/monitor-webhook", "changeTypes": ["content"] } ) monitor_id = response.json().get("id") competitor_repository.update(competitor_id, { "firecrawl_monitor_id": monitor_id }) ``` **Monitor webhook handler:** ```python @app.route('/internal/monitor-webhook', methods=['POST']) def monitor_webhook(): """ Receives Firecrawl Monitor webhook on competitor page change. Sets change_detected flag on the competitor record. Data flow: Firecrawl webhook payload → extract competitor URL → PocketBase competitor lookup by URL → change_detected = true, change_detected_at = now """ data = request.json url = data.get('url') competitor = competitor_repository.get_by_url(url) if competitor: competitor_repository.update(competitor['id'], { 'change_detected': True, 'change_detected_at': datetime.now().isoformat() }) return jsonify({'status': 'ok'}) ``` ### Docker Network All services on `pangolin_default` network. Internal service calls use container names, never localhost or host IPs. ``` flask-api → http://flask-api:5000 bettersight-worker → connects to redis://bettersight-redis:6379 bettersight-redis → redis://bettersight-redis:6379 firecrawl → http://firecrawl:3002 pocketbase → http://pocketbase:8090 ``` ### Compose Services ```yaml services: flask-api: build: ./backend container_name: flask-api command: gunicorn app:app -w 4 -b 0.0.0.0:5000 env_file: .env networks: - pangolin_default restart: unless-stopped depends_on: - bettersight-redis bettersight-worker: build: ./backend container_name: bettersight-worker command: rq worker high normal low -u redis://bettersight-redis:6379 env_file: .env networks: - pangolin_default restart: unless-stopped deploy: replicas: 3 depends_on: - bettersight-redis bettersight-redis: image: redis:alpine container_name: bettersight-redis networks: - pangolin_default restart: unless-stopped firecrawl: image: ghcr.io/mendableai/firecrawl:latest container_name: firecrawl environment: - REDIS_URL=redis://bettersight-redis:6379 - USE_DB_AUTHENTICATION=false - PORT=3002 networks: - pangolin_default restart: unless-stopped depends_on: - bettersight-redis freescout: image: freescout/freescout:latest container_name: freescout environment: - APP_URL=https://support.bettersight.io - MAIL_FROM=support@bettersight.io networks: - pangolin_default restart: unless-stopped # Pangolin routes support.bettersight.io → freescout:80 litestream: image: litestream/litestream:latest container_name: litestream command: replicate volumes: - pocketbase_data:/pb_data # same volume as PocketBase environment: - LITESTREAM_ACCESS_KEY_ID=${HETZNER_S3_ACCESS_KEY} - LITESTREAM_SECRET_ACCESS_KEY=${HETZNER_S3_SECRET_KEY} configs: - source: litestream_config target: /etc/litestream.yml networks: - pangolin_default restart: unless-stopped depends_on: - pocketbase # litestream.yml config: # dbs: # - path: /pb_data/data.db # replicas: # - url: s3://bettersight-backups/litestream # endpoint: https://fsn1.your-objectstorage.com networks: pangolin_default: external: true volumes: pocketbase_data: ``` **Litestream recovery procedure:** ```bash # Restore PocketBase database from Litestream replica docker run --rm \ -e LITESTREAM_ACCESS_KEY_ID=${HETZNER_S3_ACCESS_KEY} \ -e LITESTREAM_SECRET_ACCESS_KEY=${HETZNER_S3_SECRET_KEY} \ -v ./pb_data:/pb_data \ litestream/litestream restore \ -o /pb_data/data.db \ s3://bettersight-backups/litestream ``` Recovery point objective (RPO): seconds — not 24 hours. Recovery time objective (RTO): under 5 minutes from backup restore to running. --- ## 5. Project Structure ``` bettersight/ ├── backend/ │ ├── core/ │ │ ├── scraper.py # Playwright stealth browser — never touches industry logic │ │ ├── extractor.py # OpenRouter model cycling, JSON parsing, fallback chain │ │ ├── ai_fetch.py # AI web fetch fallback │ │ └── proxy_service.py # Proxy provider — swap here to change from Bright Data │ ├── industries/ │ │ └── adventure_travel/ │ │ ├── fields.py # STANDARD_FIELDS, NGS_FIELDS column maps │ │ ├── prompt.py # build_prompt(), inject_page_content() │ │ └── schema.py # JSON extraction schema for this vertical │ ├── config/ │ │ └── client.py # SHEET_ID, tab names, column assignments │ ├── services/ │ │ ├── licence_service.py # Domain + seat validation (no sheet_id) │ │ ├── trip_finder_service.py # find_competitor_trip_url(), score_and_rank_urls() │ │ ├── battlecard_service.py │ │ ├── embedding_service.py │ │ ├── brief_service.py # HTML email brief generation │ │ └── alert_service.py │ ├── repositories/ │ │ ├── tenant_repository.py │ │ ├── product_repository.py │ │ ├── seat_repository.py │ │ ├── scrape_repository.py │ │ └── proxy_repository.py │ ├── templates/ │ │ └── emails/ │ │ ├── welcome.html │ │ ├── trial_reminder.html │ │ ├── brief.html │ │ ├── price_alert.html │ │ └── new_product.html │ ├── api.py # Flask routes │ ├── jobs.py # RQ job functions │ ├── app.py # Entry point — imports and wires everything │ ├── requirements.txt │ └── Dockerfile ├── frontend/ # Vite + Vue 3 + Nuxt UI v4 dashboard │ ├── src/ │ │ ├── pages/ │ │ │ ├── LoginPage.vue # Auth — email/password + Google OAuth + forgot password │ │ │ ├── ResetPasswordPage.vue # /reset-password?token=xxx — PocketBase token confirm │ │ │ ├── AccountPage.vue # Subscription, seats, upgrade prompt, template download │ │ │ ├── CompetitorsPage.vue # Add/edit competitors with status badges + CSV upload │ │ │ ├── AnalysePage.vue # CORE PAGE — full analysis workflow │ │ │ ├── ActivityPage.vue # Market movement feed + alerts │ │ │ └── BattlecardsPage.vue # Battlecard view per competitor │ │ ├── components/ │ │ │ ├── DotGrid.vue # Reusable dot grid SVG — signature element │ │ │ ├── analyse/ │ │ │ │ ├── TripIntentForm.vue # Destination + duration + style + competitor checkboxes │ │ │ │ ├── UrlConfirmation.vue # Confirm or swap matched URLs before run │ │ │ │ ├── JobProgress.vue # Per-competitor real-time status during job │ │ │ │ ├── ResultsTable.vue # Side-by-side competitor comparison │ │ │ │ ├── ConfidenceBar.vue # Per-competitor confidence indicator │ │ │ │ ├── PushToSheet.vue # Tab selector + push button │ │ │ │ └── RunHistory.vue # Last 5 runs with restore option │ │ │ ├── activity/ │ │ │ │ ├── FeedItem.vue # Single activity feed row with dot grid │ │ │ │ └── AlertCard.vue # Alert card with dot grid │ │ │ ├── shared/ │ │ │ │ ├── StatCard.vue # Stat card with accent bar + dot grid │ │ │ │ ├── SeatManager.vue # Add/remove seats table │ │ │ │ ├── CompetitorForm.vue # Add/edit competitor modal form │ │ │ │ ├── BattlecardCard.vue # Single battlecard display │ │ │ │ ├── SubscriptionCard.vue │ │ │ │ ├── CsvUpload.vue # File picker + PapaParse + row validation │ │ │ │ └── ImportPreviewModal.vue # Valid/invalid preview before bulk submit │ │ ├── composables/ │ │ │ ├── useAuth.js │ │ │ ├── useTenant.js │ │ │ ├── useAnalysis.js # Analysis job lifecycle management │ │ │ ├── useCompetitors.js │ │ │ └── useOnboardingTour.js # Shepherd.js guided tour — 5 steps │ │ ├── assets/ │ │ │ └── tour.css # Custom Shepherd.js styles matching design system │ │ ├── services/ │ │ │ └── api_service.js # All Flask API calls │ │ ├── repositories/ │ │ │ ├── tenant_repository.js │ │ │ ├── seat_repository.js │ │ │ ├── competitor_repository.js │ │ │ └── analysis_repository.js # Research jobs, history, results │ │ ├── stores/ │ │ │ ├── auth_store.js # Pinia — JWT, email, tier │ │ │ └── analysis_store.js # Pinia — active job, results, history │ │ ├── router/ │ │ │ └── index.js │ │ └── main.js │ ├── package.json │ ├── vite.config.js │ └── index.html ├── appscript/ # Export integration only — 3 files │ ├── Code.gs # onOpen(), install entry point │ ├── Validation.gs # validateLicence() — email only │ ├── Export.gs # pushToSheet(), getWorkbookTabs(), detectColumns() │ └── Sidebar.html # Minimal UI — tab selector + push button only ├── n8n/ │ └── workflows/ # n8n workflow JSON exports ├── docker-compose.yml ├── .env.example └── CLAUDE.md ``` --- ## 5b. Frontend Setup — Nuxt UI v4 ### Bootstrap ```bash # Clone the dashboard-vue base template git clone https://github.com/nuxt-ui-templates/dashboard-vue frontend cd frontend npm install # Additional dependencies npm install pinia date-fns zod axios ``` ### Key dependencies (package.json) ```json { "dependencies": { "@nuxt/ui": "^4.0.0", "vue": "^3.4.0", "vue-router": "^4.0.0", "pinia": "^2.0.0", "date-fns": "^3.0.0", "zod": "^3.0.0", "axios": "^1.6.0" }, "devDependencies": { "vite": "^5.0.0", "@vitejs/plugin-vue": "^5.0.0" } } ``` ### Nuxt UI v4 component usage Use Nuxt UI components throughout — never build custom UI primitives. Key components for this dashboard: ``` UButton — all buttons UInput — all text inputs UCard — content containers UTable — seat management, competitor list UBadge — subscription status, tier labels UModal — add competitor, add seat dialogs UForm — all forms with Zod validation UAlert — error and success messages UAvatar — user display UDropdown — action menus UDivider — section separators UContainer — page layout wrapper USkeleton — loading states UNotification — toast notifications ``` ### Design tokens — align with Ledra/AccountFlow ```javascript // app.config.ts or equivalent export default { ui: { primary: 'indigo', gray: 'cool' } } ``` ### Separation of concerns — frontend ``` Pages (src/pages/) → orchestrate composables and repositories → NO direct API calls → NO business logic Composables (src/composables/) → reusable stateful logic → calls repositories → NO direct API calls Repositories (src/repositories/) → all API calls via api_service.js → returns plain data objects → NO component or store references Stores (src/stores/) → global state only (auth, tenant) → NO API calls directly → calls repositories for data Services (src/services/) → api_service.js is the single Axios instance → all endpoints configured here → NO business logic ``` ### Six pages — full dashboard **LoginPage.vue** — email/password default + Google OAuth option **AccountPage.vue** - Subscription status card (tier, renewal date, seat count, trial countdown) - **Upgrade prompt card** — shown below subscription for Analyse and Discover tier clients: ``` ┌────────────────────────────────────────────────┐ │ 🚀 Upgrade to Discover │ │ │ │ + Weekly catalogue auto-discovery │ │ + Spot new competitor trips automatically │ │ + Slack, Teams and WhatsApp alerts │ │ │ │ $499/month · 3 more seats · 21-day trial │ │ │ │ [Upgrade now →] │ └────────────────────────────────────────────────┘ ``` → [Upgrade now] creates a Stripe checkout session for the next tier → On successful upgrade: PocketBase tier updated via Stripe webhook → Upgrade card hidden for Intelligence and Enterprise clients - Download Sheet template button - Seat management table (add/remove seats) - Stripe customer portal link (manage billing, cancel) - Onboarding checklist (tracked in PocketBase) - "↺ Take the guided tour again" link **CompetitorsPage.vue** - Table of active competitors with status badges (✓ Current, ⚠ Changed, ○ Stale) - Primary market flag per competitor - Add competitor modal — name, website, catalogue URL, primary market (required) - On add → auto-registers with Firecrawl Monitor - Edit/deactivate actions - [Download CSV template] button — pre-formatted with correct headers - [Upload CSV] button — bulk import with preview and validation **AnalysePage.vue — THE CORE PAGE** - TripIntentForm — destination, duration, travel style, competitor checkboxes - [Find & Analyse] → POST /research/find-urls → UrlConfirmation screen - UrlConfirmation — matched URL per competitor, swap any URL, manual entry for no-match - [Confirm & Run] → POST /research → job_id → polling begins - JobProgress — per-competitor real-time status (queued / scraping / done / failed) - ResultsTable — side-by-side competitor comparison on completion Multi-currency display rule: → Show competitor's primary market currency as hero price (GB competitor = GBP hero, US competitor = USD hero) → Always show USD alongside in secondary muted text → Format: "£2,190 $2,780 USD" → Never currency-convert — use extracted values only → If primary market currency not extracted, fall back to USD only - ConfidenceBar — per-competitor AI confidence with low-confidence warning - RunHistory — last 5 runs, restore any run to current results view - PushToSheet — tab selector dropdown + push button (calls /sheet/push) **ActivityPage.vue** - Stat cards (price changes, drops, new products, competitors tracked) - Market movement feed — all competitor events this week with dot grid rows - Alerts panel — unread alerts with badge count in sidebar nav - Price position progress bars - Mark as read / dismiss per alert - Bell icon in topbar shows unread count **BattlecardsPage.vue** - Battlecard per competitor — positioning summary, strengths, weaknesses, pricing observation - Last updated timestamp - Topic pills (destinations, warnings, signals) - Comparable trips section (semantic matches) ### tenants ``` id auto name text email text (admin email from signup) domain text (e.g. gadventures.com — used for seat validation) tier select [analyse, discover, intelligence, enterprise] status select [trial, active, inactive] trial_ends_at date (set on signup based on tier trial period) max_seats integer (set from TIER_SEATS on signup) timezone text (IANA timezone e.g. Europe/London — captured on signup) stripe_customer_id text stripe_subscription_id text onboarding_checklist json (tracks completion of onboarding steps) data_region text (default: 'global' — for future GDPR regional routing) created autodate ``` Note: `sheet_id` binding removed — standalone Apps Script model, validation is email-only. `monitor` tier removed — dropped entirely. **Timezone capture on signup:** ```javascript // Captured automatically from browser on account creation // Never ask the PM to select a timezone manually const timezone = Intl.DateTimeFormat().resolvedOptions().timeZone // e.g. "Europe/London", "America/New_York", "Australia/Sydney" // Stored in tenants.timezone on first signup POST ``` ### tenant_seats ``` id auto tenant_id relation → tenants email text active bool last_accessed date added_by text created autodate ``` ### competitors ``` id auto tenant_id relation → tenants name text website url catalogue_url url primary_market text (ISO country code e.g. GB, US, AU — required on add) firecrawl_monitor_id text (returned by Firecrawl on monitor registration) change_detected bool (set true by monitor webhook, reset after scrape) change_detected_at date (timestamp of last detected change) active bool last_scraped date created autodate ``` ### products ``` id auto tenant_id relation → tenants competitor_id relation → competitors trip_name text url url destination text duration_days integer majority_price_usd number price_low_usd number price_high_usd number aud_price number (nullable) cad_price number (nullable) eur_price number (nullable) gbp_price number (nullable) date_used text seasonality text group_size integer meals integer service_level text activities text start_location text end_location text target_audience text hotels text comments text relevancy integer trip_code text is_new bool first_seen autodate last_seen date is_active bool embedding json (vector array for similarity matching) ``` ### products_ngs_meta ``` id auto product_id relation → products exclusive_access text group_leader text sustainability text ``` ### price_history ``` id auto tenant_id relation → tenants product_id relation → products majority_price_usd number price_low_usd number price_high_usd number aud_price number (nullable) cad_price number (nullable) eur_price number (nullable) gbp_price number (nullable) date_used text change_field text (which price field changed) change_amount number change_percent number scraped_at autodate ``` ### scrape_runs ``` id auto (also used as RQ job_id) tenant_id relation → tenants competitor_id relation → competitors (nullable for bulk runs) triggered_by select [cron, webhook, manual, sheet] status select [pending, running, complete, failed] competitors_total integer competitors_done integer results json error_log text started_at autodate completed_at date ``` ### alerts ``` id auto tenant_id relation → tenants competitor_id relation → competitors product_id relation → products (nullable) alert_type select [price_change, new_product, product_removed, new_comparable] message text change_amount number (nullable) change_percent number (nullable) delivered_dashboard bool (always true — alert always appears in dashboard) delivered_email bool (set true when included in daily digest email) delivered_slack bool (Discover tier — set true when Slack webhook fires) delivered_teams bool (Discover tier — set true when Teams webhook fires) delivered_whatsapp bool (Discover tier — set true when WhatsApp fires) read bool (PM marks as read in dashboard) created autodate ``` ### battlecards ``` id auto tenant_id relation → tenants competitor_id relation → competitors content text (AI generated battlecard text) content_json json (structured: top_routes, avg_price, positioning, strengths, weaknesses) generated_at autodate ``` ### comparable_matches ``` id auto tenant_id relation → tenants client_product text (client trip name — not a PocketBase relation) competitor_product relation → products similarity_score number price_difference number duration_difference integer first_matched autodate last_matched date dismissed bool ``` ### tenant_notifications ``` id auto tenant_id relation → tenants slack_webhook_url text (nullable — Discover tier only) teams_webhook_url text (nullable — Discover tier only) whatsapp_number text (nullable — Discover tier only) email_digest bool (default true — daily digest email) email_digest_time text (default "18:00" — local time in tenant timezone) created autodate ``` **Timezone-aware digest scheduling in n8n:** ``` Daily digest n8n workflow: → Query all active tenants with email_digest = true → For each tenant: Convert 18:00 tenant local time → UTC using tenants.timezone If current UTC time matches → send digest for this tenant → Each tenant receives their digest at 6pm their local time ``` n8n Code node timezone conversion: ```javascript // n8n Code node — per tenant digest time calculation const { DateTime } = require('luxon') const localTime = '18:00' const tenantTz = tenant.timezone || 'UTC' const sendAt = DateTime.fromFormat(localTime, 'HH:mm', { zone: tenantTz }) .toUTC() // Compare sendAt.hour and sendAt.minute against current UTC time ``` ### digest_logs ``` id auto tenant_id relation → tenants digest_type select [weekly, monthly] pdf_path text sent_at autodate recipient_email text status select [sent, failed] ``` ### proxy_overrides ``` id auto domain text (e.g. intrepidtravel.com) provider select [webshare, brightdata] blocked_count integer last_blocked date created autodate ``` ### client_trips ``` id auto tenant_id relation → tenants trip_name text destination text duration_days integer price_usd number activities text embedding json created autodate ``` --- ## 7. On-Demand Analysis — Check Flow This is the core data flow for every on-demand analysis request from the Sheet. It determines whether to serve from PocketBase (fast) or trigger a live scrape (slow). The PM always gets fresh, relevant data. The system decides the path invisibly. ``` PM triggers analysis in Sheet sidebar │ ▼ For each selected competitor: │ ├── Check PocketBase competitor record │ last_scraped < 24hrs? AND change_detected = false? │ │ │ ├── YES → FAST PATH (~seconds) │ │ Pull structured data from PocketBase products table │ │ Pass JSON to LLM for analysis narrative only (no extraction) │ │ Write results to Sheet │ │ Show: ✅ Current (updated {time}) │ │ │ └── NO → REFRESH PATH (~2 mins) │ last_scraped > 24hrs OR change_detected = true │ Trigger RQ job → Playwright scrape → AI extraction │ Update PocketBase (products, price_history) │ Reset change_detected = false │ Update last_scraped = now │ Pass fresh data to LLM for analysis narrative │ Write results to Sheet │ Show: 🔄 Refreshed just now │ ▼ Sheet sidebar status indicators: ✅ Current (updated today 6:02am) — fast path served ⚠️ Change detected (2hrs ago) — triggers refresh automatically 🔄 Refreshing... — slow path in progress ✓ Complete — results written to Sheet ``` ### Fast path — LLM role In the fast path the LLM does NOT extract data — it analyses it. PocketBase already has structured JSON. The LLM receives that JSON and generates the analysis narrative, comments, and battlecard content only. ```python def analyse_from_cache(tenant_id: str, competitor_id: str, context: dict) -> dict: """ Fast path analysis — uses cached PocketBase data instead of live scrape. LLM receives structured JSON and generates narrative analysis only. Data flow: competitor_id → PocketBase products (latest) → structured JSON + client trip context → LLM prompt (analysis only, no extraction) → narrative + comments + relevancy score returned """ ``` ### Staleness threshold 24 hours is the default staleness threshold. This is a configurable constant — adjust if client feedback indicates they need fresher data or if server load requires a longer window: ```python CACHE_STALENESS_HOURS = 24 # in core/scraper.py — adjust here only ``` --- ## 8. Trip-Intent Matching ### Overview The PM never manually enters a competitor trip URL. They describe their own product — destination, duration, travel style — and the system finds the best matching trip page on each competitor site automatically via Firecrawl /map with scoring. This feature is available in the **Analyse tier** (on demand, per job). The **Discover tier** extends this with a scheduled weekly crawl. ### Dashboard Flow ``` PM fills in the Analyse form: Product name: Inca Trail Classic Destination: Peru Duration: 15 days Travel style: Classic Competitors: ☑ Intrepid ☑ Exodus ☑ Flash Pack ☑ Kimkim PM clicks [Find & Analyse] → Flask POST /research/find-urls → Firecrawl /map runs per competitor simultaneously (async) → URL scoring selects best match per competitor Confirmation screen shown in dashboard: Intrepid Travel → intrepidtravel.com/peru-classic-15-days [✓ Use] [↗ Swap] Exodus Travels → exodustravels.com/machu-picchu-trek [✓ Use] [↗ Swap] Flash Pack → flashpack.com/peru-adventure-15d [✓ Use] [↗ Swap] Kimkim → No match found [Enter URL manually] PM confirms (or swaps any URL) → Flask POST /research with confirmed URLs → RQ job queued → analysis runs → Results displayed in dashboard ``` ### Flask Route ```python @app.route('/research/find-urls', methods=['POST']) def find_urls(): """ Finds the best matching competitor trip URL for each selected competitor given the PM's research intent (destination, duration, travel style). Runs Firecrawl /map per competitor concurrently. Body: { destination: str, duration: int, travel_style: str, competitor_ids: list[str] } Returns: { matches: [ { competitor_id, competitor_name, url, confidence, found } ] } Data flow: competitor_ids → PocketBase (get root URLs per competitor) → Firecrawl /map per competitor (concurrent, ThreadPoolExecutor) → score_and_rank_urls() per competitor → return best match per competitor with confidence score """ ``` ### URL Finding Service — `services/trip_finder_service.py` ```python def find_competitor_trip_url(root_url: str, destination: str, duration: int, travel_style: str) -> dict: """ Uses Firecrawl /map with search terms to find the most likely trip page URL on a competitor site matching the PM's research intent. Data flow: root_url + search terms → Firecrawl /map → filtered URL list → score_and_rank_urls() → { url, confidence, found } returned """ response = requests.post( "http://firecrawl:3002/v1/map", json={ "url": root_url, "search": f"{destination} {duration} days {travel_style} tour" }, timeout=15 ) urls = response.json().get("links", []) if not urls: return {"url": None, "confidence": 0, "found": False} return score_and_rank_urls(urls, destination, duration) def score_and_rank_urls(urls: list, destination: str, duration: int) -> dict: """ Scores candidate URLs by keyword match against destination and duration. Returns highest scoring URL with confidence score. Scoring rules: +1 per destination keyword found in URL +2 if duration (as integer) appears in URL +1 if travel category words appear (classic, trek, trail, etc.) Confidence = top_score / max_possible_score, capped at 1.0 If confidence < 0.3, returns found=False — prompt manual entry. """ dest_keywords = destination.lower().split() category_keywords = ['classic', 'trek', 'trail', 'safari', 'adventure', 'explorer', 'journey'] scores = [] for url in urls: url_lower = url.lower() score = 0 score += sum(1 for kw in dest_keywords if kw in url_lower) if str(duration) in url_lower: score += 2 score += sum(1 for kw in category_keywords if kw in url_lower) scores.append((score, url)) scores.sort(reverse=True) if not scores or scores[0][0] == 0: return {"url": None, "confidence": 0, "found": False} top_score, top_url = scores[0] max_possible = len(dest_keywords) + 2 + 1 confidence = min(top_score / max_possible, 1.0) return { "url": top_url, "confidence": round(confidence, 2), "found": confidence >= 0.3 } ``` ### Concurrent Execution Run /map calls for all competitors simultaneously — not sequentially: ```python from concurrent.futures import ThreadPoolExecutor, as_completed def find_all_competitor_urls(competitors: list, destination: str, duration: int, travel_style: str) -> list: """ Runs trip URL finding for all competitors concurrently. Max 5 concurrent Firecrawl /map calls — respects server limits. Data flow: competitors list → ThreadPoolExecutor (max 5) → find_competitor_trip_url() per competitor in parallel → collect results → return matches list """ results = [] with ThreadPoolExecutor(max_workers=5) as executor: futures = { executor.submit( find_competitor_trip_url, c['website'], destination, duration, travel_style ): c for c in competitors } for future in as_completed(futures): competitor = futures[future] match = future.result() results.append({ "competitor_id": competitor['id'], "competitor_name": competitor['name'], **match }) return results ``` --- ## 9. API Routes All routes require `X-API-Key` header matching `API_SECRET_KEY` env var except `/health` and `/stripe/webhook`. ### Licence and Auth ``` POST /validate-email Body: { email } Returns: { valid, tier, reason } Logic: domain check → seat count → seat registration Note: sheet_id binding removed — standalone script model POST /stripe/webhook Stripe signature verified via STRIPE_WEBHOOK_SECRET Creates/updates tenant on payment events Sets status=inactive on cancellation ``` ### Trip-Intent Matching + Research ``` POST /research/find-urls Body: { destination, duration, travel_style, competitor_ids } Returns: { matches: [ { competitor_id, name, url, confidence, found } ] } Runs Firecrawl /map concurrently per competitor POST /research Body: { tenant_id, destination, duration, travel_style, tab_type, competitors: [ { id, name, url (confirmed) } ], prompt_template? } Returns: { job_id, status: "queued" } Enqueues to RQ — never runs synchronously GET /research/status/ Returns: { status, progress, results, error } GET /research/history Query: email, limit (default 5) Returns: last N runs for tenant GET /research/history/ Returns: full stored results for a specific run POST /preview-prompt Returns: { prompt } — dry run, no scraping ``` ### Dashboard Data ``` GET /competitors Header: X-User-Email Returns: [ { id, name, url, primary_market, last_scraped, change_detected, status } ] POST /battlecards Body: { email } Returns: [ { competitor_name, content, content_json, generated_at } ] POST /comparable-trips Body: { email } Returns: [ { client_trip, competitor_trip, competitor, similarity_score, price_difference, duration_difference } ] ``` ### Apps Script Export Integration ``` POST /sheet/push Body: { email, job_id, tab_name } Triggers Apps Script pushToSheet() with stored results for job_id Returns: { success, rows_written } GET /sheet/tabs Header: X-User-Email Returns list of available tabs in PM's registered workbook ``` ### Dashboard Account ``` GET /account/ POST /account/seats DELETE /account/seats/ POST /account/competitors — add single competitor (requires: name, website, catalogue_url, primary_market) POST /account/competitors/bulk — bulk import from CSV (validates, previews, imports) GET /account/competitors/template — download blank CSV template with correct headers PUT /account/competitors/ DELETE /account/competitors/ — deactivates + deregisters Firecrawl Monitor GET /account/template PATCH /account/onboarding — update onboarding_checklist fields GET /billing/portal — Stripe customer portal session URL POST /billing/upgrade — creates Stripe checkout for tier upgrade Body: { tenant_id, target_tier } Returns: { checkout_url } POST /auth/request-password-reset — Body: { email } → triggers PocketBase reset email POST /auth/confirm-password-reset — Body: { token, password, passwordConfirm } ``` ### Internal (n8n calls) ``` POST /internal/scrape/ POST /internal/battlecard/ POST /internal/embed-products/ POST /internal/match-comparable/ POST /internal/weekly-brief/ POST /internal/monitor-webhook POST /internal/monitor/register/ POST /internal/monitor/deregister/ POST /internal/discover/ — Discover tier only ``` ### System ``` GET /health — returns { status: "ok", version } ``` --- ## 10. Concurrency — RQ Job Queue The `/research` endpoint NEVER runs synchronously. Every research job is enqueued to RQ and processed by background workers. ### Queue Priority ```python # Three priority queues — Intelligence tier gets processed first high_q = Queue('high', connection=redis_conn) # Intelligence tier normal_q = Queue('normal', connection=redis_conn) # Discover + Analyse low_q = Queue('low', connection=redis_conn) # background tasks # Worker consumes in priority order # command: rq worker high normal low -u redis://bettersight-redis:6379 ``` ### Job Function (jobs.py) The job function calls scrape and extract functions from `core/scraper.py` and `core/extractor.py`. It adds PocketBase status updates and the fast/slow path check around the existing logic. When running as an RQ job, `jobs.py` imports from `core/scraper.py` and `core/extractor.py`. Never bypass the layer structure. --- ## 9. Licence Validation Logic ### Three layers — all must pass **Layer 1 — Domain check** Extract domain from email. Look up active tenant with matching domain in PocketBase. If no match → reject with "Domain not registered". **Layer 2 — Seat count** Check if email already has an active seat for this tenant. If yes → update last_accessed, allow. If no → count active seats. If at limit → reject with "Seat limit reached". If under limit → register new seat, allow. **Layer 3 — Sheet ID binding** On first validation for a tenant, store the sheet_id in the tenants record. On subsequent validations, compare sheet_id. If mismatch → reject with "Licence bound to a different Sheet. Contact support." ### Cache Apps Script caches validation result for 6 hours via CacheService. Cache key: `BETTERSIGHT_VALID` and `BETTERSIGHT_TIER`. On cache hit — skip API call entirely. --- ## 10. Scraping Architecture ### Refactor app.py Before New Development The existing `app.py` must be refactored into the modular three-layer structure before any new development begins. Logic stays identical — it is reorganised into correct layers, not rewritten. **Core IP preserved in `core/`:** - Playwright stealth browser with cookie consent handling and lazy load scrolling - BeautifulSoup HTML cleaning - AI web fetch fallback via OpenRouter for bot-protected sites - OpenRouter model cycling with fallback chain - Multi-currency extraction **Industry config moved to `industries/adventure_travel/`:** - NGS vs Standard field differentiation - Prompt builder and injection - JSON extraction schema **Client config moved to `config/client.py`:** - Google Sheet write-back via gspread - Sheet ID, tab names, column assignments ### Modular Scraping Architecture app.py must be refactored into a three-layer modular structure before any new development begins. This separates infrastructure from industry config from client config — making the codebase maintainable, testable, and extensible to other industries without touching core scraping logic. **Target folder structure:** ``` backend/ core/ scraper.py — Playwright browser setup, stealth, waiting, HTML cleaning extractor.py — OpenRouter model cycling, JSON parsing, fallback chain ai_fetch.py — AI web fetch fallback (existing fetch_with_ai()) industries/ adventure_travel/ fields.py — STANDARD_FIELDS, NGS_FIELDS column maps prompt.py — build_prompt(), inject_page_content() schema.py — JSON extraction schema for this vertical # future verticals added here — never touch core/ config/ client.py — SHEET_ID, tab names, column assignments per client jobs.py — RQ job functions — imports from core/ and industries/ api.py — Flask routes app.py — Entry point only — imports and wires everything together ``` **The three layers:** Layer 1 — `core/` — never changes between industries or clients - Playwright stealth browser config - Smart waiting and scroll logic - Cookie consent handling - HTML cleaning via BeautifulSoup - OpenRouter model cycling and JSON parsing - AI web fetch fallback Layer 2 — `industries/{vertical}/` — swapped per vertical - Field maps (column positions in the Sheet) - Prompt template and context ("you are a travel industry analyst") - JSON extraction schema (what fields to extract) - Tab names Layer 3 — `config/client.py` — swapped per client within a vertical - Sheet ID - Column assignments per competitor - Competitor DB tab structure **To pivot to a new industry:** Create a new folder under `industries/` with three files. Core scraping logic is never touched. Sheet template changes are driven entirely by the field map. ### playwright-stealth Integration Replace the manual `add_init_script` block in `core/scraper.py` with `playwright-stealth` — a comprehensive stealth library applying 20+ evasion techniques automatically. ``` pip install playwright-stealth ``` In `core/scraper.py` — replace the existing `add_init_script` call: ```python from playwright_stealth import stealth_async async def render_page(url: str) -> dict: async with async_playwright() as p: browser = await p.chromium.launch(...) context = await browser.new_context(...) page = await context.new_page() # Replaces the manual add_init_script block entirely await stealth_async(page) # Rest of existing code unchanged await page.goto(url, ...) ``` `stealth_async` handles: navigator.webdriver spoofing, plugins array, languages, chrome runtime, canvas fingerprint, WebGL masking, audio context randomisation, and ~15 additional evasion techniques that the manual script did not cover. ### Smart Proxy Service — `core/proxy_service.py` The proxy layer is a standalone service in `core/proxy_service.py`. It is the ONLY place in the codebase that knows about any proxy provider. Switching providers or adjusting the switching logic requires changing only this file. **Two providers — smart auto-switching:** - **Primary:** Webshare (~$3/month flat, free tier available) - **Fallback:** Bright Data pay-as-you-go (~$4-8/GB, only used when Webshare is blocked) At MVP scraping scale (6 competitors × 5 clients = 30 pages/day) total proxy cost is under $5/month. Bright Data fallback costs pennies at this volume. **PocketBase collection — `proxy_overrides`** ``` id auto tenant_id relation → tenants (nullable — global overrides have no tenant) domain text (e.g. intrepidtravel.com) provider select [webshare, brightdata] blocked_count integer last_blocked date created autodate ``` **ProxyService class:** ```python class ProxyService: """ Smart proxy provider — uses Webshare by default with country targeting, auto-escalates to Bright Data for domains that block Webshare. Country targeting uses competitors.primary_market (ISO code e.g. GB, US, AU) passed through from the competitor record — no hardcoded country map in code. Data flow (first request to a domain): domain + country → proxy_overrides lookup → no override found → Webshare config with country targeting returned → scraper attempts request → if blocked → escalate_to_brightdata(domain) called → Bright Data config returned → scraper retries → override stored in PocketBase for this domain Data flow (subsequent requests to a known blocked domain): domain + country → proxy_overrides lookup → brightdata override found → Bright Data config returned immediately — no Webshare attempt """ def get_proxy_for_domain(self, domain: str, country: str = None) -> dict: """ Returns requests proxy config for a domain with optional country targeting. Routes to Bright Data if domain has been previously blocked on Webshare. """ override = proxy_repository.get_by_domain(domain) if override and override['provider'] == 'brightdata': return self._brightdata_requests() return self._webshare_requests(country=country) def get_playwright_proxy_for_domain(self, domain: str, country: str = None) -> dict: """ Returns Playwright proxy config for a domain with optional country targeting. Routes to Bright Data if domain has been previously blocked on Webshare. """ override = proxy_repository.get_by_domain(domain) if override and override['provider'] == 'brightdata': return self._brightdata_playwright() return self._webshare_playwright(country=country) def escalate_to_brightdata(self, domain: str): """ Records that Webshare was blocked on this domain. Upserts a proxy_override record so future requests skip Webshare entirely. """ existing = proxy_repository.get_by_domain(domain) if existing: proxy_repository.update(existing['id'], { 'provider': 'brightdata', 'blocked_count': existing['blocked_count'] + 1, 'last_blocked': datetime.now().isoformat() }) else: proxy_repository.create({ 'domain': domain, 'provider': 'brightdata', 'blocked_count': 1, 'last_blocked': datetime.now().isoformat() }) def _webshare_requests(self, country: str = None) -> dict: """ Webshare requests proxy config with optional country targeting. Format per Webshare docs: username-{COUNTRY}-rotate No country = general rotation across full pool. """ suffix = f"-{country.upper()}-rotate" if country else "-rotate" username = f"{WEBSHARE_USER}{suffix}" return { "http": f"http://{username}:{WEBSHARE_PASS}@p.webshare.io:80", "https": f"http://{username}:{WEBSHARE_PASS}@p.webshare.io:80" } def _webshare_playwright(self, country: str = None) -> dict: """Webshare Playwright proxy config with optional country targeting.""" suffix = f"-{country.upper()}-rotate" if country else "-rotate" return { "server": "http://p.webshare.io:80", "username": f"{WEBSHARE_USER}{suffix}", "password": WEBSHARE_PASS } def _brightdata_requests(self) -> dict: return { "http": f"http://{BRIGHTDATA_USER}:{BRIGHTDATA_PASS}@brd.superproxy.io:22225", "https": f"http://{BRIGHTDATA_USER}:{BRIGHTDATA_PASS}@brd.superproxy.io:22225" } def _brightdata_playwright(self) -> dict: return { "server": "http://brd.superproxy.io:22225", "username": BRIGHTDATA_USER, "password": BRIGHTDATA_PASS } ``` **Block detection and auto-switch in `core/scraper.py`:** ```python async def render_page(url: str, country: str = None) -> dict: """ Renders a page using the smart proxy service with country targeting. country comes from competitors.primary_market in PocketBase. Detects blocks and auto-escalates to Bright Data if Webshare fails. Data flow: url + country → domain extracted → proxy_service.get_playwright_proxy_for_domain(domain, country) → Playwright render → if blocked → escalate_to_brightdata(domain) → retry with Bright Data (no country targeting at fallback level) """ domain = extract_domain(url) proxy = proxy_service.get_playwright_proxy_for_domain(domain, country) result = await _render_with_proxy(url, proxy) if _is_blocked(result): logger.warning(f'Webshare blocked on {domain} — escalating to Bright Data') proxy_service.escalate_to_brightdata(domain) result = await _render_with_proxy(url, proxy_service._brightdata_playwright()) return result def _is_blocked(result: dict) -> bool: """ Detects common block signals from scrape result. Returns True if the response looks like a bot wall rather than real content. Signals: low char count, HTTP 403/429, Cloudflare challenge, CAPTCHA text. """ if result.get('char_count', 0) < 500: return True if result.get('status_code') in [403, 429]: return True block_signals = [ 'cf-browser-verification', 'captcha', 'access denied', 'blocked', 'robot' ] return any(s in result.get('text', '').lower() for s in block_signals) ``` **In `jobs.py` — primary_market passed through automatically:** ```python def run_research_job(job_id: str, data: dict): """ Pulls primary_market from each competitor record and passes it to scrape() for automatic country-targeted proxy routing. No manual country configuration needed per job. """ for i, competitor in enumerate(data['competitors']): competitor_record = competitor_repository.get_by_id(competitor['id']) page_data = scrape( url=competitor['url'], country=competitor_record.get('primary_market') # e.g. "GB", "US", "AU" ) # rest of job unchanged ``` **Cost summary:** ``` Webshare free tier — 10 proxies shared, enough for MVP testing Webshare paid — ~$3/month, handles the vast majority of requests Bright Data fallback — pay-as-you-go ~$4-8/GB, <$2/month at MVP scale Total proxy cost MVP — under $5/month ``` ### OpenRouter Model Cycling — `core/extractor.py` Cycles through free tier models. Falls back to a paid model if all free models fail or return rate limit errors. Never fails silently. ```python FREE_MODELS = os.getenv('OPENROUTER_FREE_MODELS', '').split(',') FALLBACK_MODEL = os.getenv('OPENROUTER_FALLBACK_MODEL', 'anthropic/claude-haiku-4-5') def extract_with_openrouter(prompt: str, content: str) -> dict: """ Cycles through free OpenRouter models in sequence. Falls back to the paid fallback model if all free models fail. Never returns None — raises ExtractorError only after all models exhausted. Data flow: prompt + content → for each FREE_MODELS: POST openrouter.ai/api/v1/chat/completions → if success: return parsed JSON → if rate limit or error: try next model → if all free models fail → try FALLBACK_MODEL (paid) → if fallback fails → raise ExtractorError, log to Sentry """ models_to_try = FREE_MODELS + [FALLBACK_MODEL] for model in models_to_try: try: response = _call_openrouter(model, prompt, content) if response: if model == FALLBACK_MODEL: logger.warning( f'Free models exhausted — used paid fallback: {model}' ) sentry_sdk.capture_message( f'OpenRouter paid fallback used', level='warning' ) return response except RateLimitError: logger.warning(f'Rate limit on {model} — trying next') continue except Exception as e: logger.error(f'Model {model} failed: {str(e)}') continue raise ExtractorError('All OpenRouter models exhausted including paid fallback') ``` Firecrawl is in the MVP stack for `/monitor` only. `/map` discovery endpoint is Intelligence tier — do not build in MVP. --- ## 11. Battlecard Generation After each scrape run completes, n8n calls `/internal/battlecard/`. ```python def generate_battlecard(competitor_id: str, tenant_id: str) -> dict: """ Pulls latest product data for a competitor from PocketBase. Passes structured data to Claude Haiku via OpenRouter. Saves generated battlecard text and JSON back to PocketBase. Returns battlecard dict for immediate use. Data flow: competitor_id → PocketBase products (latest 10) → structured prompt → OpenRouter (claude-haiku) → battlecard text + JSON → PocketBase battlecards table """ ``` Prompt template: ``` You are a competitive intelligence analyst for an adventure travel operator. Based on this competitor data, write a concise battlecard. Competitor: {name} Top trips: {top_trips} Price range: {price_low} - {price_high} USD Avg duration: {avg_duration} days Positioning: {positioning_summary} Recent changes: {recent_changes} Write: 1. One line positioning summary (max 15 words) 2. Three strengths (bullet points, max 8 words each) 3. Three weaknesses (bullet points, max 8 words each) 4. One pricing observation (max 20 words) Return ONLY valid JSON: { "positioning": "...", "strengths": ["...", "...", "..."], "weaknesses": ["...", "...", "..."], "pricing_observation": "..." } ``` --- ## 12. Semantic Trip Matching ### Embedding generation After products are stored, embed each new trip using text-embedding-3-small. Store vector as JSON in the products.embedding field. ```python def embed_trip(trip: dict) -> list: """ Generates a semantic embedding for a trip using OpenRouter. Combines trip name, destination, duration, and activities into a single text representation before embedding. Data flow: trip dict → formatted text string → OpenRouter embeddings API → vector list → stored in products.embedding """ text = f""" {trip['trip_name']} Destination: {trip['destination']} Duration: {trip['duration_days']} days Activities: {trip['activities']} Start: {trip['start_location']} """ response = openai_client.embeddings.create( model="openai/text-embedding-3-small", input=text ) return response.data[0].embedding ``` ### Similarity matching ```python def find_comparable_trips(client_trips, competitor_trips, threshold=0.75): """ Calculates cosine similarity between all client trips and all competitor trips. Returns matches above the threshold sorted by similarity score descending. Data flow: client_trips + competitor_trips (each with .embedding) → pairwise cosine similarity → filter by threshold → sort by score → list of match dicts """ ``` Threshold: 0.75. Matches stored in comparable_matches table. --- ## 13. Weekly HTML Email Brief ### No PDF — HTML email only WeasyPrint has been removed from the stack. The Monday brief is a rich Jinja2-rendered HTML email sent via Resend. Same content, zero Docker dependency, zero PDF complexity. Brief email includes: - Bettersight branded header with tenant name - Market summary (price changes, new products, competitors monitored) - Price movements table (competitor, trip, previous, current, delta, percent) - New products this week (formatted cards) - Battlecard updates summary - Link to app.bettersight.io - Unsubscribe footer ### n8n Workflow — Monday 6am ``` Schedule Trigger (Monday 06:00 AST) → HTTP POST /internal/weekly-brief/{tenant_id} for each active tenant → Flask renders brief.html Jinja2 template with weekly PocketBase data → Resend sends HTML email to tenant admin → Logged to digest_logs ``` ### Brief service ```python def send_weekly_brief(tenant_id: str): """ Renders the weekly intelligence brief as branded HTML and sends it via Resend to the tenant admin email. Data flow: tenant_id → PocketBase (price_history, products, battlecards for current week) → brief.html Jinja2 template rendered → Resend API → digest_logs record created """ data = brief_repository.get_weekly_data(tenant_id) html = render_template('emails/brief.html', **data) resend.Emails.send({ "from": "Bettersight ", "to": data['admin_email'], "subject": f"Bettersight Weekly Brief — Week of {data['week_of']}", "html": html }) brief_repository.log_digest(tenant_id, 'weekly') ``` --- ## 15. Google Apps Script ### Role — Export Integration Only The Apps Script is no longer the primary interface. The dashboard is. The script's only job is to push analysis results from the dashboard into the PM's Google Sheet workbook on demand. **Three functions only:** ``` pushToSheet(results, tabName) — writes dashboard results to selected tab getWorkbookTabs() — returns tab list for dashboard push dropdown validateLicence() — confirms PM is authorised (email check only) ``` ### Deployment Model — Standalone Script Deployed as a standalone script, not container-bound to any workbook. PM installs once via a shareable link from app.bettersight.io/account. Works in any Google Sheet workbook they open from that point forward. ### File Structure ``` appscript/ Code.gs — onOpen(), install entry point Validation.gs — validateLicence() email-only Export.gs — pushToSheet(), getWorkbookTabs() Sidebar.html — minimal UI: tab selector + push button only ``` ### Export Flow ```javascript /** * Pushes analysis results from the dashboard to the PM's selected * workbook tab. Called via the dashboard "Push to Sheet" button * which hits Flask POST /sheet/push, which calls this function. * * Writes results using header-based column detection — never * hardcoded column positions. Resilient to any sheet restructuring. * * Flow: * job_id → Flask /research/history/{job_id} → stored results → * getWorkbookTabs() shows tab selector in dashboard dropdown → * PM selects tab → POST /sheet/push → * detectColumns(sheet) → writeResults(results, columnMap, sheet) */ function pushToSheet(results, tabName) { ... } /** * Returns list of all tab names in the PM's current workbook. * Used by the dashboard to populate the "Push to Sheet" tab selector. */ function getWorkbookTabs() { return SpreadsheetApp.getActiveSpreadsheet() .getSheets() .map(s => s.getName()); } /** * Detects column positions by matching header names in row 1. * Returns { fieldName: columnIndex } map. * Falls back to STANDARD_FIELDS positions if headers not found. */ function detectColumns(sheet) { ... } ``` ### Sidebar UI (minimal) ``` ┌─────────────────────────────┐ │ 🔵 Bettersight │ │ Export results to Sheet │ │ ───────────────────────── │ │ Push to tab: │ │ [Bettersight Analysis ▾] │ │ │ │ [ Push last analysis ] │ │ ───────────────────────── │ │ Last push: Today 9:14am │ │ 5 competitors · Peru │ └─────────────────────────────┘ ``` The full analysis workflow — competitor selection, trip-intent matching, job status, results view, history, confidence indicators — all live in the dashboard. The sidebar is a push button only. The Apps Script is deployed as a **standalone script**, NOT container-bound to a specific Google Sheet file. This is the most important architectural decision for the add-on. **Why standalone:** - Works in any Google Sheet the PM opens — including their existing 20-sheet stage gate workbook - PM installs once via a shareable install link from the Bettersight dashboard - They import the Bettersight template tab into their existing workbook - The sidebar runs analysis and writes to whichever tab they select - Cross-sheet formulas in their stage gate sheet reference the Bettersight tab natively — no copy-paste ever needed after setup **What the PM receives:** 1. A template Sheet tab — downloadable from app.bettersight.io/account as a Google Sheets link. They copy this tab into their existing workbook. 2. A sidebar install link — one click installs the standalone script on their Google account. Works in any workbook from that point forward. **Sheet ID binding update:** Because the script is standalone, the licence validation no longer binds to a single Sheet ID. Instead it binds to the PM's Google account email (already implemented via domain + seat validation). Remove the sheet_id binding layer from the licence validation — it is redundant and would break the standalone model. --- ### File Structure ``` appscript/ Code.gs — onOpen(), menu setup, sidebar launch, install entry point Validation.gs — validateLicence() Research.gs — runResearch(), pollStatus(), writeResultsToSheet() Competitors.gs — loadCompetitorsFromPocketBase(), refreshCompetitorList() Settings.gs — saveSettings(), loadSettings(), getTargetSheet() Battlecards.gs — refreshBattlecards(), writeBattlecardsTab() ComparableTrips.gs — refreshComparableTrips(), writeComparableTab() Sidebar.html — sidebar UI ``` --- ### Validation Flow (Validation.gs) ```javascript /** * Validates the current user's licence against the Bettersight API. * Standalone script — validates by email only, no sheet_id binding. * * Flow: * 1. Check 6-hour cache — return true immediately if cache valid * 2. Get user email via Session.getActiveUser().getEmail() * 3. POST to /validate-email with email only * 4. On success — cache result and tier for 6 hours, return true * 5. On failure — show alert with reason, return false * * Cache keys: BETTERSIGHT_VALID, BETTERSIGHT_TIER * Cache duration: 21600 seconds (6 hours) */ function validateLicence() { ... } ``` --- ### Settings — Tab Selection and Persistence (Settings.gs) The PM selects which tab in their workbook to write results to. This setting persists across sessions via PropertiesService. ```javascript /** * Returns the target sheet tab for writing analysis results. * Reads from UserProperties — falls back to tab named * 'Bettersight Analysis' if no preference is stored. * * Flow: * UserProperties.getProperty('TARGET_TAB') → * if found: return that tab → * if not found: find 'Bettersight Analysis' tab → * if not found: prompt PM to select tab via dropdown */ function getTargetSheet() { ... } /** * Saves all sidebar settings to UserProperties so they persist * across sessions. Called automatically after each successful run. * * Settings persisted: * TARGET_TAB — which sheet tab to write to * TRAVEL_STYLE — last used travel style * PRODUCT_NAME — last used G Adventures product name * DESTINATION — last used destination * DURATION — last used duration * TAB_TYPE — Standard or NGS * SELECTED_COMPS — JSON array of last selected competitor IDs */ function saveSettings(settings) { ... } /** * Loads all persisted settings and returns them to the sidebar * so the PM's last run is pre-filled on open. */ function loadSettings() { ... } ``` --- ### Competitor Auto-Population (Competitors.gs) Competitors are loaded from PocketBase on sidebar open. The PM never manually types a URL again. ```javascript /** * Fetches the PM's competitor list from the Bettersight API. * Called on sidebar open. Returns array of competitor objects * for rendering as checkboxes in the sidebar. * * Flow: * validateLicence() → GET /competitors with email header → * returns [ { id, name, url, primary_market, last_scraped, * change_detected, status } ] → * rendered as checkboxes in sidebar with status indicators * * Status indicators shown per competitor: * ✓ Current (updated {time}) — last_scraped < 24hrs, no change * ⚠ Changed ({time} ago) — change_detected = true * ○ Stale — last_scraped > 24hrs */ function loadCompetitorsFromPocketBase() { ... } ``` --- ### Column Auto-Detection (Research.gs) Column positions are detected by reading header names, not hardcoded numbers. Makes the sheet resilient to any structural changes the PM makes. ```javascript /** * Detects the column index for each field by matching header names * in row 1 of the target sheet. Returns a field-to-column map. * Falls back to STANDARD_FIELDS hardcoded positions if headers * cannot be found — logs a warning when falling back. * * Flow: * Read row 1 of target sheet → * Build { fieldName: columnIndex } map by matching header text → * Return map for use by writeResultsToSheet() */ function detectColumns(sheet) { ... } ``` --- ### Research Flow with Per-Competitor Progress (Research.gs) ```javascript /** * Submits a research job and polls for completion. * Shows per-competitor status in the sidebar in real time. * * Flow: * 1. validateLicence() — exit if invalid * 2. getTargetSheet() — confirm which tab to write to * 3. detectColumns(sheet) — map field names to column positions * 4. loadSettings() — pre-fill job params from last run * 5. POST to /research with selected competitor IDs and job params * → receives job_id immediately * 6. Poll GET /research/status/{job_id} every 3 seconds * 7. On each poll update sidebar with per-competitor status: * ✓ Intrepid Travel — Peru Classic $2,190 * ✓ Exodus Travels — Machu Picchu $2,050 * ⟳ Flash Pack — scraping... * Kimkim — queued * 8. On complete — writeResultsToSheet() using detected columns * 9. saveSettings() — persist this run's settings for next time * 10. Show confidence indicators per competitor in sidebar * 11. Timeout after 75 attempts (5 minutes) */ function runResearch() { ... } /** * Writes extracted competitor data to the target sheet. * Uses column map from detectColumns() — never hardcoded positions. * Only writes fields that exist in the detected column map. * * Flow: * results array + columnMap + sheet → * for each competitor result: * for each field in columnMap: * sheet.getRange(fieldRow, columnMap[field]).setValue(value) */ function writeResultsToSheet(results, columnMap, sheet) { ... } ``` --- ### Single Competitor Re-Run (Research.gs) ```javascript /** * Triggers a fresh scrape and extraction for a single competitor. * Used when the PM wants to refresh one competitor without * re-running the full job. Updates only that competitor's column. * * Flow: * competitorId → POST /research with single competitor → * poll status → write result to sheet (single column only) → * update sidebar status for that competitor only * * Called from a "Refresh" button shown per competitor * in the completed run status list in the sidebar. */ function refreshSingleCompetitor(competitorId) { ... } ``` --- ### Confidence Indicators (Research.gs) ```javascript /** * Renders a confidence bar per competitor in the sidebar * after a run completes. Based on the relevancy score (1-5) * returned by the AI extraction. * * Display format: * Intrepid Travel ████████░░ 82% — Good match * Flash Pack ██████░░░░ 61% — Review recommended * Kimkim ████░░░░░░ 44% — Low confidence ⚠ * * Low confidence (< 60%) shows a warning prompt: * "The AI had low confidence extracting Flash Pack data. * The page may have changed structure. Consider a manual review." */ function renderConfidenceIndicators(results) { ... } ``` --- ### Analysis History (Research.gs) ```javascript /** * Fetches the last 5 analysis runs for the current tenant * from PocketBase. Displayed in the sidebar as a history panel. * PM can restore a previous run's data to the sheet for comparison. * * Flow: * GET /research/history?email={email}&limit=5 → * returns [ { job_id, run_at, competitors, summary } ] → * rendered as list in sidebar history panel * * Each history item shows: * Mon 9 Jun 9:14am · 5 competitors · Intrepid, Exodus, Flash Pack... * [Restore to sheet] * * Restore writes that run's stored results back to the sheet * using the same writeResultsToSheet() function. */ function loadAnalysisHistory() { ... } /** * Restores a previous run's results to the current sheet. * Fetches stored results from PocketBase by job_id. * Warns PM that this will overwrite current sheet data. */ function restoreHistoryRun(jobId) { ... } ``` --- ### Copy to Clipboard (Sidebar.html) A "Copy rows" button in the sidebar copies selected competitor rows as tab-separated values — paste-ready into any Google Sheet, Excel, or Notion table without formatting issues. ```javascript /** * Copies the selected competitor result rows as tab-separated * values to the clipboard. PM can paste directly into any sheet. * No formatting, no formulas — clean values only. * * Called from "Copy selected rows" button in sidebar. */ function copyRowsToClipboard(selectedCompetitorIds) { ... } ``` --- ### Sidebar UI Sections (Sidebar.html) ``` ┌─────────────────────────────────┐ │ 🔵 Bettersight │ │ ───────────────────────────── │ │ ANALYSIS SETTINGS │ │ Travel style [Classic ▾] │ │ Product name [____________] │ │ Destination [____________] │ │ Duration [____________] │ │ Tab type [Standard ● NGS○]│ │ Write to tab [Bettersight ▾]│ ← tab selector │ ───────────────────────────── │ │ COMPETITORS │ │ ☑ Intrepid Travel ✓ Current │ ← from PocketBase │ ☑ Exodus Travels ✓ Current │ │ ☑ Flash Pack ⚠ Changed │ │ ☐ Kimkim ○ Stale │ │ ───────────────────────────── │ │ [ Run Analysis ] │ │ [ Copy selected rows ] │ │ ───────────────────────────── │ │ LAST RUN RESULTS │ │ ✓ Intrepid ████████░░ 82% │ ← confidence │ ✓ Exodus █████████░ 91% │ │ ⚠ Flash Pk ██████░░░░ 61% │ ← low confidence warn │ [↺ Refresh Flash Pack] │ ← single re-run │ [ Copy selected rows ] │ │ ───────────────────────────── │ │ HISTORY │ │ Mon 9 Jun · 5 competitors │ │ [Restore] │ │ Mon 2 Jun · 4 competitors │ │ [Restore] │ └─────────────────────────────────┘ ``` --- ### New Flask Routes Supporting Apps Script Improvements ``` GET /competitors — returns competitor list for sidebar auto-population includes: id, name, url, primary_market, last_scraped, change_detected, status GET /research/history — last 5 runs for tenant (by email header) GET /research/history/ — full results for a specific run by job_id used by restoreHistoryRun() ``` --- ## 15. n8n Workflows ### Daily scrape (6am every day) ``` Schedule Trigger (06:00 AST) → For each active tenant: → Query PocketBase for competitors where: change_detected = true OR last_scraped > 24hrs ago → HTTP POST /internal/scrape/{tenant_id} with filtered competitor list → RQ job queued → Playwright scrape → AI extraction → PocketBase updated (products, price_history) → change_detected reset to false → last_scraped updated to now → price_history diff calculated automatically → is_new flag set on new products automatically ``` ### Battlecard regeneration (after scrape) ``` PocketBase webhook on scrape_runs status=complete → HTTP POST /internal/battlecard/{competitor_id} → Runs for each competitor in the completed scrape run ``` ### Price change alert ``` PocketBase webhook on price_history insert → Code node: check change_percent threshold (>5% triggers alert) → Creates alert record in PocketBase alerts table { alert_type: "price_change", competitor_id, product_id, change_amount, change_percent, delivered_dashboard: true } → Alert appears in dashboard activity feed immediately → n8n daily digest (6pm): collects unread alerts → Resend email summary ``` ### New product alert ``` PocketBase webhook on products insert where is_new=true → Creates alert record in PocketBase alerts table { alert_type: "new_product", competitor_id, product_id, delivered_dashboard: true } → Alert appears in dashboard activity feed immediately → Included in daily digest email (6pm) and Monday brief ``` ### [Discover tier] External notification integrations ``` When PM connects Slack/Teams/WhatsApp in account settings: → webhook_url stored in tenant_notifications table → n8n alert workflow checks tenant_notifications on each alert → If webhook configured: HTTP POST to Slack/Teams webhook URL → If Twilio configured: POST to Twilio WhatsApp API ``` ### Monday morning brief (6am) ``` Schedule Trigger (Monday 06:00 AST) → HTTP POST /internal/weekly-brief/{tenant_id} per active tenant → Flask renders brief.html Jinja2 template with weekly data → Resend sends HTML email to tenant admin email → Logged to digest_logs ``` ### Sunday night catalogue discovery (Intelligence tier — NOT in MVP) ``` DEFERRED — do not build until Intelligence tier development begins. Requires Firecrawl self-hosted which is not in the current stack. ``` --- ## 16. Stripe Integration ### Webhook events handled ``` checkout.session.completed → Create tenant in PocketBase → Set status=active → Generate welcome email with Sheet template download link customer.subscription.updated → Update tier if plan changed → Update max_seats based on new tier customer.subscription.deleted → Set tenant status=inactive → All seats blocked at next cache expiry (max 6 hours) ``` ### Tier → seat mapping ```python TIER_SEATS = { "analyse": 5, "discover": 8, "intelligence": 10, "enterprise": 999 # effectively unlimited } # Additional seats available on any tier ADDITIONAL_SEAT_PRICE = 45 # USD per seat per month # Configured as a Stripe add-on price on each subscription # Enterprise tier is not in Stripe public product list # Created manually per client as a custom Stripe price ``` --- ## 17. Environment Variables ```bash # Flask FLASK_ENV=production API_SECRET_KEY= # PocketBase POCKETBASE_URL=http://pocketbase:8090 POCKETBASE_ADMIN_EMAIL= POCKETBASE_ADMIN_PASSWORD= # OpenRouter OPENROUTER_API_KEY= OPENROUTER_FREE_MODELS=google/gemini-flash-1.5,meta-llama/llama-3.1-8b-instruct,mistralai/mistral-7b-instruct OPENROUTER_FALLBACK_MODEL=anthropic/claude-haiku-4-5 # paid fallback ~$0.25/1M tokens — used when free tier unavailable # Proxy — Primary (Webshare) WEBSHARE_USER= WEBSHARE_PASS= # Proxy — Fallback (Bright Data — pay-as-you-go, auto-escalated per domain) BRIGHTDATA_USER= BRIGHTDATA_PASS= # Proxy settings PROXY_ESCALATION_THRESHOLD=2 # blocked_count before treating domain as permanently blocked # Stripe STRIPE_SECRET_KEY= STRIPE_WEBHOOK_SECRET= STRIPE_PRICE_ANALYSE= # trial_period_days: 14 STRIPE_PRICE_DISCOVER= # trial_period_days: 21 STRIPE_PRICE_INTELLIGENCE= # trial_period_days: 30 # Enterprise: created manually per client — no trial, no env var # Transactional email RESEND_API_KEY= # Error tracking SENTRY_DSN= # Required in staging and production — never skip # Hetzner Object Storage (backups) HETZNER_S3_ACCESS_KEY= HETZNER_S3_SECRET_KEY= HETZNER_S3_ENDPOINT=https://fsn1.your-objectstorage.com HETZNER_S3_BUCKET=bettersight-backups # Gotify (owner monitoring — never client-facing) GOTIFY_URL=https://gotify.bettertend.net GOTIFY_APP_TOKEN=A78_l3r9y.ZzBr- # Google Sheets (existing from app.py) GOOGLE_SERVICE_ACCOUNT_JSON= SHEET_ID= STANDARD_TAB= NGS_TAB= ``` --- ## 18. Enterprise Software Development Standards > These standards are absolute. They are never negotiated, never skipped, > and never deferred to a later phase. ### Languages and Conventions - Backend: Python only. No mixed languages in the same service. - Frontend: JavaScript only. No TypeScript. No `.ts` files. - `snake_case` for Python. `camelCase` for JavaScript. - No abbreviations unless universally understood (id, url, pdf). ### Separation of Concerns **Backend layer hierarchy — no layer skips another:** ``` Routes (api.py) → validate input, call services, return response → NO business logic. NO direct DB calls. NO external API calls. Services (services/) → all business logic and orchestration → NO HTTP request/response objects → NO direct DB queries or external API calls → calls repositories and other services only Repositories (repositories/) → all data access (PocketBase reads/writes, external API calls) → NO business logic → NO service calls → returns plain dicts only jobs.py → imports and calls service functions only → NO direct DB access → NO business logic ``` ### Security - Every authenticated route checks `X-API-Key` against `API_SECRET_KEY` env var - Every DB query for non-admin operations is tenant-scoped (`tenant_id` filter on every PocketBase query) - All credentials from env vars — hardcoded credentials are a build failure - Stripe webhook validated via signature before any processing - No stack traces in production error responses - Seat validation checks domain and seat count — both every time ### Observability - Every route logs: method, path, tenant_id (if known), duration, status code - Every RQ job logs: job_id, tenant_id, start, completion, error if any - `/health` endpoint always returns `{ status: "ok", version }` — no auth required - Sentry configured via `SENTRY_DSN` env var in production - n8n monitors `/health` every 5 minutes — Gotify alert on failure ### Resilience - RQ job failures are logged to scrape_runs.error_log - Failed scrapes do not fail the entire batch — continue to next competitor - Side-effect failures (Gotify, email) never fail a primary operation - All external API calls (OpenRouter, Bright Data, Firecrawl, Stripe) have try/except with logged errors and graceful fallback - RQ retries failed jobs automatically up to 3 times with backoff ### Code Commenting — Mandatory **All code must be adequately commented. This is not optional.** Every function, route, service method, and job function must have: 1. A docstring stating what the function does (one sentence) 2. The data flow — what comes in, what happens to it, what goes out 3. Inline comments on every meaningful step ```python def validate_email(email: str, sheet_id: str) -> dict: """ Validates a Google user's access to Polaris via three checks: domain registration, seat count, and Sheet ID binding. Data flow: email + sheet_id → domain extraction → PocketBase tenant lookup (Layer 1) → seat count check → seat registration if new (Layer 2) → sheet_id binding on first use / comparison on subsequent (Layer 3) → { valid: bool, tier: str, reason: str } """ # Extract domain from email for tenant lookup domain = email.split("@")[1] # Layer 1: Check if domain is registered to an active or trial tenant tenant = tenant_repository.get_by_domain(domain) if not tenant: return {"valid": False, "reason": "Domain not registered"} if tenant["status"] not in VALID_STATUSES: return {"valid": False, "reason": "Subscription inactive"} # Layer 2: Check seat availability or register existing seat seat_result = seat_repository.validate_or_register(tenant, email) if not seat_result["valid"]: return seat_result return {"valid": True, "tier": tenant["tier"]} ``` ### Testing - 80% overall test coverage enforced — cannot merge below this - 100% coverage on: licence validation, Stripe webhook, seat management, rollback paths - Every repository has a mock implementation alongside the real one - Unit tests for all services - Integration tests for all API routes - Tests live in `backend/tests/` mirroring the service/repository structure ### Environments and Deployment - Three environments: development, staging, production - Nothing reaches production without passing staging - Production deploys require a tagged version — no unversioned production deploys - Dokploy handles zero-downtime deploys via rolling restart ### Data and Backup **Two-layer backup strategy — belt and braces:** **Layer 1 — Litestream (real-time WAL streaming)** - Streams every SQLite write to Hetzner Object Storage in real-time - Recovery point objective: seconds - Recovery time objective: under 5 minutes - Runs as a sidecar container alongside PocketBase **Layer 2 — Daily zip backup (point-in-time snapshots)** - Daily 2am via crontab on the VPS — survives container restarts - 7-day retention on Hetzner Object Storage - Recovery point objective: 24 hours (fallback if Litestream stream is corrupted) ```bash # /etc/cron.d/bettersight-backup 0 2 * * * root /usr/local/bin/backup-pocketbase.sh # backup-pocketbase.sh #!/bin/bash DATE=$(date +%Y%m%d) docker exec pocketbase /pb/pocketbase backup \ --dir /pb/backups --name backup_$DATE.zip rclone copy /pb/backups/ \ hetzner-s3:bettersight-backups/ \ --max-age 7d find /pb/backups -name "*.zip" -mtime +2 -delete ``` - Audit logs (alerts, digest_logs) are append-only — no update or delete permitted - Multi-step operations (scrape → store → alert) documented with rollback strategy ### Strict Rules — Never Violate - Every authenticated endpoint validates `X-API-Key` — no exceptions - Every DB query for non-admin roles includes `tenant_id` filter — no exceptions - All credentials from env vars — hardcoded values are a build failure - No layer skips another in the separation of concerns hierarchy - A side-effect failure never fails a primary operation - Every multi-step operation has a documented rollback strategy - Mock implementations ship alongside every real repository - 80% test coverage enforced by CI — cannot merge below this - 100% coverage on payment, auth, webhook, and rollback paths - Nothing reaches production without passing staging - Production deploys require a tagged version - Audit logs are append-only — no update or delete ever - Daily backup with off-server storage — no exceptions - Sentry configured in staging AND production — silent failures are unacceptable - `/health` endpoint exists and is monitored — automated alert on failure - All code and functions are adequately commented — uncommented code is a build failure - Every Flask error response is structured JSON — never plain text - Every page has a defined empty state — no blank pages ever - Rate limiting applied to every public endpoint via Flask-Limiter - PII (email addresses) never sent to Sentry — scrubbed before send - Data retention policy enforced by weekly n8n cleanup workflow - GDPR deletion endpoint exists and tested before launch - Session expiry handled gracefully — PM returned to exact page after re-login - In-progress analysis jobs survive session expiry — resume on return --- ## 19. Build Order Follow this sequence. Do not start a phase until the previous is complete and tested. ``` Phase 1 — Data foundation 1. PocketBase schema (all collections) 2. Repository layer (all repositories with mocks) 3. Service layer (licence, scrape, battlecard, embedding, brief, alert, trip_finder) 4. Flask API routes (all endpoints including /research/find-urls) 5. RQ job queue setup (jobs.py, worker config) 6. Tests for all of the above (80% floor) Phase 2 — Refactor app.py into modular structure 7. Create proxy_overrides PocketBase collection 8. Create repositories/proxy_repository.py 9. Create core/proxy_service.py — Webshare primary + Bright Data fallback 10. Create core/scraper.py — Playwright + playwright-stealth + ProxyService 11. Create core/extractor.py — OpenRouter cycling and JSON parsing 12. Create core/ai_fetch.py — AI web fetch fallback + ProxyService 13. Create industries/adventure_travel/fields.py 14. Create industries/adventure_travel/prompt.py 15. Create industries/adventure_travel/schema.py 16. Create config/client.py 17. Create services/trip_finder_service.py — find_competitor_trip_url(), score_and_rank_urls(), find_all_competitor_urls() 18. Update jobs.py to import from core/ and industries/ 19. Verify all existing functionality works identically after refactor Phase 3 — Dashboard (PRIMARY — build before Apps Script) 20. Clone nuxt-ui-templates/dashboard-vue as frontend base 21. Configure Vue Router: /login — auth page /account — subscription, seats, template download /competitors — add/edit competitor list /analyse — trip-intent form + confirmation + results (CORE PAGE) /battlecards — battlecard view /activity — market movement feed 22. Pinia auth store — tenant_id, email, tier, JWT 23. api_service.js — Axios instance with X-API-Key header 24. LoginPage.vue — email/password + Google OAuth 25. AccountPage.vue — subscription card, seats table, template download 26. CompetitorsPage.vue — competitor list with status badges, add/edit modal 27. AnalysePage.vue — THE CORE PAGE: a. Trip-intent form (destination, duration, style, competitor checkboxes) b. [Find & Analyse] button → POST /research/find-urls c. URL confirmation screen — PM confirms or swaps matches d. [Confirm & Run] → POST /research → poll status e. Per-competitor progress display during job f. Results view — side-by-side comparison table g. Confidence indicators per competitor h. [Push to Sheet ▾] button with tab selector dropdown i. Run history panel — last 5 runs with [Restore] option 28. Stripe integration — checkout, webhook, portal link 29. Resend email templates — welcome, trial reminders, daily digest, weekly brief 30. Freescout deploy — container + support.bettersight.io Pangolin route + support@bettersight.io MX record + support link in dashboard footer 31. useOnboardingTour.js — Shepherd.js 5-step guided tour 32. tour.css — custom Shepherd styles matching Bettersight design system 33. data-tour anchor attributes on nav items and key components 34. PATCH /account/onboarding route 35. Tour trigger on ActivityPage.vue onMounted 36. Tour replay link on AccountPage.vue Phase 4 — Apps Script (export integration — simple) 31. Deploy as standalone Apps Script 32. Validation.gs — validateLicence() email-only 33. Export.gs — pushToSheet(), getWorkbookTabs(), detectColumns() 34. Sidebar.html — minimal: tab selector + push button only 35. Sheet template tab — importable into any existing workbook 36. Flask POST /sheet/push + GET /sheet/tabs routes Phase 5 — Automation 37. n8n daily scrape cron — changed/stale competitors only 38. PocketBase price_history diff logic (change_amount, change_percent) 39. PocketBase is_new flag logic (first_seen detection) 40. Flask /internal/monitor-webhook 41. Flask /internal/monitor/register + /deregister 42. Auto-register Firecrawl Monitor on competitor add 43. n8n PocketBase webhook → create price_change alert record in PocketBase 44. n8n PocketBase webhook → create new_product alert record in PocketBase 45. n8n daily digest email (6pm) — collects unread alerts → Resend email 46. n8n battlecard regeneration workflow (fires on scrape_runs complete) 47. n8n Monday brief workflow (6am — Jinja2 HTML → Resend email) 48. Dashboard bell icon + unread count from alerts table 49. Mark as read / dismiss on AlertCard.vue Phase 6 — Semantic matching 50. embedding_service.py — embed_trip(), find_comparable_trips() 51. n8n embedding and matching workflows (Sunday night) 52. comparable_matches table populated and visible in dashboard Phase 7 — Ship 53. Flask-Limiter configured on all public routes 54. Sentry initialised in Flask and Vue — staging first, then production 55. GDPR deletion endpoint POST /account/delete tested end to end 56. n8n weekly data cleanup workflow (retention policy) 57. Empty states implemented on all seven dashboard pages (incl. ResetPasswordPage) 58. Session expiry handling — token refresh interceptor + return-to logic 59. In-progress job survival across session expiry 60. Password reset — "Forgot password?" on LoginPage + ResetPasswordPage.vue 61. Timezone captured on signup → stored in tenants.timezone 62. n8n daily digest uses timezone-aware send time per tenant (luxon) 63. Upgrade prompt on AccountPage — next tier card with [Upgrade now →] button 64. POST /billing/upgrade route — Stripe checkout for tier upgrade 65. Multi-currency display in ResultsTable — primary market hero + USD secondary 66. Litestream container added to docker-compose.yml — replicating PocketBase to Hetzner S3 67. Hetzner Object Storage backup configured — rclone cron daily 2am (belt + braces with Litestream) 68. robots.txt check added to core/scraper.py — check_robots_txt() before each scrape 69. OpenRouter paid fallback configured — OPENROUTER_FALLBACK_MODEL env var set 70. Scraping legal clause added to bettersight.io/terms 71. Privacy policy and terms pages live at bettersight.io/privacy and /terms 72. Google OAuth consent screen approved (requires privacy policy URL) 73. End to end test with real competitor data 74. Staging deploy 75. Tag version v1.0.0 76. Production deploy via Dokploy ``` --- ## 20. Authentication — PocketBase Native ### Default: Email + Password Standard PocketBase auth. User registers with email and password. PocketBase issues JWT on login. Stored in Pinia auth store. All Flask API calls include `Authorization: Bearer {jwt}` header. ### Option: Google OAuth PocketBase handles the full OAuth flow natively — no custom auth code. **Setup in PocketBase admin:** 1. Settings → Auth Providers → Google → Enable 2. Add Google OAuth Client ID and Secret from Google Cloud Console 3. Set redirect URL to `https://app.bettersight.io/auth/callback` **Frontend flow:** ```javascript // LoginPage.vue — Google OAuth button const loginWithGoogle = async () => { const authData = await pb.collection('users').authWithOAuth2({ provider: 'google' }) // PocketBase returns verified email — same email used for licence validation authStore.setUser(authData) router.push('/account') } ``` **Why this matters for licence validation:** Google OAuth email is verified by Google — same email registered in tenant_seats. Domain matching is perfectly reliable. No manual email entry. No typos. `Session.getActiveUser().getEmail()` in Apps Script returns the exact same email. ### Login page design - Email/password form as default (UForm + UInput + UButton) - "Continue with Google" button below the form (UButton with Google icon) - Trial messaging above the form — reads `?plan=` query param to show correct length: ```javascript // LoginPage.vue const plan = route.query.plan || 'analyse' const TRIAL_COPY = { analyse: '14-day free trial — no credit card required', discover: '21-day free trial — no credit card required', intelligence: '30-day free trial — no credit card required', } const trialCopy = TRIAL_COPY[plan] || TRIAL_COPY.analyse ``` - "Forgot password?" link below the email input field - Link to privacy policy and terms below ### Password Reset Flow PocketBase handles password reset natively via email. The frontend needs two small additions only. **LoginPage.vue — "Forgot password?" link:** ```javascript // Triggers PocketBase native password reset email async function requestPasswordReset() { await pb.collection('users').requestPasswordReset(email.value) toast.success('Reset link sent — check your inbox') } ``` **`/reset-password` route — Vue Router:** ```javascript // Reads token from URL params — PocketBase reset link format: // app.bettersight.io/reset-password?token=xxx async function confirmPasswordReset() { await pb.collection('users').confirmPasswordReset( route.query.token, newPassword.value, newPasswordConfirm.value ) router.push('/login?reset=success') } ``` **Add to Vue Router:** ```javascript { path: '/reset-password', component: ResetPasswordPage } ``` **Add `ResetPasswordPage.vue` to pages:** - New password input + confirm password input - Submit button calls `confirmPasswordReset()` - Redirect to login with success toast on completion - Error state if token is expired or invalid --- ## 21. Professional Onboarding and Brand Touches ### Transactional Emails via Resend All system emails sent via Resend from `alerts@bettersight.io`. Never use raw SMTP for user-facing emails. ```python import resend resend.api_key = os.getenv('RESEND_API_KEY') def send_welcome_email(to_email: str, tenant_name: str, template_url: str): """ Sends branded welcome email on signup with Sheet template download link. Data flow: tenant email + name + template URL → Resend API → inbox """ resend.Emails.send({ "from": "Bettersight ", "to": to_email, "subject": "Welcome to Bettersight — you're all set", "html": render_email_template('welcome', { "name": tenant_name, "template_url": template_url }) }) ``` **Email triggers and templates:** | Trigger | Template | From | |---|---|---| | Signup complete | welcome.html | alerts@bettersight.io | | Trial day 3 | checkin.html | alerts@bettersight.io | | Trial day 7 | feature_tip.html | alerts@bettersight.io | | Weekly brief | brief.html | brief@bettersight.io | | Price alert | alert.html | alerts@bettersight.io | | New product alert | alert.html | alerts@bettersight.io | | Stripe receipt | Stripe native | billing@bettersight.io | All HTML email templates in `backend/templates/emails/`. Every template includes: Bettersight logo, branded header, consistent typography, unsubscribe footer, and link to bettersight.io. ### Trial Periods Trial periods are configured per Stripe price object: ```python TRIAL_PERIODS = { "analyse": 14, # 14 days — low price point, one person decides, value visible in first run "discover": 21, # 21 days — slight committee risk at $499, needs second internal review "intelligence": 30, # 30 days — $999 needs finance approval, signal pipeline needs time to prove value "enterprise": 0, # No trial — custom demo and pilot conversation only } ``` No credit card required to start any trial — configure in Stripe checkout settings (payment_method_collection: 'if_required'). PocketBase tenant status flow: ``` signup → status = 'trial' day 12 (Analyse) / day 19 (Discover) / day 28 (Intelligence) → n8n fires trial_reminder email (2 days before expiry) trial expires → Stripe charges card (if provided) → status = 'active' trial expires → no card provided → status = 'inactive' → access blocked first Stripe charge succeeds → status = 'active' ``` Apps Script and dashboard allow access for both `trial` and `active` status: ```python VALID_STATUSES = ['active', 'trial'] ``` ### Stripe Customer Portal Self-service billing — clients manage their own subscription, upgrade, cancel. No support tickets for billing changes. ```python @app.route('/billing/portal', methods=['POST']) def billing_portal(): """ Creates a Stripe customer portal session for self-service billing. Data flow: tenant_id → PocketBase stripe_customer_id → Stripe portal session → redirect URL returned to frontend """ tenant = tenant_repository.get_by_id(request.json['tenant_id']) session = stripe.billing_portal.Session.create( customer=tenant['stripe_customer_id'], return_url='https://app.bettersight.io/account' ) return jsonify({'url': session.url}) ``` ### Onboarding Checklist Displayed on AccountPage.vue after signup. Tracked in PocketBase tenant record. Updated automatically as each action is completed. Collapses when all steps done. ``` onboarding_checklist (JSON field on tenants) { "add_competitor": false, // set true when first competitor saved "run_analysis": false, // set true when first job completes "push_to_sheet": false, // set true when first push to Sheet runs "download_template": false, // set true when template downloaded "tour_completed": false, // set true when tour reaches final step "tour_skipped": false, // set true when PM clicks Skip "tour_step": 0 // last step reached — for resume on return } ``` Each step checked off automatically when the action is completed. Full checklist completion triggers the day-3 check-in email sequence. --- ### Guided Onboarding Tour A 5-step guided tour fires automatically on first login using Shepherd.js. It is the primary first-run experience — more active than the checklist, which is passive. **Install:** ```bash npm install shepherd.js ``` **Tour triggers:** - Fires automatically on first dashboard load when `tour_completed = false AND tour_skipped = false` - "Take the tour again" link on AccountPage for PMs who skipped - Never fires again after `tour_completed = true` - If PM closes the browser mid-tour, `tour_step` stores their last position and the tour resumes from that step on next login **Tour steps:** ``` Step 1 of 5 — Welcome (no anchor element — modal overlay) ┌─────────────────────────────────────────────────┐ │ 👋 Welcome to Bettersight │ │ │ │ Let's get you to your first competitive │ │ analysis in under 5 minutes. │ │ │ │ [Skip tour] [Let's go →] │ └─────────────────────────────────────────────────┘ Step 2 of 5 — Add a competitor (anchors to Competitors sidebar nav item) ┌─────────────────────────────────────────────────┐ │ 🔍 Add your first competitor │ │ │ │ Click Competitors in the sidebar to add the │ │ first company you want to track. │ │ │ │ You'll need their website URL and their │ │ primary operating market (e.g. United Kingdom).│ │ │ │ [← Back] [Go to Competitors →] │ └─────────────────────────────────────────────────┘ → On [Go to Competitors →]: router.push('/competitors') → Tour pauses. Resumes when PM clicks [+ Add Competitor] Step 3 of 5 — Primary market (anchors to primary_market dropdown in add form) ┌─────────────────────────────────────────────────┐ │ 🌍 Set the primary market │ │ │ │ This tells Bettersight which country to │ │ use when scraping — so you see the right │ │ pricing and currency for that market. │ │ │ │ [← Back] [Next →] │ └─────────────────────────────────────────────────┘ Step 4 of 5 — Run analysis (anchors to Analyse sidebar nav item) ┌─────────────────────────────────────────────────┐ │ ▶ Run your first analysis │ │ │ │ Describe the trip you want to research — │ │ destination, duration, and travel style. │ │ │ │ Bettersight finds the matching competitor │ │ pages automatically. No URLs to copy-paste. │ │ │ │ [← Back] [Go to Analyse →] │ └─────────────────────────────────────────────────┘ → On [Go to Analyse →]: router.push('/analyse') Step 5 of 5 — Push to Sheet (anchors to PushToSheet component) ┌─────────────────────────────────────────────────┐ │ 📊 Push results to your Sheet │ │ │ │ Once your analysis is complete, push the │ │ results directly into your Google Sheet │ │ with one click. Select which tab to write to. │ │ │ │ [← Back] [Done — let's go ✓] │ └─────────────────────────────────────────────────┘ → On [Done]: set tour_completed = true in PocketBase → Tour dismissed permanently ``` **Implementation — `composables/useOnboardingTour.js`:** ```javascript import Shepherd from 'shepherd.js' import 'shepherd.js/dist/css/shepherd.css' import { useRouter } from 'vue-router' import { useTenantStore } from '@/stores/tenant_store' /** * Provides the guided onboarding tour using Shepherd.js. * Fires on first login. Resumes from last step if interrupted. * Marks tour_completed in PocketBase on final step. * * Data flow: * onboarding_checklist.tour_completed → false → * Shepherd.Tour initialised and started → * PM progresses through 5 steps → * final step → PATCH /account/onboarding { tour_completed: true } → * tour never fires again */ export function useOnboardingTour() { const router = useRouter() const tenantStore = useTenantStore() const tour = new Shepherd.Tour({ useModalOverlay: true, defaultStepOptions: { cancelIcon: { enabled: true }, classes: 'bs-tour-step', scrollTo: { behavior: 'smooth', block: 'center' }, when: { cancel: () => markTourSkipped() } } }) tour.addStep({ id: 'welcome', text: ` Welcome to Bettersight 👋

Let's get you to your first competitive analysis in under 5 minutes. `, buttons: [ { text: 'Skip tour', action: tour.cancel, classes: 'bs-btn-ghost' }, { text: "Let's go →", action: tour.next, classes: 'bs-btn-primary' } ] }) tour.addStep({ id: 'add-competitor', attachTo: { element: '[data-tour="nav-competitors"]', on: 'right' }, text: ` Add your first competitor 🔍

Click Competitors to add the first company you want to track. You'll need their website URL and primary operating market. `, buttons: [ { text: '← Back', action: tour.back, classes: 'bs-btn-ghost' }, { text: 'Go to Competitors →', action: () => { router.push('/competitors') saveTourStep(2) tour.next() }, classes: 'bs-btn-primary' } ] }) tour.addStep({ id: 'primary-market', attachTo: { element: '[data-tour="primary-market-field"]', on: 'bottom' }, text: ` Set the primary market 🌍

This tells Bettersight which country to use when scraping — so you see the right pricing and currency for that market. `, buttons: [ { text: '← Back', action: tour.back, classes: 'bs-btn-ghost' }, { text: 'Next →', action: () => { saveTourStep(3) tour.next() }, classes: 'bs-btn-primary' } ] }) tour.addStep({ id: 'run-analysis', attachTo: { element: '[data-tour="nav-analyse"]', on: 'right' }, text: ` Run your first analysis ▶

Describe the trip you want to research — destination, duration, and travel style. Bettersight finds matching competitor pages automatically. No URLs needed. `, buttons: [ { text: '← Back', action: tour.back, classes: 'bs-btn-ghost' }, { text: 'Go to Analyse →', action: () => { router.push('/analyse') saveTourStep(4) tour.next() }, classes: 'bs-btn-primary' } ] }) tour.addStep({ id: 'push-to-sheet', attachTo: { element: '[data-tour="push-to-sheet"]', on: 'top' }, text: ` Push results to your Sheet 📊

Once your analysis completes, push results directly into your Google Sheet with one click. Select which tab to write to. `, buttons: [ { text: '← Back', action: tour.back, classes: 'bs-btn-ghost' }, { text: 'Done — let\'s go ✓', action: () => { markTourCompleted() tour.complete() }, classes: 'bs-btn-primary' } ] }) async function saveTourStep(step) { await tenantStore.updateOnboarding({ tour_step: step }) } async function markTourCompleted() { await tenantStore.updateOnboarding({ tour_completed: true, tour_step: 5 }) } async function markTourSkipped() { await tenantStore.updateOnboarding({ tour_skipped: true }) } function shouldShowTour(onboarding) { return !onboarding.tour_completed && !onboarding.tour_skipped } function getResumeStep(onboarding) { return onboarding.tour_step || 0 } return { tour, shouldShowTour, getResumeStep } } ``` **Trigger in `ActivityPage.vue` (first page after login):** ```javascript // ActivityPage.vue —