# Bettersight — Development Plan > Derived from CLAUDE.md (authoritative spec, read in full) and the two > reference artifacts in this repo: `app.py` (existing scraper to be > refactored, not rewritten) and `bettersight-dashboard.html` (approved > UI mockup). This plan follows the Build Order in CLAUDE.md Section 19 > exactly — same phases, same sequence. Nothing here begins until you > approve it. --- ## 0. Guardrails that apply to every phase These are non-negotiable per CLAUDE.md Section 18 and are not repeated in every phase below, but apply throughout: - **Layering**: `api.py` routes → `services/` (business logic) → `repositories/` (data access, plain dicts only). No layer skips another. `jobs.py` calls services only. - **Every function** gets a one-sentence docstring + a "Data flow:" comment + inline comments on meaningful steps, written *before* the logic, not after. - **Every PocketBase query** for non-admin operations is filtered by `tenant_id`. No exceptions. - **Every credential** comes from an env var. Nothing hardcoded. - **Every Flask route** (except `/health` and `/stripe/webhook`) checks `X-API-Key`, and is rate-limited via Flask-Limiter. - **Every error response** is structured JSON (`error_response()` helper), never plain text or a stack trace. - **Every repository** ships with a mock implementation alongside the real one, for testing. - **Test coverage**: 80% floor overall, 100% on licence validation, Stripe webhook, seat management, rollback paths. - **Frontend**: JavaScript only (no TypeScript), Nuxt UI v4 components only (no custom primitives), pages → composables → repositories → api_service.js, with the same "no layer skips another" rule. - **UI**: light theme, dark sidebar only, dot-grid motif on every feed row / table row / alert row / stat card, per the mockup. - **Do not build** anything in CLAUDE.md Section 27 ("What NOT to Build") — Discover/Intelligence/Enterprise features, custom Sheet functions, admin dashboard, mobile app, etc. --- ## 1. Pre-flight — one thing to fix before Phase 1 `CLAUDE.md` Section 17 (Environment Variables) contains a live-looking value inline: ``` GOTIFY_APP_TOKEN=A78_l3r9y.ZzBr- ``` This violates the doc's own "all credentials from env vars — hardcoded credentials are a build failure" rule, and a real token sitting in a markdown file that may end up in git is a leak risk. **Recommendation: rotate this token in Gotify and place the new value only in `.env` (gitignored), never in CLAUDE.md.** I will not commit this value anywhere. Flagging it now so you can rotate it before this repo is pushed to a remote. I'll also create `.gitignore` (excluding `.env`, `__pycache__`, `node_modules`, `pb_data/`) and initialize git, since this directory is not currently a repo — confirm before I do this, or tell me if you'll handle git setup yourself. --- ## Phase 1 — Data Foundation **Goal:** every PocketBase collection, every repository (+ mock), every service, every Flask route, and the RQ queue skeleton exist and are tested, before any scraping or UI work begins. ### 1.1 PocketBase schema Collections to create (MVP-scope only — excludes Discover/Intelligence collections like `signal_events`, `competitor_people`, `wayback_snapshots`, `tenant_custom_fields`, which are out of scope until those tiers are built): | Collection | Source section | |---|---| | `tenants` | §6 | | `tenant_seats` | §6 | | `competitors` | §6 | | `products` | §6 | | `products_ngs_meta` | §6 | | `price_history` | §6 | | `scrape_runs` | §6 | | `alerts` | §6 | | `battlecards` | §6 | | `comparable_matches` | §6 (populated in Phase 6) | | `tenant_notifications` | §6 (email_digest fields only — Slack/Teams/WhatsApp fields stay null/unused in MVP) | | `digest_logs` | §6 | | `proxy_overrides` | §10 | | `client_trips` | §6 | | `monitoring_events` | §29 (owner-only Gotify monitoring — see note in 1.6) | All fields exactly as specified in CLAUDE.md §6/§10/§29. PocketBase admin UI or migration JSON — I'll write this as a PocketBase migration file (`pb_migrations/*.js`) so schema is versioned and re-creatable, rather than manual admin-UI clicking. **Deliverable:** `pb_migrations/001_initial_schema.js` (or similar), applied to a local PocketBase instance, verified by listing collections. ### 1.2 Repository layer (`backend/repositories/`) One file per collection, each exposing plain-dict CRUD + the specific lookups the spec calls out, each with a `_mock.py` (or in-file `Mock*` class) alongside: - `tenant_repository.py` — `get_by_domain`, `get_by_id`, `update`, `create` - `seat_repository.py` — `validate_or_register`, `count_active`, `create`, `delete` - `competitor_repository.py` — `get_by_url`, `get_by_domain`, `get_by_id`, `list_for_tenant`, `create`, `update`, `delete` - `product_repository.py` — `get_latest_for_competitor`, `create`, `update` - `scrape_repository.py` — `create`, `update`, `get_by_id` - `proxy_repository.py` — `get_by_domain`, `create`, `update` - `alert_repository.py` — `create`, `list_unread`, `mark_read` - `battlecard_repository.py` — `get_for_competitor`, `upsert` - `comparable_repository.py` — `create`, `list_for_tenant`, `dismiss` - `brief_repository.py` — `get_weekly_data`, `log_digest` - `monitoring_repository.py` — `log_event` Every repository: no business logic, no calls to services, returns plain dicts only — per §18. ### 1.3 Service layer (`backend/services/`) - `licence_service.py` — `validate_email()` (three-layer check from §9: domain → seat → ~~sheet_id~~ — **note:** §9 (first occurrence, duplicated heading) describes a 3-layer sheet_id-bound check, but §15 explicitly says sheet_id binding is removed for the standalone Apps Script model. I will implement the **standalone (no sheet_id) version** since it's the later, more specific instruction and matches the `tenants` schema note "sheet_id binding removed". Flagging this contradiction for your awareness — let me know if you intended otherwise.) - `trip_finder_service.py` — `find_competitor_trip_url()`, `score_and_rank_urls()`, `find_all_competitor_urls()` (§8, concurrent via `ThreadPoolExecutor`, max 5 workers) - `battlecard_service.py` — `generate_battlecard()` (§11) - `embedding_service.py` — `embed_trip()`, `find_comparable_trips()` (§12 — built fully in Phase 6, stubbed now) - `brief_service.py` — `send_weekly_brief()` (§13) - `alert_service.py` — `send_gotify()` (§29) + alert-record creation helpers - `analysis_service.py` — orchestrates the fast/refresh path decision (§7) — new service not explicitly named in §5 file tree but required to keep this logic out of `jobs.py` and `api.py` per the layering rule - `onboarding_service.py` — onboarding checklist merge logic (§21) - `billing_service.py` — Stripe checkout/portal/webhook handling (§16, §21) ### 1.4 Flask routes (`backend/api.py`) All routes from §9 plus the additions scattered through later sections (§15 Apps Script support routes, §21 billing/auth, §22 CSV bulk import, §23 GDPR delete). Full route list for Phase 1 (MVP-relevant only — Intelligence-tier `/internal/signal-harvest` etc. excluded): ``` POST /validate-email POST /stripe/webhook POST /research/find-urls POST /research GET /research/status/ GET /research/history GET /research/history/ POST /preview-prompt GET /competitors POST /battlecards POST /comparable-trips POST /sheet/push GET /sheet/tabs GET /account/ POST /account/seats DELETE /account/seats/ POST /account/competitors POST /account/competitors/bulk GET /account/competitors/template PUT /account/competitors/ DELETE /account/competitors/ GET /account/template PATCH /account/onboarding POST /account/delete GET /billing/portal POST /billing/upgrade POST /auth/request-password-reset POST /auth/confirm-password-reset POST /internal/scrape/ POST /internal/battlecard/ POST /internal/embed-products/ POST /internal/match-comparable/ POST /internal/weekly-brief/ POST /internal/monitor-webhook POST /internal/monitor/register/ POST /internal/monitor/deregister/ GET /health ``` Every route: validates `X-API-Key` (except `/health`, `/stripe/webhook` which use Stripe signature verification instead), rate-limited per §22's Flask-Limiter rules, structured JSON errors via `error_response()`. ### 1.5 RQ job queue (`backend/jobs.py`) - Three priority queues (`high`, `normal`, `low`) per §10 — `high` is reserved for Intelligence tier and will sit unused until that tier is built; Analyse-tier jobs go to `normal`. - `run_research_job(job_id, data)` — calls `analysis_service`, never touches DB or scraping logic directly (those come in Phase 2). - Job duration timing + Gotify alert hook (§29 Component 2) wired in now since the structure is simple and avoids a second pass later. ### 1.6 Monitoring note §29 (Gotify alerting, `monitoring_events` collection) isn't enumerated as its own line item in the §19 Build Order, but §18 lists "`/health` endpoint exists and is monitored — automated alert on failure" as a **never-violate** rule. I'm building the `monitoring_events` collection and `alert_service.send_gotify()` now in Phase 1 (cheap, foundational), but the actual n8n polling workflow that calls it is deferred to Phase 7 alongside the other ship-gate items, matching where Sentry/health monitoring appear in the Build Order. ### 1.7 Tests `backend/tests/` mirroring `services/` and `repositories/` structure. Unit tests for every service against mock repositories. Integration tests for every route. 80% floor enforced via `pytest-cov` in CI config (CI itself is out of scope unless you want a GitHub Actions workflow added — flagging as an open question below). **Phase 1 exit criteria:** schema applied, all repos + mocks written, all services written against mocks, all routes return correct shapes against mocked services, RQ worker starts and drains a trivial job, `pytest --cov` shows ≥80%. --- ## Phase 2 — Refactor `app.py` into the modular structure **Goal:** identical scraping behavior, reorganized into `core/` / `industries/adventure_travel/` / `config/client.py`. No logic changes except the explicitly-spec'd additions (playwright-stealth, smart proxy, robots.txt check). Mapping from the existing `app.py` (764 lines, read in full) to the target structure: | Existing `app.py` code | Moves to | Notes | |---|---|---| | `render_page()`, `clean_url()`, `scrape()`, tracker blocklist | `core/scraper.py` | Replace manual `add_init_script` block (lines 147–152) with `playwright_stealth.stealth_async(page)` per §10. Add `country` param + `ProxyService` integration. Add `check_robots_txt()` before each scrape (§30 — was listed as a Phase 7 item in §19 step 68, but logically belongs with the scraper rewrite in Phase 2; I'll stub the check now and confirm gating behavior with you before Phase 7 if timing matters) | | `fetch_with_ai()` | `core/ai_fetch.py` | Unchanged logic, wired through `ProxyService` per §10 | | `extract_with_groq()` (OpenRouter free-model cycling, JSON repair) | `core/extractor.py` | Renamed to `extract_with_openrouter()` per §10's spec; same model-list-with-fallback pattern, same JSON-repair fallback chain (raw parse → regex extract → comma/quote repair). Reads `OPENROUTER_FREE_MODELS` / `OPENROUTER_FALLBACK_MODEL` from env instead of the hardcoded model list in the current file | | `build_prompt()`, `inject_page_content()` | `industries/adventure_travel/prompt.py` | Unchanged | | `STANDARD_FIELDS`, `NGS_FIELDS` | `industries/adventure_travel/fields.py` | Unchanged | | (new — extracted from the JSON shape inside `build_prompt()`) | `industries/adventure_travel/schema.py` | Standard + NGS extraction schema as data, so Phase 6 custom-fields work (deferred) has a clean attach point | | `write_to_sheet()`, `get_sheet_client()`, `get_competitors_from_db()`, `SHEET_ID`/tab constants | `config/client.py` | Unchanged — this is the "swap per client" layer | | Groq/Gemini commented-out alternate providers (lines 569–630) | Dropped | Dead code per spec's "OpenRouter is the active provider" — confirm before deletion since it's clearly intentional reference material the user kept; I'll ask rather than silently delete | | Flask routes (`/health`, `/competitors`, `/preview-prompt`, `/research`) | `api.py` | Already rebuilt in Phase 1 against mocks — Phase 2 rewires them to call the real `core/`/`industries/` functions via `jobs.py` and `analysis_service` instead of running synchronously inline | ### 2.1 New pieces (not in current `app.py`) - `core/proxy_service.py` — `ProxyService` class exactly as specified in §10 (Webshare primary with country targeting, Bright Data fallback, block detection via `_is_blocked()`, auto-escalation recorded in `proxy_overrides`). - `repositories/proxy_repository.py` — already built in Phase 1. - `services/trip_finder_service.py` — already built in Phase 1 against mocks; Phase 2 wires it to the real Firecrawl `/map` endpoint. ### 2.2 Verification Run the refactored pipeline against a known competitor URL and confirm the extracted JSON is byte-for-byte equivalent in shape (same fields, same types) to what the current `app.py` produces, per §19 step 19 ("Verify all existing functionality works identically after refactor"). **Phase 2 exit criteria:** `app.py` is reduced to an entry point that only imports and wires `api.py` + `jobs.py`; a manual end-to-end scrape of one real competitor page produces equivalent output to the pre-refactor code; existing `/research` behavior (now async via RQ instead of synchronous) is confirmed working through the new job queue. --- ## Phase 3 — Dashboard (Vue 3 + Nuxt UI v4) **Goal:** the seven pages, wired to live Flask endpoints, matching `bettersight-dashboard.html` pixel-for-pixel in design language. ### 3.0 What the mockup confirms I read `bettersight-dashboard.html` in full. Key implementation facts it locks in, beyond what CLAUDE.md's prose already states: - Sidebar nav groups: **Intelligence** (Market Movement w/ unread-count badge, Comparable Trips, Battlecards w/ badge, Analyse), **Reports** (Weekly Briefs), **Setup** (Competitors, Account) — this is the route map, slightly more specific than CLAUDE.md's plain page list. - Every dot-grid instance is **hand-authored inline SVG** with per-circle opacity values forming a diagonal fade (top-left dim → bottom-right full saturation). `DotGrid.vue` needs to generate this programmatically (rows × cols × color × opacity-curve) rather than hardcoding 9-circle SVGs per usage, since it's reused at 3 different sizes (24px stat-card grids, 28px feed-row grids, 20–22px table/progress-row grids) with consistent fade math. - Pattern observed: `opacity = base + (row_index * row_step) + (col_index * col_step)`, roughly `.20 → 1.0` across a 3×3 grid, `.12 → 1.0` across the larger 6×4 stat-card grid. I'll match these curves exactly. - Topbar: search input + bell (red dot if unread alerts) + "Export" outline button + "Run Analysis" primary button — confirms Analyse is reachable from every page's topbar, not just its own nav item. "Export" isn't described anywhere in CLAUDE.md prose — I'm treating it as a generic CSV/data export of the current view and will confirm scope with you before building it, since §27 says no "REST API for external integrations" but a UI export button is a different thing. - Stat row is exactly 4 cards (Price Changes / Price Drops / New Products / Competitors) on the Activity page, matching §6 `ActivityPage.vue` spec. - Activity feed has a type filter dropdown + tab row (All / Price changes / New products) with live counts — not explicitly itemized in CLAUDE.md's ActivityPage bullet list but present in the approved mockup, so it's in scope. - Competitor table's "Δ Price" column shows the single most significant recent price move per competitor, not a full price history — confirms `GET /competitors` needs to return a lightweight summary field, not full price_history. - Price Position card (sparkline + horizontal progress bars per competitor) appears only on the mockup, and isn't named in CLAUDE.md Section 6's `ActivityPage.vue` bullets at all. This looks like a Phase 2 (comparable trips / pricing corridor) feature that depends on `client_trips` + per-destination grouping. I'm scoping it into ActivityPage.vue's component tree as `PricePositionCard.vue` since the mockup is the **approved design** per §25 ("All new pages and components must be consistent with this design... open the reference HTML file before writing component code"), but flagging that its data dependency (a "corridor" grouping concept) isn't specified anywhere in the API/schema sections — I'll need to confirm the data shape with you before building its backend support, or treat it as static/mock data initially with a follow-up ticket. ### 3.1 Bootstrap - Clone `nuxt-ui-templates/dashboard-vue` → `frontend/` - `npm install pinia date-fns zod axios papaparse shepherd.js @sentry/vue` - `app.config.ts` — color tokens (`primary: 'blue'`, `gray: 'slate'`), card/badge/button radius overrides per §25. - Global CSS variables from §25 Colour Tokens block, ported into `assets/` (the mockup's inline `