The highest-impact fix: run_research_job() only ever read entry['id'] from
confirmed competitors, silently discarding the Firecrawl-matched or
manually-swapped URL — every scrape fell back to the competitor's bare
root domain regardless of what was confirmed. This explains most of the
"unpredictable/wrong data" behavior seen against the old app.py. Threaded
confirmed_url through jobs.py -> analysis_service.py so the actually
confirmed page is what gets scraped.
Other fixes bundled in this pass:
- Proxy layer: Webshare rotating-gateway country code lowercased (was
silently failing auth), Bright Data port corrected to 33335 (current
cert), dedicated-IP candidate rotation added before falling back to the
shared gateway/Bright Data escalation.
- Trip-intent matching: score_and_rank_urls() now hard-gates on
destination keyword and excludes bare root-domain candidates, instead
of letting duration/category signals alone push a wrong-country or
homepage URL over the confidence threshold.
- force_refresh cache-bypass toggle added (opt-in, collapsed "Advanced
options"), plus a "Skip this competitor" affordance for unmatched URLs.
- De-hardcoded "G Adventures" (the hackathon reference client) out of the
extraction prompt — now uses the real tenant's company name.
- RunHistory/Apps-Script "latest results" queries filtered to
triggered_by = "manual" so per-competitor content-hash bookkeeping rows
no longer drown out real batch runs.
- Dashboard fixes: stuck-state bug on revisiting the Analyse page,
AccountPage/BattlecardsPage restacked to single-column layout, Alerts
moved to a header bell dropdown, invisible Export/Run Analysis icons
(fill -> stroke), tour popup width mismatch, horizontal-scroll bug on
cards (missing min-width: 0 in flex layout).
- New PocketBase migrations for scrape_runs (content_hash fields,
tab_type, created_by_email) and products (departures, start_days).
335 backend tests passing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Phase 5 — Automation:
- Daily scrape cron, daily digest email, and Monday brief n8n workflows
- price_history_repository.py + price diffing/is_new detection wired
into analysis_service's refresh path
- Numeric unread-count badge on the topbar bell (was a plain dot)
Change detection architecture correction:
- CLAUDE.md's Firecrawl Monitor design doesn't match the real
self-hosted product: no /v1/monitor in v1, the real /v2/monitor is
cron-scheduled and credit-metered (Supabase-gated), and self-hosted
Firecrawl is a 3-service stack (api/worker/playwright-service), not
the single container originally spec'd
- Replaced with content-hash comparison computed during the scrape
itself (core/scraper.py's compute_content_hash/normalise_html,
analysis_service's detect_change()) — no external dependency, and
the refresh path now skips LLM extraction entirely when a scrape
finds no content diff
- Fixed a real bug in the first pass of this migration: change_detected
was set and then immediately reset within the same function call,
making it unobservable to the CompetitorsPage badge, the Apps Script
sidebar, and the n8n cron filter. It now stays visible until the
next scrape confirms nothing further changed.
- Added a composite index on scrape_runs(competitor_id, content_hash,
started_at) backing the per-competitor recency lookup
- Docker Compose's firecrawl service replaced with the real
firecrawl-api/firecrawl-worker/firecrawl-playwright stack, gated
behind an opt-in profile
- Added CORS (absent from the original spec entirely) so the
browser-based dashboard can reach the API cross-origin
Phase 6 — Semantic trip matching:
- client_trips_repository.py — this collection existed in schema but
nothing ever populated it; comparable-trip matching could never
produce a real match without it
- embedding_service.save_client_trip() upserts a client_trips record
from every /research submission's trip-intent description, with
stale-embedding invalidation when destination/duration change
- /internal/embed-products now embeds client_trips as well as
competitor products; /internal/match-comparable reads both sides
from PocketBase instead of expecting client_trips in the request
body, so it's callable unattended by n8n
- comparable_matches upserts on (client_product, competitor_product)
instead of always creating — first_matched/last_matched only make
sense if repeat weekly matches update in place, and a PM's dismissed
match now survives re-matching
- embedding_and_matching_cron.json — Sunday night n8n workflow
249 tests passing, 96% coverage.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Rearchitects the Sheet integration per updated CLAUDE.md §15: a
stateless Flask backend has no way to reach into whichever Sheet a PM
has open, so the Sheet pulls its latest results on demand rather than
Flask ever pushing to it. This is also the simpler path for
non-technical users — no Sheet ID to find/paste, no service-account
sharing, just one button in the sidebar.
Apps Script (appscript/, new):
- appsscript.json — Editor Add-on manifest, spreadsheets.currentonly
scope (touches only whichever Sheet is open, nothing else in Drive)
- Code.gs — onOpen/onInstall/onHomepage, opens the sidebar
- Validation.gs — validateLicence() (6hr cache, POST /validate-email),
plus markOnboardingStepOnce_() shared helper
- Sync.gs — pullLatestResults()/getWorkbookTabs()/detectColumns(),
with a real header-alias map for every STANDARD_FIELDS column
(majorityPrice matches "Rack Rate"/"Standard Price"/etc, not just
one exact label) so silent write failures don't happen from a PM
naming a column slightly differently than expected
- Sidebar.html — tab selector + Pull latest results + last-pull status
Backend:
- GET /research/history/latest (email-resolved) replaces the removed
POST /sheet/push and GET /sheet/tabs
- GET /account/template now reads TEMPLATE_SHEET_URL instead of a
hardcoded placeholder
- PATCH /account/onboarding now accepts either a JWT session (dashboard)
or an email in the body (Apps Script has no JWT, only the PM's Google
email) — this was a real gap: Sync.gs's onboarding-flag calls would
have 401'd against the JWT-only version
- onboarding_service.DEFAULT_CHECKLIST corrected to the current schema
(template_imported/sidebar_installed/sync_from_sheet, not the stale
push_to_sheet/download_template)
- Fixed the same stale field names in CLAUDE.md §32's onboarding
runbook example payload, so it doesn't teach a future tenant-creation
script the wrong schema
Frontend:
- PushToSheet.vue removed, replaced by SyncToSheet.vue — a clipboard
copy button + instructions to use the sidebar, no tab selector (tab
selection lives in the sidebar, where the workbook is actually open)
- AccountPage.vue's template section rebuilt as three items: open
template, two-step import instructions + sidebar install link, and
a deliberately no-op verification input (never validated/fetched/
stored — the confirmation is purely the PM telling themselves it's
done, per explicit instruction not to create a dependency on their
workbook)
- useOnboardingTour.js step 5 updated to match
220 backend tests passing. Frontend production build verified clean.
Apps Script (.gs/.html) cannot be unit-tested in this environment —
no Google Apps Script runtime available locally; verified only by
careful cross-file consistency review (function names/signatures
matching across Code.gs/Validation.gs/Sync.gs/Sidebar.html).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Closes gaps identified after Phase 3: the four email templates named
in CLAUDE.md's file tree (§5) were only partially built (weekly brief
only). Adds:
- welcome.html, trial_reminder.html, price_alert.html, new_product.html
— matching the brief.html styling pattern
- services/email_service.py — send_welcome_email(),
send_trial_reminder_email(), send_price_alert_email(),
send_new_product_email(), all via the same plain-Jinja2 + Resend
pattern as brief_service.py (side-effect failures logged, never raised)
- Three new internal routes for n8n/manual-onboarding to call:
POST /internal/welcome-email/<tenant_id>
POST /internal/trial-reminder/<tenant_id>
POST /internal/alert-email/<alert_id> (dispatches to price_alert or
new_product template by alert_type, marks delivered_email on success)
- alert_repository.mark_email_delivered() to back that last route
Also adds the "Contact support" (Freescout) placeholder link CLAUDE.md
calls for in both the sidebar (visible on every page) and AccountPage,
pointing at support.bettersight.io. The actual Freescout container,
DNS/Pangolin routing, and mailbox are real infra this backend can't
provision — the link is a placeholder until that's deployed.
216 backend tests passing, 95% coverage, 100% on the new email_service.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Implements CLAUDE.md Build Order Phase 3 — the 7-page dashboard,
matching the approved bettersight-dashboard.html mockup's design
system (dot-grid motif, dark sidebar, card shadows, badges) exactly.
Scope note: nuxt-ui-templates/dashboard-vue (the template CLAUDE.md
says to clone) ships entirely in TypeScript, which directly conflicts
with §18's "JavaScript only, no .ts files" rule. Per user decision,
converted the template to plain JS and rebuilt the dashboard chrome
(sidebar/topbar) as custom components matching the mockup instead of
Nuxt UI's own dashboard components, since the mockup is the approved
design (§25) and Nuxt UI's dashboard components have their own default
look that would fight against it.
- Core plumbing: pocketbase.js client, api_service.js (X-API-Key +
JWT + 401-refresh interceptor), auth_store/analysis_store/
notifications_store (Pinia), 5 repositories, explicit vue-router
(v4, not v5 — matching CLAUDE.md's stack table) with an auth guard
- Shared components: programmatic DotGrid.vue, StatCard, EmptyState,
CompetitorForm, SeatManager, BattlecardCard, SubscriptionCard,
CsvUpload+ImportPreviewModal, FeedItem, AlertCard
- All 7 pages: Login (+Google OAuth, forgot password), ResetPassword,
Account (subscription, upgrade prompt, seats, onboarding), Competitors
(CSV bulk import), Analyse (full find-urls → confirm → poll → results
→ confidence → push-to-sheet flow with all 4 JobProgress terminal
states), Activity (stat row, feed, alerts, onboarding tour trigger),
Battlecards
- Shepherd.js 5-step onboarding tour ported verbatim from §21
- Sentry init (prod-only) + global error handler in main.js
Two backend gaps surfaced and fixed while building this:
- Added GET /account/me (dashboard had no way to learn its own
tenant_id after login) and fixed GET /account/<tenant_id> to verify
the session's tenant matches the requested one (previously any valid
X-API-Key could read any tenant's account data)
- Added GET /alerts and PATCH /alerts/<id>/read (no route existed to
list or acknowledge alerts, needed by the sidebar badge and
ActivityPage's alerts panel)
Deliberately deferred: the mockup's Price Position sparkline/corridor
card has no defined data contract anywhere in CLAUDE.md — omitted
rather than backed with fabricated numbers. /sheet/push and
/sheet/tabs still return their Phase-1 structured 501 pending Phase 4.
Verified: full production build succeeds, dev server serves the login
page correctly (screenshotted, zero console errors), and the router's
auth guard correctly redirects an unauthenticated session to
/login?return=... with no console errors. Backend: 202 tests still
passing after the two route additions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
JasonFraser