JasonFraser
/
app.bettersight.io
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Fix critical scrape-targeting bug and harden proxy, matching, and dashboard reliability 

The highest-impact fix: run_research_job() only ever read entry['id'] from
confirmed competitors, silently discarding the Firecrawl-matched or
manually-swapped URL — every scrape fell back to the competitor's bare
root domain regardless of what was confirmed. This explains most of the
"unpredictable/wrong data" behavior seen against the old app.py. Threaded
confirmed_url through jobs.py -> analysis_service.py so the actually
confirmed page is what gets scraped.

Other fixes bundled in this pass:
- Proxy layer: Webshare rotating-gateway country code lowercased (was
  silently failing auth), Bright Data port corrected to 33335 (current
  cert), dedicated-IP candidate rotation added before falling back to the
  shared gateway/Bright Data escalation.
- Trip-intent matching: score_and_rank_urls() now hard-gates on
  destination keyword and excludes bare root-domain candidates, instead
  of letting duration/category signals alone push a wrong-country or
  homepage URL over the confidence threshold.
- force_refresh cache-bypass toggle added (opt-in, collapsed "Advanced
  options"), plus a "Skip this competitor" affordance for unmatched URLs.
- De-hardcoded "G Adventures" (the hackathon reference client) out of the
  extraction prompt — now uses the real tenant's company name.
- RunHistory/Apps-Script "latest results" queries filtered to
  triggered_by = "manual" so per-competitor content-hash bookkeeping rows
  no longer drown out real batch runs.
- Dashboard fixes: stuck-state bug on revisiting the Analyse page,
  AccountPage/BattlecardsPage restacked to single-column layout, Alerts
  moved to a header bell dropdown, invisible Export/Run Analysis icons
  (fill -> stroke), tour popup width mismatch, horizontal-scroll bug on
  cards (missing min-width: 0 in flex layout).
- New PocketBase migrations for scrape_runs (content_hash fields,
  tab_type, created_by_email) and products (departures, start_days).

335 backend tests passing.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
JasonFraser 2026-07-04 17:33:05 -04:00
parent e0de2df6bf
commit ad1d7f1815
84 changed files with 4915 additions and 571 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

15
.env.example
View File

@ -12,8 +12,13 @@ REDIS_URL=redis://bettersight-redis:6379
# OpenRouter # OpenRouter
OPENROUTER_API_KEY= OPENROUTER_API_KEY=
OPENROUTER_FREE_MODELS=google/gemini-flash-1.5,meta-llama/llama-3.1-8b-instruct,mistralai/mistral-7b-instruct # MVP launch default: paid-first, no free-tier rotation (see core/extractor.py's
OPENROUTER_FALLBACK_MODEL=anthropic/claude-haiku-4-5 # USE_FREE_MODELS_FIRST comment). Set to "true" to re-enable free-tier
# cycling through FREE_MODELS before falling back to the paid model —
# FREE_MODELS is left populated below so that's a one-line switch.
OPENROUTER_USE_FREE_MODELS_FIRST=false
OPENROUTER_FREE_MODELS=openai/gpt-oss-120b:free,nvidia/nemotron-3-super-120b-a12b:free,meta-llama/llama-3.3-70b-instruct:free
OPENROUTER_FALLBACK_MODEL=deepseek/deepseek-v4-flash
# Firecrawl (self-hosted) # Firecrawl (self-hosted)
FIRECRAWL_URL=http://firecrawl:3002 FIRECRAWL_URL=http://firecrawl:3002
@ -21,6 +26,12 @@ FIRECRAWL_URL=http://firecrawl:3002
# Proxy — Primary (Webshare) # Proxy — Primary (Webshare)
WEBSHARE_USER= WEBSHARE_USER=
WEBSHARE_PASS= WEBSHARE_PASS=
# Dashboard API token (Authorization: Token <key>) — a DIFFERENT credential
# from WEBSHARE_USER/WEBSHARE_PASS above (those are proxy-gateway auth; this
# is for Webshare's account-management REST API, used to list the account's
# own dedicated IPs for candidate rotation). Generate from the Webshare
# dashboard's API/account section, not the proxy connection settings page.
WEBSHARE_API_KEY=
# Proxy — Fallback (Bright Data — pay-as-you-go, auto-escalated per domain) # Proxy — Fallback (Bright Data — pay-as-you-go, auto-escalated per domain)
BRIGHTDATA_USER= BRIGHTDATA_USER=

13
.gitignore vendored
View File

@ -24,6 +24,11 @@ npm-debug.log*
pb_data/ pb_data/
pb_migrations/*.local.js pb_migrations/*.local.js
# Local Firecrawl clone (docker-compose.yml's firecrawl-* build context —
# a third-party repo, not Bettersight code; re-clone with
# `git clone https://github.com/firecrawl/firecrawl.git`)
/firecrawl/
# OS / editor # OS / editor
.DS_Store .DS_Store
.vscode/ .vscode/
@ -31,3 +36,11 @@ pb_migrations/*.local.js
# Google service account credentials # Google service account credentials
*service-account*.json *service-account*.json
# Local Claude Code / agent tooling — not project code
.agents/
.claude/
skills-lock.json
# Personal working files, not part of the app
G Compete App V1.xlsx

658
CLAUDE.md
View File

@ -310,16 +310,94 @@ services:
### Firecrawl — real self-hosted stack ### Firecrawl — real self-hosted stack
Firecrawl self-hosted is a multi-service application, not a single Firecrawl self-hosted is a multi-service application, not a single
container. It needs api + worker + playwright-service and shares the container. It needs api + playwright-service + a NUQ Postgres + a
existing `bettersight-redis` container. No Supabase, no Postgres — RabbitMQ, and the api service shares the existing `bettersight-redis`
those are only required if you enable the credit/auth system, which container for rate limiting. No Supabase — that's only required if you
Bettersight does not use. enable the credit/auth system, which Bettersight does not use
(`USE_DB_AUTHENTICATION=false`).
There is no separate worker container. Checked upstream's own
`docker-compose.yaml` — it defines exactly one `api` service, command
`node dist/src/harness.js --start-docker`. That harness process spins
up the queue workers internally (`NUQ_WORKER_COUNT` etc — see
`apps/api/src/harness.ts`) rather than as a separate process. An
earlier version of this section documented a second `firecrawl-worker`
container running `pnpm run workers`, based on an older split-process
Firecrawl architecture — against the current codebase that container
crash-loops immediately: the runtime image's final stage only copies
`node_modules`/`dist`/`native` (see `apps/api/Dockerfile`'s runtime
stage), not `package.json`, so pnpm has no manifest to resolve
`workers` against. Do not reintroduce a second container here.
Firecrawl's queue backend (NUQ) requires a real Postgres, not just
Redis — this only surfaces once the api container actually starts:
without `NUQ_DATABASE_URL` set, `harness.ts`'s `setupNuqPostgres()`
tries to auto-provision a Postgres container by shelling out to
`docker`/`podman` from *inside* the api container itself, which has
neither installed (and isn't meant to — no Docker-in-Docker). Firecrawl
ships a purpose-built Postgres image for this at `apps/nuq-postgres/`
— `pg_cron` preloaded plus the NUQ schema (`nuq.sql`) baked in as an
initdb script — build that rather than pointing at a vanilla postgres
image, which would boot without the schema NUQ expects. Setting
`POSTGRES_HOST` to that container's name (anything other than
`localhost`) is what flips `setupNuqPostgres()` into its
docker-compose branch, which just builds the connection URL directly
and skips the Docker/Podman auto-provision path entirely — this is
the code path Firecrawl's own authors intended for exactly this setup,
not a workaround.
`NUQ_RABBITMQ_URL` is required, not optional, despite most call sites
in the API gracefully checking `if (config.NUQ_RABBITMQ_URL)` and
falling back to polling mode when absent. The harness always spawns
an `extract-worker` subprocess (`dist/src/services/extract-worker.js`)
regardless of which routes are actually used, and that subprocess's
`consumeExtractJobs()` throws immediately without it — no config
check gates whether it starts at all. The harness treats any
subprocess exit as fatal to the whole process, so without RabbitMQ
the entire api container crash-loops every ~2s even though
Bettersight never calls `/extract`. Upstream's own docker-compose.yaml
confirms this — it lists `rabbitmq` as a hard `depends_on: condition:
service_healthy` for the api service. Use the stock
`rabbitmq:3-management` image — no build needed.
Build context must be the app subdirectory itself (`apps/api`,
`apps/playwright-service-ts`, `apps/nuq-postgres`), not the repo root
— each Dockerfile's COPY instructions (e.g. `apps/api/Dockerfile`'s
`COPY sharedLibs/go-html-to-md ...`) are written relative to that
subdirectory, matching upstream firecrawl's own docker-compose.yaml
(`build: apps/api`). Pointing context at the repo root and `dockerfile:`
at the nested path resolves the Dockerfile correctly but still
resolves every COPY inside it against the repo root instead — breaks
on the first COPY that assumes otherwise. Confirmed against a real
build attempt: `context: <repo>` + `dockerfile: apps/api/Dockerfile`
fails with `"/sharedLibs/go-html-to-md": not found`; `context:
<repo>/apps/api` builds correctly. For a remote git context, Docker's
syntax puts the subdirectory after a colon in the fragment
(`#branch:subdirectory`), not as a separate `dockerfile:` field:
```yaml ```yaml
firecrawl-rabbitmq:
image: rabbitmq:3-management
container_name: firecrawl-rabbitmq
networks:
- pangolin_default
restart: unless-stopped
firecrawl-nuq-postgres:
build:
context: https://github.com/firecrawl/firecrawl.git#main:apps/nuq-postgres
container_name: firecrawl-nuq-postgres
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
- POSTGRES_DB=postgres
networks:
- pangolin_default
restart: unless-stopped
firecrawl-api: firecrawl-api:
build: build:
context: https://github.com/firecrawl/firecrawl.git#main context: https://github.com/firecrawl/firecrawl.git#main:apps/api
dockerfile: apps/api/Dockerfile
container_name: firecrawl-api container_name: firecrawl-api
environment: environment:
- PORT=3002 - PORT=3002
@ -329,35 +407,24 @@ firecrawl-api:
- PLAYWRIGHT_MICROSERVICE_URL=http://firecrawl-playwright:3000/scrape - PLAYWRIGHT_MICROSERVICE_URL=http://firecrawl-playwright:3000/scrape
- USE_DB_AUTHENTICATION=false - USE_DB_AUTHENTICATION=false
- NUM_WORKERS_PER_QUEUE=8 - NUM_WORKERS_PER_QUEUE=8
- POSTGRES_HOST=firecrawl-nuq-postgres
- POSTGRES_PORT=5432
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
- POSTGRES_DB=postgres
- NUQ_RABBITMQ_URL=amqp://firecrawl-rabbitmq:5672
depends_on: depends_on:
- bettersight-redis - bettersight-redis
- firecrawl-playwright - firecrawl-playwright
networks: - firecrawl-nuq-postgres
- pangolin_default - firecrawl-rabbitmq
restart: unless-stopped
firecrawl-worker:
build:
context: https://github.com/firecrawl/firecrawl.git#main
dockerfile: apps/api/Dockerfile
container_name: firecrawl-worker
command: pnpm run workers
environment:
- REDIS_URL=redis://bettersight-redis:6379
- REDIS_RATE_LIMIT_URL=redis://bettersight-redis:6379
- PLAYWRIGHT_MICROSERVICE_URL=http://firecrawl-playwright:3000/scrape
- USE_DB_AUTHENTICATION=false
depends_on:
- bettersight-redis
- firecrawl-playwright
networks: networks:
- pangolin_default - pangolin_default
restart: unless-stopped restart: unless-stopped
firecrawl-playwright: firecrawl-playwright:
build: build:
context: https://github.com/firecrawl/firecrawl.git#main context: https://github.com/firecrawl/firecrawl.git#main:apps/playwright-service-ts
dockerfile: apps/playwright-service-ts/Dockerfile
container_name: firecrawl-playwright container_name: firecrawl-playwright
environment: environment:
- PORT=3000 - PORT=3000
@ -372,8 +439,8 @@ The CX31 VPS (2 vCPU, 8GB RAM) sized in Section 31 handles this
alongside PocketBase, Flask, and 3 RQ workers because Firecrawl is alongside PocketBase, Flask, and 3 RQ workers because Firecrawl is
only used for /v1/map calls (trip-intent matching + weekly discovery), only used for /v1/map calls (trip-intent matching + weekly discovery),
never continuously. Only the api service is called by Bettersight — at never continuously. Only the api service is called by Bettersight — at
`http://firecrawl-api:3002/v1/map`. Worker and playwright-service are `http://firecrawl-api:3002/v1/map`. playwright-service, nuq-postgres,
internal-only. and rabbitmq are internal-only.
```yaml ```yaml
freescout: freescout:
@ -756,13 +823,23 @@ created autodate
Note: `sheet_id` binding removed — standalone Apps Script model, Note: `sheet_id` binding removed — standalone Apps Script model,
validation is email-only. `monitor` tier removed — dropped entirely. validation is email-only. `monitor` tier removed — dropped entirely.
**Timezone capture on signup:** **Timezone capture on first login (not signup — see note):**
There is no signup form in the dashboard's scope — account creation
happens via the Stripe `checkout.session.completed` webhook or the
manual onboarding runbook (§32), neither of which runs browser JS.
First login is the earliest point `Intl.DateTimeFormat` can actually
run, so that's where this happens instead: `auth_store.js`'s
`fetchTenant()` checks `tenant.timezone` after every `GET /account/me`,
and if it's still empty, captures the browser's timezone and sends it
to `PATCH /account/timezone`. Never asks the PM to select one
manually. `PATCH /account/timezone` never overwrites an
already-captured value — first write wins.
```javascript ```javascript
// Captured automatically from browser on account creation // auth_store.js — fires once per tenant, on whichever login is first
// Never ask the PM to select a timezone manually
const timezone = Intl.DateTimeFormat().resolvedOptions().timeZone const timezone = Intl.DateTimeFormat().resolvedOptions().timeZone
// e.g. "Europe/London", "America/New_York", "Australia/Sydney" // e.g. "Europe/London", "America/New_York", "Australia/Sydney"
// Stored in tenants.timezone on first signup POST
``` ```
### tenant_seats ### tenant_seats
@ -866,6 +943,10 @@ competitors_total integer
competitors_done integer competitors_done integer
results json results json
error_log text error_log text
created_by_email text (the seat/PM who submitted the run — set once at
creation from the dashboard's auth session, api.py's
research() route; not persisted for cron-triggered
rows. Shown in RunHistory.vue's "last 5 runs" list.)
started_at autodate started_at autodate
completed_at date completed_at date
``` ```
@ -966,6 +1047,18 @@ const sendAt = DateTime.fromFormat(localTime, 'HH:mm', { zone: tenantTz })
// Compare sendAt.hour and sendAt.minute against current UTC time // Compare sendAt.hour and sendAt.minute against current UTC time
``` ```
Implemented in `n8n/workflows/daily_digest.json` — runs hourly (not
once daily at a fixed UTC hour), comparing only `.hour` since
`email_digest_time` is minute-granular but the schedule itself isn't.
`email_digest`'s "default true" isn't a PocketBase-level field
default (the migration never configured one) — it's handled in the
Code node instead: a tenant with no `tenant_notifications` row at all
is treated as enabled at the default 18:00, since nothing in this
codebase creates that row yet. If something later creates a
`tenant_notifications` row without explicitly setting `email_digest`,
PocketBase's bool zero-value (`false`) would silently disable digests
for that tenant — always set it explicitly on creation.
### digest_logs ### digest_logs
``` ```
id auto id auto
@ -1047,6 +1140,21 @@ In the fast path the LLM does NOT extract data — it analyses it.
PocketBase already has structured JSON. The LLM receives that JSON PocketBase already has structured JSON. The LLM receives that JSON
and generates the analysis narrative, comments, and battlecard content only. and generates the analysis narrative, comments, and battlecard content only.
The returned dict must still carry the cached trip/pricing fields
(tripName, majorityPrice, duration, gbpPrice, etc.) alongside the fresh
narrative — the LLM call itself is constrained to return only
`{comments, relevancy}`, and those two get merged into the cached
fields, not returned as the entire result. A real "complete" job with
an empty-looking dashboard results table surfaced why this matters: an
earlier version returned the LLM's raw output as the whole result, and
since that prompt only ever asked for a narrative with no schema, the
model invented its own ad-hoc JSON shape on every call (a bare
`{"analysis": "..."}`, sometimes with extra invented keys) — never the
`tripName`/`majorityPrice`/`gbpPrice`/`duration`/`comments` shape
`ResultsTable.vue` actually renders. The fast path's whole point is to
reuse already-extracted data, not have the LLM re-derive (or
mis-derive) it.
```python ```python
def analyse_from_cache(tenant_id: str, competitor_id: str, context: dict) -> dict: def analyse_from_cache(tenant_id: str, competitor_id: str, context: dict) -> dict:
""" """
@ -1055,9 +1163,12 @@ def analyse_from_cache(tenant_id: str, competitor_id: str, context: dict) -> dic
Data flow: Data flow:
competitor_id → PocketBase products (latest) → competitor_id → PocketBase products (latest) →
structured JSON + client trip context → structured fields mapped from the cached record (trip_name →
LLM prompt (analysis only, no extraction) → tripName, majority_price_usd → majorityPrice, etc.) →
narrative + comments + relevancy score returned client trip context → LLM prompt constrained to
{comments, relevancy} only →
cached structured fields + fresh comments/relevancy merged and
returned
""" """
``` ```
@ -1071,6 +1182,74 @@ they need fresher data or if server load requires a longer window:
CACHE_STALENESS_HOURS = 24 # in core/scraper.py — adjust here only CACHE_STALENESS_HOURS = 24 # in core/scraper.py — adjust here only
``` ```
### PM-initiated cache bypass — `force_refresh`
A collapsed "Advanced options" disclosure on `TripIntentForm.vue` (closed
by default, not a prominent field) exposes a single "Force fresh scrape"
checkbox, unchecked by default. Unchecked, a run respects the normal
fast/refresh-path decision (§7) — a competitor scraped recently is served
from cache quickly. When checked, `force_refresh: true` rides through the
`/research` POST body → `jobs.py`'s per-competitor `context` dict →
`analysis_service.run_competitor_analysis()`, bypassing BOTH cache
shortcuts for that run: the initial fast-path gate (`needs_refresh()`) and
the post-scrape "content unchanged, reuse cached narrative" shortcut — so
a forced run always ends in a real `extract_with_openrouter()` call,
never `analyse_from_cache()`. Content-hash bookkeeping (`scrape_runs` row,
`change_detected` clearing) still runs normally even when forced, so
future non-forced runs keep an accurate hash history. Not persisted on
the `scrape_runs` job record — it's a one-time execution-time override
with no restore/display use case, unlike `tab_type`.
This briefly defaulted to fully-fresh (checkbox inverted to an opt-in
"prefer cache") while a real bug was being tracked down: the confirmed/
matched trip URL was silently discarded (see below), so cached data could
be for the *wrong page entirely*, not just stale. Once that was fixed, the
cache reflects the correct trip page, so the original opt-in design (fast
by default, force-fresh as the explicit override) is trustworthy again and
was reverted to.
The daily cron (`/internal/scrape/<tenant_id>`, `triggered_by='cron'`)
builds its own request body with no `force_refresh` key and no trip-intent
fields at all — `jobs.py`'s `data.get('force_refresh', False)` naturally
stays `False` there regardless of the dashboard's default, so the cron's
caching efficiency is untouched by this.
### Confirmed/matched URL actually reaches the scrape
`context['confirmed_url']` — the page confirmed at `UrlConfirmation.vue`
(a Firecrawl `/map` match or a manual swap), threaded per-competitor
through `jobs.py`'s loop — is what `run_competitor_analysis()` actually
scrapes, falling back to `competitor['website']` (the stored root domain)
only when nothing was confirmed for this run (e.g. the cron path, which
has no trip-intent context). This was a real regression fixed in this
session: `jobs.py` previously only ever read `entry['id']` from each
confirmed competitor, silently discarding `entry['url']` — every scrape
fell back to the competitor's bare root domain regardless of what was
matched or manually swapped, which is why trip-intent matching (§8)
appeared to have no effect on real analysis runs.
### Generic-homepage detection — `is_generic_page`
`services/url_utils.py`'s `is_generic_homepage(url)` flags a URL with no
meaningful path (the bare root domain, with or without a trailing slash,
query string, or `www` prefix) — used in two places:
- `score_and_rank_urls()` (§8) excludes bare-root candidates before
scoring, the same hard-gate treatment as the destination-keyword check —
a competitor whose only Firecrawl `/map` candidate is its own homepage
correctly falls through to `found: False` rather than a false positive.
- `run_competitor_analysis()` computes `is_generic_page` from whichever URL
was actually scraped this run (or the cached record's own URL, on the
fast path) and returns it as a top-level sibling of `path`/`result` —
not nested inside `result`, since it's a meta-fact about the scrape, not
an extracted trip field. `ResultsTable.vue` renders a "⚠ Generic page"
badge in the competitor's column header when set.
`UrlConfirmation.vue` also gained a per-competitor "Skip this competitor"
toggle, scoped to the `found: false` case — a PM can now exclude a
competitor with no real match from the batch entirely, instead of being
pressured into typing a low-value guess (like the competitor's own
homepage) just to unblock "Confirm & Run."
--- ---
## 8. Trip-Intent Matching ## 8. Trip-Intent Matching
@ -1179,13 +1358,33 @@ def score_and_rank_urls(urls: list, destination: str,
Scores candidate URLs by keyword match against destination Scores candidate URLs by keyword match against destination
and duration. Returns highest scoring URL with confidence score. and duration. Returns highest scoring URL with confidence score.
Scoring rules: A URL must contain at least one destination keyword to be considered
at all — this is a hard gate, not an additive bonus. Regression found
in production: a URL for a completely different country could still
cross the found threshold purely from a duration-digit match plus a
generic category-keyword match (a Turkey trip page scored 0.75
confidence against a Peru search just because it happened to be a
"9 day classic" tour) — category/duration signals only ever score
URLs that pass the destination gate, they can never compensate for
failing it.
A bare root-domain URL (no path at all — see services/url_utils.py's
is_generic_homepage()) is excluded the same way: Firecrawl's /map
candidate list commonly includes the site's own root URL, and if the
destination keyword happens to appear in the domain itself (e.g. a
competitor literally named "peru-travel.com"), the root URL could
otherwise score high enough to be returned as a confident "found"
match — silently presenting a homepage as if it were a trip-specific
page.
Scoring rules (applied only to URLs that pass both gates):
+1 per destination keyword found in URL +1 per destination keyword found in URL
+2 if duration (as integer) appears in URL +2 if duration (as integer) appears in URL
+1 if travel category words appear (classic, trek, trail, etc.) +1 if travel category words appear (classic, trek, trail, etc.)
Confidence = top_score / max_possible_score, capped at 1.0 Confidence = top_score / max_possible_score, capped at 1.0
If confidence < 0.3, returns found=False — prompt manual entry. If no URL passes both gates, or confidence < 0.3,
returns found=False — prompt manual entry.
""" """
dest_keywords = destination.lower().split() dest_keywords = destination.lower().split()
category_keywords = ['classic', 'trek', 'trail', 'safari', category_keywords = ['classic', 'trek', 'trail', 'safari',
@ -1193,16 +1392,22 @@ def score_and_rank_urls(urls: list, destination: str,
scores = [] scores = []
for url in urls: for url in urls:
if is_generic_homepage(url):
continue # bare root domain — never a trip-specific candidate
url_lower = url.lower() url_lower = url.lower()
score = 0 dest_matches = sum(1 for kw in dest_keywords if kw in url_lower)
score += sum(1 for kw in dest_keywords if kw in url_lower) if dest_matches == 0:
continue # zero destination relevance — never a candidate
score = dest_matches
if str(duration) in url_lower: if str(duration) in url_lower:
score += 2 score += 2
score += sum(1 for kw in category_keywords if kw in url_lower) score += sum(1 for kw in category_keywords if kw in url_lower)
scores.append((score, url)) scores.append((score, url))
scores.sort(reverse=True) scores.sort(reverse=True)
if not scores or scores[0][0] == 0: if not scores:
return {"url": None, "confidence": 0, "found": False} return {"url": None, "confidence": 0, "found": False}
top_score, top_url = scores[0] top_score, top_url = scores[0]
@ -1359,6 +1564,8 @@ PUT /account/competitors/<id>
DELETE /account/competitors/<id> — deactivates the competitor record DELETE /account/competitors/<id> — deactivates the competitor record
GET /account/template GET /account/template
PATCH /account/onboarding — update onboarding_checklist fields PATCH /account/onboarding — update onboarding_checklist fields
PATCH /account/timezone — captures browser timezone on first login,
never overwrites an existing value
GET /billing/portal — Stripe customer portal session URL GET /billing/portal — Stripe customer portal session URL
POST /billing/upgrade — creates Stripe checkout for tier upgrade POST /billing/upgrade — creates Stripe checkout for tier upgrade
Body: { tenant_id, target_tier } Body: { tenant_id, target_tier }
@ -1374,6 +1581,9 @@ POST /internal/battlecard/<competitor_id>
POST /internal/embed-products/<tenant_id> POST /internal/embed-products/<tenant_id>
POST /internal/match-comparable/<tenant_id> POST /internal/match-comparable/<tenant_id>
POST /internal/weekly-brief/<tenant_id> POST /internal/weekly-brief/<tenant_id>
POST /internal/daily-digest/<tenant_id>
POST /internal/welcome-email/<tenant_id>
POST /internal/data-cleanup — global, not per-tenant (§22 DATA_RETENTION)
POST /internal/discover/<tenant_id> — Discover tier only POST /internal/discover/<tenant_id> — Discover tier only
``` ```
@ -1658,8 +1868,104 @@ class ProxyService:
Data flow (subsequent requests to a known blocked domain): Data flow (subsequent requests to a known blocked domain):
domain + country → proxy_overrides lookup → brightdata override found → domain + country → proxy_overrides lookup → brightdata override found →
Bright Data config returned immediately — no Webshare attempt Bright Data config returned immediately — no Webshare attempt
Data flow (specific-IP candidate rotation — before ever escalating to
Bright Data): domain not already known-blocked → the account's own
dedicated Webshare IPs for this country fetched (cached) via
get_webshare_ip_candidates() → core/scraper.py's render_page() tries a
handful of those specific IPs in turn → only once all of them are
exhausted (or none were available) does it fall through to the single
`-rotate` gateway attempt and then Bright Data, as before.
""" """
def __init__(self):
# In-memory cache of the account's dedicated Webshare IPs, keyed by
# uppercased country code — {'expires_at': float, 'results': [...]}.
# Same time.time()-based pattern as pocketbase_client.py's token cache.
self._ip_cache = {}
def is_domain_blocked_on_webshare(self, domain: str) -> bool:
"""
Returns True if this domain already has a brightdata override —
lets render_page() skip the specific-IP candidate loop entirely for
a domain already established as Bright-Data-only.
"""
override = proxy_repository.get_by_domain(domain)
return bool(override and override.get('provider') == 'brightdata')
def get_webshare_ip_candidates(self, country: str, limit: int = WEBSHARE_IP_CANDIDATE_LIMIT) -> list:
"""
Returns up to `limit` Playwright proxy configs for specific
dedicated Webshare IPs assigned to this account, filtered to the
target country. Returns [] if none are available (no country,
API failure, or no results) — render_page() treats an empty list
as "skip straight to the existing single-gateway attempt".
Each candidate carries its OWN username/password/port from
Webshare's proxy-list API response — different from
_webshare_playwright()'s single account-level rotating credentials.
"""
if not country:
return []
results = self._fetch_webshare_ip_list(country)
if not results:
return []
chosen = random.sample(results, min(limit, len(results)))
return [
{"server": f"http://{ip['proxy_address']}:{ip['port']}",
"username": ip['username'], "password": ip['password']}
for ip in chosen
]
def _fetch_webshare_ip_list(self, country: str) -> list:
"""
Fetches (and caches) the account's own dedicated Webshare IPs for a
country via Webshare's proxy-list *management* API
(proxy.webshare.io/api/v2/proxy/list/) — a different host and
credential entirely from the proxy gateway (p.webshare.io +
WEBSHARE_USER/PASS). Authenticated with a dashboard API token
(WEBSHARE_API_KEY, "Authorization: Token <key>"), and — per
Webshare's own docs — wants the country code UPPERCASE (their
example: "FR,US"). This is the opposite case convention from
_webshare_playwright()'s gateway username suffix (lowercase) — two
unrelated Webshare systems with inconsistent casing on their end,
not a mistake here. Cached 30 min on success, 5 min on failure (a
short negative TTL bounds a retry storm within one batch run
without permanently caching what might be a transient failure).
"""
cache_key = country.upper()
cached = self._ip_cache.get(cache_key)
if cached and time.time() < cached['expires_at']:
return cached['results']
results = []
try:
url = WEBSHARE_PROXY_LIST_URL
params = {'mode': 'direct', 'country_code__in': cache_key, 'page_size': 100}
for _ in range(3): # defensive pagination cap
response = requests.get(
url, headers={'Authorization': f'Token {WEBSHARE_API_KEY}'},
params=params, timeout=10
)
response.raise_for_status()
data = response.json()
results.extend(r for r in data.get('results', []) if r.get('valid'))
url = data.get('next')
params = None
if not url:
break
self._ip_cache[cache_key] = {
'expires_at': time.time() + WEBSHARE_IP_CACHE_TTL_SECONDS, 'results': results,
}
except (requests.RequestException, ValueError) as e:
logger.warning(f'Webshare proxy list fetch failed for {country}: {str(e)[:200]}')
self._ip_cache[cache_key] = {
'expires_at': time.time() + WEBSHARE_IP_CACHE_NEGATIVE_TTL_SECONDS, 'results': [],
}
return []
return results
def get_proxy_for_domain(self, domain: str, country: str = None) -> dict: def get_proxy_for_domain(self, domain: str, country: str = None) -> dict:
""" """
Returns requests proxy config for a domain with optional country targeting. Returns requests proxy config for a domain with optional country targeting.
@ -1703,10 +2009,22 @@ class ProxyService:
def _webshare_requests(self, country: str = None) -> dict: def _webshare_requests(self, country: str = None) -> dict:
""" """
Webshare requests proxy config with optional country targeting. Webshare requests proxy config with optional country targeting.
Format per Webshare docs: username-{COUNTRY}-rotate Format per Webshare's own rotating-endpoint docs
No country = general rotation across full pool. (help.webshare.io/en/articles/8596715): username-{country}-rotate,
with the country code lowercase — their examples show `-us`, `-gb`,
`-ca`, never uppercase. `country` arrives as an uppercase ISO code
from competitors.primary_market (this schema's own convention —
"GB", "US", "AU"), so this always lowercases it before building the
username rather than assuming the caller already did. An earlier
version of this function (and the real proxy_service.py it
generated) used .upper() here — every proxied scrape failed
Webshare auth silently, which fell through to the AI web-fetch
fallback and then to an outright scrape failure, which in turn
meant analysis_service never reached build_prompt() at all for any
competitor requiring the refresh path. No country = general
rotation across full pool.
""" """
suffix = f"-{country.upper()}-rotate" if country else "-rotate" suffix = f"-{country.lower()}-rotate" if country else "-rotate"
username = f"{WEBSHARE_USER}{suffix}" username = f"{WEBSHARE_USER}{suffix}"
return { return {
"http": f"http://{username}:{WEBSHARE_PASS}@p.webshare.io:80", "http": f"http://{username}:{WEBSHARE_PASS}@p.webshare.io:80",
@ -1714,8 +2032,8 @@ class ProxyService:
} }
def _webshare_playwright(self, country: str = None) -> dict: def _webshare_playwright(self, country: str = None) -> dict:
"""Webshare Playwright proxy config with optional country targeting.""" """Webshare Playwright proxy config with optional country targeting — see _webshare_requests() for why the country code is lowercased."""
suffix = f"-{country.upper()}-rotate" if country else "-rotate" suffix = f"-{country.lower()}-rotate" if country else "-rotate"
return { return {
"server": "http://p.webshare.io:80", "server": "http://p.webshare.io:80",
"username": f"{WEBSHARE_USER}{suffix}", "username": f"{WEBSHARE_USER}{suffix}",
@ -1723,14 +2041,19 @@ class ProxyService:
} }
def _brightdata_requests(self) -> dict: def _brightdata_requests(self) -> dict:
# Port 33335, not the commonly-documented 22225 — Bright Data's
# currently valid SSL certificate requires 33335; 22225 is tied to
# their deprecated cert (phased out ahead of its 2026 expiry). Using
# 22225 is what produced a real net::ERR_CERT_AUTHORITY_INVALID on
# a live scrape before this fix.
return { return {
"http": f"http://{BRIGHTDATA_USER}:{BRIGHTDATA_PASS}@brd.superproxy.io:22225", "http": f"http://{BRIGHTDATA_USER}:{BRIGHTDATA_PASS}@brd.superproxy.io:33335",
"https": f"http://{BRIGHTDATA_USER}:{BRIGHTDATA_PASS}@brd.superproxy.io:22225" "https": f"http://{BRIGHTDATA_USER}:{BRIGHTDATA_PASS}@brd.superproxy.io:33335"
} }
def _brightdata_playwright(self) -> dict: def _brightdata_playwright(self) -> dict:
return { return {
"server": "http://brd.superproxy.io:22225", "server": "http://brd.superproxy.io:33335",
"username": BRIGHTDATA_USER, "username": BRIGHTDATA_USER,
"password": BRIGHTDATA_PASS "password": BRIGHTDATA_PASS
} }
@ -1744,13 +2067,32 @@ async def render_page(url: str, country: str = None) -> dict:
country comes from competitors.primary_market in PocketBase. country comes from competitors.primary_market in PocketBase.
Detects blocks and auto-escalates to Bright Data if Webshare fails. Detects blocks and auto-escalates to Bright Data if Webshare fails.
Before the single `-rotate` gateway attempt, tries a handful of the
account's own dedicated Webshare IPs for this country
(get_webshare_ip_candidates()) — gives IP diversity a chance to resolve
a one-off block without paying for a Bright Data request. Skipped
entirely for a domain already known (via proxy_overrides) to block the
whole Webshare account, and falls through gracefully to the existing
gateway/Bright Data flow when no candidates are available (no country
given, Webshare API unreachable, or every candidate still blocked).
Data flow: Data flow:
url + country → domain extracted → url + country → domain extracted →
is_domain_blocked_on_webshare(domain)? → no →
get_webshare_ip_candidates(country) → try each in turn, return on
first success → all exhausted or none available →
proxy_service.get_playwright_proxy_for_domain(domain, country) → proxy_service.get_playwright_proxy_for_domain(domain, country) →
Playwright render → if blocked → escalate_to_brightdata(domain) → Playwright render → if blocked → escalate_to_brightdata(domain) →
retry with Bright Data (no country targeting at fallback level) retry with Bright Data (no country targeting at fallback level)
""" """
domain = extract_domain(url) domain = extract_domain(url)
if not proxy_service.is_domain_blocked_on_webshare(domain):
for candidate_proxy in proxy_service.get_webshare_ip_candidates(country):
result = await _render_with_proxy(url, candidate_proxy)
if not _is_blocked(result):
return result
proxy = proxy_service.get_playwright_proxy_for_domain(domain, country) proxy = proxy_service.get_playwright_proxy_for_domain(domain, country)
result = await _render_with_proxy(url, proxy) result = await _render_with_proxy(url, proxy)
@ -1806,49 +2148,153 @@ Total proxy cost MVP — under $5/month
### OpenRouter Model Cycling — `core/extractor.py` ### OpenRouter Model Cycling — `core/extractor.py`
Cycles through free tier models. Falls back to a paid model if all Cycles through free tier models, retrying each one up to
free models fail or return rate limit errors. Never fails silently. `RETRY_ATTEMPTS_PER_MODEL` (2) times with backoff before moving to the
next — ported from the legacy `app.py`'s `extract_with_groq()`. An
earlier version of this section (and the Phase 2 refactor built from
it) moved to the next model on the very first error of any kind,
including a transient rate limit — that meant one rate-limit burst
could burn through the entire free-tier list in a single request with
no retry. Falls back to a paid model if every free model is
exhausted. Never fails silently.
`OPENROUTER_FREE_MODELS` model IDs need the `:free` suffix — that's
OpenRouter's actual naming convention for the free-tier variant of a
model; without it the ID may 404 as "no endpoints found" (this is
exactly what shipped in `.env.example` before — a stale, suffix-less
model list, `google/gemini-flash-1.5,meta-llama/llama-3.1-8b-instruct,
mistralai/mistral-7b-instruct`, that 404s on every real request and
silently burns the paid fallback on every single extraction). Current
working default:
```
OPENROUTER_FREE_MODELS=openai/gpt-oss-120b:free,nvidia/nemotron-3-super-120b-a12b:free,meta-llama/llama-3.3-70b-instruct:free
```
`minimax/minimax-m2.5:free` was in this list until a real test run
against a live job showed it 404ing on every single call: OpenRouter
deprecated the free-tier suffix for this model ("This model is
unavailable for free. The paid version is available now - use this
slug instead: minimax/minimax-m2.5"). Unlike a rate limit, this is not
transient — every extraction call was guaranteed to burn both retry
attempts on a model that can never succeed. Removed rather than left
in to fail predictably; OpenRouter's free-tier lineup shifts over time,
so revisit this list if extractions start failing at every free model
again.
**MVP launch runs paid-first, not free-tier-first.** The same test
run that surfaced the dead `minimax` free-tier model also showed
`openai/gpt-oss-120b:free` eating its full rate-limit backoff (5s +
10s) on nearly every extraction call — combined, a single research job
took 5m47s, which blew past the dashboard's polling timeout (raised
from 5 to 20 minutes in `analysis_store.js` as a separate fix, but the
underlying slowness is still worth avoiding). Free-tier rotation is not
reliable enough to depend on for launch, so
`OPENROUTER_USE_FREE_MODELS_FIRST=false` skips `FREE_MODELS` entirely
and calls `FALLBACK_MODEL` directly — set to `deepseek/deepseek-v4-flash`
rather than `anthropic/claude-haiku-4-5`.
`minimax/minimax-m2.5` (the paid slug) was tried here first, since it
was already the paid fallback other fixes in this session had set up —
it failed 100% of the time (10/10 calls: 2 retry attempts × 5
competitors in one real job) with `_call_openrouter`'s `_parse_json_response`
crashing on `None` content. Root cause, confirmed directly against
OpenRouter's API using the actual production prompt (real JSON schema
+ realistic page-content length, not a synthetic approximation):
`minimax/minimax-m2.5` has *mandatory* reasoning that cannot be
disabled (`reasoning: {enabled: false}` returns `400 Reasoning is
mandatory for this endpoint and cannot be disabled`), and its hidden
chain-of-thought token consumption scales unpredictably with how much
the model has to reason through — real page content demands far more
reasoning than a short, repetitive test string, so it's much more
likely to consume the entire `max_tokens=4096` budget before ever
writing the visible JSON answer, leaving `content: null`. Small/simple
test prompts against this model succeed fine, which is why isolated
smoke-testing didn't catch it — only a prompt shaped like the real
extraction call did. `deepseek/deepseek-v4-flash` was tested against
the identical real prompt (4/4 successful runs, `finish_reason: "stop"`
every time, no `None` content) and is also roughly 3.5x cheaper per
call. If a future model swap is considered, always test against the
real `build_prompt()` output at realistic page-content length — a
short synthetic prompt is not sufficient to catch a reasoning-budget
truncation failure like this one.
`_call_openrouter()` also sends `response_format={'type': 'json_object'}`
on every call — a second, unrelated real failure surfaced after the
`deepseek` switch: it would write a full prose comparative analysis
(markdown headers, tables, "Key Differentiators" sections) instead of
the requested JSON whenever a competitor's page content didn't clearly
match the requested trip, prioritizing a "helpful" explanation of the
mismatch over the format instruction. Not reproducible with a short
synthetic prompt either — only the real, long production prompt
triggered it. `response_format` constrains this at the API level
rather than relying on prompt wording alone, and is a systemic fix for
this whole class of "model deviates into prose" failure, not just the
one reproduced case. Confirmed compatible with every model in
`OPENROUTER_FREE_MODELS`/`OPENROUTER_FALLBACK_MODEL` before shipping —
a model that rejects the parameter would 400 immediately, not the 429s
(rate limiting, unrelated) seen during that compatibility check.
`FREE_MODELS` is left populated and untouched in `.env`/`.env.example`
specifically so re-enabling free-tier cycling later is a one-line
flip (`OPENROUTER_USE_FREE_MODELS_FIRST=true`) — nothing else needs
reconfiguring.
```python ```python
FREE_MODELS = os.getenv('OPENROUTER_FREE_MODELS', '').split(',') FREE_MODELS = os.getenv('OPENROUTER_FREE_MODELS', '').split(',')
FALLBACK_MODEL = os.getenv('OPENROUTER_FALLBACK_MODEL', FALLBACK_MODEL = os.getenv('OPENROUTER_FALLBACK_MODEL',
'anthropic/claude-haiku-4-5') 'anthropic/claude-haiku-4-5')
# MVP launch toggle — false skips FREE_MODELS entirely and calls
# FALLBACK_MODEL directly. See the paragraph above for why.
USE_FREE_MODELS_FIRST = os.getenv('OPENROUTER_USE_FREE_MODELS_FIRST', 'true').lower() == 'true'
RETRY_ATTEMPTS_PER_MODEL = 2
def extract_with_openrouter(prompt: str, content: str) -> dict: def extract_with_openrouter(prompt: str, content: str) -> dict:
""" """
Cycles through free OpenRouter models in sequence. Cycles through free OpenRouter models, retrying each one up to
RETRY_ATTEMPTS_PER_MODEL times before moving to the next.
Falls back to the paid fallback model if all free models fail. Falls back to the paid fallback model if all free models fail.
Never returns None — raises ExtractorError only after all models exhausted. Never returns None — raises ExtractorError only after all models exhausted.
Skips straight to FALLBACK_MODEL when USE_FREE_MODELS_FIRST is false.
Data flow: Data flow:
prompt + content → for each FREE_MODELS: prompt + content → for each model in FREE_MODELS + [FALLBACK_MODEL]
(or just [FALLBACK_MODEL] when USE_FREE_MODELS_FIRST is false):
up to RETRY_ATTEMPTS_PER_MODEL attempts:
POST openrouter.ai/api/v1/chat/completions → POST openrouter.ai/api/v1/chat/completions →
if success: return parsed JSON → if success: return parsed JSON →
if rate limit or error: try next model → if rate limit: sleep 5*(attempt+1)s, retry same model →
if credit/quota exhausted (403): move to next model immediately
(quota won't recover mid-request — retrying wastes time) →
if any other error: retry same model up to the attempt limit,
then move to next model →
if all free models fail → try FALLBACK_MODEL (paid) → if all free models fail → try FALLBACK_MODEL (paid) →
if fallback fails → raise ExtractorError, log to Sentry if fallback fails → raise ExtractorError, log to Sentry
""" """
models_to_try = FREE_MODELS + [FALLBACK_MODEL] models_to_try = FREE_MODELS + [FALLBACK_MODEL] if USE_FREE_MODELS_FIRST else [FALLBACK_MODEL]
for model in models_to_try: for model in models_to_try:
for attempt in range(RETRY_ATTEMPTS_PER_MODEL):
try: try:
response = _call_openrouter(model, prompt, content) response = _call_openrouter(model, prompt, content)
if response: if response:
if model == FALLBACK_MODEL: if USE_FREE_MODELS_FIRST and model == FALLBACK_MODEL and FREE_MODELS:
logger.warning( logger.warning(
f'Free models exhausted — used paid fallback: {model}' f'Free models exhausted — used paid fallback: {model}'
) )
sentry_sdk.capture_message( sentry_sdk.capture_message(
f'OpenRouter paid fallback used', 'OpenRouter paid fallback used',
level='warning' level='warning'
) )
return response return response
except RateLimitError: except RateLimitError:
logger.warning(f'Rate limit on {model} — trying next') wait = 5 * (attempt + 1)
continue logger.warning(f'{model} rate limited — waiting {wait}s...')
time.sleep(wait)
except CreditExhaustedError:
logger.warning(f'{model} free tier exhausted — trying next model...')
break
except Exception as e: except Exception as e:
logger.error(f'Model {model} failed: {str(e)}') logger.warning(f'{model} failed: {str(e)[:200]}')
continue
raise ExtractorError('All OpenRouter models exhausted including paid fallback') raise ExtractorError('All OpenRouter models exhausted including paid fallback')
``` ```
@ -2707,12 +3153,17 @@ POCKETBASE_ADMIN_PASSWORD=
# OpenRouter # OpenRouter
OPENROUTER_API_KEY= OPENROUTER_API_KEY=
OPENROUTER_FREE_MODELS=google/gemini-flash-1.5,meta-llama/llama-3.1-8b-instruct,mistralai/mistral-7b-instruct # MVP launch default: paid-first — see §10 "OpenRouter Model Cycling"
OPENROUTER_FALLBACK_MODEL=anthropic/claude-haiku-4-5 # paid fallback ~$0.25/1M tokens — used when free tier unavailable # for why. Set to "true" to re-enable free-tier cycling; FREE_MODELS
# stays populated below so that's a one-line switch.
OPENROUTER_USE_FREE_MODELS_FIRST=false
OPENROUTER_FREE_MODELS=openai/gpt-oss-120b:free,nvidia/nemotron-3-super-120b-a12b:free,meta-llama/llama-3.3-70b-instruct:free
OPENROUTER_FALLBACK_MODEL=deepseek/deepseek-v4-flash # paid — used directly at launch (USE_FREE_MODELS_FIRST=false), or as fallback once free tier is re-enabled. Not minimax/minimax-m2.5 — see §10, mandatory-reasoning model, 100% failure rate on the real extraction prompt
# Proxy — Primary (Webshare) # Proxy — Primary (Webshare)
WEBSHARE_USER= WEBSHARE_USER=
WEBSHARE_PASS= WEBSHARE_PASS=
WEBSHARE_API_KEY= # dashboard API token (Authorization: Token <key>) — different credential from WEBSHARE_USER/PASS above (those are proxy-gateway auth); used only to list the account's own dedicated IPs for candidate rotation. Generate from the Webshare dashboard's API/account section.
# Proxy — Fallback (Bright Data — pay-as-you-go, auto-escalated per domain) # Proxy — Fallback (Bright Data — pay-as-you-go, auto-escalated per domain)
BRIGHTDATA_USER= BRIGHTDATA_USER=
@ -2891,6 +3342,14 @@ def validate_email(email: str, sheet_id: str) -> dict:
- 7-day retention on Hetzner Object Storage - 7-day retention on Hetzner Object Storage
- Recovery point objective: 24 hours (fallback if Litestream stream is corrupted) - Recovery point objective: 24 hours (fallback if Litestream stream is corrupted)
Implemented in `scripts/backup-pocketbase.sh` and
`scripts/bettersight-backup.cron` — see that file for exact install
steps (`cp` to `/usr/local/bin/`, `chmod +x`, `cp` the crontab entry to
`/etc/cron.d/bettersight-backup`). Requires an `rclone` remote named
`hetzner-s3` configured on the VPS with the `HETZNER_S3_*` credentials.
Cron output is redirected to `/var/log/bettersight-backup.log` (a
small addition beyond the original spec below, for debuggability).
```bash ```bash
# /etc/cron.d/bettersight-backup # /etc/cron.d/bettersight-backup
0 2 * * * root /usr/local/bin/backup-pocketbase.sh 0 2 * * * root /usr/local/bin/backup-pocketbase.sh
@ -3043,7 +3502,8 @@ Phase 7 — Ship
58. Session expiry handling — token refresh interceptor + return-to logic 58. Session expiry handling — token refresh interceptor + return-to logic
59. In-progress job survival across session expiry 59. In-progress job survival across session expiry
60. Password reset — "Forgot password?" on LoginPage + ResetPasswordPage.vue 60. Password reset — "Forgot password?" on LoginPage + ResetPasswordPage.vue
61. Timezone captured on signup → stored in tenants.timezone 61. Timezone captured on first login (no signup form in scope) →
PATCH /account/timezone → stored in tenants.timezone
62. n8n daily digest uses timezone-aware send time per tenant (luxon) 62. n8n daily digest uses timezone-aware send time per tenant (luxon)
63. Upgrade prompt on AccountPage — next tier card with [Upgrade now →] button 63. Upgrade prompt on AccountPage — next tier card with [Upgrade now →] button
64. POST /billing/upgrade route — Stripe checkout for tier upgrade 64. POST /billing/upgrade route — Stripe checkout for tier upgrade
@ -4190,33 +4650,57 @@ DATA_RETENTION = {
**n8n weekly cleanup workflow:** **n8n weekly cleanup workflow:**
``` ```
Schedule Trigger (Sunday 3am) Schedule Trigger (Sunday 03:00)
→ DELETE scrape_runs older than 90 days → HTTP POST /internal/data-cleanup — one global call, not per-tenant
→ DELETE alerts older than 90 days where read = true → services/retention_service.run_weekly_cleanup():
→ DELETE price_history older than 365 days DELETE scrape_runs older than 90 days
→ Log cleanup run to audit_logs DELETE alerts older than 90 days where read = true (unread alerts
are never auto-deleted regardless of age)
DELETE price_history older than 365 days
→ counts logged (see billing_service's note on why there's no
audit_logs collection to log to)
``` ```
`weekly_data_cleanup_cron.json` in `n8n/workflows/`.
**Data deletion endpoint (GDPR Article 17 — Right to Erasure):** **Data deletion endpoint (GDPR Article 17 — Right to Erasure):**
Implemented as `services/billing_service.delete_tenant_data()`, called
from the thin `POST /account/delete` route.
```python ```python
@app.route('/account/delete', methods=['POST']) def delete_tenant_data(tenant_id):
@limiter.limit("3 per day")
def delete_account():
""" """
Permanently deletes all tenant data on request. Permanently deletes all tenant data on request.
Cancels Stripe subscription first, then purges PocketBase records. Cancels Stripe subscription first, then purges PocketBase records.
Sends confirmation email via Resend. Sends confirmation email via Resend.
Data flow: Data flow:
tenant_id → cancel Stripe subscription → tenant_id → cancel Stripe subscription (side effect, logged not
delete: tenant_seats, competitors, products, price_history, raised) → delete: tenant_seats, competitors, products,
alerts, scrape_runs, battlecards, comparable_matches → price_history, alerts, scrape_runs, battlecards,
comparable_matches, client_trips →
delete: tenant record → delete: tenant record →
send deletion confirmation email → send deletion confirmation email (side effect, logged not raised) →
log to audit_logs (audit logs are never deleted) logged
""" """
``` ```
`client_trips` (Phase 6) is included even though it predates this
section — it's tenant-scoped product data the same as everything else
on the list.
**"log to audit_logs" note:** no `audit_logs` collection exists
anywhere in this schema — the closest candidates (`alerts`,
`digest_logs`) are either on the delete list itself or hold no PII.
Rather than invent new schema for one log line, the deletion is
recorded via standard Python logging (`logger.info`, visible in
Sentry/server logs). Revisit if a real compliance audit trail is ever
required. This also resolves an apparent tension with §18's "alerts
are append-only" note — that describes normal runtime retention, not
an override of a tenant's explicit erasure request, and alerts
describe competitor products, not the PM, so they carry no PII of
their own.
**Privacy policy must state:** **Privacy policy must state:**
- What data is collected (email, usage data) - What data is collected (email, usage data)
- How long it is retained (per DATA_RETENTION above) - How long it is retained (per DATA_RETENTION above)
@ -4392,6 +4876,28 @@ VITE_ENV=production # staging | production
- Rate limit headers included in all responses (X-RateLimit-*) - Rate limit headers included in all responses (X-RateLimit-*)
- Internal routes (/internal/*) exempt from rate limiting — called by n8n only, - Internal routes (/internal/*) exempt from rate limiting — called by n8n only,
protected by X-API-Key header instead protected by X-API-Key header instead
- OPTIONS requests are exempt from rate limiting entirely (`extensions.py`'s
`_exempt_cors_preflight` request_filter) — a browser's CORS preflight is
not a real request and carries no X-API-Key, but Flask-Limiter counts it
against the same bucket as the real request unless explicitly filtered.
Discovered when analysis_store.js's 3s status-poll loop tripped a 429
on the preflight itself within under a minute: each cross-origin fetch
fires OPTIONS before the real GET, doubling counted requests against
the 30/minute default. A failed preflight has no response body, so the
browser reports it as an opaque network error — this is why it showed
up client-side as the generic "Lost connection" fallback rather than a
429 with a real message.
- Polled endpoints need their own `@limiter.limit(..., override_defaults=True)`
rather than the blanket default_limits — `/research/status/<job_id>` is
polled every 3s for up to 20 minutes (analysis_store.js's
`POLL_TIMEOUT_ATTEMPTS`), which alone exceeds the 200/hour default
before a single job even finishes. Set to 3000/hour (50/minute
sustained) — comfortably covers one job's worst case plus several
concurrent jobs/tabs, still bounded well below anything meaningful for
a runaway script against this cheap, read-only lookup. `override_defaults=True`
is required — without it, Flask-Limiter applies the route's own limit
*in addition to* the blanket defaults, not instead of them, so the
200/hour ceiling would still apply underneath a higher per-route limit.
**Price:** $999/month (target $1,499/month with case study) **Price:** $999/month (target $1,499/month with case study)
**Seats:** 10 (additional at $45/seat/month) **Seats:** 10 (additional at $45/seat/month)

103
backend/api.py
View File

@ -240,10 +240,21 @@ def research():
except Exception as e: except Exception as e:
logger.error(f'save_client_trip failed for tenant {data["tenant_id"]}: {str(e)}') logger.error(f'save_client_trip failed for tenant {data["tenant_id"]}: {str(e)}')
# Note: data may also carry 'force_refresh' (TripIntentForm.vue's
# collapsed cache-bypass toggle) — deliberately NOT persisted here.
# Unlike tab_type (needed later to reconstruct row layout on history
# restore), force_refresh is a one-time execution-time override with
# no restore/display use case; it still reaches jobs.py via the raw
# `data` dict passed to enqueue_research_job() below.
run = scrape_repository.create({ run = scrape_repository.create({
'tenant_id': data['tenant_id'], 'tenant_id': data['tenant_id'],
'triggered_by': 'manual', 'triggered_by': 'manual',
'status': 'pending', 'status': 'pending',
'tab_type': data['tab_type'],
# Whichever seat (PM) submitted this run, for RunHistory.vue's
# display — sourced from the dashboard's auth session, not looked
# up later (a seat could be removed after the run).
'created_by_email': data.get('email'),
'competitors_total': len(data['competitors']), 'competitors_total': len(data['competitors']),
'competitors_done': 0, 'competitors_done': 0,
'results': {}, 'results': {},
@ -253,6 +264,18 @@ def research():
@api_bp.route('/research/status/<job_id>', methods=['GET']) @api_bp.route('/research/status/<job_id>', methods=['GET'])
# override_defaults=True replaces (not adds to) the blanket 200/hour
# default — a polling endpoint needs a rate suited to its actual call
# pattern. analysis_store.js polls every 3s for up to 20 minutes
# (POLL_TIMEOUT_ATTEMPTS=400), so the 200/hour default alone was
# tripping a 429 partway through a single job's polling lifecycle —
# discovered when a real test job's status checks started failing with
# "ratelimit 200 per 1 hour exceeded at endpoint: api.research_status".
# 3000/hour = 50/minute sustained, comfortably covering one job's worst
# case (20/minute) plus several concurrent jobs/tabs, while still
# bounding a genuinely runaway script far below anything meaningful for
# this cheap, read-only lookup.
@limiter.limit('3000 per hour', override_defaults=True)
@require_api_key @require_api_key
def research_status(job_id): def research_status(job_id):
run = scrape_repository.get_by_id(job_id) run = scrape_repository.get_by_id(job_id)
@ -260,6 +283,7 @@ def research_status(job_id):
return error_response('not_found', 'Unknown job_id', 404) return error_response('not_found', 'Unknown job_id', 404)
return jsonify({ return jsonify({
'status': run.get('status'), 'status': run.get('status'),
'tab_type': run.get('tab_type'),
'progress': { 'progress': {
'total': run.get('competitors_total', 0), 'total': run.get('competitors_total', 0),
'done': run.get('competitors_done', 0), 'done': run.get('competitors_done', 0),
@ -269,6 +293,18 @@ def research_status(job_id):
}) })
@api_bp.route('/research/cancel/<job_id>', methods=['POST'])
@limiter.limit('20 per hour')
@require_api_key
def research_cancel(job_id):
"""PM-initiated stop — see jobs.cancel_research_job() for why this is distinct from disconnect-survival."""
from jobs import cancel_research_job
if not cancel_research_job(job_id):
return error_response('not_found', 'Unknown job_id', 404)
return jsonify({'status': 'cancelling'})
@api_bp.route('/research/history', methods=['GET']) @api_bp.route('/research/history', methods=['GET'])
@require_api_key @require_api_key
def research_history(): def research_history():
@ -324,6 +360,7 @@ def preview_prompt():
'[Competitor Name]', placeholder_page, '[Competitor Name]', placeholder_page,
data.get('productName', ''), data.get('destination', ''), data.get('productName', ''), data.get('destination', ''),
data.get('duration', ''), data.get('travelStyle', ''), data.get('duration', ''), data.get('travelStyle', ''),
company_name=data.get('companyName'),
is_ngs=data.get('tabType') == 'NGS' is_ngs=data.get('tabType') == 'NGS'
) )
return jsonify({'prompt': prompt}) return jsonify({'prompt': prompt})
@ -591,6 +628,34 @@ def account_template():
return jsonify({'template_url': TEMPLATE_SHEET_URL}) return jsonify({'template_url': TEMPLATE_SHEET_URL})
@api_bp.route('/account/timezone', methods=['PATCH'])
@require_api_key
def update_timezone():
"""
Captures the PM's browser timezone on first login when
tenants.timezone is still empty (CLAUDE.md §19 Phase 7 item 61).
There's no signup form in this dashboard's scope — account
creation happens via Stripe webhook or the manual onboarding
runbook (§32) — so first login is the earliest point browser JS
can run to capture Intl.DateTimeFormat().resolvedOptions().timeZone.
Never overwrites a timezone that's already set.
"""
tenant_id = auth_service.get_tenant_id_from_request(request)
if not tenant_id:
return error_response('unauthorized', 'Could not resolve tenant from session', 401)
tenant = tenant_repository.get_by_id(tenant_id)
if tenant.get('timezone'):
return jsonify({'timezone': tenant['timezone']})
timezone = (request.json or {}).get('timezone')
if not timezone:
return error_response('missing_field', 'timezone is required', 400)
updated = tenant_repository.update(tenant_id, {'timezone': timezone})
return jsonify({'timezone': updated['timezone']})
@api_bp.route('/account/onboarding', methods=['PATCH']) @api_bp.route('/account/onboarding', methods=['PATCH'])
@require_api_key @require_api_key
def update_onboarding(): def update_onboarding():
@ -624,32 +689,22 @@ def update_onboarding():
@require_api_key @require_api_key
def delete_account(): def delete_account():
""" """
GDPR Article 17 deletion. Cancels Stripe first, then purges every GDPR Article 17 deletion. See billing_service.delete_tenant_data()
tenant-scoped collection. Audit logs (digest_logs) are append-only for the full data flow — cancels Stripe, purges every tenant-scoped
and are intentionally NOT deleted here. collection including the tenant record itself, and sends a
deletion confirmation email. digest_logs is the one collection
intentionally left untouched (CLAUDE.md's own delete list doesn't
include it, and it holds no PII — just send timestamps/status).
""" """
data = request.json or {} data = request.json or {}
tenant_id = data.get('tenant_id') tenant_id = data.get('tenant_id')
if not tenant_id: if not tenant_id:
return error_response('missing_field', 'tenant_id is required', 400) return error_response('missing_field', 'tenant_id is required', 400)
tenant = tenant_repository.get_by_id(tenant_id) deleted = billing_service.delete_tenant_data(tenant_id)
if not tenant: if not deleted:
return error_response('not_found', 'Unknown tenant', 404) return error_response('not_found', 'Unknown tenant', 404)
if tenant.get('stripe_subscription_id'):
try:
stripe = billing_service._stripe()
stripe.Subscription.delete(tenant['stripe_subscription_id'])
except Exception as e:
logger.error(f'Stripe cancellation failed during account deletion: {str(e)}')
for seat in seat_repository.list_for_tenant(tenant_id):
seat_repository.delete(seat['id'])
for competitor in competitor_repository.list_for_tenant(tenant_id, active_only=False):
competitor_repository.update(competitor['id'], {'active': False})
tenant_repository.update(tenant_id, {'status': 'inactive'})
return jsonify({'status': 'ok'}) return jsonify({'status': 'ok'})
@ -810,6 +865,18 @@ def internal_daily_digest(tenant_id):
return jsonify({'sent': sent}) return jsonify({'sent': sent})
@api_bp.route('/internal/data-cleanup', methods=['POST'])
@require_api_key
def internal_data_cleanup():
"""
Called once (not per-tenant — this is a global sweep) by the n8n
Sunday 3am weekly cleanup workflow (CLAUDE.md §22 DATA_RETENTION).
"""
from services import retention_service
result = retention_service.run_weekly_cleanup()
return jsonify(result)
@api_bp.route('/internal/welcome-email/<tenant_id>', methods=['POST']) @api_bp.route('/internal/welcome-email/<tenant_id>', methods=['POST'])
@require_api_key @require_api_key
def internal_welcome_email(tenant_id): def internal_welcome_email(tenant_id):

30
backend/app.py
View File

@ -1,3 +1,4 @@
import os
import logging import logging
from flask import Flask from flask import Flask
from dotenv import load_dotenv from dotenv import load_dotenv
@ -5,6 +6,31 @@ from dotenv import load_dotenv
load_dotenv() load_dotenv()
logging.basicConfig(level=logging.INFO) logging.basicConfig(level=logging.INFO)
# Sentry — guarded on SENTRY_DSN being set, so earlier phases and local
# dev never silently depend on it (CLAUDE.md §22 Observability). Must
# be initialised before the Flask app is constructed so FlaskIntegration
# can instrument it.
SENTRY_DSN = os.getenv('SENTRY_DSN')
if SENTRY_DSN:
import sentry_sdk
from sentry_sdk.integrations.flask import FlaskIntegration
from sentry_sdk.integrations.rq import RqIntegration
def _scrub_sensitive_data(event, hint):
"""Removes headers (which may carry X-API-Key/session tokens) from Sentry events. PII must never reach Sentry."""
if 'request' in event:
event['request'].pop('headers', None)
return event
sentry_sdk.init(
dsn=SENTRY_DSN,
integrations=[FlaskIntegration(), RqIntegration()],
traces_sample_rate=0.1,
environment=os.getenv('FLASK_ENV', 'production'),
send_default_pii=False,
before_send=_scrub_sensitive_data,
)
from extensions import limiter, cors # noqa: E402 from extensions import limiter, cors # noqa: E402
from api import api_bp # noqa: E402 from api import api_bp # noqa: E402
@ -16,9 +42,5 @@ limiter.init_app(app)
cors.init_app(app) cors.init_app(app)
app.register_blueprint(api_bp) app.register_blueprint(api_bp)
# Sentry initialisation is a Phase 7 ship-gate item (CLAUDE.md §19 step
# 54 — "staging first, then production"). Deliberately not wired here
# yet so earlier phases never silently depend on SENTRY_DSN being set.
if __name__ == '__main__': if __name__ == '__main__':
app.run(host='0.0.0.0', port=5000, debug=False) app.run(host='0.0.0.0', port=5000, debug=False)

27
backend/core/ai_fetch.py
View File

@ -23,6 +23,20 @@ def fetch_with_ai(url):
own domain. Cost: ~$0.02/call + tokens — only fires for own domain. Cost: ~$0.02/call + tokens — only fires for
bot-protected sites, not every scrape. bot-protected sites, not every scrape.
The returned text feeds directly into build_prompt()'s PAGE CONTENT
block, which is written for plain scraped text describing one trip
— the prompt below explicitly asks for plain text (no markdown) and
a single trip, matching that shape. Discovered why this matters via
a real failure: gpt-4o-mini's *default* style for "search and
summarize" is markdown — headers, bold labels, numbered lists of
every trip it found across the domain. That got embedded into the
extraction prompt's "Return ONLY a valid JSON object, no markdown"
instruction, and the extraction model — confused about which of
several listed trips to extract, and primed by markdown-formatted
input — produced unparseable output. This is a pre-existing
condition (ported unchanged from the legacy app.py, which has the
identical prompt), not something the modular refactor introduced.
Data flow: Data flow:
url → domain extracted → OpenRouter chat completion with url → domain extracted → OpenRouter chat completion with
web_search tool (allowed_domains=[domain]) → web_search tool (allowed_domains=[domain]) →
@ -41,11 +55,18 @@ def fetch_with_ai(url):
'role': 'user', 'role': 'user',
'content': ( 'content': (
'I am a travel industry analyst conducting competitive research. ' 'I am a travel industry analyst conducting competitive research. '
'Please search for this tour page and extract the following factual data points: ' 'Search specifically for the single tour page at this exact URL — '
'not other tours or pages on this website.\n\n'
f'URL: {url}\n\n'
'Extract only the factual data points for THIS ONE tour: '
'trip name, trip code, price, duration in days, maximum group size, ' 'trip name, trip code, price, duration in days, maximum group size, '
'total included meals, hotel names, departure dates, start location, ' 'total included meals, hotel names, departure dates, start location, '
'end location, service level, and key activities. ' 'end location, service level, and key activities.\n\n'
f'Return all factual information you find.\n\nURL: {url}' 'Respond in plain text only — no markdown, no headers, no bold text, '
'no bullet points, no numbered lists. Write each fact as a short '
'"label: value" line, one per line, the way you would plainly summarise '
'a single web page. If a fact is not found, write "not found" for that '
'field rather than omitting it.'
) )
}], }],
tools=[{ tools=[{

107
backend/core/extractor.py
View File

@ -1,6 +1,7 @@
import os import os
import re import re
import json import json
import time
import logging import logging
from openai import OpenAI from openai import OpenAI
import sentry_sdk import sentry_sdk
@ -11,6 +12,15 @@ OPENROUTER_API_KEY = os.getenv('OPENROUTER_API_KEY')
FREE_MODELS = [m.strip() for m in os.getenv('OPENROUTER_FREE_MODELS', '').split(',') if m.strip()] FREE_MODELS = [m.strip() for m in os.getenv('OPENROUTER_FREE_MODELS', '').split(',') if m.strip()]
FALLBACK_MODEL = os.getenv('OPENROUTER_FALLBACK_MODEL', 'anthropic/claude-haiku-4-5') FALLBACK_MODEL = os.getenv('OPENROUTER_FALLBACK_MODEL', 'anthropic/claude-haiku-4-5')
# MVP launch toggle — set OPENROUTER_USE_FREE_MODELS_FIRST=false to skip
# FREE_MODELS entirely and call FALLBACK_MODEL directly. Added after a
# real test job took 5m47s because OpenRouter's free tier was rotating
# through rate-limited/dead models per extraction call, which blew past
# the frontend's polling timeout. Flip back to "true" (or unset — that's
# the default) once free-tier reliability is worth the latency again;
# FREE_MODELS itself is left untouched so switching back is a one-line change.
USE_FREE_MODELS_FIRST = os.getenv('OPENROUTER_USE_FREE_MODELS_FIRST', 'true').lower() == 'true'
class ExtractorError(Exception): class ExtractorError(Exception):
"""Raised when every OpenRouter model (free tier + paid fallback) has failed.""" """Raised when every OpenRouter model (free tier + paid fallback) has failed."""
@ -22,6 +32,18 @@ class RateLimitError(Exception):
pass pass
class CreditExhaustedError(Exception):
"""
Raised internally when a free model's own credit/quota is used up
for the period (403 + "credit" in the error body) — distinct from
RateLimitError: a 429 is transient and worth a short backoff+retry
on the same model, but an exhausted free-tier quota won't recover
within this request, so the caller should move to the next model
immediately instead of wasting a retry.
"""
pass
def _parse_json_response(text, model, label): def _parse_json_response(text, model, label):
""" """
Parses a model's raw text response into JSON via progressively more Parses a model's raw text response into JSON via progressively more
@ -56,6 +78,12 @@ def _parse_json_response(text, model, label):
fixed = re.sub(r'([{,])\s*([a-zA-Z0-9_]+)\s*:', r'\1"\2":', fixed) fixed = re.sub(r'([{,])\s*([a-zA-Z0-9_]+)\s*:', r'\1"\2":', fixed)
return json.loads(fixed) return json.loads(fixed)
except json.JSONDecodeError: except json.JSONDecodeError:
# All three repair passes failed — log what the model actually
# returned so a real production failure is diagnosable from
# worker logs alone. Without this, "Could not parse response"
# gives no way to tell truncation, prose-wrapped JSON, and
# genuinely malformed output apart after the fact.
logger.warning(f'{model} unparseable response for {label} (first 1000 chars): {cleaned[:1000]!r}')
raise ValueError(f'Could not parse response from {model} for {label}') raise ValueError(f'Could not parse response from {model} for {label}')
@ -69,11 +97,29 @@ def _call_openrouter(model, prompt, content):
which is harmless (the model just sees the page text once via the which is harmless (the model just sees the page text once via the
prompt and, redundantly, once more here). prompt and, redundantly, once more here).
response_format={'type': 'json_object'} constrains the model to
return valid JSON at the API level, rather than relying purely on
the prompt's "Return ONLY a valid JSON object" instruction. Added
after a real production failure: deepseek/deepseek-v4-flash would
write a full prose comparative analysis (markdown headers, tables,
"Key Differentiators" sections) instead of the requested JSON
whenever the competitor's page content didn't clearly match the
requested trip — the model prioritized being "helpful" about
explaining the mismatch over following the format instruction. This
wasn't reproducible with a short synthetic prompt, only the full
production prompt (many instructions, long page content) — so this
is a systemic, defensive fix for the whole class of "model deviates
into prose" failures, not just the one reproduced case. Confirmed
compatible with every model currently configured (OPENROUTER_FREE_MODELS
+ OPENROUTER_FALLBACK_MODEL) — a model that rejected the parameter
would 400 immediately, not the 429s seen during compatibility testing.
Data flow: Data flow:
model + prompt + content → OpenAI client (OpenRouter base_url) → model + prompt + content → OpenAI client (OpenRouter base_url) →
chat.completions.create() → raise RateLimitError on 429/rate-limit chat.completions.create() → raise RateLimitError on 429/rate-limit
responses (caller moves to the next model) → raw text → responses, CreditExhaustedError on a 403 quota-exhausted response
_parse_json_response() → parsed dict returned (caller reacts differently to each — see extract_with_openrouter) →
raw text → _parse_json_response() → parsed dict returned
""" """
client = OpenAI(api_key=OPENROUTER_API_KEY, base_url='https://openrouter.ai/api/v1') client = OpenAI(api_key=OPENROUTER_API_KEY, base_url='https://openrouter.ai/api/v1')
message = f'{prompt}\n\n{content}' if content else prompt message = f'{prompt}\n\n{content}' if content else prompt
@ -83,10 +129,13 @@ def _call_openrouter(model, prompt, content):
model=model, model=model,
messages=[{'role': 'user', 'content': message}], messages=[{'role': 'user', 'content': message}],
temperature=0.1, temperature=0.1,
max_tokens=4096 max_tokens=4096,
response_format={'type': 'json_object'}
) )
except Exception as e: except Exception as e:
error_str = str(e) error_str = str(e)
if '403' in error_str and 'credit' in error_str.lower():
raise CreditExhaustedError(error_str)
if '429' in error_str or 'rate_limit' in error_str.lower(): if '429' in error_str or 'rate_limit' in error_str.lower():
raise RateLimitError(error_str) raise RateLimitError(error_str)
raise raise
@ -95,35 +144,59 @@ def _call_openrouter(model, prompt, content):
return _parse_json_response(text, model, 'extraction') return _parse_json_response(text, model, 'extraction')
RETRY_ATTEMPTS_PER_MODEL = 2 # ported from the legacy app.py's extract_with_groq()
def extract_with_openrouter(prompt, content=''): def extract_with_openrouter(prompt, content=''):
""" """
Cycles through free OpenRouter models in sequence. Falls back to Cycles through free OpenRouter models in sequence, retrying each
the paid fallback model if all free models fail or rate-limit. one up to RETRY_ATTEMPTS_PER_MODEL times with backoff before moving
Never returns None — raises ExtractorError only after every model to the next — ported from the legacy app.py's extract_with_groq(),
(free + paid) has failed. which the Phase 2 refactor had dropped (it moved to the next model
on the very first error of any kind, including a transient 429,
burning through the whole free-tier list instantly on any rate-limit
burst). Falls back to the paid fallback model if every free model
is exhausted. Never returns None — raises ExtractorError only after
every model (free + paid) has failed.
When USE_FREE_MODELS_FIRST is false, FREE_MODELS is skipped
entirely and FALLBACK_MODEL is called directly — see that
constant's comment for why.
Data flow: Data flow:
prompt + content → for each model in FREE_MODELS + [FALLBACK_MODEL]: prompt + content → for each model in FREE_MODELS + [FALLBACK_MODEL]
(or just [FALLBACK_MODEL] when USE_FREE_MODELS_FIRST is false):
up to RETRY_ATTEMPTS_PER_MODEL attempts:
_call_openrouter() → success: return parsed JSON _call_openrouter() → success: return parsed JSON
(logging + a Sentry breadcrumb if the paid fallback had to be used) → (logging + a Sentry breadcrumb if free models were tried and
RateLimitError: log, try next model → exhausted before the paid fallback succeeded) →
any other error: log, try next model → RateLimitError: sleep 5*(attempt+1)s, retry same model →
CreditExhaustedError: log, move to next model immediately
(quota won't recover within this request — retrying wastes time) →
any other error: retry same model up to the attempt limit,
then move to next model →
all models exhausted → raise ExtractorError all models exhausted → raise ExtractorError
""" """
models_to_try = FREE_MODELS + [FALLBACK_MODEL] models_to_try = FREE_MODELS + [FALLBACK_MODEL] if USE_FREE_MODELS_FIRST else [FALLBACK_MODEL]
for model in models_to_try: for model in models_to_try:
for attempt in range(RETRY_ATTEMPTS_PER_MODEL):
try: try:
result = _call_openrouter(model, prompt, content) result = _call_openrouter(model, prompt, content)
if model == FALLBACK_MODEL and FREE_MODELS: if USE_FREE_MODELS_FIRST and model == FALLBACK_MODEL and FREE_MODELS:
logger.warning(f'Free models exhausted — used paid fallback: {model}') logger.warning(f'Free models exhausted — used paid fallback: {model}')
sentry_sdk.capture_message('OpenRouter paid fallback used', level='warning') sentry_sdk.capture_message('OpenRouter paid fallback used', level='warning')
return result return result
except RateLimitError: except RateLimitError:
logger.warning(f'Rate limit on {model} — trying next') wait = 5 * (attempt + 1)
continue logger.warning(f'{model} rate limited — waiting {wait}s (attempt {attempt + 1}/{RETRY_ATTEMPTS_PER_MODEL})...')
time.sleep(wait)
except CreditExhaustedError:
logger.warning(f'{model} free tier exhausted — trying next model...')
break
except Exception as e: except Exception as e:
logger.error(f'Model {model} failed: {str(e)}') logger.warning(f'{model} failed: {str(e)[:200]}')
continue if attempt == RETRY_ATTEMPTS_PER_MODEL - 1:
logger.warning(f'Moving to next model after {model} failed {RETRY_ATTEMPTS_PER_MODEL} times')
raise ExtractorError('All OpenRouter models exhausted including paid fallback') raise ExtractorError('All OpenRouter models exhausted including paid fallback')

169
backend/core/proxy_service.py
View File

@ -1,12 +1,37 @@
import os import os
import time
import random
import logging
from datetime import datetime, timezone from datetime import datetime, timezone
import requests
from repositories.proxy_repository import proxy_repository from repositories.proxy_repository import proxy_repository
logger = logging.getLogger(__name__)
WEBSHARE_USER = os.getenv('WEBSHARE_USER') WEBSHARE_USER = os.getenv('WEBSHARE_USER')
WEBSHARE_PASS = os.getenv('WEBSHARE_PASS') WEBSHARE_PASS = os.getenv('WEBSHARE_PASS')
# Separate from WEBSHARE_USER/WEBSHARE_PASS (proxy-gateway auth) — this is a
# Webshare *dashboard API token* (Authorization: Token <key>) for their
# account-management REST API, used only by _fetch_webshare_ip_list() to
# list the account's own dedicated IPs. Generated from the Webshare
# dashboard's API/account section, not the proxy connection settings page.
WEBSHARE_API_KEY = os.getenv('WEBSHARE_API_KEY')
BRIGHTDATA_USER = os.getenv('BRIGHTDATA_USER') BRIGHTDATA_USER = os.getenv('BRIGHTDATA_USER')
BRIGHTDATA_PASS = os.getenv('BRIGHTDATA_PASS') BRIGHTDATA_PASS = os.getenv('BRIGHTDATA_PASS')
WEBSHARE_PROXY_LIST_URL = 'https://proxy.webshare.io/api/v2/proxy/list/'
# Bounds latency (each blocked attempt costs a full Playwright render cycle,
# up to _render_with_proxy()'s own 30-60s timeouts) while still being enough
# to distinguish "one flagged IP" (fixed by 2-3 alternates from the same
# account) from "this site blocks the whole IP range" — no number of
# same-account IPs fixes that; only Bright Data's different network does.
WEBSHARE_IP_CANDIDATE_LIMIT = 3
WEBSHARE_IP_CACHE_TTL_SECONDS = 1800 # 30 min — the account's dedicated IPs rarely change
# Short on purpose: bounds a retry storm within one batch run (many domains
# targeting the same country back-to-back) without permanently caching what
# might be a transient failure (bad token, momentary API outage).
WEBSHARE_IP_CACHE_NEGATIVE_TTL_SECONDS = 300 # 5 min
class ProxyService: class ProxyService:
""" """
@ -31,8 +56,23 @@ class ProxyService:
Data flow (subsequent requests to a known blocked domain): Data flow (subsequent requests to a known blocked domain):
domain + country → proxy_overrides lookup → brightdata override domain + country → proxy_overrides lookup → brightdata override
found → Bright Data config returned immediately, no Webshare attempt found → Bright Data config returned immediately, no Webshare attempt
Data flow (specific-IP candidate rotation — before ever escalating to
Bright Data): domain not already known-blocked → the account's own
dedicated Webshare IPs for this country fetched (cached) via
get_webshare_ip_candidates() → core.scraper.render_page() tries a
handful of those specific IPs in turn → only once all of them are
exhausted (or none were available) does it fall through to the single
`-rotate` gateway attempt and then Bright Data, as before.
""" """
def __init__(self):
# In-memory cache of the account's dedicated Webshare IPs, keyed by
# uppercased country code — {'expires_at': float, 'results': [...]}.
# Same time.time()-based pattern as repositories/pocketbase_client.py's
# token cache. Not persisted anywhere — a fresh process just refetches.
self._ip_cache = {}
def get_proxy_for_domain(self, domain, country=None): def get_proxy_for_domain(self, domain, country=None):
"""Returns a `requests`-style proxy config, routed to Bright Data if previously blocked.""" """Returns a `requests`-style proxy config, routed to Bright Data if previously blocked."""
override = proxy_repository.get_by_domain(domain) override = proxy_repository.get_by_domain(domain)
@ -47,6 +87,106 @@ class ProxyService:
return self._brightdata_playwright() return self._brightdata_playwright()
return self._webshare_playwright(country=country) return self._webshare_playwright(country=country)
def is_domain_blocked_on_webshare(self, domain):
"""
Returns True if this domain already has a `brightdata` override —
i.e. Webshare (gateway or specific IPs) is already known not to
work here. Lets core.scraper.render_page() skip the specific-IP
candidate loop entirely for a domain that's already established as
Bright-Data-only, rather than re-running a 3-IP check every single
scrape once that's settled. Thin wrapper so scraper.py never reaches
into proxy_repository directly — this class stays the only place
that knows about proxy providers.
"""
override = proxy_repository.get_by_domain(domain)
return bool(override and override.get('provider') == 'brightdata')
def get_webshare_ip_candidates(self, country, limit=WEBSHARE_IP_CANDIDATE_LIMIT):
"""
Returns up to `limit` Playwright proxy configs for specific
dedicated Webshare IPs assigned to this account, filtered to the
target country. Returns [] if none are available (no country given,
API failure, or no results) — core.scraper.render_page() treats an
empty list as "skip straight to the existing single-gateway attempt",
so this always fails gracefully rather than raising.
Each candidate carries its OWN username/password/port from
Webshare's proxy-list API response — genuinely different from
_webshare_playwright()'s single account-level rotating-gateway
credentials, not a variant of it.
"""
if not country:
return []
results = self._fetch_webshare_ip_list(country)
if not results:
return []
chosen = random.sample(results, min(limit, len(results)))
return [
{
'server': f'http://{ip["proxy_address"]}:{ip["port"]}',
'username': ip['username'],
'password': ip['password'],
}
for ip in chosen
]
def _fetch_webshare_ip_list(self, country):
"""
Fetches (and caches) the account's own dedicated Webshare IPs for a
country via Webshare's proxy-list *management* API — a different
host/credential entirely from the proxy gateway (p.webshare.io +
WEBSHARE_USER/PASS). This one is proxy.webshare.io, authenticated
with a dashboard API token (WEBSHARE_API_KEY), and it wants the
country code UPPERCASE (Webshare's own docs example: "FR,US") — the
opposite case convention from _webshare_playwright()'s gateway
username suffix, which wants lowercase. These are two unrelated
Webshare systems with inconsistent casing on their end, not a
mistake here — do not "fix" them into matching each other.
Data flow:
country → cache hit (within TTL, success or negative) → cached
list returned →
cache miss → GET proxy.webshare.io/api/v2/proxy/list/ (mode=direct,
country_code__in=<COUNTRY>, page_size=100), paginated via `next` →
filter to valid=true → cache with the long TTL → list returned →
on any request/HTTP/JSON failure → logged, cached as [] with the
short negative TTL → [] returned
"""
cache_key = country.upper()
cached = self._ip_cache.get(cache_key)
if cached and time.time() < cached['expires_at']:
return cached['results']
results = []
try:
url = WEBSHARE_PROXY_LIST_URL
params = {'mode': 'direct', 'country_code__in': cache_key, 'page_size': 100}
for _ in range(3): # defensive pagination cap — a ~100-IP account should fit in one page
response = requests.get(
url, headers={'Authorization': f'Token {WEBSHARE_API_KEY}'},
params=params, timeout=10
)
response.raise_for_status()
data = response.json()
results.extend(r for r in data.get('results', []) if r.get('valid'))
url = data.get('next')
params = None # `next` already carries its own query string
if not url:
break
self._ip_cache[cache_key] = {
'expires_at': time.time() + WEBSHARE_IP_CACHE_TTL_SECONDS,
'results': results,
}
except (requests.RequestException, ValueError) as e:
logger.warning(f'Webshare proxy list fetch failed for {country}: {str(e)[:200]}')
self._ip_cache[cache_key] = {
'expires_at': time.time() + WEBSHARE_IP_CACHE_NEGATIVE_TTL_SECONDS,
'results': [],
}
return []
return results
def escalate_to_brightdata(self, domain): def escalate_to_brightdata(self, domain):
""" """
Records that Webshare was blocked on this domain. Upserts a Records that Webshare was blocked on this domain. Upserts a
@ -69,8 +209,17 @@ class ProxyService:
}) })
def _webshare_requests(self, country=None): def _webshare_requests(self, country=None):
"""Webshare `requests` proxy config. Format: username-{COUNTRY}-rotate. No country = full pool rotation.""" """
suffix = f'-{country.upper()}-rotate' if country else '-rotate' Webshare `requests` proxy config. Format: username-{country}-rotate.
Country code must be lowercase — Webshare's own rotating-endpoint
docs (help.webshare.io/en/articles/8596715) show lowercase examples
only (`-us`, `-gb`, `-ca`); an uppercase code is not a recognised
country token and fails proxy auth. `country` arrives from
competitors.primary_market, which is stored uppercase (CLAUDE.md's
schema — "GB", "US", "AU"), so this always lowercases regardless of
the input's case. No country = full pool rotation.
"""
suffix = f'-{country.lower()}-rotate' if country else '-rotate'
username = f'{WEBSHARE_USER}{suffix}' username = f'{WEBSHARE_USER}{suffix}'
return { return {
'http': f'http://{username}:{WEBSHARE_PASS}@p.webshare.io:80', 'http': f'http://{username}:{WEBSHARE_PASS}@p.webshare.io:80',
@ -78,8 +227,8 @@ class ProxyService:
} }
def _webshare_playwright(self, country=None): def _webshare_playwright(self, country=None):
"""Webshare Playwright proxy config with optional country targeting.""" """Webshare Playwright proxy config with optional country targeting — see _webshare_requests() for why the country code is lowercased."""
suffix = f'-{country.upper()}-rotate' if country else '-rotate' suffix = f'-{country.lower()}-rotate' if country else '-rotate'
return { return {
'server': 'http://p.webshare.io:80', 'server': 'http://p.webshare.io:80',
'username': f'{WEBSHARE_USER}{suffix}', 'username': f'{WEBSHARE_USER}{suffix}',
@ -87,14 +236,20 @@ class ProxyService:
} }
def _brightdata_requests(self): def _brightdata_requests(self):
# Port 33335, not the commonly-documented 22225 — Bright Data's
# currently valid SSL certificate requires 33335; 22225 is tied to
# their deprecated cert (phased out ahead of its 2026 expiry), which
# is why a live scrape hit net::ERR_CERT_AUTHORITY_INVALID on this
# host before this fix.
return { return {
'http': f'http://{BRIGHTDATA_USER}:{BRIGHTDATA_PASS}@brd.superproxy.io:22225', 'http': f'http://{BRIGHTDATA_USER}:{BRIGHTDATA_PASS}@brd.superproxy.io:33335',
'https': f'http://{BRIGHTDATA_USER}:{BRIGHTDATA_PASS}@brd.superproxy.io:22225', 'https': f'http://{BRIGHTDATA_USER}:{BRIGHTDATA_PASS}@brd.superproxy.io:33335',
} }
def _brightdata_playwright(self): def _brightdata_playwright(self):
# See _brightdata_requests() — port 33335 required for the current cert.
return { return {
'server': 'http://brd.superproxy.io:22225', 'server': 'http://brd.superproxy.io:33335',
'username': BRIGHTDATA_USER, 'username': BRIGHTDATA_USER,
'password': BRIGHTDATA_PASS, 'password': BRIGHTDATA_PASS,
} }

73
backend/core/scraper.py
View File

@ -39,7 +39,15 @@ def check_robots_txt(url):
Checks if the given URL is allowed to be scraped per robots.txt. Checks if the given URL is allowed to be scraped per robots.txt.
Returns True if scraping is permitted, False if disallowed. Returns True if scraping is permitted, False if disallowed.
Defaults to True if robots.txt cannot be fetched (permissive Defaults to True if robots.txt cannot be fetched (permissive
default) — CLAUDE.md §30 Web Scraping Legal Policy. default).
Advisory only as of this call site: scrape() logs a disallow but
does not block on it — a deliberate policy change from CLAUDE.md
§30's original "check robots.txt, skip if disallowed" position.
That section still describes the skip-on-disallow behavior; it
hasn't been updated to match. This function's result is still
computed correctly and still worth calling — only the caller's
reaction to a False result changed.
Data flow: Data flow:
url → extract root domain → fetch /robots.txt → url → extract root domain → fetch /robots.txt →
@ -141,6 +149,14 @@ async def _render_with_proxy(url, proxy):
viewport={'width': 1366, 'height': 768}, viewport={'width': 1366, 'height': 768},
locale='en-GB', locale='en-GB',
timezone_id='Europe/London', timezone_id='Europe/London',
# Every call in this function is routed through a commercial
# proxy (Webshare or Bright Data) — there's no direct/unproxied
# path here to protect by being stricter. Real competitor sites
# occasionally have their own cert misconfigurations independent
# of any proxy (same spirit as _is_blocked()'s tolerance for
# other imperfect real-world responses); this lets the scraper
# get past those rather than failing outright.
ignore_https_errors=True,
extra_http_headers={ extra_http_headers={
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
'Accept-Language': 'en-GB,en;q=0.9', 'Accept-Language': 'en-GB,en;q=0.9',
@ -232,7 +248,7 @@ async def _render_with_proxy(url, proxy):
char_count = len(text) char_count = len(text)
if char_count < 500: if char_count < 500:
logger.warning(f'Low char count ({char_count}) for {url} — will attempt AI web fetch fallback') logger.warning(f'Low char count ({char_count}) for {url} — below the AI web fetch fallback threshold')
return { return {
'text': text[:25000], 'text': text[:25000],
@ -256,13 +272,34 @@ async def render_page(url, country=None):
PocketBase. Detects blocks and auto-escalates to Bright Data if PocketBase. Detects blocks and auto-escalates to Bright Data if
Webshare fails. Webshare fails.
Before ever reaching the single `-rotate` gateway attempt, tries a
handful of the account's own dedicated Webshare IPs for this country
(proxy_service.get_webshare_ip_candidates()) — gives IP diversity a
chance to resolve a one-off block without paying for a Bright Data
request. Skipped entirely for a domain already known (via
proxy_overrides) to block the whole Webshare account, and gracefully
falls through to the existing gateway/Bright Data flow when no
candidates are available (no country given, Webshare API unreachable,
or every candidate still came back blocked).
Data flow: Data flow:
url + country → domain extracted → url + country → domain extracted →
is_domain_blocked_on_webshare(domain)? → no →
get_webshare_ip_candidates(country) → try each in turn via
_render_with_proxy(), return on first success →
all candidates exhausted or none available →
proxy_service.get_playwright_proxy_for_domain(domain, country) → proxy_service.get_playwright_proxy_for_domain(domain, country) →
_render_with_proxy() → if blocked → escalate_to_brightdata(domain) → _render_with_proxy() → if blocked → escalate_to_brightdata(domain) →
retry once with Bright Data (no country targeting at fallback level) retry once with Bright Data (no country targeting at fallback level)
""" """
domain = extract_domain(url) domain = extract_domain(url)
if not proxy_service.is_domain_blocked_on_webshare(domain):
for candidate_proxy in proxy_service.get_webshare_ip_candidates(country):
result = await _render_with_proxy(url, candidate_proxy)
if not _is_blocked(result):
return result
proxy = proxy_service.get_playwright_proxy_for_domain(domain, country) proxy = proxy_service.get_playwright_proxy_for_domain(domain, country)
result = await _render_with_proxy(url, proxy) result = await _render_with_proxy(url, proxy)
@ -276,16 +313,17 @@ async def render_page(url, country=None):
def scrape(url, country=None): def scrape(url, country=None):
""" """
Synchronous entry point for scraping one competitor page. Checks Synchronous entry point for scraping one competitor page. Logs a
robots.txt first (legal/compliance gate per §30), then renders via robots.txt disallow (does not block on it — see check_robots_txt's
Playwright, falling back to the AI web fetch path if Playwright's docstring for the policy this reflects), renders via Playwright,
char count is too low (bot-protected sites). then falls back to the AI web fetch path whenever Playwright came
back thin OR failed outright.
Data flow: Data flow:
url + country → clean_url() → check_robots_txt() → url + country → clean_url() → check_robots_txt() logged only →
disallowed: return failure immediately (no scrape attempted) → render_page() (sync wrapper via nest_asyncio) →
allowed: render_page() (sync wrapper via nest_asyncio) → char_count < 1000 (whether success=True-but-thin or an outright
char_count < 1000: fetch_with_ai() fallback → failure): fetch_with_ai() fallback →
final { text, tables, url, char_count, success } returned final { text, tables, url, char_count, success } returned
""" """
import nest_asyncio import nest_asyncio
@ -294,11 +332,7 @@ def scrape(url, country=None):
url = clean_url(url) url = clean_url(url)
if not check_robots_txt(url): if not check_robots_txt(url):
logger.warning(f'robots.txt disallows scraping {url} — skipping') logger.warning(f'robots.txt disallows scraping {url} — proceeding anyway per current policy')
return {
'text': 'Scraping disallowed by robots.txt for this URL',
'tables': '', 'url': url, 'char_count': 0, 'success': False,
}
loop = asyncio.new_event_loop() loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop) asyncio.set_event_loop(loop)
@ -307,8 +341,13 @@ def scrape(url, country=None):
finally: finally:
loop.close() loop.close()
if result.get('char_count', 0) < 1000 and result.get('success'): # Not gated on result['success'] — an outright Playwright failure
logger.info(f'Playwright char count low — trying AI web fetch for {url}') # (proxy error, cert error, timeout) is exactly the case the AI
# fallback exists for, same as a thin-but-technically-successful
# render. Previously this required success=True, which meant a
# hard render failure skipped the fallback entirely.
if result.get('char_count', 0) < 1000:
logger.info(f'Playwright content insufficient (char_count={result.get("char_count", 0)}) — trying AI web fetch for {url}')
from core.ai_fetch import fetch_with_ai from core.ai_fetch import fetch_with_ai
ai_result = fetch_with_ai(url) ai_result = fetch_with_ai(url)
if ai_result: if ai_result:

17
backend/extensions.py
View File

@ -1,4 +1,5 @@
import os import os
from flask import request
from flask_cors import CORS from flask_cors import CORS
from flask_limiter import Limiter from flask_limiter import Limiter
from flask_limiter.util import get_remote_address from flask_limiter.util import get_remote_address
@ -14,6 +15,22 @@ limiter = Limiter(
default_limits=['200 per hour', '30 per minute'], default_limits=['200 per hour', '30 per minute'],
) )
@limiter.request_filter
def _exempt_cors_preflight():
"""
Browser-generated CORS preflight (OPTIONS) requests must never count
against rate limits — discovered when the dashboard's 3s status-poll
loop (analysis_store.js) tripped a 429 on the OPTIONS preflight
within under a minute: one real GET every 3s is only 20/min, well
under the 30/min default, but each cross-origin fetch also fires an
OPTIONS preflight first, doubling the counted requests to 40/min.
The browser reports a failed preflight as an opaque network error
with no response body, which is why the frontend showed the generic
"Lost connection" fallback rather than a 429 with a real message.
"""
return request.method == 'OPTIONS'
# CORS was absent from CLAUDE.md entirely, but the Vue dashboard # CORS was absent from CLAUDE.md entirely, but the Vue dashboard
# (api_service.js) calls the Flask API directly from the browser via # (api_service.js) calls the Flask API directly from the browser via
# Axios — that's a cross-origin request the moment the dashboard and # Axios — that's a cross-origin request the moment the dashboard and

23
backend/industries/adventure_travel/prompt.py
View File

@ -5,17 +5,26 @@
def build_prompt(competitor_name, page_data, product_name, def build_prompt(competitor_name, page_data, product_name,
destination, duration, travel_style, is_ngs=False): destination, duration, travel_style, company_name=None, is_ngs=False):
""" """
Builds the full extraction prompt for one competitor page, embedding Builds the full extraction prompt for one competitor page, embedding
the scraped page content directly into the returned template. the scraped page content directly into the returned template.
company_name is the tenant's own company (jobs.py's context['company_name'],
fetched from the tenant record) — this was hardcoded to "G Adventures"
(the hackathon origin client, CLAUDE.md's "Primary reference client")
until this fix, meaning every other tenant's analysis prompt was
factually wrong about whose product it was comparing against. Defaults
to a generic placeholder only for the dry-run /preview-prompt route,
which has no tenant context to draw a real name from.
Data flow: Data flow:
competitor_name + page_data (scraped text/tables/url) + client competitor_name + page_data (scraped text/tables/url) + client
product context (name/destination/duration/style) + is_ngs flag → product context (name/destination/duration/style/company) + is_ngs
NGS-specific field block appended if is_ngs → flag → NGS-specific field block appended if is_ngs →
full prompt string returned, ready to send to core.extractor full prompt string returned, ready to send to core.extractor
""" """
company_name = company_name or 'the client'
ngs_fields = ''' ngs_fields = '''
"exclusiveAccess": "exclusive access or special unique inclusions", "exclusiveAccess": "exclusive access or special unique inclusions",
"groupLeader": "group leader and/or local expert description", "groupLeader": "group leader and/or local expert description",
@ -27,7 +36,7 @@ def build_prompt(competitor_name, page_data, product_name,
url = page_data['url'] if page_data else '[COMPETITOR URL]' url = page_data['url'] if page_data else '[COMPETITOR URL]'
return f'''You are a travel industry analyst helping G Adventures research competitors. return f'''You are a travel industry analyst helping {company_name} research competitors.
Be concise — use short phrases not full sentences for all fields except comments. Be concise — use short phrases not full sentences for all fields except comments.
Read the ENTIRE page content carefully before extracting any fields. Read the ENTIRE page content carefully before extracting any fields.
Pay particular attention to itinerary sections, departure tables, and pricing grids Pay particular attention to itinerary sections, departure tables, and pricing grids
@ -36,10 +45,10 @@ which may appear later in the content.
The content below was scraped from the {competitor_name} tour page at: {url} The content below was scraped from the {competitor_name} tour page at: {url}
This is the specific page selected as most comparable to: This is the specific page selected as most comparable to:
- G Adventures Product: {product_name} - {company_name} Product: {product_name}
- Destination: {destination} - Destination: {destination}
- Duration: ~{duration} days - Duration: ~{duration} days
- G Adventures Travel Style: {travel_style} - {company_name} Travel Style: {travel_style}
PAGE CONTENT: PAGE CONTENT:
--- ---
@ -89,7 +98,7 @@ Return ONLY a valid JSON object. No markdown, no code fences, no explanation.
"majorityPrice": standard adult USD price as number or null, "majorityPrice": standard adult USD price as number or null,
"priceLow": lowest standard USD price as number or null, "priceLow": lowest standard USD price as number or null,
"priceHigh": highest standard USD price as number or null, "priceHigh": highest standard USD price as number or null,
"comments": "analysis: key differences from G Adventures, strengths, weaknesses, pricing position", "comments": "analysis: key differences from {company_name}, strengths, weaknesses, pricing position",
"dateUsed": "specific departure date used for majority price", "dateUsed": "specific departure date used for majority price",
"audPrice": standard AUD price as number or null, "audPrice": standard AUD price as number or null,
"cadPrice": standard CAD price as number or null, "cadPrice": standard CAD price as number or null,

103
backend/jobs.py
View File

@ -5,9 +5,12 @@ from datetime import datetime, timezone
from redis import Redis from redis import Redis
from rq import Queue from rq import Queue
from rq.job import Job
from rq.exceptions import NoSuchJobError
from repositories.scrape_repository import scrape_repository from repositories.scrape_repository import scrape_repository
from repositories.competitor_repository import competitor_repository from repositories.competitor_repository import competitor_repository
from repositories.tenant_repository import tenant_repository
from services import analysis_service, alert_service, battlecard_service from services import analysis_service, alert_service, battlecard_service
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@ -42,50 +45,131 @@ def enqueue_research_job(job_id, data):
return job_id return job_id
def cancel_research_job(job_id):
"""
PM-initiated cancellation — distinct from a job merely surviving a
frontend disconnect (CLAUDE.md §18/§21: run_research_job() runs to
completion regardless of dashboard polling state, by design). This
is the opposite: the PM explicitly asked the job to stop.
RQ's own Job.cancel() only removes a job still sitting in the
queue — it has no way to interrupt code already executing inside a
worker process. So this handles both cases:
Data flow:
job_id → fetch the RQ Job record →
if still queued (never started): job.cancel() removes it from
the queue outright, and scrape_runs.status is set to
'cancelled' directly since run_research_job() will now never
run at all →
if already running (or RQ has no record — job finished, or was
never an RQ job, e.g. cancelled twice): set
scrape_runs.cancel_requested = True regardless — a running
job checks this flag between competitors and self-terminates
with status='cancelled' at the next checkpoint (see
run_research_job()'s loop) →
returns True if the scrape_runs record exists, False if job_id
is unknown entirely
"""
run = scrape_repository.get_by_id(job_id)
if not run:
return False
try:
job = Job.fetch(job_id, connection=redis_conn)
if job.get_status() in ('queued', 'deferred', 'scheduled'):
job.cancel()
scrape_repository.update(job_id, {'status': 'cancelled', 'cancel_requested': True})
return True
except NoSuchJobError:
pass
# Already running (or RQ has no record of it any more) — set the
# cooperative flag so an in-flight run_research_job() stops itself
# at its next between-competitor checkpoint.
scrape_repository.update(job_id, {'cancel_requested': True})
return True
def run_research_job(job_id, data): def run_research_job(job_id, data):
""" """
The RQ job function. Loops over every requested competitor, running The RQ job function. Loops over every requested competitor, running
the fast/refresh path decision per CLAUDE.md §7. One competitor's the fast/refresh path decision per CLAUDE.md §7. One competitor's
failure is logged and skipped — it never aborts the rest of the failure is logged and skipped — it never aborts the rest of the
batch (§18 resilience rule). Records duration and fires Gotify batch (§18 resilience rule). Records duration and fires Gotify
alerts on slow/stuck/failed completion (§29 Component 2). alerts on slow/stuck/failed completion (§29 Component 2). Checks
for PM-initiated cancellation (cancel_research_job()) once per
competitor and stops early with status='cancelled' if requested.
Data flow: Data flow:
job_id + { tenant_id, competitors, destination, duration, job_id + { tenant_id, competitors, destination, duration,
travel_style, tab_type } → travel_style, tab_type } →
scrape_runs status → running → scrape_runs status → running →
per competitor: analysis_service.run_competitor_analysis() → per competitor: check cancel_requested → stop early if set →
analysis_service.run_competitor_analysis() →
success: append to results, competitors_done += 1 → success: append to results, competitors_done += 1 →
failure: append to results.failed, continue → failure: append to results.failed, continue →
scrape_runs status → complete (or failed if every competitor failed) → scrape_runs status → complete / failed / cancelled →
duration computed → scrape_runs.duration_seconds updated → duration computed → scrape_runs.duration_seconds updated →
slow/stuck duration → Gotify alert fired slow/stuck duration → Gotify alert fired (skipped if cancelled)
""" """
start = time.time() start = time.time()
scrape_repository.update(job_id, {'status': 'running'}) scrape_repository.update(job_id, {'status': 'running'})
tenant_id = data.get('tenant_id') tenant_id = data.get('tenant_id')
competitors_input = data.get('competitors', []) competitors_input = data.get('competitors', [])
tenant = tenant_repository.get_by_id(tenant_id)
context = { context = {
'destination': data.get('destination'), 'destination': data.get('destination'),
'duration': data.get('duration'), 'duration': data.get('duration'),
'travel_style': data.get('travel_style'), 'travel_style': data.get('travel_style'),
'tab_type': data.get('tab_type'), 'tab_type': data.get('tab_type'),
'product_name': data.get('productName') or data.get('product_name'), 'product_name': data.get('productName') or data.get('product_name'),
# The tenant's own company name — build_prompt() uses this instead
# of a hardcoded brand so the prompt (and its "differences from
# {company}" comments) is correct for whichever client is actually
# running the analysis. Fetched once per job, not per competitor,
# since it's the same tenant throughout the batch.
'company_name': (tenant or {}).get('name') or 'the client',
# PM-initiated override (TripIntentForm.vue's collapsed "Advanced
# options" toggle) — bypasses both cache-shortcut points in
# analysis_service.run_competitor_analysis() for this run only.
'force_refresh': data.get('force_refresh', False),
} }
success, failed = [], [] success, failed = [], []
done = 0 done = 0
cancelled = False
for entry in competitors_input: for entry in competitors_input:
# Cancellation checkpoint — cancel_research_job() sets this flag
# for a job already running (it can't interrupt a scrape/extraction
# already in flight, only stop the next one from starting). Checked
# once per competitor rather than mid-competitor, since that's the
# natural unit of work here (scrape + AI extraction as one step).
if scrape_repository.get_by_id(job_id).get('cancel_requested'):
cancelled = True
break
competitor = competitor_repository.get_by_id(entry['id']) if isinstance(entry, dict) else None competitor = competitor_repository.get_by_id(entry['id']) if isinstance(entry, dict) else None
if not competitor: if not competitor:
failed.append({'name': entry.get('name', 'Unknown') if isinstance(entry, dict) else str(entry), failed.append({'name': entry.get('name', 'Unknown') if isinstance(entry, dict) else str(entry),
'error': 'Competitor not found'}) 'error': 'Competitor not found'})
continue continue
# The URL confirmed at UrlConfirmation.vue (Firecrawl /map match or
# a manual swap) travels per-competitor, not in the shared context —
# each competitor in the same batch can have a different confirmed
# page. Previously only entry['id'] was ever read here, so the
# confirmed/matched URL was silently discarded and every scrape fell
# back to the competitor's stored root domain regardless of what was
# matched or swapped — see analysis_service.run_competitor_analysis()'s
# target_url resolution for the other half of this fix.
confirmed_url = (entry.get('url') or '').strip() or None if isinstance(entry, dict) else None
competitor_context = {**context, 'confirmed_url': confirmed_url}
try: try:
outcome = analysis_service.run_competitor_analysis(tenant_id, competitor, context) outcome = analysis_service.run_competitor_analysis(tenant_id, competitor, competitor_context)
success.append({'competitor_id': competitor['id'], 'name': competitor['name'], **outcome}) success.append({'competitor_id': competitor['id'], 'name': competitor['name'], **outcome})
# Battlecard regeneration fires on genuinely fresh data only — # Battlecard regeneration fires on genuinely fresh data only —
@ -112,7 +196,7 @@ def run_research_job(job_id, data):
done += 1 done += 1
scrape_repository.update(job_id, {'competitors_done': done}) scrape_repository.update(job_id, {'competitors_done': done})
status = 'complete' if success or not competitors_input else 'failed' status = 'cancelled' if cancelled else ('complete' if success or not competitors_input else 'failed')
scrape_repository.update(job_id, { scrape_repository.update(job_id, {
'status': status, 'status': status,
'results': {'success': success, 'failed': failed}, 'results': {'success': success, 'failed': failed},
@ -130,6 +214,13 @@ def run_research_job(job_id, data):
duration = round(time.time() - start) duration = round(time.time() - start)
scrape_repository.update(job_id, {'duration_seconds': duration}) scrape_repository.update(job_id, {'duration_seconds': duration})
# A cancellation is intentional, not an operational problem — skip
# the slow/stuck alerts, which exist to flag jobs that are running
# long for reasons nobody chose (proxy/Playwright issues, model
# rate-limiting, etc).
if cancelled:
return
if duration > JOB_STUCK_SECONDS: if duration > JOB_STUCK_SECONDS:
alert_service.send_gotify( alert_service.send_gotify(
title='🔴 Job likely stuck', title='🔴 Job likely stuck',

40
backend/pb_migrations/002_add_cancel_support.js Normal file
View File

@ -0,0 +1,40 @@
/// <reference path="../pb_data/types.d.ts" />
//
// Adds intentional-cancellation support to scrape_runs — distinct from
// a job merely surviving a frontend disconnect (CLAUDE.md §18/§21,
// jobs.py's run_research_job already runs to completion regardless of
// the dashboard's polling state, by design). This migration adds the
// PM-facing "stop this analysis" affordance:
//
// cancel_requested bool set true by POST /research/cancel/<job_id>,
// checked by run_research_job() between
// competitors — see jobs.py's per-competitor
// loop for why this is a between-competitor
// checkpoint rather than an interrupt of a
// scrape/extraction already in flight
// status: 'cancelled' new select option, distinct from 'failed'
// so the dashboard can show "Cancelled by
// you" rather than an error state
migrate((app) => {
const collection = app.findCollectionByNameOrId('scrape_runs');
collection.fields.add(new Field({
type: 'bool',
name: 'cancel_requested',
}));
const statusField = collection.fields.getByName('status');
statusField.values = [...statusField.values, 'cancelled'];
return app.save(collection);
}, (app) => {
const collection = app.findCollectionByNameOrId('scrape_runs');
collection.fields.removeByName('cancel_requested');
const statusField = collection.fields.getByName('status');
statusField.values = statusField.values.filter((v) => v !== 'cancelled');
return app.save(collection);
});

33
backend/pb_migrations/003_add_products_missing_fields.js Normal file
View File

@ -0,0 +1,33 @@
/// <reference path="../pb_data/types.d.ts" />
//
// Adds the two `products` fields the extraction schema already asks the LLM
// for (industries/adventure_travel/prompt.py's build_prompt() JSON schema —
// `departures`, `startDays`) but which have no PocketBase column to land in,
// and a `tab_type` field on `scrape_runs` so a run's Standard-vs-NGS row
// layout survives being restored from history later (RunHistory.vue) — it
// was previously only ever held in memory for the duration of one
// run_research_job() call and never persisted anywhere.
//
// products.departures number approximate departures per year
// products.start_days text departure days of week
// scrape_runs.tab_type text 'Standard' or 'NGS' — set at job creation
migrate((app) => {
const products = app.findCollectionByNameOrId('products');
products.fields.add(new Field({ type: 'number', name: 'departures' }));
products.fields.add(new Field({ type: 'text', name: 'start_days' }));
app.save(products);
const scrapeRuns = app.findCollectionByNameOrId('scrape_runs');
scrapeRuns.fields.add(new Field({ type: 'text', name: 'tab_type' }));
return app.save(scrapeRuns);
}, (app) => {
const products = app.findCollectionByNameOrId('products');
products.fields.removeByName('departures');
products.fields.removeByName('start_days');
app.save(products);
const scrapeRuns = app.findCollectionByNameOrId('scrape_runs');
scrapeRuns.fields.removeByName('tab_type');
return app.save(scrapeRuns);
});

38
backend/pb_migrations/004_add_scrape_runs_content_hash_fields.js Normal file
View File

@ -0,0 +1,38 @@
/// <reference path="../pb_data/types.d.ts" />
//
// Adds the content-hash change-detection fields to `scrape_runs` that
// 001_initial_schema.js documents (content_hash/content_length/
// hash_algorithm + the composite (competitor_id, content_hash, started_at)
// index) but which were never actually applied to any environment that
// had already run 001 before those fields were added to that file in
// place — PocketBase tracks migrations as applied by filename, so editing
// an already-run migration's contents is silently a no-op on any instance
// that ran it before the edit. This surfaced as every single refresh-path
// competitor failing with a 400 from PocketBase, because
// scrape_repository.get_last_hash_for_url() filters on `content_hash`,
// a column that never existed on this table. Written as its own
// migration instead of re-editing 001 for exactly that reason.
migrate((app) => {
const collection = app.findCollectionByNameOrId('scrape_runs');
collection.fields.add(new Field({ type: 'text', name: 'content_hash' }));
collection.fields.add(new Field({ type: 'number', name: 'content_length' }));
collection.fields.add(new Field({ type: 'text', name: 'hash_algorithm' }));
collection.indexes = [
...collection.indexes,
'CREATE INDEX idx_scrape_runs_competitor_hash ON scrape_runs (competitor_id, content_hash, started_at)',
];
return app.save(collection);
}, (app) => {
const collection = app.findCollectionByNameOrId('scrape_runs');
collection.fields.removeByName('content_hash');
collection.fields.removeByName('content_length');
collection.fields.removeByName('hash_algorithm');
collection.indexes = collection.indexes.filter((idx) => !idx.includes('idx_scrape_runs_competitor_hash'));
return app.save(collection);
});

19
backend/pb_migrations/005_add_scrape_runs_created_by_email.js Normal file
View File

@ -0,0 +1,19 @@
/// <reference path="../pb_data/types.d.ts" />
//
// Adds `created_by_email` to `scrape_runs` — the seat (PM) who submitted
// the analysis run, so RunHistory.vue can show who ran each entry in the
// "last 5 runs" list. Sourced from the dashboard's own auth session
// (auth_store.js's authStore.email — the same PocketBase-verified email
// already used for seat/licence validation elsewhere, CLAUDE.md §9), sent
// once at job creation (api.py's research() route) rather than looked up
// later, since a seat could in principle be removed after the run.
migrate((app) => {
const collection = app.findCollectionByNameOrId('scrape_runs');
collection.fields.add(new Field({ type: 'text', name: 'created_by_email' }));
return app.save(collection);
}, (app) => {
const collection = app.findCollectionByNameOrId('scrape_runs');
collection.fields.removeByName('created_by_email');
return app.save(collection);
});

16
backend/repositories/alert_repository.py
View File

@ -42,6 +42,13 @@ class AlertRepository:
def mark_email_delivered(self, alert_id): def mark_email_delivered(self, alert_id):
return pb_client.update(COLLECTION, alert_id, {'delivered_email': True}) return pb_client.update(COLLECTION, alert_id, {'delivered_email': True})
def list_read_older_than(self, cutoff_iso):
"""Used by the weekly data-cleanup job — CLAUDE.md §22 DATA_RETENTION (90 days, read alerts only)."""
return pb_client.list(COLLECTION, filter_str=f'read = true && created < "{cutoff_iso}"', per_page=500)
def delete(self, alert_id):
return pb_client.delete(COLLECTION, alert_id)
class MockAlertRepository: class MockAlertRepository:
"""In-memory stand-in for AlertRepository.""" """In-memory stand-in for AlertRepository."""
@ -84,5 +91,14 @@ class MockAlertRepository:
self._records[alert_id]['delivered_email'] = True self._records[alert_id]['delivered_email'] = True
return self._records[alert_id] return self._records[alert_id]
def list_read_older_than(self, cutoff_iso):
return [
r for r in self._records.values()
if r.get('read') and (r.get('created') or '') < cutoff_iso
]
def delete(self, alert_id):
return self._records.pop(alert_id, None) is not None
alert_repository = MockAlertRepository() if USE_MOCK_REPOSITORIES else AlertRepository() alert_repository = MockAlertRepository() if USE_MOCK_REPOSITORIES else AlertRepository()

6
backend/repositories/battlecard_repository.py
View File

@ -30,6 +30,9 @@ class BattlecardRepository:
return pb_client.update(COLLECTION, existing['id'], data) return pb_client.update(COLLECTION, existing['id'], data)
return pb_client.create(COLLECTION, {'competitor_id': competitor_id, **data}) return pb_client.create(COLLECTION, {'competitor_id': competitor_id, **data})
def delete(self, card_id):
return pb_client.delete(COLLECTION, card_id)
class MockBattlecardRepository: class MockBattlecardRepository:
"""In-memory stand-in for BattlecardRepository.""" """In-memory stand-in for BattlecardRepository."""
@ -53,5 +56,8 @@ class MockBattlecardRepository:
self._records[record_id] = record self._records[record_id] = record
return record return record
def delete(self, card_id):
return self._records.pop(card_id, None) is not None
battlecard_repository = MockBattlecardRepository() if USE_MOCK_REPOSITORIES else BattlecardRepository() battlecard_repository = MockBattlecardRepository() if USE_MOCK_REPOSITORIES else BattlecardRepository()

6
backend/repositories/client_trips_repository.py
View File

@ -26,6 +26,9 @@ class ClientTripsRepository:
def list_for_tenant(self, tenant_id): def list_for_tenant(self, tenant_id):
return pb_client.list(COLLECTION, filter_str=f'tenant_id = "{tenant_id}"', per_page=200) return pb_client.list(COLLECTION, filter_str=f'tenant_id = "{tenant_id}"', per_page=200)
def delete(self, trip_id):
return pb_client.delete(COLLECTION, trip_id)
class MockClientTripsRepository: class MockClientTripsRepository:
"""In-memory stand-in for ClientTripsRepository.""" """In-memory stand-in for ClientTripsRepository."""
@ -54,5 +57,8 @@ class MockClientTripsRepository:
def list_for_tenant(self, tenant_id): def list_for_tenant(self, tenant_id):
return [r for r in self._records.values() if r.get('tenant_id') == tenant_id] return [r for r in self._records.values() if r.get('tenant_id') == tenant_id]
def delete(self, trip_id):
return self._records.pop(trip_id, None) is not None
client_trips_repository = MockClientTripsRepository() if USE_MOCK_REPOSITORIES else ClientTripsRepository() client_trips_repository = MockClientTripsRepository() if USE_MOCK_REPOSITORIES else ClientTripsRepository()

6
backend/repositories/comparable_repository.py
View File

@ -37,6 +37,9 @@ class ComparableRepository:
def dismiss(self, match_id): def dismiss(self, match_id):
return pb_client.update(COLLECTION, match_id, {'dismissed': True}) return pb_client.update(COLLECTION, match_id, {'dismissed': True})
def delete(self, match_id):
return pb_client.delete(COLLECTION, match_id)
class MockComparableRepository: class MockComparableRepository:
"""In-memory stand-in for ComparableRepository.""" """In-memory stand-in for ComparableRepository."""
@ -77,5 +80,8 @@ class MockComparableRepository:
self._records[match_id]['dismissed'] = True self._records[match_id]['dismissed'] = True
return self._records[match_id] return self._records[match_id]
def delete(self, match_id):
return self._records.pop(match_id, None) is not None
comparable_repository = MockComparableRepository() if USE_MOCK_REPOSITORIES else ComparableRepository() comparable_repository = MockComparableRepository() if USE_MOCK_REPOSITORIES else ComparableRepository()

7
backend/repositories/competitor_repository.py
View File

@ -36,6 +36,10 @@ class CompetitorRepository:
def update(self, competitor_id, data): def update(self, competitor_id, data):
return pb_client.update(COLLECTION, competitor_id, data) return pb_client.update(COLLECTION, competitor_id, data)
def delete(self, competitor_id):
"""Hard-deletes a competitor record — GDPR erasure only. Normal removal soft-deactivates via update()."""
return pb_client.delete(COLLECTION, competitor_id)
class MockCompetitorRepository: class MockCompetitorRepository:
"""In-memory stand-in for CompetitorRepository.""" """In-memory stand-in for CompetitorRepository."""
@ -70,5 +74,8 @@ class MockCompetitorRepository:
self._records[competitor_id] = {**self._records[competitor_id], **data} self._records[competitor_id] = {**self._records[competitor_id], **data}
return self._records[competitor_id] return self._records[competitor_id]
def delete(self, competitor_id):
return self._records.pop(competitor_id, None) is not None
competitor_repository = MockCompetitorRepository() if USE_MOCK_REPOSITORIES else CompetitorRepository() competitor_repository = MockCompetitorRepository() if USE_MOCK_REPOSITORIES else CompetitorRepository()

43
backend/repositories/ngs_meta_repository.py Normal file
View File

@ -0,0 +1,43 @@
import uuid
from repositories.pocketbase_client import pb_client
from repositories.repo_config import USE_MOCK_REPOSITORIES
COLLECTION = 'products_ngs_meta'
class NgsMetaRepository:
"""Data access for the `products_ngs_meta` collection — NGS-only extraction fields (exclusive_access, group_leader, sustainability) keyed by product_id."""
def get_by_product_id(self, product_id):
return pb_client.get_first(COLLECTION, f'product_id = "{product_id}"')
def create(self, data):
return pb_client.create(COLLECTION, data)
def update(self, record_id, data):
return pb_client.update(COLLECTION, record_id, data)
class MockNgsMetaRepository:
"""In-memory stand-in for NgsMetaRepository."""
def __init__(self):
self._records = {}
def get_by_product_id(self, product_id):
return next((r for r in self._records.values() if r.get('product_id') == product_id), None)
def create(self, data):
record_id = data.get('id') or str(uuid.uuid4())
record = {'id': record_id, **data}
self._records[record_id] = record
return record
def update(self, record_id, data):
if record_id not in self._records:
return None
self._records[record_id] = {**self._records[record_id], **data}
return self._records[record_id]
ngs_meta_repository = MockNgsMetaRepository() if USE_MOCK_REPOSITORIES else NgsMetaRepository()

20
backend/repositories/price_history_repository.py
View File

@ -16,6 +16,17 @@ class PriceHistoryRepository:
COLLECTION, filter_str=f'product_id = "{product_id}"', sort='-scraped_at', per_page=limit COLLECTION, filter_str=f'product_id = "{product_id}"', sort='-scraped_at', per_page=limit
) )
def list_for_tenant(self, tenant_id):
"""Used for GDPR erasure — every price_history row for a tenant, not just one product's."""
return pb_client.list(COLLECTION, filter_str=f'tenant_id = "{tenant_id}"', per_page=500)
def list_older_than(self, cutoff_iso):
"""Used by the weekly data-cleanup job — CLAUDE.md §22 DATA_RETENTION (365 days for price_history)."""
return pb_client.list(COLLECTION, filter_str=f'scraped_at < "{cutoff_iso}"', per_page=500)
def delete(self, record_id):
return pb_client.delete(COLLECTION, record_id)
class MockPriceHistoryRepository: class MockPriceHistoryRepository:
"""In-memory stand-in for PriceHistoryRepository.""" """In-memory stand-in for PriceHistoryRepository."""
@ -34,5 +45,14 @@ class MockPriceHistoryRepository:
items.sort(key=lambda r: r.get('scraped_at') or '', reverse=True) items.sort(key=lambda r: r.get('scraped_at') or '', reverse=True)
return items[:limit] return items[:limit]
def list_for_tenant(self, tenant_id):
return [r for r in self._records.values() if r.get('tenant_id') == tenant_id]
def list_older_than(self, cutoff_iso):
return [r for r in self._records.values() if (r.get('scraped_at') or '') < cutoff_iso]
def delete(self, record_id):
return self._records.pop(record_id, None) is not None
price_history_repository = MockPriceHistoryRepository() if USE_MOCK_REPOSITORIES else PriceHistoryRepository() price_history_repository = MockPriceHistoryRepository() if USE_MOCK_REPOSITORIES else PriceHistoryRepository()

13
backend/repositories/product_repository.py
View File

@ -37,12 +37,19 @@ class ProductRepository:
"""Returns all active products for a tenant — used for embedding/comparable matching.""" """Returns all active products for a tenant — used for embedding/comparable matching."""
return pb_client.list(COLLECTION, filter_str=f'tenant_id = "{tenant_id}" && is_active = true', per_page=500) return pb_client.list(COLLECTION, filter_str=f'tenant_id = "{tenant_id}" && is_active = true', per_page=500)
def list_all_for_tenant(self, tenant_id):
"""Returns every product for a tenant regardless of is_active — used for GDPR erasure."""
return pb_client.list(COLLECTION, filter_str=f'tenant_id = "{tenant_id}"', per_page=500)
def create(self, data): def create(self, data):
return pb_client.create(COLLECTION, data) return pb_client.create(COLLECTION, data)
def update(self, product_id, data): def update(self, product_id, data):
return pb_client.update(COLLECTION, product_id, data) return pb_client.update(COLLECTION, product_id, data)
def delete(self, product_id):
return pb_client.delete(COLLECTION, product_id)
class MockProductRepository: class MockProductRepository:
"""In-memory stand-in for ProductRepository.""" """In-memory stand-in for ProductRepository."""
@ -68,6 +75,9 @@ class MockProductRepository:
def list_for_tenant(self, tenant_id): def list_for_tenant(self, tenant_id):
return [r for r in self._records.values() if r.get('tenant_id') == tenant_id and r.get('is_active', True)] return [r for r in self._records.values() if r.get('tenant_id') == tenant_id and r.get('is_active', True)]
def list_all_for_tenant(self, tenant_id):
return [r for r in self._records.values() if r.get('tenant_id') == tenant_id]
def create(self, data): def create(self, data):
record_id = data.get('id') or str(uuid.uuid4()) record_id = data.get('id') or str(uuid.uuid4())
record = {'id': record_id, 'is_active': True, **data} record = {'id': record_id, 'is_active': True, **data}
@ -80,5 +90,8 @@ class MockProductRepository:
self._records[product_id] = {**self._records[product_id], **data} self._records[product_id] = {**self._records[product_id], **data}
return self._records[product_id] return self._records[product_id]
def delete(self, product_id):
return self._records.pop(product_id, None) is not None
product_repository = MockProductRepository() if USE_MOCK_REPOSITORIES else ProductRepository() product_repository = MockProductRepository() if USE_MOCK_REPOSITORIES else ProductRepository()

60
backend/repositories/scrape_repository.py
View File

@ -18,15 +18,40 @@ class ScrapeRepository:
return pb_client.get_one(COLLECTION, job_id) return pb_client.get_one(COLLECTION, job_id)
def list_recent_for_tenant(self, tenant_id, limit=5): def list_recent_for_tenant(self, tenant_id, limit=5):
"""Returns a tenant's most recent runs, newest first — backs GET /research/history.""" """
Returns a tenant's most recent BATCH runs, newest first — backs
GET /research/history. Filters to triggered_by="manual" —
analysis_service.run_competitor_analysis()'s refresh path creates
its own extra scrape_runs row PER COMPETITOR purely to persist a
content_hash for next time (hardcoded triggered_by='cron',
competitors_total=1, results={}), regardless of what triggered the
parent job. Without this filter those bookkeeping rows — created
within the same batch, so their started_at timestamps sort ahead
of or alongside the real job — drowned out the actual batch runs
in "last 5 runs", with misleading competitor counts and empty
`results` that broke Restore. 'manual' is the only triggered_by
value that represents a real, user-visible batch run today; if a
genuine scheduled *batch* job (distinct from this per-competitor
stub) is ever introduced, this filter needs revisiting.
"""
return pb_client.list( return pb_client.list(
COLLECTION, filter_str=f'tenant_id = "{tenant_id}"', sort='-started_at', per_page=limit COLLECTION, filter_str=f'tenant_id = "{tenant_id}" && triggered_by = "manual"',
sort='-started_at', per_page=limit
) )
def get_latest_complete_for_tenant(self, tenant_id): def get_latest_complete_for_tenant(self, tenant_id):
"""Returns the tenant's most recent completed run, or None — backs GET /research/history/latest.""" """
Returns the tenant's most recent completed BATCH run, or None —
backs GET /research/history/latest (the Apps Script sidebar's
"Pull latest results", CLAUDE.md §15). Same triggered_by="manual"
filter as list_recent_for_tenant() and for the same reason — the
per-competitor content-hash bookkeeping rows are also created with
status='complete', so without this filter the Sheet-pull
integration could silently pull a 1-competitor bookkeeping stub
instead of the PM's actual last analysis.
"""
items = pb_client.list( items = pb_client.list(
COLLECTION, filter_str=f'tenant_id = "{tenant_id}" && status = "complete"', COLLECTION, filter_str=f'tenant_id = "{tenant_id}" && triggered_by = "manual" && status = "complete"',
sort='-started_at', per_page=1 sort='-started_at', per_page=1
) )
return items[0] if items else None return items[0] if items else None
@ -44,6 +69,17 @@ class ScrapeRepository:
) )
return items[0]['content_hash'] if items else None return items[0]['content_hash'] if items else None
def list_all_for_tenant(self, tenant_id):
"""Every scrape_runs row for a tenant, no limit — used for GDPR erasure."""
return pb_client.list(COLLECTION, filter_str=f'tenant_id = "{tenant_id}"', per_page=500)
def list_older_than(self, cutoff_iso):
"""Used by the weekly data-cleanup job — CLAUDE.md §22 DATA_RETENTION (90 days for scrape_runs)."""
return pb_client.list(COLLECTION, filter_str=f'started_at < "{cutoff_iso}"', per_page=500)
def delete(self, job_id):
return pb_client.delete(COLLECTION, job_id)
class MockScrapeRepository: class MockScrapeRepository:
"""In-memory stand-in for ScrapeRepository.""" """In-memory stand-in for ScrapeRepository."""
@ -74,14 +110,17 @@ class MockScrapeRepository:
return self._records.get(job_id) return self._records.get(job_id)
def list_recent_for_tenant(self, tenant_id, limit=5): def list_recent_for_tenant(self, tenant_id, limit=5):
items = [r for r in self._records.values() if r.get('tenant_id') == tenant_id] items = [
r for r in self._records.values()
if r.get('tenant_id') == tenant_id and r.get('triggered_by') == 'manual'
]
items.sort(key=lambda r: r.get('started_at') or '', reverse=True) items.sort(key=lambda r: r.get('started_at') or '', reverse=True)
return items[:limit] return items[:limit]
def get_latest_complete_for_tenant(self, tenant_id): def get_latest_complete_for_tenant(self, tenant_id):
items = [ items = [
r for r in self._records.values() r for r in self._records.values()
if r.get('tenant_id') == tenant_id and r.get('status') == 'complete' if r.get('tenant_id') == tenant_id and r.get('triggered_by') == 'manual' and r.get('status') == 'complete'
] ]
items.sort(key=lambda r: r.get('started_at') or '', reverse=True) items.sort(key=lambda r: r.get('started_at') or '', reverse=True)
return items[0] if items else None return items[0] if items else None
@ -94,5 +133,14 @@ class MockScrapeRepository:
items.sort(key=lambda r: r.get('started_at') or '', reverse=True) items.sort(key=lambda r: r.get('started_at') or '', reverse=True)
return items[0]['content_hash'] if items else None return items[0]['content_hash'] if items else None
def list_all_for_tenant(self, tenant_id):
return [r for r in self._records.values() if r.get('tenant_id') == tenant_id]
def list_older_than(self, cutoff_iso):
return [r for r in self._records.values() if (r.get('started_at') or '') < cutoff_iso]
def delete(self, job_id):
return self._records.pop(job_id, None) is not None
scrape_repository = MockScrapeRepository() if USE_MOCK_REPOSITORIES else ScrapeRepository() scrape_repository = MockScrapeRepository() if USE_MOCK_REPOSITORIES else ScrapeRepository()

7
backend/repositories/tenant_repository.py
View File

@ -39,6 +39,10 @@ class TenantRepository:
"""Patches a tenant record and returns the updated record.""" """Patches a tenant record and returns the updated record."""
return pb_client.update(COLLECTION, tenant_id, data) return pb_client.update(COLLECTION, tenant_id, data)
def delete(self, tenant_id):
"""Hard-deletes a tenant record — GDPR Article 17 erasure only, never used in normal operation."""
return pb_client.delete(COLLECTION, tenant_id)
class MockTenantRepository: class MockTenantRepository:
"""In-memory stand-in for TenantRepository — same interface, no PocketBase dependency.""" """In-memory stand-in for TenantRepository — same interface, no PocketBase dependency."""
@ -67,5 +71,8 @@ class MockTenantRepository:
self._records[tenant_id] = {**self._records[tenant_id], **data} self._records[tenant_id] = {**self._records[tenant_id], **data}
return self._records[tenant_id] return self._records[tenant_id]
def delete(self, tenant_id):
return self._records.pop(tenant_id, None) is not None
tenant_repository = MockTenantRepository() if USE_MOCK_REPOSITORIES else TenantRepository() tenant_repository = MockTenantRepository() if USE_MOCK_REPOSITORIES else TenantRepository()

8
backend/requirements.txt
View File

@ -32,6 +32,14 @@ beautifulsoup4==4.12.3
lxml==5.3.0 lxml==5.3.0
nest_asyncio==1.6.0 nest_asyncio==1.6.0
openai==1.43.0 openai==1.43.0
# httpx is pinned below 0.28 because openai==1.43.0's internal client
# construction still passes `proxies=` to httpx.Client() — httpx 0.28
# removed that parameter entirely (replaced by `proxy=`, singular),
# which raises "Client.__init__() got an unexpected keyword argument
# 'proxies'" on every OpenRouter call the moment pip resolves an
# unpinned httpx to latest. Revisit together with the openai pin above
# if either is ever bumped.
httpx<0.28
gspread==6.1.2 gspread==6.1.2
google-auth==2.34.0 google-auth==2.34.0

197
backend/services/analysis_service.py
View File

@ -4,7 +4,9 @@ from repositories.competitor_repository import competitor_repository
from repositories.product_repository import product_repository from repositories.product_repository import product_repository
from repositories.price_history_repository import price_history_repository from repositories.price_history_repository import price_history_repository
from repositories.scrape_repository import scrape_repository from repositories.scrape_repository import scrape_repository
from repositories.ngs_meta_repository import ngs_meta_repository
from services import alert_service from services import alert_service
from services.url_utils import is_generic_homepage
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@ -49,15 +51,31 @@ def needs_refresh(competitor):
def analyse_from_cache(tenant_id, competitor_id, context): def analyse_from_cache(tenant_id, competitor_id, context):
""" """
FAST path analysis — uses cached PocketBase data instead of a live FAST path analysis — uses cached PocketBase data instead of a live
scrape. The LLM receives already-structured JSON and generates the scrape. The LLM's only job is to write a fresh comparative narrative
analysis narrative, comments, and relevancy score only — it does against the client's *current* context and reassess relevancy — it
NOT re-extract fields from raw page content. does NOT re-extract trip/pricing fields from raw page content, since
those already exist on the cached `products` record.
The structured fields (tripName, price, duration, etc.) returned
here come directly from that cached record, not from the LLM call.
An earlier version returned the LLM's raw output as the entire
result — since the prompt only ever asked for a narrative, with no
schema, the model would invent its own ad-hoc JSON shape (bare
`{"analysis": "..."}`, or with extra invented keys) that never
matched what ResultsTable.vue actually renders (item.result.tripName,
.majorityPrice/.gbpPrice, .duration, .comments) — the dashboard
showed a "complete" job with an empty-looking results table because
none of those fields were ever present. Now the LLM output is
merged into (not used as) the returned dict.
Data flow: Data flow:
competitor_id → PocketBase products (latest) → competitor_id → PocketBase products (latest) →
structured JSON + client trip context (destination/duration/style) → structured fields mapped from the cached record (trip_name →
LLM prompt (analysis only, no extraction, via core.extractor) → tripName, majority_price_usd → majorityPrice, etc.) →
narrative + comments + relevancy score returned client trip context (destination/duration/style) → LLM prompt
constrained to {comments, relevancy} only (via core.extractor) →
cached structured fields + fresh comments/relevancy merged and
returned
""" """
products = product_repository.get_latest_for_competitor(competitor_id, limit=1) products = product_repository.get_latest_for_competitor(competitor_id, limit=1)
if not products: if not products:
@ -71,10 +89,56 @@ def analyse_from_cache(tenant_id, competitor_id, context):
prompt = ( prompt = (
'You are a travel industry analyst. Given this already-extracted ' 'You are a travel industry analyst. Given this already-extracted '
'competitor trip data, write a short comparative analysis against ' 'competitor trip data, assess it against '
f"the client's product context: {context}.\n\nCompetitor data:\n{latest}" f"the client's product context: {context}.\n\nCompetitor data:\n{latest}\n\n"
'Return ONLY a valid JSON object:\n'
'{\n'
' "comments": "short comparative analysis: key differences, strengths, weaknesses, pricing position",\n'
' "relevancy": relevancy score 1-5 as integer\n'
'}'
) )
return extract_with_openrouter(prompt, content='') narrative = extract_with_openrouter(prompt, content='')
result = {
'link': latest.get('url'),
'tripName': latest.get('trip_name'),
'tripCode': latest.get('trip_code'),
'serviceLevel': latest.get('service_level'),
'groupSize': latest.get('group_size'),
'duration': latest.get('duration_days'),
'meals': latest.get('meals'),
'startLocation': latest.get('start_location'),
'endLocation': latest.get('end_location'),
'departures': latest.get('departures'),
'seasonality': latest.get('seasonality'),
'startDays': latest.get('start_days'),
'targetAudience': latest.get('target_audience'),
'hotels': latest.get('hotels'),
'activities': latest.get('activities'),
'majorityPrice': latest.get('majority_price_usd'),
'priceLow': latest.get('price_low_usd'),
'priceHigh': latest.get('price_high_usd'),
'dateUsed': latest.get('date_used'),
'audPrice': latest.get('aud_price'),
'cadPrice': latest.get('cad_price'),
'eurPrice': latest.get('eur_price'),
'gbpPrice': latest.get('gbp_price'),
'comments': narrative.get('comments', ''),
'relevancy': narrative.get('relevancy', latest.get('relevancy')),
}
# NGS fields live on a separate products_ngs_meta record (see
# _save_ngs_meta()) — only look it up when this run is actually NGS,
# to avoid an extra PocketBase round trip on the common Standard-tab
# fast path.
if context.get('tab_type') == 'NGS':
ngs_meta = ngs_meta_repository.get_by_product_id(latest['id'])
if ngs_meta:
result['exclusiveAccess'] = ngs_meta.get('exclusive_access')
result['groupLeader'] = ngs_meta.get('group_leader')
result['sustainability'] = ngs_meta.get('sustainability')
return result
def detect_change(competitor_id, new_hash): def detect_change(competitor_id, new_hash):
@ -110,31 +174,60 @@ def run_competitor_analysis(tenant_id, competitor, context):
caught by the caller so one competitor's failure never aborts the caught by the caller so one competitor's failure never aborts the
whole batch (CLAUDE.md §18 resilience rule). whole batch (CLAUDE.md §18 resilience rule).
context['force_refresh'] (PM-initiated, via TripIntentForm.vue's
collapsed "Advanced options" toggle) bypasses BOTH cache-shortcut
points below — the initial needs_refresh() fast-path gate, and the
post-scrape "content unchanged, reuse cached narrative" shortcut — so
a forced run always ends up calling extract_with_openrouter() for real,
never analyse_from_cache(). The content-hash bookkeeping itself
(scrape_runs row, change_detected clearing) still runs normally even
when forced, so future non-forced runs keep an accurate hash history.
Data flow: Data flow:
competitor + context → needs_refresh() decision → competitor + context → needs_refresh() decision (or force_refresh) →
FAST: analyse_from_cache() (seconds) → FAST: analyse_from_cache() (seconds) →
REFRESH: core.scraper.scrape() → compute_content_hash() → REFRESH: core.scraper.scrape() → compute_content_hash() →
detect_change() against the last recorded hash → detect_change() against the last recorded hash →
unchanged: change_detected cleared (confirmed stable since last unchanged (and not force_refresh): change_detected cleared
look), skip LLM extraction, reuse cached narrative, bump (confirmed stable since last look), skip LLM extraction, reuse
last_scraped only (no wasted extraction cost) → cached narrative, bump last_scraped only (no wasted extraction
changed: core.extractor.extract_with_openrouter() → cost) →
changed (or force_refresh): core.extractor.extract_with_openrouter() →
PocketBase products/price_history updated → PocketBase products/price_history updated →
last_scraped bumped, change_detected left True (stays visible last_scraped bumped, change_detected left True (stays visible
until the next scrape confirms no further change) → until the next scrape confirms no further change) →
result dict returned result dict returned
""" """
if not needs_refresh(competitor): force_refresh = context.get('force_refresh', False)
logger.info(f'{competitor["name"]} — fast path (cache fresh)')
return {'path': 'fast', 'result': analyse_from_cache(tenant_id, competitor['id'], context)}
logger.info(f'{competitor["name"]} — refresh path (stale or changed)') if not needs_refresh(competitor) and not force_refresh:
logger.info(f'{competitor["name"]} — fast path (cache fresh)')
fast_result = analyse_from_cache(tenant_id, competitor['id'], context)
return {
'path': 'fast', 'result': fast_result,
'is_generic_page': is_generic_homepage(fast_result.get('link')),
}
logger.info(f'{competitor["name"]} — refresh path (force_refresh)' if force_refresh else f'{competitor["name"]} — refresh path (stale or changed)')
# Phase 2 modules — imported lazily, same reasoning as above. # Phase 2 modules — imported lazily, same reasoning as above.
from core.scraper import scrape, compute_content_hash from core.scraper import scrape, compute_content_hash
from core.extractor import extract_with_openrouter from core.extractor import extract_with_openrouter
from industries.adventure_travel.prompt import build_prompt from industries.adventure_travel.prompt import build_prompt
page_data = scrape(competitor['website'], country=competitor.get('primary_market')) # context['confirmed_url'] is the page confirmed at UrlConfirmation.vue
# (a Firecrawl /map match or a manual swap) — falls back to the
# competitor's stored root domain only when nothing was confirmed for
# this run (e.g. the daily cron path, which has no trip-intent context
# at all). Regression fix: this used to always scrape
# competitor['website'] unconditionally, silently discarding whatever
# trip-specific URL was actually matched/confirmed for this run.
target_url = context.get('confirmed_url') or competitor['website']
# Computed once here and reused by both possible refresh-path returns
# below — a meta-fact about which page was actually scraped this run,
# not an extracted trip field, so it's a top-level sibling of `result`/
# `path`/`product` in the returned dict rather than nested inside `result`.
is_generic = is_generic_homepage(target_url)
page_data = scrape(target_url, country=competitor.get('primary_market'))
if not page_data.get('success'): if not page_data.get('success'):
raise ValueError(page_data.get('text', 'Scrape failed')) raise ValueError(page_data.get('text', 'Scrape failed'))
@ -153,7 +246,7 @@ def run_competitor_analysis(tenant_id, competitor, context):
'results': {}, 'completed_at': datetime.now(timezone.utc).isoformat(), 'results': {}, 'completed_at': datetime.now(timezone.utc).isoformat(),
}) })
if not changed: if not changed and not force_refresh:
# Confirmed no further change since the last scrape — this is # Confirmed no further change since the last scrape — this is
# the moment change_detected gets cleared. If it was never true # the moment change_detected gets cleared. If it was never true
# to begin with, this is a harmless no-op. # to begin with, this is a harmless no-op.
@ -162,12 +255,25 @@ def run_competitor_analysis(tenant_id, competitor, context):
'change_detected': False, 'change_detected': False,
'last_scraped': datetime.now(timezone.utc).isoformat(), 'last_scraped': datetime.now(timezone.utc).isoformat(),
}) })
return {'path': 'refresh', 'result': analyse_from_cache(tenant_id, competitor['id'], context), 'unchanged': True} return {
'path': 'refresh', 'result': analyse_from_cache(tenant_id, competitor['id'], context),
'unchanged': True, 'is_generic_page': is_generic,
}
if not changed:
# force_refresh bypasses the "reuse cache" shortcut, but the hash
# comparison itself is still accurate — content really is
# unchanged — so change_detected is still cleared here. Only the
# decision to skip real extraction is bypassed; falls through to
# a real extraction below instead of returning early.
logger.info(f'{competitor["name"]} — unchanged but force_refresh requested, running real extraction anyway')
competitor_repository.update(competitor['id'], {'change_detected': False})
prompt = build_prompt( prompt = build_prompt(
competitor['name'], page_data, competitor['name'], page_data,
context.get('product_name'), context.get('destination'), context.get('product_name'), context.get('destination'),
context.get('duration'), context.get('travel_style'), context.get('duration'), context.get('travel_style'),
company_name=context.get('company_name'),
is_ngs=context.get('tab_type') == 'NGS' is_ngs=context.get('tab_type') == 'NGS'
) )
extracted = extract_with_openrouter(prompt, content=page_data['text']) extracted = extract_with_openrouter(prompt, content=page_data['text'])
@ -186,7 +292,7 @@ def run_competitor_analysis(tenant_id, competitor, context):
'last_scraped': datetime.now(timezone.utc).isoformat(), 'last_scraped': datetime.now(timezone.utc).isoformat(),
}) })
return {'path': 'refresh', 'result': extracted, 'product': product} return {'path': 'refresh', 'result': extracted, 'product': product, 'is_generic_page': is_generic}
def _save_scrape_result(tenant_id, competitor, extracted, page_data): def _save_scrape_result(tenant_id, competitor, extracted, page_data):
@ -227,6 +333,24 @@ def _save_scrape_result(tenant_id, competitor, extracted, page_data):
'date_used': extracted.get('dateUsed'), 'date_used': extracted.get('dateUsed'),
'comments': extracted.get('comments'), 'comments': extracted.get('comments'),
'relevancy': extracted.get('relevancy'), 'relevancy': extracted.get('relevancy'),
# These all already have PocketBase columns and were previously
# extracted by the LLM (industries/adventure_travel/prompt.py's
# build_prompt() JSON schema) but silently dropped on write — the
# dashboard's transposed results grid (ResultsTable.vue /
# resultFields.js) needs every one of these to mirror the legacy
# Sheet's row layout.
'trip_code': extracted.get('tripCode'),
'service_level': extracted.get('serviceLevel'),
'group_size': extracted.get('groupSize'),
'meals': extracted.get('meals'),
'start_location': extracted.get('startLocation'),
'end_location': extracted.get('endLocation'),
'departures': extracted.get('departures'),
'seasonality': extracted.get('seasonality'),
'start_days': extracted.get('startDays'),
'target_audience': extracted.get('targetAudience'),
'hotels': extracted.get('hotels'),
'activities': extracted.get('activities'),
'last_seen': now, 'last_seen': now,
'is_active': True, 'is_active': True,
} }
@ -246,6 +370,7 @@ def _save_scrape_result(tenant_id, competitor, extracted, page_data):
f"{competitor['name']} added a new product: {trip_name or 'Unnamed trip'}", f"{competitor['name']} added a new product: {trip_name or 'Unnamed trip'}",
product_id=product['id'], product_id=product['id'],
) )
_save_ngs_meta(product['id'], extracted)
return product return product
product = product_repository.update(existing['id'], {**common_fields, 'is_new': False}) product = product_repository.update(existing['id'], {**common_fields, 'is_new': False})
@ -279,4 +404,32 @@ def _save_scrape_result(tenant_id, competitor, extracted, page_data):
product_id=product['id'], change_amount=change_amount, change_percent=change_percent, product_id=product['id'], change_amount=change_amount, change_percent=change_percent,
) )
_save_ngs_meta(product['id'], extracted)
return product return product
def _save_ngs_meta(product_id, extracted):
"""
Upserts NGS-only extraction fields (exclusiveAccess/groupLeader/
sustainability) into products_ngs_meta, keyed on product_id. Only
fires when build_prompt() actually asked for these fields (is_ngs=True
in run_competitor_analysis()) — inferred here from their presence in
`extracted` rather than threading a tab_type/context param through
_save_scrape_result()'s already-long signature, since a Standard-tab
extraction never has these keys at all. Silently no-ops for a
Standard-tab run.
"""
if not any(extracted.get(k) is not None for k in ('exclusiveAccess', 'groupLeader', 'sustainability')):
return
fields = {
'product_id': product_id,
'exclusive_access': extracted.get('exclusiveAccess'),
'group_leader': extracted.get('groupLeader'),
'sustainability': extracted.get('sustainability'),
}
existing = ngs_meta_repository.get_by_product_id(product_id)
if existing:
ngs_meta_repository.update(existing['id'], fields)
else:
ngs_meta_repository.create(fields)

123
backend/services/billing_service.py
View File

@ -78,9 +78,21 @@ def create_upgrade_checkout(tenant_id, target_tier):
if not tenant: if not tenant:
raise ValueError(f'Unknown tenant: {tenant_id}') raise ValueError(f'Unknown tenant: {tenant_id}')
# Stripe rejects an empty-string `customer` param outright (it only
# accepts a real customer id or the param being omitted entirely) —
# every trial/manually-onboarded tenant (CLAUDE.md §32) starts with
# no Stripe customer at all, so this must be conditional. When
# there's no customer yet, `customer_email` pre-fills checkout and
# Stripe creates the customer during this session; the resulting
# id is captured in handle_checkout_completed() below.
checkout_kwargs = {}
if tenant.get('stripe_customer_id'):
checkout_kwargs['customer'] = tenant['stripe_customer_id']
elif tenant.get('email'):
checkout_kwargs['customer_email'] = tenant['email']
stripe = _stripe() stripe = _stripe()
session = stripe.checkout.Session.create( session = stripe.checkout.Session.create(
customer=tenant.get('stripe_customer_id'),
mode='subscription', mode='subscription',
line_items=[{'price': os.getenv(price_env_var), 'quantity': 1}], line_items=[{'price': os.getenv(price_env_var), 'quantity': 1}],
payment_method_collection='if_required', payment_method_collection='if_required',
@ -88,6 +100,7 @@ def create_upgrade_checkout(tenant_id, target_tier):
metadata={'tenant_id': tenant_id, 'target_tier': target_tier}, metadata={'tenant_id': tenant_id, 'target_tier': target_tier},
success_url='https://app.bettersight.io/account?upgrade=success', success_url='https://app.bettersight.io/account?upgrade=success',
cancel_url='https://app.bettersight.io/account?upgrade=cancelled', cancel_url='https://app.bettersight.io/account?upgrade=cancelled',
**checkout_kwargs,
) )
return session.url return session.url
@ -118,7 +131,7 @@ def handle_checkout_completed(event_data):
logger.info('checkout.session.completed with no upgrade metadata — ignoring (manual onboarding flow)') logger.info('checkout.session.completed with no upgrade metadata — ignoring (manual onboarding flow)')
return return
tenant_repository.update(tenant_id, { updates = {
'tier': target_tier, 'tier': target_tier,
'max_seats': TIER_SEATS.get(target_tier, 5), 'max_seats': TIER_SEATS.get(target_tier, 5),
'stripe_subscription_id': event_data.get('subscription'), 'stripe_subscription_id': event_data.get('subscription'),
@ -127,7 +140,16 @@ def handle_checkout_completed(event_data):
(datetime.now(timezone.utc) + timedelta(days=TRIAL_PERIODS[target_tier])).isoformat() (datetime.now(timezone.utc) + timedelta(days=TRIAL_PERIODS[target_tier])).isoformat()
if TRIAL_PERIODS.get(target_tier, 0) > 0 else None if TRIAL_PERIODS.get(target_tier, 0) > 0 else None
), ),
}) }
# Captures the customer id Stripe created during this checkout for
# tenants that had none yet — without this, the tenant would still
# have no stripe_customer_id after their very first upgrade, and
# both the next upgrade and the billing portal route would fail
# the same way this one just did.
if event_data.get('customer'):
updates['stripe_customer_id'] = event_data['customer']
tenant_repository.update(tenant_id, updates)
def handle_subscription_updated(event_data, tenant_id): def handle_subscription_updated(event_data, tenant_id):
@ -144,3 +166,98 @@ def handle_subscription_updated(event_data, tenant_id):
def handle_subscription_deleted(tenant_id): def handle_subscription_deleted(tenant_id):
"""Marks a tenant inactive on cancellation — seats block at next Apps Script cache expiry (6hrs).""" """Marks a tenant inactive on cancellation — seats block at next Apps Script cache expiry (6hrs)."""
tenant_repository.update(tenant_id, {'status': 'inactive'}) tenant_repository.update(tenant_id, {'status': 'inactive'})
def delete_tenant_data(tenant_id):
"""
GDPR Article 17 (Right to Erasure) — permanently deletes a tenant
and every tenant-scoped collection, per CLAUDE.md §22.
CLAUDE.md's spec says to "log to audit_logs" as a separate step,
but no such collection exists anywhere in the schema — the closest
real candidates (alerts, digest_logs) are themselves in the delete
list. This logs the deletion via standard Python logging (visible
in Sentry/server logs) instead of inventing new PocketBase schema
for a single audit line; revisit if a real compliance audit trail
is ever required.
§18's "audit logs are append-only" note about alerts/digest_logs
describes normal runtime retention — it doesn't override a
tenant's explicit erasure request, and alerts contain no PII of
their own (they describe competitor products, not the PM).
Data flow:
tenant_id → cancel Stripe subscription (side-effect, logged not
raised) → delete every tenant-scoped collection row → delete the
tenant record itself → send deletion confirmation email
(side-effect) → logged
"""
from repositories.seat_repository import seat_repository
from repositories.competitor_repository import competitor_repository
from repositories.product_repository import product_repository
from repositories.price_history_repository import price_history_repository
from repositories.scrape_repository import scrape_repository
from repositories.alert_repository import alert_repository
from repositories.battlecard_repository import battlecard_repository
from repositories.comparable_repository import comparable_repository
from repositories.client_trips_repository import client_trips_repository
from services import email_service
tenant = tenant_repository.get_by_id(tenant_id)
if not tenant:
return False
if tenant.get('stripe_subscription_id'):
try:
stripe = _stripe()
stripe.Subscription.delete(tenant['stripe_subscription_id'])
except Exception as e:
logger.error(f'Stripe cancellation failed during account deletion: {str(e)}')
for seat in seat_repository.list_for_tenant(tenant_id):
seat_repository.delete(seat['id'])
for product in product_repository.list_all_for_tenant(tenant_id):
product_repository.delete(product['id'])
for record in price_history_repository.list_for_tenant(tenant_id):
price_history_repository.delete(record['id'])
for run in scrape_repository.list_all_for_tenant(tenant_id):
scrape_repository.delete(run['id'])
for alert in alert_repository.list_recent(tenant_id, limit=1000):
alert_repository.delete(alert['id'])
for card in battlecard_repository.list_for_tenant(tenant_id):
battlecard_repository.delete(card['id'])
for match in comparable_repository.list_for_tenant(tenant_id, include_dismissed=True):
comparable_repository.delete(match['id'])
for trip in client_trips_repository.list_for_tenant(tenant_id):
client_trips_repository.delete(trip['id'])
# Competitors deleted last — nothing else references them by the
# time this runs, and hard-deleting here (unlike the soft-deactivate
# used by normal /account/competitors/<id> removal) is correct only
# because the whole tenant is being erased.
for competitor in competitor_repository.list_for_tenant(tenant_id, active_only=False):
competitor_repository.delete(competitor['id'])
admin_email = tenant.get('email')
tenant_name = tenant.get('name')
tenant_repository.delete(tenant_id)
# Erasure is already complete by this point — a failure sending the
# confirmation is a side effect only, never allowed to surface as a
# failure of the deletion itself.
if admin_email:
try:
email_service.send_account_deleted_email(admin_email, tenant_name)
except Exception as e:
logger.error(f'Deletion confirmation email failed for tenant {tenant_id}: {str(e)}')
logger.info(f'GDPR erasure completed for tenant {tenant_id} ({tenant_name})')
return True

12
backend/services/email_service.py
View File

@ -105,6 +105,18 @@ def send_new_product_email(to_email, alert):
}) })
def send_account_deleted_email(to_email, tenant_name):
"""
Sends the GDPR Article 17 deletion confirmation — CLAUDE.md §22
"send deletion confirmation email", the last thing to happen after
every tenant-scoped collection has actually been purged.
"""
return _send('account_deleted.html', to_email, 'Your Bettersight account has been deleted', {
'name': tenant_name,
'tenant_name': tenant_name,
})
def send_daily_digest_email(to_email, tenant_name, alerts): def send_daily_digest_email(to_email, tenant_name, alerts):
""" """
Sends the 6pm daily digest — one email summarising every alert Sends the 6pm daily digest — one email summarising every alert

62
backend/services/retention_service.py Normal file
View File

@ -0,0 +1,62 @@
import logging
from datetime import datetime, timedelta, timezone
from repositories.scrape_repository import scrape_repository
from repositories.price_history_repository import price_history_repository
from repositories.alert_repository import alert_repository
logger = logging.getLogger(__name__)
# CLAUDE.md §22 DATA_RETENTION — scrape_runs and alerts (read only) at
# 90 days, price_history at 365. tenant_data (0 = retained until an
# explicit deletion request) has no cron counterpart — that's
# billing_service.delete_tenant_data(). audit_logs (730 days) has no
# implementation here — see billing_service's note on why no
# audit_logs collection exists in this schema.
DATA_RETENTION_DAYS = {
'scrape_runs': 90,
'price_history': 365,
'alerts': 90,
}
def run_weekly_cleanup():
"""
Deletes data past its retention window — the Sunday 3am n8n
workflow's only job is to trigger this. All real logic lives here,
not in n8n, matching every other internal route in this codebase.
Data flow:
now → per-collection cutoff = now - retention_days →
scrape_runs older than cutoff: deleted →
alerts older than cutoff AND read=true: deleted (unread alerts
are never auto-deleted, regardless of age) →
price_history older than cutoff: deleted →
counts logged and returned
"""
now = datetime.now(timezone.utc)
scrape_cutoff = (now - timedelta(days=DATA_RETENTION_DAYS['scrape_runs'])).isoformat()
scrape_runs_deleted = 0
for run in scrape_repository.list_older_than(scrape_cutoff):
scrape_repository.delete(run['id'])
scrape_runs_deleted += 1
alert_cutoff = (now - timedelta(days=DATA_RETENTION_DAYS['alerts'])).isoformat()
alerts_deleted = 0
for alert in alert_repository.list_read_older_than(alert_cutoff):
alert_repository.delete(alert['id'])
alerts_deleted += 1
price_history_cutoff = (now - timedelta(days=DATA_RETENTION_DAYS['price_history'])).isoformat()
price_history_deleted = 0
for record in price_history_repository.list_older_than(price_history_cutoff):
price_history_repository.delete(record['id'])
price_history_deleted += 1
result = {
'scrape_runs_deleted': scrape_runs_deleted,
'alerts_deleted': alerts_deleted,
'price_history_deleted': price_history_deleted,
}
logger.info(f'Weekly data cleanup: {result}')
return result

40
backend/services/trip_finder_service.py
View File

@ -2,6 +2,7 @@ import os
import logging import logging
import requests import requests
from concurrent.futures import ThreadPoolExecutor, as_completed from concurrent.futures import ThreadPoolExecutor, as_completed
from services.url_utils import is_generic_homepage
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@ -17,28 +18,55 @@ def score_and_rank_urls(urls, destination, duration):
Scores candidate URLs by keyword match against destination and Scores candidate URLs by keyword match against destination and
duration, returning the highest-scoring URL with a confidence score. duration, returning the highest-scoring URL with a confidence score.
A URL must mention at least one destination keyword to be considered
at all — this is a hard gate, not an additive bonus. Regression found
in production: a URL for a completely different country could still
cross CONFIDENCE_FOUND_THRESHOLD purely from a duration-digit match
plus a generic category-keyword match (e.g. a Turkey trip page scored
0.75 confidence against a Peru search just because it happened to be a
"9 day classic" tour) — destination relevance must be a prerequisite,
not something duration/category signals alone can compensate for.
A bare root-domain URL (no path at all) is excluded the same way, for
the same reason: Firecrawl's /map candidate list commonly includes the
site's own root URL, and if the destination keyword happens to appear
in the domain itself (e.g. a competitor literally named
"peru-travel.com"), the root URL could otherwise score high enough to
be returned as a confident "found" match — silently presenting a
homepage as if it were a trip-specific page, the exact failure mode
this whole scoring function exists to prevent.
Data flow: Data flow:
urls + destination + duration → urls + destination + duration →
per-url score: +1 per destination keyword in URL, +2 if duration per-url: zero destination-keyword matches, or a bare root domain →
appears in URL, +1 per travel-category keyword in URL → excluded entirely →
surviving urls scored: +1 per destination keyword in URL, +2 if
duration appears in URL, +1 per travel-category keyword in URL →
sort by score → confidence = top_score / max_possible_score → sort by score → confidence = top_score / max_possible_score →
{ url, confidence, found } — found=False (prompt manual entry) { url, confidence, found } — found=False (prompt manual entry)
when confidence is below CONFIDENCE_FOUND_THRESHOLD when no url survives the gate, or confidence is below
CONFIDENCE_FOUND_THRESHOLD
""" """
dest_keywords = destination.lower().split() dest_keywords = destination.lower().split()
scores = [] scores = []
for url in urls: for url in urls:
if is_generic_homepage(url):
continue # bare root domain — never a trip-specific candidate
url_lower = url.lower() url_lower = url.lower()
score = 0 dest_matches = sum(1 for kw in dest_keywords if kw in url_lower)
score += sum(1 for kw in dest_keywords if kw in url_lower) if dest_matches == 0:
continue # zero destination relevance — never a candidate, regardless of duration/category signal
score = dest_matches
if str(duration) in url_lower: if str(duration) in url_lower:
score += 2 score += 2
score += sum(1 for kw in CATEGORY_KEYWORDS if kw in url_lower) score += sum(1 for kw in CATEGORY_KEYWORDS if kw in url_lower)
scores.append((score, url)) scores.append((score, url))
scores.sort(key=lambda pair: pair[0], reverse=True) scores.sort(key=lambda pair: pair[0], reverse=True)
if not scores or scores[0][0] == 0: if not scores:
return {'url': None, 'confidence': 0, 'found': False} return {'url': None, 'confidence': 0, 'found': False}
top_score, top_url = scores[0] top_score, top_url = scores[0]

18
backend/services/url_utils.py Normal file
View File

@ -0,0 +1,18 @@
from urllib.parse import urlparse
def is_generic_homepage(url):
"""
True when `url` has no meaningful path — the bare root domain (with or
without a trailing slash, query string, or www prefix), not a
trip-specific page. Used to flag/exclude scrapes and matches that landed
on a competitor's homepage instead of an actual trip page.
Known limitation: only catches a structurally empty path — not a "soft"
generic page like /home or /en/. Not worth guarding against
speculatively; revisit if that turns out to be a real case in practice.
"""
if not url:
return False
parsed = urlparse(url if '://' in url else f'//{url}')
return (parsed.path or '').rstrip('/') == ''

42
backend/templates/emails/account_deleted.html Normal file
View File

@ -0,0 +1,42 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Your Bettersight account has been deleted</title>
</head>
<body style="margin:0;padding:0;background:#EEF1F6;font-family:Arial,sans-serif;color:#0D1526;">
<table width="100%" cellpadding="0" cellspacing="0" style="max-width:600px;margin:0 auto;background:#FFFFFF;">
<tr>
<td style="padding:24px 28px;border-bottom:1px solid #E5E9F0;">
<span style="font-size:18px;font-weight:800;color:#0D1526;">Better<span style="color:#00D4FF;">sight</span></span>
</td>
</tr>
<tr>
<td style="padding:28px;">
<h1 style="font-size:18px;margin:0 0 14px;">Your account has been deleted</h1>
<p style="font-size:13.5px;color:#2D3A52;line-height:1.7;margin:0 0 18px;">
Hi {{ name }},<br><br>
As requested, your Bettersight account for {{ tenant_name }} and all associated
data — competitors, tracked products, price history, alerts, battlecards, and
comparable trip matches — has been permanently deleted.
</p>
<p style="font-size:13px;color:#6B7A99;line-height:1.6;margin:0 0 20px;">
Your subscription has been cancelled. If this was a mistake or you have any
questions, reply to this email or contact support within the next few days —
once deleted, this data cannot be recovered.
</p>
</td>
</tr>
<tr>
<td style="padding:16px 28px;font-size:11px;color:#9BA8C0;border-top:1px solid #E5E9F0;">
Bettersight · Better sight, better moves.
<br>
Questions? Reply to this email or visit
<a href="https://support.bettersight.io" style="color:#9BA8C0;">support.bettersight.io</a>
</td>
</tr>
</table>
</body>
</html>

43
backend/tests/conftest.py
View File

@ -14,6 +14,25 @@ os.environ.setdefault('FIRECRAWL_URL', 'http://firecrawl.invalid')
os.environ.setdefault('GOTIFY_URL', 'http://gotify.invalid') os.environ.setdefault('GOTIFY_URL', 'http://gotify.invalid')
os.environ.setdefault('GOTIFY_APP_TOKEN', 'test-token') os.environ.setdefault('GOTIFY_APP_TOKEN', 'test-token')
# Force-cleared (not setdefault) rather than left alone: app.py's
# load_dotenv() runs on every `client`/`app` fixture use and picks up
# whatever real credentials happen to be sitting in backend/.env on
# the machine running the suite — nothing above protects these, since
# they were never set here before. With real values, SENTRY_DSN
# actually initialises Sentry and ships real events during test runs,
# and STRIPE_PRICE_DISCOVER stops being "unset" for the one test that
# depends on it being absent. Setting them to '' here wins that race —
# load_dotenv() never overrides an already-present env var, even an
# empty one — so tests stay hermetic regardless of what's in .env.
for _leaky_var in (
'SENTRY_DSN', 'STRIPE_SECRET_KEY', 'STRIPE_WEBHOOK_SECRET',
'STRIPE_PRICE_ANALYSE', 'STRIPE_PRICE_DISCOVER', 'STRIPE_PRICE_INTELLIGENCE',
'RESEND_API_KEY', 'OPENROUTER_API_KEY', 'WEBSHARE_USER', 'WEBSHARE_PASS',
'BRIGHTDATA_USER', 'BRIGHTDATA_PASS', 'HETZNER_S3_ACCESS_KEY', 'HETZNER_S3_SECRET_KEY',
'GOOGLE_SERVICE_ACCOUNT_JSON', 'SHEET_ID',
):
os.environ[_leaky_var] = ''
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..')) sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..'))
import pytest import pytest
@ -36,6 +55,27 @@ def api_headers():
return {'X-API-Key': 'test-api-key'} return {'X-API-Key': 'test-api-key'}
@pytest.fixture(autouse=True)
def _reset_rate_limiter():
"""
Flask-Limiter's memory:// storage is a module-level singleton
(extensions.py's `limiter` is created once at import time) — its
hit counters persist across the whole pytest session, not per
test. Without this, any route with a tight limit (e.g. "3 per
day") starts failing tests with 429s once enough tests across the
whole suite happen to call it, regardless of which test "used up"
the quota.
"""
yield
from extensions import limiter
# Only initialised once some test in this session has actually used
# the `app`/`client` fixture (app.py's limiter.init_app(app)) — pure
# service-layer tests never touch Flask at all, and limiter.reset()
# asserts on a storage backend that was never attached.
if getattr(limiter, '_storage', None) is not None:
limiter.reset()
@pytest.fixture(autouse=True) @pytest.fixture(autouse=True)
def _reset_mock_repositories(): def _reset_mock_repositories():
""" """
@ -55,12 +95,13 @@ def _reset_mock_repositories():
from repositories.monitoring_repository import monitoring_repository from repositories.monitoring_repository import monitoring_repository
from repositories.price_history_repository import price_history_repository from repositories.price_history_repository import price_history_repository
from repositories.client_trips_repository import client_trips_repository from repositories.client_trips_repository import client_trips_repository
from repositories.ngs_meta_repository import ngs_meta_repository
for repo in [ for repo in [
tenant_repository, seat_repository, competitor_repository, product_repository, tenant_repository, seat_repository, competitor_repository, product_repository,
scrape_repository, proxy_repository, alert_repository, battlecard_repository, scrape_repository, proxy_repository, alert_repository, battlecard_repository,
comparable_repository, monitoring_repository, price_history_repository, comparable_repository, monitoring_repository, price_history_repository,
client_trips_repository, client_trips_repository, ngs_meta_repository,
]: ]:
if hasattr(repo, '_records'): if hasattr(repo, '_records'):
repo._records.clear() repo._records.clear()

29
backend/tests/core/test_ai_fetch.py
View File

@ -74,3 +74,32 @@ def test_fetch_with_ai_appends_citation_content(monkeypatch):
result = ai_fetch.fetch_with_ai('https://intrepidtravel.com/peru') result = ai_fetch.fetch_with_ai('https://intrepidtravel.com/peru')
assert 'extra citation text' in result['text'] assert 'extra citation text' in result['text']
def test_fetch_with_ai_prompt_forbids_markdown_and_scopes_to_single_page(monkeypatch):
"""
Regression test — a real failure showed gpt-4o-mini defaulting to a
markdown-formatted, multi-trip summary of the whole domain when the
prompt didn't explicitly forbid that. The resulting markdown then
confused the downstream extraction model (build_prompt()'s "Return
ONLY a valid JSON object, no markdown" instruction) into producing
unparseable output. The prompt sent here must explicitly rule both
of those out.
"""
captured = {}
class _FakeClient:
class chat:
class completions:
@staticmethod
def create(**kwargs):
captured['messages'] = kwargs['messages']
return _FakeResponse(_FakeMessage('x' * 400))
monkeypatch.setattr(ai_fetch, 'OpenAI', lambda api_key, base_url: _FakeClient())
ai_fetch.fetch_with_ai('https://intrepidtravel.com/peru')
prompt_text = captured['messages'][0]['content']
assert 'no markdown' in prompt_text.lower()
assert 'this exact url' in prompt_text.lower() or 'this one tour' in prompt_text.lower()

110
backend/tests/core/test_extractor.py
View File

@ -1,6 +1,8 @@
import pytest import pytest
from core import extractor from core import extractor
from core.extractor import extract_with_openrouter, ExtractorError, RateLimitError, _parse_json_response from core.extractor import (
extract_with_openrouter, ExtractorError, RateLimitError, CreditExhaustedError, _parse_json_response,
)
def test_parse_json_response_handles_clean_json(): def test_parse_json_response_handles_clean_json():
@ -52,6 +54,19 @@ def test_call_openrouter_raises_rate_limit_error(monkeypatch):
extractor._call_openrouter('some-model', 'prompt text', '') extractor._call_openrouter('some-model', 'prompt text', '')
def test_call_openrouter_raises_credit_exhausted_error(monkeypatch):
class _FakeClient:
class chat:
class completions:
@staticmethod
def create(**kwargs):
raise Exception('403 Forbidden: insufficient credit balance for this model')
monkeypatch.setattr(extractor, 'OpenAI', lambda api_key, base_url: _FakeClient())
with pytest.raises(CreditExhaustedError):
extractor._call_openrouter('some-model', 'prompt text', '')
def test_call_openrouter_reraises_other_errors(monkeypatch): def test_call_openrouter_reraises_other_errors(monkeypatch):
class _FakeClient: class _FakeClient:
class chat: class chat:
@ -84,9 +99,42 @@ def test_call_openrouter_appends_content_when_present(monkeypatch):
assert 'EXTRA CONTENT' in captured['messages'][0]['content'] assert 'EXTRA CONTENT' in captured['messages'][0]['content']
def test_extract_with_openrouter_tries_next_model_on_rate_limit(monkeypatch): def test_call_openrouter_forces_json_object_response_format(monkeypatch):
"""
Regression test — deepseek/deepseek-v4-flash wrote a full prose
comparative analysis instead of the requested JSON object on a real
production call, whenever the competitor's content didn't clearly
match the requested trip. response_format=json_object constrains
this at the API level rather than relying on prompt wording alone.
"""
captured = {}
class _FakeClient:
class chat:
class completions:
@staticmethod
def create(**kwargs):
captured.update(kwargs)
return _FakeCompletionResponse('{"ok": true}')
monkeypatch.setattr(extractor, 'OpenAI', lambda api_key, base_url: _FakeClient())
extractor._call_openrouter('some-model', 'prompt', '')
assert captured['response_format'] == {'type': 'json_object'}
def test_extract_with_openrouter_retries_same_model_on_rate_limit_before_moving_on(monkeypatch):
"""
Regression test — the Phase 2 refactor moved to the next model on
the very first rate-limit response, dropping the legacy app.py's
retry-with-backoff behavior. A model-free-1 that's merely rate
limited (not exhausted) should get RETRY_ATTEMPTS_PER_MODEL tries
before extract_with_openrouter gives up on it.
"""
monkeypatch.setattr(extractor, 'FREE_MODELS', ['model-free-1', 'model-free-2']) monkeypatch.setattr(extractor, 'FREE_MODELS', ['model-free-1', 'model-free-2'])
monkeypatch.setattr(extractor, 'FALLBACK_MODEL', 'model-paid') monkeypatch.setattr(extractor, 'FALLBACK_MODEL', 'model-paid')
sleeps = []
monkeypatch.setattr(extractor.time, 'sleep', lambda seconds: sleeps.append(seconds))
calls = [] calls = []
@ -101,14 +149,40 @@ def test_extract_with_openrouter_tries_next_model_on_rate_limit(monkeypatch):
result = extract_with_openrouter('prompt') result = extract_with_openrouter('prompt')
assert result == {'tripName': 'ok'} assert result == {'tripName': 'ok'}
assert calls == ['model-free-1', 'model-free-2'] # Retried model-free-1 up to the attempt limit before moving to model-free-2.
assert calls == ['model-free-1', 'model-free-1', 'model-free-2']
assert sleeps == [5, 10] # 5 * (attempt + 1) for attempts 0 and 1
def test_extract_with_openrouter_falls_back_to_paid_model_and_logs_sentry(monkeypatch): def test_extract_with_openrouter_moves_on_immediately_on_credit_exhausted(monkeypatch):
"""CreditExhaustedError shouldn't retry the same model — the quota won't recover mid-request."""
monkeypatch.setattr(extractor, 'FREE_MODELS', ['model-free-1', 'model-free-2'])
monkeypatch.setattr(extractor, 'FALLBACK_MODEL', 'model-paid')
calls = []
def _fake_call(model, prompt, content):
calls.append(model)
if model == 'model-free-1':
raise CreditExhaustedError('quota exhausted')
return {'tripName': 'ok'}
monkeypatch.setattr(extractor, '_call_openrouter', _fake_call)
result = extract_with_openrouter('prompt')
assert result == {'tripName': 'ok'}
assert calls == ['model-free-1', 'model-free-2'] # only one attempt on model-free-1
def test_extract_with_openrouter_retries_same_model_on_generic_error_before_moving_on(monkeypatch):
monkeypatch.setattr(extractor, 'FREE_MODELS', ['model-free-1']) monkeypatch.setattr(extractor, 'FREE_MODELS', ['model-free-1'])
monkeypatch.setattr(extractor, 'FALLBACK_MODEL', 'model-paid') monkeypatch.setattr(extractor, 'FALLBACK_MODEL', 'model-paid')
calls = []
def _fake_call(model, prompt, content): def _fake_call(model, prompt, content):
calls.append(model)
if model == 'model-free-1': if model == 'model-free-1':
raise Exception('exhausted') raise Exception('exhausted')
return {'tripName': 'from-fallback'} return {'tripName': 'from-fallback'}
@ -120,6 +194,7 @@ def test_extract_with_openrouter_falls_back_to_paid_model_and_logs_sentry(monkey
result = extract_with_openrouter('prompt') result = extract_with_openrouter('prompt')
assert result == {'tripName': 'from-fallback'} assert result == {'tripName': 'from-fallback'}
assert calls == ['model-free-1', 'model-free-1', 'model-paid']
assert sentry_calls == ['OpenRouter paid fallback used'] assert sentry_calls == ['OpenRouter paid fallback used']
@ -130,3 +205,30 @@ def test_extract_with_openrouter_raises_when_all_models_exhausted(monkeypatch):
with pytest.raises(ExtractorError): with pytest.raises(ExtractorError):
extract_with_openrouter('prompt') extract_with_openrouter('prompt')
def test_extract_with_openrouter_skips_free_models_when_disabled(monkeypatch):
"""
MVP launch toggle — OPENROUTER_USE_FREE_MODELS_FIRST=false must call
FALLBACK_MODEL directly and never touch FREE_MODELS at all, so a
dead/rate-limited free-tier lineup can't add latency or fail a job.
"""
monkeypatch.setattr(extractor, 'FREE_MODELS', ['model-free-1', 'model-free-2'])
monkeypatch.setattr(extractor, 'FALLBACK_MODEL', 'model-paid')
monkeypatch.setattr(extractor, 'USE_FREE_MODELS_FIRST', False)
calls = []
def _fake_call(model, prompt, content):
calls.append(model)
return {'tripName': 'ok'}
monkeypatch.setattr(extractor, '_call_openrouter', _fake_call)
sentry_calls = []
monkeypatch.setattr(extractor.sentry_sdk, 'capture_message', lambda msg, level=None: sentry_calls.append(msg))
result = extract_with_openrouter('prompt')
assert result == {'tripName': 'ok'}
assert calls == ['model-paid'] # free models never attempted
assert sentry_calls == [] # no "free models exhausted" warning — none were tried

212
backend/tests/core/test_proxy_service.py
View File

@ -1,31 +1,54 @@
import core.proxy_service as proxy_service_module
from core.proxy_service import ProxyService from core.proxy_service import ProxyService
def test_webshare_requests_config_with_country(): def test_webshare_requests_config_with_country():
service = ProxyService() service = ProxyService()
config = service._webshare_requests(country='gb') config = service._webshare_requests(country='gb')
assert '-GB-rotate' in config['http'] assert '-gb-rotate' in config['http']
assert config['http'] == config['https'] assert config['http'] == config['https']
def test_webshare_requests_config_lowercases_uppercase_country():
# competitors.primary_market is stored as an uppercase ISO code
# (CLAUDE.md's schema — "GB", "US", "AU"), but Webshare's rotating
# endpoint only recognises lowercase country codes in the username
# (help.webshare.io/en/articles/8596715 — examples are `-us`, `-gb`,
# `-ca`, never uppercase). An uppercase suffix here previously caused
# every proxied scrape to fail Webshare auth silently.
service = ProxyService()
config = service._webshare_requests(country='GB')
assert '-gb-rotate' in config['http']
assert '-GB-rotate' not in config['http']
def test_webshare_requests_config_without_country(): def test_webshare_requests_config_without_country():
service = ProxyService() service = ProxyService()
config = service._webshare_requests() config = service._webshare_requests()
assert '-rotate' in config['http'] assert '-rotate' in config['http']
assert '-GB-' not in config['http'] assert '-gb-' not in config['http']
def test_webshare_playwright_config(): def test_webshare_playwright_config():
service = ProxyService() service = ProxyService()
config = service._webshare_playwright(country='us') config = service._webshare_playwright(country='us')
assert config['server'] == 'http://p.webshare.io:80' assert config['server'] == 'http://p.webshare.io:80'
assert config['username'].endswith('-US-rotate') assert config['username'].endswith('-us-rotate')
def test_webshare_playwright_config_lowercases_uppercase_country():
service = ProxyService()
config = service._webshare_playwright(country='US')
assert config['username'].endswith('-us-rotate')
def test_brightdata_requests_and_playwright_configs(): def test_brightdata_requests_and_playwright_configs():
# Port 33335, not the commonly-documented 22225 — Bright Data's currently
# valid SSL cert requires 33335; 22225 is the deprecated cert's port,
# which produced a real net::ERR_CERT_AUTHORITY_INVALID on a live scrape.
service = ProxyService() service = ProxyService()
assert service._brightdata_requests()['http'].startswith('http://') assert service._brightdata_requests()['http'].startswith('http://')
assert service._brightdata_playwright()['server'] == 'http://brd.superproxy.io:22225' assert service._brightdata_playwright()['server'] == 'http://brd.superproxy.io:33335'
def test_get_proxy_for_domain_uses_webshare_when_no_override(monkeypatch): def test_get_proxy_for_domain_uses_webshare_when_no_override(monkeypatch):
@ -52,7 +75,7 @@ def test_get_playwright_proxy_for_domain_uses_brightdata_when_override_exists(mo
lambda domain: {'id': 'o1', 'provider': 'brightdata'} lambda domain: {'id': 'o1', 'provider': 'brightdata'}
) )
config = service.get_playwright_proxy_for_domain('flashpack.com') config = service.get_playwright_proxy_for_domain('flashpack.com')
assert config['server'] == 'http://brd.superproxy.io:22225' assert config['server'] == 'http://brd.superproxy.io:33335'
def test_escalate_to_brightdata_creates_new_override(monkeypatch): def test_escalate_to_brightdata_creates_new_override(monkeypatch):
@ -85,3 +108,182 @@ def test_escalate_to_brightdata_increments_existing_override(monkeypatch):
assert updated['id'] == 'o1' assert updated['id'] == 'o1'
assert updated['blocked_count'] == 3 assert updated['blocked_count'] == 3
assert updated['provider'] == 'brightdata' assert updated['provider'] == 'brightdata'
# ── is_domain_blocked_on_webshare ────────────────────────────────────────
def test_is_domain_blocked_on_webshare_true_when_brightdata_override(monkeypatch):
service = ProxyService()
monkeypatch.setattr(
'core.proxy_service.proxy_repository.get_by_domain',
lambda domain: {'id': 'o1', 'provider': 'brightdata'}
)
assert service.is_domain_blocked_on_webshare('flashpack.com') is True
def test_is_domain_blocked_on_webshare_false_when_no_override(monkeypatch):
service = ProxyService()
monkeypatch.setattr('core.proxy_service.proxy_repository.get_by_domain', lambda domain: None)
assert service.is_domain_blocked_on_webshare('intrepidtravel.com') is False
def test_is_domain_blocked_on_webshare_false_when_webshare_override(monkeypatch):
service = ProxyService()
monkeypatch.setattr(
'core.proxy_service.proxy_repository.get_by_domain',
lambda domain: {'id': 'o1', 'provider': 'webshare'}
)
assert service.is_domain_blocked_on_webshare('intrepidtravel.com') is False
# ── get_webshare_ip_candidates / _fetch_webshare_ip_list ─────────────────
class _FakeResponse:
def __init__(self, json_data, status_code=200):
self._json_data = json_data
self.status_code = status_code
def raise_for_status(self):
if self.status_code >= 400:
raise proxy_service_module.requests.HTTPError(f'{self.status_code}')
def json(self):
return self._json_data
def _fake_ip(suffix, valid=True, country='GB'):
return {
'id': f'd-{suffix}', 'username': f'user{suffix}', 'password': f'pass{suffix}',
'proxy_address': f'1.2.3.{suffix}', 'port': 8000 + suffix,
'valid': valid, 'country_code': country,
}
def test_get_webshare_ip_candidates_returns_empty_without_country():
service = ProxyService()
assert service.get_webshare_ip_candidates(None) == []
def test_fetch_webshare_ip_list_sends_uppercase_country_code(monkeypatch):
# Webshare's proxy-list *management* API wants uppercase country codes
# (their own docs example: "FR,US") — the opposite convention from the
# rotating-gateway username suffix, which wants lowercase. These are two
# separate Webshare systems; do not "fix" them to match each other.
service = ProxyService()
captured = {}
def fake_get(url, headers=None, params=None, timeout=None):
captured['params'] = params
return _FakeResponse({'results': [_fake_ip(1)], 'next': None})
monkeypatch.setattr(proxy_service_module.requests, 'get', fake_get)
service.get_webshare_ip_candidates('gb')
assert captured['params']['country_code__in'] == 'GB'
def test_get_webshare_ip_candidates_filters_valid_and_respects_limit(monkeypatch):
service = ProxyService()
ips = [_fake_ip(1), _fake_ip(2, valid=False), _fake_ip(3), _fake_ip(4), _fake_ip(5)]
monkeypatch.setattr(
proxy_service_module.requests, 'get',
lambda *a, **k: _FakeResponse({'results': ips, 'next': None})
)
candidates = service.get_webshare_ip_candidates('GB', limit=2)
assert len(candidates) == 2
# The invalid one (suffix 2) must never appear, since it's filtered
# before the random.sample() call — every candidate has real fields.
for c in candidates:
assert 'password' in c and c['password'] != 'pass2'
def test_get_webshare_ip_candidates_dict_shape_uses_per_ip_credentials(monkeypatch):
# Each Webshare list entry carries its OWN username/password/port —
# genuinely different from _webshare_playwright()'s single account-level
# rotating-gateway credentials at p.webshare.io:80.
service = ProxyService()
monkeypatch.setattr(
proxy_service_module.requests, 'get',
lambda *a, **k: _FakeResponse({'results': [_fake_ip(7)], 'next': None})
)
candidates = service.get_webshare_ip_candidates('GB', limit=1)
assert candidates == [{'server': 'http://1.2.3.7:8007', 'username': 'user7', 'password': 'pass7'}]
def test_get_webshare_ip_candidates_returns_empty_on_api_failure(monkeypatch):
service = ProxyService()
def raising_get(*a, **k):
raise proxy_service_module.requests.RequestException('connection refused')
monkeypatch.setattr(proxy_service_module.requests, 'get', raising_get)
assert service.get_webshare_ip_candidates('GB') == []
def test_get_webshare_ip_candidates_returns_empty_when_no_results_for_country(monkeypatch):
service = ProxyService()
monkeypatch.setattr(
proxy_service_module.requests, 'get',
lambda *a, **k: _FakeResponse({'results': [], 'next': None})
)
assert service.get_webshare_ip_candidates('GB') == []
def test_fetch_webshare_ip_list_caches_within_ttl(monkeypatch):
service = ProxyService()
call_count = {'n': 0}
def fake_get(*a, **k):
call_count['n'] += 1
return _FakeResponse({'results': [_fake_ip(1)], 'next': None})
monkeypatch.setattr(proxy_service_module.requests, 'get', fake_get)
monkeypatch.setattr(proxy_service_module.time, 'time', lambda: 1000.0)
service.get_webshare_ip_candidates('GB')
service.get_webshare_ip_candidates('GB')
assert call_count['n'] == 1 # second call served from cache
def test_fetch_webshare_ip_list_refetches_after_ttl_expires(monkeypatch):
service = ProxyService()
call_count = {'n': 0}
current_time = {'t': 1000.0}
def fake_get(*a, **k):
call_count['n'] += 1
return _FakeResponse({'results': [_fake_ip(1)], 'next': None})
monkeypatch.setattr(proxy_service_module.requests, 'get', fake_get)
monkeypatch.setattr(proxy_service_module.time, 'time', lambda: current_time['t'])
service.get_webshare_ip_candidates('GB')
current_time['t'] += proxy_service_module.WEBSHARE_IP_CACHE_TTL_SECONDS + 1
service.get_webshare_ip_candidates('GB')
assert call_count['n'] == 2
def test_fetch_webshare_ip_list_negative_cache_uses_shorter_ttl(monkeypatch):
service = ProxyService()
call_count = {'n': 0}
current_time = {'t': 1000.0}
def raising_get(*a, **k):
call_count['n'] += 1
raise proxy_service_module.requests.RequestException('down')
monkeypatch.setattr(proxy_service_module.requests, 'get', raising_get)
monkeypatch.setattr(proxy_service_module.time, 'time', lambda: current_time['t'])
service.get_webshare_ip_candidates('GB')
current_time['t'] += proxy_service_module.WEBSHARE_IP_CACHE_NEGATIVE_TTL_SECONDS + 1
service.get_webshare_ip_candidates('GB')
assert call_count['n'] == 2 # negative cache expired, retried

159
backend/tests/core/test_scraper.py
View File

@ -84,11 +84,22 @@ def test_check_robots_txt_respects_disallow(monkeypatch):
assert scraper.check_robots_txt('https://intrepidtravel.com/peru') is False assert scraper.check_robots_txt('https://intrepidtravel.com/peru') is False
def test_scrape_skips_when_robots_txt_disallows(monkeypatch): def test_scrape_proceeds_despite_robots_txt_disallow(monkeypatch):
"""
robots.txt disallow is logged only, not enforced — a deliberate
policy change from the original "skip on disallow" behavior (see
check_robots_txt's docstring). scrape() must still render the page.
"""
monkeypatch.setattr(scraper, 'check_robots_txt', lambda url: False) monkeypatch.setattr(scraper, 'check_robots_txt', lambda url: False)
async def _fake_render_page(url, country=None):
return {'text': 'x' * 2000, 'tables': '', 'url': url, 'char_count': 2000, 'success': True}
monkeypatch.setattr(scraper, 'render_page', _fake_render_page)
result = scraper.scrape('https://intrepidtravel.com/peru') result = scraper.scrape('https://intrepidtravel.com/peru')
assert result['success'] is False assert result['success'] is True
assert 'robots.txt' in result['text'] assert result['char_count'] == 2000
def test_scrape_returns_playwright_result_when_content_is_sufficient(monkeypatch): def test_scrape_returns_playwright_result_when_content_is_sufficient(monkeypatch):
@ -126,6 +137,8 @@ def test_scrape_falls_back_to_ai_fetch_when_char_count_low(monkeypatch):
def test_render_page_uses_webshare_proxy_when_not_blocked(monkeypatch): def test_render_page_uses_webshare_proxy_when_not_blocked(monkeypatch):
monkeypatch.setattr(scraper.proxy_service, 'is_domain_blocked_on_webshare', lambda domain: False)
monkeypatch.setattr(scraper.proxy_service, 'get_webshare_ip_candidates', lambda country=None: [])
monkeypatch.setattr( monkeypatch.setattr(
scraper.proxy_service, 'get_playwright_proxy_for_domain', scraper.proxy_service, 'get_playwright_proxy_for_domain',
lambda domain, country=None: {'server': 'webshare-proxy'} lambda domain, country=None: {'server': 'webshare-proxy'}
@ -148,6 +161,8 @@ def test_render_page_uses_webshare_proxy_when_not_blocked(monkeypatch):
def test_render_page_escalates_to_brightdata_when_blocked(monkeypatch): def test_render_page_escalates_to_brightdata_when_blocked(monkeypatch):
monkeypatch.setattr(scraper.proxy_service, 'is_domain_blocked_on_webshare', lambda domain: False)
monkeypatch.setattr(scraper.proxy_service, 'get_webshare_ip_candidates', lambda country=None: [])
monkeypatch.setattr( monkeypatch.setattr(
scraper.proxy_service, 'get_playwright_proxy_for_domain', scraper.proxy_service, 'get_playwright_proxy_for_domain',
lambda domain, country=None: {'server': 'webshare-proxy'} lambda domain, country=None: {'server': 'webshare-proxy'}
@ -173,6 +188,144 @@ def test_render_page_escalates_to_brightdata_when_blocked(monkeypatch):
assert call_log == [{'server': 'webshare-proxy'}, {'server': 'brightdata-proxy'}] assert call_log == [{'server': 'webshare-proxy'}, {'server': 'brightdata-proxy'}]
def test_render_page_returns_on_first_successful_specific_ip(monkeypatch):
monkeypatch.setattr(scraper.proxy_service, 'is_domain_blocked_on_webshare', lambda domain: False)
monkeypatch.setattr(
scraper.proxy_service, 'get_webshare_ip_candidates',
lambda country=None: [{'server': 'ip-1'}, {'server': 'ip-2'}]
)
gateway_called = []
monkeypatch.setattr(
scraper.proxy_service, 'get_playwright_proxy_for_domain',
lambda domain, country=None: gateway_called.append(domain) or {'server': 'webshare-gateway'}
)
escalated = []
monkeypatch.setattr(scraper.proxy_service, 'escalate_to_brightdata', lambda domain: escalated.append(domain))
calls = []
async def _fake_render_with_proxy(url, proxy):
calls.append(proxy)
return {'char_count': 5000, 'status_code': 200, 'text': 'good content', 'success': True, 'url': url}
monkeypatch.setattr(scraper, '_render_with_proxy', _fake_render_with_proxy)
result = _run_async(scraper.render_page('https://intrepidtravel.com/peru', country='GB'))
assert result['text'] == 'good content'
assert calls == [{'server': 'ip-1'}] # only the first candidate was tried
assert gateway_called == [] # gateway/Bright Data never reached
assert escalated == []
def test_render_page_tries_multiple_specific_ips_then_falls_back_to_rotate_gateway(monkeypatch):
monkeypatch.setattr(scraper.proxy_service, 'is_domain_blocked_on_webshare', lambda domain: False)
monkeypatch.setattr(
scraper.proxy_service, 'get_webshare_ip_candidates',
lambda country=None: [{'server': 'ip-1'}, {'server': 'ip-2'}, {'server': 'ip-3'}]
)
monkeypatch.setattr(
scraper.proxy_service, 'get_playwright_proxy_for_domain',
lambda domain, country=None: {'server': 'webshare-gateway'}
)
escalated = []
monkeypatch.setattr(scraper.proxy_service, 'escalate_to_brightdata', lambda domain: escalated.append(domain))
call_log = []
async def _fake_render_with_proxy(url, proxy):
call_log.append(proxy)
if proxy['server'] == 'webshare-gateway':
return {'char_count': 5000, 'status_code': 200, 'text': 'good content', 'success': True, 'url': url}
return {'char_count': 50, 'status_code': 403, 'text': 'blocked', 'success': True, 'url': url}
monkeypatch.setattr(scraper, '_render_with_proxy', _fake_render_with_proxy)
result = _run_async(scraper.render_page('https://intrepidtravel.com/peru', country='GB'))
assert result['text'] == 'good content'
assert call_log == [{'server': 'ip-1'}, {'server': 'ip-2'}, {'server': 'ip-3'}, {'server': 'webshare-gateway'}]
assert escalated == [] # gateway succeeded — never needed Bright Data
def test_render_page_falls_through_to_rotate_gateway_when_ip_candidates_empty(monkeypatch):
monkeypatch.setattr(scraper.proxy_service, 'is_domain_blocked_on_webshare', lambda domain: False)
monkeypatch.setattr(scraper.proxy_service, 'get_webshare_ip_candidates', lambda country=None: [])
monkeypatch.setattr(
scraper.proxy_service, 'get_playwright_proxy_for_domain',
lambda domain, country=None: {'server': 'webshare-gateway'}
)
calls = []
async def _fake_render_with_proxy(url, proxy):
calls.append(proxy)
return {'char_count': 5000, 'status_code': 200, 'text': 'good content', 'success': True, 'url': url}
monkeypatch.setattr(scraper, '_render_with_proxy', _fake_render_with_proxy)
monkeypatch.setattr(scraper.proxy_service, 'escalate_to_brightdata', lambda domain: None)
result = _run_async(scraper.render_page('https://intrepidtravel.com/peru', country='GB'))
assert result['text'] == 'good content'
assert calls == [{'server': 'webshare-gateway'}]
def test_render_page_skips_specific_ip_layer_when_domain_already_blocked(monkeypatch):
monkeypatch.setattr(scraper.proxy_service, 'is_domain_blocked_on_webshare', lambda domain: True)
candidates_called = []
monkeypatch.setattr(
scraper.proxy_service, 'get_webshare_ip_candidates',
lambda country=None: candidates_called.append(country) or []
)
monkeypatch.setattr(
scraper.proxy_service, 'get_playwright_proxy_for_domain',
lambda domain, country=None: {'server': 'brightdata-direct'}
)
async def _fake_render_with_proxy(url, proxy):
return {'char_count': 5000, 'status_code': 200, 'text': 'good content', 'success': True, 'url': url}
monkeypatch.setattr(scraper, '_render_with_proxy', _fake_render_with_proxy)
monkeypatch.setattr(scraper.proxy_service, 'escalate_to_brightdata', lambda domain: None)
result = _run_async(scraper.render_page('https://intrepidtravel.com/peru', country='GB'))
assert result['text'] == 'good content'
assert candidates_called == [] # get_webshare_ip_candidates never invoked
def test_scrape_falls_back_to_ai_fetch_when_render_fails_outright(monkeypatch):
"""
Regression test — the AI fallback used to require success=True,
which meant an outright Playwright failure (proxy error, cert
error, timeout — char_count=0, success=False) skipped the AI
fallback entirely, even though that's exactly the case it exists
for. Confirmed against a real ERR_CERT_AUTHORITY_INVALID failure.
"""
import sys
import types
monkeypatch.setattr(scraper, 'check_robots_txt', lambda url: True)
async def _fake_render_page(url, country=None):
return {
'text': 'Render failed: Page.goto: net::ERR_CERT_AUTHORITY_INVALID',
'tables': '', 'url': url, 'char_count': 0, 'success': False,
}
monkeypatch.setattr(scraper, 'render_page', _fake_render_page)
fake_ai_fetch = types.ModuleType('core.ai_fetch')
fake_ai_fetch.fetch_with_ai = lambda url: {
'text': 'ai-fetched content', 'tables': '', 'url': url, 'char_count': 2000, 'success': True
}
monkeypatch.setitem(sys.modules, 'core.ai_fetch', fake_ai_fetch)
result = scraper.scrape('https://intrepidtravel.com/peru')
assert result['text'] == 'ai-fetched content'
assert result['success'] is True
def test_scrape_keeps_playwright_result_when_ai_fetch_returns_none(monkeypatch): def test_scrape_keeps_playwright_result_when_ai_fetch_returns_none(monkeypatch):
import sys import sys
import types import types

373
backend/tests/services/test_analysis_service.py
View File

@ -16,6 +16,7 @@ from repositories.tenant_repository import tenant_repository
from repositories.alert_repository import alert_repository from repositories.alert_repository import alert_repository
from repositories.price_history_repository import price_history_repository from repositories.price_history_repository import price_history_repository
from repositories.scrape_repository import scrape_repository from repositories.scrape_repository import scrape_repository
from repositories.ngs_meta_repository import ngs_meta_repository
def _install_fake_module(monkeypatch, dotted_name, **attrs): def _install_fake_module(monkeypatch, dotted_name, **attrs):
@ -91,15 +92,68 @@ def test_analyse_from_cache_uses_latest_product(monkeypatch):
captured = {} captured = {}
_install_fake_module( _install_fake_module(
monkeypatch, 'core.extractor', monkeypatch, 'core.extractor',
extract_with_openrouter=lambda prompt, content: captured.update(prompt=prompt) or {'narrative': 'ok'} extract_with_openrouter=lambda prompt, content: captured.update(prompt=prompt) or {'comments': 'ok', 'relevancy': 4}
) )
result = analysis_service.analyse_from_cache(tenant['id'], competitor['id'], {'destination': 'Peru'}) result = analysis_service.analyse_from_cache(tenant['id'], competitor['id'], {'destination': 'Peru'})
assert result == {'narrative': 'ok'} assert result['tripName'] == 'Peru Classic'
assert result['comments'] == 'ok'
assert result['relevancy'] == 4
assert 'Peru Classic' in captured['prompt'] assert 'Peru Classic' in captured['prompt']
def test_analyse_from_cache_returns_structured_fields_resultstable_expects(monkeypatch):
"""
Regression test — a real "complete" job showed an empty-looking
results table because analyse_from_cache() returned only whatever
ad-hoc shape the LLM invented for a bare narrative prompt, never
the cached product's own tripName/price/duration fields that
ResultsTable.vue actually renders (item.result.tripName,
.majorityPrice/.gbpPrice, .duration, .comments).
"""
tenant, competitor = _make_tenant_and_competitor()
product_repository.create({
'tenant_id': tenant['id'], 'competitor_id': competitor['id'],
'trip_name': 'Inca Trail Classic', 'trip_code': 'INC15',
'duration_days': 15, 'majority_price_usd': 2780,
'price_low_usd': 2500, 'price_high_usd': 3100,
'gbp_price': 2190, 'last_seen': '2026-01-01',
})
_install_fake_module(
monkeypatch, 'core.extractor',
extract_with_openrouter=lambda prompt, content: {'comments': 'strong match', 'relevancy': 5}
)
result = analysis_service.analyse_from_cache(tenant['id'], competitor['id'], {'destination': 'Peru'})
assert result['tripName'] == 'Inca Trail Classic'
assert result['tripCode'] == 'INC15'
assert result['duration'] == 15
assert result['majorityPrice'] == 2780
assert result['priceLow'] == 2500
assert result['priceHigh'] == 3100
assert result['gbpPrice'] == 2190
assert result['comments'] == 'strong match'
assert result['relevancy'] == 5
def test_analyse_from_cache_falls_back_to_cached_relevancy_when_llm_omits_it(monkeypatch):
tenant, competitor = _make_tenant_and_competitor()
product_repository.create({
'tenant_id': tenant['id'], 'competitor_id': competitor['id'],
'trip_name': 'Peru Classic', 'relevancy': 3, 'last_seen': '2026-01-01',
})
_install_fake_module(
monkeypatch, 'core.extractor',
extract_with_openrouter=lambda prompt, content: {'comments': 'ok'} # no relevancy key
)
result = analysis_service.analyse_from_cache(tenant['id'], competitor['id'], {'destination': 'Peru'})
assert result['relevancy'] == 3
def test_run_competitor_analysis_takes_fast_path_when_fresh(monkeypatch): def test_run_competitor_analysis_takes_fast_path_when_fresh(monkeypatch):
fresh = (datetime.now(timezone.utc) - timedelta(hours=1)).isoformat() fresh = (datetime.now(timezone.utc) - timedelta(hours=1)).isoformat()
tenant, competitor = _make_tenant_and_competitor(change_detected=False, last_scraped=fresh) tenant, competitor = _make_tenant_and_competitor(change_detected=False, last_scraped=fresh)
@ -157,6 +211,104 @@ def test_run_competitor_analysis_takes_refresh_path_and_persists_product(monkeyp
assert alerts[0]['product_id'] == result['product']['id'] assert alerts[0]['product_id'] == result['product']['id']
def test_run_competitor_analysis_uses_confirmed_url_over_competitor_website(monkeypatch):
# Regression test — run_competitor_analysis() used to always scrape
# competitor['website'] unconditionally, silently discarding whatever
# trip-specific URL was actually matched/confirmed at UrlConfirmation.vue
# for this run. context['confirmed_url'] must win when present.
tenant, competitor = _make_tenant_and_competitor(change_detected=True)
captured_urls = []
def fake_scrape(url, country=None):
captured_urls.append(url)
return {'success': True, 'url': url, 'text': 'scraped content'}
_install_fake_module(monkeypatch, 'core.scraper', scrape=fake_scrape)
_install_fake_module(
monkeypatch, 'core.extractor',
extract_with_openrouter=lambda prompt, content: {'tripName': 'Peru Classic', 'majorityPrice': 2190}
)
_install_fake_module(monkeypatch, 'industries.adventure_travel.prompt', build_prompt=lambda *a, **k: '')
analysis_service.run_competitor_analysis(tenant['id'], competitor, {
'confirmed_url': 'https://intrepidtravel.com/peru-classic-15-days',
})
assert captured_urls == ['https://intrepidtravel.com/peru-classic-15-days']
assert captured_urls[0] != competitor['website']
def test_run_competitor_analysis_falls_back_to_website_without_confirmed_url(monkeypatch):
tenant, competitor = _make_tenant_and_competitor(change_detected=True)
captured_urls = []
def fake_scrape(url, country=None):
captured_urls.append(url)
return {'success': True, 'url': url, 'text': 'scraped content'}
_install_fake_module(monkeypatch, 'core.scraper', scrape=fake_scrape)
_install_fake_module(
monkeypatch, 'core.extractor',
extract_with_openrouter=lambda prompt, content: {'tripName': 'Peru Classic', 'majorityPrice': 2190}
)
_install_fake_module(monkeypatch, 'industries.adventure_travel.prompt', build_prompt=lambda *a, **k: '')
analysis_service.run_competitor_analysis(tenant['id'], competitor, {})
assert captured_urls == [competitor['website']]
def test_run_competitor_analysis_flags_generic_page_for_root_domain(monkeypatch):
tenant, competitor = _make_tenant_and_competitor(change_detected=True)
fake_page_data = {'success': True, 'url': 'https://intrepidtravel.com', 'text': 'homepage content'}
_install_fake_module(monkeypatch, 'core.scraper', scrape=lambda url, country=None: fake_page_data)
_install_fake_module(
monkeypatch, 'core.extractor',
extract_with_openrouter=lambda prompt, content: {'tripName': ''}
)
_install_fake_module(monkeypatch, 'industries.adventure_travel.prompt', build_prompt=lambda *a, **k: '')
result = analysis_service.run_competitor_analysis(tenant['id'], competitor, {
'confirmed_url': 'https://intrepidtravel.com',
})
assert result['is_generic_page'] is True
def test_run_competitor_analysis_no_generic_page_flag_for_trip_specific_url(monkeypatch):
tenant, competitor = _make_tenant_and_competitor(change_detected=True)
fake_page_data = {
'success': True, 'url': 'https://intrepidtravel.com/peru-classic-15-days', 'text': 'trip page content',
}
_install_fake_module(monkeypatch, 'core.scraper', scrape=lambda url, country=None: fake_page_data)
_install_fake_module(
monkeypatch, 'core.extractor',
extract_with_openrouter=lambda prompt, content: {'tripName': 'Peru Classic'}
)
_install_fake_module(monkeypatch, 'industries.adventure_travel.prompt', build_prompt=lambda *a, **k: '')
result = analysis_service.run_competitor_analysis(tenant['id'], competitor, {
'confirmed_url': 'https://intrepidtravel.com/peru-classic-15-days',
})
assert result['is_generic_page'] is False
def test_run_competitor_analysis_fast_path_flags_generic_page_from_cached_url(monkeypatch):
fresh = (datetime.now(timezone.utc) - timedelta(hours=1)).isoformat()
tenant, competitor = _make_tenant_and_competitor(change_detected=False, last_scraped=fresh)
product_repository.create({
'tenant_id': tenant['id'], 'competitor_id': competitor['id'],
'trip_name': '', 'url': 'https://intrepidtravel.com',
})
_install_fake_module(monkeypatch, 'core.extractor', extract_with_openrouter=lambda p, content: {'ok': True})
result = analysis_service.run_competitor_analysis(tenant['id'], competitor, {})
assert result['path'] == 'fast'
assert result['is_generic_page'] is True
def test_save_scrape_result_updates_existing_product_without_duplicating(monkeypatch): def test_save_scrape_result_updates_existing_product_without_duplicating(monkeypatch):
tenant, competitor = _make_tenant_and_competitor() tenant, competitor = _make_tenant_and_competitor()
existing = product_repository.create({ existing = product_repository.create({
@ -253,6 +405,164 @@ def test_save_scrape_result_price_raised_alert_wording(monkeypatch):
assert alerts[0]['change_percent'] > 0 assert alerts[0]['change_percent'] > 0
def test_save_scrape_result_persists_full_field_set():
"""
Regression test — _save_scrape_result()'s common_fields dict used to
silently drop service_level/group_size/meals/start_location/
end_location/departures/seasonality/start_days/target_audience/
hotels/activities/trip_code on write, even though every one of these
already has a products column and the LLM already returns them
(industries/adventure_travel/prompt.py's build_prompt() schema).
ResultsTable.vue's transposed sheet-layout grid needs every one of
these to render (frontend/src/config/resultFields.js).
"""
tenant, competitor = _make_tenant_and_competitor()
extracted = {
'tripName': 'Inca Trail Classic', 'tripCode': 'INC15',
'serviceLevel': 'Premium', 'groupSize': 12, 'duration': 15, 'meals': 14,
'startLocation': 'Lima', 'endLocation': 'Cusco', 'departures': 52,
'seasonality': 'Year-round', 'startDays': 'Saturday, Monday',
'targetAudience': 'Premium travellers', 'hotels': 'Comfortable hotels',
'activities': 'City tours, Machu Picchu', 'majorityPrice': 2780, 'duration_days': 15,
}
product = analysis_service._save_scrape_result(
tenant['id'], competitor, extracted, {'url': competitor['website']}
)
assert product['trip_code'] == 'INC15'
assert product['service_level'] == 'Premium'
assert product['group_size'] == 12
assert product['meals'] == 14
assert product['start_location'] == 'Lima'
assert product['end_location'] == 'Cusco'
assert product['departures'] == 52
assert product['seasonality'] == 'Year-round'
assert product['start_days'] == 'Saturday, Monday'
assert product['target_audience'] == 'Premium travellers'
assert product['hotels'] == 'Comfortable hotels'
assert product['activities'] == 'City tours, Machu Picchu'
def test_save_scrape_result_upserts_ngs_meta_when_ngs_fields_present():
tenant, competitor = _make_tenant_and_competitor()
extracted = {
'tripName': 'Nat Geo Signature', 'majorityPrice': 4500,
'exclusiveAccess': 'After-hours museum access',
'groupLeader': 'Local archaeologist guide',
'sustainability': 'Carbon-offset flights included',
}
product = analysis_service._save_scrape_result(
tenant['id'], competitor, extracted, {'url': competitor['website']}
)
meta = ngs_meta_repository.get_by_product_id(product['id'])
assert meta is not None
assert meta['exclusive_access'] == 'After-hours museum access'
assert meta['group_leader'] == 'Local archaeologist guide'
assert meta['sustainability'] == 'Carbon-offset flights included'
def test_save_scrape_result_updates_existing_ngs_meta_without_duplicating():
tenant, competitor = _make_tenant_and_competitor()
extracted = {'tripName': 'Nat Geo Signature', 'exclusiveAccess': 'Version 1'}
product = analysis_service._save_scrape_result(
tenant['id'], competitor, extracted, {'url': competitor['website']}
)
analysis_service._save_scrape_result(
tenant['id'], competitor,
{'tripName': 'Nat Geo Signature', 'exclusiveAccess': 'Version 2'},
{'url': competitor['website']},
)
meta = ngs_meta_repository.get_by_product_id(product['id'])
assert meta['exclusive_access'] == 'Version 2'
def test_save_scrape_result_skips_ngs_meta_for_standard_run():
tenant, competitor = _make_tenant_and_competitor()
extracted = {'tripName': 'Peru Classic', 'majorityPrice': 2190}
product = analysis_service._save_scrape_result(
tenant['id'], competitor, extracted, {'url': competitor['website']}
)
assert ngs_meta_repository.get_by_product_id(product['id']) is None
def test_analyse_from_cache_returns_full_field_set(monkeypatch):
tenant, competitor = _make_tenant_and_competitor()
product_repository.create({
'tenant_id': tenant['id'], 'competitor_id': competitor['id'],
'trip_name': 'Inca Trail Classic', 'url': 'https://intrepidtravel.com/inca-trail',
'service_level': 'Premium', 'group_size': 12, 'duration_days': 15, 'meals': 14,
'start_location': 'Lima', 'end_location': 'Cusco', 'departures': 52,
'seasonality': 'Year-round', 'start_days': 'Saturday', 'target_audience': 'Premium travellers',
'hotels': 'Comfortable hotels', 'activities': 'City tours', 'date_used': '2026-08-15',
'last_seen': '2026-01-01',
})
_install_fake_module(
monkeypatch, 'core.extractor',
extract_with_openrouter=lambda prompt, content: {'comments': 'ok', 'relevancy': 4}
)
result = analysis_service.analyse_from_cache(tenant['id'], competitor['id'], {'destination': 'Peru'})
assert result['link'] == 'https://intrepidtravel.com/inca-trail'
assert result['serviceLevel'] == 'Premium'
assert result['groupSize'] == 12
assert result['meals'] == 14
assert result['startLocation'] == 'Lima'
assert result['endLocation'] == 'Cusco'
assert result['departures'] == 52
assert result['seasonality'] == 'Year-round'
assert result['startDays'] == 'Saturday'
assert result['targetAudience'] == 'Premium travellers'
assert result['hotels'] == 'Comfortable hotels'
assert result['activities'] == 'City tours'
assert result['dateUsed'] == '2026-08-15'
def test_analyse_from_cache_includes_ngs_fields_when_tab_type_ngs(monkeypatch):
tenant, competitor = _make_tenant_and_competitor()
product = product_repository.create({
'tenant_id': tenant['id'], 'competitor_id': competitor['id'],
'trip_name': 'Nat Geo Signature', 'last_seen': '2026-01-01',
})
ngs_meta_repository.create({
'product_id': product['id'], 'exclusive_access': 'After-hours access',
'group_leader': 'Local expert', 'sustainability': 'Offset flights',
})
_install_fake_module(
monkeypatch, 'core.extractor',
extract_with_openrouter=lambda prompt, content: {'comments': 'ok', 'relevancy': 4}
)
result = analysis_service.analyse_from_cache(
tenant['id'], competitor['id'], {'destination': 'Peru', 'tab_type': 'NGS'}
)
assert result['exclusiveAccess'] == 'After-hours access'
assert result['groupLeader'] == 'Local expert'
assert result['sustainability'] == 'Offset flights'
def test_analyse_from_cache_skips_ngs_lookup_for_standard_tab(monkeypatch):
tenant, competitor = _make_tenant_and_competitor()
product_repository.create({
'tenant_id': tenant['id'], 'competitor_id': competitor['id'],
'trip_name': 'Peru Classic', 'last_seen': '2026-01-01',
})
_install_fake_module(
monkeypatch, 'core.extractor',
extract_with_openrouter=lambda prompt, content: {'comments': 'ok', 'relevancy': 4}
)
result = analysis_service.analyse_from_cache(
tenant['id'], competitor['id'], {'destination': 'Peru', 'tab_type': 'Standard'}
)
assert 'exclusiveAccess' not in result
def test_detect_change_true_on_first_scrape(): def test_detect_change_true_on_first_scrape():
_, competitor = _make_tenant_and_competitor() _, competitor = _make_tenant_and_competitor()
assert analysis_service.detect_change(competitor['id'], 'somehash') is True assert analysis_service.detect_change(competitor['id'], 'somehash') is True
@ -329,6 +639,65 @@ def test_run_competitor_analysis_refresh_path_skips_extraction_when_hash_unchang
assert updated['change_detected'] is False assert updated['change_detected'] is False
def test_force_refresh_bypasses_fast_path_gate_even_when_fresh(monkeypatch):
fresh = (datetime.now(timezone.utc) - timedelta(hours=1)).isoformat()
tenant, competitor = _make_tenant_and_competitor(change_detected=False, last_scraped=fresh)
product_repository.create({
'tenant_id': tenant['id'], 'competitor_id': competitor['id'], 'trip_name': 'Peru Classic',
})
fake_page_data = {'success': True, 'url': competitor['website'], 'text': 'fresh content from a real scrape'}
_install_fake_module(monkeypatch, 'core.scraper', scrape=lambda url, country=None: fake_page_data)
_install_fake_module(
monkeypatch, 'core.extractor',
extract_with_openrouter=lambda prompt, content: {'tripName': 'Peru Classic', 'majorityPrice': 2190}
)
_install_fake_module(monkeypatch, 'industries.adventure_travel.prompt', build_prompt=lambda *a, **k: '')
# This competitor looks fresh — needs_refresh() alone would take the
# fast path — but force_refresh must override that decision entirely.
result = analysis_service.run_competitor_analysis(tenant['id'], competitor, {'force_refresh': True})
assert result['path'] == 'refresh'
assert result.get('product') is not None # proves _save_scrape_result() ran — a real extraction, not cache reuse
def test_force_refresh_bypasses_unchanged_shortcut_even_when_hash_matches(monkeypatch):
tenant, competitor = _make_tenant_and_competitor(change_detected=False)
product_repository.create({
'tenant_id': tenant['id'], 'competitor_id': competitor['id'], 'trip_name': 'Peru Classic',
})
from core.scraper import compute_content_hash
fake_page_data = {'success': True, 'url': competitor['website'], 'text': 'unchanged content'}
_install_fake_module(monkeypatch, 'core.scraper', scrape=lambda url, country=None: fake_page_data)
prior_hash = compute_content_hash('unchanged content')
scrape_repository.create({
'tenant_id': tenant['id'], 'competitor_id': competitor['id'], 'triggered_by': 'cron',
'status': 'complete', 'content_hash': prior_hash, 'competitors_total': 1, 'competitors_done': 1,
})
calls_with_content = []
_install_fake_module(
monkeypatch, 'core.extractor',
extract_with_openrouter=lambda prompt, content: calls_with_content.append(content) or {
'tripName': 'Peru Classic', 'majorityPrice': 2190
}
)
_install_fake_module(monkeypatch, 'industries.adventure_travel.prompt', build_prompt=lambda *a, **k: '')
result = analysis_service.run_competitor_analysis(tenant['id'], competitor, {'force_refresh': True})
assert result['path'] == 'refresh'
assert not result.get('unchanged') # the cache-reuse shortcut was bypassed, not taken
assert calls_with_content == ['unchanged content'] # a REAL extraction ran with the full page text, not ''
updated = competitor_repository.get_by_id(competitor['id'])
# Bookkeeping (hash-confirmed-stable flag clear) still happens even
# though the cache-reuse decision itself was bypassed.
assert updated['change_detected'] is False
def test_change_detected_stays_visible_until_next_scrape_confirms_stable(monkeypatch): def test_change_detected_stays_visible_until_next_scrape_confirms_stable(monkeypatch):
""" """
Regression test for the bug where change_detected was set True by Regression test for the bug where change_detected was set True by

87
backend/tests/services/test_billing_service.py
View File

@ -81,3 +81,90 @@ def test_upgrade_checkout_rejects_unconfigured_tier(monkeypatch):
assert False, 'expected ValueError' assert False, 'expected ValueError'
except ValueError as e: except ValueError as e:
assert 'No Stripe price configured' in str(e) assert 'No Stripe price configured' in str(e)
class _FakeCheckoutSession:
url = 'https://checkout.stripe.com/fake-session'
def _fake_stripe(monkeypatch, captured):
"""
Stands in for services.billing_service._stripe() — captures the
kwargs checkout.Session.create() was called with instead of
hitting the real Stripe API, so the customer/customer_email
branching can be asserted directly.
"""
fake = type('_FakeStripe', (), {
'checkout': type('_Checkout', (), {
'Session': type('_Session', (), {
'create': staticmethod(lambda **kwargs: captured.update(kwargs) or _FakeCheckoutSession())
})
})
})
monkeypatch.setattr(billing_service, '_stripe', lambda: fake)
def test_upgrade_checkout_omits_customer_when_tenant_has_none(monkeypatch):
"""
Regression test — Stripe rejects an empty-string `customer` param
outright (raises InvalidRequestError), and every trial/manually-
onboarded tenant (CLAUDE.md §32) starts with stripe_customer_id=""
rather than unset. This must be omitted, not passed empty.
"""
tenant = tenant_repository.create({
'name': 'G Adventures', 'domain': 'gadventures.com', 'email': 'pm@gadventures.com',
'tier': 'analyse', 'status': 'trial', 'max_seats': 5, 'stripe_customer_id': '',
})
monkeypatch.setenv('STRIPE_PRICE_DISCOVER', 'price_123')
captured = {}
_fake_stripe(monkeypatch, captured)
url = billing_service.create_upgrade_checkout(tenant['id'], 'discover')
assert url == 'https://checkout.stripe.com/fake-session'
assert 'customer' not in captured
assert captured['customer_email'] == 'pm@gadventures.com'
def test_upgrade_checkout_uses_existing_customer_id_when_present(monkeypatch):
tenant = tenant_repository.create({
'name': 'G Adventures', 'domain': 'gadventures.com', 'email': 'pm@gadventures.com',
'tier': 'analyse', 'status': 'active', 'max_seats': 5, 'stripe_customer_id': 'cus_existing123',
})
monkeypatch.setenv('STRIPE_PRICE_DISCOVER', 'price_123')
captured = {}
_fake_stripe(monkeypatch, captured)
billing_service.create_upgrade_checkout(tenant['id'], 'discover')
assert captured['customer'] == 'cus_existing123'
assert 'customer_email' not in captured
def test_checkout_completed_captures_new_stripe_customer_id():
tenant = tenant_repository.create({
'name': 'G Adventures', 'domain': 'gadventures.com', 'tier': 'analyse',
'status': 'trial', 'max_seats': 5, 'stripe_customer_id': '',
})
billing_service.handle_checkout_completed({
'metadata': {'tenant_id': tenant['id'], 'target_tier': 'discover'},
'subscription': 'sub_123',
'customer': 'cus_new456',
})
assert tenant_repository.get_by_id(tenant['id'])['stripe_customer_id'] == 'cus_new456'
def test_checkout_completed_does_not_clobber_customer_id_when_event_lacks_one():
tenant = tenant_repository.create({
'name': 'G Adventures', 'domain': 'gadventures.com', 'tier': 'analyse',
'status': 'active', 'max_seats': 5, 'stripe_customer_id': 'cus_existing123',
})
billing_service.handle_checkout_completed({
'metadata': {'tenant_id': tenant['id'], 'target_tier': 'intelligence'},
'subscription': 'sub_123',
})
assert tenant_repository.get_by_id(tenant['id'])['stripe_customer_id'] == 'cus_existing123'

225
backend/tests/services/test_jobs.py
View File

@ -45,6 +45,105 @@ def test_run_research_job_marks_complete_on_success(monkeypatch):
assert gotify_calls == [] # fast completion, no slow/stuck alert assert gotify_calls == [] # fast completion, no slow/stuck alert
def test_run_research_job_threads_confirmed_url_per_competitor(monkeypatch):
# Regression test — entry['url'] (the URL confirmed at
# UrlConfirmation.vue, whether a Firecrawl match or a manual swap) used
# to be silently discarded; only entry['id'] was ever read.
tenant, competitor, run = _setup()
captured_contexts = []
monkeypatch.setattr(
'services.analysis_service.run_competitor_analysis',
lambda tenant_id, comp, ctx: captured_contexts.append(ctx) or {'path': 'refresh', 'result': {}}
)
monkeypatch.setattr('services.alert_service.send_gotify', lambda **kw: None)
jobs.run_research_job(run['id'], {
'tenant_id': tenant['id'],
'competitors': [{'id': competitor['id'], 'name': competitor['name'], 'url': 'https://intrepidtravel.com/peru-classic-15-days'}],
})
assert captured_contexts[0]['confirmed_url'] == 'https://intrepidtravel.com/peru-classic-15-days'
def test_run_research_job_confirmed_url_blank_normalizes_to_none(monkeypatch):
tenant, competitor, run = _setup()
captured_contexts = []
monkeypatch.setattr(
'services.analysis_service.run_competitor_analysis',
lambda tenant_id, comp, ctx: captured_contexts.append(ctx) or {'path': 'refresh', 'result': {}}
)
monkeypatch.setattr('services.alert_service.send_gotify', lambda **kw: None)
jobs.run_research_job(run['id'], {
'tenant_id': tenant['id'],
'competitors': [{'id': competitor['id'], 'name': competitor['name'], 'url': ' '}],
})
assert captured_contexts[0]['confirmed_url'] is None
def test_run_research_job_threads_force_refresh_into_context(monkeypatch):
# context is hand-built from `data` in run_research_job(), not `data`
# itself — a regression guard against a future refactor silently
# dropping a key, the same class of bug that would have hidden
# tab_type/product_name if unguarded.
tenant, competitor, run = _setup()
captured_contexts = []
monkeypatch.setattr(
'services.analysis_service.run_competitor_analysis',
lambda tenant_id, comp, ctx: captured_contexts.append(ctx) or {'path': 'refresh', 'result': {}}
)
monkeypatch.setattr('services.alert_service.send_gotify', lambda **kw: None)
jobs.run_research_job(run['id'], {
'tenant_id': tenant['id'], 'competitors': [{'id': competitor['id']}], 'force_refresh': True,
})
assert captured_contexts[0]['force_refresh'] is True
def test_run_research_job_threads_real_tenant_company_name_into_context(monkeypatch):
# Regression test — build_prompt() used to hardcode "G Adventures"
# regardless of which tenant was actually running the analysis.
# context['company_name'] must reflect this tenant's real name.
tenant = tenant_repository.create({
'name': 'Wild Horizons Travel', 'domain': 'wildhorizons.example', 'tier': 'analyse',
'status': 'active', 'max_seats': 5,
})
competitor = competitor_repository.create({
'tenant_id': tenant['id'], 'name': 'Intrepid Travel', 'website': 'https://intrepidtravel.com',
'catalogue_url': 'https://intrepidtravel.com/adventures', 'primary_market': 'GB', 'active': True,
})
run = scrape_repository.create({
'tenant_id': tenant['id'], 'triggered_by': 'manual', 'status': 'pending',
'competitors_total': 1, 'competitors_done': 0, 'results': {},
})
captured_contexts = []
monkeypatch.setattr(
'services.analysis_service.run_competitor_analysis',
lambda tenant_id, comp, ctx: captured_contexts.append(ctx) or {'path': 'refresh', 'result': {}}
)
monkeypatch.setattr('services.alert_service.send_gotify', lambda **kw: None)
jobs.run_research_job(run['id'], {'tenant_id': tenant['id'], 'competitors': [{'id': competitor['id']}]})
assert captured_contexts[0]['company_name'] == 'Wild Horizons Travel'
def test_run_research_job_defaults_force_refresh_to_false(monkeypatch):
tenant, competitor, run = _setup()
captured_contexts = []
monkeypatch.setattr(
'services.analysis_service.run_competitor_analysis',
lambda tenant_id, comp, ctx: captured_contexts.append(ctx) or {'path': 'fast', 'result': {}}
)
monkeypatch.setattr('services.alert_service.send_gotify', lambda **kw: None)
jobs.run_research_job(run['id'], {'tenant_id': tenant['id'], 'competitors': [{'id': competitor['id']}]})
assert captured_contexts[0]['force_refresh'] is False
def test_run_research_job_regenerates_battlecard_on_refresh_path_only(monkeypatch): def test_run_research_job_regenerates_battlecard_on_refresh_path_only(monkeypatch):
tenant, competitor, run = _setup() tenant, competitor, run = _setup()
battlecard_calls = [] battlecard_calls = []
@ -210,6 +309,132 @@ def test_run_research_job_handles_unknown_competitor_id(monkeypatch):
assert updated['results']['failed'][0]['error'] == 'Competitor not found' assert updated['results']['failed'][0]['error'] == 'Competitor not found'
def test_run_research_job_stops_early_when_cancel_requested_from_the_start(monkeypatch):
tenant, competitor, run = _setup()
scrape_repository.update(run['id'], {'cancel_requested': True})
calls = []
monkeypatch.setattr(
'services.analysis_service.run_competitor_analysis',
lambda *a, **k: calls.append(1) or {'path': 'fast', 'result': {}}
)
gotify_calls = []
monkeypatch.setattr('services.alert_service.send_gotify', lambda **kw: gotify_calls.append(kw))
jobs.run_research_job(run['id'], {'tenant_id': tenant['id'], 'competitors': [{'id': competitor['id']}]})
updated = scrape_repository.get_by_id(run['id'])
assert updated['status'] == 'cancelled'
assert calls == [] # never even attempted the one competitor
assert gotify_calls == [] # cancellation isn't a failure or slow/stuck condition
def test_run_research_job_cancellation_preserves_prior_progress(monkeypatch):
tenant = tenant_repository.create({
'name': 'G Adventures', 'domain': 'gadventures.com', 'tier': 'analyse',
'status': 'active', 'max_seats': 5,
})
first = competitor_repository.create({
'tenant_id': tenant['id'], 'name': 'Exodus', 'website': 'https://exodustravels.com',
'catalogue_url': 'https://exodustravels.com/trips', 'primary_market': 'GB', 'active': True,
})
second = competitor_repository.create({
'tenant_id': tenant['id'], 'name': 'Flash Pack', 'website': 'https://flashpack.com',
'catalogue_url': 'https://flashpack.com/adventures', 'primary_market': 'US', 'active': True,
})
run = scrape_repository.create({
'tenant_id': tenant['id'], 'triggered_by': 'manual', 'status': 'pending',
'competitors_total': 2, 'competitors_done': 0, 'results': {},
})
# Simulates the PM clicking Cancel while the first competitor is
# still being analysed — cancel_requested flips mid-job, and the
# checkpoint before the *second* competitor should catch it. Flips
# the flag directly (rather than via cancel_research_job(), which
# would need a real Redis connection to fetch the RQ job record) —
# cancel_research_job()'s own behavior is covered separately below.
def _fake_analysis(tenant_id, comp, ctx):
if comp['name'] == 'Exodus':
scrape_repository.update(run['id'], {'cancel_requested': True})
return {'path': 'fast', 'result': {}}
monkeypatch.setattr('services.analysis_service.run_competitor_analysis', _fake_analysis)
monkeypatch.setattr('services.alert_service.send_gotify', lambda **kw: None)
jobs.run_research_job(run['id'], {
'tenant_id': tenant['id'], 'competitors': [{'id': first['id']}, {'id': second['id']}]
})
updated = scrape_repository.get_by_id(run['id'])
assert updated['status'] == 'cancelled'
assert len(updated['results']['success']) == 1 # Exodus finished before cancellation landed
assert updated['results']['success'][0]['name'] == 'Exodus'
def test_cancel_research_job_returns_false_for_unknown_job_id():
assert jobs.cancel_research_job('does-not-exist') is False
def test_cancel_research_job_cancels_still_queued_job(monkeypatch):
tenant, competitor, run = _setup()
class _FakeJob:
cancelled = False
def get_status(self):
return 'queued'
def cancel(self):
self.cancelled = True
fake_job = _FakeJob()
monkeypatch.setattr(jobs.Job, 'fetch', lambda job_id, connection=None: fake_job)
result = jobs.cancel_research_job(run['id'])
assert result is True
assert fake_job.cancelled is True
updated = scrape_repository.get_by_id(run['id'])
assert updated['status'] == 'cancelled' # will never run at all — safe to set immediately
assert updated['cancel_requested'] is True
def test_cancel_research_job_sets_flag_when_already_running(monkeypatch):
tenant, competitor, run = _setup()
scrape_repository.update(run['id'], {'status': 'running'})
class _FakeJob:
def get_status(self):
return 'started' # already executing — RQ can't interrupt this
def cancel(self):
raise AssertionError('must not attempt to cancel a job already executing')
monkeypatch.setattr(jobs.Job, 'fetch', lambda job_id, connection=None: _FakeJob())
result = jobs.cancel_research_job(run['id'])
assert result is True
updated = scrape_repository.get_by_id(run['id'])
assert updated['cancel_requested'] is True
assert updated['status'] == 'running' # unchanged — the running job itself flips this at its next checkpoint
def test_cancel_research_job_sets_flag_when_rq_has_no_record(monkeypatch):
"""Job already finished (or was never queued) by the time cancel is requested — still a safe no-op."""
tenant, competitor, run = _setup()
def _raise_no_such_job(job_id, connection=None):
raise jobs.NoSuchJobError()
monkeypatch.setattr(jobs.Job, 'fetch', _raise_no_such_job)
result = jobs.cancel_research_job(run['id'])
assert result is True
updated = scrape_repository.get_by_id(run['id'])
assert updated['cancel_requested'] is True
def test_enqueue_research_job_returns_job_id(monkeypatch): def test_enqueue_research_job_returns_job_id(monkeypatch):
captured = {} captured = {}
monkeypatch.setattr( monkeypatch.setattr(

75
backend/tests/services/test_retention_service.py Normal file
View File

@ -0,0 +1,75 @@
"""
retention_service.run_weekly_cleanup() — CLAUDE.md §22 DATA_RETENTION.
Uses explicit ISO date strings (not the mocks' internal sequence
counters) since these tests are specifically about real date-based
filtering, matching the pattern test_api_extended.py already uses for
scrape_runs' started_at.
"""
from datetime import datetime, timedelta, timezone
from services import retention_service
from repositories.scrape_repository import scrape_repository
from repositories.alert_repository import alert_repository
from repositories.price_history_repository import price_history_repository
OLD = (datetime.now(timezone.utc) - timedelta(days=200)).isoformat()
RECENT = (datetime.now(timezone.utc) - timedelta(days=1)).isoformat()
def test_deletes_scrape_runs_older_than_90_days():
old_run = scrape_repository.create({'tenant_id': 't1', 'triggered_by': 'cron', 'status': 'complete', 'started_at': OLD})
recent_run = scrape_repository.create({'tenant_id': 't1', 'triggered_by': 'cron', 'status': 'complete', 'started_at': RECENT})
result = retention_service.run_weekly_cleanup()
assert result['scrape_runs_deleted'] == 1
assert scrape_repository.get_by_id(old_run['id']) is None
assert scrape_repository.get_by_id(recent_run['id']) is not None
def test_deletes_only_read_alerts_older_than_90_days():
old_read = alert_repository.create({
'tenant_id': 't1', 'competitor_id': 'c1', 'alert_type': 'new_product',
'message': 'x', 'read': True, 'created': OLD,
})
old_unread = alert_repository.create({
'tenant_id': 't1', 'competitor_id': 'c1', 'alert_type': 'new_product',
'message': 'x', 'read': False, 'created': OLD,
})
recent_read = alert_repository.create({
'tenant_id': 't1', 'competitor_id': 'c1', 'alert_type': 'new_product',
'message': 'x', 'read': True, 'created': RECENT,
})
result = retention_service.run_weekly_cleanup()
assert result['alerts_deleted'] == 1
assert alert_repository.get_by_id(old_read['id']) is None
assert alert_repository.get_by_id(old_unread['id']) is not None # unread never auto-deleted, regardless of age
assert alert_repository.get_by_id(recent_read['id']) is not None
def test_deletes_price_history_older_than_365_days():
very_old = (datetime.now(timezone.utc) - timedelta(days=400)).isoformat()
old_record = price_history_repository.create({'tenant_id': 't1', 'product_id': 'p1', 'scraped_at': very_old})
recent_record = price_history_repository.create({'tenant_id': 't1', 'product_id': 'p1', 'scraped_at': RECENT})
result = retention_service.run_weekly_cleanup()
assert result['price_history_deleted'] == 1
assert price_history_repository.list_for_tenant('t1') == [recent_record]
assert old_record not in price_history_repository.list_for_tenant('t1')
def test_price_history_at_200_days_survives_365_day_retention():
"""200-day-old price_history is past the 90-day scrape_runs/alerts window but under the 365-day price_history one."""
record = price_history_repository.create({'tenant_id': 't1', 'product_id': 'p1', 'scraped_at': OLD})
result = retention_service.run_weekly_cleanup()
assert result['price_history_deleted'] == 0
assert price_history_repository.list_for_tenant('t1') == [record]
def test_returns_zero_counts_when_nothing_to_clean():
result = retention_service.run_weekly_cleanup()
assert result == {'scrape_runs_deleted': 0, 'alerts_deleted': 0, 'price_history_deleted': 0}

34
backend/tests/services/test_trip_finder_service.py
View File

@ -32,6 +32,40 @@ def test_low_confidence_match_is_not_found():
assert result['found'] is False assert result['found'] is False
def test_wrong_destination_with_duration_and_category_match_is_not_found():
# Regression test — a Turkey URL previously scored 3/4 = 0.75 confidence
# against a Peru search purely from duration ('9') + category ('classic')
# matches, despite zero actual destination relevance. This is the exact
# bug that served a Turkey trip for a Peru search via the fast path in
# production. Destination match is now a hard requirement, not an
# additive bonus duration/category can compensate for.
urls = ['https://someexplorer.com/turkey-classic-cultural-tour-9-days']
result = trip_finder_service.score_and_rank_urls(urls, 'Peru', 9)
assert result == {'url': None, 'confidence': 0, 'found': False}
def test_bare_root_domain_url_is_never_a_candidate_even_with_destination_in_domain_name():
# Regression test — a bare root domain could otherwise score high
# enough to be a confident "found" match if the destination keyword
# happens to appear in the domain itself (e.g. a competitor literally
# named "peru-travel.com"), silently presenting a homepage as if it
# were a trip-specific page. Root-domain URLs are excluded before
# scoring, same as the destination-keyword gate.
urls = ['https://www.peru-travel.com/', 'https://www.peru-travel.com']
result = trip_finder_service.score_and_rank_urls(urls, 'Peru', 15)
assert result == {'url': None, 'confidence': 0, 'found': False}
def test_root_domain_gate_does_not_exclude_urls_with_a_real_path():
# Sanity check — the gate only excludes bare root domains, not
# legitimate trip pages that also happen to be on a destination-named
# domain.
urls = ['https://www.peru-travel.com/', 'https://www.peru-travel.com/classic-15-days']
result = trip_finder_service.score_and_rank_urls(urls, 'Peru', 15)
assert result['url'] == 'https://www.peru-travel.com/classic-15-days'
assert result['found'] is True
def test_find_competitor_trip_url_handles_firecrawl_failure_gracefully(monkeypatch): def test_find_competitor_trip_url_handles_firecrawl_failure_gracefully(monkeypatch):
import requests import requests

19
backend/tests/services/test_url_utils.py Normal file
View File

@ -0,0 +1,19 @@
from services.url_utils import is_generic_homepage
def test_bare_root_domain_is_generic():
assert is_generic_homepage('https://www.explore.co.uk/') is True
assert is_generic_homepage('https://www.explore.co.uk') is True
def test_root_domain_with_query_string_is_generic():
assert is_generic_homepage('https://www.explore.co.uk/?ref=homepage') is True
def test_trip_specific_path_is_not_generic():
assert is_generic_homepage('https://www.explore.co.uk/peru-classic-15-days') is False
def test_none_or_empty_url_is_not_generic():
assert is_generic_homepage(None) is False
assert is_generic_homepage('') is False

16
backend/tests/test_api.py
View File

@ -135,6 +135,22 @@ def test_research_enqueues_job_without_touching_real_redis(client, api_headers,
assert captured['job_id'] == response.json['job_id'] assert captured['job_id'] == response.json['job_id']
def test_research_persists_created_by_email(client, api_headers, monkeypatch):
import jobs
from repositories.scrape_repository import scrape_repository
tenant = _make_tenant()
monkeypatch.setattr(jobs, 'enqueue_research_job', lambda job_id, data: job_id)
response = client.post('/research', json={
'tenant_id': tenant['id'], 'destination': 'Peru', 'duration': 15,
'travel_style': 'Classic', 'tab_type': 'Standard', 'competitors': [],
'email': 'pm@gadventures.com',
}, headers=api_headers)
run = scrape_repository.get_by_id(response.json['job_id'])
assert run['created_by_email'] == 'pm@gadventures.com'
def test_research_persists_client_trip_from_trip_intent(client, api_headers, monkeypatch): def test_research_persists_client_trip_from_trip_intent(client, api_headers, monkeypatch):
from repositories.client_trips_repository import client_trips_repository from repositories.client_trips_repository import client_trips_repository
import jobs import jobs

230
backend/tests/test_api_extended.py
View File

@ -85,6 +85,45 @@ def test_get_my_account_returns_own_tenant(client, api_headers, monkeypatch):
assert response.json['name'] == 'G Adventures' assert response.json['name'] == 'G Adventures'
def test_update_timezone_requires_session(client, api_headers):
response = client.patch('/account/timezone', json={'timezone': 'Europe/London'}, headers=api_headers)
assert response.status_code == 401
def test_update_timezone_requires_field(client, api_headers, monkeypatch):
tenant = _make_tenant()
monkeypatch.setattr('api.auth_service.get_tenant_id_from_request', lambda req: tenant['id'])
response = client.patch('/account/timezone', json={}, headers=api_headers)
assert response.status_code == 400
assert response.json['code'] == 'missing_field'
def test_update_timezone_sets_when_empty(client, api_headers, monkeypatch):
tenant = _make_tenant()
monkeypatch.setattr('api.auth_service.get_tenant_id_from_request', lambda req: tenant['id'])
response = client.patch('/account/timezone', json={'timezone': 'Europe/London'}, headers=api_headers)
assert response.status_code == 200
assert response.json['timezone'] == 'Europe/London'
assert tenant_repository.get_by_id(tenant['id'])['timezone'] == 'Europe/London'
def test_update_timezone_never_overwrites_existing(client, api_headers, monkeypatch):
tenant = tenant_repository.create({
'name': 'G Adventures', 'domain': 'gadventures.com', 'tier': 'analyse',
'status': 'active', 'max_seats': 5, 'timezone': 'America/Toronto',
})
monkeypatch.setattr('api.auth_service.get_tenant_id_from_request', lambda req: tenant['id'])
response = client.patch('/account/timezone', json={'timezone': 'Europe/London'}, headers=api_headers)
assert response.status_code == 200
assert response.json['timezone'] == 'America/Toronto' # unchanged — first-write-wins
def test_update_competitor(client, api_headers): def test_update_competitor(client, api_headers):
tenant = _make_tenant() tenant = _make_tenant()
competitor = competitor_repository.create({ competitor = competitor_repository.create({
@ -176,20 +215,80 @@ def test_delete_account_requires_tenant_id(client, api_headers):
assert response.status_code == 400 assert response.status_code == 400
def test_delete_account_marks_inactive_and_deactivates_competitors(client, api_headers): def test_delete_account_unknown_tenant_404(client, api_headers):
response = client.post('/account/delete', json={'tenant_id': 'does-not-exist'}, headers=api_headers)
assert response.status_code == 404
def test_delete_account_erases_every_tenant_scoped_collection(client, api_headers, monkeypatch):
from repositories.product_repository import product_repository
from repositories.price_history_repository import price_history_repository
from repositories.alert_repository import alert_repository
from repositories.battlecard_repository import battlecard_repository
from repositories.comparable_repository import comparable_repository
from repositories.client_trips_repository import client_trips_repository
tenant = _make_tenant() tenant = _make_tenant()
competitor = competitor_repository.create({ competitor = competitor_repository.create({
'tenant_id': tenant['id'], 'name': 'Intrepid', 'website': 'https://intrepidtravel.com', 'tenant_id': tenant['id'], 'name': 'Intrepid', 'website': 'https://intrepidtravel.com',
'catalogue_url': 'https://intrepidtravel.com/adventures', 'primary_market': 'GB', 'active': True, 'catalogue_url': 'https://intrepidtravel.com/adventures', 'primary_market': 'GB', 'active': True,
}) })
seat_repository.create({'tenant_id': tenant['id'], 'email': 'pm@gadventures.com', 'active': True}) seat_repository.create({'tenant_id': tenant['id'], 'email': 'pm@gadventures.com', 'active': True})
product = product_repository.create({
'tenant_id': tenant['id'], 'competitor_id': competitor['id'], 'trip_name': 'Peru Classic',
})
price_history_repository.create({'tenant_id': tenant['id'], 'product_id': product['id'], 'majority_price_usd': 2190})
scrape_repository.create({
'tenant_id': tenant['id'], 'triggered_by': 'manual', 'status': 'complete',
'competitors_total': 1, 'competitors_done': 1,
})
alert = alert_repository.create({
'tenant_id': tenant['id'], 'competitor_id': competitor['id'], 'alert_type': 'new_product', 'message': 'x',
})
battlecard_repository.upsert(competitor['id'], {'tenant_id': tenant['id'], 'content': 'x'})
comparable_repository.create({
'tenant_id': tenant['id'], 'client_product': 'Inca Trail', 'competitor_product': product['id'],
'similarity_score': 0.9,
})
client_trips_repository.create({'tenant_id': tenant['id'], 'trip_name': 'Inca Trail'})
sent = []
monkeypatch.setattr('services.email_service.send_account_deleted_email', lambda email, name: sent.append(email))
response = client.post('/account/delete', json={'tenant_id': tenant['id']}, headers=api_headers) response = client.post('/account/delete', json={'tenant_id': tenant['id']}, headers=api_headers)
assert response.status_code == 200 assert response.status_code == 200
assert tenant_repository.get_by_id(tenant['id'])['status'] == 'inactive' assert tenant_repository.get_by_id(tenant['id']) is None # hard-deleted, not just deactivated
assert competitor_repository.get_by_id(competitor['id'])['active'] is False assert competitor_repository.get_by_id(competitor['id']) is None
assert seat_repository.list_for_tenant(tenant['id']) == [] assert seat_repository.list_for_tenant(tenant['id']) == []
assert product_repository.list_all_for_tenant(tenant['id']) == []
assert price_history_repository.list_for_tenant(tenant['id']) == []
assert scrape_repository.list_all_for_tenant(tenant['id']) == []
assert alert_repository.get_by_id(alert['id']) is None
assert battlecard_repository.list_for_tenant(tenant['id']) == []
assert comparable_repository.list_for_tenant(tenant['id'], include_dismissed=True) == []
assert client_trips_repository.list_for_tenant(tenant['id']) == []
assert sent == ['pm@gadventures.com']
def test_delete_account_side_effect_failures_dont_block_erasure(client, api_headers, monkeypatch):
"""Stripe cancellation and the confirmation email are side effects — a failure in either never blocks erasure."""
tenant = _make_tenant(stripe_subscription_id='sub_123')
monkeypatch.setattr(
'services.billing_service._stripe',
lambda: type('_S', (), {'Subscription': type('_Sub', (), {
'delete': staticmethod(lambda *a, **k: (_ for _ in ()).throw(RuntimeError('stripe down')))
})})()
)
monkeypatch.setattr(
'services.email_service.send_account_deleted_email',
lambda *a, **k: (_ for _ in ()).throw(RuntimeError('resend down'))
)
response = client.post('/account/delete', json={'tenant_id': tenant['id']}, headers=api_headers)
assert response.status_code == 200
assert tenant_repository.get_by_id(tenant['id']) is None
# ── Billing ──────────────────────────────────────────────────────── # ── Billing ────────────────────────────────────────────────────────
@ -267,6 +366,16 @@ def test_internal_daily_digest(client, api_headers, monkeypatch):
assert response.json['sent'] is True assert response.json['sent'] is True
def test_internal_data_cleanup(client, api_headers, monkeypatch):
monkeypatch.setattr(
'services.retention_service.run_weekly_cleanup',
lambda: {'scrape_runs_deleted': 3, 'alerts_deleted': 1, 'price_history_deleted': 0}
)
response = client.post('/internal/data-cleanup', headers=api_headers)
assert response.status_code == 200
assert response.json == {'scrape_runs_deleted': 3, 'alerts_deleted': 1, 'price_history_deleted': 0}
def test_internal_welcome_email_unknown_tenant_404(client, api_headers): def test_internal_welcome_email_unknown_tenant_404(client, api_headers):
response = client.post('/internal/welcome-email/does-not-exist', headers=api_headers) response = client.post('/internal/welcome-email/does-not-exist', headers=api_headers)
assert response.status_code == 404 assert response.status_code == 404
@ -517,6 +626,68 @@ def test_research_history_detail_unknown_404(client, api_headers):
assert response.status_code == 404 assert response.status_code == 404
def test_research_cancel_known_job(client, api_headers, monkeypatch):
tenant = _make_tenant()
run = scrape_repository.create({
'tenant_id': tenant['id'], 'triggered_by': 'manual', 'status': 'running',
'competitors_total': 2, 'competitors_done': 1, 'results': {},
})
calls = []
monkeypatch.setattr('jobs.cancel_research_job', lambda job_id: calls.append(job_id) or True)
response = client.post(f'/research/cancel/{run["id"]}', headers=api_headers)
assert response.status_code == 200
assert response.json['status'] == 'cancelling'
assert calls == [run['id']]
def test_research_cancel_unknown_job_404(client, api_headers, monkeypatch):
monkeypatch.setattr('jobs.cancel_research_job', lambda job_id: False)
response = client.post('/research/cancel/nope', headers=api_headers)
assert response.status_code == 404
# ── Rate limiting — CORS preflight exemption ───────────────────────
def test_options_preflight_never_rate_limited(client):
"""
Regression test — the dashboard's 3s status-poll loop tripped a 429
on the browser's own CORS preflight (OPTIONS) within under a minute:
each cross-origin GET also fires an OPTIONS first, doubling counted
requests against the 30/minute default limit. OPTIONS must never be
rate limited — it carries no X-API-Key and isn't a real request.
"""
for _ in range(50): # comfortably past the 30/minute default limit
response = client.options('/research/status/anything')
assert response.status_code != 429
def test_get_requests_still_rate_limited(client, api_headers):
"""
The OPTIONS exemption must not accidentally exempt real requests too.
Uses /research/history rather than /research/status — the latter now
has its own override_defaults=True limit (see api.py) specifically
because it's polled, so it wouldn't trip the blanket default here.
"""
responses = [client.get('/research/history', query_string={'email': 'pm@test.com'}, headers=api_headers)
for _ in range(35)]
assert any(r.status_code == 429 for r in responses)
def test_research_status_survives_a_full_poll_cycle_worth_of_requests(client, api_headers):
"""
Regression test — a real test job's 20-minute poll loop (3s interval,
analysis_store.js's POLL_TIMEOUT_ATTEMPTS=400) tripped the blanket
200/hour default well before the job even finished, because
research_status had no override. 250 requests here comfortably
exceeds the old 200/hour default and must still all succeed under
the route's own 3000/hour override.
"""
responses = [client.get('/research/status/nope', headers=api_headers) for _ in range(250)]
assert all(r.status_code == 404 for r in responses) # 404 (unknown job), never 429
def test_research_history_for_tenant(client, api_headers): def test_research_history_for_tenant(client, api_headers):
tenant = _make_tenant() tenant = _make_tenant()
scrape_repository.create({ scrape_repository.create({
@ -528,6 +699,59 @@ def test_research_history_for_tenant(client, api_headers):
assert len(response.json) == 1 assert len(response.json) == 1
def test_research_history_excludes_per_competitor_bookkeeping_rows(client, api_headers):
# Regression test — analysis_service.run_competitor_analysis()'s
# refresh path creates its own scrape_runs row PER COMPETITOR purely
# to persist a content_hash for next time (hardcoded
# triggered_by='cron', competitors_total=1, results={}), regardless of
# what triggered the parent job. Without filtering these out, "last 5
# runs" showed misleading 1-competitor entries with no restorable
# results instead of the real batch run.
tenant = _make_tenant()
real_run = scrape_repository.create({
'tenant_id': tenant['id'], 'triggered_by': 'manual', 'status': 'complete',
'competitors_total': 5, 'competitors_done': 5,
'results': {'success': [{'name': 'Intrepid'}], 'failed': []},
})
for _ in range(5):
scrape_repository.create({
'tenant_id': tenant['id'], 'triggered_by': 'cron', 'status': 'complete',
'competitors_total': 1, 'competitors_done': 1, 'results': {},
'content_hash': 'somehash',
})
response = client.get('/research/history', query_string={'email': 'pm@gadventures.com'}, headers=api_headers)
assert response.status_code == 200
assert len(response.json) == 1
assert response.json[0]['id'] == real_run['id']
def test_research_history_latest_excludes_per_competitor_bookkeeping_rows(client, api_headers):
tenant = _make_tenant()
real_run = scrape_repository.create({
'tenant_id': tenant['id'], 'triggered_by': 'manual', 'status': 'complete',
'competitors_total': 5, 'competitors_done': 5,
'results': {'success': [{'name': 'Intrepid'}], 'failed': []},
'started_at': '2026-06-01T00:00:00Z',
})
# Created after the real run, same as the actual per-competitor loop —
# this is exactly the case that could previously win the "latest
# complete" lookup used by the Apps Script sidebar's pull integration.
scrape_repository.create({
'tenant_id': tenant['id'], 'triggered_by': 'cron', 'status': 'complete',
'competitors_total': 1, 'competitors_done': 1, 'results': {},
'content_hash': 'somehash', 'started_at': '2026-06-01T00:05:00Z',
})
response = client.get(
'/research/history/latest', query_string={'email': 'pm@gadventures.com'}, headers=api_headers
)
assert response.status_code == 200
assert response.json['id'] == real_run['id']
def test_battlecards_route(client, api_headers): def test_battlecards_route(client, api_headers):
tenant = _make_tenant() tenant = _make_tenant()
competitor = competitor_repository.create({ competitor = competitor_repository.create({

50
backend/tests/test_app.py Normal file
View File

@ -0,0 +1,50 @@
"""
app.py's Sentry initialisation is conditional on SENTRY_DSN at module
import time — Python caches `sys.modules['app']`, so re-importing it
inside the same pytest process with a different env var wouldn't
actually re-run the conditional. Each case is verified in a fresh
subprocess instead.
"""
import os
import subprocess
import sys
def _run_import_check(env_overrides, assertion_snippet):
env = {
**os.environ,
'USE_MOCK_REPOSITORIES': 'true',
'API_SECRET_KEY': 'test-api-key',
'POCKETBASE_URL': 'http://pocketbase.invalid',
'REDIS_URL': 'redis://localhost:6379/0',
'RATELIMIT_STORAGE_URI': 'memory://',
'FIRECRAWL_URL': 'http://firecrawl.invalid',
**env_overrides,
}
backend_dir = os.path.join(os.path.dirname(__file__), '..')
result = subprocess.run(
[sys.executable, '-c', assertion_snippet],
cwd=backend_dir, env=env, capture_output=True, text=True, timeout=30,
)
assert result.returncode == 0, result.stderr
return result.stdout
def test_app_boots_without_sentry_dsn():
env = {k: v for k, v in os.environ.items() if k != 'SENTRY_DSN'}
env.pop('SENTRY_DSN', None)
out = _run_import_check(
{'SENTRY_DSN': ''},
"from app import app; import sentry_sdk; "
"print('client_none' if sentry_sdk.Hub.current.client is None else 'client_set')"
)
assert 'client_none' in out
def test_app_boots_with_sentry_dsn_set():
out = _run_import_check(
{'SENTRY_DSN': 'https://abc123@o0.ingest.sentry.io/0', 'FLASK_ENV': 'staging'},
"from app import app; import sentry_sdk; "
"print('client_none' if sentry_sdk.Hub.current.client is None else 'client_set')"
)
assert 'client_set' in out

22
backend/tests/test_industries_adventure_travel.py
View File

@ -20,6 +20,28 @@ def test_build_prompt_embeds_page_content_and_context():
assert '"exclusiveAccess"' not in result assert '"exclusiveAccess"' not in result
def test_build_prompt_uses_real_company_name_not_hardcoded_g_adventures():
# Regression test — build_prompt() hardcoded "G Adventures" (the
# hackathon-origin reference client) into every tenant's prompt,
# regardless of who was actually running the analysis.
page_data = {'text': 'content', 'tables': '', 'url': 'https://example.com'}
result = prompt.build_prompt(
'Competitor', page_data, 'Peru Trek', 'Peru', 10, 'Classic',
company_name='Wild Horizons Travel',
)
assert 'Wild Horizons Travel' in result
assert 'G Adventures' not in result
def test_build_prompt_falls_back_to_generic_placeholder_without_company_name():
# Covers the dry-run /preview-prompt route, which has no tenant
# context to draw a real company name from.
page_data = {'text': 'content', 'tables': '', 'url': 'https://example.com'}
result = prompt.build_prompt('Competitor', page_data, 'Peru Trek', 'Peru', 10, 'Classic')
assert 'the client' in result
assert 'G Adventures' not in result
def test_build_prompt_includes_ngs_fields_when_flagged(): def test_build_prompt_includes_ngs_fields_when_flagged():
page_data = {'text': 'content', 'tables': '', 'url': 'https://example.com'} page_data = {'text': 'content', 'tables': '', 'url': 'https://example.com'}
result = prompt.build_prompt('Competitor', page_data, 'Product', 'Peru', 10, 'NGS', is_ngs=True) result = prompt.build_prompt('Competitor', page_data, 'Product', 'Peru', 10, 'NGS', is_ngs=True)

125
docker-compose.yml
View File

@ -61,16 +61,94 @@ services:
# ── Opt-in only — real self-hosted Firecrawl is a 3-service stack # ── Opt-in only — real self-hosted Firecrawl is a 3-service stack
# built from source (no stable published image — see CLAUDE.md §4's # built from source (no stable published image — see CLAUDE.md §4's
# "Firecrawl — real self-hosted stack" note). The build is slow # "Firecrawl — real self-hosted stack" note). Builds from a local
# (playwright-service alone pulls in Chromium + fonts), and Firecrawl # clone at ./firecrawl (`git clone https://github.com/firecrawl/
# is only used for on-demand /v1/map trip-intent matching — not # firecrawl.git`) rather than a remote git build context — this
# required for viewing dashboard pages. Start explicitly with # environment's network couldn't reliably clone the repo itself
# `docker compose --profile firecrawl up -d` when trip-intent # (git's smart-HTTP protocol specifically; plain HTTPS to GitHub's
# matching needs to be exercised locally. # API and to PyPI both worked fine, so this was not general
# bandwidth throttling). Re-point back to the GitHub URL below if
# building somewhere with normal git connectivity, and drop the
# local clone from the repo (it's gitignored, not committed).
#
# Build context is the app subdirectory itself (./firecrawl/apps/api),
# not the repo root — apps/api/Dockerfile's COPY instructions (e.g.
# `COPY sharedLibs/go-html-to-md ...`) are written relative to
# apps/api/, matching upstream firecrawl's own docker-compose.yaml
# (`build: apps/api`). Pointing context at the repo root with
# `dockerfile: apps/api/Dockerfile` resolves the Dockerfile from the
# right place but still resolves every COPY inside it against the
# repo root, not apps/api/ — breaks on the first COPY that assumes
# otherwise.
# Firecrawl's NUQ queue backend needs a real Postgres, not just Redis —
# discovered only once firecrawl-api actually started: without
# NUQ_DATABASE_URL set, harness.ts's setupNuqPostgres() tries to
# auto-provision a Postgres container by shelling out to `docker`/
# `podman` from *inside* the firecrawl-api container itself, which
# obviously has neither installed and isn't meant to (no
# Docker-in-Docker here). Firecrawl ships a purpose-built Postgres
# image for this at apps/nuq-postgres/ — pg_cron preloaded plus the
# NUQ schema (nuq.sql) baked in as an initdb script — build that
# rather than pointing at a vanilla postgres image, which would boot
# without the schema NUQ expects.
#
# Setting POSTGRES_HOST to this container's name (anything other
# than "localhost") is what flips setupNuqPostgres() into its
# docker-compose branch, which just builds the connection URL
# directly and skips the Docker/Podman auto-provision path entirely
# — this is the code path Firecrawl's own authors intended for
# exactly this setup, not a workaround.
#
# NUQ_RABBITMQ_URL turned out not to be optional in practice, despite
# most call sites gracefully checking `if (config.NUQ_RABBITMQ_URL)`
# and falling back to polling mode when absent. The harness always
# spawns an extract-worker subprocess (dist/src/services/extract-worker.js)
# regardless of which routes are actually used, and that subprocess's
# consumeExtractJobs() throws immediately without it — no config
# check gates whether it starts. The harness treats any subprocess
# exit as fatal to the whole process, so without RabbitMQ the entire
# firecrawl-api container crash-looped every ~2s even though
# Bettersight never calls /extract. Upstream's own docker-compose.yaml
# confirms this isn't optional either — it lists rabbitmq as a hard
# `depends_on: condition: service_healthy` for the api service. Stock
# image, no build needed.
firecrawl-rabbitmq:
image: rabbitmq:3-management
container_name: firecrawl-rabbitmq
profiles: ["firecrawl"]
networks:
- bettersight_default
restart: unless-stopped
firecrawl-nuq-postgres:
build:
context: ./firecrawl/apps/nuq-postgres
container_name: firecrawl-nuq-postgres
profiles: ["firecrawl"]
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
- POSTGRES_DB=postgres
networks:
- bettersight_default
restart: unless-stopped
# No separate firecrawl-worker service — checked upstream's own
# docker-compose.yaml (firecrawl/docker-compose.yaml) and it defines
# only one `api` container, command `node dist/src/harness.js
# --start-docker`. That harness process spins up the queue workers
# internally (NUQ_WORKER_COUNT etc — see apps/api/src/harness.ts)
# rather than running as a separate process. A prior version of this
# file ran a second container on `pnpm run workers`, which crash
# looped: the runtime image only carries node_modules/dist/native
# (no package.json — see the Dockerfile's runtime stage), so pnpm
# had nothing to run against. This matches the older split-process
# Firecrawl architecture CLAUDE.md's §4 was written against, not the
# current one — do not reintroduce a second container here.
firecrawl-api: firecrawl-api:
build: build:
context: https://github.com/firecrawl/firecrawl.git#main context: ./firecrawl/apps/api
dockerfile: apps/api/Dockerfile # context: https://github.com/firecrawl/firecrawl.git#main&context=apps/api
container_name: firecrawl-api container_name: firecrawl-api
profiles: ["firecrawl"] profiles: ["firecrawl"]
environment: environment:
@ -81,36 +159,25 @@ services:
- PLAYWRIGHT_MICROSERVICE_URL=http://firecrawl-playwright:3000/scrape - PLAYWRIGHT_MICROSERVICE_URL=http://firecrawl-playwright:3000/scrape
- USE_DB_AUTHENTICATION=false - USE_DB_AUTHENTICATION=false
- NUM_WORKERS_PER_QUEUE=8 - NUM_WORKERS_PER_QUEUE=8
- POSTGRES_HOST=firecrawl-nuq-postgres
- POSTGRES_PORT=5432
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
- POSTGRES_DB=postgres
- NUQ_RABBITMQ_URL=amqp://firecrawl-rabbitmq:5672
depends_on: depends_on:
- bettersight-redis - bettersight-redis
- firecrawl-playwright - firecrawl-playwright
networks: - firecrawl-nuq-postgres
- bettersight_default - firecrawl-rabbitmq
restart: unless-stopped
firecrawl-worker:
build:
context: https://github.com/firecrawl/firecrawl.git#main
dockerfile: apps/api/Dockerfile
container_name: firecrawl-worker
profiles: ["firecrawl"]
command: pnpm run workers
environment:
- REDIS_URL=redis://bettersight-redis:6379
- REDIS_RATE_LIMIT_URL=redis://bettersight-redis:6379
- PLAYWRIGHT_MICROSERVICE_URL=http://firecrawl-playwright:3000/scrape
- USE_DB_AUTHENTICATION=false
depends_on:
- bettersight-redis
- firecrawl-playwright
networks: networks:
- bettersight_default - bettersight_default
restart: unless-stopped restart: unless-stopped
firecrawl-playwright: firecrawl-playwright:
build: build:
context: https://github.com/firecrawl/firecrawl.git#main context: ./firecrawl/apps/playwright-service-ts
dockerfile: apps/playwright-service-ts/Dockerfile # context: https://github.com/firecrawl/firecrawl.git#main&context=apps/playwright-service-ts
container_name: firecrawl-playwright container_name: firecrawl-playwright
profiles: ["firecrawl"] profiles: ["firecrawl"]
environment: environment:

14
frontend/src/assets/css/main.css
View File

@ -175,9 +175,21 @@ body {
.cg-right { display: flex; flex-direction: column; gap: 18px; } .cg-right { display: flex; flex-direction: column; gap: 18px; }
/* ─── CARD ────────────────────────────────────────────────────── */ /* ─── CARD ────────────────────────────────────────────────────── */
.card { background: var(--surface); border-radius: var(--card-radius); box-shadow: var(--card-shadow); overflow: hidden; transition: box-shadow .2s; } /* min-width: 0 is required here, not cosmetic — a .card is a flex item
inside .cg-left (display:flex;flex-direction:column). Flex items default
to min-width:auto, which lets their content's intrinsic width (e.g. a
wide table) grow the card itself past its container instead of clipping
— so a nested overflow-x:auto wrapper never gets a chance to show a
scrollbar, because the card just expands and the whole page scrolls
instead. This is what silently broke horizontal scrolling in every
table/grid card (ResultsTable.vue, SeatManager.vue, CompetitorsPage.vue). */
.card { background: var(--surface); border-radius: var(--card-radius); box-shadow: var(--card-shadow); overflow: hidden; transition: box-shadow .2s; min-width: 0; }
.card:hover { box-shadow: var(--card-hover); } .card:hover { box-shadow: var(--card-hover); }
/* Shared wrapper for any table/grid wider than its card — apply around
<table> (or similar) whenever content can exceed the card's width. */
.tbl-scroll { overflow-x: auto; }
.ch { padding: 16px 20px; border-bottom: 1px solid var(--border); display: flex; align-items: center; justify-content: space-between; } .ch { padding: 16px 20px; border-bottom: 1px solid var(--border); display: flex; align-items: center; justify-content: space-between; }
.ct { font-size: 14px; font-weight: 800; letter-spacing: -.3px; color: var(--t1); } .ct { font-size: 14px; font-weight: 800; letter-spacing: -.3px; color: var(--t1); }
.cs { font-size: 12px; color: var(--t3); margin-top: 2px; } .cs { font-size: 12px; color: var(--t3); margin-top: 2px; }

26
frontend/src/assets/css/tour.css
View File

@ -1,5 +1,21 @@
/* Custom Shepherd.js styles matching the Bettersight design system — /* Custom Shepherd.js styles matching the Bettersight design system —
ported from CLAUDE.md §21 verbatim. */ ported from CLAUDE.md §21 verbatim, plus two fixes:
1. Shepherd's own base stylesheet caps the OUTER positioned element at
max-width:400px (.shepherd-element), while this file only ever
constrained the INNER .shepherd-content to 320px — the outer box
could render wider than its visible content, showing as blank space
on one side of the card. Matching the outer element's max-width to
the inner content's fixes it.
2. None of the 5 tour steps pass a `title`, but cancelIcon.enabled:true
(useOnboardingTour.js) makes Shepherd still render a full header bar
for just the × button — heavier than the plain bordered-card mockups
in CLAUDE.md §21. Tightened rather than removed, since steps 2-4 have
no other way to abandon the tour early besides repeated Back clicks. */
.bs-tour-step.shepherd-element {
max-width: 320px;
}
.bs-tour-step .shepherd-content { .bs-tour-step .shepherd-content {
border-radius: 14px; border-radius: 14px;
box-shadow: var(--card-hover); box-shadow: var(--card-hover);
@ -8,6 +24,14 @@
max-width: 320px; max-width: 320px;
} }
.bs-tour-step .shepherd-header {
padding: 8px 8px 0 0;
}
.bs-tour-step .shepherd-cancel-icon {
font-size: 1.4em;
}
.bs-tour-step .shepherd-text { .bs-tour-step .shepherd-text {
font-size: 13.5px; font-size: 13.5px;
line-height: 1.6; line-height: 1.6;

31
frontend/src/components/analyse/JobProgress.vue
View File

@ -10,12 +10,12 @@ const FAILURE_MESSAGES = {
} }
const props = defineProps({ const props = defineProps({
status: { type: String, required: true }, // 'queued' | 'running' | 'complete' | 'failed' status: { type: String, required: true }, // 'queued' | 'running' | 'cancelling' | 'complete' | 'failed' | 'cancelled'
progress: { type: Object, required: true }, progress: { type: Object, required: true },
results: { type: Object, default: null }, results: { type: Object, default: null },
error: { type: String, default: null } error: { type: String, default: null }
}) })
const emit = defineEmits(['retry', 'skip']) const emit = defineEmits(['retry', 'skip', 'cancel'])
function failureMessage(errorText) { function failureMessage(errorText) {
const key = Object.keys(FAILURE_MESSAGES).find((k) => errorText?.toLowerCase().includes(k.replace('_', ' '))) const key = Object.keys(FAILURE_MESSAGES).find((k) => errorText?.toLowerCase().includes(k.replace('_', ' ')))
@ -29,16 +29,22 @@ function failureMessage(errorText) {
<div class="ct"> <div class="ct">
<template v-if="status === 'queued'">Queued…</template> <template v-if="status === 'queued'">Queued…</template>
<template v-else-if="status === 'running'">Analysing competitors…</template> <template v-else-if="status === 'running'">Analysing competitors…</template>
<template v-else-if="status === 'cancelling'">Cancelling…</template>
<template v-else-if="status === 'complete'">Analysis complete</template> <template v-else-if="status === 'complete'">Analysis complete</template>
<template v-else>Analysis failed</template> <template v-else-if="status === 'cancelled'">Analysis cancelled</template>
<template v-else-if="status === 'failed'">Analysis failed</template>
<template v-else>Analysis starting…</template>
</div> </div>
<div v-if="status === 'running' || status === 'queued'" class="cs"> <div v-if="status === 'running' || status === 'queued' || status === 'cancelling'" class="cs">
{{ progress.done }} of {{ progress.total }} done {{ progress.done }} of {{ progress.total }} done
</div> </div>
</div> </div>
<div v-if="status === 'queued'" style="padding:32px;text-align:center;color:var(--t3);font-size:13px;"> <div v-if="status === 'queued'" style="padding:32px;text-align:center;color:var(--t3);font-size:13px;">
Queued — waiting for a worker to pick this job up… Queued — waiting for a worker to pick this job up…
<div style="margin-top:14px;">
<UButton size="xs" color="neutral" variant="ghost" @click="emit('cancel')">Cancel</UButton>
</div>
</div> </div>
<div v-else-if="status === 'running'" class="pr"> <div v-else-if="status === 'running'" class="pr">
@ -47,6 +53,23 @@ function failureMessage(errorText) {
<div class="pr-track"><div class="pr-fill blue" :style="{ width: `${(progress.done / Math.max(progress.total, 1)) * 100}%` }" /></div> <div class="pr-track"><div class="pr-fill blue" :style="{ width: `${(progress.done / Math.max(progress.total, 1)) * 100}%` }" /></div>
<div class="pr-val">{{ progress.done }}/{{ progress.total }}</div> <div class="pr-val">{{ progress.done }}/{{ progress.total }}</div>
</div> </div>
<UButton size="xs" color="neutral" variant="ghost" style="margin-left:12px;" @click="emit('cancel')">Cancel</UButton>
</div>
<div v-else-if="status === 'cancelling'" style="padding:32px;text-align:center;color:var(--t3);font-size:13px;">
Stopping — finishing the competitor currently in progress, then it will stop.
</div>
<div v-else-if="status === 'cancelled'" style="padding:24px 20px;">
<div style="display:flex;align-items:center;gap:8px;margin-bottom:6px;">
<span style="color:var(--t3);">○</span>
<strong style="font-size:13px;">You cancelled this analysis</strong>
</div>
<p style="font-size:12.5px;color:var(--t3);margin:0 0 10px;">
{{ progress.done }} of {{ progress.total }} competitors finished before it stopped
— results for those are saved below.
</p>
<UButton size="xs" color="primary" variant="soft" @click="emit('retry')">↺ Run again</UButton>
</div> </div>
<template v-else-if="status === 'complete' && results"> <template v-else-if="status === 'complete' && results">

79
frontend/src/components/analyse/ResultsTable.vue
View File

@ -1,49 +1,64 @@
<script setup> <script setup>
// Multi-currency display rule (CLAUDE.md §6 AnalysePage spec): show // Transposed comparison grid mirroring the legacy Google Sheet's layout —
// the competitor's own primary-market currency as the hero price, // one row per field, one column per competitor — see
// USD always alongside in muted secondary text, never converted. // frontend/src/config/resultFields.js for the row order/labels, verified
const CURRENCY_MAP = { // directly against the real completed run in "G Compete App V1.xlsx".
GB: { field: 'gbpPrice', symbol: '£' }, import { computed } from 'vue'
AU: { field: 'audPrice', symbol: 'A$' }, import { getRowsForTabType, rawValue, formatForDisplay } from '../../config/resultFields'
CA: { field: 'cadPrice', symbol: 'C$' },
DE: { field: 'eurPrice', symbol: '€' }, FR: { field: 'eurPrice', symbol: '€' },
ES: { field: 'eurPrice', symbol: '€' }, IT: { field: 'eurPrice', symbol: '€' }, NL: { field: 'eurPrice', symbol: '€' }
}
defineProps({ const props = defineProps({
results: { type: Array, required: true } // [{ competitor_id, name, path, result, product }] results: { type: Array, required: true }, // [{ competitor_id, name, path, result, product, is_generic_page }]
tabType: { type: String, default: 'Standard' }
}) })
function formatPrice(extracted, primaryMarket) { const rows = computed(() => getRowsForTabType(props.tabType))
if (!extracted) return '—'
const usd = extracted.majorityPrice ?? extracted.majority_price_usd
const mapping = CURRENCY_MAP[primaryMarket]
const heroValue = mapping ? extracted[mapping.field] : null
if (heroValue != null) {
const usdText = usd != null ? ` $${Number(usd).toLocaleString()} USD` : ''
return `${mapping.symbol}${Number(heroValue).toLocaleString()}${usdText}`
}
return usd != null ? `$${Number(usd).toLocaleString()} USD` : '—'
}
</script> </script>
<template> <template>
<div class="card"> <div class="card">
<div class="ch"><div class="ct">Results</div></div> <div class="ch"><div class="ct">Results</div></div>
<table class="tbl"> <div class="results-scroll">
<table class="tbl results-grid">
<thead> <thead>
<tr><th style="padding-left:20px;">Competitor</th><th>Trip</th><th>Price</th><th>Duration</th><th>Comments</th></tr> <tr>
<th class="label-col">Field</th>
<th v-for="item in results" :key="item.competitor_id">
{{ item.name }}
<span
v-if="item.is_generic_page"
class="badge bg-amber"
title="Scraped page is the competitor's homepage, not a trip-specific page — treat this result with caution"
>⚠ Generic page</span>
</th>
</tr>
</thead> </thead>
<tbody> <tbody>
<tr v-for="item in results" :key="item.competitor_id"> <tr v-for="row in rows" :key="row.key">
<td style="padding-left:20px;"><div class="comp-name">{{ item.name }}</div></td> <td class="label-col">{{ row.label }}</td>
<td>{{ item.result?.tripName || '—' }}</td> <td v-for="item in results" :key="item.competitor_id">
<td class="pdelta">{{ formatPrice(item.result, item.primary_market) }}</td> {{ formatForDisplay(rawValue(row, item), row.format) }}
<td>{{ item.result?.duration ?? '—' }} days</td> </td>
<td style="max-width:280px;color:var(--t3);font-size:12px;">{{ item.result?.comments || '—' }}</td>
</tr> </tr>
</tbody> </tbody>
</table> </table>
</div> </div>
</div>
</template> </template>
<style scoped>
.results-scroll { overflow-x: auto; }
.results-grid th, .results-grid td { white-space: nowrap; max-width: 260px; overflow: hidden; text-overflow: ellipsis; padding-left: 16px; padding-right: 16px; }
.results-grid td { white-space: normal; }
.label-col {
position: sticky;
left: 0;
z-index: 1;
background: var(--surface-2);
color: var(--t2);
font-weight: 600;
min-width: 220px;
max-width: 220px;
white-space: normal !important;
}
.results-grid thead th.label-col { z-index: 2; }
</style>

12
frontend/src/components/analyse/RunHistory.vue
View File

@ -1,8 +1,16 @@
<script setup> <script setup>
import { format } from 'date-fns'
defineProps({ defineProps({
history: { type: Array, required: true } history: { type: Array, required: true }
}) })
const emit = defineEmits(['restore']) const emit = defineEmits(['restore'])
function formatRunTime(startedAt) {
if (!startedAt) return 'Unknown time'
const parsed = new Date(startedAt)
return Number.isNaN(parsed.getTime()) ? 'Unknown time' : format(parsed, 'MMM d, h:mm a')
}
</script> </script>
<template> <template>
@ -11,8 +19,8 @@ const emit = defineEmits(['restore'])
<div v-if="!history.length" style="padding:20px;color:var(--t3);font-size:12.5px;">No previous runs yet.</div> <div v-if="!history.length" style="padding:20px;color:var(--t3);font-size:12.5px;">No previous runs yet.</div>
<div v-for="run in history" :key="run.id" class="fi" style="padding:0;"> <div v-for="run in history" :key="run.id" class="fi" style="padding:0;">
<div class="fi-body" style="padding:12px 16px;"> <div class="fi-body" style="padding:12px 16px;">
<div class="fi-title">{{ run.started_at }} · {{ run.competitors_total }} competitors</div> <div class="fi-title">{{ formatRunTime(run.started_at) }} · {{ run.competitors_total }} competitors</div>
<div class="fi-meta">Status: {{ run.status }}</div> <div class="fi-meta">Status: {{ run.status }}<template v-if="run.created_by_email"> · Run by {{ run.created_by_email }}</template></div>
</div> </div>
<div class="fi-right"> <div class="fi-right">
<UButton size="xs" color="neutral" variant="ghost" @click="emit('restore', run.id)">Restore</UButton> <UButton size="xs" color="neutral" variant="ghost" @click="emit('restore', run.id)">Restore</UButton>

28
frontend/src/components/analyse/SyncToSheet.vue
View File

@ -1,5 +1,6 @@
<script setup> <script setup>
import { ref } from 'vue' import { ref } from 'vue'
import { getRowsForTabType, rawValue } from '../../config/resultFields'
// Replaces PushToSheet.vue per the Sheet-initiated pull architecture // Replaces PushToSheet.vue per the Sheet-initiated pull architecture
// (CLAUDE.md §15): a stateless Flask backend has no way to reach into // (CLAUDE.md §15): a stateless Flask backend has no way to reach into
@ -7,22 +8,33 @@ import { ref } from 'vue'
// action here at all — only a clipboard copy plus a reminder that the // action here at all — only a clipboard copy plus a reminder that the
// actual write happens from inside the Sheet's own Bettersight sidebar // actual write happens from inside the Sheet's own Bettersight sidebar
// via [Pull latest results]. // via [Pull latest results].
//
// copyRows() emits the SAME transposed shape as the legacy Sheet — one
// line per field row, tab-separated per competitor — so a PM can paste
// this straight into the real "Standard"/"NGS Competitive Tab" starting
// at the label column as a manual fallback if the Sheet-pull integration
// ever fails. See frontend/src/config/resultFields.js for the row order,
// verified against the real completed run in "G Compete App V1.xlsx".
const props = defineProps({ const props = defineProps({
results: { type: Array, required: true } // [{ name, result, primary_market }] results: { type: Array, required: true }, // [{ name, result, primary_market }]
tabType: { type: String, default: 'Standard' }
}) })
const copied = ref(false) const copied = ref(false)
function sanitize(value) {
if (value == null) return ''
return String(value).replace(/\t|\n/g, ' ')
}
async function copyRows() { async function copyRows() {
const rows = props.results.map((item) => [ const rows = getRowsForTabType(props.tabType)
item.name, const lines = rows.map((row) => [
item.result?.tripName || '', row.label,
item.result?.majorityPrice ?? '', ...props.results.map((item) => sanitize(rawValue(row, item))),
item.result?.duration ?? '',
(item.result?.comments || '').replace(/\t|\n/g, ' '),
].join('\t')) ].join('\t'))
await navigator.clipboard.writeText(rows.join('\n')) await navigator.clipboard.writeText(lines.join('\n'))
copied.value = true copied.value = true
// Not an onboarding checklist event — sync_from_sheet only flips on a // Not an onboarding checklist event — sync_from_sheet only flips on a
// real pullLatestResults() success from inside the Sheet itself. // real pullLatestResults() success from inside the Sheet itself.

30
frontend/src/components/analyse/TripIntentForm.vue
View File

@ -15,6 +15,19 @@ const duration = ref('')
const travelStyle = ref('Classic') const travelStyle = ref('Classic')
const tabType = ref('Standard') const tabType = ref('Standard')
const selectedIds = ref([]) const selectedIds = ref([])
const showAdvanced = ref(false)
// Unchecked by default — respects the normal fast/refresh-path decision
// (CLAUDE.md §7), so a competitor scraped recently is served from cache
// quickly. Checking this bypasses both cache shortcuts in
// analysis_service.run_competitor_analysis() for a genuinely fresh
// scrape+extraction every time, at the cost of speed. Reverted back to
// opt-in after a brief default-fresh experiment — that was working around
// a since-fixed bug where the confirmed/matched trip URL was silently
// discarded and the wrong page got cached; now that the correct page is
// what's actually cached, the normal fast-path behavior is trustworthy
// again and the original CLAUDE.md §7 design (fast when unchanged, live
// when stale) is worth having back.
const forceRefresh = ref(false)
function toggle(id) { function toggle(id) {
const i = selectedIds.value.indexOf(id) const i = selectedIds.value.indexOf(id)
@ -29,7 +42,8 @@ function submit() {
duration: Number(duration.value), duration: Number(duration.value),
travelStyle: travelStyle.value, travelStyle: travelStyle.value,
tabType: tabType.value, tabType: tabType.value,
competitorIds: selectedIds.value competitorIds: selectedIds.value,
forceRefresh: forceRefresh.value
}) })
} }
</script> </script>
@ -71,6 +85,20 @@ function submit() {
</div> </div>
</div> </div>
<UCollapsible v-model:open="showAdvanced">
<UButton color="neutral" variant="ghost" size="xs">
{{ showAdvanced ? '▾' : '▸' }} Advanced options
</UButton>
<template #content>
<div style="padding-top:10px;">
<label style="display:flex;align-items:center;gap:8px;font-size:12.5px;color:var(--t3);cursor:pointer;">
<UCheckbox v-model="forceRefresh" />
Force fresh scrape (ignore cache — always re-scrape and re-analyse live; slower)
</label>
</div>
</template>
</UCollapsible>
<UButton <UButton
type="submit" type="submit"
color="primary" color="primary"

34
frontend/src/components/analyse/UrlConfirmation.vue
View File

@ -9,13 +9,25 @@ const emit = defineEmits(['confirm', 'back'])
const editableUrls = ref({}) const editableUrls = ref({})
const editingId = ref(null) const editingId = ref(null)
// Competitors with no real match shouldn't force the PM into typing a
// guess (like the competitor's own homepage) just to unblock the run —
// skipping excludes them from the batch entirely instead.
const skipped = ref(new Set())
watch(() => props.matches, (matches) => { watch(() => props.matches, (matches) => {
editableUrls.value = Object.fromEntries(matches.map((m) => [m.competitor_id, m.url || ''])) editableUrls.value = Object.fromEntries(matches.map((m) => [m.competitor_id, m.url || '']))
}, { immediate: true }) }, { immediate: true })
function toggleSkip(id) {
if (skipped.value.has(id)) skipped.value.delete(id)
else skipped.value.add(id)
skipped.value = new Set(skipped.value) // reassign — Set mutation isn't tracked in-place
}
function confirm() { function confirm() {
const competitors = props.matches.map((m) => ({ const competitors = props.matches
.filter((m) => !skipped.value.has(m.competitor_id))
.map((m) => ({
id: m.competitor_id, id: m.competitor_id,
name: m.competitor_name, name: m.competitor_name,
url: editableUrls.value[m.competitor_id] url: editableUrls.value[m.competitor_id]
@ -23,7 +35,10 @@ function confirm() {
emit('confirm', competitors) emit('confirm', competitors)
} }
const allResolved = () => props.matches.every((m) => editableUrls.value[m.competitor_id]?.trim()) const allResolved = () => props.matches.every((m) =>
skipped.value.has(m.competitor_id) || editableUrls.value[m.competitor_id]?.trim()
)
const anyIncluded = () => props.matches.some((m) => !skipped.value.has(m.competitor_id))
</script> </script>
<template> <template>
@ -47,15 +62,24 @@ const allResolved = () => props.matches.every((m) => editableUrls.value[m.compet
<UButton size="xs" color="neutral" variant="ghost" @click="editingId = match.competitor_id">↗ Swap</UButton> <UButton size="xs" color="neutral" variant="ghost" @click="editingId = match.competitor_id">↗ Swap</UButton>
</template> </template>
<template v-else> <template v-else>
<div style="flex:1;font-size:12.5px;color:var(--red);">No match found</div> <div v-if="!skipped.has(match.competitor_id)" style="flex:1;font-size:12.5px;color:var(--red);">No match found</div>
<UButton size="xs" color="primary" variant="soft" @click="editingId = match.competitor_id">Enter URL manually</UButton> <div v-else style="flex:1;font-size:12.5px;color:var(--t3);">Skipped — will not be included in this run</div>
<UButton
v-if="!skipped.has(match.competitor_id)"
size="xs" color="primary" variant="soft"
@click="editingId = match.competitor_id"
>Enter URL manually</UButton>
<UButton
size="xs" color="neutral" :variant="skipped.has(match.competitor_id) ? 'soft' : 'ghost'"
@click="toggleSkip(match.competitor_id)"
>{{ skipped.has(match.competitor_id) ? '↺ Undo skip' : 'Skip this competitor' }}</UButton>
</template> </template>
</div> </div>
</div> </div>
<div style="padding:16px 20px;display:flex;justify-content:space-between;"> <div style="padding:16px 20px;display:flex;justify-content:space-between;">
<UButton color="neutral" variant="ghost" @click="emit('back')">← Back</UButton> <UButton color="neutral" variant="ghost" @click="emit('back')">← Back</UButton>
<UButton color="primary" :loading="submitting" :disabled="!allResolved()" @click="confirm">Confirm & Run</UButton> <UButton color="primary" :loading="submitting" :disabled="!allResolved() || !anyIncluded()" @click="confirm">Confirm & Run</UButton>
</div> </div>
</div> </div>
</template> </template>

2
frontend/src/components/layout/Sidebar.vue
View File

@ -7,7 +7,7 @@ const authStore = useAuthStore()
const notificationsStore = useNotificationsStore() const notificationsStore = useNotificationsStore()
onMounted(() => { onMounted(() => {
notificationsStore.fetchUnreadCount() notificationsStore.fetchUnread()
}) })
const initials = computed(() => { const initials = computed(() => {

33
frontend/src/components/layout/Topbar.vue
View File

@ -1,6 +1,7 @@
<script setup> <script setup>
import { useRouter } from 'vue-router' import { useRouter } from 'vue-router'
import { useNotificationsStore } from '../../stores/notifications_store' import { useNotificationsStore } from '../../stores/notifications_store'
import AlertCard from '../activity/AlertCard.vue'
defineProps({ defineProps({
title: { type: String, required: true }, title: { type: String, required: true },
@ -12,6 +13,10 @@ const emit = defineEmits(['export'])
const router = useRouter() const router = useRouter()
const notificationsStore = useNotificationsStore() const notificationsStore = useNotificationsStore()
function goToActivity() {
router.push('/activity')
}
</script> </script>
<template> <template>
@ -25,16 +30,38 @@ const notificationsStore = useNotificationsStore()
<svg width="13" height="13" viewBox="0 0 16 16" fill="#9BA8C0"><path d="M11.5 7a4.5 4.5 0 10-9 0 4.5 4.5 0 009 0zm-.8 4.3l3 3-1.1 1.1-3-3a6 6 0 111.1-1.1z" /></svg> <svg width="13" height="13" viewBox="0 0 16 16" fill="#9BA8C0"><path d="M11.5 7a4.5 4.5 0 10-9 0 4.5 4.5 0 009 0zm-.8 4.3l3 3-1.1 1.1-3-3a6 6 0 111.1-1.1z" /></svg>
<input type="text" placeholder="Search…"> <input type="text" placeholder="Search…">
</div> </div>
<button class="tbell" @click="router.push('/activity')"> <UPopover>
<button class="tbell" @click="notificationsStore.fetchUnread()">
<svg width="15" height="15" viewBox="0 0 16 16" fill="#6B7A99"><path d="M8 1a5.5 5.5 0 00-5.5 5.5c0 2.1-.6 3.7-1.5 4.5h14c-.9-.8-1.5-2.4-1.5-4.5A5.5 5.5 0 008 1zM6.5 13a1.5 1.5 0 003 0h-3z" /></svg> <svg width="15" height="15" viewBox="0 0 16 16" fill="#6B7A99"><path d="M8 1a5.5 5.5 0 00-5.5 5.5c0 2.1-.6 3.7-1.5 4.5h14c-.9-.8-1.5-2.4-1.5-4.5A5.5 5.5 0 008 1zM6.5 13a1.5 1.5 0 003 0h-3z" /></svg>
<div v-if="notificationsStore.unreadCount > 0" class="tbell-dot">{{ notificationsStore.unreadCount }}</div> <div v-if="notificationsStore.unreadCount > 0" class="tbell-dot">{{ notificationsStore.unreadCount }}</div>
</button> </button>
<template #content>
<div style="width:336px;max-height:420px;overflow-y:auto;">
<div class="ch">
<div><div class="ct">Alerts</div><div class="cs">Requires attention</div></div>
<span v-if="notificationsStore.unreadCount" class="badge bg-red">{{ notificationsStore.unreadCount }} new</span>
</div>
<div v-if="!notificationsStore.unreadAlerts.length" style="padding:20px;color:var(--t3);font-size:12.5px;">
You're all caught up.
</div>
<AlertCard
v-for="alert in notificationsStore.unreadAlerts"
:key="alert.id"
:alert="alert"
@read="notificationsStore.markRead"
/>
<div style="padding:10px 16px;border-top:1px solid var(--border);">
<a style="font-size:12.5px;color:var(--blue-vivid);cursor:pointer;" @click="goToActivity">View all activity →</a>
</div>
</div>
</template>
</UPopover>
<button v-if="showExport" class="tbtn tbtn-outline" @click="emit('export')"> <button v-if="showExport" class="tbtn tbtn-outline" @click="emit('export')">
<svg width="13" height="13" viewBox="0 0 16 16" fill="currentColor"><path d="M3 4h10M3 8h10M3 12h6" /></svg> <svg width="13" height="13" viewBox="0 0 16 16" fill="none"><path d="M3 4h10M3 8h10M3 12h6" stroke="currentColor" stroke-width="1.5" stroke-linecap="round" /></svg>
Export Export
</button> </button>
<button v-if="showRunAnalysis" class="tbtn tbtn-primary" @click="router.push('/analyse')"> <button v-if="showRunAnalysis" class="tbtn tbtn-primary" @click="router.push('/analyse')">
<svg width="13" height="13" viewBox="0 0 16 16" fill="white"><path d="M8 3v10M3 8h10" /></svg> <svg width="13" height="13" viewBox="0 0 16 16" fill="none"><path d="M8 3v10M3 8h10" stroke="white" stroke-width="1.5" stroke-linecap="round" /></svg>
Run Analysis Run Analysis
</button> </button>
</div> </div>

2
frontend/src/components/shared/SeatManager.vue
View File

@ -25,6 +25,7 @@ function addSeat() {
</div> </div>
</div> </div>
<div class="tbl-scroll">
<table class="tbl"> <table class="tbl">
<thead> <thead>
<tr><th style="padding-left:20px;">Email</th><th>Last accessed</th><th></th></tr> <tr><th style="padding-left:20px;">Email</th><th>Last accessed</th><th></th></tr>
@ -39,6 +40,7 @@ function addSeat() {
</tr> </tr>
</tbody> </tbody>
</table> </table>
</div>
<div style="padding:14px 20px;border-top:1px solid var(--border);display:flex;gap:8px;"> <div style="padding:14px 20px;border-top:1px solid var(--border);display:flex;gap:8px;">
<UInput <UInput

138
frontend/src/config/resultFields.js Normal file
View File

@ -0,0 +1,138 @@
// Single source of truth for the dashboard's results grid — mirrors the
// legacy Google Sheet's transposed layout (fields down the rows, one
// competitor per column) verified directly against the real completed run
// in "G Compete App V1.xlsx". Row order/labels match
// backend/industries/adventure_travel/fields.py's STANDARD_FIELDS/NGS_FIELDS
// exactly, extended with display metadata so ResultsTable.vue and
// SyncToSheet.vue both render/copy from this one list instead of
// duplicating row order in two places.
//
// Row shape: { key, label, format, source, compute }
// format — 'text' | 'number' | 'currency', controls display formatting
// source — 'name' (competitor's own name) or 'result' (item.result[key]);
// defaults to 'result' when omitted
// compute — present only on derived rows (Total inventory, Price/day) —
// a (result) => value function; when present, key/source are
// ignored for value lookup
//
// Sheet row 7 ("Map Image") is intentionally omitted — never scraped, no
// dashboard equivalent.
function divOrNull(numerator, denominator) {
if (numerator == null || denominator == null || denominator === 0) return null
return numerator / denominator
}
function mulOrNull(a, b) {
if (a == null || b == null) return null
return a * b
}
function insertAfter(rows, afterKey, inserted) {
const index = rows.findIndex((row) => row.key === afterKey)
return [...rows.slice(0, index + 1), ...inserted, ...rows.slice(index + 1)]
}
export const STANDARD_ROWS = [
{ key: 'company', label: 'Company', format: 'text', source: 'name' },
{ key: 'link', label: 'Link', format: 'text' },
{ key: 'tripName', label: 'Trip Name', format: 'text' },
{ key: 'tripCode', label: 'Trip Code', format: 'text' },
{ key: 'serviceLevel', label: 'Service Level', format: 'text' },
{ key: 'groupSize', label: 'Group size', format: 'number' },
{ key: 'duration', label: 'Duration (days)', format: 'number' },
{ key: 'meals', label: 'Meals', format: 'number' },
{ key: 'startLocation', label: 'Start Location', format: 'text' },
{ key: 'endLocation', label: 'End Location', format: 'text' },
{ key: 'departures', label: '# Departures Offered per year', format: 'number' },
{
key: 'totalInventory',
label: 'Total inventory',
format: 'number',
compute: (r) => mulOrNull(r?.departures, r?.groupSize),
},
{ key: 'seasonality', label: 'Seasonality (what are the price bands in terms of dates?)', format: 'text' },
{ key: 'startDays', label: 'Start days of week', format: 'text' },
{ key: 'targetAudience', label: 'Target audience if known', format: 'text' },
{ key: 'hotels', label: 'Hotels (if applicable)', format: 'text' },
{ key: 'activities', label: 'Included Activities', format: 'text' },
{ key: 'relevancy', label: 'Relevancy rating', format: 'number' },
{ key: 'majorityPrice', label: 'Majority Price (USD)', format: 'currency' },
{ key: 'priceLow', label: 'Price low (USD)', format: 'currency' },
{ key: 'priceHigh', label: 'Price high (USD)', format: 'currency' },
{
key: 'pricePerDay',
label: 'Price per day (USD)',
format: 'currency',
compute: (r) => divOrNull(r?.majorityPrice, r?.duration),
},
{ key: 'comments', label: 'Suggested changes for G/ Competitive Comments', format: 'text' },
{ key: 'dateUsed', label: 'Date Used for Majority Price', format: 'text' },
{ key: 'audPrice', label: 'AUD Majority price', format: 'currency' },
{ key: 'cadPrice', label: 'CAD Majority price', format: 'currency' },
{ key: 'eurPrice', label: 'EUR Majority price', format: 'currency' },
{ key: 'gbpPrice', label: 'GBP Majority price', format: 'currency' },
{
key: 'audPricePerDay',
label: 'AUD Price/day',
format: 'currency',
compute: (r) => divOrNull(r?.audPrice, r?.duration),
},
{
key: 'cadPricePerDay',
label: 'CAD Price/day',
format: 'currency',
compute: (r) => divOrNull(r?.cadPrice, r?.duration),
},
{
key: 'eurPricePerDay',
label: 'EUR Price/day',
format: 'currency',
compute: (r) => divOrNull(r?.eurPrice, r?.duration),
},
{
key: 'gbpPricePerDay',
label: 'GBP Price/day',
format: 'currency',
compute: (r) => divOrNull(r?.gbpPrice, r?.duration),
},
]
export const NGS_ROWS = insertAfter(STANDARD_ROWS, 'activities', [
{ key: 'exclusiveAccess', label: 'Exclusive Access/Special Unique Inclusions', format: 'text' },
{ key: 'groupLeader', label: 'Group Leader and/or Local Expert Description', format: 'text' },
{ key: 'sustainability', label: 'Sustainability Inclusions (if applicable)', format: 'text' },
])
export function getRowsForTabType(tabType) {
return tabType === 'NGS' ? NGS_ROWS : STANDARD_ROWS
}
/**
* Resolves a row's raw (unformatted) value for one result item —
* shared by ResultsTable.vue (display) and SyncToSheet.vue (clipboard
* copy) so the two never diverge on how a value is derived.
*/
export function rawValue(row, item) {
if (row.compute) return row.compute(item.result)
if (row.source === 'name') return item.name
return item.result?.[row.key]
}
/**
* Formats a raw value for on-screen display only — SyncToSheet.vue's
* clipboard copy uses rawValue() directly (unformatted) so pasted
* numbers land as real numbers in Excel's own currency-formatted cells.
*/
export function formatForDisplay(value, format) {
if (value == null || value === '') return '—'
if (format === 'currency') {
const num = Number(value)
return Number.isNaN(num) ? '—' : `$${num.toLocaleString(undefined, { minimumFractionDigits: 2, maximumFractionDigits: 2 })}`
}
if (format === 'number') {
const num = Number(value)
return Number.isNaN(num) ? '—' : num.toLocaleString()
}
return value
}

13
frontend/src/pages/AccountPage.vue
View File

@ -102,8 +102,7 @@ function replayTour() {
<DashboardLayout title="Account" subtitle="Subscription, seats and setup" :show-run-analysis="false"> <DashboardLayout title="Account" subtitle="Subscription, seats and setup" :show-run-analysis="false">
<div v-if="loading" style="padding:40px;text-align:center;color:var(--t3);">Loading…</div> <div v-if="loading" style="padding:40px;text-align:center;color:var(--t3);">Loading…</div>
<div v-else class="cg"> <div v-else class="cg-left">
<div class="cg-left">
<SubscriptionCard :tenant="authStore.tenant" @manage-billing="manageBilling" /> <SubscriptionCard :tenant="authStore.tenant" @manage-billing="manageBilling" />
<div v-if="upgradePitch" class="card" style="padding:20px;"> <div v-if="upgradePitch" class="card" style="padding:20px;">
@ -158,9 +157,8 @@ function replayTour() {
<span class="badge bg-green">✓ Connected</span> <span class="badge bg-green">✓ Connected</span>
<span style="font-size:12.5px;color:var(--t3);">Your Sheet template is set up.</span> <span style="font-size:12.5px;color:var(--t3);">Your Sheet template is set up.</span>
</div> </div>
</div>
<div class="cg-right"> <div class="account-bottom-row">
<SeatManager <SeatManager
:seats="authStore.tenant?.seats || []" :seats="authStore.tenant?.seats || []"
:max-seats="authStore.tenant?.max_seats || 0" :max-seats="authStore.tenant?.max_seats || 0"
@ -189,3 +187,10 @@ function replayTour() {
</div> </div>
</DashboardLayout> </DashboardLayout>
</template> </template>
<style scoped>
.account-bottom-row { display: grid; grid-template-columns: 1fr 1fr; gap: 18px; }
@media (max-width: 1100px) {
.account-bottom-row { grid-template-columns: 1fr; }
}
</style>

31
frontend/src/pages/ActivityPage.vue
View File

@ -5,31 +5,25 @@ import DashboardLayout from '../components/layout/DashboardLayout.vue'
import StatCard from '../components/shared/StatCard.vue' import StatCard from '../components/shared/StatCard.vue'
import EmptyState from '../components/shared/EmptyState.vue' import EmptyState from '../components/shared/EmptyState.vue'
import FeedItem from '../components/activity/FeedItem.vue' import FeedItem from '../components/activity/FeedItem.vue'
import AlertCard from '../components/activity/AlertCard.vue'
import { useAuthStore } from '../stores/auth_store' import { useAuthStore } from '../stores/auth_store'
import { useNotificationsStore } from '../stores/notifications_store'
import { useOnboardingTour } from '../composables/useOnboardingTour' import { useOnboardingTour } from '../composables/useOnboardingTour'
import * as alertRepository from '../repositories/alert_repository' import * as alertRepository from '../repositories/alert_repository'
import * as competitorRepository from '../repositories/competitor_repository' import * as competitorRepository from '../repositories/competitor_repository'
const authStore = useAuthStore() const authStore = useAuthStore()
const notificationsStore = useNotificationsStore()
const { tour, shouldShowTour, getResumeStep } = useOnboardingTour() const { tour, shouldShowTour, getResumeStep } = useOnboardingTour()
const loading = ref(true) const loading = ref(true)
const allAlerts = ref([]) const allAlerts = ref([])
const unreadAlerts = ref([])
const competitorCount = ref(0) const competitorCount = ref(0)
const activeTab = ref('all') const activeTab = ref('all')
onMounted(async () => { onMounted(async () => {
const [alerts, unread, competitors] = await Promise.all([ const [alerts, competitors] = await Promise.all([
alertRepository.listAlerts({ limit: 100 }), alertRepository.listAlerts({ limit: 100 }),
alertRepository.listAlerts({ unreadOnly: true }),
competitorRepository.listCompetitors(authStore.email) competitorRepository.listCompetitors(authStore.email)
]) ])
allAlerts.value = alerts allAlerts.value = alerts
unreadAlerts.value = unread
competitorCount.value = competitors.length competitorCount.value = competitors.length
loading.value = false loading.value = false
@ -71,14 +65,6 @@ function feedBadge(alert) {
function timeAgo(ts) { function timeAgo(ts) {
return ts ? formatDistanceToNow(new Date(ts), { addSuffix: true }) : '' return ts ? formatDistanceToNow(new Date(ts), { addSuffix: true }) : ''
} }
async function markRead(alertId) {
await alertRepository.markAlertRead(alertId)
unreadAlerts.value = unreadAlerts.value.filter((a) => a.id !== alertId)
const alert = allAlerts.value.find((a) => a.id === alertId)
if (alert) alert.read = true
notificationsStore.decrementUnread()
}
</script> </script>
<template> <template>
@ -113,7 +99,6 @@ async function markRead(alertId) {
</StatCard> </StatCard>
</div> </div>
<div class="cg">
<div class="cg-left"> <div class="cg-left">
<EmptyState <EmptyState
v-if="!allAlerts.length" v-if="!allAlerts.length"
@ -150,20 +135,6 @@ async function markRead(alertId) {
/> />
</div> </div>
</div> </div>
<div class="cg-right">
<div class="card">
<div class="ch">
<div><div class="ct">Alerts</div><div class="cs">Requires attention</div></div>
<span v-if="unreadAlerts.length" class="badge bg-red">{{ unreadAlerts.length }} new</span>
</div>
<div v-if="!unreadAlerts.length" style="padding:20px;color:var(--t3);font-size:12.5px;">
You're all caught up.
</div>
<AlertCard v-for="alert in unreadAlerts" :key="alert.id" :alert="alert" @read="markRead" />
</div>
</div>
</div>
</template> </template>
</DashboardLayout> </DashboardLayout>
</template> </template>

51
frontend/src/pages/AnalysePage.vue
View File

@ -25,6 +25,10 @@ const confirming = ref(false)
const matches = ref([]) const matches = ref([])
const intent = ref(null) const intent = ref(null)
const displayResults = ref([]) const displayResults = ref([])
// competitor_id -> confirmed URL from the last UrlConfirmation.vue submit —
// so a single-competitor retry reuses the same matched/swapped page rather
// than falling back to the competitor's generic root domain.
const confirmedUrls = ref({})
onMounted(async () => { onMounted(async () => {
competitors.value = await competitorRepository.listCompetitors(authStore.email) competitors.value = await competitorRepository.listCompetitors(authStore.email)
@ -32,11 +36,23 @@ onMounted(async () => {
await analysisStore.loadHistory(authStore.email) await analysisStore.loadHistory(authStore.email)
analysisStore.checkForPendingJob() analysisStore.checkForPendingJob()
if (analysisStore.activeJobId) step.value = 'progress' // analysisStore is a Pinia singleton — it outlives this page's own
// mount/unmount cycle, so activeJobId from a PAST completed run is
// still sitting there on a fresh visit (only an explicit "Run another
// analysis"/stopPolling() clears it). Checking activeJobId alone used
// to force step='progress' even for a long-finished job, showing a
// stuck "Analysis complete" header with no form and no results (the
// 'progress'-step JobProgress usage never passes :results/:error) until
// a full page refresh reset the store. Only resume the live progress
// view when the job is genuinely still in flight.
const IN_FLIGHT_STATUSES = ['queued', 'running', 'cancelling']
if (analysisStore.activeJobId && IN_FLIGHT_STATUSES.includes(analysisStore.status)) {
step.value = 'progress'
}
}) })
watch(() => analysisStore.status, (status) => { watch(() => analysisStore.status, (status) => {
if (status === 'complete' || status === 'failed') { if (status === 'complete' || status === 'failed' || status === 'cancelled') {
buildDisplayResults() buildDisplayResults()
step.value = 'results' step.value = 'results'
} }
@ -68,6 +84,7 @@ async function handleIntentSubmit(formIntent) {
async function handleConfirm(confirmedCompetitors) { async function handleConfirm(confirmedCompetitors) {
confirming.value = true confirming.value = true
confirmedUrls.value = Object.fromEntries(confirmedCompetitors.map((c) => [c.id, c.url]))
try { try {
const { job_id } = await analysisRepository.runResearch({ const { job_id } = await analysisRepository.runResearch({
tenant_id: authStore.tenantId, tenant_id: authStore.tenantId,
@ -76,7 +93,9 @@ async function handleConfirm(confirmedCompetitors) {
travel_style: intent.value.travelStyle, travel_style: intent.value.travelStyle,
tab_type: intent.value.tabType, tab_type: intent.value.tabType,
productName: intent.value.productName, productName: intent.value.productName,
competitors: confirmedCompetitors competitors: confirmedCompetitors,
force_refresh: intent.value.forceRefresh,
email: authStore.email
}) })
analysisStore.startPolling(job_id) analysisStore.startPolling(job_id)
step.value = 'progress' step.value = 'progress'
@ -97,8 +116,18 @@ function startOver() {
} }
async function retryCompetitor(item) { async function retryCompetitor(item) {
// JobProgress emits a bare 'retry' (no item) from its whole-job
// failed/cancelled states — that's a full re-run, not a single
// competitor, so there's nothing to look up here.
if (!item) return startOver()
const competitor = competitors.value.find((c) => c.id === item.competitor_id) const competitor = competitors.value.find((c) => c.id === item.competitor_id)
if (!competitor) return if (!competitor) return
// Reuse the URL actually confirmed for this competitor in the original
// batch (a Firecrawl match or a manual swap) — falls back to the
// competitor's root domain only if it was never part of a confirmed batch
// this session (e.g. added after matching ran).
const retryUrl = confirmedUrls.value[competitor.id] || competitor.url
const { job_id } = await analysisRepository.runResearch({ const { job_id } = await analysisRepository.runResearch({
tenant_id: authStore.tenantId, tenant_id: authStore.tenantId,
destination: intent.value.destination, destination: intent.value.destination,
@ -106,7 +135,9 @@ async function retryCompetitor(item) {
travel_style: intent.value.travelStyle, travel_style: intent.value.travelStyle,
tab_type: intent.value.tabType, tab_type: intent.value.tabType,
productName: intent.value.productName, productName: intent.value.productName,
competitors: [{ id: competitor.id, name: competitor.name, url: competitor.url }] competitors: [{ id: competitor.id, name: competitor.name, url: retryUrl }],
force_refresh: intent.value.forceRefresh,
email: authStore.email
}) })
const retryResult = await pollUntilDone(job_id) const retryResult = await pollUntilDone(job_id)
const success = retryResult?.results?.success?.[0] const success = retryResult?.results?.success?.[0]
@ -151,8 +182,7 @@ async function restoreFromHistory(jobId) {
action-to="/competitors" action-to="/competitors"
/> />
<div v-else class="cg"> <div v-else class="cg-left">
<div class="cg-left">
<TripIntentForm <TripIntentForm
v-if="step === 'form'" v-if="step === 'form'"
:competitors="competitors" :competitors="competitors"
@ -173,6 +203,7 @@ async function restoreFromHistory(jobId) {
:status="analysisStore.status" :status="analysisStore.status"
:progress="analysisStore.progress" :progress="analysisStore.progress"
@retry="retryCompetitor" @retry="retryCompetitor"
@cancel="analysisStore.cancelAnalysis"
/> />
<template v-else-if="step === 'results'"> <template v-else-if="step === 'results'">
@ -183,8 +214,9 @@ async function restoreFromHistory(jobId) {
:error="analysisStore.error" :error="analysisStore.error"
@retry="retryCompetitor" @retry="retryCompetitor"
@skip="skipCompetitor" @skip="skipCompetitor"
@cancel="analysisStore.cancelAnalysis"
/> />
<ResultsTable v-if="displayResults.length" :results="displayResults" /> <ResultsTable v-if="displayResults.length" :results="displayResults" :tab-type="analysisStore.tabType" />
<div v-if="displayResults.length" class="card"> <div v-if="displayResults.length" class="card">
<div class="ch"><div class="ct">Confidence</div></div> <div class="ch"><div class="ct">Confidence</div></div>
<ConfidenceBar <ConfidenceBar
@ -194,14 +226,11 @@ async function restoreFromHistory(jobId) {
:relevancy="item.result?.relevancy" :relevancy="item.result?.relevancy"
/> />
</div> </div>
<SyncToSheet v-if="displayResults.length" :results="displayResults" /> <SyncToSheet v-if="displayResults.length" :results="displayResults" :tab-type="analysisStore.tabType" />
<UButton color="neutral" variant="ghost" @click="startOver">← Run another analysis</UButton> <UButton color="neutral" variant="ghost" @click="startOver">← Run another analysis</UButton>
</template> </template>
</div>
<div class="cg-right">
<RunHistory :history="analysisStore.history" @restore="restoreFromHistory" /> <RunHistory :history="analysisStore.history" @restore="restoreFromHistory" />
</div> </div>
</div>
</DashboardLayout> </DashboardLayout>
</template> </template>

6
frontend/src/pages/BattlecardsPage.vue
View File

@ -35,15 +35,12 @@ onMounted(async () => {
action-to="/analyse" action-to="/analyse"
/> />
<div v-else class="cg"> <div v-else class="cg-left">
<div class="cg-left">
<div class="card"> <div class="card">
<div class="ch"><div class="ct">Battlecards</div><div class="cs">{{ battlecards.length }} competitors</div></div> <div class="ch"><div class="ct">Battlecards</div><div class="cs">{{ battlecards.length }} competitors</div></div>
<BattlecardCard v-for="card in battlecards" :key="card.competitor_name" :battlecard="card" /> <BattlecardCard v-for="card in battlecards" :key="card.competitor_name" :battlecard="card" />
</div> </div>
</div>
<div class="cg-right">
<div class="card"> <div class="card">
<div class="ch"><div class="ct">Comparable trips</div><div class="cs">Semantic matches</div></div> <div class="ch"><div class="ct">Comparable trips</div><div class="cs">Semantic matches</div></div>
<div v-if="!comparableTrips.length" style="padding:20px;color:var(--t3);font-size:12.5px;"> <div v-if="!comparableTrips.length" style="padding:20px;color:var(--t3);font-size:12.5px;">
@ -60,6 +57,5 @@ onMounted(async () => {
</div> </div>
</div> </div>
</div> </div>
</div>
</DashboardLayout> </DashboardLayout>
</template> </template>

2
frontend/src/pages/CompetitorsPage.vue
View File

@ -109,6 +109,7 @@ async function confirmImport(rows) {
</div> </div>
</div> </div>
<div class="tbl-scroll">
<table class="tbl"> <table class="tbl">
<thead> <thead>
<tr> <tr>
@ -144,6 +145,7 @@ async function confirmImport(rows) {
</tbody> </tbody>
</table> </table>
</div> </div>
</div>
<CompetitorForm v-model:open="formOpen" :competitor="editingCompetitor" @submit="handleSubmit" /> <CompetitorForm v-model:open="formOpen" :competitor="editingCompetitor" @submit="handleSubmit" />
<ImportPreviewModal v-model:open="previewOpen" :valid="previewValid" :invalid="previewInvalid" @confirm="confirmImport" /> <ImportPreviewModal v-model:open="previewOpen" :valid="previewValid" :invalid="previewInvalid" @confirm="confirmImport" />

5
frontend/src/repositories/analysis_repository.js
View File

@ -17,6 +17,11 @@ export async function getStatus(jobId) {
return data return data
} }
export async function cancelJob(jobId) {
const { data } = await apiService.post(`/research/cancel/${jobId}`)
return data
}
export async function getHistory(email, limit = 5) { export async function getHistory(email, limit = 5) {
const { data } = await apiService.get('/research/history', { params: { email, limit } }) const { data } = await apiService.get('/research/history', { params: { email, limit } })
return data return data

5
frontend/src/repositories/tenant_repository.js
View File

@ -19,6 +19,11 @@ export async function updateOnboarding(updates) {
return data.onboarding_checklist return data.onboarding_checklist
} }
export async function captureTimezone(timezone) {
const { data } = await apiService.patch('/account/timezone', { timezone })
return data.timezone
}
export async function getTemplateUrl() { export async function getTemplateUrl() {
const { data } = await apiService.get('/account/template') const { data } = await apiService.get('/account/template')
return data.template_url return data.template_url

59
frontend/src/stores/analysis_store.js
View File

@ -3,7 +3,27 @@ import * as analysisRepository from '../repositories/analysis_repository'
const PENDING_JOB_KEY = 'pending_job_id' const PENDING_JOB_KEY = 'pending_job_id'
const POLL_INTERVAL_MS = 3000 const POLL_INTERVAL_MS = 3000
const POLL_TIMEOUT_ATTEMPTS = 100 // ~5 minutes at 3s per CLAUDE.md §15 Research.gs timeout precedent // 20 minutes at 3s/poll — raised from the original 5-minute (100-attempt)
// value borrowed from CLAUDE.md §15's Research.gs precedent. A real MVP
// test job took 5m47s to complete (OpenRouter free-tier model rotation
// under load), so the 5-minute ceiling was reporting "failed" on jobs
// that were still succeeding in the background moments later.
const POLL_TIMEOUT_ATTEMPTS = 400
// The backend's scrape_runs.status starts life as 'pending' (api.py's
// research() route, and the PocketBase select field itself — see
// pb_migrations/001_initial_schema.js) and only becomes 'running' once an
// RQ worker actually dequeues the job (jobs.py's run_research_job()). Every
// status poll while a job is still sitting in the queue therefore returns
// 'pending' verbatim — but JobProgress.vue only recognises 'queued' for
// that state, so an unnormalised 'pending' fell through its template's
// final catch-all branch and rendered "Analysis failed" for however long
// the job waited for a free worker. Normalising here, once, means every
// consumer of this store only ever sees 'queued' — never the backend's
// internal 'pending' spelling.
function normalizeStatus(status) {
return status === 'pending' ? 'queued' : status
}
// Active job lifecycle + last-5-runs history. Job progress survives a // Active job lifecycle + last-5-runs history. Job progress survives a
// session expiry mid-analysis (CLAUDE.md §21): the job_id is stashed // session expiry mid-analysis (CLAUDE.md §21): the job_id is stashed
@ -12,9 +32,16 @@ const POLL_TIMEOUT_ATTEMPTS = 100 // ~5 minutes at 3s per CLAUDE.md §15 Researc
export const useAnalysisStore = defineStore('analysis', { export const useAnalysisStore = defineStore('analysis', {
state: () => ({ state: () => ({
activeJobId: null, activeJobId: null,
status: null, // 'queued' | 'running' | 'complete' | 'failed' status: null, // 'queued' | 'running' | 'complete' | 'failed' | 'cancelled'
progress: { total: 0, done: 0 }, progress: { total: 0, done: 0 },
results: null, results: null,
// Standard vs NGS row layout for ResultsTable.vue/SyncToSheet.vue —
// populated from the backend (the source of truth, since it's what
// was actually persisted on scrape_runs at job creation) rather than
// held only in the page's local form state, so a run restored from
// history (which never went through TripIntentForm this session)
// still renders the correct row set.
tabType: null,
error: null, error: null,
history: [], history: [],
_pollTimer: null, _pollTimer: null,
@ -40,12 +67,13 @@ export const useAnalysisStore = defineStore('analysis', {
this._clearTimer() this._clearTimer()
try { try {
const data = await analysisRepository.getStatus(this.activeJobId) const data = await analysisRepository.getStatus(this.activeJobId)
this.status = data.status this.status = normalizeStatus(data.status)
this.progress = data.progress || { total: 0, done: 0 } this.progress = data.progress || { total: 0, done: 0 }
this.error = data.error this.error = data.error
if (data.status === 'complete' || data.status === 'failed') { if (data.status === 'complete' || data.status === 'failed' || data.status === 'cancelled') {
this.results = data.results this.results = data.results
this.tabType = data.tab_type || this.tabType
localStorage.removeItem(PENDING_JOB_KEY) localStorage.removeItem(PENDING_JOB_KEY)
return return
} }
@ -78,6 +106,26 @@ export const useAnalysisStore = defineStore('analysis', {
this.activeJobId = null this.activeJobId = null
}, },
// PM-initiated stop — distinct from stopPolling(), which just stops
// watching (the backend job keeps running either way — CLAUDE.md
// §18/§21's disconnect-survival design). Cancellation is cooperative
// on the backend (jobs.py's run_research_job() only checks for it
// between competitors, so a scrape/extraction already in flight
// still finishes) — this sets a transient 'cancelling' status and
// keeps polling rather than guessing the final state client-side,
// so the eventual 'cancelled' status and whatever partial results
// did complete land correctly from the backend.
async cancelAnalysis() {
if (!this.activeJobId) return
try {
await analysisRepository.cancelJob(this.activeJobId)
} catch (err) {
this.error = err?.response?.data?.message || 'Could not cancel — please try again.'
return
}
this.status = 'cancelling'
},
checkForPendingJob() { checkForPendingJob() {
const pending = localStorage.getItem(PENDING_JOB_KEY) const pending = localStorage.getItem(PENDING_JOB_KEY)
if (pending) this.resumePolling(pending) if (pending) this.resumePolling(pending)
@ -90,8 +138,9 @@ export const useAnalysisStore = defineStore('analysis', {
async restoreFromHistory(jobId) { async restoreFromHistory(jobId) {
const run = await analysisRepository.getHistoryDetail(jobId) const run = await analysisRepository.getHistoryDetail(jobId)
this.activeJobId = run.id this.activeJobId = run.id
this.status = run.status this.status = normalizeStatus(run.status)
this.results = run.results this.results = run.results
this.tabType = run.tab_type || null
} }
} }
}) })

13
frontend/src/stores/auth_store.js
View File

@ -62,6 +62,19 @@ export const useAuthStore = defineStore('auth', {
async fetchTenant() { async fetchTenant() {
this.email = pb.authStore.model?.email ?? null this.email = pb.authStore.model?.email ?? null
this.tenant = await tenantRepository.getMe() this.tenant = await tenantRepository.getMe()
// No signup form exists in this dashboard's scope (account
// creation is via Stripe webhook or the manual onboarding
// runbook), so first login is the earliest point browser JS can
// capture the PM's timezone. Once persisted, getMe() returns it
// going forward and this never fires again for that tenant.
if (this.tenant && !this.tenant.timezone) {
try {
const timezone = Intl.DateTimeFormat().resolvedOptions().timeZone
this.tenant.timezone = await tenantRepository.captureTimezone(timezone)
} catch {
// Side effect — never blocks login if this fails.
}
}
}, },
async restoreSession() { async restoreSession() {

22
frontend/src/stores/notifications_store.js
View File

@ -2,22 +2,26 @@ import { defineStore } from 'pinia'
import * as alertRepository from '../repositories/alert_repository' import * as alertRepository from '../repositories/alert_repository'
// Small, dedicated store (not in CLAUDE.md's explicit stores list, but // Small, dedicated store (not in CLAUDE.md's explicit stores list, but
// the sidebar's unread-alert badge is visible on every page, not just // the sidebar's unread-alert badge — and now the Topbar's alerts
// ActivityPage, so its count needs a home outside any single page's // dropdown — are visible on every page, not just ActivityPage, so both
// local state. // the count AND the full alert list need a home outside any single
// page's local state.
export const useNotificationsStore = defineStore('notifications', { export const useNotificationsStore = defineStore('notifications', {
state: () => ({ state: () => ({
unreadCount: 0 unreadCount: 0,
unreadAlerts: []
}), }),
actions: { actions: {
async fetchUnreadCount() { async fetchUnread() {
const unread = await alertRepository.listAlerts({ unreadOnly: true }) this.unreadAlerts = await alertRepository.listAlerts({ unreadOnly: true })
this.unreadCount = unread.length this.unreadCount = this.unreadAlerts.length
}, },
decrementUnread() { async markRead(alertId) {
if (this.unreadCount > 0) this.unreadCount -= 1 await alertRepository.markAlertRead(alertId)
this.unreadAlerts = this.unreadAlerts.filter((a) => a.id !== alertId)
this.unreadCount = Math.max(0, this.unreadCount - 1)
} }
} }
}) })

129
legal/privacy-policy.md Normal file
View File

@ -0,0 +1,129 @@
<!--
DRAFT — not legal advice. This is a starting point built directly from
CLAUDE.md §22's stated requirements (what must be disclosed, retention
periods from DATA_RETENTION, deletion process, data region) plus
standard SaaS privacy-policy structure. Have this reviewed by an
actual lawyer before publishing to bettersight.io/privacy — GDPR/CCPA
compliance, the exact subprocessor list, and the governing-law/contact
details below all need a real legal review pass, not just an LLM pass.
Fields marked [BRACKETED] need your input before this is ready to publish.
-->
# Bettersight Privacy Policy
**Last updated:** [DATE]
Bettersight ("Bettersight," "we," "us," or "our") is operated by
Undo Designs LLC. This policy explains what data we collect from you
(the paying client and your team, collectively "you" or "PM"), why we
collect it, how long we keep it, and how you can request its deletion.
## 1. What We Collect
**Account and identity data**
- Your name and email address (used for login, licence validation,
and billing)
- Your organisation's domain (used to validate team member access)
- Your timezone (captured automatically from your browser on first
login, used only to send your daily/weekly briefs at the right
local time)
**Usage data**
- Competitor websites and catalogue URLs you add to your account
- Research job history (destinations, durations, travel styles you've
searched, and the resulting job records)
- Login and session activity, for security and support purposes
**Payment data**
- Billing is handled entirely by Stripe. We never see or store your
card number — Stripe stores your payment method and shares with us
only your subscription status, tier, and billing history.
**Competitor and market data**
- Bettersight scrapes publicly available pages from competitor
websites you specify, extracting pricing and trip information. This
data describes third-party businesses' public offerings — it is not
personal data about you, and it is not obtained from any
password-protected or non-public source. See our
[Terms of Service](/terms) for the full scraping policy.
## 2. Why We Collect It
We use your data solely to operate Bettersight: authenticating you,
running the competitive analyses you request, sending you the alerts
and briefs you've configured, and processing your subscription
payment. We do not sell your data, and we do not use it to train any
third-party AI model beyond the narrow, per-request use described in
Section 4.
## 3. How Long We Keep It
| Data | Retention |
|---|---|
| Research job records (scrape_runs) | 90 days |
| Price history | 365 days |
| Read/dismissed alerts | 90 days (unread alerts are kept until you read them) |
| Your account and account data generally | Retained until you request deletion (Section 6) |
Unread alerts, your competitor list, and your account itself are kept
for as long as your subscription is active, since they are the core
data your subscription exists to maintain.
## 4. Who We Share It With
We use the following subprocessors to operate Bettersight. Each
processes only the minimum data needed for its function:
| Subprocessor | Purpose | Data shared |
|---|---|---|
| Stripe | Payment processing | Your name, email, payment method |
| Resend | Transactional email delivery | Your email address, email content |
| OpenRouter | AI extraction and analysis of scraped competitor pages | Scraped competitor page content only — never your personal data |
| Sentry | Error monitoring | Technical error data, with your email address stripped before it reaches Sentry |
| Hetzner | Server hosting and database storage | All account and usage data (see Section 5 — data location) |
| Webshare / Bright Data | Proxy infrastructure for scraping competitor pages | No personal data — used only to route scraping traffic |
We do not share your data with any other third party, and we do not
sell it.
## 5. Where Your Data Is Stored
Bettersight's servers and database are hosted on Hetzner
infrastructure in the EU region. Backups are stored in Hetzner Object
Storage, also in the EU.
## 6. Your Rights and How to Request Deletion
You may request permanent deletion of your account and all associated
data at any time by emailing **support@bettersight.io** or using the
delete option in your Account settings. On request, we will:
- Cancel your active subscription
- Permanently delete your competitors, tracked products, price
history, alerts, battlecards, comparable trip matches, and your
account record itself
- Send you a confirmation email once deletion is complete
Deletion is permanent and cannot be undone. If you are in the EU/UK,
you have the right under GDPR to request a copy of your data, request
correction of inaccurate data, or object to processing — contact
support@bettersight.io for any of these requests.
## 7. Cookies and Tracking
Bettersight's dashboard uses only the session cookie/token required to
keep you logged in. We do not use third-party advertising or tracking
cookies.
## 8. Changes to This Policy
We'll update the "Last updated" date above if this policy changes
materially, and notify active clients by email of any change that
affects how their data is used.
## 9. Contact
Questions about this policy or your data: **support@bettersight.io**
[Undo Designs LLC — company address / registration details]

132
legal/terms-of-service.md Normal file
View File

@ -0,0 +1,132 @@
<!--
DRAFT — not legal advice. Section 4 below is CLAUDE.md §30's required
scraping clause, reproduced verbatim (do not paraphrase it — it's the
liability-shifting language the whole scraping legal position depends
on). Everything else is standard SaaS ToS structure built around it.
Have this reviewed by an actual lawyer before publishing to
bettersight.io/terms.
Fields marked [BRACKETED] need your input before this is ready to publish.
-->
# Bettersight Terms of Service
**Last updated:** [DATE]
These Terms of Service ("Terms") govern your use of Bettersight, a
competitive intelligence platform operated by Undo Designs LLC
("Undo Designs," "we," "us," or "our"). By creating an account or
using Bettersight, you agree to these Terms.
## 1. The Service
Bettersight automates competitor price tracking, trip discovery, and
market analysis for adventure travel operators. You describe the
competitor websites and products you want tracked; Bettersight
retrieves publicly available data from those sites on your behalf and
delivers structured intelligence to your dashboard and connected
Google Sheet.
## 2. Subscription, Billing, and Trials
- Bettersight is offered on the tiers and pricing published at
[bettersight.io], billed monthly via Stripe.
- Trial periods (14/21/30 days depending on tier) require no credit
card to start. If a card is on file when the trial ends, your
subscription converts automatically to a paid plan; if not, your
account is paused until you subscribe.
- You can add or remove seats, upgrade your tier, or cancel at any
time from your Account page or the Stripe customer portal.
- Fees are non-refundable except where required by law.
## 3. Acceptable Use
You agree to use Bettersight only for legitimate competitive
intelligence purposes related to your own business, and not to:
- Share your account access beyond the seats included in your plan
- Use Bettersight to target websites you do not have a legitimate
competitive interest in monitoring
- Attempt to circumvent seat limits, rate limits, or other technical
restrictions on the Service
- Use the Service in a way that violates applicable law
We may suspend or terminate accounts that violate this section.
## 4. Competitor Data Retrieval — Your Responsibility
By using Bettersight, you (the client) acknowledge and agree that:
(a) Bettersight retrieves publicly available data from third-party
websites on your behalf as instructed by you. You are solely
responsible for ensuring that your use of Bettersight and any data
retrieved through it complies with the terms of service of any
third-party website, applicable law, and any other relevant
obligations.
(b) Bettersight does not access any password-protected,
subscription-gated, or otherwise non-public content. All data
retrieved is publicly accessible to any internet user without
authentication.
(c) Bettersight limits its retrieval activity to a frequency
consistent with normal human browsing behaviour and does not engage in
denial-of-service or scraping activity that would unreasonably burden
any third-party server.
(d) Undo Designs LLC accepts no liability for any claims arising from
a third party in connection with data retrieved from their website on
your behalf. You agree to indemnify and hold harmless Undo Designs LLC
from any such claims.
## 5. Data Ownership
You own the data you enter into Bettersight (your competitor list,
research history, and any custom configuration). We own the
Bettersight platform, its underlying software, and the structured
intelligence formatting it produces. On account deletion, your data is
permanently erased per our [Privacy Policy](/privacy) — we do not
retain a copy for our own use afterward.
## 6. Third-Party Services
Bettersight relies on third-party infrastructure (Stripe for payment,
Resend for email, OpenRouter for AI-assisted analysis, Hetzner for
hosting, and proxy providers for competitor page retrieval). We are
not responsible for outages or failures originating from these
providers, though we will make reasonable efforts to maintain service
continuity.
## 7. Disclaimer and Limitation of Liability
Bettersight is provided "as is." Competitive intelligence extracted by
Bettersight is generated by automated scraping and AI analysis and may
occasionally be incomplete or inaccurate — you should use it as a
research aid, not as the sole basis for pricing or strategic
decisions. To the maximum extent permitted by law, Undo Designs LLC's
total liability for any claim arising from your use of Bettersight is
limited to the fees you paid in the 12 months preceding the claim.
## 8. Termination
You may cancel your subscription at any time. We may suspend or
terminate your account for violation of these Terms, non-payment, or
if required by law. On termination, Section 4's obligations and
Section 5's data ownership terms survive.
## 9. Changes to These Terms
We'll update the "Last updated" date above if these Terms change
materially, and notify active clients by email of any change that
affects their rights or obligations.
## 10. Governing Law
These Terms are governed by the laws of [STATE/COUNTRY OF
INCORPORATION], without regard to conflict-of-law principles.
## 11. Contact
Questions about these Terms: **support@bettersight.io**
[Undo Designs LLC — company address / registration details]

70
n8n/workflows/daily_digest.json
View File

@ -1,14 +1,14 @@
{ {
"name": "Bettersight — Daily Digest (6pm)", "name": "Bettersight — Daily Digest (timezone-aware, hourly check)",
"nodes": [ "nodes": [
{ {
"parameters": { "parameters": {
"rule": { "rule": {
"interval": [{ "field": "cronExpression", "expression": "0 18 * * *" }] "interval": [{ "field": "cronExpression", "expression": "0 * * * *" }]
} }
}, },
"id": "schedule-trigger", "id": "schedule-trigger",
"name": "Schedule Trigger (18:00 daily)", "name": "Schedule Trigger (hourly, on the hour)",
"type": "n8n-nodes-base.scheduleTrigger", "type": "n8n-nodes-base.scheduleTrigger",
"typeVersion": 1.2, "typeVersion": 1.2,
"position": [0, 0] "position": [0, 0]
@ -37,20 +37,42 @@
} }
}, },
{ {
"parameters": { "fieldToSplitOut": "items" }, "parameters": {
"id": "split-tenants", "method": "GET",
"name": "Split into items", "url": "={{ $env.POCKETBASE_URL }}/api/collections/tenant_notifications/records",
"type": "n8n-nodes-base.splitOut", "sendQuery": true,
"typeVersion": 1, "queryParameters": {
"position": [440, 0] "parameters": [{ "name": "perPage", "value": "200" }]
},
"authentication": "genericCredentialType",
"genericAuthType": "httpHeaderAuth"
},
"id": "list-tenant-notifications",
"name": "List tenant_notifications (PocketBase)",
"type": "n8n-nodes-base.httpRequest",
"typeVersion": 4.2,
"position": [220, 160],
"credentials": {
"httpHeaderAuth": { "id": "pocketbase-admin-auth", "name": "PocketBase Admin Auth" }
}
},
{
"parameters": {
"jsCode": "const { DateTime } = require('luxon')\n\n// CLAUDE.md §6 timezone-aware digest scheduling — this Code node is\n// the direct implementation of the snippet documented there. Runs\n// every hour on the hour; for each active tenant, converts their\n// configured local send time (tenant_notifications.email_digest_time,\n// default \"18:00\") from their timezone (tenants.timezone) to UTC, and\n// keeps only the tenants whose target UTC hour matches the current\n// hour. A tenant with no tenant_notifications row yet gets the\n// schema's stated defaults (email_digest=true, email_digest_time=\"18:00\")\n// rather than being silently skipped forever.\nconst tenants = $('List active tenants (PocketBase)').first().json.items || []\nconst notifications = $('List tenant_notifications (PocketBase)').first().json.items || []\nconst notifByTenant = Object.fromEntries(notifications.map(n => [n.tenant_id, n]))\n\nconst nowUtc = DateTime.utc()\nconst due = []\n\nfor (const tenant of tenants) {\n const notif = notifByTenant[tenant.id] || {}\n const emailDigestEnabled = notif.email_digest !== false // default true\n if (!emailDigestEnabled) continue\n\n const localTime = notif.email_digest_time || '18:00'\n const tenantTz = tenant.timezone || 'UTC'\n const sendAtUtc = DateTime.fromFormat(localTime, 'HH:mm', { zone: tenantTz }).toUTC()\n\n if (sendAtUtc.hour === nowUtc.hour) {\n due.push({ id: tenant.id })\n }\n}\n\nreturn due.map(t => ({ json: t }))"
},
"id": "filter-due-tenants",
"name": "Filter tenants due this hour (luxon)",
"type": "n8n-nodes-base.code",
"typeVersion": 2,
"position": [660, 0]
}, },
{ {
"parameters": { "batchSize": 1 }, "parameters": { "batchSize": 1 },
"id": "loop-tenants", "id": "loop-tenants",
"name": "Loop Over Tenants", "name": "Loop Over Due Tenants",
"type": "n8n-nodes-base.splitInBatches", "type": "n8n-nodes-base.splitInBatches",
"typeVersion": 3, "typeVersion": 3,
"position": [660, 0] "position": [880, 0]
}, },
{ {
"parameters": { "parameters": {
@ -65,32 +87,40 @@
"name": "POST /internal/daily-digest/{tenant_id}", "name": "POST /internal/daily-digest/{tenant_id}",
"type": "n8n-nodes-base.httpRequest", "type": "n8n-nodes-base.httpRequest",
"typeVersion": 4.2, "typeVersion": 4.2,
"position": [880, 0] "position": [1100, 0]
} }
], ],
"connections": { "connections": {
"Schedule Trigger (18:00 daily)": { "Schedule Trigger (hourly, on the hour)": {
"main": [[{ "node": "List active tenants (PocketBase)", "type": "main", "index": 0 }]] "main": [
[
{ "node": "List active tenants (PocketBase)", "type": "main", "index": 0 },
{ "node": "List tenant_notifications (PocketBase)", "type": "main", "index": 0 }
]
]
}, },
"List active tenants (PocketBase)": { "List active tenants (PocketBase)": {
"main": [[{ "node": "Split into items", "type": "main", "index": 0 }]] "main": [[{ "node": "Filter tenants due this hour (luxon)", "type": "main", "index": 0 }]]
}, },
"Split into items": { "List tenant_notifications (PocketBase)": {
"main": [[{ "node": "Loop Over Tenants", "type": "main", "index": 0 }]] "main": [[{ "node": "Filter tenants due this hour (luxon)", "type": "main", "index": 0 }]]
}, },
"Loop Over Tenants": { "Filter tenants due this hour (luxon)": {
"main": [[{ "node": "Loop Over Due Tenants", "type": "main", "index": 0 }]]
},
"Loop Over Due Tenants": {
"main": [ "main": [
[], [],
[{ "node": "POST /internal/daily-digest/{tenant_id}", "type": "main", "index": 0 }] [{ "node": "POST /internal/daily-digest/{tenant_id}", "type": "main", "index": 0 }]
] ]
}, },
"POST /internal/daily-digest/{tenant_id}": { "POST /internal/daily-digest/{tenant_id}": {
"main": [[{ "node": "Loop Over Tenants", "type": "main", "index": 0 }]] "main": [[{ "node": "Loop Over Due Tenants", "type": "main", "index": 0 }]]
} }
}, },
"active": false, "active": false,
"settings": { "executionOrder": "v1" }, "settings": { "executionOrder": "v1" },
"meta": { "meta": {
"description": "CLAUDE.md §15 daily digest — one Flask call per active tenant at 6pm. All the real logic (which alerts to include, rendering, sending, marking delivered_email) lives in services/alert_service.py's send_daily_digest(); this workflow only iterates tenants and triggers it. Not yet imported/run against a live n8n instance." "description": "CLAUDE.md §6 timezone-aware daily digest — runs hourly, converts each tenant's tenant_notifications.email_digest_time (default 18:00) from tenants.timezone to UTC via luxon, and only triggers /internal/daily-digest for tenants whose local send hour matches the current UTC hour. Replaces the earlier version of this workflow which fired at one fixed UTC hour for every tenant regardless of their timezone. All digest content/rendering logic lives in services/alert_service.py's send_daily_digest(); this workflow only decides WHEN to call it per tenant. Both PocketBase list nodes connect their output to the same Code node input, so n8n waits for both branches to complete before it runs; the Code node then reads each dataset via named node references ($('List active tenants (PocketBase)') / $('List tenant_notifications (PocketBase)')) rather than a wired Merge node. Not yet imported/run against a live n8n instance."
} }
} }

42
n8n/workflows/weekly_data_cleanup_cron.json Normal file
View File

@ -0,0 +1,42 @@
{
"name": "Bettersight — Weekly Data Cleanup (Sunday 3am)",
"nodes": [
{
"parameters": {
"rule": {
"interval": [{ "field": "cronExpression", "expression": "0 3 * * 0" }]
}
},
"id": "schedule-trigger",
"name": "Schedule Trigger (Sunday 03:00)",
"type": "n8n-nodes-base.scheduleTrigger",
"typeVersion": 1.2,
"position": [0, 0]
},
{
"parameters": {
"method": "POST",
"url": "={{ $env.BETTERSIGHT_API_BASE_URL }}/internal/data-cleanup",
"sendHeaders": true,
"headerParameters": {
"parameters": [{ "name": "X-API-Key", "value": "={{ $env.API_SECRET_KEY }}" }]
}
},
"id": "data-cleanup",
"name": "POST /internal/data-cleanup",
"type": "n8n-nodes-base.httpRequest",
"typeVersion": 4.2,
"position": [220, 0]
}
],
"connections": {
"Schedule Trigger (Sunday 03:00)": {
"main": [[{ "node": "POST /internal/data-cleanup", "type": "main", "index": 0 }]]
}
},
"active": false,
"settings": { "executionOrder": "v1" },
"meta": {
"description": "CLAUDE.md §22 DATA_RETENTION weekly cleanup — a single global call, not per-tenant, since the deletion sweep runs across all tenants' scrape_runs (>90 days), read alerts (>90 days), and price_history (>365 days) in one pass. All real logic lives in services/retention_service.py's run_weekly_cleanup(); this workflow only triggers it on schedule. Not yet imported/run against a live n8n instance."
}
}

31
scripts/backup-pocketbase.sh Executable file
View File

@ -0,0 +1,31 @@
#!/bin/bash
#
# Daily PocketBase point-in-time backup — CLAUDE.md §18 "Layer 2 —
# Daily zip backup". Belt-and-braces alongside Litestream's real-time
# WAL streaming (Layer 1): this is the fallback if the Litestream
# stream itself is ever corrupted, so it deliberately does NOT depend
# on Litestream being healthy.
#
# Installed on the Hetzner VPS at /usr/local/bin/backup-pocketbase.sh,
# triggered by the crontab entry in bettersight-backup.cron.
#
# Requires: rclone configured with a remote named "hetzner-s3" pointing
# at the Hetzner Object Storage bucket (see .env.example's
# HETZNER_S3_* vars for the credentials that remote should use).
set -euo pipefail
DATE=$(date +%Y%m%d)
BACKUP_NAME="backup_${DATE}.zip"
docker exec pocketbase /pb/pocketbase backup \
--dir /pb/backups --name "${BACKUP_NAME}"
rclone copy /pb/backups/ \
hetzner-s3:bettersight-backups/ \
--max-age 7d
# 7-day retention on Hetzner Object Storage (via --max-age above) plus
# a tighter 2-day local retention on the VPS itself — the local copy
# only needs to survive long enough for the rclone upload to succeed.
find /pb/backups -name "*.zip" -mtime +2 -delete

10
scripts/bettersight-backup.cron Normal file
View File

@ -0,0 +1,10 @@
# Daily PocketBase backup — CLAUDE.md §18 Layer 2 (belt + braces with Litestream)
#
# Install on the Hetzner VPS at /etc/cron.d/bettersight-backup after
# copying scripts/backup-pocketbase.sh to /usr/local/bin/ and chmod +x'ing it:
#
# cp scripts/backup-pocketbase.sh /usr/local/bin/backup-pocketbase.sh
# chmod +x /usr/local/bin/backup-pocketbase.sh
# cp scripts/bettersight-backup.cron /etc/cron.d/bettersight-backup
#
0 2 * * * root /usr/local/bin/backup-pocketbase.sh >> /var/log/bettersight-backup.log 2>&1

23
z_starting_files/brightdata_proxy_ca/New SSL certifcate - MUST BE USED WITH PORT 33335/BrightData SSL certificate (port 33335).crt Normal file
View File

@ -0,0 +1,23 @@
-----BEGIN CERTIFICATE-----
MIID1DCCArygAwIBAgIUeAKxfu5Akc2vs63HjJCeOqPVpC8wDQYJKoZIhvcNAQEL
BQAwVDELMAkGA1UEBhMCSUwxCzAJBgNVBAgMAklMMRQwEgYDVQQKDAtCcmlnaHQg
RGF0YTEiMCAGA1UEAwwZQnJpZ2h0IERhdGEgUHJveHkgUm9vdCBDQTAeFw0yNDA5
MTYxNTE3NTVaFw0zNDA5MTQxNTE3NTVaMFQxCzAJBgNVBAYTAklMMQswCQYDVQQI
DAJJTDEUMBIGA1UECgwLQnJpZ2h0IERhdGExIjAgBgNVBAMMGUJyaWdodCBEYXRh
IFByb3h5IFJvb3QgQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC/
ECdI43fgos9VnyJecnW2KsnClOMuzACmxSdCYzfl/2RgS/d4HludE8hXjkbT8ive
Nb5+Wa6fAB3cOmDPXTdYkHqGbWGGap3YQhsyC5DqxZwKPgWBshIP4kai8CL8UzRh
rDz75KW5G12amr7ozQxYnrkyr7DIz3Reac/9ShuZVr8iRdQI2yJCeP1uw0VYt77m
3dvw2XVar1m79TawBXws1twORbu5I+qA31YSDEs4CAfTcK+405TcZWLngjUb2Rtj
EHF/qHcQuXrS5rpAskXWj0n45UFqPg5stJRnigT/DzV4BhOkvSNKh4TAg1202TIT
/OCDF9heAks6jVjAHx1DAgMBAAGjgZ0wgZowHQYDVR0OBBYEFOmeH8XNfK2OcWef
fztuOmD+XHbjMB8GA1UdIwQYMBaAFOmeH8XNfK2OcWeffztuOmD+XHbjMA8GA1Ud
EwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgGGMDcGA1UdHwQwMC4wLKAqoCiGJmh0
dHA6Ly9jcmwuYnJpZ2h0ZGF0YS5jb20vcHJveHlfY2EuY3JsMA0GCSqGSIb3DQEB
CwUAA4IBAQAqmunmZM+/9ClQjzXaHmBu3I/eLqqWOssnJYA7tfK+EVXAgUYULJ5u
BO3FIWdhSX1HvmOz7nodNru7z2JDngAHVmCVpLJ82a8s8RK1OPtDZE+j50UVNk/G
McXVFFrL1AyAL6TgANTdGYtNNHHIb2HcOnYNwDqIvqJZn43FF9UWSY1sh3vOMbmI
geXK0RG6XMVKu7/abhWFBZBTFNsbuu8Qf4C+SGwvO6qHVmqV1iLgGlgnYNhksOMq
M3lc6F9H0Qpe91+5mMvfienrWPYOOvFFKPMCX714cZGUUlKpGoO5FibqVCmYf8kY
7TxA62alCsJJv4K1np6UXdPmCHQVV9yI
-----END CERTIFICATE-----

BIN
z_starting_files/brightdata_proxy_ca/New SSL certifcate - MUST BE USED WITH PORT 33335/BrightData_SSL_Certificate_Installer_signed.exe Normal file
View File

Binary file not shown.

14
z_starting_files/brightdata_proxy_ca/New SSL certifcate - MUST BE USED WITH PORT 33335/README.txt Normal file
View File

@ -0,0 +1,14 @@
This is the *new* SSL certificate, effective from September 2024. Please use this certificate for all new projects and avoid using the previous certificate.
IMPORTANT: to use this certificate, you must use port 33335 when connecting to Bright Data's proxy network, even if documentation and sample codes provided mention port 22225. Documentation will gradually be updated to port 33335 as the old certificate is being phased out.
You have a few options to use this certificate:
- If you write code, you can simply load it into your existing code.
- If you are using Windows and third-party tools that allow you to configure proxy settings (like AdsPower, Proxifier, Multilogin, Undetectable etc.) you can install the certificate on your computer using the installer.
Again, remember: you MUST use port 33335 when connecting to Bright Data's proxy network, even if documentation and sample codes provided mention port 22225.
Full information regarding how to use the SSL certificate is available on https://docs.brightdata.com/general/account/ssl-certificate

33
z_starting_files/brightdata_proxy_ca/Old Certficate - DEPRECATED, DONT USE/DEPRECATED BrightData SSL certificate (port 22225).crt Normal file
View File

@ -0,0 +1,33 @@
-----BEGIN CERTIFICATE-----
MIIFozCCA4ugAwIBAgIJAPnnIqmvvTArMA0GCSqGSIb3DQEBBQUAMD8xCzAJBgNV
BAYTAklMMQswCQYDVQQIEwJJTDENMAsGA1UEChMESG9sYTEUMBIGA1UEAxMLbHVt
aW5hdGkuaW8wHhcNMTYwOTI3MTQyODM4WhcNMjYwOTI1MTQyODM4WjA/MQswCQYD
VQQGEwJJTDELMAkGA1UECBMCSUwxDTALBgNVBAoTBEhvbGExFDASBgNVBAMTC2x1
bWluYXRpLmlvMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAtiqw0DuX
g5g7BC+Cr7mZvgXB7CsJ10YFb2xwoDZlHHJ8G0KEMUeNiY9EjPR8ZIHlnjGJehsW
PUvJeSAoDnT+fh4udWyUJ3VSqDTyGpu4DpfLBwaaZP/fq45UeR0oLs3ZJd6joDss
AjJdQbdBPJj/57MjwbF+jddP6qm9XbCWjYzl1uxdMVjloetyRUgkhkh2ALp/VtK8
hUj/XgvD/Y1souKYs5DKayJTn+GM6MlSOUBQ0+b8yUbDb/9vjbHlX4pZ8gbgSEFf
xUV49Sxd6EhRXzFw4TERQVut0cgojmRmrgXXwc4kJi0Uvtc6tV/hJeH2yRS84Ehg
feY5dcJVc69ILYfGrNmwbFvf5aHZPWFG0kIcy9iMMk+3wSUaBP+FAYyd0i+PJTxy
5Jfmhs6BHowuEr0zgL+xge+/RCEbVUPvA6w9DWYbpqckZUh9sPga3JcHjaHGs6Cz
dnjEShgmlBm0DL6JMumLWFJrjztsm56Huuai0F5pwyrsyq8fbK6Sp18sq5/vH3Vy
t2XAj4EIFvpWHZjuocCe5/5vAbkSXjQ5HEIS+SyVhlFriCy5Mf3fTyMFqwm3tbZv
jEooumi0/9F2WvisUgheC1uatZ8M+Pzi+Kp3x2SSS992KWs0M35GEstiB09RkNHe
GItI6qxqY/Npw5u6lBE6Z28ISwvuet1a4vMCAwEAAaOBoTCBnjAdBgNVHQ4EFgQU
Wq7PsMnq2tuDhTV0oUW4jjzvLTcwbwYDVR0jBGgwZoAUWq7PsMnq2tuDhTV0oUW4
jjzvLTehQ6RBMD8xCzAJBgNVBAYTAklMMQswCQYDVQQIEwJJTDENMAsGA1UEChME
SG9sYTEUMBIGA1UEAxMLbHVtaW5hdGkuaW+CCQD55yKpr70wKzAMBgNVHRMEBTAD
AQH/MA0GCSqGSIb3DQEBBQUAA4ICAQA3oT4lrUErSqXjQtDUINo62KcJWs4kjEd8
qXZdl/HVim06nOG6DFZCSh8JngFi4MFmSzGlBGxe1pXaYArtekfLWmhwoVoJiiaA
DAAPItcZNlA9zIORyLZlrXlIuP5xzsb9PbnNWhd9xJHksHGoHDPHAW/KI/GJdjQv
uuCyObvv1IgGvfHbv4lXGCwQuU0OBGXv1kfZtAqUS+ei5zkK+nY0qc3L3Ce+Ow6h
/haDe0FDoT7zkwnEHu/ExCGSR3lNnyBAewlPVMzbJznuPMU3FFA3MHT7IcHxJWff
r8jOXo3qXWqd+T2oDO02KUR2ZVolI8FGx6yIKfLwWnj+eR2dfdMx0tUX4F6mRi4N
zGmhhIIHtViAMf59tBL7az26C8DGfX0p4oECpKtc86u5bYTbRZ1xrf6t/wFqqgB/
RVqn9IhSfXNZtxBn8G0odR8sPIiBxJKvkLMDKoAEeErwd0yqnr8FplskFuPn0FY5
N7n7dj5cHoSUtSAkM6bHCFY+XVtUoy6xisTAobajHvU3e2cDVKizC/ocUbHbTJgh
nevnzyTtKL2w820PDmI7plFN3wR3epd4kTAP5KT196Pjwjg+Dqgt2OnGAafKr+Qr
o2cdIF5MbULVkux4RKzpNKaoDtrnvC1jROM5s1R0Lb96dQcS/VwmyX22lKdbbY9F
ij5GZar9JA==
-----END CERTIFICATE-----

7
z_starting_files/brightdata_proxy_ca/Old Certficate - DEPRECATED, DONT USE/README.txt Normal file
View File

@ -0,0 +1,7 @@
This SSL certificate will expire in 2026 and is being phased out.
**Please do not use this certificate for new projects**
We recommend that you migrate to the new certificate (available inside the zip file) before the previous one expires.
Keep in mind that the new certificate requires you to use port 33335, meaning your proxy host will be from now on: brd.superproxy.io:33335

20
z_starting_files/brightdata_proxy_ca/README - USE THE NEW CERTIFICATE.txt Normal file
View File

@ -0,0 +1,20 @@
Please use the *new* SSL certificate for all new projects and avoid using the previous certificate. the new certificate is effective from September 2024
IMPORTANT: to use the *new* certificate, you must use port 33335 when connecting to Bright Data's proxy network, even if documentation and sample codes provided mention port 22225. Documentation will gradually be updated to port 33335 as the old certificate is being phased out.
You have a few options to use this certificate:
- If you write code, you can simply load it into your existing code.
- If you are using Windows and third-party tools that allow you to configure proxy settings (like AdsPower, Proxifier, Multilogin, Undetectable etc.) you can install the certificate on your computer using the installer.
Again, remember: when using the new certificate you *MUST use port 33335* when connecting to Bright Data's proxy network, even if documentation and sample codes provided mention port 22225.
Full information regarding how to use the SSL certificate is available on https://docs.brightdata.com/general/account/ssl-certificate
--------------------------------------------------------------------------
The old SSL certificate will expire in 2026 and is being phased out.
**Please DO NOT use the old certificate for new projects**
Again, Keep in mind that the new certificate requires you to use port 33335, meaning your proxy host will be from now on: brd.superproxy.io:33335