Add per-seat trip watchlist, fix cross-URL caching bug, and improve results table UX
The biggest fix: change-detection and the fast-path cache lookup were scoped only by competitor_id, not by which specific trip page was scraped. Since trip-intent matching lets a PM scrape a different URL per run, get_last_hash_for_url() compared each new scrape against whatever page happened to be hashed most recently for that competitor — almost always a different page — so change_detected never settled back to False and the true no-scrape fast path was unreachable in practice. Scoped scrape_runs/products lookups by (competitor_id, url) instead, and threaded a cleaned confirmed_url consistently through needs_refresh(), detect_change(), and analyse_from_cache(). Verified end-to-end with a regression test reproducing the exact reported scenario. New: a per-seat, tier-capped trip watchlist (1 trip/seat at Analyse). The nightly cron now re-scrapes only the specific trips a PM chose to track instead of every competitor's homepage (a page already flagged low-signal by is_generic_homepage()) — bounded, predictable volume, and every resulting alert is about a trip someone actually cares about. Reuses the existing confirmed_url/per-url caching machinery with zero changes needed to jobs.py's job loop. Also fixes a real gap: auth_service.py resolved the caller's active seat only to validate the JWT and then discarded it — every route before this only ever needed tenant_id. get_seat_from_request() now returns it. Also fixed the Firecrawl /map timeout (15s → 30s; a large real site's sitemap fetch took ~21s and was silently read as "no match found"), a job left stuck at "Stopping…" forever after a worker process was killed mid-run (status endpoint now self-heals via the underlying RQ job state), and several results-table issues: hardcoded "G Adventures" in the extraction prompt and a literal "G" in the comments column label (replaced with the real tenant name), non-clickable trip links, a cache-source badge that was silently clipped on most columns, a missing view into the client's own scraped trip data, and bolding for Strengths/Weaknesses/Differences/Pricing in the comments text. 370+ new/updated backend tests; full suite green. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
parent
ad1d7f1815
commit
a25909a459
149
CLAUDE.md
149
CLAUDE.md
|
|
@ -1250,6 +1250,71 @@ competitor with no real match from the batch entirely, instead of being
|
||||||
pressured into typing a low-value guess (like the competitor's own
|
pressured into typing a low-value guess (like the competitor's own
|
||||||
homepage) just to unblock "Confirm & Run."
|
homepage) just to unblock "Confirm & Run."
|
||||||
|
|
||||||
|
### ResultsTable.vue — clickable links, cache-source pill, own-trip column
|
||||||
|
|
||||||
|
- The `link` row (every column, including the own-trip column below) now
|
||||||
|
renders as a real `<a target="_blank" rel="noopener">` instead of
|
||||||
|
plain text, wherever a value is present — opens the actual scraped
|
||||||
|
page in a new tab.
|
||||||
|
- Every column header shows a subtle blue database SVG icon next to the
|
||||||
|
name whenever this run's data came from PocketBase (`path === 'fast'`
|
||||||
|
or `unchanged: true`) — a fresh live scrape gets no icon at all, since
|
||||||
|
that's the default expectation and doesn't need flagging. Gives the PM
|
||||||
|
direct, per-competitor visibility into which path §7's fast/refresh
|
||||||
|
decision actually took, rather than having to infer it. Went through
|
||||||
|
two iterations: first a text badge ("From PocketBase"), which was too
|
||||||
|
noisy and also got silently clipped on any column whose name+badge
|
||||||
|
combination didn't fit the shared `nowrap` + `text-overflow: ellipsis`
|
||||||
|
rule (only ever appeared on whichever competitor had the shortest
|
||||||
|
name) — replaced with an icon-only indicator, and header cells now
|
||||||
|
wrap (name renders in its own `div`) instead of truncating.
|
||||||
|
- A dedicated "🏠 {tenant name}" column renders BEFORE the first
|
||||||
|
competitor column whenever `context['own_trip']` was populated this
|
||||||
|
run (client self-scrape feature, §8's trip-intent matching applied to
|
||||||
|
the tenant's own site) — same row layout, same cache pill (based on
|
||||||
|
`own_trip.cached`, set by `embedding_service.scrape_own_trip()`), same
|
||||||
|
clickable link. Omitted entirely when no own-trip URL was confirmed
|
||||||
|
this run (no website configured, or explicitly skipped) — this was
|
||||||
|
previously invisible in the dashboard even when the feature was
|
||||||
|
working, since `own_trip` was only ever consumed internally by
|
||||||
|
`build_prompt()` and never returned to the frontend at all.
|
||||||
|
`jobs.py`'s `run_research_job()` now includes it as
|
||||||
|
`results['own_trip']` on the persisted `scrape_runs` record, so it
|
||||||
|
also survives into RunHistory restores.
|
||||||
|
- The `comments` row label was hardcoded to `"Suggested changes for
|
||||||
|
G/ Competitive Comments"` — a leftover literal "G" from "G Adventures"
|
||||||
|
(the hackathon reference client), showing every tenant's own table as
|
||||||
|
if it belonged to a company literally named "G". `resultFields.js`'s
|
||||||
|
label is now `"Suggested changes for {company}/ Competitive
|
||||||
|
Comments"`, substituted at render time via `formatLabel(label,
|
||||||
|
companyName)` — used by both `ResultsTable.vue` (display) and
|
||||||
|
`SyncToSheet.vue` (clipboard copy) via a new `companyName` prop each,
|
||||||
|
wired from `AnalysePage.vue`'s `authStore.tenant?.name`.
|
||||||
|
- `analyse_from_cache()`'s (§7 fast path) narrative prompt used to dump
|
||||||
|
the raw `context` dict (Python repr) into the prompt with no priority
|
||||||
|
framing — `own_trip`'s presence was easy for the model to under-weight
|
||||||
|
against the typed destination/duration/style fields sitting right next
|
||||||
|
to it in the same blob. It now explicitly builds a `client_context_desc`
|
||||||
|
string that's grounded in `context['own_trip']` (the tenant's own REAL
|
||||||
|
extracted trip) FIRST when present, falling back to the PM's typed
|
||||||
|
form fields only when no own-trip data was returned for this run —
|
||||||
|
matching `build_prompt()`'s existing own_trip-first priority on the
|
||||||
|
refresh path, so both paths ground `comments` the same way.
|
||||||
|
- The `comments` cell bolds every comparison-section label
|
||||||
|
`build_prompt()`'s own JSON schema instruction asks the LLM for —
|
||||||
|
"(Key) Differences:", "Strength(s):", "Weakness(es):", "Pricing
|
||||||
|
(Position):" (case-insensitive) — wherever they occur in every
|
||||||
|
column's comments text, so a PM can skim multiple competitors' analysis
|
||||||
|
at a glance. Includes the own-trip column, though that one is always
|
||||||
|
blank since `build_own_trip_prompt()` never asks for a `comments`
|
||||||
|
field (nothing to compare against at extraction time). `ResultsTable.vue`'s
|
||||||
|
`boldKeywords()` HTML-escapes the text first, then re-inserts only its
|
||||||
|
own `<strong>` tags around an exact keyword match, so `v-html` can't be
|
||||||
|
used to inject arbitrary markup even though the underlying text
|
||||||
|
originates from an LLM extraction. Purely a display change —
|
||||||
|
`SyncToSheet.vue`'s clipboard copy (plain text, pasted into the real
|
||||||
|
Sheet template) is untouched.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 8. Trip-Intent Matching
|
## 8. Trip-Intent Matching
|
||||||
|
|
@ -3015,21 +3080,89 @@ GET /research/history/<id> — full results for a specific run by job_id
|
||||||
|
|
||||||
## 15. n8n Workflows
|
## 15. n8n Workflows
|
||||||
|
|
||||||
### Daily scrape (6am every day)
|
### Daily scrape (6am every day) — targets the trip watchlist, not competitor homepages
|
||||||
|
|
||||||
```
|
```
|
||||||
Schedule Trigger (06:00 AST)
|
Schedule Trigger (06:00 AST)
|
||||||
→ For each active tenant:
|
→ For each active tenant:
|
||||||
→ Query PocketBase for competitors where:
|
→ Query PocketBase for that tenant's trip_watchlist items (every
|
||||||
change_detected = true OR last_scraped > 24hrs ago
|
seat's watched trips — no staleness pre-filter; every watched
|
||||||
→ HTTP POST /internal/scrape/{tenant_id} with filtered competitor list
|
trip is sent every night)
|
||||||
→ RQ job queued → Playwright scrape → AI extraction
|
→ HTTP POST /internal/scrape/{tenant_id} with
|
||||||
→ PocketBase updated (products, price_history)
|
competitors: [{ id: competitor_id, url: watched_url }, ...] —
|
||||||
→ change_detected reset to false
|
one entry per watched trip (a competitor can appear more than
|
||||||
→ last_scraped updated to now
|
once if two seats watch two different trips on it)
|
||||||
|
→ RQ job queued → jobs.py's confirmed_url threading scrapes the
|
||||||
|
SPECIFIC watched url per entry (analysis_service.needs_refresh()'s
|
||||||
|
per-url freshness check decides fast-vs-refresh internally,
|
||||||
|
exactly as a manual dashboard run would)
|
||||||
|
→ PocketBase updated (products, price_history) for that specific trip
|
||||||
→ price_history diff calculated automatically
|
→ price_history diff calculated automatically
|
||||||
→ is_new flag set on new products automatically
|
→ is_new flag set on new products automatically
|
||||||
```
|
```
|
||||||
|
|
||||||
|
This replaced an earlier design that queried `competitors` directly
|
||||||
|
(`change_detected = true || last_scraped > 24hrs`) and re-scraped each
|
||||||
|
competitor's root `website` every night — a page
|
||||||
|
`services/url_utils.py`'s `is_generic_homepage()` already flags as
|
||||||
|
low-signal and non-trip-specific, with no natural bound on nightly
|
||||||
|
volume. The trip watchlist (below) replaces it with a small, predictable
|
||||||
|
ceiling and alerts that are always about a trip a PM explicitly chose to
|
||||||
|
track.
|
||||||
|
|
||||||
|
### Trip watchlist — per-seat, tier-capped nightly monitoring
|
||||||
|
|
||||||
|
Each **active seat** (not the tenant as a whole) can pin a small number
|
||||||
|
of specific already-analyzed competitor trips to a personal watchlist —
|
||||||
|
`trip_watchlist` PocketBase collection (migration 008), one row per
|
||||||
|
watched trip, `seat_id` + `competitor_id` + `url` + `trip_name`. The cap
|
||||||
|
is per-seat and tier-based
|
||||||
|
(`services/watchlist_service.py`'s `WATCHLIST_CAP_PER_SEAT`):
|
||||||
|
|
||||||
|
```python
|
||||||
|
WATCHLIST_CAP_PER_SEAT = {
|
||||||
|
'analyse': 1, # a fully-seated ($349, 5 seats) tenant tops out
|
||||||
|
# at 5 real trip scrapes/night — same ceiling as
|
||||||
|
# scraping every competitor once, but every
|
||||||
|
# alert is now about a trip a PM chose to track
|
||||||
|
'discover': 2, # placeholder — Discover isn't built yet (§27)
|
||||||
|
'intelligence': 3, # placeholder — layered on top of that tier's
|
||||||
|
'enterprise': 3, # full catalogue auto-discovery, not instead of it
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Routes** (`api.py`): `GET /watchlist` returns every watched item for
|
||||||
|
the session's tenant (all seats', with the watching seat's email
|
||||||
|
denormalized for display — visibility is tenant-wide even though the
|
||||||
|
cap is per-seat); `POST /watchlist` (`{competitor_id, url, trip_name}`)
|
||||||
|
resolves the calling seat via `auth_service.get_seat_from_request()` and
|
||||||
|
enforces the cap, 400 `watchlist_full` with a clear message on breach;
|
||||||
|
`DELETE /watchlist/<id>` removes an item — any active seat on the tenant
|
||||||
|
may remove any item, no per-seat ownership lock (matching this app's
|
||||||
|
existing no-fine-grained-permissions convention).
|
||||||
|
|
||||||
|
`auth_service.get_tenant_id_from_request()` used to look up the caller's
|
||||||
|
active seat purely to validate the JWT, then discard it — every
|
||||||
|
authenticated route before this feature only ever needed `tenant_id`.
|
||||||
|
`get_seat_from_request()` now returns `(tenant_id, seat)` instead, with
|
||||||
|
`get_tenant_id_from_request()` kept as a thin backward-compatible
|
||||||
|
wrapper around it.
|
||||||
|
|
||||||
|
**Frontend**: `ResultsTable.vue` gets a "☆ Watch" / "★ Watching" toggle
|
||||||
|
per competitor column (not shown on the own-trip column — watching your
|
||||||
|
own trip isn't meaningful), using the row-derived `link`/`tripName`
|
||||||
|
values so it works identically for fast-path and refresh-path results.
|
||||||
|
`ActivityPage.vue`'s new `WatchlistCard.vue` lists every tenant item
|
||||||
|
(trip name, watched-by email, remove button) plus an "X of Y slots
|
||||||
|
used" indicator for the current seat (mirrors
|
||||||
|
`WATCHLIST_CAP_PER_SEAT` client-side for display only — the server
|
||||||
|
enforces the real cap).
|
||||||
|
|
||||||
|
Not built in this pass: no ad-hoc "watch any URL" entry (v1 only lets a
|
||||||
|
PM watch a trip that already appears in a completed analysis result),
|
||||||
|
and Discover/Intelligence's actual cap numbers are placeholders, not
|
||||||
|
load-bearing until those tiers are built.
|
||||||
|
|
||||||
### Battlecard regeneration (after scrape)
|
### Battlecard regeneration (after scrape)
|
||||||
```
|
```
|
||||||
PocketBase webhook on scrape_runs status=complete
|
PocketBase webhook on scrape_runs status=complete
|
||||||
|
|
|
||||||
195
backend/api.py
195
backend/api.py
|
|
@ -1,8 +1,10 @@
|
||||||
import os
|
import os
|
||||||
import logging
|
import logging
|
||||||
import time
|
import time
|
||||||
|
from datetime import datetime, timezone
|
||||||
from functools import wraps
|
from functools import wraps
|
||||||
from urllib.parse import urlparse
|
from urllib.parse import urlparse
|
||||||
|
from concurrent.futures import ThreadPoolExecutor
|
||||||
|
|
||||||
import requests
|
import requests
|
||||||
from flask import Blueprint, request, jsonify, Response
|
from flask import Blueprint, request, jsonify, Response
|
||||||
|
|
@ -17,6 +19,7 @@ from repositories.scrape_repository import scrape_repository
|
||||||
from repositories.battlecard_repository import battlecard_repository
|
from repositories.battlecard_repository import battlecard_repository
|
||||||
from repositories.comparable_repository import comparable_repository
|
from repositories.comparable_repository import comparable_repository
|
||||||
from repositories.alert_repository import alert_repository
|
from repositories.alert_repository import alert_repository
|
||||||
|
from repositories.watchlist_repository import watchlist_repository
|
||||||
|
|
||||||
from services import licence_service
|
from services import licence_service
|
||||||
from services import trip_finder_service
|
from services import trip_finder_service
|
||||||
|
|
@ -28,6 +31,7 @@ from services import auth_service
|
||||||
from services import analysis_service
|
from services import analysis_service
|
||||||
from services import email_service
|
from services import email_service
|
||||||
from services import alert_service
|
from services import alert_service
|
||||||
|
from services import watchlist_service
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
@ -197,7 +201,20 @@ def confirm_password_reset():
|
||||||
@limiter.limit('20 per hour')
|
@limiter.limit('20 per hour')
|
||||||
@require_api_key
|
@require_api_key
|
||||||
def find_urls():
|
def find_urls():
|
||||||
"""Runs Firecrawl /map concurrently per selected competitor to find matching trip URLs."""
|
"""
|
||||||
|
Runs Firecrawl /map concurrently per selected competitor to find
|
||||||
|
matching trip URLs, and — when the caller supplies tenant_id and that
|
||||||
|
tenant has a website on file — also finds the tenant's own matching
|
||||||
|
trip page the same way (client self-scrape feature). tenant_id is
|
||||||
|
optional and not in `required`; omitting it (or a tenant with no
|
||||||
|
website configured) simply omits `own_trip` from the response,
|
||||||
|
which the frontend treats as "not available for this tenant."
|
||||||
|
|
||||||
|
The own-trip lookup is kicked off in its own thread BEFORE the
|
||||||
|
competitor batch, then joined after — so it overlaps with the
|
||||||
|
competitor ThreadPoolExecutor instead of adding its own /map call's
|
||||||
|
latency on top of the whole competitor batch sequentially.
|
||||||
|
"""
|
||||||
data = request.json or {}
|
data = request.json or {}
|
||||||
required = ['destination', 'duration', 'travel_style', 'competitor_ids']
|
required = ['destination', 'duration', 'travel_style', 'competitor_ids']
|
||||||
missing = [f for f in required if f not in data]
|
missing = [f for f in required if f not in data]
|
||||||
|
|
@ -207,10 +224,30 @@ def find_urls():
|
||||||
competitors = [
|
competitors = [
|
||||||
c for c in (competitor_repository.get_by_id(cid) for cid in data['competitor_ids']) if c
|
c for c in (competitor_repository.get_by_id(cid) for cid in data['competitor_ids']) if c
|
||||||
]
|
]
|
||||||
|
|
||||||
|
own_trip_future = None
|
||||||
|
own_trip_executor = None
|
||||||
|
tenant_id = data.get('tenant_id')
|
||||||
|
if tenant_id:
|
||||||
|
tenant = tenant_repository.get_by_id(tenant_id)
|
||||||
|
website = (tenant or {}).get('website')
|
||||||
|
if website:
|
||||||
|
own_trip_executor = ThreadPoolExecutor(max_workers=1)
|
||||||
|
own_trip_future = own_trip_executor.submit(
|
||||||
|
trip_finder_service.find_own_trip_url,
|
||||||
|
website, data['destination'], data['duration'], data['travel_style']
|
||||||
|
)
|
||||||
|
|
||||||
matches = trip_finder_service.find_all_competitor_urls(
|
matches = trip_finder_service.find_all_competitor_urls(
|
||||||
competitors, data['destination'], data['duration'], data['travel_style']
|
competitors, data['destination'], data['duration'], data['travel_style']
|
||||||
)
|
)
|
||||||
return jsonify({'matches': matches})
|
|
||||||
|
own_trip = None
|
||||||
|
if own_trip_future:
|
||||||
|
own_trip = own_trip_future.result()
|
||||||
|
own_trip_executor.shutdown()
|
||||||
|
|
||||||
|
return jsonify({'matches': matches, 'own_trip': own_trip})
|
||||||
|
|
||||||
|
|
||||||
@api_bp.route('/research', methods=['POST'])
|
@api_bp.route('/research', methods=['POST'])
|
||||||
|
|
@ -281,6 +318,10 @@ def research_status(job_id):
|
||||||
run = scrape_repository.get_by_id(job_id)
|
run = scrape_repository.get_by_id(job_id)
|
||||||
if not run:
|
if not run:
|
||||||
return error_response('not_found', 'Unknown job_id', 404)
|
return error_response('not_found', 'Unknown job_id', 404)
|
||||||
|
|
||||||
|
if run.get('status') in ('running', 'cancelling'):
|
||||||
|
run = _reconcile_orphaned_job(job_id, run)
|
||||||
|
|
||||||
return jsonify({
|
return jsonify({
|
||||||
'status': run.get('status'),
|
'status': run.get('status'),
|
||||||
'tab_type': run.get('tab_type'),
|
'tab_type': run.get('tab_type'),
|
||||||
|
|
@ -293,6 +334,65 @@ def research_status(job_id):
|
||||||
})
|
})
|
||||||
|
|
||||||
|
|
||||||
|
def _reconcile_orphaned_job(job_id, run):
|
||||||
|
"""
|
||||||
|
Self-heals a scrape_runs row stuck at status='running' (or the PM
|
||||||
|
having requested cancellation, cancel_requested=True, while it's
|
||||||
|
still nominally 'running') because the RQ worker process actually
|
||||||
|
handling it died outright — e.g. `docker compose up -d --build
|
||||||
|
bettersight-worker` recreating the container mid-job — before
|
||||||
|
run_research_job()'s own try/except/finally ever got to write a
|
||||||
|
terminal status. Cancellation is cooperative (jobs.cancel_research_job()'s
|
||||||
|
docstring): a running job only notices cancel_requested at its next
|
||||||
|
between-competitor checkpoint, which assumes the SAME process stays
|
||||||
|
alive to reach it. A hard-killed worker never reaches that checkpoint
|
||||||
|
again, so without this check the dashboard polls a job that will
|
||||||
|
never change state, showing "Stopping…" (or a stuck progress bar)
|
||||||
|
forever. Checked lazily here, on the exact endpoint the dashboard
|
||||||
|
already polls every 3s (analysis_store.js), rather than a separate
|
||||||
|
background sweep.
|
||||||
|
|
||||||
|
Data flow:
|
||||||
|
job_id → fetch the underlying RQ Job from Redis →
|
||||||
|
RQ status is 'started'/'queued'/'deferred'/'scheduled' → still
|
||||||
|
genuinely in flight → run returned unchanged →
|
||||||
|
RQ status is 'failed', 'finished', or the job record is missing
|
||||||
|
entirely (NoSuchJobError — expired from Redis, or the worker
|
||||||
|
was killed before RQ even wrote a heartbeat) → this job will
|
||||||
|
never update scrape_runs itself anymore → mark it 'cancelled'
|
||||||
|
(cancel_requested was set) or 'failed' (otherwise) now →
|
||||||
|
updated run record returned
|
||||||
|
"""
|
||||||
|
import redis
|
||||||
|
from jobs import redis_conn
|
||||||
|
from rq.job import Job
|
||||||
|
from rq.exceptions import NoSuchJobError
|
||||||
|
|
||||||
|
try:
|
||||||
|
rq_status = Job.fetch(job_id, connection=redis_conn).get_status()
|
||||||
|
except NoSuchJobError:
|
||||||
|
rq_status = None
|
||||||
|
except redis.exceptions.RedisError as e:
|
||||||
|
# Can't reach Redis to check — don't guess. Leave the row alone
|
||||||
|
# and let a later poll (once Redis is reachable again) decide,
|
||||||
|
# rather than risk marking a genuinely still-running job failed.
|
||||||
|
logger.warning(f'Could not reach Redis to reconcile job {job_id}: {str(e)}')
|
||||||
|
return run
|
||||||
|
|
||||||
|
if rq_status in ('started', 'queued', 'deferred', 'scheduled'):
|
||||||
|
return run # genuinely still in flight — nothing to reconcile
|
||||||
|
|
||||||
|
new_status = 'cancelled' if run.get('cancel_requested') else 'failed'
|
||||||
|
logger.warning(
|
||||||
|
f'Job {job_id} orphaned (RQ status: {rq_status}) — reconciling scrape_runs to "{new_status}"'
|
||||||
|
)
|
||||||
|
return scrape_repository.update(job_id, {
|
||||||
|
'status': new_status,
|
||||||
|
'error_log': run.get('error_log') or 'Job was interrupted before it could finish (worker restarted or crashed).',
|
||||||
|
'completed_at': datetime.now(timezone.utc).isoformat(),
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
@api_bp.route('/research/cancel/<job_id>', methods=['POST'])
|
@api_bp.route('/research/cancel/<job_id>', methods=['POST'])
|
||||||
@limiter.limit('20 per hour')
|
@limiter.limit('20 per hour')
|
||||||
@require_api_key
|
@require_api_key
|
||||||
|
|
@ -454,6 +554,75 @@ def mark_alert_read(alert_id):
|
||||||
return jsonify(alert_repository.mark_read(alert_id))
|
return jsonify(alert_repository.mark_read(alert_id))
|
||||||
|
|
||||||
|
|
||||||
|
# ── TRIP WATCHLIST ───────────────────────────────────────────────────
|
||||||
|
# Per-seat, tier-capped list of specific trips the nightly n8n cron
|
||||||
|
# re-scrapes instead of each competitor's root website — see
|
||||||
|
# services/watchlist_service.py's WATCHLIST_CAP_PER_SEAT and
|
||||||
|
# n8n/workflows/daily_scrape_cron.json.
|
||||||
|
|
||||||
|
@api_bp.route('/watchlist', methods=['GET'])
|
||||||
|
@require_api_key
|
||||||
|
def get_watchlist():
|
||||||
|
"""
|
||||||
|
Returns every watched item for the session's tenant (every seat's,
|
||||||
|
not just the caller's) — the cap is per-seat, but visibility is
|
||||||
|
tenant-wide so teammates can see what each other is tracking.
|
||||||
|
"""
|
||||||
|
tenant_id = auth_service.get_tenant_id_from_request(request)
|
||||||
|
if not tenant_id:
|
||||||
|
return error_response('unauthorized', 'Could not resolve tenant from session', 401)
|
||||||
|
return jsonify(watchlist_service.list_watchlist_for_tenant(tenant_id))
|
||||||
|
|
||||||
|
|
||||||
|
@api_bp.route('/watchlist', methods=['POST'])
|
||||||
|
@limiter.limit('30 per hour')
|
||||||
|
@require_api_key
|
||||||
|
def add_watchlist_item():
|
||||||
|
"""
|
||||||
|
Pins a specific competitor trip (from a completed analysis result)
|
||||||
|
to the calling seat's personal watchlist. 400 with a clear message
|
||||||
|
if that seat is already at WATCHLIST_CAP_PER_SEAT for its tier.
|
||||||
|
"""
|
||||||
|
tenant_id, seat = auth_service.get_seat_from_request(request)
|
||||||
|
if not tenant_id or not seat:
|
||||||
|
return error_response('unauthorized', 'Could not resolve seat from session', 401)
|
||||||
|
|
||||||
|
data = request.json or {}
|
||||||
|
required = ['competitor_id', 'url']
|
||||||
|
missing = [f for f in required if not data.get(f)]
|
||||||
|
if missing:
|
||||||
|
return error_response('missing_field', f'Missing fields: {", ".join(missing)}', 400)
|
||||||
|
|
||||||
|
competitor = competitor_repository.get_by_id(data['competitor_id'])
|
||||||
|
if not competitor or competitor.get('tenant_id') != tenant_id:
|
||||||
|
return error_response('not_found', 'Unknown competitor', 404)
|
||||||
|
|
||||||
|
tenant = tenant_repository.get_by_id(tenant_id)
|
||||||
|
result = watchlist_service.add_to_watchlist(
|
||||||
|
tenant, seat, data['competitor_id'], data['url'], data.get('trip_name')
|
||||||
|
)
|
||||||
|
if not result['success']:
|
||||||
|
return error_response('watchlist_full', result['reason'], 400)
|
||||||
|
return jsonify(result['item']), 201
|
||||||
|
|
||||||
|
|
||||||
|
@api_bp.route('/watchlist/<item_id>', methods=['DELETE'])
|
||||||
|
@limiter.limit('30 per hour')
|
||||||
|
@require_api_key
|
||||||
|
def remove_watchlist_item(item_id):
|
||||||
|
"""Removes a watched trip — any active seat on the tenant may remove any item."""
|
||||||
|
tenant_id = auth_service.get_tenant_id_from_request(request)
|
||||||
|
if not tenant_id:
|
||||||
|
return error_response('unauthorized', 'Could not resolve tenant from session', 401)
|
||||||
|
|
||||||
|
item = watchlist_repository.get_by_id(item_id)
|
||||||
|
if not item or item.get('tenant_id') != tenant_id:
|
||||||
|
return error_response('not_found', 'Unknown watchlist item', 404)
|
||||||
|
|
||||||
|
watchlist_service.remove_from_watchlist(item_id)
|
||||||
|
return jsonify({'status': 'ok'})
|
||||||
|
|
||||||
|
|
||||||
# ── APPS SCRIPT SHEET SYNC ──────────────────────────────────────────
|
# ── APPS SCRIPT SHEET SYNC ──────────────────────────────────────────
|
||||||
# No dedicated routes here per CLAUDE.md §15 — the Sheet-initiated pull
|
# No dedicated routes here per CLAUDE.md §15 — the Sheet-initiated pull
|
||||||
# uses GET /research/history/latest (above) and GET
|
# uses GET /research/history/latest (above) and GET
|
||||||
|
|
@ -656,6 +825,28 @@ def update_timezone():
|
||||||
return jsonify({'timezone': updated['timezone']})
|
return jsonify({'timezone': updated['timezone']})
|
||||||
|
|
||||||
|
|
||||||
|
@api_bp.route('/account/website', methods=['PATCH'])
|
||||||
|
@require_api_key
|
||||||
|
def update_website():
|
||||||
|
"""
|
||||||
|
Sets or changes the tenant's own storefront URL — used by the
|
||||||
|
client self-scrape feature to find/scrape the tenant's own matching
|
||||||
|
trip page the same way competitor pages are matched/scraped. Unlike
|
||||||
|
/account/timezone, this is freely editable at any time (a PM may
|
||||||
|
reasonably correct or change their site later), not first-write-wins.
|
||||||
|
"""
|
||||||
|
tenant_id = auth_service.get_tenant_id_from_request(request)
|
||||||
|
if not tenant_id:
|
||||||
|
return error_response('unauthorized', 'Could not resolve tenant from session', 401)
|
||||||
|
|
||||||
|
website = (request.json or {}).get('website')
|
||||||
|
if not website:
|
||||||
|
return error_response('missing_field', 'website is required', 400)
|
||||||
|
|
||||||
|
updated = tenant_repository.update(tenant_id, {'website': website})
|
||||||
|
return jsonify({'website': updated['website']})
|
||||||
|
|
||||||
|
|
||||||
@api_bp.route('/account/onboarding', methods=['PATCH'])
|
@api_bp.route('/account/onboarding', methods=['PATCH'])
|
||||||
@require_api_key
|
@require_api_key
|
||||||
def update_onboarding():
|
def update_onboarding():
|
||||||
|
|
|
||||||
|
|
@ -5,7 +5,7 @@
|
||||||
|
|
||||||
|
|
||||||
def build_prompt(competitor_name, page_data, product_name,
|
def build_prompt(competitor_name, page_data, product_name,
|
||||||
destination, duration, travel_style, company_name=None, is_ngs=False):
|
destination, duration, travel_style, company_name=None, is_ngs=False, own_trip=None):
|
||||||
"""
|
"""
|
||||||
Builds the full extraction prompt for one competitor page, embedding
|
Builds the full extraction prompt for one competitor page, embedding
|
||||||
the scraped page content directly into the returned template.
|
the scraped page content directly into the returned template.
|
||||||
|
|
@ -18,11 +18,23 @@ def build_prompt(competitor_name, page_data, product_name,
|
||||||
to a generic placeholder only for the dry-run /preview-prompt route,
|
to a generic placeholder only for the dry-run /preview-prompt route,
|
||||||
which has no tenant context to draw a real name from.
|
which has no tenant context to draw a real name from.
|
||||||
|
|
||||||
|
own_trip is the tenant's own trip, ACTUALLY extracted from their own
|
||||||
|
site by embedding_service.scrape_own_trip() (see jobs.py's
|
||||||
|
context['own_trip']) — not just the PM's typed destination/duration/
|
||||||
|
style strings. When present, the comparison block and the `comments`
|
||||||
|
field instruction reference these real facts instead of only the
|
||||||
|
typed intent, so the LLM compares against ground truth rather than a
|
||||||
|
guess. None (the default) preserves the exact prior behavior for
|
||||||
|
every existing call site and for tenants who haven't confirmed an
|
||||||
|
own-trip URL for this run.
|
||||||
|
|
||||||
Data flow:
|
Data flow:
|
||||||
competitor_name + page_data (scraped text/tables/url) + client
|
competitor_name + page_data (scraped text/tables/url) + client
|
||||||
product context (name/destination/duration/style/company) + is_ngs
|
product context (name/destination/duration/style/company) + is_ngs
|
||||||
flag → NGS-specific field block appended if is_ngs →
|
flag + optional own_trip (real extracted client data) →
|
||||||
full prompt string returned, ready to send to core.extractor
|
NGS-specific field block appended if is_ngs → own_trip block
|
||||||
|
appended if present → full prompt string returned, ready to send
|
||||||
|
to core.extractor
|
||||||
"""
|
"""
|
||||||
company_name = company_name or 'the client'
|
company_name = company_name or 'the client'
|
||||||
ngs_fields = '''
|
ngs_fields = '''
|
||||||
|
|
@ -36,6 +48,24 @@ def build_prompt(competitor_name, page_data, product_name,
|
||||||
|
|
||||||
url = page_data['url'] if page_data else '[COMPETITOR URL]'
|
url = page_data['url'] if page_data else '[COMPETITOR URL]'
|
||||||
|
|
||||||
|
own_trip_block = ''
|
||||||
|
comments_target = f'{company_name}'
|
||||||
|
if own_trip:
|
||||||
|
own_trip_name = own_trip.get('tripName') or product_name
|
||||||
|
own_trip_price = own_trip.get('majorityPrice')
|
||||||
|
own_trip_duration = own_trip.get('duration') or duration
|
||||||
|
own_trip_activities = own_trip.get('activities')
|
||||||
|
own_trip_block = f'''
|
||||||
|
|
||||||
|
{company_name}'s OWN trip, actually extracted from their own site (real
|
||||||
|
data, not a typed description) — use this as the ground truth for comparison:
|
||||||
|
- Trip name: {own_trip_name}
|
||||||
|
- Price: {own_trip_price if own_trip_price is not None else 'not extracted'} USD (majority price)
|
||||||
|
- Duration: {own_trip_duration} days
|
||||||
|
- Service level: {own_trip.get('serviceLevel') or 'not extracted'}
|
||||||
|
- Activities: {own_trip_activities or 'not extracted'}'''
|
||||||
|
comments_target = f"{company_name}'s own {own_trip_name} (${own_trip_price}, {own_trip_duration} days: {own_trip_activities})"
|
||||||
|
|
||||||
return f'''You are a travel industry analyst helping {company_name} research competitors.
|
return f'''You are a travel industry analyst helping {company_name} research competitors.
|
||||||
Be concise — use short phrases not full sentences for all fields except comments.
|
Be concise — use short phrases not full sentences for all fields except comments.
|
||||||
Read the ENTIRE page content carefully before extracting any fields.
|
Read the ENTIRE page content carefully before extracting any fields.
|
||||||
|
|
@ -48,7 +78,7 @@ This is the specific page selected as most comparable to:
|
||||||
- {company_name} Product: {product_name}
|
- {company_name} Product: {product_name}
|
||||||
- Destination: {destination}
|
- Destination: {destination}
|
||||||
- Duration: ~{duration} days
|
- Duration: ~{duration} days
|
||||||
- {company_name} Travel Style: {travel_style}
|
- {company_name} Travel Style: {travel_style}{own_trip_block}
|
||||||
|
|
||||||
PAGE CONTENT:
|
PAGE CONTENT:
|
||||||
---
|
---
|
||||||
|
|
@ -98,7 +128,110 @@ Return ONLY a valid JSON object. No markdown, no code fences, no explanation.
|
||||||
"majorityPrice": standard adult USD price as number or null,
|
"majorityPrice": standard adult USD price as number or null,
|
||||||
"priceLow": lowest standard USD price as number or null,
|
"priceLow": lowest standard USD price as number or null,
|
||||||
"priceHigh": highest standard USD price as number or null,
|
"priceHigh": highest standard USD price as number or null,
|
||||||
"comments": "analysis: key differences from {company_name}, strengths, weaknesses, pricing position",
|
"comments": "analysis: key differences from {comments_target}, strengths, weaknesses, pricing position",
|
||||||
|
"dateUsed": "specific departure date used for majority price",
|
||||||
|
"audPrice": standard AUD price as number or null,
|
||||||
|
"cadPrice": standard CAD price as number or null,
|
||||||
|
"eurPrice": standard EUR price as number or null,
|
||||||
|
"gbpPrice": standard GBP price as number or null
|
||||||
|
}}'''
|
||||||
|
|
||||||
|
|
||||||
|
def build_own_trip_prompt(page_data, product_name, destination, duration,
|
||||||
|
travel_style, company_name, is_ngs=False):
|
||||||
|
"""
|
||||||
|
Builds the extraction prompt for the TENANT'S OWN trip page —
|
||||||
|
embedding_service.scrape_own_trip() calls this, not build_prompt(),
|
||||||
|
because the framing is different: this is not "researching a
|
||||||
|
competitor," it's extracting structured data from the client's own
|
||||||
|
product so later competitor extractions (build_prompt()'s own_trip
|
||||||
|
param) have real ground truth to compare against instead of typed
|
||||||
|
strings.
|
||||||
|
|
||||||
|
Shares the same pricing/meals extraction rules as build_prompt() so
|
||||||
|
the numbers are directly comparable, but the returned JSON schema
|
||||||
|
drops `relevancy` (relevance to itself is meaningless) and `comments`
|
||||||
|
(nothing to compare against yet at this stage).
|
||||||
|
|
||||||
|
Data flow:
|
||||||
|
page_data (scraped from the tenant's own confirmed/matched URL) +
|
||||||
|
typed intent (product_name/destination/duration/travel_style) +
|
||||||
|
company_name + is_ngs → full prompt string, ready to send to
|
||||||
|
core.extractor.extract_with_openrouter()
|
||||||
|
"""
|
||||||
|
ngs_fields = '''
|
||||||
|
"exclusiveAccess": "exclusive access or special unique inclusions",
|
||||||
|
"groupLeader": "group leader and/or local expert description",
|
||||||
|
"sustainability": "sustainability inclusions or null",''' if is_ngs else ''
|
||||||
|
|
||||||
|
content = page_data['text'] if page_data else '[PAGE CONTENT WILL BE INSERTED HERE]'
|
||||||
|
if page_data and page_data.get('tables'):
|
||||||
|
content += f'\n\nTABLE DATA:\n{page_data["tables"]}'
|
||||||
|
|
||||||
|
url = page_data['url'] if page_data else '[OWN TRIP URL]'
|
||||||
|
|
||||||
|
return f'''You are a travel industry analyst extracting structured trip data
|
||||||
|
from {company_name}'s own tour page — this is {company_name}'s own product,
|
||||||
|
not a competitor's. Be concise — use short phrases, not full sentences.
|
||||||
|
Read the ENTIRE page content carefully before extracting any fields.
|
||||||
|
Pay particular attention to itinerary sections, departure tables, and pricing grids
|
||||||
|
which may appear later in the content.
|
||||||
|
|
||||||
|
The content below was scraped from {company_name}'s own tour page at: {url}
|
||||||
|
|
||||||
|
This is the page selected as matching:
|
||||||
|
- Product: {product_name}
|
||||||
|
- Destination: {destination}
|
||||||
|
- Duration: ~{duration} days
|
||||||
|
- Travel Style: {travel_style}
|
||||||
|
|
||||||
|
PAGE CONTENT:
|
||||||
|
---
|
||||||
|
{content}
|
||||||
|
---
|
||||||
|
|
||||||
|
CRITICAL PRICING RULES:
|
||||||
|
- majorityPrice: Standard/regular adult price for the most common departure.
|
||||||
|
IGNORE sale, promotional, early bird, or "was/now" pricing. Use non-discounted rack rate only.
|
||||||
|
- Prices may appear as "from $X", "per person from $X", or inside a departures table or pricing grid.
|
||||||
|
Extract the most commonly listed standard adult price regardless of how it is labelled on the page.
|
||||||
|
- priceLow: Lowest standard price across all departures shown (not sale price).
|
||||||
|
- priceHigh: Highest standard price across all departures shown (not sale price).
|
||||||
|
- If a site requires clicking "see more departures" to show all dates, note this in seasonality
|
||||||
|
and use only prices visible on the main listing page.
|
||||||
|
- Multi-currency: only fill if the price is explicitly listed in that currency on the page.
|
||||||
|
Do not convert. If the page shows GBP only, fill gbpPrice and leave USD fields null.
|
||||||
|
|
||||||
|
MEALS RULE:
|
||||||
|
- meals: Return the TOTAL number of included meals as a single integer only. No text description.
|
||||||
|
- Meals may be listed individually per day in the itinerary section
|
||||||
|
(e.g. "Day 1: Breakfast", "Day 3: Breakfast and Dinner").
|
||||||
|
Count every individual meal mention across all itinerary days.
|
||||||
|
Do not rely on a summary box — read through the full itinerary to count.
|
||||||
|
- Count each meal type separately: 1 breakfast + 1 lunch + 1 dinner = 3 meals.
|
||||||
|
|
||||||
|
Return ONLY a valid JSON object. No markdown, no code fences, no explanation.
|
||||||
|
|
||||||
|
{{
|
||||||
|
"company": "{company_name}",
|
||||||
|
"link": "{url}",
|
||||||
|
"tripName": "exact trip name from the page",
|
||||||
|
"tripCode": "trip code if shown, otherwise null",
|
||||||
|
"serviceLevel": "accommodation service level",
|
||||||
|
"groupSize": max group size as integer,
|
||||||
|
"duration": number of days as integer,
|
||||||
|
"meals": total included meals as integer,{ngs_fields}
|
||||||
|
"startLocation": "departure city",
|
||||||
|
"endLocation": "end city",
|
||||||
|
"departures": approximate departures per year as integer or null,
|
||||||
|
"seasonality": "price bands by season. Note if more departures are hidden behind a button.",
|
||||||
|
"startDays": "departure days of week",
|
||||||
|
"targetAudience": "target traveller",
|
||||||
|
"hotels": "hotel types or properties",
|
||||||
|
"activities": "key included activities",
|
||||||
|
"majorityPrice": standard adult USD price as number or null,
|
||||||
|
"priceLow": lowest standard USD price as number or null,
|
||||||
|
"priceHigh": highest standard USD price as number or null,
|
||||||
"dateUsed": "specific departure date used for majority price",
|
"dateUsed": "specific departure date used for majority price",
|
||||||
"audPrice": standard AUD price as number or null,
|
"audPrice": standard AUD price as number or null,
|
||||||
"cadPrice": standard CAD price as number or null,
|
"cadPrice": standard CAD price as number or null,
|
||||||
|
|
|
||||||
|
|
@ -11,7 +11,7 @@ from rq.exceptions import NoSuchJobError
|
||||||
from repositories.scrape_repository import scrape_repository
|
from repositories.scrape_repository import scrape_repository
|
||||||
from repositories.competitor_repository import competitor_repository
|
from repositories.competitor_repository import competitor_repository
|
||||||
from repositories.tenant_repository import tenant_repository
|
from repositories.tenant_repository import tenant_repository
|
||||||
from services import analysis_service, alert_service, battlecard_service
|
from services import analysis_service, alert_service, battlecard_service, embedding_service
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
@ -137,6 +137,22 @@ def run_research_job(job_id, data):
|
||||||
'force_refresh': data.get('force_refresh', False),
|
'force_refresh': data.get('force_refresh', False),
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# Client self-scrape — if the PM confirmed/swapped a URL for the
|
||||||
|
# tenant's own trip page (UrlConfirmation.vue's "Your trip page" row),
|
||||||
|
# scrape and extract it once here so every competitor's build_prompt()
|
||||||
|
# call below gets real ground-truth data instead of only typed
|
||||||
|
# destination/duration/style strings. A side-effect failure (site
|
||||||
|
# blocked, extraction error) must never fail the whole batch — logged
|
||||||
|
# and left as None, which build_prompt() treats identically to a
|
||||||
|
# tenant who never confirmed an own-trip URL for this run.
|
||||||
|
own_trip_url = data.get('own_trip_url')
|
||||||
|
context['own_trip'] = None
|
||||||
|
if own_trip_url:
|
||||||
|
try:
|
||||||
|
context['own_trip'] = embedding_service.scrape_own_trip(tenant_id, own_trip_url, context)
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f'scrape_own_trip failed for tenant {tenant_id}: {str(e)}')
|
||||||
|
|
||||||
success, failed = [], []
|
success, failed = [], []
|
||||||
done = 0
|
done = 0
|
||||||
cancelled = False
|
cancelled = False
|
||||||
|
|
@ -199,7 +215,11 @@ def run_research_job(job_id, data):
|
||||||
status = 'cancelled' if cancelled else ('complete' if success or not competitors_input else 'failed')
|
status = 'cancelled' if cancelled else ('complete' if success or not competitors_input else 'failed')
|
||||||
scrape_repository.update(job_id, {
|
scrape_repository.update(job_id, {
|
||||||
'status': status,
|
'status': status,
|
||||||
'results': {'success': success, 'failed': failed},
|
# own_trip surfaces the tenant's own extracted trip data (or None
|
||||||
|
# if no website was confirmed this run) to the dashboard — before
|
||||||
|
# this it was only ever consumed internally by build_prompt(),
|
||||||
|
# with no way for the PM to see it was actually scraped and used.
|
||||||
|
'results': {'success': success, 'failed': failed, 'own_trip': context.get('own_trip')},
|
||||||
'error_log': '; '.join(f['error'] for f in failed) if failed else '',
|
'error_log': '; '.join(f['error'] for f in failed) if failed else '',
|
||||||
'completed_at': datetime.now(timezone.utc).isoformat(),
|
'completed_at': datetime.now(timezone.utc).isoformat(),
|
||||||
})
|
})
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,39 @@
|
||||||
|
/// <reference path="../pb_data/types.d.ts" />
|
||||||
|
//
|
||||||
|
// Adds the fields needed to scrape the TENANT's OWN trip page, mirroring
|
||||||
|
// how competitors are scraped, so build_prompt()'s comparison and the
|
||||||
|
// `comments` field are grounded in a real extracted trip instead of only
|
||||||
|
// the PM's typed destination/duration/style strings.
|
||||||
|
//
|
||||||
|
// tenants.website url the tenant's own storefront root — distinct
|
||||||
|
// from `domain` (bare text, used only for
|
||||||
|
// email-domain seat matching, never a URL)
|
||||||
|
// client_trips.url text confirmed/matched own-trip page for a run
|
||||||
|
// client_trips.extracted json full structured extraction blob (mirrors
|
||||||
|
// products' shape — tripName/price/duration/
|
||||||
|
// activities/serviceLevel/etc.)
|
||||||
|
// client_trips.last_scraped date drives the same 24hr freshness reuse as
|
||||||
|
// competitors (core/scraper.py's
|
||||||
|
// CACHE_STALENESS_HOURS)
|
||||||
|
|
||||||
|
migrate((app) => {
|
||||||
|
const tenants = app.findCollectionByNameOrId('tenants');
|
||||||
|
tenants.fields.add(new Field({ type: 'url', name: 'website' }));
|
||||||
|
app.save(tenants);
|
||||||
|
|
||||||
|
const clientTrips = app.findCollectionByNameOrId('client_trips');
|
||||||
|
clientTrips.fields.add(new Field({ type: 'text', name: 'url' }));
|
||||||
|
clientTrips.fields.add(new Field({ type: 'json', name: 'extracted' }));
|
||||||
|
clientTrips.fields.add(new Field({ type: 'date', name: 'last_scraped' }));
|
||||||
|
return app.save(clientTrips);
|
||||||
|
}, (app) => {
|
||||||
|
const tenants = app.findCollectionByNameOrId('tenants');
|
||||||
|
tenants.fields.removeByName('website');
|
||||||
|
app.save(tenants);
|
||||||
|
|
||||||
|
const clientTrips = app.findCollectionByNameOrId('client_trips');
|
||||||
|
clientTrips.fields.removeByName('url');
|
||||||
|
clientTrips.fields.removeByName('extracted');
|
||||||
|
clientTrips.fields.removeByName('last_scraped');
|
||||||
|
return app.save(clientTrips);
|
||||||
|
});
|
||||||
|
|
@ -0,0 +1,40 @@
|
||||||
|
/// <reference path="../pb_data/types.d.ts" />
|
||||||
|
//
|
||||||
|
// Adds `scraped_url` to `scrape_runs` — fixes a real bug where content-hash
|
||||||
|
// change detection was scoped only by competitor_id, not by which specific
|
||||||
|
// page was actually scraped. Once trip-intent matching (CLAUDE.md §8) let a
|
||||||
|
// PM scrape different trip pages for the same competitor across different
|
||||||
|
// analysis runs, get_last_hash_for_url(competitor_id) compared each new
|
||||||
|
// scrape's hash against whatever page happened to be scraped MOST
|
||||||
|
// RECENTLY for that competitor — almost always a different page — so
|
||||||
|
// change_detected kept getting set back to True on nearly every run, and
|
||||||
|
// needs_refresh() (which gates the fast path on change_detected being
|
||||||
|
// False) never let the true cache-only fast path trigger in practice.
|
||||||
|
// Confirmed directly against real scrape_runs history: the same
|
||||||
|
// competitor's recorded content_hash changed on 5 of 6 consecutive scrapes
|
||||||
|
// over one test session, even though re-scraping the exact same URL twice
|
||||||
|
// in immediate succession produced an identical hash both times.
|
||||||
|
//
|
||||||
|
// See services/analysis_service.py's detect_change()/needs_refresh()/
|
||||||
|
// run_competitor_analysis() and repositories/scrape_repository.py's
|
||||||
|
// get_last_hash_for_url() for the corresponding code fix.
|
||||||
|
|
||||||
|
migrate((app) => {
|
||||||
|
const collection = app.findCollectionByNameOrId('scrape_runs');
|
||||||
|
|
||||||
|
collection.fields.add(new Field({ type: 'text', name: 'scraped_url' }));
|
||||||
|
|
||||||
|
collection.indexes = [
|
||||||
|
...collection.indexes,
|
||||||
|
'CREATE INDEX idx_scrape_runs_url_hash ON scrape_runs (competitor_id, scraped_url, content_hash, started_at)',
|
||||||
|
];
|
||||||
|
|
||||||
|
return app.save(collection);
|
||||||
|
}, (app) => {
|
||||||
|
const collection = app.findCollectionByNameOrId('scrape_runs');
|
||||||
|
|
||||||
|
collection.fields.removeByName('scraped_url');
|
||||||
|
collection.indexes = collection.indexes.filter((idx) => !idx.includes('idx_scrape_runs_url_hash'));
|
||||||
|
|
||||||
|
return app.save(collection);
|
||||||
|
});
|
||||||
|
|
@ -0,0 +1,62 @@
|
||||||
|
/// <reference path="../pb_data/types.d.ts" />
|
||||||
|
//
|
||||||
|
// Adds `trip_watchlist` — the per-seat trip-monitoring feature. Each
|
||||||
|
// active seat can pin a small, tier-capped number of specific already-
|
||||||
|
// analyzed competitor trips (WATCHLIST_CAP_PER_SEAT in
|
||||||
|
// services/watchlist_service.py — 1 per seat at Analyse). The nightly
|
||||||
|
// n8n cron re-scrapes exactly these URLs instead of each competitor's
|
||||||
|
// root website, replacing an unbounded, low-signal daily scrape (a
|
||||||
|
// homepage `is_generic_homepage()` already flags as non-trip-specific)
|
||||||
|
// with a small, predictable, and meaningful one — every resulting
|
||||||
|
// price/product alert is about a trip a PM explicitly chose to track.
|
||||||
|
//
|
||||||
|
// Reuses the exact same confirmed_url / per-url content-hash caching
|
||||||
|
// already built for manual trip-intent analysis this session — no
|
||||||
|
// schema change needed on products/scrape_runs/competitors, and no
|
||||||
|
// jobs.py/analysis_service.py code change either. See
|
||||||
|
// n8n/workflows/daily_scrape_cron.json for the cron-side wiring.
|
||||||
|
|
||||||
|
migrate((app) => {
|
||||||
|
const tenants = app.findCollectionByNameOrId('tenants');
|
||||||
|
const tenantSeats = app.findCollectionByNameOrId('tenant_seats');
|
||||||
|
const competitors = app.findCollectionByNameOrId('competitors');
|
||||||
|
|
||||||
|
const rel = (name, collection, opts = {}) => ({
|
||||||
|
name, type: 'relation', collectionId: collection.id,
|
||||||
|
cascadeDelete: false, minSelect: 0, maxSelect: 1, ...opts,
|
||||||
|
});
|
||||||
|
const text = (name, opts = {}) => ({ name, type: 'text', ...opts });
|
||||||
|
const autodate = (name, onCreate, onUpdate = false) => ({
|
||||||
|
name, type: 'autodate', onCreate, onUpdate,
|
||||||
|
});
|
||||||
|
|
||||||
|
const collection = new Collection({
|
||||||
|
type: 'base',
|
||||||
|
name: 'trip_watchlist',
|
||||||
|
fields: [
|
||||||
|
rel('tenant_id', tenants, { required: true }),
|
||||||
|
rel('seat_id', tenantSeats, { required: true }),
|
||||||
|
rel('competitor_id', competitors, { required: true }),
|
||||||
|
text('url', { required: true }),
|
||||||
|
text('trip_name'),
|
||||||
|
autodate('added_at', true),
|
||||||
|
],
|
||||||
|
indexes: [
|
||||||
|
'CREATE INDEX idx_trip_watchlist_seat ON trip_watchlist (seat_id)',
|
||||||
|
'CREATE INDEX idx_trip_watchlist_tenant ON trip_watchlist (tenant_id)',
|
||||||
|
],
|
||||||
|
// Same access model as every other tenant-scoped collection — Flask
|
||||||
|
// is the only direct PocketBase client, enforcing tenant_id/seat_id
|
||||||
|
// scoping itself.
|
||||||
|
listRule: null,
|
||||||
|
viewRule: null,
|
||||||
|
createRule: null,
|
||||||
|
updateRule: null,
|
||||||
|
deleteRule: null,
|
||||||
|
});
|
||||||
|
|
||||||
|
return app.save(collection);
|
||||||
|
}, (app) => {
|
||||||
|
const collection = app.findCollectionByNameOrId('trip_watchlist');
|
||||||
|
if (collection) app.delete(collection);
|
||||||
|
});
|
||||||
|
|
@ -33,6 +33,23 @@ class ProductRepository:
|
||||||
f'competitor_id = "{competitor_id}" && trip_name = "{safe_name}" && is_active = true'
|
f'competitor_id = "{competitor_id}" && trip_name = "{safe_name}" && is_active = true'
|
||||||
)
|
)
|
||||||
|
|
||||||
|
def get_by_competitor_and_url(self, competitor_id, url):
|
||||||
|
"""
|
||||||
|
Finds the active product matching this exact competitor+URL —
|
||||||
|
used by needs_refresh()/analyse_from_cache() to check freshness
|
||||||
|
for and serve the SPECIFIC trip page confirmed for this run, not
|
||||||
|
just whichever product happens to be most recently scraped for
|
||||||
|
this competitor overall (which could be a completely different
|
||||||
|
trip if the PM has researched more than one product here).
|
||||||
|
"""
|
||||||
|
if not url:
|
||||||
|
return None
|
||||||
|
safe_url = url.replace('"', '\\"')
|
||||||
|
return pb_client.get_first(
|
||||||
|
COLLECTION,
|
||||||
|
f'competitor_id = "{competitor_id}" && url = "{safe_url}" && is_active = true'
|
||||||
|
)
|
||||||
|
|
||||||
def list_for_tenant(self, tenant_id):
|
def list_for_tenant(self, tenant_id):
|
||||||
"""Returns all active products for a tenant — used for embedding/comparable matching."""
|
"""Returns all active products for a tenant — used for embedding/comparable matching."""
|
||||||
return pb_client.list(COLLECTION, filter_str=f'tenant_id = "{tenant_id}" && is_active = true', per_page=500)
|
return pb_client.list(COLLECTION, filter_str=f'tenant_id = "{tenant_id}" && is_active = true', per_page=500)
|
||||||
|
|
@ -72,6 +89,16 @@ class MockProductRepository:
|
||||||
and r.get('is_active', True)
|
and r.get('is_active', True)
|
||||||
), None)
|
), None)
|
||||||
|
|
||||||
|
def get_by_competitor_and_url(self, competitor_id, url):
|
||||||
|
if not url:
|
||||||
|
return None
|
||||||
|
return next((
|
||||||
|
r for r in self._records.values()
|
||||||
|
if r.get('competitor_id') == competitor_id
|
||||||
|
and r.get('url') == url
|
||||||
|
and r.get('is_active', True)
|
||||||
|
), None)
|
||||||
|
|
||||||
def list_for_tenant(self, tenant_id):
|
def list_for_tenant(self, tenant_id):
|
||||||
return [r for r in self._records.values() if r.get('tenant_id') == tenant_id and r.get('is_active', True)]
|
return [r for r in self._records.values() if r.get('tenant_id') == tenant_id and r.get('is_active', True)]
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -56,15 +56,28 @@ class ScrapeRepository:
|
||||||
)
|
)
|
||||||
return items[0] if items else None
|
return items[0] if items else None
|
||||||
|
|
||||||
def get_last_hash_for_url(self, competitor_id):
|
def get_last_hash_for_url(self, competitor_id, url):
|
||||||
"""
|
"""
|
||||||
Returns the most recently recorded content_hash for this
|
Returns the most recently recorded content_hash for this exact
|
||||||
competitor's prior scrape, or None if it has never been hashed
|
competitor+url combination, or None if this specific page has
|
||||||
before. Backs detect_change()'s content-hash comparison.
|
never been hashed before. Backs detect_change()'s content-hash
|
||||||
|
comparison.
|
||||||
|
|
||||||
|
Scoped by url as well as competitor_id — CLAUDE.md §8's
|
||||||
|
trip-intent matching means one competitor can have many
|
||||||
|
different confirmed trip URLs scraped over time. Comparing
|
||||||
|
against whichever page happened to be scraped MOST RECENTLY for
|
||||||
|
this competitor (the old, url-blind lookup) meant a fresh scrape
|
||||||
|
of URL A got compared against URL B's hash whenever B was
|
||||||
|
scraped more recently — almost always a mismatch, which kept
|
||||||
|
change_detected stuck True and made the true cache-only fast
|
||||||
|
path unreachable in practice. Confirmed directly against real
|
||||||
|
scrape_runs history before this fix.
|
||||||
"""
|
"""
|
||||||
|
safe_url = url.replace('"', '\\"')
|
||||||
items = pb_client.list(
|
items = pb_client.list(
|
||||||
COLLECTION,
|
COLLECTION,
|
||||||
filter_str=f'competitor_id = "{competitor_id}" && content_hash != ""',
|
filter_str=f'competitor_id = "{competitor_id}" && scraped_url = "{safe_url}" && content_hash != ""',
|
||||||
sort='-started_at', per_page=1
|
sort='-started_at', per_page=1
|
||||||
)
|
)
|
||||||
return items[0]['content_hash'] if items else None
|
return items[0]['content_hash'] if items else None
|
||||||
|
|
@ -125,10 +138,10 @@ class MockScrapeRepository:
|
||||||
items.sort(key=lambda r: r.get('started_at') or '', reverse=True)
|
items.sort(key=lambda r: r.get('started_at') or '', reverse=True)
|
||||||
return items[0] if items else None
|
return items[0] if items else None
|
||||||
|
|
||||||
def get_last_hash_for_url(self, competitor_id):
|
def get_last_hash_for_url(self, competitor_id, url):
|
||||||
items = [
|
items = [
|
||||||
r for r in self._records.values()
|
r for r in self._records.values()
|
||||||
if r.get('competitor_id') == competitor_id and r.get('content_hash')
|
if r.get('competitor_id') == competitor_id and r.get('scraped_url') == url and r.get('content_hash')
|
||||||
]
|
]
|
||||||
items.sort(key=lambda r: r.get('started_at') or '', reverse=True)
|
items.sort(key=lambda r: r.get('started_at') or '', reverse=True)
|
||||||
return items[0]['content_hash'] if items else None
|
return items[0]['content_hash'] if items else None
|
||||||
|
|
|
||||||
|
|
@ -21,6 +21,11 @@ class SeatRepository:
|
||||||
f'tenant_id = "{tenant_id}" && email = "{safe_email}" && active = true'
|
f'tenant_id = "{tenant_id}" && email = "{safe_email}" && active = true'
|
||||||
)
|
)
|
||||||
|
|
||||||
|
def get_by_id(self, seat_id):
|
||||||
|
"""Returns a seat record by id, or None — used to denormalize the
|
||||||
|
watching seat's email onto a watchlist item for display."""
|
||||||
|
return pb_client.get_one(COLLECTION, seat_id)
|
||||||
|
|
||||||
def count_active(self, tenant_id):
|
def count_active(self, tenant_id):
|
||||||
"""Returns the count of active seats for a tenant."""
|
"""Returns the count of active seats for a tenant."""
|
||||||
return len(pb_client.list(
|
return len(pb_client.list(
|
||||||
|
|
@ -58,6 +63,9 @@ class MockSeatRepository:
|
||||||
if r.get('tenant_id') == tenant_id and r.get('email') == email and r.get('active')
|
if r.get('tenant_id') == tenant_id and r.get('email') == email and r.get('active')
|
||||||
), None)
|
), None)
|
||||||
|
|
||||||
|
def get_by_id(self, seat_id):
|
||||||
|
return self._records.get(seat_id)
|
||||||
|
|
||||||
def count_active(self, tenant_id):
|
def count_active(self, tenant_id):
|
||||||
return len([
|
return len([
|
||||||
r for r in self._records.values()
|
r for r in self._records.values()
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,74 @@
|
||||||
|
import uuid
|
||||||
|
from repositories.pocketbase_client import pb_client
|
||||||
|
from repositories.repo_config import USE_MOCK_REPOSITORIES
|
||||||
|
|
||||||
|
COLLECTION = 'trip_watchlist'
|
||||||
|
|
||||||
|
|
||||||
|
class WatchlistRepository:
|
||||||
|
"""
|
||||||
|
Data access for the `trip_watchlist` collection — pure CRUD only, per
|
||||||
|
the no-business-logic-in-repositories rule. Cap enforcement and
|
||||||
|
ownership decisions live in services/watchlist_service.py.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def create(self, data):
|
||||||
|
return pb_client.create(COLLECTION, data)
|
||||||
|
|
||||||
|
def delete(self, item_id):
|
||||||
|
return pb_client.delete(COLLECTION, item_id)
|
||||||
|
|
||||||
|
def get_by_id(self, item_id):
|
||||||
|
return pb_client.get_one(COLLECTION, item_id)
|
||||||
|
|
||||||
|
def list_for_tenant(self, tenant_id):
|
||||||
|
"""All watched items for a tenant (every seat's), newest first."""
|
||||||
|
return pb_client.list(
|
||||||
|
COLLECTION, filter_str=f'tenant_id = "{tenant_id}"',
|
||||||
|
sort='-added_at', per_page=200
|
||||||
|
)
|
||||||
|
|
||||||
|
def count_for_seat(self, seat_id):
|
||||||
|
"""Returns how many watchlist slots this seat currently occupies."""
|
||||||
|
return len(pb_client.list(
|
||||||
|
COLLECTION, filter_str=f'seat_id = "{seat_id}"', per_page=200
|
||||||
|
))
|
||||||
|
|
||||||
|
|
||||||
|
class MockWatchlistRepository:
|
||||||
|
"""In-memory stand-in for WatchlistRepository."""
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
self._records = {}
|
||||||
|
# Mirrors MockScrapeRepository's monotonic counter — real
|
||||||
|
# PocketBase auto-populates added_at via autodate(true), and
|
||||||
|
# every "newest first" sort here relies on strict ordering.
|
||||||
|
self._sequence = 0
|
||||||
|
|
||||||
|
def create(self, data):
|
||||||
|
record_id = data.get('id') or str(uuid.uuid4())
|
||||||
|
self._sequence += 1
|
||||||
|
record = {
|
||||||
|
'id': record_id,
|
||||||
|
'added_at': data.get('added_at') or f'{self._sequence:020d}',
|
||||||
|
**data,
|
||||||
|
}
|
||||||
|
self._records[record_id] = record
|
||||||
|
return record
|
||||||
|
|
||||||
|
def delete(self, item_id):
|
||||||
|
return self._records.pop(item_id, None) is not None
|
||||||
|
|
||||||
|
def get_by_id(self, item_id):
|
||||||
|
return self._records.get(item_id)
|
||||||
|
|
||||||
|
def list_for_tenant(self, tenant_id):
|
||||||
|
items = [r for r in self._records.values() if r.get('tenant_id') == tenant_id]
|
||||||
|
items.sort(key=lambda r: r.get('added_at') or '', reverse=True)
|
||||||
|
return items
|
||||||
|
|
||||||
|
def count_for_seat(self, seat_id):
|
||||||
|
return len([r for r in self._records.values() if r.get('seat_id') == seat_id])
|
||||||
|
|
||||||
|
|
||||||
|
watchlist_repository = MockWatchlistRepository() if USE_MOCK_REPOSITORIES else WatchlistRepository()
|
||||||
|
|
@ -15,17 +15,60 @@ logger = logging.getLogger(__name__)
|
||||||
PRICE_CHANGE_ALERT_THRESHOLD_PERCENT = 5
|
PRICE_CHANGE_ALERT_THRESHOLD_PERCENT = 5
|
||||||
|
|
||||||
|
|
||||||
def needs_refresh(competitor):
|
def needs_refresh(competitor, confirmed_url=None):
|
||||||
"""
|
"""
|
||||||
Decides whether a competitor needs a live re-scrape (REFRESH path)
|
Decides whether a competitor's specific confirmed page needs a live
|
||||||
or can be served from PocketBase (FAST path).
|
re-scrape (REFRESH path) or can be served from PocketBase (FAST path).
|
||||||
|
|
||||||
Data flow:
|
When confirmed_url is given (trip-intent matching, CLAUDE.md §8),
|
||||||
|
freshness is judged against the cached PRODUCT for that exact URL,
|
||||||
|
not the competitor's global change_detected/last_scraped flags.
|
||||||
|
Those flags reflect whichever page was scraped MOST RECENTLY for
|
||||||
|
this competitor — which may be a completely different trip than the
|
||||||
|
one being asked about now. Gating on them directly made the fast
|
||||||
|
path unreachable (or worse, risked serving the wrong cached trip)
|
||||||
|
whenever a PM researched more than one product for the same
|
||||||
|
competitor across different runs — confirmed directly against real
|
||||||
|
scrape_runs history, where the same competitor's recorded hash
|
||||||
|
changed on nearly every run despite re-scraping one specific URL
|
||||||
|
twice in a row producing an identical hash both times.
|
||||||
|
|
||||||
|
Falls back to the original competitor-wide check when no
|
||||||
|
confirmed_url is available — the daily cron path (§15), which
|
||||||
|
scrapes one canonical page per competitor and has no trip-intent
|
||||||
|
context at all, is unaffected by this fix.
|
||||||
|
|
||||||
|
Data flow (confirmed_url given):
|
||||||
|
confirmed_url → product_repository.get_by_competitor_and_url() →
|
||||||
|
no cached product for this exact page → True (refresh) →
|
||||||
|
cached product's last_seen missing/stale (> CACHE_STALENESS_HOURS)
|
||||||
|
→ True (refresh) → otherwise → False (fast path is safe)
|
||||||
|
|
||||||
|
Data flow (no confirmed_url — cron path, unchanged from before):
|
||||||
competitor record →
|
competitor record →
|
||||||
change_detected flag set? → True (refresh) →
|
change_detected flag set? → True (refresh) →
|
||||||
last_scraped missing or older than CACHE_STALENESS_HOURS? → True (refresh) →
|
last_scraped missing or older than CACHE_STALENESS_HOURS? → True (refresh) →
|
||||||
otherwise → False (fast path is safe)
|
otherwise → False (fast path is safe)
|
||||||
"""
|
"""
|
||||||
|
# CACHE_STALENESS_HOURS' canonical, adjustable home is core/scraper.py
|
||||||
|
# per CLAUDE.md §7 ("in core/scraper.py — adjust here only") — imported
|
||||||
|
# lazily here for the same reason every other core.* reference in this
|
||||||
|
# file is lazy: this module must stay importable independent of Phase 2.
|
||||||
|
from core.scraper import CACHE_STALENESS_HOURS
|
||||||
|
|
||||||
|
if confirmed_url:
|
||||||
|
product = product_repository.get_by_competitor_and_url(competitor['id'], confirmed_url)
|
||||||
|
if not product:
|
||||||
|
return True
|
||||||
|
last_seen = product.get('last_seen')
|
||||||
|
if not last_seen:
|
||||||
|
return True
|
||||||
|
try:
|
||||||
|
seen_at = datetime.fromisoformat(last_seen.replace('Z', '+00:00'))
|
||||||
|
except (ValueError, AttributeError):
|
||||||
|
return True
|
||||||
|
return (datetime.now(timezone.utc) - seen_at) > timedelta(hours=CACHE_STALENESS_HOURS)
|
||||||
|
|
||||||
if competitor.get('change_detected'):
|
if competitor.get('change_detected'):
|
||||||
return True
|
return True
|
||||||
|
|
||||||
|
|
@ -38,12 +81,6 @@ def needs_refresh(competitor):
|
||||||
except (ValueError, AttributeError):
|
except (ValueError, AttributeError):
|
||||||
return True
|
return True
|
||||||
|
|
||||||
# CACHE_STALENESS_HOURS' canonical, adjustable home is core/scraper.py
|
|
||||||
# per CLAUDE.md §7 ("in core/scraper.py — adjust here only") — imported
|
|
||||||
# lazily here for the same reason every other core.* reference in this
|
|
||||||
# file is lazy: this module must stay importable independent of Phase 2.
|
|
||||||
from core.scraper import CACHE_STALENESS_HOURS
|
|
||||||
|
|
||||||
age = datetime.now(timezone.utc) - scraped_at
|
age = datetime.now(timezone.utc) - scraped_at
|
||||||
return age > timedelta(hours=CACHE_STALENESS_HOURS)
|
return age > timedelta(hours=CACHE_STALENESS_HOURS)
|
||||||
|
|
||||||
|
|
@ -76,21 +113,59 @@ def analyse_from_cache(tenant_id, competitor_id, context):
|
||||||
constrained to {comments, relevancy} only (via core.extractor) →
|
constrained to {comments, relevancy} only (via core.extractor) →
|
||||||
cached structured fields + fresh comments/relevancy merged and
|
cached structured fields + fresh comments/relevancy merged and
|
||||||
returned
|
returned
|
||||||
|
|
||||||
|
context['confirmed_url'], when present, selects the cached product
|
||||||
|
for that EXACT trip page (product_repository.get_by_competitor_and_url())
|
||||||
|
rather than whichever product happens to be most recently scraped for
|
||||||
|
this competitor overall — the same competitor can have cached data
|
||||||
|
for several different trips, and "most recent" is not necessarily the
|
||||||
|
one this run is actually about. Falls back to "latest for competitor"
|
||||||
|
when no confirmed_url is given (cron path) or no product matches it yet.
|
||||||
"""
|
"""
|
||||||
|
confirmed_url = context.get('confirmed_url')
|
||||||
|
latest = product_repository.get_by_competitor_and_url(competitor_id, confirmed_url) if confirmed_url else None
|
||||||
|
if not latest:
|
||||||
products = product_repository.get_latest_for_competitor(competitor_id, limit=1)
|
products = product_repository.get_latest_for_competitor(competitor_id, limit=1)
|
||||||
if not products:
|
if not products:
|
||||||
raise ValueError('No cached product data available for fast path')
|
raise ValueError('No cached product data available for fast path')
|
||||||
|
|
||||||
latest = products[0]
|
latest = products[0]
|
||||||
|
|
||||||
# core.extractor is a Phase 2 module — imported lazily so this
|
# core.extractor is a Phase 2 module — imported lazily so this
|
||||||
# service is importable before the scraping engine refactor lands.
|
# service is importable before the scraping engine refactor lands.
|
||||||
from core.extractor import extract_with_openrouter
|
from core.extractor import extract_with_openrouter
|
||||||
|
|
||||||
|
# Ground the comparison in the tenant's own REAL extracted trip
|
||||||
|
# (context['own_trip'], from embedding_service.scrape_own_trip())
|
||||||
|
# whenever it's available — only falling back to the PM's typed
|
||||||
|
# destination/duration/style/product name when no own-trip data was
|
||||||
|
# returned for this run (no website confirmed, or the scrape/
|
||||||
|
# extraction failed). Previously this dumped the raw `context` dict
|
||||||
|
# (Python repr) into the prompt with no priority framing at all, so
|
||||||
|
# own_trip's presence was easy for the model to under-weight against
|
||||||
|
# the typed fields sitting right next to it in the same blob.
|
||||||
|
own_trip = context.get('own_trip')
|
||||||
|
company_name = context.get('company_name') or 'the client'
|
||||||
|
if own_trip:
|
||||||
|
own_trip_name = own_trip.get('tripName') or context.get('product_name')
|
||||||
|
client_context_desc = (
|
||||||
|
f"{company_name}'s own {own_trip_name} — ACTUAL extracted data from "
|
||||||
|
f"their own site, not a typed description: "
|
||||||
|
f"{own_trip.get('majorityPrice', 'price not extracted')} USD, "
|
||||||
|
f"{own_trip.get('duration') or context.get('duration')} days, "
|
||||||
|
f"activities: {own_trip.get('activities') or 'not extracted'}"
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
client_context_desc = (
|
||||||
|
f"{company_name}'s typed product intent (no own-site data confirmed "
|
||||||
|
f"this run): product={context.get('product_name')}, "
|
||||||
|
f"destination={context.get('destination')}, "
|
||||||
|
f"duration={context.get('duration')} days, style={context.get('travel_style')}"
|
||||||
|
)
|
||||||
|
|
||||||
prompt = (
|
prompt = (
|
||||||
'You are a travel industry analyst. Given this already-extracted '
|
'You are a travel industry analyst. Given this already-extracted '
|
||||||
'competitor trip data, assess it against '
|
'competitor trip data, assess it against '
|
||||||
f"the client's product context: {context}.\n\nCompetitor data:\n{latest}\n\n"
|
f"{client_context_desc}.\n\nCompetitor data:\n{latest}\n\n"
|
||||||
'Return ONLY a valid JSON object:\n'
|
'Return ONLY a valid JSON object:\n'
|
||||||
'{\n'
|
'{\n'
|
||||||
' "comments": "short comparative analysis: key differences, strengths, weaknesses, pricing position",\n'
|
' "comments": "short comparative analysis: key differences, strengths, weaknesses, pricing position",\n'
|
||||||
|
|
@ -141,21 +216,28 @@ def analyse_from_cache(tenant_id, competitor_id, context):
|
||||||
return result
|
return result
|
||||||
|
|
||||||
|
|
||||||
def detect_change(competitor_id, new_hash):
|
def detect_change(competitor_id, url, new_hash):
|
||||||
"""
|
"""
|
||||||
Compares new_hash against the last content_hash recorded for this
|
Compares new_hash against the last content_hash recorded for this
|
||||||
competitor. On a diff: sets competitor.change_detected = true and
|
exact competitor+url. On a diff: sets competitor.change_detected =
|
||||||
change_detected_at = now. Replaces Firecrawl Monitor's webhook role
|
true and change_detected_at = now. Replaces Firecrawl Monitor's
|
||||||
(CLAUDE.md §4/§10) — the daily/on-demand scrape itself is now the
|
webhook role (CLAUDE.md §4/§10) — the daily/on-demand scrape itself
|
||||||
only place a change is ever detected.
|
is now the only place a change is ever detected.
|
||||||
|
|
||||||
|
Scoped by url, not just competitor_id — see
|
||||||
|
scrape_repository.get_last_hash_for_url()'s docstring for why
|
||||||
|
comparing against whatever page was scraped most recently for this
|
||||||
|
competitor (regardless of which page that was) made change_detected
|
||||||
|
effectively never settle back to False once a competitor had more
|
||||||
|
than one trip page scraped over time.
|
||||||
|
|
||||||
Data flow:
|
Data flow:
|
||||||
competitor_id + new_hash → scrape_repository.get_last_hash_for_url() →
|
competitor_id + url + new_hash → scrape_repository.get_last_hash_for_url() →
|
||||||
no prior hash: True (first scrape — always process) →
|
no prior hash for this url: True (first scrape of this page — always process) →
|
||||||
hash differs: competitor_repository.update() sets change flags → True →
|
hash differs: competitor_repository.update() sets change flags → True →
|
||||||
hash matches: no update → False
|
hash matches: no update → False
|
||||||
"""
|
"""
|
||||||
last_hash = scrape_repository.get_last_hash_for_url(competitor_id)
|
last_hash = scrape_repository.get_last_hash_for_url(competitor_id, url)
|
||||||
if last_hash is None:
|
if last_hash is None:
|
||||||
return True
|
return True
|
||||||
if last_hash != new_hash:
|
if last_hash != new_hash:
|
||||||
|
|
@ -184,10 +266,11 @@ def run_competitor_analysis(tenant_id, competitor, context):
|
||||||
when forced, so future non-forced runs keep an accurate hash history.
|
when forced, so future non-forced runs keep an accurate hash history.
|
||||||
|
|
||||||
Data flow:
|
Data flow:
|
||||||
competitor + context → needs_refresh() decision (or force_refresh) →
|
competitor + context → target_url resolved and cleaned →
|
||||||
|
needs_refresh() decision, scoped to target_url (or force_refresh) →
|
||||||
FAST: analyse_from_cache() (seconds) →
|
FAST: analyse_from_cache() (seconds) →
|
||||||
REFRESH: core.scraper.scrape() → compute_content_hash() →
|
REFRESH: core.scraper.scrape() → compute_content_hash() →
|
||||||
detect_change() against the last recorded hash →
|
detect_change() against the last recorded hash FOR THIS URL →
|
||||||
unchanged (and not force_refresh): change_detected cleared
|
unchanged (and not force_refresh): change_detected cleared
|
||||||
(confirmed stable since last look), skip LLM extraction, reuse
|
(confirmed stable since last look), skip LLM extraction, reuse
|
||||||
cached narrative, bump last_scraped only (no wasted extraction
|
cached narrative, bump last_scraped only (no wasted extraction
|
||||||
|
|
@ -200,9 +283,40 @@ def run_competitor_analysis(tenant_id, competitor, context):
|
||||||
"""
|
"""
|
||||||
force_refresh = context.get('force_refresh', False)
|
force_refresh = context.get('force_refresh', False)
|
||||||
|
|
||||||
if not needs_refresh(competitor) and not force_refresh:
|
# Phase 2 module — imported lazily, same reasoning as the other
|
||||||
|
# core.* imports further down in this function.
|
||||||
|
from core.scraper import clean_url
|
||||||
|
|
||||||
|
# context['confirmed_url'] is the page confirmed at UrlConfirmation.vue
|
||||||
|
# (a Firecrawl /map match or a manual swap) — falls back to the
|
||||||
|
# competitor's stored root domain only when nothing was confirmed for
|
||||||
|
# this run (e.g. the daily cron path, which has no trip-intent context
|
||||||
|
# at all). Regression fix: this used to always scrape
|
||||||
|
# competitor['website'] unconditionally, silently discarding whatever
|
||||||
|
# trip-specific URL was actually matched/confirmed for this run.
|
||||||
|
#
|
||||||
|
# Cleaned up front (matching what scrape() does internally) so every
|
||||||
|
# url-scoped lookup below (needs_refresh, detect_change,
|
||||||
|
# analyse_from_cache) compares against the same normalised form
|
||||||
|
# _save_scrape_result() actually persists as products.url — a
|
||||||
|
# trailing-slash or tracking-param difference would otherwise
|
||||||
|
# silently defeat the url match and force an unnecessary refresh.
|
||||||
|
# Cleaned regardless of source (confirmed_url or the website
|
||||||
|
# fallback) so scrape_runs.scraped_url/products.url are always
|
||||||
|
# consistent with each other either way.
|
||||||
|
raw_confirmed_url = context.get('confirmed_url')
|
||||||
|
target_url = clean_url(raw_confirmed_url or competitor['website'])
|
||||||
|
# Only url-scope the fast-path/cache lookups below when this run
|
||||||
|
# actually had an explicit confirmed_url (trip-intent matching) —
|
||||||
|
# the cron/website-fallback path keeps using the competitor-wide
|
||||||
|
# change_detected/last_scraped flags, unchanged from before this fix.
|
||||||
|
confirmed_url = target_url if raw_confirmed_url else None
|
||||||
|
|
||||||
|
if not needs_refresh(competitor, confirmed_url=confirmed_url) and not force_refresh:
|
||||||
logger.info(f'{competitor["name"]} — fast path (cache fresh)')
|
logger.info(f'{competitor["name"]} — fast path (cache fresh)')
|
||||||
fast_result = analyse_from_cache(tenant_id, competitor['id'], context)
|
fast_result = analyse_from_cache(
|
||||||
|
tenant_id, competitor['id'], {**context, 'confirmed_url': confirmed_url}
|
||||||
|
)
|
||||||
return {
|
return {
|
||||||
'path': 'fast', 'result': fast_result,
|
'path': 'fast', 'result': fast_result,
|
||||||
'is_generic_page': is_generic_homepage(fast_result.get('link')),
|
'is_generic_page': is_generic_homepage(fast_result.get('link')),
|
||||||
|
|
@ -214,14 +328,6 @@ def run_competitor_analysis(tenant_id, competitor, context):
|
||||||
from core.extractor import extract_with_openrouter
|
from core.extractor import extract_with_openrouter
|
||||||
from industries.adventure_travel.prompt import build_prompt
|
from industries.adventure_travel.prompt import build_prompt
|
||||||
|
|
||||||
# context['confirmed_url'] is the page confirmed at UrlConfirmation.vue
|
|
||||||
# (a Firecrawl /map match or a manual swap) — falls back to the
|
|
||||||
# competitor's stored root domain only when nothing was confirmed for
|
|
||||||
# this run (e.g. the daily cron path, which has no trip-intent context
|
|
||||||
# at all). Regression fix: this used to always scrape
|
|
||||||
# competitor['website'] unconditionally, silently discarding whatever
|
|
||||||
# trip-specific URL was actually matched/confirmed for this run.
|
|
||||||
target_url = context.get('confirmed_url') or competitor['website']
|
|
||||||
# Computed once here and reused by both possible refresh-path returns
|
# Computed once here and reused by both possible refresh-path returns
|
||||||
# below — a meta-fact about which page was actually scraped this run,
|
# below — a meta-fact about which page was actually scraped this run,
|
||||||
# not an extracted trip field, so it's a top-level sibling of `result`/
|
# not an extracted trip field, so it's a top-level sibling of `result`/
|
||||||
|
|
@ -232,17 +338,19 @@ def run_competitor_analysis(tenant_id, competitor, context):
|
||||||
raise ValueError(page_data.get('text', 'Scrape failed'))
|
raise ValueError(page_data.get('text', 'Scrape failed'))
|
||||||
|
|
||||||
content_hash = compute_content_hash(page_data['text'])
|
content_hash = compute_content_hash(page_data['text'])
|
||||||
changed = detect_change(competitor['id'], content_hash)
|
changed = detect_change(competitor['id'], target_url, content_hash)
|
||||||
|
|
||||||
# Every refresh scrape gets its own scrape_runs record purely to
|
# Every refresh scrape gets its own scrape_runs record purely to
|
||||||
# persist content_hash for the next comparison — the outer batch
|
# persist content_hash (and the url it was hashed from) for the next
|
||||||
# job's scrape_runs row (created before run_research_job) tracks
|
# comparison — the outer batch job's scrape_runs row (created before
|
||||||
# overall job status, not per-competitor hash history.
|
# run_research_job) tracks overall job status, not per-competitor
|
||||||
|
# hash history.
|
||||||
scrape_repository.create({
|
scrape_repository.create({
|
||||||
'tenant_id': tenant_id, 'competitor_id': competitor['id'],
|
'tenant_id': tenant_id, 'competitor_id': competitor['id'],
|
||||||
'triggered_by': 'cron', 'status': 'complete',
|
'triggered_by': 'cron', 'status': 'complete',
|
||||||
'content_hash': content_hash, 'content_length': len(page_data['text']),
|
'content_hash': content_hash, 'content_length': len(page_data['text']),
|
||||||
'hash_algorithm': 'sha256', 'competitors_total': 1, 'competitors_done': 1,
|
'hash_algorithm': 'sha256', 'scraped_url': target_url,
|
||||||
|
'competitors_total': 1, 'competitors_done': 1,
|
||||||
'results': {}, 'completed_at': datetime.now(timezone.utc).isoformat(),
|
'results': {}, 'completed_at': datetime.now(timezone.utc).isoformat(),
|
||||||
})
|
})
|
||||||
|
|
||||||
|
|
@ -256,7 +364,8 @@ def run_competitor_analysis(tenant_id, competitor, context):
|
||||||
'last_scraped': datetime.now(timezone.utc).isoformat(),
|
'last_scraped': datetime.now(timezone.utc).isoformat(),
|
||||||
})
|
})
|
||||||
return {
|
return {
|
||||||
'path': 'refresh', 'result': analyse_from_cache(tenant_id, competitor['id'], context),
|
'path': 'refresh',
|
||||||
|
'result': analyse_from_cache(tenant_id, competitor['id'], {**context, 'confirmed_url': confirmed_url}),
|
||||||
'unchanged': True, 'is_generic_page': is_generic,
|
'unchanged': True, 'is_generic_page': is_generic,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -274,7 +383,8 @@ def run_competitor_analysis(tenant_id, competitor, context):
|
||||||
context.get('product_name'), context.get('destination'),
|
context.get('product_name'), context.get('destination'),
|
||||||
context.get('duration'), context.get('travel_style'),
|
context.get('duration'), context.get('travel_style'),
|
||||||
company_name=context.get('company_name'),
|
company_name=context.get('company_name'),
|
||||||
is_ngs=context.get('tab_type') == 'NGS'
|
is_ngs=context.get('tab_type') == 'NGS',
|
||||||
|
own_trip=context.get('own_trip'),
|
||||||
)
|
)
|
||||||
extracted = extract_with_openrouter(prompt, content=page_data['text'])
|
extracted = extract_with_openrouter(prompt, content=page_data['text'])
|
||||||
product = _save_scrape_result(tenant_id, competitor, extracted, page_data)
|
product = _save_scrape_result(tenant_id, competitor, extracted, page_data)
|
||||||
|
|
|
||||||
|
|
@ -3,44 +3,58 @@ from repositories.tenant_repository import tenant_repository
|
||||||
from repositories.seat_repository import seat_repository
|
from repositories.seat_repository import seat_repository
|
||||||
|
|
||||||
|
|
||||||
def get_tenant_id_from_request(request):
|
def get_seat_from_request(request):
|
||||||
"""
|
"""
|
||||||
Resolves the tenant_id for a dashboard request from its
|
Resolves both the tenant_id AND the specific active seat record for a
|
||||||
`Authorization: Bearer {jwt}` header. The JWT itself is issued and
|
dashboard request from its `Authorization: Bearer {jwt}` header. The
|
||||||
verified by PocketBase (the dashboard authenticates directly against
|
JWT itself is issued and verified by PocketBase (the dashboard
|
||||||
PocketBase) — this backend doesn't hold the JWT signing secret, so
|
authenticates directly against PocketBase) — this backend doesn't
|
||||||
it validates the token by asking PocketBase to refresh it, then
|
hold the JWT signing secret, so it validates the token by asking
|
||||||
resolves the returned email to a tenant via the existing seat record.
|
PocketBase to refresh it, then resolves the returned email to a
|
||||||
|
tenant and seat.
|
||||||
|
|
||||||
|
Needed by seat-scoped features (the trip watchlist's per-seat cap)
|
||||||
|
where "which tenant" isn't enough — every other authenticated route
|
||||||
|
only ever needed tenant_id, so the seat record this lookup already
|
||||||
|
finds was previously discarded (see get_tenant_id_from_request()
|
||||||
|
below, now a thin wrapper around this).
|
||||||
|
|
||||||
Data flow:
|
Data flow:
|
||||||
request.headers['Authorization'] → strip "Bearer " prefix →
|
request.headers['Authorization'] → strip "Bearer " prefix →
|
||||||
PocketBase POST /api/collections/users/auth-refresh →
|
PocketBase POST /api/collections/users/auth-refresh →
|
||||||
{ record: { email } } → seat lookup by email across known tenants
|
{ record: { email } } → seat lookup by email across known tenants
|
||||||
(via domain) → tenant_id returned, or None if unresolvable
|
(via domain) → (tenant_id, seat) returned, or (None, None) if
|
||||||
|
unresolvable
|
||||||
"""
|
"""
|
||||||
auth_header = request.headers.get('Authorization', '')
|
auth_header = request.headers.get('Authorization', '')
|
||||||
if not auth_header.startswith('Bearer '):
|
if not auth_header.startswith('Bearer '):
|
||||||
return None
|
return None, None
|
||||||
token = auth_header[len('Bearer '):]
|
token = auth_header[len('Bearer '):]
|
||||||
|
|
||||||
record = pb_client.auth_refresh(token)
|
record = pb_client.auth_refresh(token)
|
||||||
if not record or not record.get('email'):
|
if not record or not record.get('email'):
|
||||||
return None
|
return None, None
|
||||||
|
|
||||||
email = record['email']
|
email = record['email']
|
||||||
domain = email.split('@', 1)[1].lower() if '@' in email else None
|
domain = email.split('@', 1)[1].lower() if '@' in email else None
|
||||||
if not domain:
|
if not domain:
|
||||||
return None
|
return None, None
|
||||||
|
|
||||||
tenant = tenant_repository.get_by_domain(domain)
|
tenant = tenant_repository.get_by_domain(domain)
|
||||||
if not tenant:
|
if not tenant:
|
||||||
return None
|
return None, None
|
||||||
|
|
||||||
# Confirm this email actually holds an active seat on the tenant —
|
# Confirm this email actually holds an active seat on the tenant —
|
||||||
# a verified PocketBase login alone isn't sufficient authorization,
|
# a verified PocketBase login alone isn't sufficient authorization,
|
||||||
# the email must also be a registered, active seat.
|
# the email must also be a registered, active seat.
|
||||||
seat = seat_repository.get_active(tenant['id'], email)
|
seat = seat_repository.get_active(tenant['id'], email)
|
||||||
if not seat:
|
if not seat:
|
||||||
return None
|
return None, None
|
||||||
|
|
||||||
return tenant['id']
|
return tenant['id'], seat
|
||||||
|
|
||||||
|
|
||||||
|
def get_tenant_id_from_request(request):
|
||||||
|
"""Back-compat wrapper — most routes only need the tenant_id, not the seat."""
|
||||||
|
tenant_id, _ = get_seat_from_request(request)
|
||||||
|
return tenant_id
|
||||||
|
|
|
||||||
|
|
@ -10,7 +10,8 @@ EMBEDDING_MODEL = 'openai/text-embedding-3-small'
|
||||||
SIMILARITY_THRESHOLD = 0.75
|
SIMILARITY_THRESHOLD = 0.75
|
||||||
|
|
||||||
|
|
||||||
def save_client_trip(tenant_id, destination, duration, travel_style, product_name=None):
|
def save_client_trip(tenant_id, destination, duration, travel_style, product_name=None,
|
||||||
|
url=None, extracted=None, price_usd=None, activities=None):
|
||||||
"""
|
"""
|
||||||
Upserts a client_trips record from a PM's trip-intent description
|
Upserts a client_trips record from a PM's trip-intent description
|
||||||
(CLAUDE.md §8's TripIntentForm — destination/duration/travel_style,
|
(CLAUDE.md §8's TripIntentForm — destination/duration/travel_style,
|
||||||
|
|
@ -19,12 +20,19 @@ def save_client_trip(tenant_id, destination, duration, travel_style, product_nam
|
||||||
it, find_comparable_trips() has nothing on the client side to match
|
it, find_comparable_trips() has nothing on the client side to match
|
||||||
competitor products against.
|
competitor products against.
|
||||||
|
|
||||||
|
url/extracted/price_usd/activities are optional and only ever passed
|
||||||
|
by scrape_own_trip() below, once the tenant's own trip page has
|
||||||
|
actually been scraped and extracted — the original typed-values-only
|
||||||
|
call site in api.py's /research route passes none of these, so its
|
||||||
|
behavior is unchanged.
|
||||||
|
|
||||||
Dedupes on (tenant_id, trip_name) so re-running analysis for the
|
Dedupes on (tenant_id, trip_name) so re-running analysis for the
|
||||||
same product doesn't create duplicate rows — it just keeps the
|
same product doesn't create duplicate rows — it just keeps the
|
||||||
existing one current.
|
existing one current.
|
||||||
|
|
||||||
Data flow:
|
Data flow:
|
||||||
destination/duration/travel_style/product_name →
|
destination/duration/travel_style/product_name (+ optional
|
||||||
|
url/extracted/price_usd/activities from a real scrape) →
|
||||||
trip_name resolved (product_name if given, else a
|
trip_name resolved (product_name if given, else a
|
||||||
destination+style fallback) →
|
destination+style fallback) →
|
||||||
client_trips_repository.get_by_tenant_and_name() →
|
client_trips_repository.get_by_tenant_and_name() →
|
||||||
|
|
@ -38,6 +46,15 @@ def save_client_trip(tenant_id, destination, duration, travel_style, product_nam
|
||||||
'destination': destination,
|
'destination': destination,
|
||||||
'duration_days': duration,
|
'duration_days': duration,
|
||||||
}
|
}
|
||||||
|
if url is not None:
|
||||||
|
payload['url'] = url
|
||||||
|
if extracted is not None:
|
||||||
|
payload['extracted'] = extracted
|
||||||
|
if price_usd is not None:
|
||||||
|
payload['price_usd'] = price_usd
|
||||||
|
if activities is not None:
|
||||||
|
payload['activities'] = activities
|
||||||
|
|
||||||
existing = client_trips_repository.get_by_tenant_and_name(tenant_id, trip_name)
|
existing = client_trips_repository.get_by_tenant_and_name(tenant_id, trip_name)
|
||||||
if existing:
|
if existing:
|
||||||
# A stale embedding would silently never get regenerated —
|
# A stale embedding would silently never get regenerated —
|
||||||
|
|
@ -49,6 +66,96 @@ def save_client_trip(tenant_id, destination, duration, travel_style, product_nam
|
||||||
return client_trips_repository.create(payload)
|
return client_trips_repository.create(payload)
|
||||||
|
|
||||||
|
|
||||||
|
def scrape_own_trip(tenant_id, website_url, context):
|
||||||
|
"""
|
||||||
|
Scrapes and extracts the tenant's OWN confirmed/matched trip page —
|
||||||
|
the same Firecrawl-matched-or-manually-swapped URL pattern already
|
||||||
|
used for competitors, pointed at the tenant's own site instead. Gives
|
||||||
|
build_prompt()'s own_trip param and client_trips real ground-truth
|
||||||
|
data instead of only the PM's typed destination/duration/style.
|
||||||
|
|
||||||
|
Reuses the same CACHE_STALENESS_HOURS freshness window as competitors
|
||||||
|
(core/scraper.py) so the tenant's own site isn't rescraped on every
|
||||||
|
single analysis run — only when the cached client_trips row is stale
|
||||||
|
or context['force_refresh'] is set, both exactly mirroring
|
||||||
|
analysis_service.needs_refresh()'s competitor logic.
|
||||||
|
|
||||||
|
Never raises — a side-effect failure here (site blocked, proxy
|
||||||
|
error, extraction failure) must not fail the whole batch (CLAUDE.md
|
||||||
|
§18). Logs and returns None on any failure, which build_prompt()
|
||||||
|
treats identically to a tenant who never confirmed an own-trip URL.
|
||||||
|
|
||||||
|
The returned dict always carries a `cached` bool (True when served
|
||||||
|
from client_trips.extracted without a live scrape, False when just
|
||||||
|
freshly extracted) — surfaced to the dashboard via jobs.py's
|
||||||
|
results['own_trip'] so ResultsTable.vue can show the same "from
|
||||||
|
PocketBase vs new scrape" pill on the own-trip column as every
|
||||||
|
competitor column. `link` is always overwritten with the real
|
||||||
|
website_url actually scraped, rather than trusting the LLM's own
|
||||||
|
echoed `"link"` field verbatim — same reasoning as
|
||||||
|
analysis_service._save_scrape_result() using page_data['url'], not
|
||||||
|
extracted.get('link'), for a competitor's stored product URL.
|
||||||
|
|
||||||
|
Data flow:
|
||||||
|
website_url + context → trip_name resolved exactly as
|
||||||
|
save_client_trip() does (product_name, else destination+style) →
|
||||||
|
existing client_trips row fresh (and not force_refresh)? → return
|
||||||
|
its cached `extracted` blob (cached=True) → otherwise:
|
||||||
|
core.scraper.scrape() → build_own_trip_prompt() →
|
||||||
|
core.extractor.extract_with_openrouter() → link corrected to the
|
||||||
|
real website_url → save_client_trip() with
|
||||||
|
url/extracted/price_usd/activities → extracted dict returned
|
||||||
|
(cached=False)
|
||||||
|
"""
|
||||||
|
from datetime import datetime, timedelta, timezone
|
||||||
|
|
||||||
|
trip_name = context.get('product_name') or f"{context.get('destination')} {context.get('travel_style')}".strip()
|
||||||
|
existing = client_trips_repository.get_by_tenant_and_name(tenant_id, trip_name)
|
||||||
|
force_refresh = context.get('force_refresh', False)
|
||||||
|
|
||||||
|
if existing and existing.get('extracted') and not force_refresh:
|
||||||
|
last_scraped = existing.get('last_scraped')
|
||||||
|
if last_scraped:
|
||||||
|
try:
|
||||||
|
from core.scraper import CACHE_STALENESS_HOURS
|
||||||
|
scraped_at = datetime.fromisoformat(last_scraped.replace('Z', '+00:00'))
|
||||||
|
if datetime.now(timezone.utc) - scraped_at <= timedelta(hours=CACHE_STALENESS_HOURS):
|
||||||
|
logger.info(f'Own trip for tenant {tenant_id} — cache fresh, reusing extracted data')
|
||||||
|
return {**existing['extracted'], 'cached': True}
|
||||||
|
except (ValueError, AttributeError):
|
||||||
|
pass # unparseable timestamp — fall through and rescrape
|
||||||
|
|
||||||
|
try:
|
||||||
|
from core.scraper import scrape
|
||||||
|
from core.extractor import extract_with_openrouter
|
||||||
|
from industries.adventure_travel.prompt import build_own_trip_prompt
|
||||||
|
|
||||||
|
page_data = scrape(website_url)
|
||||||
|
if not page_data.get('success'):
|
||||||
|
logger.warning(f'Own-trip scrape failed for tenant {tenant_id}: {page_data.get("text")}')
|
||||||
|
return None
|
||||||
|
|
||||||
|
prompt = build_own_trip_prompt(
|
||||||
|
page_data, context.get('product_name'), context.get('destination'),
|
||||||
|
context.get('duration'), context.get('travel_style'),
|
||||||
|
context.get('company_name'), is_ngs=context.get('tab_type') == 'NGS',
|
||||||
|
)
|
||||||
|
extracted = extract_with_openrouter(prompt, content=page_data['text'])
|
||||||
|
extracted['link'] = website_url
|
||||||
|
|
||||||
|
saved = save_client_trip(
|
||||||
|
tenant_id, context.get('destination'), context.get('duration'),
|
||||||
|
context.get('travel_style'), product_name=context.get('product_name'),
|
||||||
|
url=website_url, extracted=extracted,
|
||||||
|
price_usd=extracted.get('majorityPrice'), activities=extracted.get('activities'),
|
||||||
|
)
|
||||||
|
client_trips_repository.update(saved['id'], {'last_scraped': datetime.now(timezone.utc).isoformat()})
|
||||||
|
return {**extracted, 'cached': False}
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f'scrape_own_trip failed for tenant {tenant_id}: {str(e)}')
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
def embed_trip(trip):
|
def embed_trip(trip):
|
||||||
"""
|
"""
|
||||||
Generates a semantic embedding for a trip using OpenRouter's
|
Generates a semantic embedding for a trip using OpenRouter's
|
||||||
|
|
|
||||||
|
|
@ -11,6 +11,16 @@ FIRECRAWL_URL = os.getenv('FIRECRAWL_URL', 'http://firecrawl:3002')
|
||||||
CATEGORY_KEYWORDS = ['classic', 'trek', 'trail', 'safari', 'adventure', 'explorer', 'journey']
|
CATEGORY_KEYWORDS = ['classic', 'trek', 'trail', 'safari', 'adventure', 'explorer', 'journey']
|
||||||
CONFIDENCE_FOUND_THRESHOLD = 0.3
|
CONFIDENCE_FOUND_THRESHOLD = 0.3
|
||||||
MAX_CONCURRENT_MAP_CALLS = 5
|
MAX_CONCURRENT_MAP_CALLS = 5
|
||||||
|
# Firecrawl /map fetches the target site's sitemap.xml before scoring
|
||||||
|
# candidates — a real production site with a large catalogue (thousands
|
||||||
|
# of URLs, e.g. a tenant's own established storefront rather than a
|
||||||
|
# smaller competitor test site) can take well over 15s just for that
|
||||||
|
# fetch. Confirmed directly: a real /map call against a large site's
|
||||||
|
# sitemap took ~21.6s, which a 15s timeout silently turned into a false
|
||||||
|
# "no match found" (requests.RequestException swallowed below) even
|
||||||
|
# though a retry with more time succeeded in ~2s. 30s covers this with
|
||||||
|
# margin while still failing within a bounded time on a truly dead site.
|
||||||
|
FIRECRAWL_MAP_TIMEOUT_SECONDS = 30
|
||||||
|
|
||||||
|
|
||||||
def score_and_rank_urls(urls, destination, duration):
|
def score_and_rank_urls(urls, destination, duration):
|
||||||
|
|
@ -97,7 +107,7 @@ def find_competitor_trip_url(root_url, destination, duration, travel_style):
|
||||||
response = requests.post(
|
response = requests.post(
|
||||||
f'{FIRECRAWL_URL}/v1/map',
|
f'{FIRECRAWL_URL}/v1/map',
|
||||||
json={'url': root_url, 'search': f'{destination} {duration} days {travel_style} tour'},
|
json={'url': root_url, 'search': f'{destination} {duration} days {travel_style} tour'},
|
||||||
timeout=15
|
timeout=FIRECRAWL_MAP_TIMEOUT_SECONDS
|
||||||
)
|
)
|
||||||
response.raise_for_status()
|
response.raise_for_status()
|
||||||
urls = response.json().get('links', [])
|
urls = response.json().get('links', [])
|
||||||
|
|
@ -111,6 +121,23 @@ def find_competitor_trip_url(root_url, destination, duration, travel_style):
|
||||||
return score_and_rank_urls(urls, destination, duration)
|
return score_and_rank_urls(urls, destination, duration)
|
||||||
|
|
||||||
|
|
||||||
|
def find_own_trip_url(website, destination, duration, travel_style):
|
||||||
|
"""
|
||||||
|
Finds the tenant's own best-matching trip page on their own site,
|
||||||
|
using the exact same Firecrawl /map + scoring gate as competitor
|
||||||
|
matching (score_and_rank_urls()'s destination-keyword hard gate and
|
||||||
|
root-domain exclusion apply unchanged) — named separately purely for
|
||||||
|
call-site clarity ("our own site" vs "a competitor's site"), not
|
||||||
|
because the matching behavior differs.
|
||||||
|
|
||||||
|
Data flow:
|
||||||
|
website (tenant's own storefront root) + destination/duration/
|
||||||
|
travel_style → find_competitor_trip_url() → { url, confidence,
|
||||||
|
found }
|
||||||
|
"""
|
||||||
|
return find_competitor_trip_url(website, destination, duration, travel_style)
|
||||||
|
|
||||||
|
|
||||||
def find_all_competitor_urls(competitors, destination, duration, travel_style):
|
def find_all_competitor_urls(competitors, destination, duration, travel_style):
|
||||||
"""
|
"""
|
||||||
Runs trip URL finding for all selected competitors concurrently —
|
Runs trip URL finding for all selected competitors concurrently —
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,88 @@
|
||||||
|
import logging
|
||||||
|
from repositories.watchlist_repository import watchlist_repository
|
||||||
|
from repositories.seat_repository import seat_repository
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
# Per-seat cap on watched trips, keyed by tenant tier — bounds the nightly
|
||||||
|
# cron's scrape volume predictably (seats x cap) instead of the old
|
||||||
|
# "re-scrape every competitor's homepage" approach. 1/seat at Analyse
|
||||||
|
# means a fully-seated ($349, 5 seats) tenant tops out at 5 real trip
|
||||||
|
# scrapes a night, same ceiling as scraping every competitor once, but
|
||||||
|
# now every alert is about a trip a PM actually chose to track instead of
|
||||||
|
# homepage noise. Discover/Intelligence values are placeholders — those
|
||||||
|
# tiers aren't built yet (CLAUDE.md §27) and will also get full catalogue
|
||||||
|
# auto-discovery on top, so the watchlist there is "pin your priorities,"
|
||||||
|
# not the only detection mechanism.
|
||||||
|
WATCHLIST_CAP_PER_SEAT = {
|
||||||
|
'analyse': 1,
|
||||||
|
'discover': 2,
|
||||||
|
'intelligence': 3,
|
||||||
|
'enterprise': 3,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def add_to_watchlist(tenant, seat, competitor_id, url, trip_name):
|
||||||
|
"""
|
||||||
|
Adds a specific competitor trip to a seat's personal watchlist,
|
||||||
|
enforcing WATCHLIST_CAP_PER_SEAT[tenant['tier']] against the seat's
|
||||||
|
current count first — same count-vs-cap shape as
|
||||||
|
licence_service.validate_email()'s seat check and api.py's
|
||||||
|
add_seat(). Never silently overwrites or evicts an existing item.
|
||||||
|
|
||||||
|
Data flow:
|
||||||
|
tenant + seat + competitor_id/url/trip_name →
|
||||||
|
watchlist_repository.count_for_seat(seat['id']) →
|
||||||
|
count >= cap for this tier → {'success': False, 'reason': ...} →
|
||||||
|
otherwise → watchlist_repository.create() →
|
||||||
|
{'success': True, 'item': created_record}
|
||||||
|
"""
|
||||||
|
cap = WATCHLIST_CAP_PER_SEAT.get(tenant.get('tier'), 1)
|
||||||
|
current_count = watchlist_repository.count_for_seat(seat['id'])
|
||||||
|
if current_count >= cap:
|
||||||
|
return {
|
||||||
|
'success': False,
|
||||||
|
'reason': f'Your watchlist is full ({current_count}/{cap}). Remove a trip to add another.',
|
||||||
|
}
|
||||||
|
|
||||||
|
item = watchlist_repository.create({
|
||||||
|
'tenant_id': tenant['id'],
|
||||||
|
'seat_id': seat['id'],
|
||||||
|
'competitor_id': competitor_id,
|
||||||
|
'url': url,
|
||||||
|
'trip_name': trip_name,
|
||||||
|
})
|
||||||
|
return {'success': True, 'item': item}
|
||||||
|
|
||||||
|
|
||||||
|
def remove_from_watchlist(item_id):
|
||||||
|
"""
|
||||||
|
Removes a watchlist item outright. Any active seat on the tenant may
|
||||||
|
remove any item — no per-seat ownership lock, matching this app's
|
||||||
|
existing no-fine-grained-permissions convention (e.g. CompetitorsPage
|
||||||
|
edits aren't restricted to whoever added a competitor either).
|
||||||
|
Returns True if a record was deleted, False if item_id was unknown.
|
||||||
|
"""
|
||||||
|
return watchlist_repository.delete(item_id)
|
||||||
|
|
||||||
|
|
||||||
|
def list_watchlist_for_tenant(tenant_id):
|
||||||
|
"""
|
||||||
|
Returns every watched item for a tenant (all seats'), each enriched
|
||||||
|
with the watching seat's email for display — a PM should be able to
|
||||||
|
see what teammates are tracking even though the cap itself is
|
||||||
|
per-seat. A seat that's since been removed (seat_repository.get_by_id
|
||||||
|
returns None) still shows the item with watched_by_email=None rather
|
||||||
|
than raising, so a removed teammate's picks don't break the whole list.
|
||||||
|
|
||||||
|
Data flow:
|
||||||
|
tenant_id → watchlist_repository.list_for_tenant() →
|
||||||
|
per item: seat_repository.get_by_id(item['seat_id']) →
|
||||||
|
item + watched_by_email merged → list returned
|
||||||
|
"""
|
||||||
|
items = watchlist_repository.list_for_tenant(tenant_id)
|
||||||
|
enriched = []
|
||||||
|
for item in items:
|
||||||
|
seat = seat_repository.get_by_id(item.get('seat_id'))
|
||||||
|
enriched.append({**item, 'watched_by_email': seat.get('email') if seat else None})
|
||||||
|
return enriched
|
||||||
|
|
@ -96,12 +96,13 @@ def _reset_mock_repositories():
|
||||||
from repositories.price_history_repository import price_history_repository
|
from repositories.price_history_repository import price_history_repository
|
||||||
from repositories.client_trips_repository import client_trips_repository
|
from repositories.client_trips_repository import client_trips_repository
|
||||||
from repositories.ngs_meta_repository import ngs_meta_repository
|
from repositories.ngs_meta_repository import ngs_meta_repository
|
||||||
|
from repositories.watchlist_repository import watchlist_repository
|
||||||
|
|
||||||
for repo in [
|
for repo in [
|
||||||
tenant_repository, seat_repository, competitor_repository, product_repository,
|
tenant_repository, seat_repository, competitor_repository, product_repository,
|
||||||
scrape_repository, proxy_repository, alert_repository, battlecard_repository,
|
scrape_repository, proxy_repository, alert_repository, battlecard_repository,
|
||||||
comparable_repository, monitoring_repository, price_history_repository,
|
comparable_repository, monitoring_repository, price_history_repository,
|
||||||
client_trips_repository, ngs_meta_repository,
|
client_trips_repository, ngs_meta_repository, watchlist_repository,
|
||||||
]:
|
]:
|
||||||
if hasattr(repo, '_records'):
|
if hasattr(repo, '_records'):
|
||||||
repo._records.clear()
|
repo._records.clear()
|
||||||
|
|
|
||||||
|
|
@ -19,6 +19,7 @@ from repositories.brief_repository import BriefRepository
|
||||||
from repositories.monitoring_repository import MonitoringRepository
|
from repositories.monitoring_repository import MonitoringRepository
|
||||||
from repositories.price_history_repository import PriceHistoryRepository
|
from repositories.price_history_repository import PriceHistoryRepository
|
||||||
from repositories.client_trips_repository import ClientTripsRepository
|
from repositories.client_trips_repository import ClientTripsRepository
|
||||||
|
from repositories.watchlist_repository import WatchlistRepository
|
||||||
|
|
||||||
|
|
||||||
class _FakePbClient:
|
class _FakePbClient:
|
||||||
|
|
@ -77,6 +78,7 @@ def test_seat_repository_delegates_correctly(monkeypatch):
|
||||||
repo = SeatRepository()
|
repo = SeatRepository()
|
||||||
|
|
||||||
assert repo.get_active('t1', 'pm@gadventures.com') == fake.get_first_return
|
assert repo.get_active('t1', 'pm@gadventures.com') == fake.get_first_return
|
||||||
|
assert repo.get_by_id('s1') == fake.get_one_return
|
||||||
assert repo.count_active('t1') == 2
|
assert repo.count_active('t1') == 2
|
||||||
assert repo.list_for_tenant('t1') == fake.list_return
|
assert repo.list_for_tenant('t1') == fake.list_return
|
||||||
assert repo.create({'email': 'x'}) == fake.create_return
|
assert repo.create({'email': 'x'}) == fake.create_return
|
||||||
|
|
@ -108,6 +110,7 @@ def test_product_repository_delegates_correctly(monkeypatch):
|
||||||
assert repo.create({'trip_name': 'x'}) == fake.create_return
|
assert repo.create({'trip_name': 'x'}) == fake.create_return
|
||||||
assert repo.update('p1', {'majority_price_usd': 100}) == fake.update_return
|
assert repo.update('p1', {'majority_price_usd': 100}) == fake.update_return
|
||||||
assert repo.get_by_competitor_and_name('c1', 'Peru Classic') == fake.get_first_return
|
assert repo.get_by_competitor_and_name('c1', 'Peru Classic') == fake.get_first_return
|
||||||
|
assert repo.get_by_competitor_and_url('c1', 'https://intrepidtravel.com/peru') == fake.get_first_return
|
||||||
|
|
||||||
|
|
||||||
def test_product_repository_get_by_competitor_and_name_none_without_trip_name(monkeypatch):
|
def test_product_repository_get_by_competitor_and_name_none_without_trip_name(monkeypatch):
|
||||||
|
|
@ -119,6 +122,15 @@ def test_product_repository_get_by_competitor_and_name_none_without_trip_name(mo
|
||||||
assert repo.get_by_competitor_and_name('c1', '') is None
|
assert repo.get_by_competitor_and_name('c1', '') is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_product_repository_get_by_competitor_and_url_none_without_url(monkeypatch):
|
||||||
|
fake = _FakePbClient()
|
||||||
|
monkeypatch.setattr('repositories.product_repository.pb_client', fake)
|
||||||
|
repo = ProductRepository()
|
||||||
|
|
||||||
|
assert repo.get_by_competitor_and_url('c1', None) is None
|
||||||
|
assert repo.get_by_competitor_and_url('c1', '') is None
|
||||||
|
|
||||||
|
|
||||||
def test_scrape_repository_delegates_correctly(monkeypatch):
|
def test_scrape_repository_delegates_correctly(monkeypatch):
|
||||||
fake = _FakePbClient()
|
fake = _FakePbClient()
|
||||||
fake.list_return = [{'id': 'r1'}]
|
fake.list_return = [{'id': 'r1'}]
|
||||||
|
|
@ -147,7 +159,7 @@ def test_scrape_repository_get_last_hash_for_url_returns_hash(monkeypatch):
|
||||||
monkeypatch.setattr('repositories.scrape_repository.pb_client', fake)
|
monkeypatch.setattr('repositories.scrape_repository.pb_client', fake)
|
||||||
repo = ScrapeRepository()
|
repo = ScrapeRepository()
|
||||||
|
|
||||||
assert repo.get_last_hash_for_url('c1') == 'abc123'
|
assert repo.get_last_hash_for_url('c1', 'https://intrepidtravel.com/peru') == 'abc123'
|
||||||
|
|
||||||
|
|
||||||
def test_scrape_repository_get_last_hash_for_url_returns_none_when_never_hashed(monkeypatch):
|
def test_scrape_repository_get_last_hash_for_url_returns_none_when_never_hashed(monkeypatch):
|
||||||
|
|
@ -156,7 +168,7 @@ def test_scrape_repository_get_last_hash_for_url_returns_none_when_never_hashed(
|
||||||
monkeypatch.setattr('repositories.scrape_repository.pb_client', fake)
|
monkeypatch.setattr('repositories.scrape_repository.pb_client', fake)
|
||||||
repo = ScrapeRepository()
|
repo = ScrapeRepository()
|
||||||
|
|
||||||
assert repo.get_last_hash_for_url('c1') is None
|
assert repo.get_last_hash_for_url('c1', 'https://intrepidtravel.com/peru') is None
|
||||||
|
|
||||||
|
|
||||||
def test_proxy_repository_delegates_correctly(monkeypatch):
|
def test_proxy_repository_delegates_correctly(monkeypatch):
|
||||||
|
|
@ -268,3 +280,17 @@ def test_price_history_repository_delegates_correctly(monkeypatch):
|
||||||
assert repo.create({'majority_price_usd': 2190}) == fake.create_return
|
assert repo.create({'majority_price_usd': 2190}) == fake.create_return
|
||||||
assert repo.list_for_product('p1') == fake.list_return
|
assert repo.list_for_product('p1') == fake.list_return
|
||||||
assert fake.calls[-1][1] == 'price_history'
|
assert fake.calls[-1][1] == 'price_history'
|
||||||
|
|
||||||
|
|
||||||
|
def test_watchlist_repository_delegates_correctly(monkeypatch):
|
||||||
|
fake = _FakePbClient()
|
||||||
|
fake.list_return = [{'id': 'w1'}, {'id': 'w2'}]
|
||||||
|
monkeypatch.setattr('repositories.watchlist_repository.pb_client', fake)
|
||||||
|
repo = WatchlistRepository()
|
||||||
|
|
||||||
|
assert repo.create({'seat_id': 's1'}) == fake.create_return
|
||||||
|
assert repo.get_by_id('w1') == fake.get_one_return
|
||||||
|
assert repo.list_for_tenant('t1') == fake.list_return
|
||||||
|
assert repo.count_for_seat('s1') == 2
|
||||||
|
assert repo.delete('w1') is True
|
||||||
|
assert fake.calls[-1] == ('delete', 'trip_watchlist', 'w1')
|
||||||
|
|
|
||||||
|
|
@ -77,6 +77,136 @@ def test_needs_refresh_true_on_malformed_timestamp():
|
||||||
assert analysis_service.needs_refresh({'change_detected': False, 'last_scraped': 'not-a-date'}) is True
|
assert analysis_service.needs_refresh({'change_detected': False, 'last_scraped': 'not-a-date'}) is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_needs_refresh_url_scoped_false_when_product_for_url_is_fresh():
|
||||||
|
# A competitor-wide change_detected=True (leftover from a DIFFERENT
|
||||||
|
# trip page's scrape) must NOT force a refresh for a url we already
|
||||||
|
# have fresh cached data for — the whole point of url-scoping.
|
||||||
|
_, competitor = _make_tenant_and_competitor(change_detected=True)
|
||||||
|
fresh = (datetime.now(timezone.utc) - timedelta(hours=1)).isoformat()
|
||||||
|
product_repository.create({
|
||||||
|
'tenant_id': 't1', 'competitor_id': competitor['id'], 'trip_name': 'Peru Classic',
|
||||||
|
'url': 'https://intrepidtravel.com/peru', 'last_seen': fresh,
|
||||||
|
})
|
||||||
|
|
||||||
|
assert analysis_service.needs_refresh(competitor, confirmed_url='https://intrepidtravel.com/peru') is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_needs_refresh_url_scoped_true_when_no_product_for_that_url():
|
||||||
|
_, competitor = _make_tenant_and_competitor(change_detected=False)
|
||||||
|
fresh = (datetime.now(timezone.utc) - timedelta(hours=1)).isoformat()
|
||||||
|
product_repository.create({
|
||||||
|
'tenant_id': 't1', 'competitor_id': competitor['id'], 'trip_name': 'Vietnam Explorer',
|
||||||
|
'url': 'https://intrepidtravel.com/vietnam', 'last_seen': fresh,
|
||||||
|
})
|
||||||
|
|
||||||
|
# Fresh data exists, but for a different page entirely.
|
||||||
|
assert analysis_service.needs_refresh(competitor, confirmed_url='https://intrepidtravel.com/peru') is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_needs_refresh_url_scoped_true_when_product_for_url_is_stale():
|
||||||
|
_, competitor = _make_tenant_and_competitor(change_detected=False)
|
||||||
|
stale = (datetime.now(timezone.utc) - timedelta(hours=48)).isoformat()
|
||||||
|
product_repository.create({
|
||||||
|
'tenant_id': 't1', 'competitor_id': competitor['id'], 'trip_name': 'Peru Classic',
|
||||||
|
'url': 'https://intrepidtravel.com/peru', 'last_seen': stale,
|
||||||
|
})
|
||||||
|
|
||||||
|
assert analysis_service.needs_refresh(competitor, confirmed_url='https://intrepidtravel.com/peru') is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_analyse_from_cache_prefers_product_matching_confirmed_url_over_latest(monkeypatch):
|
||||||
|
# Regression test — analyse_from_cache() used to always grab whichever
|
||||||
|
# product was scraped most recently for the competitor, which could be
|
||||||
|
# a completely different trip than the one this run is actually about.
|
||||||
|
tenant, competitor = _make_tenant_and_competitor()
|
||||||
|
product_repository.create({
|
||||||
|
'tenant_id': tenant['id'], 'competitor_id': competitor['id'], 'trip_name': 'Peru Classic',
|
||||||
|
'url': 'https://intrepidtravel.com/peru', 'last_seen': '2026-01-01',
|
||||||
|
})
|
||||||
|
# Scraped later — "latest for competitor" would wrongly pick this one.
|
||||||
|
product_repository.create({
|
||||||
|
'tenant_id': tenant['id'], 'competitor_id': competitor['id'], 'trip_name': 'Vietnam Explorer',
|
||||||
|
'url': 'https://intrepidtravel.com/vietnam', 'last_seen': '2026-02-01',
|
||||||
|
})
|
||||||
|
_install_fake_module(
|
||||||
|
monkeypatch, 'core.extractor',
|
||||||
|
extract_with_openrouter=lambda prompt, content: {'comments': 'ok', 'relevancy': 4}
|
||||||
|
)
|
||||||
|
|
||||||
|
result = analysis_service.analyse_from_cache(
|
||||||
|
tenant['id'], competitor['id'], {'destination': 'Peru', 'confirmed_url': 'https://intrepidtravel.com/peru'}
|
||||||
|
)
|
||||||
|
|
||||||
|
assert result['tripName'] == 'Peru Classic'
|
||||||
|
|
||||||
|
|
||||||
|
def test_run_competitor_analysis_reuses_cache_for_one_url_even_after_a_different_url_changed(monkeypatch):
|
||||||
|
# End-to-end regression test for the exact bug reported in
|
||||||
|
# production: a PM re-runs analysis on a URL they already researched,
|
||||||
|
# without checking "force fresh scrape", after a DIFFERENT trip page
|
||||||
|
# for the same competitor genuinely changed in between. Before this
|
||||||
|
# fix, that other page's change flipped change_detected back to True
|
||||||
|
# competitor-wide, which made needs_refresh() force a real re-scrape
|
||||||
|
# on literally every subsequent run for this competitor, regardless
|
||||||
|
# of which URL was actually being asked about.
|
||||||
|
tenant, competitor = _make_tenant_and_competitor()
|
||||||
|
peru_url = 'https://intrepidtravel.com/peru'
|
||||||
|
vietnam_url = 'https://intrepidtravel.com/vietnam'
|
||||||
|
|
||||||
|
_install_fake_module(monkeypatch, 'industries.adventure_travel.prompt', build_prompt=lambda *a, **k: '')
|
||||||
|
|
||||||
|
scraped_text = {'url': None}
|
||||||
|
|
||||||
|
def fake_scrape(url, country=None):
|
||||||
|
return {'success': True, 'url': url, 'text': scraped_text['url']}
|
||||||
|
_install_fake_module(monkeypatch, 'core.scraper', scrape=fake_scrape)
|
||||||
|
|
||||||
|
def fake_extract(prompt, content):
|
||||||
|
if content == 'peru v1':
|
||||||
|
return {'tripName': 'Peru Classic', 'majorityPrice': 2190}
|
||||||
|
if content in ('vietnam v1', 'vietnam v2'):
|
||||||
|
return {'tripName': 'Vietnam Explorer', 'majorityPrice': 1890}
|
||||||
|
return {'narrative': 'cached'} # analyse_from_cache's narrative-only call
|
||||||
|
_install_fake_module(monkeypatch, 'core.extractor', extract_with_openrouter=fake_extract)
|
||||||
|
|
||||||
|
# Run 1 — scrape Peru for the first time (real extraction, no prior
|
||||||
|
# hash to compare against for this url — establishes its baseline).
|
||||||
|
scraped_text['url'] = 'peru v1'
|
||||||
|
result1 = analysis_service.run_competitor_analysis(tenant['id'], competitor, {'confirmed_url': peru_url})
|
||||||
|
assert result1['path'] == 'refresh'
|
||||||
|
assert result1['result']['tripName'] == 'Peru Classic'
|
||||||
|
|
||||||
|
# Run 2 — scrape Vietnam for the first time (its own baseline —
|
||||||
|
# different url, no prior hash for IT either, so still no diff yet).
|
||||||
|
competitor = competitor_repository.get_by_id(competitor['id'])
|
||||||
|
scraped_text['url'] = 'vietnam v1'
|
||||||
|
result2 = analysis_service.run_competitor_analysis(tenant['id'], competitor, {'confirmed_url': vietnam_url})
|
||||||
|
assert result2['path'] == 'refresh'
|
||||||
|
|
||||||
|
# Run 3 — Vietnam's content genuinely changes. force_refresh here
|
||||||
|
# only to bypass the fast-path gate (Vietnam's own product would
|
||||||
|
# otherwise still look fresh from run 2) so the hash comparison
|
||||||
|
# actually runs against the new content — this is what sets the
|
||||||
|
# competitor-wide change_detected flag True, rooted entirely in
|
||||||
|
# Vietnam's page and nothing to do with Peru.
|
||||||
|
competitor = competitor_repository.get_by_id(competitor['id'])
|
||||||
|
scraped_text['url'] = 'vietnam v2'
|
||||||
|
analysis_service.run_competitor_analysis(
|
||||||
|
tenant['id'], competitor, {'confirmed_url': vietnam_url, 'force_refresh': True}
|
||||||
|
)
|
||||||
|
competitor = competitor_repository.get_by_id(competitor['id'])
|
||||||
|
assert competitor['change_detected'] is True # confirms the "leftover" state this fix must see past
|
||||||
|
|
||||||
|
# Run 4 — re-run Peru again, same URL and same content as run 1, no
|
||||||
|
# force_refresh. Must hit the FAST path (no re-scrape) since Peru's
|
||||||
|
# own cached product is fresh — despite the competitor-wide
|
||||||
|
# change_detected flag still being True from Vietnam's change.
|
||||||
|
result4 = analysis_service.run_competitor_analysis(tenant['id'], competitor, {'confirmed_url': peru_url})
|
||||||
|
|
||||||
|
assert result4['path'] == 'fast'
|
||||||
|
assert result4['result']['tripName'] == 'Peru Classic'
|
||||||
|
|
||||||
|
|
||||||
def test_analyse_from_cache_raises_without_cached_products():
|
def test_analyse_from_cache_raises_without_cached_products():
|
||||||
_, competitor = _make_tenant_and_competitor()
|
_, competitor = _make_tenant_and_competitor()
|
||||||
with pytest.raises(ValueError):
|
with pytest.raises(ValueError):
|
||||||
|
|
@ -103,6 +233,54 @@ def test_analyse_from_cache_uses_latest_product(monkeypatch):
|
||||||
assert 'Peru Classic' in captured['prompt']
|
assert 'Peru Classic' in captured['prompt']
|
||||||
|
|
||||||
|
|
||||||
|
def test_analyse_from_cache_grounds_comments_in_own_trip_when_present(monkeypatch):
|
||||||
|
# Regression test — the comparison prompt used to dump the raw
|
||||||
|
# context dict verbatim with no priority framing, so own_trip's
|
||||||
|
# presence was easy to under-weight against the typed fields sitting
|
||||||
|
# right next to it. own_trip must be the primary ground truth.
|
||||||
|
tenant, competitor = _make_tenant_and_competitor()
|
||||||
|
product_repository.create({
|
||||||
|
'tenant_id': tenant['id'], 'competitor_id': competitor['id'],
|
||||||
|
'trip_name': 'Peru Classic', 'last_seen': '2026-01-01',
|
||||||
|
})
|
||||||
|
captured = {}
|
||||||
|
_install_fake_module(
|
||||||
|
monkeypatch, 'core.extractor',
|
||||||
|
extract_with_openrouter=lambda prompt, content: captured.update(prompt=prompt) or {'comments': 'ok', 'relevancy': 4}
|
||||||
|
)
|
||||||
|
|
||||||
|
analysis_service.analyse_from_cache(tenant['id'], competitor['id'], {
|
||||||
|
'destination': 'Peru', 'product_name': 'Inca Trail', 'company_name': 'Wild Horizons Travel',
|
||||||
|
'own_trip': {'tripName': 'Inca Trail Classic', 'majorityPrice': 1999, 'duration': 15, 'activities': 'Trekking'},
|
||||||
|
})
|
||||||
|
|
||||||
|
assert 'ACTUAL extracted data' in captured['prompt']
|
||||||
|
assert "Wild Horizons Travel's own Inca Trail Classic" in captured['prompt']
|
||||||
|
assert '1999' in captured['prompt']
|
||||||
|
|
||||||
|
|
||||||
|
def test_analyse_from_cache_falls_back_to_typed_form_data_without_own_trip(monkeypatch):
|
||||||
|
tenant, competitor = _make_tenant_and_competitor()
|
||||||
|
product_repository.create({
|
||||||
|
'tenant_id': tenant['id'], 'competitor_id': competitor['id'],
|
||||||
|
'trip_name': 'Peru Classic', 'last_seen': '2026-01-01',
|
||||||
|
})
|
||||||
|
captured = {}
|
||||||
|
_install_fake_module(
|
||||||
|
monkeypatch, 'core.extractor',
|
||||||
|
extract_with_openrouter=lambda prompt, content: captured.update(prompt=prompt) or {'comments': 'ok', 'relevancy': 4}
|
||||||
|
)
|
||||||
|
|
||||||
|
analysis_service.analyse_from_cache(tenant['id'], competitor['id'], {
|
||||||
|
'destination': 'Peru', 'duration': 15, 'travel_style': 'Classic',
|
||||||
|
'product_name': 'Inca Trail', 'company_name': 'Wild Horizons Travel',
|
||||||
|
})
|
||||||
|
|
||||||
|
assert 'typed product intent' in captured['prompt']
|
||||||
|
assert 'ACTUAL extracted data' not in captured['prompt']
|
||||||
|
assert 'Inca Trail' in captured['prompt']
|
||||||
|
|
||||||
|
|
||||||
def test_analyse_from_cache_returns_structured_fields_resultstable_expects(monkeypatch):
|
def test_analyse_from_cache_returns_structured_fields_resultstable_expects(monkeypatch):
|
||||||
"""
|
"""
|
||||||
Regression test — a real "complete" job showed an empty-looking
|
Regression test — a real "complete" job showed an empty-looking
|
||||||
|
|
@ -565,7 +743,7 @@ def test_analyse_from_cache_skips_ngs_lookup_for_standard_tab(monkeypatch):
|
||||||
|
|
||||||
def test_detect_change_true_on_first_scrape():
|
def test_detect_change_true_on_first_scrape():
|
||||||
_, competitor = _make_tenant_and_competitor()
|
_, competitor = _make_tenant_and_competitor()
|
||||||
assert analysis_service.detect_change(competitor['id'], 'somehash') is True
|
assert analysis_service.detect_change(competitor['id'], 'https://intrepidtravel.com/peru', 'somehash') is True
|
||||||
# First scrape never had a prior hash to compare against, so it
|
# First scrape never had a prior hash to compare against, so it
|
||||||
# shouldn't flag change_detected — there's nothing to diff against.
|
# shouldn't flag change_detected — there's nothing to diff against.
|
||||||
assert competitor_repository.get_by_id(competitor['id'])['change_detected'] is False
|
assert competitor_repository.get_by_id(competitor['id'])['change_detected'] is False
|
||||||
|
|
@ -575,10 +753,11 @@ def test_detect_change_true_and_flags_competitor_on_diff():
|
||||||
_, competitor = _make_tenant_and_competitor()
|
_, competitor = _make_tenant_and_competitor()
|
||||||
scrape_repository.create({
|
scrape_repository.create({
|
||||||
'tenant_id': 't1', 'competitor_id': competitor['id'], 'triggered_by': 'cron',
|
'tenant_id': 't1', 'competitor_id': competitor['id'], 'triggered_by': 'cron',
|
||||||
'status': 'complete', 'content_hash': 'old-hash', 'competitors_total': 1, 'competitors_done': 1,
|
'status': 'complete', 'content_hash': 'old-hash', 'scraped_url': 'https://intrepidtravel.com/peru',
|
||||||
|
'competitors_total': 1, 'competitors_done': 1,
|
||||||
})
|
})
|
||||||
|
|
||||||
changed = analysis_service.detect_change(competitor['id'], 'new-hash')
|
changed = analysis_service.detect_change(competitor['id'], 'https://intrepidtravel.com/peru', 'new-hash')
|
||||||
|
|
||||||
assert changed is True
|
assert changed is True
|
||||||
updated = competitor_repository.get_by_id(competitor['id'])
|
updated = competitor_repository.get_by_id(competitor['id'])
|
||||||
|
|
@ -590,10 +769,40 @@ def test_detect_change_false_when_hash_unchanged():
|
||||||
_, competitor = _make_tenant_and_competitor()
|
_, competitor = _make_tenant_and_competitor()
|
||||||
scrape_repository.create({
|
scrape_repository.create({
|
||||||
'tenant_id': 't1', 'competitor_id': competitor['id'], 'triggered_by': 'cron',
|
'tenant_id': 't1', 'competitor_id': competitor['id'], 'triggered_by': 'cron',
|
||||||
'status': 'complete', 'content_hash': 'same-hash', 'competitors_total': 1, 'competitors_done': 1,
|
'status': 'complete', 'content_hash': 'same-hash', 'scraped_url': 'https://intrepidtravel.com/peru',
|
||||||
|
'competitors_total': 1, 'competitors_done': 1,
|
||||||
})
|
})
|
||||||
|
|
||||||
assert analysis_service.detect_change(competitor['id'], 'same-hash') is False
|
assert analysis_service.detect_change(competitor['id'], 'https://intrepidtravel.com/peru', 'same-hash') is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_detect_change_scoped_by_url_not_just_competitor():
|
||||||
|
# Regression test for the exact bug reported in production: two
|
||||||
|
# different trip pages scraped for the same competitor must keep
|
||||||
|
# independent hash histories. The old, url-blind lookup compared a
|
||||||
|
# fresh scrape of URL A against whichever page was hashed MOST
|
||||||
|
# RECENTLY for this competitor (URL B here) — almost always a
|
||||||
|
# mismatch, which kept change_detected permanently stuck True.
|
||||||
|
_, competitor = _make_tenant_and_competitor()
|
||||||
|
scrape_repository.create({
|
||||||
|
'tenant_id': 't1', 'competitor_id': competitor['id'], 'triggered_by': 'cron', 'status': 'complete',
|
||||||
|
'content_hash': 'hash-for-peru', 'scraped_url': 'https://intrepidtravel.com/peru',
|
||||||
|
'competitors_total': 1, 'competitors_done': 1,
|
||||||
|
})
|
||||||
|
# Created (and thus "most recently scraped") second — the old,
|
||||||
|
# url-blind lookup would compare against this row instead.
|
||||||
|
scrape_repository.create({
|
||||||
|
'tenant_id': 't1', 'competitor_id': competitor['id'], 'triggered_by': 'cron', 'status': 'complete',
|
||||||
|
'content_hash': 'hash-for-vietnam', 'scraped_url': 'https://intrepidtravel.com/vietnam',
|
||||||
|
'competitors_total': 1, 'competitors_done': 1,
|
||||||
|
})
|
||||||
|
|
||||||
|
# Re-scraping the Peru page again with the exact same content
|
||||||
|
# (unchanged) must report unchanged, even though Vietnam's hash
|
||||||
|
# (the most recently recorded row overall) differs entirely.
|
||||||
|
changed = analysis_service.detect_change(competitor['id'], 'https://intrepidtravel.com/peru', 'hash-for-peru')
|
||||||
|
|
||||||
|
assert changed is False
|
||||||
|
|
||||||
|
|
||||||
def test_run_competitor_analysis_refresh_path_skips_extraction_when_hash_unchanged(monkeypatch):
|
def test_run_competitor_analysis_refresh_path_skips_extraction_when_hash_unchanged(monkeypatch):
|
||||||
|
|
@ -609,7 +818,8 @@ def test_run_competitor_analysis_refresh_path_skips_extraction_when_hash_unchang
|
||||||
prior_hash = compute_content_hash('unchanged content')
|
prior_hash = compute_content_hash('unchanged content')
|
||||||
scrape_repository.create({
|
scrape_repository.create({
|
||||||
'tenant_id': tenant['id'], 'competitor_id': competitor['id'], 'triggered_by': 'cron',
|
'tenant_id': tenant['id'], 'competitor_id': competitor['id'], 'triggered_by': 'cron',
|
||||||
'status': 'complete', 'content_hash': prior_hash, 'competitors_total': 1, 'competitors_done': 1,
|
'status': 'complete', 'content_hash': prior_hash, 'scraped_url': competitor['website'],
|
||||||
|
'competitors_total': 1, 'competitors_done': 1,
|
||||||
})
|
})
|
||||||
|
|
||||||
# extract_with_openrouter backs both the full-page extraction call
|
# extract_with_openrouter backs both the full-page extraction call
|
||||||
|
|
@ -675,7 +885,8 @@ def test_force_refresh_bypasses_unchanged_shortcut_even_when_hash_matches(monkey
|
||||||
prior_hash = compute_content_hash('unchanged content')
|
prior_hash = compute_content_hash('unchanged content')
|
||||||
scrape_repository.create({
|
scrape_repository.create({
|
||||||
'tenant_id': tenant['id'], 'competitor_id': competitor['id'], 'triggered_by': 'cron',
|
'tenant_id': tenant['id'], 'competitor_id': competitor['id'], 'triggered_by': 'cron',
|
||||||
'status': 'complete', 'content_hash': prior_hash, 'competitors_total': 1, 'competitors_done': 1,
|
'status': 'complete', 'content_hash': prior_hash, 'scraped_url': competitor['website'],
|
||||||
|
'competitors_total': 1, 'competitors_done': 1,
|
||||||
})
|
})
|
||||||
|
|
||||||
calls_with_content = []
|
calls_with_content = []
|
||||||
|
|
|
||||||
|
|
@ -54,3 +54,57 @@ def test_returns_tenant_id_for_valid_active_seat(monkeypatch):
|
||||||
|
|
||||||
result = auth_service.get_tenant_id_from_request(_FakeRequest('Bearer validtoken'))
|
result = auth_service.get_tenant_id_from_request(_FakeRequest('Bearer validtoken'))
|
||||||
assert result == tenant['id']
|
assert result == tenant['id']
|
||||||
|
|
||||||
|
|
||||||
|
# ── get_seat_from_request ──────────────────────────────────────────
|
||||||
|
|
||||||
|
def test_get_seat_from_request_returns_none_none_without_bearer_prefix():
|
||||||
|
assert auth_service.get_seat_from_request(_FakeRequest('not-bearer-token')) == (None, None)
|
||||||
|
|
||||||
|
|
||||||
|
def test_get_seat_from_request_returns_none_none_when_no_active_seat(monkeypatch):
|
||||||
|
tenant_repository.create({
|
||||||
|
'name': 'G Adventures', 'domain': 'gadventures.com', 'tier': 'analyse',
|
||||||
|
'status': 'active', 'max_seats': 5,
|
||||||
|
})
|
||||||
|
monkeypatch.setattr(
|
||||||
|
'services.auth_service.pb_client.auth_refresh',
|
||||||
|
lambda token: {'email': 'pm@gadventures.com'}
|
||||||
|
)
|
||||||
|
assert auth_service.get_seat_from_request(_FakeRequest('Bearer sometoken')) == (None, None)
|
||||||
|
|
||||||
|
|
||||||
|
def test_get_seat_from_request_returns_tenant_id_and_seat_record(monkeypatch):
|
||||||
|
tenant = tenant_repository.create({
|
||||||
|
'name': 'G Adventures', 'domain': 'gadventures.com', 'tier': 'analyse',
|
||||||
|
'status': 'active', 'max_seats': 5,
|
||||||
|
})
|
||||||
|
seat = seat_repository.create({'tenant_id': tenant['id'], 'email': 'pm@gadventures.com', 'active': True})
|
||||||
|
monkeypatch.setattr(
|
||||||
|
'services.auth_service.pb_client.auth_refresh',
|
||||||
|
lambda token: {'email': 'pm@gadventures.com'}
|
||||||
|
)
|
||||||
|
|
||||||
|
tenant_id, resolved_seat = auth_service.get_seat_from_request(_FakeRequest('Bearer validtoken'))
|
||||||
|
|
||||||
|
assert tenant_id == tenant['id']
|
||||||
|
assert resolved_seat['id'] == seat['id']
|
||||||
|
assert resolved_seat['email'] == 'pm@gadventures.com'
|
||||||
|
|
||||||
|
|
||||||
|
def test_get_tenant_id_from_request_delegates_to_get_seat_from_request(monkeypatch):
|
||||||
|
# Regression test — get_tenant_id_from_request() is now a thin
|
||||||
|
# wrapper; confirm it still returns just the tenant_id half.
|
||||||
|
tenant = tenant_repository.create({
|
||||||
|
'name': 'G Adventures', 'domain': 'gadventures.com', 'tier': 'analyse',
|
||||||
|
'status': 'active', 'max_seats': 5,
|
||||||
|
})
|
||||||
|
seat_repository.create({'tenant_id': tenant['id'], 'email': 'pm@gadventures.com', 'active': True})
|
||||||
|
monkeypatch.setattr(
|
||||||
|
'services.auth_service.pb_client.auth_refresh',
|
||||||
|
lambda token: {'email': 'pm@gadventures.com'}
|
||||||
|
)
|
||||||
|
|
||||||
|
result = auth_service.get_tenant_id_from_request(_FakeRequest('Bearer validtoken'))
|
||||||
|
assert result == tenant['id']
|
||||||
|
assert not isinstance(result, tuple)
|
||||||
|
|
|
||||||
|
|
@ -120,3 +120,116 @@ def test_save_client_trip_keeps_embedding_when_nothing_relevant_changed():
|
||||||
updated = embedding_service.save_client_trip('t1', 'Peru', 15, 'Classic', product_name='Inca Trail')
|
updated = embedding_service.save_client_trip('t1', 'Peru', 15, 'Classic', product_name='Inca Trail')
|
||||||
|
|
||||||
assert updated['embedding'] == [0.1, 0.2, 0.3] # unchanged inputs — no need to invalidate
|
assert updated['embedding'] == [0.1, 0.2, 0.3] # unchanged inputs — no need to invalidate
|
||||||
|
|
||||||
|
|
||||||
|
def test_save_client_trip_persists_url_extracted_price_and_activities_when_given():
|
||||||
|
trip = embedding_service.save_client_trip(
|
||||||
|
't1', 'Peru', 15, 'Classic', product_name='Inca Trail',
|
||||||
|
url='https://client.com/inca-trail', extracted={'tripName': 'Inca Trail'},
|
||||||
|
price_usd=1999, activities='Trekking, Camping',
|
||||||
|
)
|
||||||
|
assert trip['url'] == 'https://client.com/inca-trail'
|
||||||
|
assert trip['extracted'] == {'tripName': 'Inca Trail'}
|
||||||
|
assert trip['price_usd'] == 1999
|
||||||
|
assert trip['activities'] == 'Trekking, Camping'
|
||||||
|
|
||||||
|
|
||||||
|
def test_save_client_trip_omits_own_trip_fields_when_not_given():
|
||||||
|
# The original typed-values-only call site (api.py's /research route)
|
||||||
|
# must be unaffected — no url/extracted/price_usd/activities keys at all.
|
||||||
|
trip = embedding_service.save_client_trip('t1', 'Peru', 15, 'Classic', product_name='Inca Trail')
|
||||||
|
assert 'url' not in trip
|
||||||
|
assert 'extracted' not in trip
|
||||||
|
assert 'price_usd' not in trip
|
||||||
|
assert 'activities' not in trip
|
||||||
|
|
||||||
|
|
||||||
|
def _fake_scrape(success=True, text='Peru Classic 15 days...'):
|
||||||
|
def _scrape(url):
|
||||||
|
return {'success': success, 'text': text, 'tables': '', 'url': url}
|
||||||
|
return _scrape
|
||||||
|
|
||||||
|
|
||||||
|
def test_scrape_own_trip_scrapes_and_extracts_when_no_cached_row(monkeypatch):
|
||||||
|
monkeypatch.setattr('core.scraper.scrape', _fake_scrape(), raising=False)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
'core.extractor.extract_with_openrouter',
|
||||||
|
lambda prompt, content='': {'tripName': 'Inca Trail Classic', 'majorityPrice': 1999, 'activities': 'Trekking'},
|
||||||
|
raising=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
context = {
|
||||||
|
'destination': 'Peru', 'duration': 15, 'travel_style': 'Classic',
|
||||||
|
'product_name': 'Inca Trail Classic', 'company_name': 'Wild Horizons', 'tab_type': 'Standard',
|
||||||
|
}
|
||||||
|
result = embedding_service.scrape_own_trip('t1', 'https://client.com/inca-trail', context)
|
||||||
|
|
||||||
|
assert result == {
|
||||||
|
'tripName': 'Inca Trail Classic', 'majorityPrice': 1999, 'activities': 'Trekking',
|
||||||
|
'link': 'https://client.com/inca-trail', 'cached': False,
|
||||||
|
}
|
||||||
|
saved = client_trips_repository.get_by_tenant_and_name('t1', 'Inca Trail Classic')
|
||||||
|
assert saved['url'] == 'https://client.com/inca-trail'
|
||||||
|
assert saved['price_usd'] == 1999
|
||||||
|
assert saved['last_scraped'] is not None
|
||||||
|
|
||||||
|
|
||||||
|
def test_scrape_own_trip_reuses_fresh_cached_extraction_without_rescraping(monkeypatch):
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
client_trips_repository.create({
|
||||||
|
'id': 'ct1', 'tenant_id': 't1', 'trip_name': 'Inca Trail Classic',
|
||||||
|
'destination': 'Peru', 'duration_days': 15,
|
||||||
|
'extracted': {'tripName': 'Inca Trail Classic', 'majorityPrice': 1999},
|
||||||
|
'last_scraped': datetime.now(timezone.utc).isoformat(),
|
||||||
|
})
|
||||||
|
|
||||||
|
calls = []
|
||||||
|
monkeypatch.setattr('core.scraper.scrape', lambda url: calls.append(url) or _fake_scrape()(url), raising=False)
|
||||||
|
|
||||||
|
context = {'destination': 'Peru', 'duration': 15, 'travel_style': 'Classic', 'product_name': 'Inca Trail Classic'}
|
||||||
|
result = embedding_service.scrape_own_trip('t1', 'https://client.com/inca-trail', context)
|
||||||
|
|
||||||
|
assert result == {'tripName': 'Inca Trail Classic', 'majorityPrice': 1999, 'cached': True}
|
||||||
|
assert calls == [] # cache was fresh — scrape() never called
|
||||||
|
|
||||||
|
|
||||||
|
def test_scrape_own_trip_rescrapes_when_force_refresh_even_if_fresh(monkeypatch):
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
client_trips_repository.create({
|
||||||
|
'id': 'ct1', 'tenant_id': 't1', 'trip_name': 'Inca Trail Classic',
|
||||||
|
'destination': 'Peru', 'duration_days': 15,
|
||||||
|
'extracted': {'tripName': 'stale'}, 'last_scraped': datetime.now(timezone.utc).isoformat(),
|
||||||
|
})
|
||||||
|
monkeypatch.setattr('core.scraper.scrape', _fake_scrape(), raising=False)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
'core.extractor.extract_with_openrouter',
|
||||||
|
lambda prompt, content='': {'tripName': 'fresh'}, raising=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
context = {
|
||||||
|
'destination': 'Peru', 'duration': 15, 'travel_style': 'Classic',
|
||||||
|
'product_name': 'Inca Trail Classic', 'force_refresh': True,
|
||||||
|
}
|
||||||
|
result = embedding_service.scrape_own_trip('t1', 'https://client.com/inca-trail', context)
|
||||||
|
|
||||||
|
assert result == {'tripName': 'fresh', 'link': 'https://client.com/inca-trail', 'cached': False}
|
||||||
|
|
||||||
|
|
||||||
|
def test_scrape_own_trip_returns_none_on_scrape_failure_without_raising(monkeypatch):
|
||||||
|
monkeypatch.setattr('core.scraper.scrape', _fake_scrape(success=False, text='Blocked'), raising=False)
|
||||||
|
|
||||||
|
context = {'destination': 'Peru', 'duration': 15, 'travel_style': 'Classic', 'product_name': 'Inca Trail Classic'}
|
||||||
|
result = embedding_service.scrape_own_trip('t1', 'https://client.com/inca-trail', context)
|
||||||
|
|
||||||
|
assert result is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_scrape_own_trip_returns_none_and_never_raises_on_unexpected_error(monkeypatch):
|
||||||
|
def _raise(url):
|
||||||
|
raise RuntimeError('boom')
|
||||||
|
monkeypatch.setattr('core.scraper.scrape', _raise, raising=False)
|
||||||
|
|
||||||
|
context = {'destination': 'Peru', 'duration': 15, 'travel_style': 'Classic', 'product_name': 'Inca Trail Classic'}
|
||||||
|
result = embedding_service.scrape_own_trip('t1', 'https://client.com/inca-trail', context)
|
||||||
|
|
||||||
|
assert result is None
|
||||||
|
|
|
||||||
|
|
@ -130,6 +130,104 @@ def test_run_research_job_threads_real_tenant_company_name_into_context(monkeypa
|
||||||
assert captured_contexts[0]['company_name'] == 'Wild Horizons Travel'
|
assert captured_contexts[0]['company_name'] == 'Wild Horizons Travel'
|
||||||
|
|
||||||
|
|
||||||
|
def test_run_research_job_threads_own_trip_result_into_context(monkeypatch):
|
||||||
|
tenant, competitor, run = _setup()
|
||||||
|
captured_contexts = []
|
||||||
|
monkeypatch.setattr(
|
||||||
|
'services.analysis_service.run_competitor_analysis',
|
||||||
|
lambda tenant_id, comp, ctx: captured_contexts.append(ctx) or {'path': 'refresh', 'result': {}}
|
||||||
|
)
|
||||||
|
monkeypatch.setattr('services.alert_service.send_gotify', lambda **kw: None)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
'services.embedding_service.scrape_own_trip',
|
||||||
|
lambda tenant_id, website_url, ctx: {'tripName': 'Inca Trail Classic', 'majorityPrice': 1999},
|
||||||
|
)
|
||||||
|
|
||||||
|
jobs.run_research_job(run['id'], {
|
||||||
|
'tenant_id': tenant['id'], 'competitors': [{'id': competitor['id']}],
|
||||||
|
'own_trip_url': 'https://wildhorizons.com/inca-trail',
|
||||||
|
})
|
||||||
|
|
||||||
|
assert captured_contexts[0]['own_trip'] == {'tripName': 'Inca Trail Classic', 'majorityPrice': 1999}
|
||||||
|
|
||||||
|
|
||||||
|
def test_run_research_job_surfaces_own_trip_in_persisted_results(monkeypatch):
|
||||||
|
# Regression test — own_trip was computed and fed into build_prompt()
|
||||||
|
# but never actually returned to the dashboard, so there was no way
|
||||||
|
# for a PM to confirm their own site was really scraped and used.
|
||||||
|
tenant, competitor, run = _setup()
|
||||||
|
monkeypatch.setattr(
|
||||||
|
'services.analysis_service.run_competitor_analysis',
|
||||||
|
lambda tenant_id, comp, ctx: {'path': 'refresh', 'result': {}}
|
||||||
|
)
|
||||||
|
monkeypatch.setattr('services.alert_service.send_gotify', lambda **kw: None)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
'services.embedding_service.scrape_own_trip',
|
||||||
|
lambda tenant_id, website_url, ctx: {'tripName': 'Inca Trail Classic', 'majorityPrice': 1999, 'cached': False},
|
||||||
|
)
|
||||||
|
|
||||||
|
jobs.run_research_job(run['id'], {
|
||||||
|
'tenant_id': tenant['id'], 'competitors': [{'id': competitor['id']}],
|
||||||
|
'own_trip_url': 'https://wildhorizons.com/inca-trail',
|
||||||
|
})
|
||||||
|
|
||||||
|
updated = scrape_repository.get_by_id(run['id'])
|
||||||
|
assert updated['results']['own_trip'] == {'tripName': 'Inca Trail Classic', 'majorityPrice': 1999, 'cached': False}
|
||||||
|
|
||||||
|
|
||||||
|
def test_run_research_job_own_trip_is_none_in_results_without_own_trip_url(monkeypatch):
|
||||||
|
tenant, competitor, run = _setup()
|
||||||
|
monkeypatch.setattr(
|
||||||
|
'services.analysis_service.run_competitor_analysis',
|
||||||
|
lambda tenant_id, comp, ctx: {'path': 'refresh', 'result': {}}
|
||||||
|
)
|
||||||
|
monkeypatch.setattr('services.alert_service.send_gotify', lambda **kw: None)
|
||||||
|
|
||||||
|
jobs.run_research_job(run['id'], {'tenant_id': tenant['id'], 'competitors': [{'id': competitor['id']}]})
|
||||||
|
|
||||||
|
updated = scrape_repository.get_by_id(run['id'])
|
||||||
|
assert updated['results']['own_trip'] is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_run_research_job_own_trip_defaults_to_none_without_own_trip_url(monkeypatch):
|
||||||
|
tenant, competitor, run = _setup()
|
||||||
|
captured_contexts = []
|
||||||
|
monkeypatch.setattr(
|
||||||
|
'services.analysis_service.run_competitor_analysis',
|
||||||
|
lambda tenant_id, comp, ctx: captured_contexts.append(ctx) or {'path': 'refresh', 'result': {}}
|
||||||
|
)
|
||||||
|
monkeypatch.setattr('services.alert_service.send_gotify', lambda **kw: None)
|
||||||
|
|
||||||
|
jobs.run_research_job(run['id'], {'tenant_id': tenant['id'], 'competitors': [{'id': competitor['id']}]})
|
||||||
|
|
||||||
|
assert captured_contexts[0]['own_trip'] is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_run_research_job_own_trip_failure_does_not_fail_the_batch(monkeypatch):
|
||||||
|
# A side-effect failure (site blocked, extraction error) must never
|
||||||
|
# fail the whole batch — CLAUDE.md §18 resilience rule.
|
||||||
|
tenant, competitor, run = _setup()
|
||||||
|
captured_contexts = []
|
||||||
|
monkeypatch.setattr(
|
||||||
|
'services.analysis_service.run_competitor_analysis',
|
||||||
|
lambda tenant_id, comp, ctx: captured_contexts.append(ctx) or {'path': 'refresh', 'result': {}}
|
||||||
|
)
|
||||||
|
monkeypatch.setattr('services.alert_service.send_gotify', lambda **kw: None)
|
||||||
|
|
||||||
|
def _raise(tenant_id, website_url, ctx):
|
||||||
|
raise RuntimeError('boom')
|
||||||
|
monkeypatch.setattr('services.embedding_service.scrape_own_trip', _raise)
|
||||||
|
|
||||||
|
jobs.run_research_job(run['id'], {
|
||||||
|
'tenant_id': tenant['id'], 'competitors': [{'id': competitor['id']}],
|
||||||
|
'own_trip_url': 'https://wildhorizons.com/inca-trail',
|
||||||
|
})
|
||||||
|
|
||||||
|
assert captured_contexts[0]['own_trip'] is None
|
||||||
|
updated = scrape_repository.get_by_id(run['id'])
|
||||||
|
assert updated['status'] == 'complete'
|
||||||
|
|
||||||
|
|
||||||
def test_run_research_job_defaults_force_refresh_to_false(monkeypatch):
|
def test_run_research_job_defaults_force_refresh_to_false(monkeypatch):
|
||||||
tenant, competitor, run = _setup()
|
tenant, competitor, run = _setup()
|
||||||
captured_contexts = []
|
captured_contexts = []
|
||||||
|
|
|
||||||
|
|
@ -77,6 +77,46 @@ def test_find_competitor_trip_url_handles_firecrawl_failure_gracefully(monkeypat
|
||||||
assert result == {'url': None, 'confidence': 0, 'found': False}
|
assert result == {'url': None, 'confidence': 0, 'found': False}
|
||||||
|
|
||||||
|
|
||||||
|
def test_find_competitor_trip_url_uses_generous_map_timeout(monkeypatch):
|
||||||
|
# Regression test — a large real production site's sitemap.xml fetch
|
||||||
|
# can take well over 15s (confirmed directly: ~21.6s for a real site),
|
||||||
|
# which a too-tight timeout silently turns into a false "no match
|
||||||
|
# found" instead of a real answer. Guards against re-tightening this.
|
||||||
|
captured = {}
|
||||||
|
|
||||||
|
class _FakeResponse:
|
||||||
|
def raise_for_status(self):
|
||||||
|
pass
|
||||||
|
|
||||||
|
def json(self):
|
||||||
|
return {'links': []}
|
||||||
|
|
||||||
|
def _fake_post(url, json, timeout):
|
||||||
|
captured['timeout'] = timeout
|
||||||
|
return _FakeResponse()
|
||||||
|
|
||||||
|
monkeypatch.setattr(trip_finder_service.requests, 'post', _fake_post)
|
||||||
|
trip_finder_service.find_competitor_trip_url('https://gadventures.com', 'Peru', 15, 'Classic')
|
||||||
|
|
||||||
|
assert captured['timeout'] == trip_finder_service.FIRECRAWL_MAP_TIMEOUT_SECONDS
|
||||||
|
assert captured['timeout'] >= 30
|
||||||
|
|
||||||
|
|
||||||
|
def test_find_own_trip_url_delegates_to_find_competitor_trip_url(monkeypatch):
|
||||||
|
calls = []
|
||||||
|
|
||||||
|
def _fake_find(root_url, destination, duration, travel_style):
|
||||||
|
calls.append((root_url, destination, duration, travel_style))
|
||||||
|
return {'url': f'{root_url}/peru-classic-15-days', 'confidence': 0.9, 'found': True}
|
||||||
|
|
||||||
|
monkeypatch.setattr(trip_finder_service, 'find_competitor_trip_url', _fake_find)
|
||||||
|
|
||||||
|
result = trip_finder_service.find_own_trip_url('https://gadventures.com', 'Peru', 15, 'Classic')
|
||||||
|
|
||||||
|
assert calls == [('https://gadventures.com', 'Peru', 15, 'Classic')]
|
||||||
|
assert result == {'url': 'https://gadventures.com/peru-classic-15-days', 'confidence': 0.9, 'found': True}
|
||||||
|
|
||||||
|
|
||||||
def test_find_all_competitor_urls_runs_concurrently_and_preserves_competitor_ids(monkeypatch):
|
def test_find_all_competitor_urls_runs_concurrently_and_preserves_competitor_ids(monkeypatch):
|
||||||
def _fake_find(root_url, destination, duration, travel_style):
|
def _fake_find(root_url, destination, duration, travel_style):
|
||||||
return {'url': f'{root_url}/match', 'confidence': 0.8, 'found': True}
|
return {'url': f'{root_url}/match', 'confidence': 0.8, 'found': True}
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,139 @@
|
||||||
|
from services import watchlist_service
|
||||||
|
from repositories.tenant_repository import tenant_repository
|
||||||
|
from repositories.seat_repository import seat_repository
|
||||||
|
from repositories.competitor_repository import competitor_repository
|
||||||
|
from repositories.watchlist_repository import watchlist_repository
|
||||||
|
|
||||||
|
|
||||||
|
def _make_tenant_and_seat(tier='analyse'):
|
||||||
|
tenant = tenant_repository.create({
|
||||||
|
'name': 'Wild Horizons Travel', 'domain': 'wildhorizons.com', 'tier': tier,
|
||||||
|
'status': 'active', 'max_seats': 5,
|
||||||
|
})
|
||||||
|
seat = seat_repository.create({'tenant_id': tenant['id'], 'email': 'pm@wildhorizons.com', 'active': True})
|
||||||
|
return tenant, seat
|
||||||
|
|
||||||
|
|
||||||
|
def _make_competitor(tenant_id):
|
||||||
|
return competitor_repository.create({
|
||||||
|
'tenant_id': tenant_id, 'name': 'Intrepid Travel', 'website': 'https://intrepidtravel.com',
|
||||||
|
'catalogue_url': 'https://intrepidtravel.com/adventures', 'primary_market': 'GB', 'active': True,
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
|
def test_add_to_watchlist_succeeds_under_cap():
|
||||||
|
tenant, seat = _make_tenant_and_seat()
|
||||||
|
competitor = _make_competitor(tenant['id'])
|
||||||
|
|
||||||
|
result = watchlist_service.add_to_watchlist(
|
||||||
|
tenant, seat, competitor['id'], 'https://intrepidtravel.com/peru-classic-15-days', 'Peru Classic'
|
||||||
|
)
|
||||||
|
|
||||||
|
assert result['success'] is True
|
||||||
|
assert result['item']['competitor_id'] == competitor['id']
|
||||||
|
assert result['item']['url'] == 'https://intrepidtravel.com/peru-classic-15-days'
|
||||||
|
assert watchlist_repository.count_for_seat(seat['id']) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_add_to_watchlist_blocked_at_analyse_cap_of_one():
|
||||||
|
tenant, seat = _make_tenant_and_seat(tier='analyse')
|
||||||
|
competitor = _make_competitor(tenant['id'])
|
||||||
|
watchlist_service.add_to_watchlist(tenant, seat, competitor['id'], 'https://intrepidtravel.com/peru', 'Peru')
|
||||||
|
|
||||||
|
result = watchlist_service.add_to_watchlist(
|
||||||
|
tenant, seat, competitor['id'], 'https://intrepidtravel.com/vietnam', 'Vietnam'
|
||||||
|
)
|
||||||
|
|
||||||
|
assert result['success'] is False
|
||||||
|
assert '1/1' in result['reason']
|
||||||
|
assert watchlist_repository.count_for_seat(seat['id']) == 1 # unchanged — never silently overwrites
|
||||||
|
|
||||||
|
|
||||||
|
def test_add_to_watchlist_respects_higher_discover_cap():
|
||||||
|
tenant, seat = _make_tenant_and_seat(tier='discover')
|
||||||
|
competitor = _make_competitor(tenant['id'])
|
||||||
|
|
||||||
|
first = watchlist_service.add_to_watchlist(tenant, seat, competitor['id'], 'https://intrepidtravel.com/peru', 'Peru')
|
||||||
|
second = watchlist_service.add_to_watchlist(tenant, seat, competitor['id'], 'https://intrepidtravel.com/vietnam', 'Vietnam')
|
||||||
|
third = watchlist_service.add_to_watchlist(tenant, seat, competitor['id'], 'https://intrepidtravel.com/kenya', 'Kenya')
|
||||||
|
|
||||||
|
assert first['success'] is True
|
||||||
|
assert second['success'] is True
|
||||||
|
assert third['success'] is False # Discover cap is 2/seat
|
||||||
|
|
||||||
|
|
||||||
|
def test_add_to_watchlist_defaults_to_cap_of_one_for_unknown_tier():
|
||||||
|
tenant, seat = _make_tenant_and_seat(tier='unknown-future-tier')
|
||||||
|
competitor = _make_competitor(tenant['id'])
|
||||||
|
|
||||||
|
first = watchlist_service.add_to_watchlist(tenant, seat, competitor['id'], 'https://intrepidtravel.com/peru', 'Peru')
|
||||||
|
second = watchlist_service.add_to_watchlist(tenant, seat, competitor['id'], 'https://intrepidtravel.com/vietnam', 'Vietnam')
|
||||||
|
|
||||||
|
assert first['success'] is True
|
||||||
|
assert second['success'] is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_two_seats_each_get_their_own_independent_slot():
|
||||||
|
tenant, seat_a = _make_tenant_and_seat()
|
||||||
|
seat_b = seat_repository.create({'tenant_id': tenant['id'], 'email': 'other@wildhorizons.com', 'active': True})
|
||||||
|
competitor = _make_competitor(tenant['id'])
|
||||||
|
|
||||||
|
result_a = watchlist_service.add_to_watchlist(tenant, seat_a, competitor['id'], 'https://intrepidtravel.com/peru', 'Peru')
|
||||||
|
result_b = watchlist_service.add_to_watchlist(tenant, seat_b, competitor['id'], 'https://intrepidtravel.com/vietnam', 'Vietnam')
|
||||||
|
|
||||||
|
assert result_a['success'] is True
|
||||||
|
assert result_b['success'] is True # seat_a being at cap doesn't block seat_b
|
||||||
|
|
||||||
|
|
||||||
|
def test_remove_from_watchlist_frees_the_slot():
|
||||||
|
tenant, seat = _make_tenant_and_seat()
|
||||||
|
competitor = _make_competitor(tenant['id'])
|
||||||
|
added = watchlist_service.add_to_watchlist(tenant, seat, competitor['id'], 'https://intrepidtravel.com/peru', 'Peru')
|
||||||
|
|
||||||
|
removed = watchlist_service.remove_from_watchlist(added['item']['id'])
|
||||||
|
|
||||||
|
assert removed is True
|
||||||
|
assert watchlist_repository.count_for_seat(seat['id']) == 0
|
||||||
|
second = watchlist_service.add_to_watchlist(tenant, seat, competitor['id'], 'https://intrepidtravel.com/vietnam', 'Vietnam')
|
||||||
|
assert second['success'] is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_remove_from_watchlist_returns_false_for_unknown_item():
|
||||||
|
assert watchlist_service.remove_from_watchlist('does-not-exist') is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_list_watchlist_for_tenant_enriches_with_seat_email():
|
||||||
|
tenant, seat = _make_tenant_and_seat()
|
||||||
|
competitor = _make_competitor(tenant['id'])
|
||||||
|
watchlist_service.add_to_watchlist(tenant, seat, competitor['id'], 'https://intrepidtravel.com/peru', 'Peru Classic')
|
||||||
|
|
||||||
|
items = watchlist_service.list_watchlist_for_tenant(tenant['id'])
|
||||||
|
|
||||||
|
assert len(items) == 1
|
||||||
|
assert items[0]['watched_by_email'] == 'pm@wildhorizons.com'
|
||||||
|
assert items[0]['trip_name'] == 'Peru Classic'
|
||||||
|
|
||||||
|
|
||||||
|
def test_list_watchlist_for_tenant_shows_items_from_every_seat():
|
||||||
|
tenant, seat_a = _make_tenant_and_seat()
|
||||||
|
seat_b = seat_repository.create({'tenant_id': tenant['id'], 'email': 'other@wildhorizons.com', 'active': True})
|
||||||
|
competitor = _make_competitor(tenant['id'])
|
||||||
|
watchlist_service.add_to_watchlist(tenant, seat_a, competitor['id'], 'https://intrepidtravel.com/peru', 'Peru')
|
||||||
|
watchlist_service.add_to_watchlist(tenant, seat_b, competitor['id'], 'https://intrepidtravel.com/vietnam', 'Vietnam')
|
||||||
|
|
||||||
|
items = watchlist_service.list_watchlist_for_tenant(tenant['id'])
|
||||||
|
|
||||||
|
emails = {item['watched_by_email'] for item in items}
|
||||||
|
assert emails == {'pm@wildhorizons.com', 'other@wildhorizons.com'}
|
||||||
|
|
||||||
|
|
||||||
|
def test_list_watchlist_for_tenant_handles_removed_seat_gracefully():
|
||||||
|
tenant, seat = _make_tenant_and_seat()
|
||||||
|
competitor = _make_competitor(tenant['id'])
|
||||||
|
watchlist_service.add_to_watchlist(tenant, seat, competitor['id'], 'https://intrepidtravel.com/peru', 'Peru')
|
||||||
|
seat_repository.delete(seat['id']) # seat later removed from the team
|
||||||
|
|
||||||
|
items = watchlist_service.list_watchlist_for_tenant(tenant['id'])
|
||||||
|
|
||||||
|
assert len(items) == 1
|
||||||
|
assert items[0]['watched_by_email'] is None
|
||||||
|
|
@ -113,6 +113,53 @@ def test_research_find_urls_requires_fields(client, api_headers):
|
||||||
assert response.json['code'] == 'missing_field'
|
assert response.json['code'] == 'missing_field'
|
||||||
|
|
||||||
|
|
||||||
|
def test_research_find_urls_omits_own_trip_without_tenant_id(client, api_headers, monkeypatch):
|
||||||
|
monkeypatch.setattr(
|
||||||
|
'api.trip_finder_service.find_all_competitor_urls',
|
||||||
|
lambda competitors, destination, duration, travel_style: [],
|
||||||
|
)
|
||||||
|
response = client.post('/research/find-urls', json={
|
||||||
|
'destination': 'Peru', 'duration': 15, 'travel_style': 'Classic', 'competitor_ids': [],
|
||||||
|
}, headers=api_headers)
|
||||||
|
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert response.json['own_trip'] is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_research_find_urls_omits_own_trip_when_tenant_has_no_website(client, api_headers, monkeypatch):
|
||||||
|
tenant = _make_tenant()
|
||||||
|
monkeypatch.setattr(
|
||||||
|
'api.trip_finder_service.find_all_competitor_urls',
|
||||||
|
lambda competitors, destination, duration, travel_style: [],
|
||||||
|
)
|
||||||
|
response = client.post('/research/find-urls', json={
|
||||||
|
'tenant_id': tenant['id'], 'destination': 'Peru', 'duration': 15,
|
||||||
|
'travel_style': 'Classic', 'competitor_ids': [],
|
||||||
|
}, headers=api_headers)
|
||||||
|
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert response.json['own_trip'] is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_research_find_urls_includes_own_trip_when_tenant_has_website(client, api_headers, monkeypatch):
|
||||||
|
tenant = _make_tenant(website='https://wildhorizons.com')
|
||||||
|
monkeypatch.setattr(
|
||||||
|
'api.trip_finder_service.find_all_competitor_urls',
|
||||||
|
lambda competitors, destination, duration, travel_style: [],
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
'api.trip_finder_service.find_own_trip_url',
|
||||||
|
lambda website, destination, duration, travel_style: {'url': f'{website}/peru-classic', 'confidence': 0.9, 'found': True},
|
||||||
|
)
|
||||||
|
response = client.post('/research/find-urls', json={
|
||||||
|
'tenant_id': tenant['id'], 'destination': 'Peru', 'duration': 15,
|
||||||
|
'travel_style': 'Classic', 'competitor_ids': [],
|
||||||
|
}, headers=api_headers)
|
||||||
|
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert response.json['own_trip'] == {'url': 'https://wildhorizons.com/peru-classic', 'confidence': 0.9, 'found': True}
|
||||||
|
|
||||||
|
|
||||||
def test_research_enqueues_job_without_touching_real_redis(client, api_headers, monkeypatch):
|
def test_research_enqueues_job_without_touching_real_redis(client, api_headers, monkeypatch):
|
||||||
import jobs
|
import jobs
|
||||||
tenant = _make_tenant()
|
tenant = _make_tenant()
|
||||||
|
|
|
||||||
|
|
@ -124,6 +124,35 @@ def test_update_timezone_never_overwrites_existing(client, api_headers, monkeypa
|
||||||
assert response.json['timezone'] == 'America/Toronto' # unchanged — first-write-wins
|
assert response.json['timezone'] == 'America/Toronto' # unchanged — first-write-wins
|
||||||
|
|
||||||
|
|
||||||
|
def test_update_website_requires_session(client, api_headers):
|
||||||
|
response = client.patch('/account/website', json={'website': 'https://client.com'}, headers=api_headers)
|
||||||
|
assert response.status_code == 401
|
||||||
|
|
||||||
|
|
||||||
|
def test_update_website_requires_field(client, api_headers, monkeypatch):
|
||||||
|
tenant = _make_tenant()
|
||||||
|
monkeypatch.setattr('api.auth_service.get_tenant_id_from_request', lambda req: tenant['id'])
|
||||||
|
|
||||||
|
response = client.patch('/account/website', json={}, headers=api_headers)
|
||||||
|
|
||||||
|
assert response.status_code == 400
|
||||||
|
assert response.json['code'] == 'missing_field'
|
||||||
|
|
||||||
|
|
||||||
|
def test_update_website_is_freely_editable_unlike_timezone(client, api_headers, monkeypatch):
|
||||||
|
tenant = tenant_repository.create({
|
||||||
|
'name': 'Wild Horizons', 'domain': 'wildhorizons.com', 'tier': 'analyse',
|
||||||
|
'status': 'active', 'max_seats': 5, 'website': 'https://old.wildhorizons.com',
|
||||||
|
})
|
||||||
|
monkeypatch.setattr('api.auth_service.get_tenant_id_from_request', lambda req: tenant['id'])
|
||||||
|
|
||||||
|
response = client.patch('/account/website', json={'website': 'https://wildhorizons.com'}, headers=api_headers)
|
||||||
|
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert response.json['website'] == 'https://wildhorizons.com'
|
||||||
|
assert tenant_repository.get_by_id(tenant['id'])['website'] == 'https://wildhorizons.com'
|
||||||
|
|
||||||
|
|
||||||
def test_update_competitor(client, api_headers):
|
def test_update_competitor(client, api_headers):
|
||||||
tenant = _make_tenant()
|
tenant = _make_tenant()
|
||||||
competitor = competitor_repository.create({
|
competitor = competitor_repository.create({
|
||||||
|
|
@ -610,17 +639,101 @@ def test_research_status_unknown_job_404(client, api_headers):
|
||||||
assert response.status_code == 404
|
assert response.status_code == 404
|
||||||
|
|
||||||
|
|
||||||
def test_research_status_known_job(client, api_headers):
|
def test_research_status_known_job(client, api_headers, monkeypatch):
|
||||||
|
import jobs
|
||||||
tenant = _make_tenant()
|
tenant = _make_tenant()
|
||||||
run = scrape_repository.create({
|
run = scrape_repository.create({
|
||||||
'tenant_id': tenant['id'], 'triggered_by': 'manual', 'status': 'running',
|
'tenant_id': tenant['id'], 'triggered_by': 'manual', 'status': 'running',
|
||||||
'competitors_total': 2, 'competitors_done': 1, 'results': {},
|
'competitors_total': 2, 'competitors_done': 1, 'results': {},
|
||||||
})
|
})
|
||||||
|
# Genuinely still executing — the reconciliation check must leave it alone.
|
||||||
|
monkeypatch.setattr(jobs.Job, 'fetch', lambda job_id, connection=None: types.SimpleNamespace(get_status=lambda: 'started'))
|
||||||
|
|
||||||
response = client.get(f'/research/status/{run["id"]}', headers=api_headers)
|
response = client.get(f'/research/status/{run["id"]}', headers=api_headers)
|
||||||
assert response.status_code == 200
|
assert response.status_code == 200
|
||||||
|
assert response.json['status'] == 'running'
|
||||||
assert response.json['progress'] == {'total': 2, 'done': 1}
|
assert response.json['progress'] == {'total': 2, 'done': 1}
|
||||||
|
|
||||||
|
|
||||||
|
def test_research_status_reconciles_orphaned_job_to_cancelled(client, api_headers, monkeypatch):
|
||||||
|
# Regression test — a PM clicks Cancel, then the worker process
|
||||||
|
# handling the job is killed outright (e.g. a container rebuild)
|
||||||
|
# before run_research_job() ever reaches its cooperative cancel
|
||||||
|
# checkpoint. Without reconciliation, scrape_runs stays 'running'
|
||||||
|
# forever and the dashboard shows "Stopping…" indefinitely.
|
||||||
|
import jobs
|
||||||
|
tenant = _make_tenant()
|
||||||
|
run = scrape_repository.create({
|
||||||
|
'tenant_id': tenant['id'], 'triggered_by': 'manual', 'status': 'running',
|
||||||
|
'cancel_requested': True, 'competitors_total': 5, 'competitors_done': 3, 'results': {},
|
||||||
|
})
|
||||||
|
monkeypatch.setattr(jobs.Job, 'fetch', lambda job_id, connection=None: types.SimpleNamespace(get_status=lambda: 'failed'))
|
||||||
|
|
||||||
|
response = client.get(f'/research/status/{run["id"]}', headers=api_headers)
|
||||||
|
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert response.json['status'] == 'cancelled'
|
||||||
|
updated = scrape_repository.get_by_id(run['id'])
|
||||||
|
assert updated['status'] == 'cancelled'
|
||||||
|
assert updated['completed_at'] is not None
|
||||||
|
|
||||||
|
|
||||||
|
def test_research_status_reconciles_orphaned_job_to_failed_without_cancel_request(client, api_headers, monkeypatch):
|
||||||
|
# Same orphaned-worker scenario, but the PM never clicked cancel —
|
||||||
|
# an uncaught exception or a killed worker should surface as a real
|
||||||
|
# failure, not silently stay 'running' forever either.
|
||||||
|
import jobs
|
||||||
|
tenant = _make_tenant()
|
||||||
|
run = scrape_repository.create({
|
||||||
|
'tenant_id': tenant['id'], 'triggered_by': 'manual', 'status': 'running',
|
||||||
|
'competitors_total': 5, 'competitors_done': 2, 'results': {},
|
||||||
|
})
|
||||||
|
monkeypatch.setattr(jobs.Job, 'fetch', lambda job_id, connection=None: types.SimpleNamespace(get_status=lambda: 'failed'))
|
||||||
|
|
||||||
|
response = client.get(f'/research/status/{run["id"]}', headers=api_headers)
|
||||||
|
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert response.json['status'] == 'failed'
|
||||||
|
assert response.json['error']
|
||||||
|
|
||||||
|
|
||||||
|
def test_research_status_reconciles_when_rq_has_no_record_at_all(client, api_headers, monkeypatch):
|
||||||
|
import jobs
|
||||||
|
tenant = _make_tenant()
|
||||||
|
run = scrape_repository.create({
|
||||||
|
'tenant_id': tenant['id'], 'triggered_by': 'manual', 'status': 'running',
|
||||||
|
'cancel_requested': True, 'competitors_total': 3, 'competitors_done': 1, 'results': {},
|
||||||
|
})
|
||||||
|
|
||||||
|
def _raise_no_such_job(job_id, connection=None):
|
||||||
|
raise jobs.NoSuchJobError()
|
||||||
|
monkeypatch.setattr(jobs.Job, 'fetch', _raise_no_such_job)
|
||||||
|
|
||||||
|
response = client.get(f'/research/status/{run["id"]}', headers=api_headers)
|
||||||
|
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert response.json['status'] == 'cancelled'
|
||||||
|
|
||||||
|
|
||||||
|
def test_research_status_does_not_touch_terminal_jobs(client, api_headers, monkeypatch):
|
||||||
|
"""A job already complete/failed/cancelled must never hit the RQ reconciliation path at all."""
|
||||||
|
import jobs
|
||||||
|
tenant = _make_tenant()
|
||||||
|
run = scrape_repository.create({
|
||||||
|
'tenant_id': tenant['id'], 'triggered_by': 'manual', 'status': 'complete',
|
||||||
|
'competitors_total': 1, 'competitors_done': 1, 'results': {'success': []},
|
||||||
|
})
|
||||||
|
|
||||||
|
def _fail_if_called(job_id, connection=None):
|
||||||
|
raise AssertionError('must not check RQ for an already-terminal job')
|
||||||
|
monkeypatch.setattr(jobs.Job, 'fetch', _fail_if_called)
|
||||||
|
|
||||||
|
response = client.get(f'/research/status/{run["id"]}', headers=api_headers)
|
||||||
|
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert response.json['status'] == 'complete'
|
||||||
|
|
||||||
|
|
||||||
def test_research_history_detail_unknown_404(client, api_headers):
|
def test_research_history_detail_unknown_404(client, api_headers):
|
||||||
response = client.get('/research/history/nope', headers=api_headers)
|
response = client.get('/research/history/nope', headers=api_headers)
|
||||||
assert response.status_code == 404
|
assert response.status_code == 404
|
||||||
|
|
@ -841,6 +954,165 @@ def test_mark_alert_read_success(client, api_headers, monkeypatch):
|
||||||
assert response.json['read'] is True
|
assert response.json['read'] is True
|
||||||
|
|
||||||
|
|
||||||
|
# ── Trip watchlist ─────────────────────────────────────────────────
|
||||||
|
|
||||||
|
def _make_competitor(tenant_id, **overrides):
|
||||||
|
data = {
|
||||||
|
'tenant_id': tenant_id, 'name': 'Intrepid Travel', 'website': 'https://intrepidtravel.com',
|
||||||
|
'catalogue_url': 'https://intrepidtravel.com/adventures', 'primary_market': 'GB', 'active': True,
|
||||||
|
}
|
||||||
|
data.update(overrides)
|
||||||
|
return competitor_repository.create(data)
|
||||||
|
|
||||||
|
|
||||||
|
def test_get_watchlist_requires_session(client, api_headers):
|
||||||
|
response = client.get('/watchlist', headers=api_headers)
|
||||||
|
assert response.status_code == 401
|
||||||
|
|
||||||
|
|
||||||
|
def test_get_watchlist_returns_tenant_items(client, api_headers, monkeypatch):
|
||||||
|
from services import watchlist_service
|
||||||
|
tenant = _make_tenant()
|
||||||
|
seat = seat_repository.create({'tenant_id': tenant['id'], 'email': 'pm@gadventures.com', 'active': True})
|
||||||
|
competitor = _make_competitor(tenant['id'])
|
||||||
|
watchlist_service.add_to_watchlist(tenant, seat, competitor['id'], 'https://intrepidtravel.com/peru', 'Peru Classic')
|
||||||
|
monkeypatch.setattr('api.auth_service.get_tenant_id_from_request', lambda req: tenant['id'])
|
||||||
|
|
||||||
|
response = client.get('/watchlist', headers=api_headers)
|
||||||
|
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert len(response.json) == 1
|
||||||
|
assert response.json[0]['watched_by_email'] == 'pm@gadventures.com'
|
||||||
|
|
||||||
|
|
||||||
|
def test_add_watchlist_item_requires_session(client, api_headers):
|
||||||
|
response = client.post('/watchlist', json={'competitor_id': 'c1', 'url': 'x'}, headers=api_headers)
|
||||||
|
assert response.status_code == 401
|
||||||
|
|
||||||
|
|
||||||
|
def test_add_watchlist_item_requires_fields(client, api_headers, monkeypatch):
|
||||||
|
tenant = _make_tenant()
|
||||||
|
seat = seat_repository.create({'tenant_id': tenant['id'], 'email': 'pm@gadventures.com', 'active': True})
|
||||||
|
monkeypatch.setattr('api.auth_service.get_seat_from_request', lambda req: (tenant['id'], seat))
|
||||||
|
|
||||||
|
response = client.post('/watchlist', json={}, headers=api_headers)
|
||||||
|
|
||||||
|
assert response.status_code == 400
|
||||||
|
assert response.json['code'] == 'missing_field'
|
||||||
|
|
||||||
|
|
||||||
|
def test_add_watchlist_item_unknown_competitor_404(client, api_headers, monkeypatch):
|
||||||
|
tenant = _make_tenant()
|
||||||
|
seat = seat_repository.create({'tenant_id': tenant['id'], 'email': 'pm@gadventures.com', 'active': True})
|
||||||
|
monkeypatch.setattr('api.auth_service.get_seat_from_request', lambda req: (tenant['id'], seat))
|
||||||
|
|
||||||
|
response = client.post(
|
||||||
|
'/watchlist', json={'competitor_id': 'does-not-exist', 'url': 'https://x.com/y'}, headers=api_headers
|
||||||
|
)
|
||||||
|
|
||||||
|
assert response.status_code == 404
|
||||||
|
|
||||||
|
|
||||||
|
def test_add_watchlist_item_rejects_other_tenants_competitor(client, api_headers, monkeypatch):
|
||||||
|
tenant = _make_tenant()
|
||||||
|
other_tenant = _make_tenant(domain='otherco.com', email='pm@otherco.com')
|
||||||
|
competitor = _make_competitor(other_tenant['id'])
|
||||||
|
seat = seat_repository.create({'tenant_id': tenant['id'], 'email': 'pm@gadventures.com', 'active': True})
|
||||||
|
monkeypatch.setattr('api.auth_service.get_seat_from_request', lambda req: (tenant['id'], seat))
|
||||||
|
|
||||||
|
response = client.post(
|
||||||
|
'/watchlist', json={'competitor_id': competitor['id'], 'url': 'https://intrepidtravel.com/peru'},
|
||||||
|
headers=api_headers
|
||||||
|
)
|
||||||
|
|
||||||
|
assert response.status_code == 404
|
||||||
|
|
||||||
|
|
||||||
|
def test_add_watchlist_item_success(client, api_headers, monkeypatch):
|
||||||
|
tenant = _make_tenant()
|
||||||
|
seat = seat_repository.create({'tenant_id': tenant['id'], 'email': 'pm@gadventures.com', 'active': True})
|
||||||
|
competitor = _make_competitor(tenant['id'])
|
||||||
|
monkeypatch.setattr('api.auth_service.get_seat_from_request', lambda req: (tenant['id'], seat))
|
||||||
|
|
||||||
|
response = client.post(
|
||||||
|
'/watchlist',
|
||||||
|
json={'competitor_id': competitor['id'], 'url': 'https://intrepidtravel.com/peru', 'trip_name': 'Peru Classic'},
|
||||||
|
headers=api_headers
|
||||||
|
)
|
||||||
|
|
||||||
|
assert response.status_code == 201
|
||||||
|
assert response.json['url'] == 'https://intrepidtravel.com/peru'
|
||||||
|
assert response.json['seat_id'] == seat['id']
|
||||||
|
|
||||||
|
|
||||||
|
def test_add_watchlist_item_blocked_at_cap(client, api_headers, monkeypatch):
|
||||||
|
tenant = _make_tenant() # analyse tier — cap is 1/seat
|
||||||
|
seat = seat_repository.create({'tenant_id': tenant['id'], 'email': 'pm@gadventures.com', 'active': True})
|
||||||
|
competitor = _make_competitor(tenant['id'])
|
||||||
|
monkeypatch.setattr('api.auth_service.get_seat_from_request', lambda req: (tenant['id'], seat))
|
||||||
|
|
||||||
|
client.post(
|
||||||
|
'/watchlist', json={'competitor_id': competitor['id'], 'url': 'https://intrepidtravel.com/peru'},
|
||||||
|
headers=api_headers
|
||||||
|
)
|
||||||
|
response = client.post(
|
||||||
|
'/watchlist', json={'competitor_id': competitor['id'], 'url': 'https://intrepidtravel.com/vietnam'},
|
||||||
|
headers=api_headers
|
||||||
|
)
|
||||||
|
|
||||||
|
assert response.status_code == 400
|
||||||
|
assert response.json['code'] == 'watchlist_full'
|
||||||
|
|
||||||
|
|
||||||
|
def test_remove_watchlist_item_requires_session(client, api_headers):
|
||||||
|
response = client.delete('/watchlist/w1', headers=api_headers)
|
||||||
|
assert response.status_code == 401
|
||||||
|
|
||||||
|
|
||||||
|
def test_remove_watchlist_item_unknown_404(client, api_headers, monkeypatch):
|
||||||
|
tenant = _make_tenant()
|
||||||
|
monkeypatch.setattr('api.auth_service.get_tenant_id_from_request', lambda req: tenant['id'])
|
||||||
|
response = client.delete('/watchlist/does-not-exist', headers=api_headers)
|
||||||
|
assert response.status_code == 404
|
||||||
|
|
||||||
|
|
||||||
|
def test_remove_watchlist_item_rejects_other_tenants_item(client, api_headers, monkeypatch):
|
||||||
|
from services import watchlist_service
|
||||||
|
tenant = _make_tenant()
|
||||||
|
other_tenant = _make_tenant(domain='otherco.com', email='pm@otherco.com')
|
||||||
|
other_seat = seat_repository.create({'tenant_id': other_tenant['id'], 'email': 'pm@otherco.com', 'active': True})
|
||||||
|
other_competitor = _make_competitor(other_tenant['id'])
|
||||||
|
added = watchlist_service.add_to_watchlist(
|
||||||
|
other_tenant, other_seat, other_competitor['id'], 'https://intrepidtravel.com/peru', 'Peru'
|
||||||
|
)
|
||||||
|
monkeypatch.setattr('api.auth_service.get_tenant_id_from_request', lambda req: tenant['id'])
|
||||||
|
|
||||||
|
response = client.delete(f'/watchlist/{added["item"]["id"]}', headers=api_headers)
|
||||||
|
|
||||||
|
assert response.status_code == 404
|
||||||
|
|
||||||
|
|
||||||
|
def test_remove_watchlist_item_success_frees_the_slot(client, api_headers, monkeypatch):
|
||||||
|
from services import watchlist_service
|
||||||
|
tenant = _make_tenant()
|
||||||
|
seat = seat_repository.create({'tenant_id': tenant['id'], 'email': 'pm@gadventures.com', 'active': True})
|
||||||
|
competitor = _make_competitor(tenant['id'])
|
||||||
|
added = watchlist_service.add_to_watchlist(
|
||||||
|
tenant, seat, competitor['id'], 'https://intrepidtravel.com/peru', 'Peru'
|
||||||
|
)
|
||||||
|
monkeypatch.setattr('api.auth_service.get_tenant_id_from_request', lambda req: tenant['id'])
|
||||||
|
|
||||||
|
response = client.delete(f'/watchlist/{added["item"]["id"]}', headers=api_headers)
|
||||||
|
|
||||||
|
assert response.status_code == 200
|
||||||
|
monkeypatch.setattr('api.auth_service.get_seat_from_request', lambda req: (tenant['id'], seat))
|
||||||
|
retry = client.post(
|
||||||
|
'/watchlist', json={'competitor_id': competitor['id'], 'url': 'https://intrepidtravel.com/vietnam'},
|
||||||
|
headers=api_headers
|
||||||
|
)
|
||||||
|
assert retry.status_code == 201
|
||||||
|
|
||||||
|
|
||||||
def test_preview_prompt(client, api_headers, monkeypatch):
|
def test_preview_prompt(client, api_headers, monkeypatch):
|
||||||
_install_fake_module(
|
_install_fake_module(
|
||||||
monkeypatch, 'industries.adventure_travel.prompt',
|
monkeypatch, 'industries.adventure_travel.prompt',
|
||||||
|
|
|
||||||
|
|
@ -42,6 +42,52 @@ def test_build_prompt_falls_back_to_generic_placeholder_without_company_name():
|
||||||
assert 'G Adventures' not in result
|
assert 'G Adventures' not in result
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_prompt_enriches_comparison_with_real_own_trip_data_when_given():
|
||||||
|
page_data = {'text': 'content', 'tables': '', 'url': 'https://example.com'}
|
||||||
|
own_trip = {
|
||||||
|
'tripName': 'Inca Trail Classic', 'majorityPrice': 1999,
|
||||||
|
'duration': 15, 'activities': 'Trekking, Camping', 'serviceLevel': 'Mid-range',
|
||||||
|
}
|
||||||
|
result = prompt.build_prompt(
|
||||||
|
'Competitor', page_data, 'Inca Trail Classic', 'Peru', 15, 'Classic',
|
||||||
|
company_name='Wild Horizons Travel', own_trip=own_trip,
|
||||||
|
)
|
||||||
|
assert 'Inca Trail Classic' in result
|
||||||
|
assert '1999' in result
|
||||||
|
assert 'Trekking, Camping' in result
|
||||||
|
assert "Wild Horizons Travel's own Inca Trail Classic" in result
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_prompt_falls_back_to_generic_comments_wording_without_own_trip():
|
||||||
|
page_data = {'text': 'content', 'tables': '', 'url': 'https://example.com'}
|
||||||
|
result = prompt.build_prompt(
|
||||||
|
'Competitor', page_data, 'Inca Trail Classic', 'Peru', 15, 'Classic',
|
||||||
|
company_name='Wild Horizons Travel',
|
||||||
|
)
|
||||||
|
assert 'key differences from Wild Horizons Travel,' in result
|
||||||
|
assert 'own site' not in result
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_own_trip_prompt_frames_extraction_as_the_clients_own_page_not_a_competitor():
|
||||||
|
page_data = {'text': 'Peru Classic 15 days itinerary...', 'tables': '', 'url': 'https://client.com/peru'}
|
||||||
|
result = prompt.build_own_trip_prompt(
|
||||||
|
page_data, 'Inca Trail Classic', 'Peru', 15, 'Classic', 'Wild Horizons Travel',
|
||||||
|
)
|
||||||
|
assert "Wild Horizons Travel's own tour page" in result
|
||||||
|
assert 'not a competitor' in result
|
||||||
|
assert '"tripName"' in result
|
||||||
|
assert '"relevancy"' not in result
|
||||||
|
assert '"comments"' not in result
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_own_trip_prompt_includes_ngs_fields_when_flagged():
|
||||||
|
page_data = {'text': 'content', 'tables': '', 'url': 'https://client.com/peru'}
|
||||||
|
result = prompt.build_own_trip_prompt(
|
||||||
|
page_data, 'Product', 'Peru', 10, 'NGS', 'Wild Horizons Travel', is_ngs=True,
|
||||||
|
)
|
||||||
|
assert '"exclusiveAccess"' in result
|
||||||
|
|
||||||
|
|
||||||
def test_build_prompt_includes_ngs_fields_when_flagged():
|
def test_build_prompt_includes_ngs_fields_when_flagged():
|
||||||
page_data = {'text': 'content', 'tables': '', 'url': 'https://example.com'}
|
page_data = {'text': 'content', 'tables': '', 'url': 'https://example.com'}
|
||||||
result = prompt.build_prompt('Competitor', page_data, 'Product', 'Peru', 10, 'NGS', is_ngs=True)
|
result = prompt.build_prompt('Competitor', page_data, 'Product', 'Peru', 10, 'NGS', is_ngs=True)
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,55 @@
|
||||||
|
<script setup>
|
||||||
|
import { computed } from 'vue'
|
||||||
|
|
||||||
|
// Mirror of backend/services/watchlist_service.py's WATCHLIST_CAP_PER_SEAT
|
||||||
|
// — display only, the server is the actual source of truth/enforcement
|
||||||
|
// (see AnalysePage.vue's handleWatch() surfacing the real error message
|
||||||
|
// on a 400 watchlist_full response). Kept here purely so this card can
|
||||||
|
// show "X of Y slots used" without an extra round trip.
|
||||||
|
const WATCHLIST_CAP_PER_SEAT = { analyse: 1, discover: 2, intelligence: 3, enterprise: 3 }
|
||||||
|
|
||||||
|
const props = defineProps({
|
||||||
|
items: { type: Array, required: true }, // [{ id, trip_name, watched_by_email }]
|
||||||
|
currentSeatEmail: { type: String, default: '' },
|
||||||
|
tier: { type: String, default: 'analyse' }
|
||||||
|
})
|
||||||
|
defineEmits(['remove'])
|
||||||
|
|
||||||
|
const cap = computed(() => WATCHLIST_CAP_PER_SEAT[props.tier] ?? 1)
|
||||||
|
const mySlotsUsed = computed(() => props.items.filter((i) => i.watched_by_email === props.currentSeatEmail).length)
|
||||||
|
</script>
|
||||||
|
|
||||||
|
<template>
|
||||||
|
<div class="card">
|
||||||
|
<div class="ch">
|
||||||
|
<div>
|
||||||
|
<div class="ct">Your Watchlist</div>
|
||||||
|
<div class="cs">{{ mySlotsUsed }} of {{ cap }} of your slots used · {{ items.length }} watched tenant-wide</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div v-if="!items.length" style="padding:24px;text-align:center;color:var(--t3);font-size:12.5px;">
|
||||||
|
No trips watched yet — click "☆ Watch" on any analysis result to have it re-checked every night.
|
||||||
|
</div>
|
||||||
|
<div v-for="item in items" :key="item.id" class="watch-row">
|
||||||
|
<div style="flex:1;min-width:0;">
|
||||||
|
<div style="font-weight:600;font-size:13px;overflow:hidden;text-overflow:ellipsis;white-space:nowrap;">
|
||||||
|
{{ item.trip_name || 'Untitled trip' }}
|
||||||
|
</div>
|
||||||
|
<div style="font-size:11.5px;color:var(--t3);">
|
||||||
|
Watched by {{ item.watched_by_email || 'a former teammate' }}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<UButton size="xs" color="neutral" variant="ghost" @click="$emit('remove', item.id)">Remove</UButton>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</template>
|
||||||
|
|
||||||
|
<style scoped>
|
||||||
|
.watch-row {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 12px;
|
||||||
|
padding: 12px 20px;
|
||||||
|
border-top: 1px solid var(--border);
|
||||||
|
}
|
||||||
|
</style>
|
||||||
|
|
@ -4,14 +4,84 @@
|
||||||
// frontend/src/config/resultFields.js for the row order/labels, verified
|
// frontend/src/config/resultFields.js for the row order/labels, verified
|
||||||
// directly against the real completed run in "G Compete App V1.xlsx".
|
// directly against the real completed run in "G Compete App V1.xlsx".
|
||||||
import { computed } from 'vue'
|
import { computed } from 'vue'
|
||||||
import { getRowsForTabType, rawValue, formatForDisplay } from '../../config/resultFields'
|
import { getRowsForTabType, rawValue, formatForDisplay, formatLabel } from '../../config/resultFields'
|
||||||
|
|
||||||
const props = defineProps({
|
const props = defineProps({
|
||||||
results: { type: Array, required: true }, // [{ competitor_id, name, path, result, product, is_generic_page }]
|
results: { type: Array, required: true }, // [{ competitor_id, name, path, unchanged, result, product, is_generic_page }]
|
||||||
tabType: { type: String, default: 'Standard' }
|
tabType: { type: String, default: 'Standard' },
|
||||||
|
// The tenant's own extracted trip (client self-scrape feature) — null
|
||||||
|
// when no website was confirmed for this run. Shown as its own
|
||||||
|
// column before every competitor so it's visible that this data was
|
||||||
|
// actually fetched and grounding the analysis, not just typed intent.
|
||||||
|
ownTrip: { type: Object, default: null },
|
||||||
|
ownTripLabel: { type: String, default: 'Your trip' },
|
||||||
|
companyName: { type: String, default: '' },
|
||||||
|
// Tenant-wide watchlist items (every seat's) — used only to determine
|
||||||
|
// whether the CURRENT seat (currentSeatEmail) already watches a given
|
||||||
|
// competitor's trip, so the button reads "★ Watching" vs "☆ Watch".
|
||||||
|
watchlist: { type: Array, default: () => [] },
|
||||||
|
currentSeatEmail: { type: String, default: '' }
|
||||||
})
|
})
|
||||||
|
|
||||||
|
const emit = defineEmits(['watch', 'unwatch'])
|
||||||
|
|
||||||
|
// Not shown on the own-trip column — watching your own trip isn't
|
||||||
|
// meaningful, only competitor trips are monitored.
|
||||||
|
function watchedItemFor(item) {
|
||||||
|
const url = rawValue({ key: 'link' }, item)
|
||||||
|
if (!url) return null
|
||||||
|
return props.watchlist.find((w) =>
|
||||||
|
w.competitor_id === item.competitor_id && w.url === url && w.watched_by_email === props.currentSeatEmail
|
||||||
|
) || null
|
||||||
|
}
|
||||||
|
|
||||||
|
function toggleWatch(item) {
|
||||||
|
const existing = watchedItemFor(item)
|
||||||
|
if (existing) {
|
||||||
|
emit('unwatch', { itemId: existing.id })
|
||||||
|
return
|
||||||
|
}
|
||||||
|
emit('watch', {
|
||||||
|
competitorId: item.competitor_id,
|
||||||
|
url: rawValue({ key: 'link' }, item),
|
||||||
|
tripName: rawValue({ key: 'tripName' }, item),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
const rows = computed(() => getRowsForTabType(props.tabType))
|
const rows = computed(() => getRowsForTabType(props.tabType))
|
||||||
|
// Synthetic "item" shape so rawValue()/the cache pill work identically
|
||||||
|
// for the own-trip column and every competitor column.
|
||||||
|
const ownTripItem = computed(() => ({
|
||||||
|
name: props.ownTripLabel, result: props.ownTrip, path: props.ownTrip?.cached ? 'fast' : 'refresh',
|
||||||
|
}))
|
||||||
|
|
||||||
|
// Only the cached case gets an indicator — that's the noteworthy state
|
||||||
|
// worth flagging (§7's fast path served data instead of a live scrape);
|
||||||
|
// a fresh scrape is the default expectation and needs no badge.
|
||||||
|
function isFromCache(item) {
|
||||||
|
return item.path === 'fast' || !!item.unchanged
|
||||||
|
}
|
||||||
|
|
||||||
|
function escapeHtml(str) {
|
||||||
|
return str.replace(/&/g, '&').replace(/</g, '<').replace(/>/g, '>')
|
||||||
|
}
|
||||||
|
|
||||||
|
// Bolds the comparison-section labels build_prompt()'s own JSON schema
|
||||||
|
// instruction asks the LLM for — "key differences from {company}, "
|
||||||
|
// "strengths, weaknesses, pricing position" — wherever they appear in
|
||||||
|
// the comments row, so a PM can skim multiple competitors' analysis at
|
||||||
|
// a glance. Only the comments cell ever renders via v-html — text is
|
||||||
|
// HTML-escaped first, then only our own <strong> tags are reinserted
|
||||||
|
// around an exact keyword match, so this can't be used to inject
|
||||||
|
// arbitrary markup even though the underlying text originates from an
|
||||||
|
// LLM extraction.
|
||||||
|
function boldKeywords(value) {
|
||||||
|
if (value == null || value === '') return '—'
|
||||||
|
return escapeHtml(String(value)).replace(
|
||||||
|
/((?:key )?differences:|strengths?:|weakness(?:es)?:|pricing(?: position)?:)/gi,
|
||||||
|
'<strong>$1</strong>'
|
||||||
|
)
|
||||||
|
}
|
||||||
</script>
|
</script>
|
||||||
|
|
||||||
<template>
|
<template>
|
||||||
|
|
@ -22,21 +92,62 @@ const rows = computed(() => getRowsForTabType(props.tabType))
|
||||||
<thead>
|
<thead>
|
||||||
<tr>
|
<tr>
|
||||||
<th class="label-col">Field</th>
|
<th class="label-col">Field</th>
|
||||||
|
<th v-if="ownTrip" class="own-trip-col">
|
||||||
|
<div class="th-name">
|
||||||
|
🏠 {{ ownTripLabel }}
|
||||||
|
<span v-if="isFromCache(ownTripItem)" class="db-icon" title="Served from cached data — no live scrape this run">
|
||||||
|
<svg width="13" height="13" viewBox="0 0 24 24" fill="none" stroke="var(--blue-vivid)" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
|
||||||
|
<ellipse cx="12" cy="5" rx="9" ry="3" />
|
||||||
|
<path d="M21 12c0 1.66-4 3-9 3s-9-1.34-9-3" />
|
||||||
|
<path d="M3 5v14c0 1.66 4 3 9 3s9-1.34 9-3V5" />
|
||||||
|
</svg>
|
||||||
|
</span>
|
||||||
|
</div>
|
||||||
|
</th>
|
||||||
<th v-for="item in results" :key="item.competitor_id">
|
<th v-for="item in results" :key="item.competitor_id">
|
||||||
|
<div class="th-name">
|
||||||
{{ item.name }}
|
{{ item.name }}
|
||||||
|
<span v-if="isFromCache(item)" class="db-icon" title="Served from cached data — no live scrape this run">
|
||||||
|
<svg width="13" height="13" viewBox="0 0 24 24" fill="none" stroke="var(--blue-vivid)" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
|
||||||
|
<ellipse cx="12" cy="5" rx="9" ry="3" />
|
||||||
|
<path d="M21 12c0 1.66-4 3-9 3s-9-1.34-9-3" />
|
||||||
|
<path d="M3 5v14c0 1.66 4 3 9 3s9-1.34 9-3V5" />
|
||||||
|
</svg>
|
||||||
|
</span>
|
||||||
|
</div>
|
||||||
|
<div class="th-badges">
|
||||||
|
<button
|
||||||
|
type="button" class="watch-btn" :class="{ active: !!watchedItemFor(item) }"
|
||||||
|
:title="watchedItemFor(item) ? 'Remove from your watchlist' : 'Watch this trip — the nightly cron will re-check it for changes'"
|
||||||
|
@click="toggleWatch(item)"
|
||||||
|
>{{ watchedItemFor(item) ? '★ Watching' : '☆ Watch' }}</button>
|
||||||
<span
|
<span
|
||||||
v-if="item.is_generic_page"
|
v-if="item.is_generic_page"
|
||||||
class="badge bg-amber"
|
class="badge bg-amber"
|
||||||
title="Scraped page is the competitor's homepage, not a trip-specific page — treat this result with caution"
|
title="Scraped page is the competitor's homepage, not a trip-specific page — treat this result with caution"
|
||||||
>⚠ Generic page</span>
|
>⚠ Generic page</span>
|
||||||
|
</div>
|
||||||
</th>
|
</th>
|
||||||
</tr>
|
</tr>
|
||||||
</thead>
|
</thead>
|
||||||
<tbody>
|
<tbody>
|
||||||
<tr v-for="row in rows" :key="row.key">
|
<tr v-for="row in rows" :key="row.key">
|
||||||
<td class="label-col">{{ row.label }}</td>
|
<td class="label-col">{{ formatLabel(row.label, companyName) }}</td>
|
||||||
|
<td v-if="ownTrip" class="own-trip-col">
|
||||||
|
<a
|
||||||
|
v-if="row.key === 'link' && rawValue(row, ownTripItem)"
|
||||||
|
:href="rawValue(row, ownTripItem)" target="_blank" rel="noopener"
|
||||||
|
>{{ rawValue(row, ownTripItem) }}</a>
|
||||||
|
<span v-else-if="row.key === 'comments'" v-html="boldKeywords(rawValue(row, ownTripItem))"></span>
|
||||||
|
<template v-else>{{ formatForDisplay(rawValue(row, ownTripItem), row.format) }}</template>
|
||||||
|
</td>
|
||||||
<td v-for="item in results" :key="item.competitor_id">
|
<td v-for="item in results" :key="item.competitor_id">
|
||||||
{{ formatForDisplay(rawValue(row, item), row.format) }}
|
<a
|
||||||
|
v-if="row.key === 'link' && rawValue(row, item)"
|
||||||
|
:href="rawValue(row, item)" target="_blank" rel="noopener"
|
||||||
|
>{{ rawValue(row, item) }}</a>
|
||||||
|
<span v-else-if="row.key === 'comments'" v-html="boldKeywords(rawValue(row, item))"></span>
|
||||||
|
<template v-else>{{ formatForDisplay(rawValue(row, item), row.format) }}</template>
|
||||||
</td>
|
</td>
|
||||||
</tr>
|
</tr>
|
||||||
</tbody>
|
</tbody>
|
||||||
|
|
@ -61,4 +172,36 @@ const rows = computed(() => getRowsForTabType(props.tabType))
|
||||||
white-space: normal !important;
|
white-space: normal !important;
|
||||||
}
|
}
|
||||||
.results-grid thead th.label-col { z-index: 2; }
|
.results-grid thead th.label-col { z-index: 2; }
|
||||||
|
.own-trip-col { background: var(--surface-2); }
|
||||||
|
.results-grid a { color: var(--blue-vivid); text-decoration: none; }
|
||||||
|
.results-grid a:hover { text-decoration: underline; }
|
||||||
|
.results-grid .badge { font-weight: 700; }
|
||||||
|
/* Header cells wrap instead of clipping — a long competitor name plus
|
||||||
|
its cache-source badge previously overflowed the shared th/td
|
||||||
|
max-width + ellipsis rule, silently hiding the badge on any column
|
||||||
|
whose name didn't happen to fit within 260px. */
|
||||||
|
.results-grid th {
|
||||||
|
white-space: normal;
|
||||||
|
overflow: visible;
|
||||||
|
text-overflow: clip;
|
||||||
|
vertical-align: bottom;
|
||||||
|
padding-top: 10px;
|
||||||
|
padding-bottom: 10px;
|
||||||
|
}
|
||||||
|
.th-name { font-weight: 700; margin-bottom: 4px; display: flex; align-items: center; gap: 5px; }
|
||||||
|
.th-badges { display: flex; flex-wrap: wrap; gap: 4px; align-items: center; }
|
||||||
|
.db-icon { display: inline-flex; opacity: 0.85; }
|
||||||
|
.watch-btn {
|
||||||
|
font-size: 11px;
|
||||||
|
font-weight: 700;
|
||||||
|
padding: 2px 8px;
|
||||||
|
border-radius: 20px;
|
||||||
|
border: 1px solid var(--border-2);
|
||||||
|
background: var(--surface);
|
||||||
|
color: var(--t3);
|
||||||
|
cursor: pointer;
|
||||||
|
line-height: 17px;
|
||||||
|
}
|
||||||
|
.watch-btn:hover { border-color: var(--blue-vivid); color: var(--blue-vivid); }
|
||||||
|
.watch-btn.active { background: var(--blue-light); border-color: var(--blue-vivid); color: var(--blue-vivid); }
|
||||||
</style>
|
</style>
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
<script setup>
|
<script setup>
|
||||||
import { ref } from 'vue'
|
import { ref } from 'vue'
|
||||||
import { getRowsForTabType, rawValue } from '../../config/resultFields'
|
import { getRowsForTabType, rawValue, formatLabel } from '../../config/resultFields'
|
||||||
|
|
||||||
// Replaces PushToSheet.vue per the Sheet-initiated pull architecture
|
// Replaces PushToSheet.vue per the Sheet-initiated pull architecture
|
||||||
// (CLAUDE.md §15): a stateless Flask backend has no way to reach into
|
// (CLAUDE.md §15): a stateless Flask backend has no way to reach into
|
||||||
|
|
@ -17,7 +17,8 @@ import { getRowsForTabType, rawValue } from '../../config/resultFields'
|
||||||
// verified against the real completed run in "G Compete App V1.xlsx".
|
// verified against the real completed run in "G Compete App V1.xlsx".
|
||||||
const props = defineProps({
|
const props = defineProps({
|
||||||
results: { type: Array, required: true }, // [{ name, result, primary_market }]
|
results: { type: Array, required: true }, // [{ name, result, primary_market }]
|
||||||
tabType: { type: String, default: 'Standard' }
|
tabType: { type: String, default: 'Standard' },
|
||||||
|
companyName: { type: String, default: '' }
|
||||||
})
|
})
|
||||||
|
|
||||||
const copied = ref(false)
|
const copied = ref(false)
|
||||||
|
|
@ -30,7 +31,7 @@ function sanitize(value) {
|
||||||
async function copyRows() {
|
async function copyRows() {
|
||||||
const rows = getRowsForTabType(props.tabType)
|
const rows = getRowsForTabType(props.tabType)
|
||||||
const lines = rows.map((row) => [
|
const lines = rows.map((row) => [
|
||||||
row.label,
|
formatLabel(row.label, props.companyName),
|
||||||
...props.results.map((item) => sanitize(rawValue(row, item))),
|
...props.results.map((item) => sanitize(rawValue(row, item))),
|
||||||
].join('\t'))
|
].join('\t'))
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -3,6 +3,12 @@ import { ref, watch } from 'vue'
|
||||||
|
|
||||||
const props = defineProps({
|
const props = defineProps({
|
||||||
matches: { type: Array, required: true },
|
matches: { type: Array, required: true },
|
||||||
|
// The tenant's own matching trip page (client self-scrape feature) —
|
||||||
|
// null when the tenant has no website configured, or find-urls was
|
||||||
|
// called without a tenant_id. Independent of `matches`: it never
|
||||||
|
// blocks "Confirm & Run" and isn't folded into the competitor-keyed
|
||||||
|
// editableUrls/skipped state below.
|
||||||
|
ownTrip: { type: Object, default: null },
|
||||||
submitting: { type: Boolean, default: false }
|
submitting: { type: Boolean, default: false }
|
||||||
})
|
})
|
||||||
const emit = defineEmits(['confirm', 'back'])
|
const emit = defineEmits(['confirm', 'back'])
|
||||||
|
|
@ -14,10 +20,19 @@ const editingId = ref(null)
|
||||||
// skipping excludes them from the batch entirely instead.
|
// skipping excludes them from the batch entirely instead.
|
||||||
const skipped = ref(new Set())
|
const skipped = ref(new Set())
|
||||||
|
|
||||||
|
const ownTripUrl = ref('')
|
||||||
|
const ownTripEditing = ref(false)
|
||||||
|
const ownTripSkipped = ref(false)
|
||||||
|
|
||||||
watch(() => props.matches, (matches) => {
|
watch(() => props.matches, (matches) => {
|
||||||
editableUrls.value = Object.fromEntries(matches.map((m) => [m.competitor_id, m.url || '']))
|
editableUrls.value = Object.fromEntries(matches.map((m) => [m.competitor_id, m.url || '']))
|
||||||
}, { immediate: true })
|
}, { immediate: true })
|
||||||
|
|
||||||
|
watch(() => props.ownTrip, (ownTrip) => {
|
||||||
|
ownTripUrl.value = ownTrip?.url || ''
|
||||||
|
ownTripSkipped.value = false
|
||||||
|
}, { immediate: true })
|
||||||
|
|
||||||
function toggleSkip(id) {
|
function toggleSkip(id) {
|
||||||
if (skipped.value.has(id)) skipped.value.delete(id)
|
if (skipped.value.has(id)) skipped.value.delete(id)
|
||||||
else skipped.value.add(id)
|
else skipped.value.add(id)
|
||||||
|
|
@ -32,7 +47,8 @@ function confirm() {
|
||||||
name: m.competitor_name,
|
name: m.competitor_name,
|
||||||
url: editableUrls.value[m.competitor_id]
|
url: editableUrls.value[m.competitor_id]
|
||||||
}))
|
}))
|
||||||
emit('confirm', competitors)
|
const ownTripUrlToSend = ownTripSkipped.value ? null : (ownTripUrl.value.trim() || null)
|
||||||
|
emit('confirm', { competitors, ownTripUrl: ownTripUrlToSend })
|
||||||
}
|
}
|
||||||
|
|
||||||
const allResolved = () => props.matches.every((m) =>
|
const allResolved = () => props.matches.every((m) =>
|
||||||
|
|
@ -47,6 +63,43 @@ const anyIncluded = () => props.matches.some((m) => !skipped.value.has(m.competi
|
||||||
<div class="ct">Confirm matched pages</div>
|
<div class="ct">Confirm matched pages</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
<div v-if="ownTrip" class="pr" style="align-items:center;background:var(--surface-2);">
|
||||||
|
<div class="pr-inner" style="gap:16px;">
|
||||||
|
<div class="pr-label" style="width:140px;">🏠 Your trip page</div>
|
||||||
|
<template v-if="ownTripEditing">
|
||||||
|
<UInput v-model="ownTripUrl" class="flex-1" placeholder="https://…" />
|
||||||
|
<UButton size="xs" color="primary" @click="ownTripEditing = false">Done</UButton>
|
||||||
|
</template>
|
||||||
|
<template v-else-if="ownTripUrl && !ownTripSkipped">
|
||||||
|
<div style="flex:1;font-size:12.5px;color:var(--t3);overflow:hidden;text-overflow:ellipsis;white-space:nowrap;">
|
||||||
|
{{ ownTripUrl }}
|
||||||
|
</div>
|
||||||
|
<span v-if="ownTrip.found" class="badge bg-green">✓ {{ Math.round((ownTrip.confidence || 0) * 100) }}%</span>
|
||||||
|
<UButton size="xs" color="neutral" variant="ghost" @click="ownTripEditing = true">↗ Swap</UButton>
|
||||||
|
</template>
|
||||||
|
<template v-else>
|
||||||
|
<div v-if="!ownTripSkipped" style="flex:1;font-size:12.5px;color:var(--red);">No match found</div>
|
||||||
|
<div v-else style="flex:1;font-size:12.5px;color:var(--t3);">Skipped — comparisons will use typed details only</div>
|
||||||
|
<UButton
|
||||||
|
v-if="!ownTripSkipped"
|
||||||
|
size="xs" color="primary" variant="soft"
|
||||||
|
@click="ownTripEditing = true"
|
||||||
|
>Enter URL manually</UButton>
|
||||||
|
<UButton
|
||||||
|
size="xs" color="neutral" :variant="ownTripSkipped ? 'soft' : 'ghost'"
|
||||||
|
@click="ownTripSkipped = !ownTripSkipped"
|
||||||
|
>{{ ownTripSkipped ? '↺ Undo skip' : 'Skip' }}</UButton>
|
||||||
|
</template>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div v-else class="pr" style="align-items:center;">
|
||||||
|
<div class="pr-inner" style="gap:16px;">
|
||||||
|
<div style="flex:1;font-size:12px;color:var(--t3);">
|
||||||
|
Add your company website in Account settings to ground this analysis in your real trip page.
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
<div v-for="match in matches" :key="match.competitor_id" class="pr" style="align-items:center;">
|
<div v-for="match in matches" :key="match.competitor_id" class="pr" style="align-items:center;">
|
||||||
<div class="pr-inner" style="gap:16px;">
|
<div class="pr-inner" style="gap:16px;">
|
||||||
<div class="pr-label" style="width:140px;">{{ match.competitor_name }}</div>
|
<div class="pr-label" style="width:140px;">{{ match.competitor_name }}</div>
|
||||||
|
|
|
||||||
|
|
@ -66,7 +66,11 @@ export const STANDARD_ROWS = [
|
||||||
format: 'currency',
|
format: 'currency',
|
||||||
compute: (r) => divOrNull(r?.majorityPrice, r?.duration),
|
compute: (r) => divOrNull(r?.majorityPrice, r?.duration),
|
||||||
},
|
},
|
||||||
{ key: 'comments', label: 'Suggested changes for G/ Competitive Comments', format: 'text' },
|
// '{company}' is substituted at render time via formatLabel() — was
|
||||||
|
// hardcoded to the literal letter "G" (a leftover from "G Adventures",
|
||||||
|
// the hackathon reference client), showing every tenant's own results
|
||||||
|
// table as if it belonged to a company literally named "G".
|
||||||
|
{ key: 'comments', label: 'Suggested changes for {company}/ Competitive Comments', format: 'text' },
|
||||||
{ key: 'dateUsed', label: 'Date Used for Majority Price', format: 'text' },
|
{ key: 'dateUsed', label: 'Date Used for Majority Price', format: 'text' },
|
||||||
{ key: 'audPrice', label: 'AUD Majority price', format: 'currency' },
|
{ key: 'audPrice', label: 'AUD Majority price', format: 'currency' },
|
||||||
{ key: 'cadPrice', label: 'CAD Majority price', format: 'currency' },
|
{ key: 'cadPrice', label: 'CAD Majority price', format: 'currency' },
|
||||||
|
|
@ -108,6 +112,16 @@ export function getRowsForTabType(tabType) {
|
||||||
return tabType === 'NGS' ? NGS_ROWS : STANDARD_ROWS
|
return tabType === 'NGS' ? NGS_ROWS : STANDARD_ROWS
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Substitutes the '{company}' token in a row label with the tenant's
|
||||||
|
* real company name — shared by ResultsTable.vue and SyncToSheet.vue so
|
||||||
|
* the displayed table and the copied clipboard rows never diverge.
|
||||||
|
* Falls back to a generic phrase when the tenant name isn't loaded yet.
|
||||||
|
*/
|
||||||
|
export function formatLabel(label, companyName) {
|
||||||
|
return label.replace('{company}', companyName || 'your company')
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Resolves a row's raw (unformatted) value for one result item —
|
* Resolves a row's raw (unformatted) value for one result item —
|
||||||
* shared by ResultsTable.vue (display) and SyncToSheet.vue (clipboard
|
* shared by ResultsTable.vue (display) and SyncToSheet.vue (clipboard
|
||||||
|
|
|
||||||
|
|
@ -42,6 +42,7 @@ const appsScriptInstallUrl = import.meta.env.VITE_APPS_SCRIPT_INSTALL_URL
|
||||||
|
|
||||||
onMounted(async () => {
|
onMounted(async () => {
|
||||||
await authStore.fetchTenant()
|
await authStore.fetchTenant()
|
||||||
|
websiteInput.value = authStore.tenant?.website || ''
|
||||||
loading.value = false
|
loading.value = false
|
||||||
})
|
})
|
||||||
|
|
||||||
|
|
@ -80,6 +81,27 @@ async function openTemplate() {
|
||||||
window.open(url, '_blank')
|
window.open(url, '_blank')
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// The tenant's own storefront — used by the "Your trip page" step in
|
||||||
|
// Analyse to find and scrape the client's own matching trip, so
|
||||||
|
// analysis comments compare against real extracted data instead of
|
||||||
|
// only what was typed into the intent form. Freely editable, unlike
|
||||||
|
// timezone's first-write-wins capture.
|
||||||
|
const websiteInput = ref('')
|
||||||
|
const websiteSaving = ref(false)
|
||||||
|
const websiteSaved = ref(false)
|
||||||
|
|
||||||
|
async function saveWebsite() {
|
||||||
|
websiteSaving.value = true
|
||||||
|
websiteSaved.value = false
|
||||||
|
try {
|
||||||
|
await tenantRepository.updateWebsite(websiteInput.value.trim())
|
||||||
|
await authStore.fetchTenant()
|
||||||
|
websiteSaved.value = true
|
||||||
|
} finally {
|
||||||
|
websiteSaving.value = false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// Genuinely no-op on the pasted value per design: we never validate it
|
// Genuinely no-op on the pasted value per design: we never validate it
|
||||||
// against Google, never fetch it, never store it beyond this local ref.
|
// against Google, never fetch it, never store it beyond this local ref.
|
||||||
// The confirmation is psychological — the PM telling themselves they've
|
// The confirmation is psychological — the PM telling themselves they've
|
||||||
|
|
@ -105,6 +127,19 @@ function replayTour() {
|
||||||
<div v-else class="cg-left">
|
<div v-else class="cg-left">
|
||||||
<SubscriptionCard :tenant="authStore.tenant" @manage-billing="manageBilling" />
|
<SubscriptionCard :tenant="authStore.tenant" @manage-billing="manageBilling" />
|
||||||
|
|
||||||
|
<div class="card" style="padding:18px 20px;display:flex;flex-direction:column;gap:10px;">
|
||||||
|
<div class="ct">Your company website</div>
|
||||||
|
<p style="font-size:12px;color:var(--t3);margin:0;">
|
||||||
|
Used to find and scrape your own matching trip page during analysis, so
|
||||||
|
comments compare against your real trip data instead of just what you typed.
|
||||||
|
</p>
|
||||||
|
<div style="display:flex;gap:8px;">
|
||||||
|
<UInput v-model="websiteInput" placeholder="https://yourcompany.com" class="flex-1" />
|
||||||
|
<UButton color="primary" :loading="websiteSaving" @click="saveWebsite">Save</UButton>
|
||||||
|
</div>
|
||||||
|
<p v-if="websiteSaved" style="font-size:12px;color:var(--green-dark);margin:0;">✓ Saved</p>
|
||||||
|
</div>
|
||||||
|
|
||||||
<div v-if="upgradePitch" class="card" style="padding:20px;">
|
<div v-if="upgradePitch" class="card" style="padding:20px;">
|
||||||
<div style="font-size:14px;font-weight:800;margin-bottom:12px;">🚀 {{ upgradePitch.title }}</div>
|
<div style="font-size:14px;font-weight:800;margin-bottom:12px;">🚀 {{ upgradePitch.title }}</div>
|
||||||
<ul style="margin:0 0 14px;padding:0;list-style:none;display:flex;flex-direction:column;gap:6px;">
|
<ul style="margin:0 0 14px;padding:0;list-style:none;display:flex;flex-direction:column;gap:6px;">
|
||||||
|
|
|
||||||
|
|
@ -5,10 +5,12 @@ import DashboardLayout from '../components/layout/DashboardLayout.vue'
|
||||||
import StatCard from '../components/shared/StatCard.vue'
|
import StatCard from '../components/shared/StatCard.vue'
|
||||||
import EmptyState from '../components/shared/EmptyState.vue'
|
import EmptyState from '../components/shared/EmptyState.vue'
|
||||||
import FeedItem from '../components/activity/FeedItem.vue'
|
import FeedItem from '../components/activity/FeedItem.vue'
|
||||||
|
import WatchlistCard from '../components/activity/WatchlistCard.vue'
|
||||||
import { useAuthStore } from '../stores/auth_store'
|
import { useAuthStore } from '../stores/auth_store'
|
||||||
import { useOnboardingTour } from '../composables/useOnboardingTour'
|
import { useOnboardingTour } from '../composables/useOnboardingTour'
|
||||||
import * as alertRepository from '../repositories/alert_repository'
|
import * as alertRepository from '../repositories/alert_repository'
|
||||||
import * as competitorRepository from '../repositories/competitor_repository'
|
import * as competitorRepository from '../repositories/competitor_repository'
|
||||||
|
import * as watchlistRepository from '../repositories/watchlist_repository'
|
||||||
|
|
||||||
const authStore = useAuthStore()
|
const authStore = useAuthStore()
|
||||||
const { tour, shouldShowTour, getResumeStep } = useOnboardingTour()
|
const { tour, shouldShowTour, getResumeStep } = useOnboardingTour()
|
||||||
|
|
@ -17,14 +19,17 @@ const loading = ref(true)
|
||||||
const allAlerts = ref([])
|
const allAlerts = ref([])
|
||||||
const competitorCount = ref(0)
|
const competitorCount = ref(0)
|
||||||
const activeTab = ref('all')
|
const activeTab = ref('all')
|
||||||
|
const watchlist = ref([])
|
||||||
|
|
||||||
onMounted(async () => {
|
onMounted(async () => {
|
||||||
const [alerts, competitors] = await Promise.all([
|
const [alerts, competitors, watched] = await Promise.all([
|
||||||
alertRepository.listAlerts({ limit: 100 }),
|
alertRepository.listAlerts({ limit: 100 }),
|
||||||
competitorRepository.listCompetitors(authStore.email)
|
competitorRepository.listCompetitors(authStore.email),
|
||||||
|
watchlistRepository.listWatchlist()
|
||||||
])
|
])
|
||||||
allAlerts.value = alerts
|
allAlerts.value = alerts
|
||||||
competitorCount.value = competitors.length
|
competitorCount.value = competitors.length
|
||||||
|
watchlist.value = watched
|
||||||
loading.value = false
|
loading.value = false
|
||||||
|
|
||||||
const onboarding = authStore.onboarding
|
const onboarding = authStore.onboarding
|
||||||
|
|
@ -35,6 +40,11 @@ onMounted(async () => {
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|
||||||
|
async function removeWatchlistItem(itemId) {
|
||||||
|
await watchlistRepository.removeFromWatchlist(itemId)
|
||||||
|
watchlist.value = await watchlistRepository.listWatchlist()
|
||||||
|
}
|
||||||
|
|
||||||
const priceChanges = computed(() => allAlerts.value.filter((a) => a.alert_type === 'price_change'))
|
const priceChanges = computed(() => allAlerts.value.filter((a) => a.alert_type === 'price_change'))
|
||||||
const priceDrops = computed(() => priceChanges.value.filter((a) => (a.change_percent ?? 0) < 0))
|
const priceDrops = computed(() => priceChanges.value.filter((a) => (a.change_percent ?? 0) < 0))
|
||||||
const newProducts = computed(() => allAlerts.value.filter((a) => a.alert_type === 'new_product'))
|
const newProducts = computed(() => allAlerts.value.filter((a) => a.alert_type === 'new_product'))
|
||||||
|
|
@ -100,6 +110,12 @@ function timeAgo(ts) {
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<div class="cg-left">
|
<div class="cg-left">
|
||||||
|
<WatchlistCard
|
||||||
|
:items="watchlist"
|
||||||
|
:current-seat-email="authStore.email"
|
||||||
|
:tier="authStore.tier"
|
||||||
|
@remove="removeWatchlistItem"
|
||||||
|
/>
|
||||||
<EmptyState
|
<EmptyState
|
||||||
v-if="!allAlerts.length"
|
v-if="!allAlerts.length"
|
||||||
icon="📡"
|
icon="📡"
|
||||||
|
|
|
||||||
|
|
@ -13,6 +13,7 @@ import { useAuthStore } from '../stores/auth_store'
|
||||||
import { useAnalysisStore } from '../stores/analysis_store'
|
import { useAnalysisStore } from '../stores/analysis_store'
|
||||||
import * as competitorRepository from '../repositories/competitor_repository'
|
import * as competitorRepository from '../repositories/competitor_repository'
|
||||||
import * as analysisRepository from '../repositories/analysis_repository'
|
import * as analysisRepository from '../repositories/analysis_repository'
|
||||||
|
import * as watchlistRepository from '../repositories/watchlist_repository'
|
||||||
|
|
||||||
const authStore = useAuthStore()
|
const authStore = useAuthStore()
|
||||||
const analysisStore = useAnalysisStore()
|
const analysisStore = useAnalysisStore()
|
||||||
|
|
@ -23,16 +24,28 @@ const step = ref('form') // 'form' | 'confirm' | 'progress' | 'results'
|
||||||
const findingUrls = ref(false)
|
const findingUrls = ref(false)
|
||||||
const confirming = ref(false)
|
const confirming = ref(false)
|
||||||
const matches = ref([])
|
const matches = ref([])
|
||||||
|
// The tenant's own matching trip page (client self-scrape) — null when
|
||||||
|
// the tenant has no website configured. Independent of `matches`.
|
||||||
|
const ownTrip = ref(null)
|
||||||
const intent = ref(null)
|
const intent = ref(null)
|
||||||
const displayResults = ref([])
|
const displayResults = ref([])
|
||||||
// competitor_id -> confirmed URL from the last UrlConfirmation.vue submit —
|
// competitor_id -> confirmed URL from the last UrlConfirmation.vue submit —
|
||||||
// so a single-competitor retry reuses the same matched/swapped page rather
|
// so a single-competitor retry reuses the same matched/swapped page rather
|
||||||
// than falling back to the competitor's generic root domain.
|
// than falling back to the competitor's generic root domain.
|
||||||
const confirmedUrls = ref({})
|
const confirmedUrls = ref({})
|
||||||
|
// The own-trip URL confirmed in the same submit, if any — threaded into
|
||||||
|
// single-competitor retries too, so a retry gets the same real
|
||||||
|
// ground-truth grounding as the original batch run.
|
||||||
|
const confirmedOwnTripUrl = ref(null)
|
||||||
|
// Tenant-wide watchlist (every seat's) — ResultsTable.vue uses this plus
|
||||||
|
// authStore.email to show "★ Watching" vs "☆ Watch" for the current seat.
|
||||||
|
const watchlist = ref([])
|
||||||
|
const watchlistError = ref('')
|
||||||
|
|
||||||
onMounted(async () => {
|
onMounted(async () => {
|
||||||
competitors.value = await competitorRepository.listCompetitors(authStore.email)
|
competitors.value = await competitorRepository.listCompetitors(authStore.email)
|
||||||
loadingCompetitors.value = false
|
loadingCompetitors.value = false
|
||||||
|
watchlist.value = await watchlistRepository.listWatchlist()
|
||||||
await analysisStore.loadHistory(authStore.email)
|
await analysisStore.loadHistory(authStore.email)
|
||||||
|
|
||||||
analysisStore.checkForPendingJob()
|
analysisStore.checkForPendingJob()
|
||||||
|
|
@ -70,21 +83,25 @@ async function handleIntentSubmit(formIntent) {
|
||||||
intent.value = formIntent
|
intent.value = formIntent
|
||||||
findingUrls.value = true
|
findingUrls.value = true
|
||||||
try {
|
try {
|
||||||
matches.value = await analysisRepository.findUrls({
|
const result = await analysisRepository.findUrls({
|
||||||
|
tenantId: authStore.tenantId,
|
||||||
destination: formIntent.destination,
|
destination: formIntent.destination,
|
||||||
duration: formIntent.duration,
|
duration: formIntent.duration,
|
||||||
travelStyle: formIntent.travelStyle,
|
travelStyle: formIntent.travelStyle,
|
||||||
competitorIds: formIntent.competitorIds
|
competitorIds: formIntent.competitorIds
|
||||||
})
|
})
|
||||||
|
matches.value = result.matches
|
||||||
|
ownTrip.value = result.ownTrip
|
||||||
step.value = 'confirm'
|
step.value = 'confirm'
|
||||||
} finally {
|
} finally {
|
||||||
findingUrls.value = false
|
findingUrls.value = false
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
async function handleConfirm(confirmedCompetitors) {
|
async function handleConfirm({ competitors: confirmedCompetitors, ownTripUrl }) {
|
||||||
confirming.value = true
|
confirming.value = true
|
||||||
confirmedUrls.value = Object.fromEntries(confirmedCompetitors.map((c) => [c.id, c.url]))
|
confirmedUrls.value = Object.fromEntries(confirmedCompetitors.map((c) => [c.id, c.url]))
|
||||||
|
confirmedOwnTripUrl.value = ownTripUrl
|
||||||
try {
|
try {
|
||||||
const { job_id } = await analysisRepository.runResearch({
|
const { job_id } = await analysisRepository.runResearch({
|
||||||
tenant_id: authStore.tenantId,
|
tenant_id: authStore.tenantId,
|
||||||
|
|
@ -94,6 +111,7 @@ async function handleConfirm(confirmedCompetitors) {
|
||||||
tab_type: intent.value.tabType,
|
tab_type: intent.value.tabType,
|
||||||
productName: intent.value.productName,
|
productName: intent.value.productName,
|
||||||
competitors: confirmedCompetitors,
|
competitors: confirmedCompetitors,
|
||||||
|
own_trip_url: ownTripUrl,
|
||||||
force_refresh: intent.value.forceRefresh,
|
force_refresh: intent.value.forceRefresh,
|
||||||
email: authStore.email
|
email: authStore.email
|
||||||
})
|
})
|
||||||
|
|
@ -113,6 +131,7 @@ function startOver() {
|
||||||
analysisStore.stopPolling()
|
analysisStore.stopPolling()
|
||||||
step.value = 'form'
|
step.value = 'form'
|
||||||
matches.value = []
|
matches.value = []
|
||||||
|
ownTrip.value = null
|
||||||
}
|
}
|
||||||
|
|
||||||
async function retryCompetitor(item) {
|
async function retryCompetitor(item) {
|
||||||
|
|
@ -136,6 +155,7 @@ async function retryCompetitor(item) {
|
||||||
tab_type: intent.value.tabType,
|
tab_type: intent.value.tabType,
|
||||||
productName: intent.value.productName,
|
productName: intent.value.productName,
|
||||||
competitors: [{ id: competitor.id, name: competitor.name, url: retryUrl }],
|
competitors: [{ id: competitor.id, name: competitor.name, url: retryUrl }],
|
||||||
|
own_trip_url: confirmedOwnTripUrl.value,
|
||||||
force_refresh: intent.value.forceRefresh,
|
force_refresh: intent.value.forceRefresh,
|
||||||
email: authStore.email
|
email: authStore.email
|
||||||
})
|
})
|
||||||
|
|
@ -167,6 +187,24 @@ async function restoreFromHistory(jobId) {
|
||||||
buildDisplayResults()
|
buildDisplayResults()
|
||||||
step.value = 'results'
|
step.value = 'results'
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async function handleWatch({ competitorId, url, tripName }) {
|
||||||
|
watchlistError.value = ''
|
||||||
|
try {
|
||||||
|
await watchlistRepository.addToWatchlist({ competitorId, url, tripName })
|
||||||
|
watchlist.value = await watchlistRepository.listWatchlist()
|
||||||
|
} catch (err) {
|
||||||
|
// watchlist_full is the expected, common case (cap reached for this
|
||||||
|
// seat's tier) — surfaced inline rather than as a generic error.
|
||||||
|
watchlistError.value = err?.response?.data?.message || 'Could not add to your watchlist — please try again.'
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function handleUnwatch({ itemId }) {
|
||||||
|
watchlistError.value = ''
|
||||||
|
await watchlistRepository.removeFromWatchlist(itemId)
|
||||||
|
watchlist.value = await watchlistRepository.listWatchlist()
|
||||||
|
}
|
||||||
</script>
|
</script>
|
||||||
|
|
||||||
<template>
|
<template>
|
||||||
|
|
@ -193,6 +231,7 @@ async function restoreFromHistory(jobId) {
|
||||||
<UrlConfirmation
|
<UrlConfirmation
|
||||||
v-else-if="step === 'confirm'"
|
v-else-if="step === 'confirm'"
|
||||||
:matches="matches"
|
:matches="matches"
|
||||||
|
:own-trip="ownTrip"
|
||||||
:submitting="confirming"
|
:submitting="confirming"
|
||||||
@confirm="handleConfirm"
|
@confirm="handleConfirm"
|
||||||
@back="backToForm"
|
@back="backToForm"
|
||||||
|
|
@ -216,7 +255,19 @@ async function restoreFromHistory(jobId) {
|
||||||
@skip="skipCompetitor"
|
@skip="skipCompetitor"
|
||||||
@cancel="analysisStore.cancelAnalysis"
|
@cancel="analysisStore.cancelAnalysis"
|
||||||
/>
|
/>
|
||||||
<ResultsTable v-if="displayResults.length" :results="displayResults" :tab-type="analysisStore.tabType" />
|
<p v-if="watchlistError" style="color:var(--red);font-size:12.5px;margin:0;">{{ watchlistError }}</p>
|
||||||
|
<ResultsTable
|
||||||
|
v-if="displayResults.length"
|
||||||
|
:results="displayResults"
|
||||||
|
:tab-type="analysisStore.tabType"
|
||||||
|
:own-trip="analysisStore.results?.own_trip"
|
||||||
|
:own-trip-label="authStore.tenant?.name"
|
||||||
|
:company-name="authStore.tenant?.name"
|
||||||
|
:watchlist="watchlist"
|
||||||
|
:current-seat-email="authStore.email"
|
||||||
|
@watch="handleWatch"
|
||||||
|
@unwatch="handleUnwatch"
|
||||||
|
/>
|
||||||
<div v-if="displayResults.length" class="card">
|
<div v-if="displayResults.length" class="card">
|
||||||
<div class="ch"><div class="ct">Confidence</div></div>
|
<div class="ch"><div class="ct">Confidence</div></div>
|
||||||
<ConfidenceBar
|
<ConfidenceBar
|
||||||
|
|
@ -226,7 +277,12 @@ async function restoreFromHistory(jobId) {
|
||||||
:relevancy="item.result?.relevancy"
|
:relevancy="item.result?.relevancy"
|
||||||
/>
|
/>
|
||||||
</div>
|
</div>
|
||||||
<SyncToSheet v-if="displayResults.length" :results="displayResults" :tab-type="analysisStore.tabType" />
|
<SyncToSheet
|
||||||
|
v-if="displayResults.length"
|
||||||
|
:results="displayResults"
|
||||||
|
:tab-type="analysisStore.tabType"
|
||||||
|
:company-name="authStore.tenant?.name"
|
||||||
|
/>
|
||||||
<UButton color="neutral" variant="ghost" @click="startOver">← Run another analysis</UButton>
|
<UButton color="neutral" variant="ghost" @click="startOver">← Run another analysis</UButton>
|
||||||
</template>
|
</template>
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,10 +1,12 @@
|
||||||
import apiService from '../services/api_service'
|
import apiService from '../services/api_service'
|
||||||
|
|
||||||
export async function findUrls({ destination, duration, travelStyle, competitorIds }) {
|
export async function findUrls({ tenantId, destination, duration, travelStyle, competitorIds }) {
|
||||||
const { data } = await apiService.post('/research/find-urls', {
|
const { data } = await apiService.post('/research/find-urls', {
|
||||||
destination, duration, travel_style: travelStyle, competitor_ids: competitorIds
|
tenant_id: tenantId, destination, duration, travel_style: travelStyle, competitor_ids: competitorIds
|
||||||
})
|
})
|
||||||
return data.matches
|
// own_trip is null when the tenant has no website configured (or the
|
||||||
|
// caller omitted tenant_id) — the caller treats it as "not available."
|
||||||
|
return { matches: data.matches, ownTrip: data.own_trip }
|
||||||
}
|
}
|
||||||
|
|
||||||
export async function runResearch(payload) {
|
export async function runResearch(payload) {
|
||||||
|
|
|
||||||
|
|
@ -24,6 +24,11 @@ export async function captureTimezone(timezone) {
|
||||||
return data.timezone
|
return data.timezone
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export async function updateWebsite(website) {
|
||||||
|
const { data } = await apiService.patch('/account/website', { website })
|
||||||
|
return data.website
|
||||||
|
}
|
||||||
|
|
||||||
export async function getTemplateUrl() {
|
export async function getTemplateUrl() {
|
||||||
const { data } = await apiService.get('/account/template')
|
const { data } = await apiService.get('/account/template')
|
||||||
return data.template_url
|
return data.template_url
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,21 @@
|
||||||
|
import apiService from '../services/api_service'
|
||||||
|
|
||||||
|
// All Flask API calls for the per-seat trip watchlist — see
|
||||||
|
// backend/services/watchlist_service.py's WATCHLIST_CAP_PER_SEAT.
|
||||||
|
|
||||||
|
export async function listWatchlist() {
|
||||||
|
const { data } = await apiService.get('/watchlist')
|
||||||
|
return data
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function addToWatchlist({ competitorId, url, tripName }) {
|
||||||
|
const { data } = await apiService.post('/watchlist', {
|
||||||
|
competitor_id: competitorId, url, trip_name: tripName
|
||||||
|
})
|
||||||
|
return data
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function removeFromWatchlist(itemId) {
|
||||||
|
const { data } = await apiService.delete(`/watchlist/${itemId}`)
|
||||||
|
return data
|
||||||
|
}
|
||||||
|
|
@ -58,13 +58,13 @@
|
||||||
{
|
{
|
||||||
"parameters": {
|
"parameters": {
|
||||||
"method": "GET",
|
"method": "GET",
|
||||||
"url": "={{ $env.POCKETBASE_URL }}/api/collections/competitors/records",
|
"url": "={{ $env.POCKETBASE_URL }}/api/collections/trip_watchlist/records",
|
||||||
"sendQuery": true,
|
"sendQuery": true,
|
||||||
"queryParameters": {
|
"queryParameters": {
|
||||||
"parameters": [
|
"parameters": [
|
||||||
{
|
{
|
||||||
"name": "filter",
|
"name": "filter",
|
||||||
"value": "={{ 'tenant_id = \"' + $json.id + '\" && active = true && (change_detected = true || last_scraped = \"\" || last_scraped < \"' + $now.minus(24, 'hours').toISO() + '\")' }}"
|
"value": "={{ 'tenant_id = \"' + $json.id + '\"' }}"
|
||||||
},
|
},
|
||||||
{ "name": "perPage", "value": "200" }
|
{ "name": "perPage", "value": "200" }
|
||||||
]
|
]
|
||||||
|
|
@ -72,11 +72,12 @@
|
||||||
"authentication": "genericCredentialType",
|
"authentication": "genericCredentialType",
|
||||||
"genericAuthType": "httpHeaderAuth"
|
"genericAuthType": "httpHeaderAuth"
|
||||||
},
|
},
|
||||||
"id": "list-stale-competitors",
|
"id": "list-watchlist-items",
|
||||||
"name": "List stale/changed competitors",
|
"name": "List watchlist items",
|
||||||
"type": "n8n-nodes-base.httpRequest",
|
"type": "n8n-nodes-base.httpRequest",
|
||||||
"typeVersion": 4.2,
|
"typeVersion": 4.2,
|
||||||
"position": [880, -80],
|
"position": [880, -80],
|
||||||
|
"notes": "Deliberately no staleness pre-filter (no change_detected/last_scraped check) — every watched trip is sent every night, and analysis_service.needs_refresh()'s existing per-url freshness check decides fast-vs-refresh internally, exactly as a manual dashboard run would. The watchlist's own per-seat cap (services/watchlist_service.py's WATCHLIST_CAP_PER_SEAT) is what bounds nightly volume now, not a staleness filter — replaces the old 'competitors' collection query that targeted each competitor's root website instead of a specific trip page.",
|
||||||
"credentials": {
|
"credentials": {
|
||||||
"httpHeaderAuth": { "id": "pocketbase-admin-auth", "name": "PocketBase Admin Auth" }
|
"httpHeaderAuth": { "id": "pocketbase-admin-auth", "name": "PocketBase Admin Auth" }
|
||||||
}
|
}
|
||||||
|
|
@ -95,8 +96,8 @@
|
||||||
"combinator": "and"
|
"combinator": "and"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"id": "has-stale-competitors",
|
"id": "has-watchlist-items",
|
||||||
"name": "Any stale competitors?",
|
"name": "Any watched trips?",
|
||||||
"type": "n8n-nodes-base.if",
|
"type": "n8n-nodes-base.if",
|
||||||
"typeVersion": 2,
|
"typeVersion": 2,
|
||||||
"position": [1100, -80]
|
"position": [1100, -80]
|
||||||
|
|
@ -112,7 +113,7 @@
|
||||||
"sendBody": true,
|
"sendBody": true,
|
||||||
"bodyParameters": {
|
"bodyParameters": {
|
||||||
"parameters": [
|
"parameters": [
|
||||||
{ "name": "competitors", "value": "={{ $json.items.map(c => ({ id: c.id, name: c.name })) }}" }
|
{ "name": "competitors", "value": "={{ $json.items.map(w => ({ id: w.competitor_id, url: w.url })) }}" }
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
|
|
@ -120,7 +121,8 @@
|
||||||
"name": "POST /internal/scrape/{tenant_id}",
|
"name": "POST /internal/scrape/{tenant_id}",
|
||||||
"type": "n8n-nodes-base.httpRequest",
|
"type": "n8n-nodes-base.httpRequest",
|
||||||
"typeVersion": 4.2,
|
"typeVersion": 4.2,
|
||||||
"position": [1320, -80]
|
"position": [1320, -80],
|
||||||
|
"notes": "Body shape now carries `url` per entry (the specific watched trip page) instead of just `{id, name}` — jobs.py's run_research_job() already threads entry['url'] through as confirmed_url per competitor (fixed earlier this session for manual trip-intent analysis), and already handles multiple entries sharing the same competitor_id independently and correctly (two seats watching two different trips on the same competitor). `name` is intentionally omitted — jobs.py only ever uses it as a fallback label when competitor_repository.get_by_id() finds nothing; the real name is always looked up fresh on the happy path."
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
"connections": {
|
"connections": {
|
||||||
|
|
@ -136,13 +138,13 @@
|
||||||
"Loop Over Tenants": {
|
"Loop Over Tenants": {
|
||||||
"main": [
|
"main": [
|
||||||
[],
|
[],
|
||||||
[{ "node": "List stale/changed competitors", "type": "main", "index": 0 }]
|
[{ "node": "List watchlist items", "type": "main", "index": 0 }]
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
"List stale/changed competitors": {
|
"List watchlist items": {
|
||||||
"main": [[{ "node": "Any stale competitors?", "type": "main", "index": 0 }]]
|
"main": [[{ "node": "Any watched trips?", "type": "main", "index": 0 }]]
|
||||||
},
|
},
|
||||||
"Any stale competitors?": {
|
"Any watched trips?": {
|
||||||
"main": [
|
"main": [
|
||||||
[{ "node": "POST /internal/scrape/{tenant_id}", "type": "main", "index": 0 }],
|
[{ "node": "POST /internal/scrape/{tenant_id}", "type": "main", "index": 0 }],
|
||||||
[]
|
[]
|
||||||
|
|
@ -155,6 +157,6 @@
|
||||||
"active": false,
|
"active": false,
|
||||||
"settings": { "executionOrder": "v1" },
|
"settings": { "executionOrder": "v1" },
|
||||||
"meta": {
|
"meta": {
|
||||||
"description": "CLAUDE.md §15 daily scrape cron — queries PocketBase directly for each active tenant's stale/changed competitors (last_scraped > 24hrs OR change_detected = true), then triggers a scrape job per tenant via the existing Flask /internal/scrape/<tenant_id> route. Not yet imported/run against a live n8n instance — verify the PocketBase filter syntax and credential wiring before enabling."
|
"description": "CLAUDE.md §15 daily scrape cron — queries PocketBase directly for each active tenant's trip_watchlist items (every seat's watched trips, no staleness pre-filter — see 'List watchlist items' node notes) and triggers a scrape job per tenant via the existing Flask /internal/scrape/<tenant_id> route, one entry per watched trip URL. Replaces the earlier version that re-scraped each competitor's root website nightly — that page is flagged low-signal by services/url_utils.py's is_generic_homepage() and had no natural volume cap. Not yet imported/run against a live n8n instance — verify the PocketBase filter syntax and credential wiring before enabling."
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue