app.bettersight.io/n8n/workflows/daily_scrape_cron.json

{
  "name": "Bettersight — Daily Scrape Cron",
  "nodes": [
    {
      "parameters": {
        "rule": {
          "interval": [{ "field": "cronExpression", "expression": "0 6 * * *" }]
        }
      },
      "id": "schedule-trigger",
      "name": "Schedule Trigger (06:00 daily)",
      "type": "n8n-nodes-base.scheduleTrigger",
      "typeVersion": 1.2,
      "position": [0, 0]
    },
    {
      "parameters": {
        "method": "GET",
        "url": "={{ $env.POCKETBASE_URL }}/api/collections/tenants/records",
        "sendQuery": true,
        "queryParameters": {
          "parameters": [
            { "name": "filter", "value": "status = 'active' || status = 'trial'" },
            { "name": "perPage", "value": "200" }
          ]
        },
        "authentication": "genericCredentialType",
        "genericAuthType": "httpHeaderAuth"
      },
      "id": "list-active-tenants",
      "name": "List active tenants (PocketBase)",
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 4.2,
      "position": [220, 0],
      "credentials": {
        "httpHeaderAuth": { "id": "pocketbase-admin-auth", "name": "PocketBase Admin Auth" }
      },
      "notes": "Header credential should carry the PocketBase admin session token — see repositories/pocketbase_client.py's own auth flow for the equivalent Python-side pattern."
    },
    {
      "parameters": { "fieldToSplitOut": "items" },
      "id": "split-tenants",
      "name": "Split into items",
      "type": "n8n-nodes-base.splitOut",
      "typeVersion": 1,
      "position": [440, 0]
    },
    {
      "parameters": {
        "batchSize": 1
      },
      "id": "loop-tenants",
      "name": "Loop Over Tenants",
      "type": "n8n-nodes-base.splitInBatches",
      "typeVersion": 3,
      "position": [660, 0]
    },
    {
      "parameters": {
        "method": "GET",
        "url": "={{ $env.POCKETBASE_URL }}/api/collections/trip_watchlist/records",
        "sendQuery": true,
        "queryParameters": {
          "parameters": [
            {
              "name": "filter",
              "value": "={{ 'tenant_id = \"' + $json.id + '\"' }}"
            },
            { "name": "perPage", "value": "200" }
          ]
        },
        "authentication": "genericCredentialType",
        "genericAuthType": "httpHeaderAuth"
      },
      "id": "list-watchlist-items",
      "name": "List watchlist items",
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 4.2,
      "position": [880, -80],
      "notes": "Deliberately no staleness pre-filter (no change_detected/last_scraped check) — every watched trip is sent every night, and analysis_service.needs_refresh()'s existing per-url freshness check decides fast-vs-refresh internally, exactly as a manual dashboard run would. The watchlist's own per-seat cap (services/watchlist_service.py's WATCHLIST_CAP_PER_SEAT) is what bounds nightly volume now, not a staleness filter — replaces the old 'competitors' collection query that targeted each competitor's root website instead of a specific trip page.",
      "credentials": {
        "httpHeaderAuth": { "id": "pocketbase-admin-auth", "name": "PocketBase Admin Auth" }
      }
    },
    {
      "parameters": {
        "conditions": {
          "options": { "caseSensitive": true, "leftValue": "", "typeValidation": "loose" },
          "conditions": [
            {
              "leftValue": "={{ $json.items.length }}",
              "rightValue": 0,
              "operator": { "type": "number", "operation": "gt" }
            }
          ],
          "combinator": "and"
        }
      },
      "id": "has-watchlist-items",
      "name": "Any watched trips?",
      "type": "n8n-nodes-base.if",
      "typeVersion": 2,
      "position": [1100, -80]
    },
    {
      "parameters": {
        "method": "POST",
        "url": "={{ $env.BETTERSIGHT_API_BASE_URL }}/internal/scrape/{{ $('Loop Over Tenants').item.json.id }}",
        "sendHeaders": true,
        "headerParameters": {
          "parameters": [{ "name": "X-API-Key", "value": "={{ $env.API_SECRET_KEY }}" }]
        },
        "sendBody": true,
        "bodyParameters": {
          "parameters": [
            { "name": "competitors", "value": "={{ $json.items.map(w => ({ id: w.competitor_id, url: w.url })) }}" }
          ]
        }
      },
      "id": "trigger-scrape",
      "name": "POST /internal/scrape/{tenant_id}",
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 4.2,
      "position": [1320, -80],
      "notes": "Body shape now carries `url` per entry (the specific watched trip page) instead of just `{id, name}` — jobs.py's run_research_job() already threads entry['url'] through as confirmed_url per competitor (fixed earlier this session for manual trip-intent analysis), and already handles multiple entries sharing the same competitor_id independently and correctly (two seats watching two different trips on the same competitor). `name` is intentionally omitted — jobs.py only ever uses it as a fallback label when competitor_repository.get_by_id() finds nothing; the real name is always looked up fresh on the happy path."
    }
  ],
  "connections": {
    "Schedule Trigger (06:00 daily)": {
      "main": [[{ "node": "List active tenants (PocketBase)", "type": "main", "index": 0 }]]
    },
    "List active tenants (PocketBase)": {
      "main": [[{ "node": "Split into items", "type": "main", "index": 0 }]]
    },
    "Split into items": {
      "main": [[{ "node": "Loop Over Tenants", "type": "main", "index": 0 }]]
    },
    "Loop Over Tenants": {
      "main": [
        [],
        [{ "node": "List watchlist items", "type": "main", "index": 0 }]
      ]
    },
    "List watchlist items": {
      "main": [[{ "node": "Any watched trips?", "type": "main", "index": 0 }]]
    },
    "Any watched trips?": {
      "main": [
        [{ "node": "POST /internal/scrape/{tenant_id}", "type": "main", "index": 0 }],
        []
      ]
    },
    "POST /internal/scrape/{tenant_id}": {
      "main": [[{ "node": "Loop Over Tenants", "type": "main", "index": 0 }]]
    }
  },
  "active": false,
  "settings": { "executionOrder": "v1" },
  "meta": {
    "description": "CLAUDE.md §15 daily scrape cron — queries PocketBase directly for each active tenant's trip_watchlist items (every seat's watched trips, no staleness pre-filter — see 'List watchlist items' node notes) and triggers a scrape job per tenant via the existing Flask /internal/scrape/<tenant_id> route, one entry per watched trip URL. Replaces the earlier version that re-scraped each competitor's root website nightly — that page is flagged low-signal by services/url_utils.py's is_generic_homepage() and had no natural volume cap. Not yet imported/run against a live n8n instance — verify the PocketBase filter syntax and credential wiring before enabling."
  }
}