Help API Docs

Trigger crawls & pull data out as JSON.

Kick off a fresh crawl when you deploy, then read every audit’s data — pages, issues, broken links, images, scripts and CSS — through a simple HTTP API. Point a script or an AI agent at the results, and fix issues automatically.

EU1 (Dev/Test): https://eu1.website-toolkit.co.ukEU2 (Live): https://eu2.website-toolkit.co.uk

Check your dashboard or contact Dave to confirm which server your domain is assigned to.

Quick start

The API lets you start a crawl for a domain and read the data a crawl produced. Here’s how to pull the broken links from the latest crawl.

BASE="https://eu1.website-toolkit.co.uk"
KEY="your-api-key"

# 1. What crawls exist for a domain?
curl -s -H "Authorization: Bearer $KEY" \
  "$BASE/api/v1/example.com"

# 2. Pull the issues from the most recent crawl
curl -s -H "Authorization: Bearer $KEY" \
  "$BASE/api/v1/example.com/latest/issues"

# 3. Pull just the broken links
curl -s -H "Authorization: Bearer $KEY" \
  "$BASE/api/v1/example.com/latest/broken-links"

Authentication

Every request must send an API key as a Bearer token in the Authorization header.

Authorization: Bearer your-api-key

🔒 Tenant Isolation

A key only sees crawls owned by its own tenant. Cross-tenant access returns a 404 Not Found so the API never reveals whether data exists.

Trigger a crawl

Start a full crawl of a domain — ideal for GitHub Actions or deploy webhooks. Use GET or POST.

POST /api/crawl-site/{domain} — recommended
GET /api/crawl-site/{token}/{domain} — for plain webhooks
# Recommended header auth
curl -s -X POST -H "Authorization: Bearer $KEY" \
  "$BASE/api/crawl-site/example.com"

GitHub Actions Example

name: Re-crawl on deploy
on:
  push:
    branches: [main]
jobs:
  trigger-crawl:
    runs-on: ubuntu-latest
    steps:
      - run: |
          curl -sS -X POST \
            -H "Authorization: Bearer ${{ secrets.CRAWLER_KEY }}" \
            --fail-with-body \
            "https://eu1.website-toolkit.co.uk/api/crawl-site/example.com"

Agent Data API

The primary read-only API for pulling audit data. Keyed by an agency API key from the admin console.

Discovery

GET /api/v1/{domain}

Lists recent crawls and available data types for a domain.

Data Feed

GET /api/v1/{domain}/{crawlId|latest}/{type}

Pagination: ?page=1&pageSize=100 (max 1000).

TypeDescription
pagesEvery crawled page: status, title, H1, load time, size, redirect target, SEO issues.
issuesDetected problems: type, the page it was found on, a recommendation, destination.
linksEvery link checked: link URL, owning page, link text, exists flag.
imagesEvery image: image URL, page, alt text, size, exists flag.
cssEvery stylesheet: URL, page, size, exists flag.
scriptsEvery script: URL, page, location, size, exists flag.
broken-linksOnly the links whose target failed.
broken-imagesOnly the images that failed to load.
broken-cssOnly the stylesheets that failed.
broken-scriptsOnly the scripts that failed.
wordsSite-level content-integrity word list (not crawl-scoped).
summaryThe crawl’s headline metrics (single object, not paginated).

Column reference

pages   : content_status, content_url, page_title, heading_1, load_time_ms,
          size_kb, page_redirects_to, seo_issues
issues  : id, issue_type, found_on_url, recommendation, destination, screenshot_path
links   : link_exists, link_url, owning_page_url, link_text, link_advisory, screenshot_path
images  : image_exists, image_url, image_page_url, image_alt, size_kb
css     : css_exists, css_url, css_page_url, size_kb
scripts : script_exists, script_url, script_page_url, script_location, size_kb

Client API & MCP

A read/write API keyed by a client’s own token. Perfect for AI assistants that need to read audits, mark issues fixed, and re-check URLs.

GET /api/client-api/v1/domains — list domains
GET /api/client-api/v1/{domain}/crawls — recent audits
GET /api/client-api/v1/{domain}/{crawlId|latest}/{type} — read results
POST /api/client-api/v1/{domain}/audit — start audit (details)
GET /api/client-api/v1/{domain}/{crawlId|latest}/report.pdf — download report PDF (details)
POST /api/client-api/v1/{domain}/{crawlId}/issues/resolve — mark fixed
POST /api/client-api/v1/{domain}/{crawlId}/issues/unresolve — undo mark fixed
GET /api/client-api/v1/{domain}/broken-urls — get URLs for re-check
POST /api/client-api/v1/{domain}/{crawlId}/page-status — report re-check results
GET /api/client-api/v1/{domain}/sitemap-check — check sitemap

Mark fixed / undo take { "issueId": 42 }. An issueId is only valid within its own crawl, so always pass the crawlId it came from — an id from another crawl returns 404.

MCP Protocol

Exposed over POST /mcp for Claude Code and Claude Desktop. The start_audit tool accepts the same optional checksOff list as the REST endpoint.

claude mcp add --transport http wsc-crawler \
  https://eu1.website-toolkit.co.uk/mcp \
  --header "Authorization: Bearer $TOKEN"

Start an audit

POST /api/client-api/v1/{domain}/audit — { "mode": "basic" | "advanced", "checksOff"?: [...] }

mode is basic or advanced — advanced is the full audit, the same one the portal’s Run full audit button starts. Any other value returns 400.

The call returns straight away with the new crawlId; the audit runs in the background and can take minutes to hours. Poll GET /{domain}/crawls for its status rather than waiting on the response.

No need to run a batch one at a time: queue every audit, then poll /crawls — they run in parallel as capacity allows. Limited to 30 audits per 15 minutes per token.

Switching checks off (optional)

checksOff switches off optional report checks for this one audit, on top of whatever the domain’s saved Report Settings in the client portal already switch off. The saved settings aren’t changed. Valid keys:

KeyLeaves out
large_imageOversized images
slow_pageSlow pages
not_found_page404 page check
orphaned_pageOrphaned pages
redirect_chainRedirect chains
redirected_linkLinks that redirect
missing_redirectOld pages with no redirect
seoAll SEO issues
seo_lengthTitle / description length only
social_imageSocial share images
duplicate_copyDuplicate copy
dns_changeDNS changes
open_graphOpen Graph tags
structured_dataStructured data
llms_txtllms.txt
ai_freshnessAI freshness signals
ai_training_blockedAI training crawlers blocked
ai_optoutAI opt-out signals

An unknown key returns 400 listing the valid ones, rather than silently leaving the check on. Broken pages, broken resources and security findings can’t be switched off. The response echoes the checksOff it applied.

# Start a full audit
curl -s -X POST -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"mode": "advanced"}' \
  "$BASE/api/client-api/v1/example.com/audit"

# Start one without image-size and SEO-length findings
curl -s -X POST -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"mode": "advanced", "checksOff": ["large_image", "seo_length"]}' \
  "$BASE/api/client-api/v1/example.com/audit"

Download the report PDF

GET /api/client-api/v1/{domain}/{crawlId|latest}/report.pdf

Returns the crawl-report email as an A4 PDF, without its buttons and with the homepage thumbnail included. It’s a copy of the email as sent, including the SSL, domain expiry and PageSpeed figures. Audits finished before this endpoint existed are rebuilt from the stored results instead, and show those three as N/A.

An audit that hasn’t finished returns 409 — poll /crawls until it has. Each PDF is rendered on demand, so downloads have their own limit of 60 per 15 minutes per token. REST only; there’s no MCP equivalent, as MCP results are JSON.

# Download the latest finished audit's report
curl -s -H "Authorization: Bearer $TOKEN" -o example.com.pdf \
  "$BASE/api/client-api/v1/example.com/latest/report.pdf"

Monitoring

GET /api/client-api/v1/{domain}/monitoring — status
POST /api/client-api/v1/{domain}/monitoring — request
POST /api/client-api/v1/{domain}/monitoring/cancel — cancel

Client Feedback

Manage feedback items left via the widget.

GET /api/client-api/v1/feedback — overview
GET /api/client-api/v1/{domain}/feedback — list items
POST /api/client-api/v1/{domain}/feedback/{itemId}/status — set status
POST /api/client-api/v1/{domain}/feedback/{itemId}/note — add note

Launch & Migration

Manage checklists and redirect maps.

Checklists

GET /api/client-api/v1/{domain}/projects — list projects
POST /api/client-api/v1/{domain}/projects — start project
GET /api/client-api/v1/{domain}/projects/{type}/tasks — list tasks
POST /api/client-api/v1/{domain}/projects/{type}/tasks/{taskKey}/state — set state
POST /api/client-api/v1/{domain}/projects/{type}/recheck — re-run auto-check

Redirects

GET /api/client-api/v1/{domain}/redirects — list redirects
POST /api/client-api/v1/{domain}/redirects/import — import from crawl
POST /api/client-api/v1/{domain}/redirects/check — check all
POST /api/client-api/v1/{domain}/redirects/{rowId} — update row

Time Tracking

Log and report time entries against projects.

GET /api/client-api/v1/time/report — pull report
POST /api/client-api/v1/time/entries — log entry

Usage & Account

GET /api/client-api/v1/account — usage & limits
POST /api/client-api/v1/domains — add domain

Response shapes

These are the payloads returned by the Client API, transcribed field-for-field from the query columns each operation selects.

GET /account

{
  "budget": {
    "budget": 20000, "used": 4300, "reserved": 0, "remaining": 15700,
    "windowStart": "2026-06-15T00:00:00.000Z", "resetsAt": "2026-06-22T00:00:00.000Z"
  },
  "residentialProxy": { "cap": 5000, "used": 120, "remaining": 4880 },
  "monitoring": { "slotsTotal": 10, "slotsUsed": 3, "slotsRemaining": 7 }
}

GET /{domain}/crawls

{
  "domain": "example.com",
  "crawls": [
    {
      "crawlId": "8f1c2e4a-...", "startTime": "...", "endTime": "...", "status": "Completed",
      "mode": "advanced", "pages": 84, "brokenPages": 1, "brokenLinks": 3,
      "brokenImages": 0, "brokenCss": 0, "brokenJs": 0, "warnings": 12
    }
  ]
}

GET /{domain}/monitoring

{
  "domain": "example.com", "monitored": true, "pending": false,
  "schedule": "0 6 * * 1", "lastCheckedAt": "2026-06-20T06:00:00.000Z",
  "lastCrawlId": "8f1c2e4a-...", "maxPages": 200
}

GET /{domain}/feedback (one item)

{
  "id": "...", "pageUrl": "https://example.com/pricing", "pagePath": "/pricing",
  "note": "typo in the second bullet", "status": "open",
  "viewport": "1440x900", "createdAt": "...",
  "attachments": [
    { "id": "...", "kind": "screenshot", "mime": "image/webp", "bytes": 48213,
      "url": "https://.../api/project/{editToken}/feedback/{attachmentId}" }
  ],
  "targetSelector": ".hero h1", "category": "content", "hidden": false,
  "internalNote": null, "noteSentAt": null
}

GET /{domain}/projects/{type}/tasks (one task)

{
  "key": "dns-live", "type": "Pre", "category": "DNS", "group": "Go-live",
  "task": "Point DNS at the new host", "auto": true, "custom": false,
  "edited": false, "state": "open", "source": null, "hidden": false
}

GET /{domain}/redirects (one row)

{
  "id": "...", "oldUrl": "https://old.example.com/about-us",
  "expectedNewUrl": "https://new.example.com/about",
  "lastStatus": 301, "resolvedTo": "https://new.example.com/about",
  "result": "ok_301", "hidden": false
}

Examples

Node.js — Pull every issue across all pages

async function fetchAll(domain, crawlId, type) {
  const out = [];
  for (let page = 1; ; page++) {
    const r = await fetch(`${BASE}/api/v1/${domain}/${crawlId}/${type}?page=${page}&pageSize=500`, { headers });
    const { rows, total, pageSize } = await r.json();
    out.push(...rows);
    if (page * pageSize >= total) break;
  }
  return out;
}

const issues = await fetchAll('example.com', 'latest', 'issues');

Bash — Audit a batch of domains, then download each report

# Queue every audit up front — they run in parallel
for d in example.com example.org; do
  curl -s -X POST -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" -d '{"mode": "advanced"}' \
    "$BASE/api/client-api/v1/$d/audit"
done

# Later, once /crawls shows them finished
for d in example.com example.org; do
  curl -s -H "Authorization: Bearer $TOKEN" -o "$d.pdf" \
    "$BASE/api/client-api/v1/$d/latest/report.pdf"
done

Error responses

StatusMeaning
400Invalid domain, unknown type, bad request params, or an unknown mode / checksOff key.
401Missing or unrecognised API key / token.
403Domain not on account or disallowed origin.
404No data found, cross-tenant access, or an issueId from a different crawl.
409Audit not finished yet (report PDF) — poll /crawls and retry.
429Rate limit exceeded (e.g. 30 audits or 60 PDF downloads per 15 minutes per token).
500Internal server error.

Need help or a new feature?

Need an API key for the beta service, or want a request type that isn’t here yet? We’d love to hear what you’re building.