HelpAPI Docs

Trigger crawls & pull data out as JSON.

Kick off a fresh crawl when you deploy, then read every audit’s data — pages, issues, broken links, images, scripts and CSS — through a simple HTTP API. Point a script or an AI agent at the results, and fix issues automatically.

EU1 (Dev/Test):https://eu1.website-toolkit.co.uk
EU2 (Live):https://eu2.website-toolkit.co.uk

Check your dashboard or contact Dave to confirm which server your domain is assigned to.

Quick start

The API lets you start a crawl for a domain and read the data a crawl produced. Here’s how to pull the broken links from the latest crawl.

BASE="https://eu1.website-toolkit.co.uk"
KEY="your-api-key"

# 1. What crawls exist for a domain?
curl -s -H "Authorization: Bearer $KEY" \
  "$BASE/api/v1/example.com"

# 2. Pull the issues from the most recent crawl
curl -s -H "Authorization: Bearer $KEY" \
  "$BASE/api/v1/example.com/latest/issues"

# 3. Pull just the broken links
curl -s -H "Authorization: Bearer $KEY" \
  "$BASE/api/v1/example.com/latest/broken-links"

Authentication

Every request must send an API key as a Bearer token in the Authorization header.

Authorization: Bearer your-api-key

Tenant Isolation

A key only sees crawls owned by its own tenant. Cross-tenant access returns a 404 Not Found so the API never reveals whether data exists.

Trigger a crawl

Start a full crawl of a domain — ideal for GitHub Actions or deploy webhooks. Use GET or POST.

POST/api/crawl-site/{domain}— recommended
GET/api/crawl-site/{token}/{domain}— for plain webhooks
# Recommended header auth
curl -s -X POST -H "Authorization: Bearer $KEY" \
  "$BASE/api/crawl-site/example.com"

GitHub Actions Example

name: Re-crawl on deploy
on:
  push:
    branches: [main]
jobs:
  trigger-crawl:
    runs-on: ubuntu-latest
    steps:
      - run: |
          curl -sS -X POST \
            -H "Authorization: Bearer ${{ secrets.CRAWLER_KEY }}" \
            --fail-with-body \
            "https://eu1.website-toolkit.co.uk/api/crawl-site/example.com"

Agent Data API

The primary read-only API for pulling audit data. Keyed by an agency API key from the admin console.

Discovery

GET/api/v1/{domain}

Lists recent crawls and available data types for a domain.

Data Feed

GET/api/v1/{domain}/{crawlId|latest}/{type}

Pagination: ?page=1&pageSize=100 (max 1000).

TypeDescription
pagesEvery crawled page: status, title, H1, load time, size, redirect target, SEO issues.
issuesDetected problems: type, the page it was found on, a recommendation, destination.
linksEvery link checked: link URL, owning page, link text, exists flag.
imagesEvery image: image URL, page, alt text, size, exists flag.
cssEvery stylesheet: URL, page, size, exists flag.
scriptsEvery script: URL, page, location, size, exists flag.
broken-linksOnly the links whose target failed.
broken-imagesOnly the images that failed to load.
broken-cssOnly the stylesheets that failed.
broken-scriptsOnly the scripts that failed.
wordsSite-level content-integrity word list (not crawl-scoped).
summaryThe crawl’s headline metrics (single object, not paginated).

Column reference

pages   : content_status, content_url, page_title, heading_1, load_time_ms,
          size_kb, page_redirects_to, seo_issues
issues  : id, issue_type, found_on_url, recommendation, destination, screenshot_path
links   : link_exists, link_url, owning_page_url, link_text, link_advisory, screenshot_path
images  : image_exists, image_url, image_page_url, image_alt, size_kb
css     : css_exists, css_url, css_page_url, size_kb
scripts : script_exists, script_url, script_page_url, script_location, size_kb

Client API & MCP

A read/write API keyed by a client's own token. Perfect for AI assistants that need to read audits, mark issues fixed, and re-check URLs.

GET/api/client-api/v1/domains— list domains
GET/api/client-api/v1/{domain}/crawls— recent audits
GET/api/client-api/v1/{domain}/{crawlId|latest}/{type}— read results
POST/api/client-api/v1/{domain}/audit— start audit
POST/api/client-api/v1/{domain}/{crawlId}/issues/resolve— mark fixed
GET/api/client-api/v1/{domain}/broken-urls— get URLs for re-check
POST/api/client-api/v1/{domain}/{crawlId}/page-status— report re-check results
GET/api/client-api/v1/{domain}/sitemap-check— check sitemap

MCP Protocol

Exposed over POST /mcp for Claude Code and Claude Desktop.

claude mcp add --transport http wsc-crawler \
  https://eu1.website-toolkit.co.uk/mcp \
  --header "Authorization: Bearer $TOKEN"

Monitoring

GET/api/client-api/v1/{domain}/monitoring— status
POST/api/client-api/v1/{domain}/monitoring— request
POST/api/client-api/v1/{domain}/monitoring/cancel— cancel

Client Feedback

Manage feedback items left via the widget.

GET/api/client-api/v1/feedback— overview
GET/api/client-api/v1/{domain}/feedback— list items
POST/api/client-api/v1/{domain}/feedback/{itemId}/status— set status
POST/api/client-api/v1/{domain}/feedback/{itemId}/note— add note

Launch & Migration

Manage checklists and redirect maps.

Checklists

GET/api/client-api/v1/{domain}/projects— list projects
POST/api/client-api/v1/{domain}/projects— start project
GET/api/client-api/v1/{domain}/projects/{type}/tasks— list tasks
POST/api/client-api/v1/{domain}/projects/{type}/tasks/{taskKey}/state— set state
POST/api/client-api/v1/{domain}/projects/{type}/recheck— re-run auto-check

Redirects

GET/api/client-api/v1/{domain}/redirects— list redirects
POST/api/client-api/v1/{domain}/redirects/import— import from crawl
POST/api/client-api/v1/{domain}/redirects/check— check all
POST/api/client-api/v1/{domain}/redirects/{rowId}— update row

Time Tracking

Log and report time entries against projects.

GET/api/client-api/v1/time/report— pull report
POST/api/client-api/v1/time/entries— log entry

Usage & Account

GET/api/client-api/v1/account— usage & limits
POST/api/client-api/v1/domains— add domain

Response shapes

These are the payloads returned by the Client API, transcribed field-for-field from the query columns each operation selects.

GET /account

{
  "budget": { 
    "budget": 20000, "used": 4300, "reserved": 0, "remaining": 15700,
    "windowStart": "2026-06-15T00:00:00.000Z", "resetsAt": "2026-06-22T00:00:00.000Z" 
  },
  "residentialProxy": { "cap": 5000, "used": 120, "remaining": 4880 },
  "monitoring": { "slotsTotal": 10, "slotsUsed": 3, "slotsRemaining": 7 }
}

GET /{domain}/crawls

{
  "domain": "example.com",
  "crawls": [
    { 
      "crawlId": "8f1c2e4a-...", "startTime": "...", "endTime": "...", "status": "Completed",
      "mode": "advanced", "pages": 84, "brokenPages": 1, "brokenLinks": 3,
      "brokenImages": 0, "brokenCss": 0, "brokenJs": 0, "warnings": 12 
    }
  ]
}

GET /{domain}/monitoring

{
  "domain": "example.com", "monitored": true, "pending": false,
  "schedule": "0 6 * * 1", "lastCheckedAt": "2026-06-20T06:00:00.000Z",
  "lastCrawlId": "8f1c2e4a-...", "maxPages": 200
}

GET /{domain}/feedback (one item)

{
  "id": "...", "pageUrl": "https://example.com/pricing", "pagePath": "/pricing",
  "note": "typo in the second bullet", "status": "open",
  "viewport": "1440x900", "createdAt": "...",
  "attachments": [ 
    { "id": "...", "kind": "screenshot", "mime": "image/webp", "bytes": 48213,
      "url": "https://.../api/project/{editToken}/feedback/{attachmentId}" } 
  ],
  "targetSelector": ".hero h1", "category": "content", "hidden": false,
  "internalNote": null, "noteSentAt": null
}

GET /{domain}/projects/{type}/tasks (one task)

{
  "key": "dns-live", "type": "Pre", "category": "DNS", "group": "Go-live",
  "task": "Point DNS at the new host", "auto": true, "custom": false,
  "edited": false, "state": "open", "source": null, "hidden": false
}

GET /{domain}/redirects (one row)

{
  "id": "...", "oldUrl": "https://old.example.com/about-us",
  "expectedNewUrl": "https://new.example.com/about",
  "lastStatus": 301, "resolvedTo": "https://new.example.com/about",
  "result": "ok_301", "hidden": false
}

Examples

Node.js — Pull every issue across all pages

async function fetchAll(domain, crawlId, type) {
  const out = [];
  for (let page = 1; ; page++) {
    const r = await fetch(`${BASE}/api/v1/${domain}/${crawlId}/${type}?page=${page}&pageSize=500`, { headers });
    const { rows, total, pageSize } = await r.json();
    out.push(...rows);
    if (page * pageSize >= total) break;
  }
  return out;
}

const issues = await fetchAll('example.com', 'latest', 'issues');

Error responses

StatusMeaning
400Invalid domain, unknown type, or bad request params.
401Missing or unrecognised API key / token.
403Domain not on account or disallowed origin.
404No data found or cross-tenant access.
429Rate limit exceeded.
500Internal server error.

Need help or a new feature?

Need an API key for the beta service, or want a request type that isn’t here yet? We’d love to hear what you’re building.

Email Support