Trigger crawls & pull data out as JSON.
Kick off a fresh crawl when you deploy, then read every audit’s data — pages, issues, broken links, images, scripts and CSS — through a simple HTTP API. Point a script or an AI agent at the results, and fix issues automatically.
https://eu1.website-toolkit.co.ukEU2 (Live): https://eu2.website-toolkit.co.ukCheck your dashboard or contact Dave to confirm which server your domain is assigned to.
Base URLs
EU1 (Dev/Test)
https://eu1.website-toolkit.co.uk
EU2 (Live Customers)
https://eu2.website-toolkit.co.uk
Quick start
The API lets you start a crawl for a domain and read the data a crawl produced. Here’s how to pull the broken links from the latest crawl.
BASE="https://eu1.website-toolkit.co.uk"
KEY="your-api-key"
# 1. What crawls exist for a domain?
curl -s -H "Authorization: Bearer $KEY" \
"$BASE/api/v1/example.com"
# 2. Pull the issues from the most recent crawl
curl -s -H "Authorization: Bearer $KEY" \
"$BASE/api/v1/example.com/latest/issues"
# 3. Pull just the broken links
curl -s -H "Authorization: Bearer $KEY" \
"$BASE/api/v1/example.com/latest/broken-links"
Authentication
Every request must send an API key as a Bearer token in the Authorization header.
Authorization: Bearer your-api-key
🔒 Tenant Isolation
A key only sees crawls owned by its own tenant. Cross-tenant access returns a 404 Not Found so the API never reveals whether data exists.
Trigger a crawl
Start a full crawl of a domain — ideal for GitHub Actions or deploy webhooks. Use GET or POST.
/api/crawl-site/{domain} — recommended/api/crawl-site/{token}/{domain} — for plain webhooks# Recommended header auth
curl -s -X POST -H "Authorization: Bearer $KEY" \
"$BASE/api/crawl-site/example.com"
GitHub Actions Example
name: Re-crawl on deploy
on:
push:
branches: [main]
jobs:
trigger-crawl:
runs-on: ubuntu-latest
steps:
- run: |
curl -sS -X POST \
-H "Authorization: Bearer ${{ secrets.CRAWLER_KEY }}" \
--fail-with-body \
"https://eu1.website-toolkit.co.uk/api/crawl-site/example.com"
Agent Data API
The primary read-only API for pulling audit data. Keyed by an agency API key from the admin console.
Discovery
/api/v1/{domain}Lists recent crawls and available data types for a domain.
Data Feed
/api/v1/{domain}/{crawlId|latest}/{type}Pagination: ?page=1&pageSize=100 (max 1000).
| Type | Description |
|---|---|
pages | Every crawled page: status, title, H1, load time, size, redirect target, SEO issues. |
issues | Detected problems: type, the page it was found on, a recommendation, destination. |
links | Every link checked: link URL, owning page, link text, exists flag. |
images | Every image: image URL, page, alt text, size, exists flag. |
css | Every stylesheet: URL, page, size, exists flag. |
scripts | Every script: URL, page, location, size, exists flag. |
broken-links | Only the links whose target failed. |
broken-images | Only the images that failed to load. |
broken-css | Only the stylesheets that failed. |
broken-scripts | Only the scripts that failed. |
words | Site-level content-integrity word list (not crawl-scoped). |
summary | The crawl’s headline metrics (single object, not paginated). |
Column reference
pages : content_status, content_url, page_title, heading_1, load_time_ms,
size_kb, page_redirects_to, seo_issues
issues : id, issue_type, found_on_url, recommendation, destination, screenshot_path
links : link_exists, link_url, owning_page_url, link_text, link_advisory, screenshot_path
images : image_exists, image_url, image_page_url, image_alt, size_kb
css : css_exists, css_url, css_page_url, size_kb
scripts : script_exists, script_url, script_page_url, script_location, size_kb
Client API & MCP
A read/write API keyed by a client’s own token. Perfect for AI assistants that need to read audits, mark issues fixed, and re-check URLs.
/api/client-api/v1/domains — list domains/api/client-api/v1/{domain}/crawls — recent audits/api/client-api/v1/{domain}/{crawlId|latest}/{type} — read results/api/client-api/v1/{domain}/{crawlId}/issues/resolve — mark fixed/api/client-api/v1/{domain}/{crawlId}/issues/unresolve — undo mark fixed/api/client-api/v1/{domain}/broken-urls — get URLs for re-check/api/client-api/v1/{domain}/{crawlId}/page-status — report re-check results/api/client-api/v1/{domain}/sitemap-check — check sitemapMark fixed / undo take { "issueId": 42 }. An issueId is only valid within its own crawl, so always pass the crawlId it came from — an id from another crawl returns 404.
MCP Protocol
Exposed over POST /mcp for Claude Code and Claude Desktop. The start_audit tool accepts the same optional checksOff list as the REST endpoint.
claude mcp add --transport http wsc-crawler \
https://eu1.website-toolkit.co.uk/mcp \
--header "Authorization: Bearer $TOKEN"
Start an audit
/api/client-api/v1/{domain}/audit — { "mode": "basic" | "advanced", "checksOff"?: [...] }mode is basic or advanced — advanced is the full audit, the same one the portal’s Run full audit button starts. Any other value returns 400.
The call returns straight away with the new crawlId; the audit runs in the background and can take minutes to hours. Poll GET /{domain}/crawls for its status rather than waiting on the response.
No need to run a batch one at a time: queue every audit, then poll /crawls — they run in parallel as capacity allows. Limited to 30 audits per 15 minutes per token.
Switching checks off (optional)
checksOff switches off optional report checks for this one audit, on top of whatever the domain’s saved Report Settings in the client portal already switch off. The saved settings aren’t changed. Valid keys:
| Key | Leaves out |
|---|---|
large_image | Oversized images |
slow_page | Slow pages |
not_found_page | 404 page check |
orphaned_page | Orphaned pages |
redirect_chain | Redirect chains |
redirected_link | Links that redirect |
missing_redirect | Old pages with no redirect |
seo | All SEO issues |
seo_length | Title / description length only |
social_image | Social share images |
duplicate_copy | Duplicate copy |
dns_change | DNS changes |
open_graph | Open Graph tags |
structured_data | Structured data |
llms_txt | llms.txt |
ai_freshness | AI freshness signals |
ai_training_blocked | AI training crawlers blocked |
ai_optout | AI opt-out signals |
An unknown key returns 400 listing the valid ones, rather than silently leaving the check on. Broken pages, broken resources and security findings can’t be switched off. The response echoes the checksOff it applied.
# Start a full audit
curl -s -X POST -H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"mode": "advanced"}' \
"$BASE/api/client-api/v1/example.com/audit"
# Start one without image-size and SEO-length findings
curl -s -X POST -H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"mode": "advanced", "checksOff": ["large_image", "seo_length"]}' \
"$BASE/api/client-api/v1/example.com/audit"
Download the report PDF
/api/client-api/v1/{domain}/{crawlId|latest}/report.pdfReturns the crawl-report email as an A4 PDF, without its buttons and with the homepage thumbnail included. It’s a copy of the email as sent, including the SSL, domain expiry and PageSpeed figures. Audits finished before this endpoint existed are rebuilt from the stored results instead, and show those three as N/A.
An audit that hasn’t finished returns 409 — poll /crawls until it has. Each PDF is rendered on demand, so downloads have their own limit of 60 per 15 minutes per token. REST only; there’s no MCP equivalent, as MCP results are JSON.
# Download the latest finished audit's report
curl -s -H "Authorization: Bearer $TOKEN" -o example.com.pdf \
"$BASE/api/client-api/v1/example.com/latest/report.pdf"
Monitoring
/api/client-api/v1/{domain}/monitoring — status/api/client-api/v1/{domain}/monitoring — request/api/client-api/v1/{domain}/monitoring/cancel — cancelClient Feedback
Manage feedback items left via the widget.
/api/client-api/v1/feedback — overview/api/client-api/v1/{domain}/feedback — list items/api/client-api/v1/{domain}/feedback/{itemId}/status — set status/api/client-api/v1/{domain}/feedback/{itemId}/note — add noteLaunch & Migration
Manage checklists and redirect maps.
Checklists
/api/client-api/v1/{domain}/projects — list projects/api/client-api/v1/{domain}/projects — start project/api/client-api/v1/{domain}/projects/{type}/tasks — list tasks/api/client-api/v1/{domain}/projects/{type}/tasks/{taskKey}/state — set state/api/client-api/v1/{domain}/projects/{type}/recheck — re-run auto-checkRedirects
/api/client-api/v1/{domain}/redirects — list redirects/api/client-api/v1/{domain}/redirects/import — import from crawl/api/client-api/v1/{domain}/redirects/check — check all/api/client-api/v1/{domain}/redirects/{rowId} — update rowTime Tracking
Log and report time entries against projects.
/api/client-api/v1/time/report — pull report/api/client-api/v1/time/entries — log entryUsage & Account
/api/client-api/v1/account — usage & limits/api/client-api/v1/domains — add domainResponse shapes
These are the payloads returned by the Client API, transcribed field-for-field from the query columns each operation selects.
GET /account
{
"budget": {
"budget": 20000, "used": 4300, "reserved": 0, "remaining": 15700,
"windowStart": "2026-06-15T00:00:00.000Z", "resetsAt": "2026-06-22T00:00:00.000Z"
},
"residentialProxy": { "cap": 5000, "used": 120, "remaining": 4880 },
"monitoring": { "slotsTotal": 10, "slotsUsed": 3, "slotsRemaining": 7 }
}
GET /{domain}/crawls
{
"domain": "example.com",
"crawls": [
{
"crawlId": "8f1c2e4a-...", "startTime": "...", "endTime": "...", "status": "Completed",
"mode": "advanced", "pages": 84, "brokenPages": 1, "brokenLinks": 3,
"brokenImages": 0, "brokenCss": 0, "brokenJs": 0, "warnings": 12
}
]
}
GET /{domain}/monitoring
{
"domain": "example.com", "monitored": true, "pending": false,
"schedule": "0 6 * * 1", "lastCheckedAt": "2026-06-20T06:00:00.000Z",
"lastCrawlId": "8f1c2e4a-...", "maxPages": 200
}
GET /{domain}/feedback (one item)
{
"id": "...", "pageUrl": "https://example.com/pricing", "pagePath": "/pricing",
"note": "typo in the second bullet", "status": "open",
"viewport": "1440x900", "createdAt": "...",
"attachments": [
{ "id": "...", "kind": "screenshot", "mime": "image/webp", "bytes": 48213,
"url": "https://.../api/project/{editToken}/feedback/{attachmentId}" }
],
"targetSelector": ".hero h1", "category": "content", "hidden": false,
"internalNote": null, "noteSentAt": null
}
GET /{domain}/projects/{type}/tasks (one task)
{
"key": "dns-live", "type": "Pre", "category": "DNS", "group": "Go-live",
"task": "Point DNS at the new host", "auto": true, "custom": false,
"edited": false, "state": "open", "source": null, "hidden": false
}
GET /{domain}/redirects (one row)
{
"id": "...", "oldUrl": "https://old.example.com/about-us",
"expectedNewUrl": "https://new.example.com/about",
"lastStatus": 301, "resolvedTo": "https://new.example.com/about",
"result": "ok_301", "hidden": false
}
Examples
Node.js — Pull every issue across all pages
async function fetchAll(domain, crawlId, type) {
const out = [];
for (let page = 1; ; page++) {
const r = await fetch(`${BASE}/api/v1/${domain}/${crawlId}/${type}?page=${page}&pageSize=500`, { headers });
const { rows, total, pageSize } = await r.json();
out.push(...rows);
if (page * pageSize >= total) break;
}
return out;
}
const issues = await fetchAll('example.com', 'latest', 'issues');
Bash — Audit a batch of domains, then download each report
# Queue every audit up front — they run in parallel
for d in example.com example.org; do
curl -s -X POST -H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" -d '{"mode": "advanced"}' \
"$BASE/api/client-api/v1/$d/audit"
done
# Later, once /crawls shows them finished
for d in example.com example.org; do
curl -s -H "Authorization: Bearer $TOKEN" -o "$d.pdf" \
"$BASE/api/client-api/v1/$d/latest/report.pdf"
done
Error responses
| Status | Meaning |
|---|---|
400 | Invalid domain, unknown type, bad request params, or an unknown mode / checksOff key. |
401 | Missing or unrecognised API key / token. |
403 | Domain not on account or disallowed origin. |
404 | No data found, cross-tenant access, or an issueId from a different crawl. |
409 | Audit not finished yet (report PDF) — poll /crawls and retry. |
429 | Rate limit exceeded (e.g. 30 audits or 60 PDF downloads per 15 minutes per token). |
500 | Internal server error. |
Need help or a new feature?
Need an API key for the beta service, or want a request type that isn’t here yet? We’d love to hear what you’re building.