Website-Toolkit.co.uk | Useful Tools for Freelancers & Agencies

Quick start

The API lets you start a crawl for a domain and read the data a crawl produced. Here’s how to pull the broken links from the latest crawl.

BASE="https://eu1.website-toolkit.co.uk"
KEY="your-api-key"

# 1. What crawls exist for a domain?
curl -s -H "Authorization: Bearer $KEY" \
  "$BASE/api/v1/example.com"

# 2. Pull the issues from the most recent crawl
curl -s -H "Authorization: Bearer $KEY" \
  "$BASE/api/v1/example.com/latest/issues"

# 3. Pull just the broken links
curl -s -H "Authorization: Bearer $KEY" \
  "$BASE/api/v1/example.com/latest/broken-links"

Authentication

Every request must send an API key as a Bearer token in the Authorization header.

Authorization: Bearer your-api-key

Tenant Isolation

A key only sees crawls owned by its own tenant. Cross-tenant access returns a 404 Not Found so the API never reveals whether data exists.

Trigger a crawl

Start a full crawl of a domain — ideal for GitHub Actions or deploy webhooks. Use GET or POST.

POST/api/crawl-site/{domain}— recommended

GET/api/crawl-site/{token}/{domain}— for plain webhooks

# Recommended header auth
curl -s -X POST -H "Authorization: Bearer $KEY" \
  "$BASE/api/crawl-site/example.com"

GitHub Actions Example

name: Re-crawl on deploy
on:
  push:
    branches: [main]
jobs:
  trigger-crawl:
    runs-on: ubuntu-latest
    steps:
      - run: |
          curl -sS -X POST \
            -H "Authorization: Bearer ${{ secrets.CRAWLER_KEY }}" \
            --fail-with-body \
            "https://eu1.website-toolkit.co.uk/api/crawl-site/example.com"

Agent Data API

The primary read-only API for pulling audit data. Keyed by an agency API key from the admin console.

Discovery

GET/api/v1/{domain}

Lists recent crawls and available data types for a domain.

Data Feed

GET/api/v1/{domain}/{crawlId|latest}/{type}

Pagination: ?page=1&pageSize=100 (max 1000).

Type	Description
`pages`	Every crawled page: status, title, H1, load time, size, redirect target, SEO issues.
`issues`	Detected problems: type, the page it was found on, a recommendation, destination.
`links`	Every link checked: link URL, owning page, link text, exists flag.
`images`	Every image: image URL, page, alt text, size, exists flag.
`css`	Every stylesheet: URL, page, size, exists flag.
`scripts`	Every script: URL, page, location, size, exists flag.
`broken-links`	Only the links whose target failed.
`broken-images`	Only the images that failed to load.
`broken-css`	Only the stylesheets that failed.
`broken-scripts`	Only the scripts that failed.
`words`	Site-level content-integrity word list (not crawl-scoped).
`summary`	The crawl’s headline metrics (single object, not paginated).

Column reference

pages   : content_status, content_url, page_title, heading_1, load_time_ms,
          size_kb, page_redirects_to, seo_issues
issues  : id, issue_type, found_on_url, recommendation, destination, screenshot_path
links   : link_exists, link_url, owning_page_url, link_text, link_advisory, screenshot_path
images  : image_exists, image_url, image_page_url, image_alt, size_kb
css     : css_exists, css_url, css_page_url, size_kb
scripts : script_exists, script_url, script_page_url, script_location, size_kb

Client API + MCP

A read/write API keyed by a client's own token. Perfect for AI assistants that need to read audits and mark issues fixed.

GET/api/client-api/v1/domains— list domains

POST/api/client-api/v1/{domain}/audit— start audit

POST/api/client-api/v1/{domain}/{crawlId}/issues/resolve— mark fixed

MCP Protocol

Exposed over POST /mcp for Claude Code and Claude Desktop.

claude mcp add --transport http wsc-crawler \
  https://eu1.website-toolkit.co.uk/mcp \
  --header "Authorization: Bearer $TOKEN"

Examples

Node.js — Pull every issue across all pages

async function fetchAll(domain, crawlId, type) {
  const out = [];
  for (let page = 1; ; page++) {
    const r = await fetch(`${BASE}/api/v1/${domain}/${crawlId}/${type}?page=${page}&pageSize=500`, { headers });
    const { rows, total, pageSize } = await r.json();
    out.push(...rows);
    if (page * pageSize >= total) break;
  }
  return out;
}

const issues = await fetchAll('example.com', 'latest', 'issues');

Error responses

Status	Meaning
`400`	Invalid domain, unknown type, or bad request params.
`401`	Missing or unrecognised API key / token.
`403`	Domain not on account or disallowed origin.
`404`	No data found or cross-tenant access.
`429`	Rate limit exceeded.
`500`	Internal server error.

Need help or a new feature?

Need an API key for the beta service, or want a request type that isn’t here yet? We’d love to hear what you’re building.

Email Support

Trigger crawls & pull data out as JSON.