ScrapeUnblocker Documentation

ScrapeUnblocker tries every available bypass route before giving up. When it does return a non-2xx, the status code tells you exactly what went wrong. This guide covers the playbook for each.

403 - blocked on every route

We tried every bypass route we have for this domain, and every one of them was blocked. Recovery:

Switch proxy_country. This is the highest-yield fix. Many sites apply country-specific bot rules.

curl -X POST "https://api.scrapeunblocker.com/getPageSource?url=...&proxy_country=de" \
  -H "x-scrapeunblocker-key: YOUR_API_KEY"

Wait a few minutes and retry. Rate-based blocks expire on their own.
Lower request volume. If you’re hitting the same domain hard, the block is probably your traffic pattern, not the IP.
Contact support. If the same URL 403s consistently, the domain may need a custom plugin. The help center is the right place.

403 is never an auth problem. Invalid API keys return 401. If you’re seeing 401, see Authentication.

503 - no capacity on our side

We had no API capacity for the request right now: all capacity was busy past the queue limit, or none was available (No server is available to handle this request), or the render path was unavailable (Render unavailable.). This is a problem on our side, not the target’s - and it is not billed. Recovery: wait and retry. Exponential backoff is appropriate:

import time
import requests

def get_with_backoff(url, key, max_attempts=5):
    for i in range(max_attempts):
        r = requests.post(
            "https://api.scrapeunblocker.com/getPageSource",
            params={"url": url},
            headers={"x-scrapeunblocker-key": key},
            timeout=125,
        )
        if r.status_code not in (500, 502, 503):
            return r
        time.sleep(min(60, 2 ** i))
    return r

500 (internal error) and 502 with an HTML body (our gateway lost the connection to the scraping backend) are the same class - ours, free, retry with backoff.

523 - the target is unreachable

The target site answered nobody over any of our exits: it is down, or it refuses connections in bursts. This is not a bot block, and it is not billed. A site that accepts the connection but is slow to answer is not a 523 - we keep waiting for it within the call’s time cap. Recovery: retry after a pause (tens of seconds to minutes). Rotating proxy_country rarely helps - the site itself is not answering.

Dead URLs - `X-Origin-Status`

If the target says the page does not exist, you get the target’s own 404 or 410 with its page as the body (it can be empty) and an X-Origin-Status header carrying that status. If its origin returns its own 5xx error page - or the target answers its own 401, 451 or an empty 500 - you get a 200 with that body and the same header. Either way we reached the site; the URL is dead. It is billed like any delivered page, and retrying will not change it - check the header and drop or fix the URL. A 404 without X-Origin-Status is ours (a wrong API path), not the target’s. A URL whose domain does not exist returns 400 Hostname cannot be resolved (also billed). See Errors.

504 - timeout

The request reached its time cap: Request timed out. on /getPageSource (60 s, 90 s with steps), or the SERP / plugin time budget on /serpApi and the dedicated plugin endpoints. Usually the target is slow for this URL. Recovery:

Retry. Transient network issues account for most 504s.
Pick a different proxy_country. Some country pools are slower or under heavier rotation.
On /serpApi, lower pages_to_check. Each additional page extends the time needed.

A 504 with an HTML body comes from our gateway - that one is on our side. If 504s hit many different domains at once, treat it like a 503.

422 - validation error

You sent a bad parameter. The response body pinpoints the field:

{
  "detail": [
    {
      "loc": ["query", "url"],
      "msg": "field required",
      "type": "value_error.missing"
    }
  ]
}

Recovery: fix the request. 422 is not retryable - retrying the same bad request will fail the same way.

Retry policy that works

For most production workloads, this policy handles these error classes correctly:

import time
import requests

RETRYABLE = {408, 500, 502, 503, 504, 523}

def scrape(url, key, max_attempts=4):
    last = None
    for attempt in range(max_attempts):
        r = requests.post(
            "https://api.scrapeunblocker.com/getPageSource",
            params={"url": url},
            headers={"x-scrapeunblocker-key": key},
            timeout=125,
        )
        if r.status_code == 200:
            return r
        last = r
        if r.status_code == 403 and attempt == 0:
            # one shot at rotating country
            r = requests.post(
                "https://api.scrapeunblocker.com/getPageSource",
                params={"url": url, "proxy_country": "us"},
                headers={"x-scrapeunblocker-key": key},
                timeout=125,
            )
            if r.status_code == 200:
                return r
            last = r
        if r.status_code in RETRYABLE:
            time.sleep(2 ** attempt)
            continue
        break
    last.raise_for_status()

This:

Retries 408/500/502/503/504/523 with exponential backoff (failed calls are free).
Tries one country rotation on 403.
Fails fast on 401, 422, and other terminal errors.

Get Started

No-code area

Spider Cloud

Plugins

AI Assistants

Guides

Code Examples

Handling failures

403 - blocked on every route

503 - no capacity on our side

523 - the target is unreachable

Dead URLs - `X-Origin-Status`

504 - timeout

422 - validation error

Retry policy that works

​403 - blocked on every route

​503 - no capacity on our side

​523 - the target is unreachable

​Dead URLs - X-Origin-Status

​504 - timeout

​422 - validation error

​Retry policy that works

403 - blocked on every route

503 - no capacity on our side

523 - the target is unreachable

Dead URLs - `X-Origin-Status`

504 - timeout

422 - validation error

Retry policy that works