> ## Documentation Index
> Fetch the complete documentation index at: https://docs.scrapeunblocker.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Changelog
> Notable changes to the ScrapeUnblocker API.
This site now lives at **docs.scrapeunblocker.com**. Every page keeps its
path: links to `developers.scrapeunblocker.com` redirect permanently (301) to
the same page here, so old bookmarks, SDK READMEs and search results keep
working. The [API Reference](/api-reference/introduction) is part of this
site, and the raw OpenAPI 3.1 file is served at
[`/api-reference/openapi.json`](/api-reference/openapi.json) and, as before
on this hostname, at [`/openapi.json`](/openapi.json).
ChatGPT can now use the hosted [MCP server](/sdks/mcp) as a custom connector -
there's a dedicated, screenshot-by-screenshot [ChatGPT setup guide](/sdks/chatgpt).
It connects with your API key in the URL (No Auth) from **chatgpt.com** (web,
not the desktop app), under **Settings → Plugins → Developer mode**.
The MCP server exposes **four** tools: `fetch_html`, `fetch_parsed`,
`google_search` and `list_elements`.
**[Meta Ad Library plugin](/plugins/meta-ad-library).** A new endpoint, `POST
/ads/meta-ad-library`, returns a Facebook/Instagram advertiser's ads from Meta's
public Ad Library as structured JSON: `ad_text`, `title`, `cta_text`/`cta_type`,
`display_format` (image/video/DCO/carousel), `creatives` (image and video URLs),
`link_url` (the landing page), `platforms`, and `started_running`/`ended` dates.
Give it an advertiser - a Page name or a numeric Page ID - and a `country`;
filter with `active_status` and `media_type`, and bound the result with
`max_ads`. Unlike Meta's official Ad Library API - which returns full data only
for political and social-issue ads - this reads the same public library anyone
can open, so it covers ordinary commercial advertisers. A competitor-ads tool.
Available in the [Python](/sdks/python), [Node.js](/sdks/nodejs),
[PHP](/sdks/php) and [Ruby](/sdks/ruby) clients.
**Drive a real browser after the page loads.**
[`getPageSource`](/guides/page-source) takes two new parameters, documented in
[browser steps and element discovery](/guides/browser-steps):
* **`steps`** - a JSON array of actions the API runs in a real browser once the
page has loaded, before returning the resulting HTML: `wait_for`,
`wait_for_text`, `wait`, `click`, `type` (human-like keystrokes), `select`,
`press_key` and `scroll`. Fill a search form and read the results, click a
"load more" button, scroll an infinite list. Bounded to \~10 steps and a
\~30-second total budget; a failed step returns **HTTP 422** with `step_index`,
`action`, `reason`, `selector` and the page `html` at the point of failure.
Steps have side effects and are not idempotent - treat the call like a POST.
* **`list_elements=true`** - the discovery half. Returns the page's interactive
elements as JSON (`{ url, count, elements }`) instead of HTML, each element
carrying `tag`, `text`, `id`, `name`, `type`, `placeholder`, `aria_label`,
`href` and a ready-to-use `selector` you can drop straight into a `steps`
action. List the elements, build the steps, run them - a clean loop for AI
agents. Read-only and idempotent.
Both land in the official [Python](/sdks/python), [Node.js](/sdks/nodejs),
[PHP](/sdks/php) and [Ruby](/sdks/ruby) clients: pass a native array/list for
`steps` and a boolean for `list_elements`.
**[Amazon plugin](/plugins/amazon).** Two new endpoints: `POST
/marketplace/amazon-product` returns one product by ASIN or URL (title, brand,
**price + currency**, list price and savings, availability, rating, review
count, seller, feature bullets, categories and images), and `POST
/marketplace/amazon-search` returns a keyword search's result cards (ASIN,
title, price, rating, review count, clean product URL, sponsored / Prime
flags) on any of 20 regional marketplaces.
Prices come back in the **right currency automatically**: `proxy_country`
defaults to the marketplace's home country (`amazon.com` -> US, `amazon.de` ->
DE, ...) and the request is routed through our Oxylabs ISP + iProyal ISP pool
in that country, so `amazon.com` returns USD and `amazon.de` returns EUR with
no configuration. No Selling Partner / Product Advertising API account needed.
**API v3 - Google SERP overhaul.** [`/serpApi`](/guides/serp-scraping) is now
served by our own **su\_browser** engine over the ISP proxy pool. Callers use
it exactly as before, with two user-facing changes:
* **`proxy_country` falls back instead of erroring.** It is matched against the
countries we can serve (`BE`, `CA`, `CH`, `CN`, `DE`, `DK`, `ES`, `FR`, `GB`,
`HK`, `HR`, `IE`, `IT`, `JP`, `KR`, `LT`, `NL`, `NO`, `PL`, `RO`, `SE`, `SG`,
`TW`, `US`). Ask for a country we do not have an exit for and the request
still succeeds from an available exit, with two extra response fields:
`requestedCountry` (the ISO2 you asked for) and `note` (a human-readable
explanation of the fallback).
* **[`getPageSource`](/guides/page-source) now redirects Google search URLs.** A
`google./search` path, or a `google.` homepage with a `q=` query,
returns **HTTP 400** pointing you to `POST /serpApi?keyword=`. Other
Google URLs (Maps, etc.) are unaffected.
**[Temu search](/plugins/temu#search) takes a `sort`.** `POST
/goods/temu-search?keyword=...&sort=...` now orders results using Temu's own
sort: `relevance` (default), `best_selling`, `recent`, `price_asc` (price low
to high) or `price_desc` (price high to low). It is server-side, so the whole
result set comes back sorted, not just a re-order of the page. The chosen
`sort` is echoed in the response.
**[Temu product](/plugins/temu#product) returns more fields** - all from the
same document, at no extra request. `POST /goods/temu-product` now also
includes:
* **`reviews`** - per-review `rating` and `date`. Temu masks the review author
and text in the crawler document (they come back as `****`); the rating and
date are real.
* **`video`** - `{ contentUrl, thumbnailUrl, uploadDate }` when the product
ships a video, else `null`.
* **`variantGroupId`** - the Temu product-group ID that ties variants of the
same item together.
* **`itemCondition`** - schema.org condition URL, e.g. `.../NewCondition`.
All additive - existing fields are unchanged. The image list still carries the
1-2 images Temu puts in the crawler document, not the full gallery.
**Spider Cloud has a Python client and a command line.** Everything the
dashboard does - deploy a project, run and watch jobs, schedule them, pull the
scraped items - now works from a script.
```bash theme={null}
pip install scrapeunblocker-cloud
```
* **`su-cloud` CLI:** `deploy`, `run`, `jobs`, `logs`, `items`, `download`,
`stats`, `schedule-add`, `destination-add` and more.
* **`SpiderCloudClient` library:** projects, deploys, jobs, schedules and
destinations, plus `job.items()` to stream data by cursor and
`job.download()` for a full `.jsonl.gz` export.
* Authenticates with a per-account **API token**, created in the dashboard
under **Spider Cloud → API tokens**. Pure standard library, no runtime
dependencies. See [SDK & CLI](/spider-cloud/sdk).
**[Temu search](/plugins/temu#search) is faster and wider.** `POST
/goods/temu-search` now returns about **40 product URLs per keyword** in \~2s
(up from \~14, and no third-party search in the loop). Search is discovery only
* fetch each URL's data through `POST /goods/temu-product`. The `details`
parameter has been removed.
**New plugin: [Temu](/plugins/temu).** Two endpoints for Temu, which never
serves product data to a browser (the page hydrates through a call that refuses
automation). The plugin reads the `ld+json` document Temu serves to search
crawlers instead - the only view that carries the real fields.
* **`POST /goods/temu-product?url=...`** returns a single product as JSON:
`name`, `description`, `brand`, `price`, `priceCurrency`, `availability`,
`rating`, `reviewCount` and `images`.
* **`POST /goods/temu-search?keyword=...`** returns product URLs taken straight
from Temu's own search page (no third-party search in the loop). With
`details=true` it also resolves the top results to full product objects.
Temu meters the crawler view, so the plugin is built for **targeted** use
(price checks, monitoring, comparison), enforces its own fleet-wide rate limit,
and answers `429` when the budget is spent. Price currency follows the exit
region. See [the Temu plugin page](/plugins/temu).
**ScrapeUnblocker installs into [Claude Code](https://claude.com/claude-code)
with two commands.**
```
/plugin marketplace add ScrapeUnblocker/claude-code-plugin
/plugin install scrapeunblocker@scrapeunblocker
```
You get the three MCP tools (`fetch_html`, `fetch_parsed`, `google_search`), a
`/scrape-url` command, and reference skills Claude pulls in while writing
scraping code for your project. Your API key is prompted for once when the
plugin is enabled and stored in your operating system's keychain - no
environment variable to set, no config file to edit. See
[Claude Code Plugin](/sdks/claude-code).
Five fixes, all from one round of outside testing.
**`parsed_data=true` now works on TikTok and Instagram.** Both sites build
their pages in the browser, so there was almost nothing in the markup to read
and the response came back as little more than a page title - even though the
HTML we handed you carried every number. Dedicated extractors now read the
embedded payload, which also means **exact integers** instead of the rounded
prose the page paints: a TikTok video returns `plays`, `likes`, `comments`,
`shares` and `saves`; an Instagram post returns exact `likes` and `comments`,
the real `author.username` and a real `posted_at`. See
[Parsed data](/guides/parsed-data#social-pages-tiktok-and-instagram).
**Google Local returns `phone` and `website`.** The phone used to arrive glued
to the end of `hours`; it is now its own field and `hours` is clean. `website`
is new. Paid listings are flagged with `sponsored` and no longer report
"Sponsored" as the business name. See
[Google Local](/plugins/google-local).
**Skyscanner flights honour `market`, `locale` and `currency`.** A request for
`market=TR&locale=tr-TR¤cy=TRY` came back priced in GBP with en-GB deep
links; it now returns Turkish lira and Turkish place names.
**`/serpApi` no longer fails a query that has an answer.** A search whose
results page did not finish rendering was returned as a successful but empty
fetch, which surfaced as `502 No SERP results collected` on a query that
worked on the next call. That page is now discarded and retried. Separately, a
search where Google genuinely reports no matches - normal for a narrow `site:`
query - returns `200` with an empty result set instead of a `502`.
**`POST /marketplace/ebay-search`** turns any eBay search into structured
JSON on 19 regional marketplaces, from `ebay.com` to `ebay.com.au`. Each
listing comes back with title, numeric price and currency, condition (plus a
normalised `conditionCode`), seller username and feedback, shipping cost,
sold / watcher / bid counts, image and a clean item URL:
```bash theme={null}
curl -X POST "https://api.scrapeunblocker.com/marketplace/ebay-search?keyword=iphone%2013&condition=used&sort=newly_listed" \
-H "x-scrapeunblocker-key: YOUR_API_KEY"
```
Filter with `condition`, `listing_type`, `min_price`/`max_price`,
`free_shipping`, `seller` or `category`; sort with `sort`; page with
`page`/`page_size` (60, 120 or 240 per page).
When eBay finds no exact match it still serves a page of loosely-related
suggestions. Those are never passed off as results: the response sets
`exactMatches: false` and explains it in `notice`.
The plugin is live in the [dashboard](https://app.scrapeunblocker.com/dashboard/plugins),
in all four SDKs on **0.1.8**, and documented at [eBay Search](/plugins/ebay).
```bash theme={null}
pip install scrapeunblocker==0.1.8
npm install scrapeunblocker@0.1.8
gem install scrapeunblocker -v 0.1.8
composer require scrapeunblocker/client:0.1.8
```
**`/serpApi` now returns every paid unit on the page.** Google had started
serving part of its ad inventory from a wrapper inside the organic column
rather than the two containers we read, so those units were missing: a German
insurance query carried six ads and came back with four. Ads are now found
wherever they sit and sorted into `topAds` / `bottomAds` by their position
relative to the organic results.
**`organicResultsCount` was double the real number.** Two of the paths used to
collect organic results reached the same entries at different depths, so every
result was returned twice - positions 1-9 followed by an identical 10-18. The
list is now deduplicated, which means the count you get for an unchanged query
will be about half of what it was, and correct. Paid blocks that Google places
inside the organic column are excluded from `organic` as well.
**New field: `aiOverview`.** Google's AI-written summary above the results,
with the hostnames it cites, or `null` when the search has none:
```json theme={null}
"aiOverview": {
"text": "Web scraping is an automated method used to extract data...",
"sources": ["reddit.com", "en.wikipedia.org"]
}
```
It is read from the page we already fetched, never waited for, so it adds no
latency. See [SERP scraping](/guides/serp-scraping).
**Faster, too.** The extraction step used to wait a flat 1.2 s for ads that
measurement showed arrive within \~0.4 s, and Google's "unusual traffic"
challenge went undetected whenever it was served at the search URL instead of
redirecting - so a burnt exit was retried instead of replaced. Both fixed.
**`wait_after_load` works again.** The engine now serving SERP was ignoring it
entirely. It is also no longer something you should need: extraction waits for
the ads on its own, so use `wait_after_load` only when you deliberately want a
longer settle.
**[Spider Cloud](/spider-cloud/overview)** now has its own section: host your
own Scrapy project with us, deploy it straight from a git repository, run and
schedule spiders, and read the items back - with ScrapeUnblocker routing built
in, so a spider that gets blocked elsewhere needs no proxy or fingerprint code
here.
Eight pages, starting with
[Getting started](/spider-cloud/getting-started), which walks one repository
from an empty dashboard to a finished run with data in it. The one to read
before your first run is
[routing modes](/spider-cloud/routing) - the choice between `off`, `marked`
and `all` decides what a job costs:
```python theme={null}
# `marked` routes only what the spider asks for, and nothing else.
yield scrapy.Request(url, meta={"unblock": True})
```
`all` sends **every** request through the API, so a spider walking a hundred
thousand detail pages makes a hundred thousand paid calls. `marked` exists so
a producer can be unblocked without dragging its consumer onto the paid path.
Also covered: [projects and settings](/spider-cloud/projects) (including why
some projects cannot see environment variables at all),
[deploys and refs](/spider-cloud/deploys),
[jobs](/spider-cloud/jobs), [schedules](/spider-cloud/schedules) and
[delivery destinations](/spider-cloud/destinations).
The **Node.js, Ruby and PHP SDKs** now carry the same typed billing errors
released for Python earlier today, and all four are on **0.1.6** so their
version numbers line up from here on:
```bash theme={null}
pip install scrapeunblocker==0.1.6
npm install scrapeunblocker@0.1.6
gem install scrapeunblocker -v 0.1.6
composer require scrapeunblocker/client:0.1.6
```
Each one gains `PaymentRequired*` for `402` (with the quota, credit-limit and
failed-payment subclasses), the no-subscription flavour of `401`, and typed
errors for `404`, `408`, `415` and `422`. Nothing was removed, so existing
`APIError` / `ApiException` handlers keep working. See the
[Node.js](/sdks/nodejs#error-handling), [Ruby](/sdks/ruby#error-handling) and
[PHP](/sdks/php#error-handling) guides.
**`scrapeunblocker` 0.1.6** for Python raises a typed exception for every
documented status code. `402` previously arrived as a bare `APIError`; it now
raises `PaymentRequiredError`, with a subclass per billing block -
`QuotaExceededError`, `CreditLimitExceededError` and `PaymentFailedError` -
so each can be handled separately:
```python theme={null}
except QuotaExceededError: ... # plan quota used up for this period
except CreditLimitExceededError: ... # unpaid balance past the credit limit
except PaymentFailedError: ... # card declined three times
```
Also new: `NoSubscriptionError` (the `401` that means the key is fine but the
account has no plan), plus `NotFoundError`, `BrowserTimeoutError`,
`UnsupportedContentError` and `ValidationError`. Nothing was removed and every
new class derives from `APIError`, so existing handlers keep working.
See the [Python SDK guide](/sdks/python#error-handling).
The [Errors](/errors) page now documents **HTTP `402`**, which was missing from
the status table entirely. A `402` has three distinct plain-text bodies, each
with its own fix:
* `Quota exceeded` - the plan's requests for this billing period are used up
* `Credit limit exceeded` - the unpaid balance has grown past the account's
credit limit
* `Payment failed - update payment method` - a card payment for an open
invoice has been declined three times
All three clear on their own once the underlying billing state changes -
access returns within about a minute, with no key change needed. Note that
subscribing to a new plan does not clear a failed payment: the old unpaid
invoice has to be settled. See
[402 payment required](/errors#402-payment-required).
**HTTP `401`** is now documented properly too, with its two bodies -
`Unauthorized` (key not recognised) and `No valid subscription` (key is fine,
account has no active plan) - plus the neighbouring cases: omitting the
`x-scrapeunblocker-key` header entirely returns `400`, not `401`, and neither
`401` nor `402` counts against your quota. See
[401 unauthorized](/errors#401-unauthorized).
A new plugin returns **oopbuy.com product-search listings** as structured
JSON - title (English and Chinese), USD and CNY price, monthly sales count,
image and a product-detail link, in about a second:
```
POST /goods/oopbuy-search?keyword=hoodie&channel=1688&sort=best_selling
```
Pick the source marketplace with `channel` (`1688`, `taobao`, `official`),
order with `sort`, and page with `page` / `page_size`. Note that oopbuy
trademark-blocks brand-name keywords (nike, adidas) at its own backend -
those come back as a successful `200` with `keywordRejected: true` and an
empty `results` array (oopbuy's own genuine response), not an error.
Available in the
[dashboard](https://app.scrapeunblocker.com/dashboard/plugins), the
[API reference](/api-reference/introduction), and all four SDKs. See
[oopbuy Search plugin](/plugins/oopbuy-search).
A new plugin returns **Google Local (Maps) business listings** as structured
JSON - name, rating, review count, price level, category, address, opening
hours and a top review snippet:
```
POST /maps/google-local?keyword=coffee shops in chicago&proxy_country=US
```
It runs on our warm, anti-bot-cleared Google sessions, so you get the local
pack without a Google Places API key or billing project. Available in the
[dashboard](https://app.scrapeunblocker.com/dashboard/plugins), the
[API reference](/api-reference/introduction), and all four SDKs. See
[Google Local plugin](/plugins/google-local).
The three Skyscanner plugin endpoints are now part of the public OpenAPI spec,
so they show up in the [API Reference](/api-reference/introduction) and in any
OpenAPI-aware client (Postman, Insomnia, code generators):
```
POST /flights/skyscanner-quotes
POST /hotels/skyscanner-quotes
POST /carhire/skyscanner-quotes
```
Nothing changed about how they work - they were already documented and
available in all four SDKs. See [Skyscanner plugins](/plugins/skyscanner).
`getPageSource` no longer reports two cases as `403 Blocked by bot protection`
when no block was involved:
* A URL that serves an **image or other non-HTML resource** now returns
**`415`** naming the content type. For images, use `getImage`.
* A **Skyscanner flights / hotels / car-hire search URL** now returns **`400`**
pointing at the plugin that serves it, with the endpoint in the
`X-Recommended-Endpoint` header. Other Skyscanner pages are unaffected.
Also fixed: `getImage` returned `404 No image found` for every URL. It now
returns the PNG.
Added a **hosted** MCP connector so you can use ScrapeUnblocker in **claude.ai**
(web and mobile) with no local install - just add a custom connector with your key:
```
https://mcp.scrapeunblocker.com/mcp?key=YOUR_API_KEY
```
See the [MCP server guide](/sdks/mcp#hosted-connector-claude-ai-web-mobile).
Released the official **MCP server** so [Claude](https://claude.ai) (Desktop, Code, and the web app) and any other [Model Context Protocol](https://modelcontextprotocol.io) client can fetch any web page through ScrapeUnblocker, using your own API key:
```bash theme={null}
npx -y scrapeunblocker-mcp
```
Three tools: `fetch_html`, `fetch_parsed`, and `google_search`. See the [MCP server guide](/sdks/mcp).
Released official client libraries for four languages, each wrapping `getPageSource` (HTML / parsed JSON / cookies), `serpApi`, `getImage`, and the Skyscanner plugins - with typed errors and automatic retries:
```bash theme={null}
pip install scrapeunblocker # Python
npm install scrapeunblocker # Node.js
gem install scrapeunblocker # Ruby
composer require scrapeunblocker/client # PHP
```
See the [Python](/sdks/python), [Node.js](/sdks/nodejs), [PHP](/sdks/php) and [Ruby](/sdks/ruby) guides.
Introduced **Plugins** - ready-made scrapers for specific sites that return clean, structured JSON. Available in the [dashboard](https://app.scrapeunblocker.com/dashboard/plugins) (no code) and via the API for volume.
First release covers three [Skyscanner plugins](/plugins/skyscanner):
* **Flights** - `POST /flights/skyscanner-quotes`
* **Hotels** - `POST /hotels/skyscanner-quotes`
* **Car Hire** - `POST /carhire/skyscanner-quotes`
All three share your existing API key and plan quota. See the [Plugins overview](/plugins/overview).
Public documentation launched at `developers.scrapeunblocker.com`.
Three endpoints documented:
* `POST /getPageSource`
* `POST /serpApi`
* `POST /getImage`
All other paths remain internal and are not part of the public API contract.