ScrapeUnblocker Documentation

The official Ruby client wraps the ScrapeUnblocker API in a small, dependency-free gem (standard library only) with typed errors and automatic retries.

gem install scrapeunblocker

Or in a Gemfile: gem "scrapeunblocker". Requires Ruby 2.7+. Store your API key in an environment variable so the client picks it up automatically:

export SCRAPEUNBLOCKER_KEY="YOUR_API_KEY"

Quickstart

require "scrapeunblocker"

su = ScrapeUnblocker::Client.new # reads SCRAPEUNBLOCKER_KEY, or Client.new(api_key: "YOUR_API_KEY")

html = su.get_page_source("https://example.com")

Get Markdown for LLMs

get_markdown (0.13.0+) returns the page as clean Markdown for LLMs, agents and RAG pipelines: headings, lists, tables, absolute links and images with alt text, without scripts, styles, navigation or footer. By default you get the main content only. It takes the same options as get_page_source except list_elements, and is billed the same.

md = su.get_markdown("https://en.wikipedia.org/wiki/Web_scraping")

# The whole page, not only the main content
full = su.get_markdown(url, main_content_only: false)

See Markdown for LLMs for what is kept and what is dropped.

Get parsed JSON instead of HTML

product = su.get_parsed("https://www.amazon.com/dp/B08N5WRWNW")

puts product.page_type # "product"
p product.data         # the fields

# If a parse ever comes back wrong, force a fresh set of rules:
fresh = su.get_parsed(url, refresh_rules: true, rules_hint: "price is missing")

When the page loads but nothing can be extracted from it, get_parsed does not raise: it returns a result with page type unknown and empty data, together with the rendered HTML (the API’s data_extracted: false and html fields). The call is billed like a page-source call. See when nothing can be extracted. See the parsed data guide for response shapes.

Scrape a Google SERP

serp = su.serp("best running shoes", pages_to_check: 2)

serp["organic"].each do |result|
  puts "#{result['position']} #{result['title']} #{result['url']}"
end

Force a country

result = su.get_parsed("https://www.amazon.de/dp/B08N5WRWNW", proxy_country: "de")

Capture cookies and the proxy used

page = su.get_page_with_cookies("https://example.com")
puts page.html
p page.cookies
puts page.proxy

Fetch an image

bytes = su.get_image("https://example.com/photo.jpg")
File.binwrite("photo.jpg", bytes)

Drive the browser with steps

Pass a native array of actions to run in a real browser after the page loads. You get back the HTML of the resulting state.

html = su.get_page_source(
  "https://example.com",
  steps: [
    { action: "type", selector: "input[name=q]", value: "wireless headphones" },
    { action: "press_key", value: "Enter" },
    { action: "wait_for", selector: ".search-results", timeout_ms: 10000 },
  ]
)

Discover the selectors first with list_elements: true, which makes get_page_source return the page’s interactive elements as a parsed Hash instead of HTML:

page = su.get_page_source("https://example.com", list_elements: true)

page["elements"].each do |el|
  puts [el["tag"], el["selector"], el["text"]].join(" | ")
end

Each element’s selector drops straight into a steps action. Full action reference in the browser steps guide.

Amazon

Product and search data as JSON, priced in the marketplace’s own currency:

product = su.amazon_product(asin: "B0BSHF7WHW", marketplace: "amazon.com")
puts [product["title"], product["price"], product["currency"], product["rating"]].join(" | ")

results = su.amazon_search("wireless headphones", sort: "price_asc")
results["results"].each { |item| puts [item["title"], item["price"], item["currency"], item["asin"]].join(" | ") }

proxy_country defaults to the marketplace’s home country (amazon.com -> US, amazon.de -> DE), so Amazon prices come back in the right currency. Full field list on the Amazon plugin page.

eBay search

Listings from any of the 19 regional eBay marketplaces as JSON:

items = su.ebay_search("iphone 13", marketplace: "ebay.com", condition: "used", sort: "newly_listed")

if items["exactMatches"]
  items["results"].each do |item|
    puts [item["title"], item["price"], item["currency"], item["condition"]].join(" | ")
  end
end

exactMatches is false when eBay found nothing for the keyword and answered with its own loosely-related suggestions instead. Full field list on the eBay Search plugin page.

Skyscanner plugins

locations = su.skyscanner.flight_locations("London")

flights = su.skyscanner.flights(
  origin: "London", dest: "New York",
  depart_date: "2026-09-01", adults: 1, currency: "USD"
)

hotels = su.skyscanner.hotels(destination: "Madrid", checkin: "2026-09-01", checkout: "2026-09-03")
cars = su.skyscanner.carhire(pickup: "Madrid", pickup_datetime: "2026-09-01T10:00", dropoff_datetime: "2026-09-03T10:00")

Every other plugin

The gem has a method for every plugin (0.10.0 and later) - the plugin’s main input goes in positionally, everything else as keyword options, and options you leave out are not sent, so the API’s defaults apply. Each plugin page shows its method next to cURL. Methods for newer plugins come with newer releases: the Google Hotels, Google News, Google Ads Transparency, Shopee, Taobao, Upwork, Smyths Toys and Wish methods need 0.14.0 or later.

products = su.target_search("coffee maker", sort: "price_low", max_results: 48)
milk = su.woolworths_product("888137")                     # one product, with nutrition
trends = su.google_trends(%w[coffee tea], geo: "US")       # a list is sent comma-separated
reviews = su.trustpilot_reviews("www.nike.com", stars: [4, 5])
hotels = su.google_hotels("Lisbon", check_in: "2026-11-20", check_out: "2026-11-22") # 0.14.0+

# Any plugin endpoint by path, parameters named as the API names them
data = su.plugin("/marketplace/target-search", query: "tv", max_results: 24)

Error handling

Non-2xx responses raise typed errors, all subclasses of ScrapeUnblocker::Error. Transient failures (429, 502, 503, 504 and network errors) are retried automatically; a 401 or 402 is never retried, because it clears when the key or the billing state changes.

begin
  html = su.get_page_source("https://example.com")
rescue ScrapeUnblocker::BlockedError
  # 403: the target blocked every bypass path (not billed)
rescue ScrapeUnblocker::PaymentRequiredError
  # 402: quota, credit limit, or a failed payment - fix billing
rescue ScrapeUnblocker::RateLimitError
  # 429: slow down
rescue ScrapeUnblocker::UpstreamOutageError
  # 503: the target site itself is down - retry later
end

Error	Status	Meaning
`InvalidRequestError`	400	Bad URL, unsupported scheme, or the API key header was not sent
`AuthenticationError`	401	Key not recognised - typo, stray whitespace, or a rotated key
`NoSubscriptionError`	401	Key is fine, but the account has no active plan
`PaymentRequiredError`	402	Billing block - base class for the four below
`QuotaExceededError`	402	The plan’s requests for this period are used up
`CreditLimitExceededError`	402	Unpaid balance is past the account’s credit limit
`BudgetExceededError`	402	This period’s spend reached the monthly budget limit you set in your profile (from SDK 0.9.0)
`PaymentFailedError`	402	A card payment was declined three times
`BlockedError`	403	Blocked by bot protection on every path
`NotFoundError`	404	A plugin lookup found nothing
`TargetNotFoundError`	404 / 410	The target site says the page does not exist (`X-Origin-Status` set). Billed; carries `origin_status`, `html` and `destination_url`. Subclass of `NotFoundError`, never retried
`BrowserTimeoutError`	408	Our browser run timed out before the page was ready
`UnsupportedContentError`	415	The URL serves something other than HTML (page methods) or other than an image (`get_image`)
`ValidationError`	422	Missing or wrong-typed parameter; `body` holds the `detail` array
`RateLimitError`	429	Too many requests
`UpstreamOutageError`	503	The target origin is down
`ServerError`	5xx	Unexpected server error, including a 504 upstream timeout
`TimeoutError`	-	This client gave up locally before the API answered
`ConnectionError`	-	Could not reach the API

The 402 and 404-422 errors were added in 0.1.6; before that they arrived as a bare APIError. Nothing was removed, so rescue ScrapeUnblocker::APIError written against earlier versions keeps working. See all status codes in Errors.

Tune the retry count and timeout on the client:

ScrapeUnblocker::Client.new(timeout: 180, max_retries: 2)

Raw HTTP (without the gem)

The API is plain HTTP, so Ruby’s standard library is enough - no gem required.

require "net/http"
require "json"

uri = URI("https://api.scrapeunblocker.com/getPageSource?url=https://example.com&parsed_data=true")
req = Net::HTTP::Post.new(uri)
req["x-scrapeunblocker-key"] = ENV["SCRAPEUNBLOCKER_KEY"]
res = Net::HTTP.start(uri.host, uri.port, use_ssl: true) { |http| http.request(req) }
payload = JSON.parse(res.body)
puts payload["data"]["page_type"]

The endpoints are getPageSource, serpApi, and getImage; every parameter shown above maps to a query parameter of the same name.

Get Started

No-code area

Spider Cloud

Plugins

AI Assistants

Guides

Code Examples

Ruby

Quickstart

Get Markdown for LLMs

Get parsed JSON instead of HTML

Scrape a Google SERP

Force a country

Capture cookies and the proxy used

Fetch an image

Drive the browser with steps

Amazon

eBay search

Skyscanner plugins

Every other plugin

Error handling

Raw HTTP (without the gem)

​Quickstart

​Get Markdown for LLMs

​Get parsed JSON instead of HTML

​Scrape a Google SERP

​Force a country

​Capture cookies and the proxy used

​Fetch an image

​Drive the browser with steps

​Amazon

​eBay search

​Skyscanner plugins

​Every other plugin

​Error handling

​Raw HTTP (without the gem)

Quickstart

Get Markdown for LLMs

Get parsed JSON instead of HTML

Scrape a Google SERP

Force a country

Capture cookies and the proxy used

Fetch an image

Drive the browser with steps

Amazon

eBay search

Skyscanner plugins

Every other plugin

Error handling

Raw HTTP (without the gem)