All Collections
Getting Started
How to start using Web Scraper API
How to start using Web Scraper API
Josh avatar
Written by Josh
Updated over a week ago

Create an API user

Begin by creating an account on the Oxylabs dashboard. After that, select a pricing plan or pick a free trial, and then create an API user.

Make a cURL request

You can use the below cURL code for testing:

curl 'https://realtime.oxylabs.io/v1/queries' --user
'USERNAME:PASSWORD' -H 'Content-Type: application/json' -d
'{"source": "universal", "url":
"https://sandbox.oxylabs.io/products/", "geo_location": "United States"}'

Replace the USERNAME and PASSWORD with the API credentials you’ve just created, and then run the request on Postman or any other setup. The output will contain an HTML file of sandbox.oxylabs.io/products with data localized for the United States.

See this short video tutorial demonstrating how to use Web Scraper API:

🔬Test our APIs in the Scraper APIs Playground, which is accessible via the dashboard.

Integration methods

There are three different methods you can use to integrate Web Scraper API :

  • Realtime – provides a synchronous method where you have to keep the connection open until the job is finished.

  • Push-Pull – enables an asynchronous method where you’ll have to make another request to the API to retrieve results once the job is finished.

  • Proxy Endpoint – offers a synchronous method where you can use our endpoint as a proxy.

🔎 For a detailed look at these methods and their benefits, refer to the respective integration pages or see this comprehensive blog post.

Targets and parameters

Web Scraper API can effectively gather public data from any website, including online travel agencies, real estate platforms, and other domains. Visit these pages to learn more about the parameters you can use and see code examples. Details about other settings, like JavaScript rendering and user agents, can be found on the Global Parameter Values page.

Additional free features

If you want to extend the API functionality in your projects, you can use the following features integrated into Web Scraper API:

  • Headless Browser – renders JavaScript and lets you define custom browser instructions, such as entering text, clicking elements, scrolling pages, and more.

  • Custom Parser – enables you to create your own parsing and data processing logic that’s executed on a raw scraping result.

  • Web Crawler – crawls any site, allowing you to select useful content and receive it in bulk. You can use it to perform URL discovery, crawl all pages on a site, index all URLs on a domain, and more.

  • Scheduler – provides a way for you to automate recurring scraping and parsing tasks by creating schedules.


🙌 Need assistance? Contact support via live chat or send a message to [email protected].

🎯 Want a custom solution or a free trial? Contact sales by booking a call. For any questions, such as custom pricing, advice, or a free trial, drop us a line at [email protected].

Did this answer your question?