You can integrate the Web Scraper API using one of the following three methods:
Push-Pull – the most reliable data delivery method. Using it, you provide us with your job parameters and we give you a job id that can be used to get content from /results endpoint at a later point. You are able to check if the job is completed yourself or set up a listener accepting POST requests, in which case we’d send you a callback message once the job is ready to be reclaimed.
In addition, Push-Pull offers the following possibilities: Single Query, Check Job Status, Retrieve Job Content, Batch Query, Get Notifier IP Address List, Upload to Storage, and Callback.
Realtime – this method allows you to submit your request and receive data back on the same HTTPS connection straight away.
Proxy Endpoint – this method only takes completely formed URLs instead of parameters such as domain and search query. You can send extra information in the request headers, such as location and language. Use our entry node as a proxy, authenticate with Web Scraper API credentials, and ignore certificates. Your data will reach you on the same open connection.
Read our Quick Start Guide for more detailed explanations and code samples.