Not all website owners are happy to let you collect their public data.
Particularly when you need large volumes of data from a single site, many, in fact, try to prevent scraping of their data. Some site creators are exceptionally creative with the means they’ll use to stop web crawlers.
Stratalis has an answer to virtually every type of attempt to block data-gathering robots. Such attempts include:
- Text in images
- IP blocking
- Randomised content or structure
The number one rule we stand by is to respect the websites that we crawl by keeping our data acquisition slow enough so that it does not burden the remote server. Beyond this, it is our own secret recipe!