GrabzIt
Tools to Capture and Convert the Web

Does GrabzIt's Web Scraper respect robots.txt files?

Our Web Scraper has to respect the rules found in a websites robots.txt file. One of the main reasons for this, other than being nice, is that web scrapers that don't follow the robots.txt file can find themselves being blacklisted by a honeypot service.

These services use the robots.txt to tell a web scraper not to visit a certain file that is linked to from the website. If the web scraper still visits the file then the web scrapers IP address is blacklisted, preventing the web scraper visiting the web site in the future.

Back to SupportMore scraping questions
Try all our premium features for free with a 7 day free trial. Then from $5.99 a month, unless cancelled.
  • More Captures
  • More Features
  • More API's
  • Bigger Scrapes
  • Bigger Screenshots
  • Bigger Everything
Start Free Trial