numbers incorrect
There are no forum posts on this topic! Why don't you write one?
i performed a scrape and it said it captured 388 pages, but the download only gave me 189 items, including at least 6 duplicates. anyone know how to fix this or rerun it and get a complete result?
The problem I'm having is it's not grabbing or scraping everything, it's missing a lot of the pages that are on the site.
Hi Nathan,
This is normal, the scrape count is just the number of pages that are visited. This maybe because the scraper was sent there by a web page, it may not even be on your website if there is a automatic redirection that forced the scraper to visit a web page.
Duplicates can be removed by setting the ignore duplicate pages option which will try and ignore pages based on how similar the content is to previously visited pages.
It has to be done this way because, with the complexity of some website structures. For instance some web pages might have a structure like https://www.example.com/index?a=1 and https://www.example.com/index?a=2 that has the same content.