HTTrack Website Copier
Free software offline browser - FORUM
Subject: Is it possible to crawl in increments of 1000?
Author: marine88
Date: 04/25/2024 14:55
 
For example, let's say you have 100,000 links and content on your site.
The site is blocking crawling bots, so crawling more than 1500 will result in
an IP block.
To avoid this, I want HTTrack to pause after processing 1000 links, and then I
want to use a VPN and continue working with another IP. I can write a simple
macro to do this, but the important part is "HTTrack pauses after processing
1000 links".

How can I do this?
 
Reply


All articles

Subject Author Date
Is it possible to crawl in increments of 1000?

04/25/2024 14:55




6

Created with FORUM 2.0.11