| Hi all. I am a relative newby to this, I was wondering if someone with
experience could give some advice. I want to mirror some Swedish real estate
sites (www.boliga.se; www.booli.se; www.hemnet.se) plus all outside sites 3
links deep. I am a small investor and want to record the listing data on a
weekly basis.
These sites are huge - tens of thousands of house and condo listings.
Question 1) Can HTTrack realistically mirror such a huge site?Question 2) How
do I estimate how big the mirrored site will be? Will I need to get a TB-size
external drive?Question 3) I have a mobile broadband connection that averages
6 Mbits/s. How long will uploading one of these sites take?Question 4) How do
I check if robot.txt will stop me from mirroring?
Any and all answers really appreciated.
| |