HTTrack Website Copier
Free software offline browser - FORUM
Subject: Can't download a site from development server
Author: Andrew White
Date: 01/27/2014 01:05
I can't, with Httrack or a couple of other downloading spiders, download a new
site I'm having built from the development website. I get a 503 server not
available error.

Presumably the dev site is set up to forbid spiders (the developer is on
holiday!) The pages load fine with a browser. I've set httrack to ignore the
robots.txt and to use a couple of different user agent names, still the same.

How to make it work? Apart from the robots.txt, how does the server know it is
a spider and not a browser? I am only trying to download a single page, so
it's not the bandwidth or anything like that.



All articles

Subject Author Date
Can't download a site from development server

01/27/2014 01:05


Created with FORUM 2.0.11