| Subject: Re: Error: "" (500) after 2 retries at link... |
Author: William Roeder |
Date: 05/15/2009 04:29 |
| | > sorry for that. The correct link is
> <http://wl4.wealth-lab.com/cgi-bin/WealthLab.DLL/cate>
> gories and this is what I used in httrack.
>
> Why not?
robots.txt says:
User-agent: *
Disallow: /cgi-bin/
The site doesn't want you to. You'll have to override robots.txt setting
| |
|
|
|
|
e
Created with FORUM 2.0.11
