Subject: Re: Error: "" (500) after 2 retries at link...
Author: Tom
Date: 05/20/2009 14:12
> robots.txt says:
> User-agent: *
> Disallow: /cgi-bin/
>
> The site doesn't want you to. You'll have to
> override robots.txt setting
>
As I wrote in my initial posting: I already tried "ignore rules in
robots.txt".
Any workaround?
