> A robots.txt file is a convention used by sites to
> tell spiders (like google and httrack) what files
> they want to allow or forbid them from spidering.
>
> You can tell HTTrack to ignore the robots.txt file
> if you have a legitimate reason to spider the site.
Thanks Mike... We own the site and our host wont help us. I have tried many
forms of exclusion but nothing as worked.
the pcglobal.ca/index.txt file reads:
User-agent: *
Disallow: / |