HTTrack Website Copier
Free software offline browser - FORUM
Subject: Cant copy a web site with .pl ending
Author: Daniel
Date: 10/22/2015 16:35
 

I am tying to copy an archive that is made to be copyed and i cant get it to
work . They say ---- >

" automated downloading and indexing of archive content using crawlers,
spiders, and robots has been enabled. If the crawler identifies itself using
its original referer (or if the referer contains the word "bot") it will be
provided with static and unique links that prevent the otherwise inevitable
multiple crawling of pages. It is essential to follow the archive's robots.txt
file and <meta name="robots"> directives for this. (In a nutshell: do not
crawl URLs containing ".pl?". "

The site is ---- > <https://the-hive.archive.erowid.org/forum/forums.pl>
 
Reply


All articles

Subject Author Date
Cant copy a web site with .pl ending 10/22/2015 16:35




b

Created with FORUM 2.0.11