|
I am tying to copy an archive that is made to be copyed and i cant get it to
work . They say ---- >
" automated downloading and indexing of archive content using crawlers,
spiders, and robots has been enabled. If the crawler identifies itself using
its original referer (or if the referer contains the word "bot") it will be
provided with static and unique links that prevent the otherwise inevitable
multiple crawling of pages. It is essential to follow the archive's robots.txt
file and <meta name="robots"> directives for this. (In a nutshell: do not
crawl URLs containing ".pl?". "
The site is ---- > <https://the-hive.archive.erowid.org/forum/forums.pl> | |