| Hi!
I'm trying to mirror a site and it seems robots.txt on the site is doing stuff
that won't let me get certain images on the site. If I disable robots.txt
rules, index.html is modified in a way that it shouldn't be.
The site I'm trying to mirror is: <http://www.evilbible.com/>
I can not get these 2 files:
www.evilbible.com/images/redwhitebluegradient.jpg
www.evilbible.com/images/evil_bible_banner.jpg
If robots.txt is disabled, then you will see "images/evil_bible_banner.html"
as an image in the index.html which doesn't even exist on the site.
What can I do to get the 2 images? They won't show up in my browser if I paste
the URL in the address bar. | |