A small addon, robots.txt helps search engines so on smaller sites its possible
for robots.txt to not exist thus requireing the option to be disabled. HTTRACK
should just spit out an error instead of completely stopping the archive.