> Also; Is there a way to bypass the /robots.txt check
options -> spider
> using HTTRACK? 
HTTrack Users Guide By Fred Cohen - <http://httrack.com/html/fcguide.html>
-sN follow robots.txt and meta robots tags (0=never,1=sometimes,* 2=always)
(--robots[=N])

Change the browser ID to a normal browser not google. I use Mozilla/4.0
(compatible; MSIE 5.5; Windows NT 5.0)