HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: How to use HTTrack on .aspx?id=*-sites?
Author: William Roeder
Date: 06/16/2010 19:12
 
> Could you clarify your answer, what is for you
> "override the robot.txt"?
-s0 or options -> spider -> spider = no robots.txt

> In may case I try with the next link and i didn't
> work.
> <http://www.sick.com/group/EN/home/products/product_p>
> ortfolio/Pages/product_portfolio.aspx

That site doesn't have a robots.txt
Always post the log file line two so we know exactly what settings you used.
The links on that site are absolute. By default httrack only does down, so
most you don't get. +www.sick.com/* or options -> experts -> travel mode =
both.
The site is mostly javascript. Make sure you have
options -> links -> Attempt to detect all = checked
options -> links -> get non-html file related = checked
FAQ: javascript no full support.
 
Reply Create subthread


All articles

Subject Author Date
Re: How to use HTTrack on .aspx?id=*-sites?

06/12/2010 04:39
Re: How to use HTTrack on .aspx?id=*-sites?

06/16/2010 14:21
Re: How to use HTTrack on .aspx?id=*-sites?

06/16/2010 19:12




5

Created with FORUM 2.0.11