> > Ok, I'm trying to mirror a site that tells engines
> 
> > like httrack to not go down to certain
> directories. 
> > Which version of httrack allows me to complete 
> ignore 
> > these files and go down into this certain
> directory?> 
> See options/spider/spider: robots.txt -> 'never'
> 
> But also ensure that you set proper bandwidth
> limiter 
> if you are crawling big files or a large number of 
> generated pages (robots.txt are often used to avoid
> 
> server overload)
> 

> > Ok, I'm trying to mirror a site that tells engines
> 
> > like httrack to not go down to certain
> directories. 
> > Which version of httrack allows me to complete 
> ignore 
> > these files and go down into this certain
> directory?> 
> See options/spider/spider: robots.txt -> 'never'
> 
> But also ensure that you set proper bandwidth
> limiter 
> if you are crawling big files or a large number of 
> generated pages (robots.txt are often used to avoid
> 
> server overload)
> 

> > Ok, I'm trying to mirror a site that tells engines
> 
> > like httrack to not go down to certain
> directories. 
> > Which version of httrack allows me to complete 
> ignore 
> > these files and go down into this certain
> directory?> 
> See options/spider/spider: robots.txt -> 'never'
> 
> But also ensure that you set proper bandwidth
> limiter 
> if you are crawling big files or a large number of 
> generated pages (robots.txt are often used to avoid
> 
> server overload)
> 

> > Ok, I'm trying to mirror a site that tells engines
> 
> > like httrack to not go down to certain
> directories. 
> > Which version of httrack allows me to complete 
> ignore 
> > these files and go down into this certain
> directory?> 
> See options/spider/spider: robots.txt -> 'never'
> 
> But also ensure that you set proper bandwidth
> limiter 
> if you are crawling big files or a large number of 
> generated pages (robots.txt are often used to avoid
> 
> server overload)
>