HTTrack Website Copier
Free software offline browser - FORUM
Subject: Re: Website images not being downloaded
Author: Nijaz
Date: 08/26/2020 18:52
 
It is also good not to use -s0 because sometimes robots.txt will only help you
skip downloading what you won't need, and reduce bandwidth for you so you
don't get banned. For example @trakker you should use robots.txt for
mobile.twitter.com because it is good. You can always find what is in
robots.txt by going to url you wanted to download, keep only domain part of
url, and add robots.txt to path, for example for twitter it is
<https://mobile.twitter.com/robots.txt>
You'll notice crawl delay parameter is set to 1 second, some servers will ban
you if you don't follow such rules or robots.
Of course sometimes robots.txt may be your enemy, preventing you from
downloading anything, in such case it is good to not follow robots, by using
-s0 in case of command line.
@Matthew in your case robots.txt is preventing you from downlaoding images so
you should not follow it, here is it:
<https://system16.com/robots.txt>
 
Reply Create subthread


All articles

Subject Author Date
Website images not being downloaded

08/24/2020 14:26
Re: Website images not being downloaded

08/25/2020 14:21
Re: Website images not being downloaded

08/26/2020 00:18
Re: Website images not being downloaded

08/26/2020 00:21
Re: Website images not being downloaded

08/26/2020 15:33
Re: Website images not being downloaded

08/26/2020 18:52




d

Created with FORUM 2.0.11