| It is also good not to use -s0 because sometimes robots.txt will only help you
skip downloading what you won't need, and reduce bandwidth for you so you
don't get banned. For example @trakker you should use robots.txt for
mobile.twitter.com because it is good. You can always find what is in
robots.txt by going to url you wanted to download, keep only domain part of
url, and add robots.txt to path, for example for twitter it is
<https://mobile.twitter.com/robots.txt>
You'll notice crawl delay parameter is set to 1 second, some servers will ban
you if you don't follow such rules or robots.
Of course sometimes robots.txt may be your enemy, preventing you from
downloading anything, in such case it is good to not follow robots, by using
-s0 in case of command line.
@Matthew in your case robots.txt is preventing you from downlaoding images so
you should not follow it, here is it:
<https://system16.com/robots.txt> | |