> is there an
> option to get all *.jpg or *.gif and override this?
The only solution is to override robots.txt ('Set
Options' / 'Spider' / spider: no robots.txt)
But ensure that you don't clobber the bandwidth by setting
up limits (Set Options / Limits and Set Options / Flow
control)
You can also add additional scan rules (Set Options / Scan
rules):
+*.jpg +*.gif
But these rules should be activated by default
|