In the options menu of your task there is a tab "Spider". Inside there is a
ComboBox with the Label "Spider" which is blank by default. There you can
select wheteher you want to follow robots.txt or not.