Always post the command used (log file line two or doit.log)
Most likey you didn't use the near flag (get non-html files related) and/or
extended parsing.