Could you clarify your answer, what is for you "override the robot.txt"?
In may case I try with the next link and i didn't work.
<http://www.sick.com/group/EN/home/products/product_portfolio/Pages/product_portfolio.aspx>