When I write down a website name, it works fine and downloads everything, but
If I add something at the end of the link, it only downloads the robot.txt

<http://mysite.org>         works fine
<http://mysite.org/page/>    doesn't work

Note: If I download <http://mysite.org> , it also downloads
<http://mysite.org/page> but takes a long time