Dear trakker and Nijaz,

Thanks to both of you for your suggestions. 
Regarding the cloudfront links, my problem wasn't that it was missing
cloudfront links but was downloading the rest, it was downloading nothing form
the subdirectories of the main link.
I will try Nijaz's method and hopefully it works :)