I am having a difficult time trying to figure out how to use the program to
spider thru the given site and grab the hosted .zip files. They are zips of
recipes users upload and I would love to just grab them all quickly rather
then spend tedious amounts of time going thru one by one. I'm running into two
problems:
1 - the program returns an error saying robot.txt tells it not to search
anywhere that returns */Forum, can I change this property to allow it to?2 - I
can't get it to grab the files on other sites that host .exe's such as
sourceforge.net.

Can this be done given the posing problems?