I'm trying to grab a full thread from a forum here
www.bay12forums.com/smf/index.php?topic=19239.
The pages increment by 15 the end of the url as in:
www.bay12forums.com/smf/index.php?topic=19239.15
www.bay12forums.com/smf/index.php?topic=19239.30
www.bay12forums.com/smf/index.php?topic=19239.45
ect...
I tried putting in
www.bay12forums.com/smf/index.php?topic=19239.
as the web address and under the scan rules I have
-*
+www.bay12forums.com/smf/index.php?topic=19239.*
+mime:image/*
and put not to follow robots.txt. Should I leave max mirroring blank to get
the full thread? I tried max set to 1 and I'm only getting the first page. and
the rest are going external. |