| > Hallo, I need a hint. I'm starting to use this
> program. What I didn't understand is if I can use it
> for the spidering a list of URL (directory as
Yahoo).
Many dynamic pages (including some yahoo groups) are
very hard to handle. For some sites, you'll need to
use the 'capture url' feature, or define additional
filters. Note that some websites also setup robots.txt
rules to prevent from spidering them.
| |