> So if I remove all +filters (+*.png +*.gif +*.jpg
> +*.css +*.js -ad.doubleclick.net/*
> -mime:application/foobar) it should stop crawling
> other website unnecessarily? When I had looked at
You need to learn to read. I said they were unnecessary. I didn't say they
would change anything.
> the progress in the morning it was crawling a
> completely different domain. I'm going to try again
Based on what you've posted it shouldn't be.
|