| Subject: Re: search engines |
Author: geatz |
Date: 02/22/2005 10:48 |
| | action:download web site(s)
put the 10 google.fr urls in "web addresses: (url)"
set options...
- limits(maximum mirroring depth: 2)
- spider (spider:no robots.txt rules)
- scan rules:
-* +*.png +*.gif +*.jpg +*.css +*.js -ad.doubleclick.net/*
- browser ID - browser"identity":
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Win 9x 4.90) | |
|
|
|
|
d
Created with FORUM 2.0.11