HTTrack Website Copier
Free software offline browser - FORUM
Subject: Can I copy or update site from sitemap.xml?
Author: zjzengdongyang
Date: 08/12/2014 06:23
 
Can I copy or update site from sitemap.xml?For example I download
msdn.microsoft.com/library/*.aspx.
I found we want to copy page all in
msdn.microsoft.com/Sitemaps/MSDN_LIBRARY__ZH-CN_*.xml
I think download any page from it can gain time and save computer resource.
After it can improve the page coverage and improve website copy time's and
space's expected accuracy(for example I can know I should download how many
page from net).
I think it can obey robots.txt.
Because I found this xml in sitemapindex.xml from robots.txt.
I hope you can improves this functionality.I think this is very useful.
 
Reply


All articles

Subject Author Date
Can I copy or update site from sitemap.xml? 08/12/2014 06:23
Re: Can I copy or update site from sitemap.xml? 08/12/2014 06:36




2

Created with FORUM 2.0.11