HTTrack Website Copier
Free software offline browser - FORUM
Subject: plugging function for HEAD requests
Author: Allen Day
Date: 05/04/2012 22:48
 
Hi,

I was just reading the "plugging function" docs at
<http://www.httrack.com/html/plug.html> and didn't understand how to implement
my function, specifically for mirroring binary files (images).

I want to do this: for a given URL, HEAD request the first N bytes, md5sum
them, and using this md5sum decide whether or not to proceed with a full GET
of all bytes.  It looks like "check_link" is close to what I want, but it
doesn't appear to have the referring page context I need (as many web servers
block requests with an empty or invalid "Referer" header to prevent
hot-linking).

It looks like I might be able to use one of the html processing plugs
(check_html, preprocess, postprocess), but this doesn't really seem
appropriate ; I'm not trying to modify any HTML, just make a decision of
whether or not to follow/retrieve a particular link.

-Allen
 
Reply


All articles

Subject Author Date
plugging function for HEAD requests

05/04/2012 22:48
Re: plugging function for HEAD requests

05/05/2012 10:30
Re: plugging function for HEAD requests

05/05/2012 12:03




8

Created with FORUM 2.0.11