HTTrack Website Copier
Free software offline browser - FORUM
Subject: WinHTTrack (GUI) not working with robots.txt
Author: Enforcer
Date: 03/16/2025 00:12
 
Hi --
I'm trying to use WinHTTrack (GUI version, Windows 10 64-bit) to back up my
Neocities site (https://draggianuniverse.neocities.org) which worked
previously, but isn't now. I have my robots.txt set to block all crawlers, but
even telling WinHTTrack to ignore robots.txt is still resulting in it failing
to access anything -- it isn't even mirroring the homepage; it's just saying
that the site is empty when saved. It worked as recently as October 2024 and I
reinstalled to restore default settings, but it didn't fix the issue. Any idea
what's going wrong here? Do I need to temporarily delete my site's robots.txt
for WinHTTrack to work?
 
Reply


All articles

Subject Author Date
WinHTTrack (GUI) not working with robots.txt

03/16/2025 00:12
Re: WinHTTrack (GUI) not working with robots.txt

03/16/2025 00:15




6

Created with FORUM 2.0.11