HTTrack Website Copier
Free software offline browser - FORUM
Subject: Unable to get HTTrack past login redirect page
Author: GChristy
Date: 08/27/2012 00:10
 
Originally, this was going to be a "Please help me!" post, but I figured out my
issue while I was working on it (isn't that how it always works in the end?).
Hopefully this will help someone:

Hello, I have been working at this for the past 3 days on and off. I am trying
to archive a few pages of technical materials from <https://dtt.dell.com>. 

*Please understand that I have every right to do this and the materials are
usually provided for me in a zip format to download and save as a complete
website archive. However, there are some older training materials that weren't
archived into a offline website for download in a *.zip.

If you visit the page, you will see that it uses HTTPS and active server
pages. In fact, every page within the site that I am trying to archive is an
ASP.

I am using WinHTTrack 3.46 on Windows 7 x64.
IPv6 is disabled - because at first I was trying to use form based capture
with the built in proxy redirect in WinHTTrack. Obviously this won't work with
HTTPS as I found.

Now, on to what I have done:

1. I have tried capturing the form data for the login. This doesn't work with
HTTPS as I read in many threads and tried to make it work many times.

2. I have tried using the HTTP authentication method like so:
<https://user:pass@dtt.dell.com/ifr/eql_upskill/index.asp> - My thoughts were
that this was worth a try because if I log out of the website, and then place
a URL that points to a specific page within the dtt.dell.com website, it will
automatically redirect me to the main page to log in and then redirect me to
the page I wanted to go to. For example:
<https://dtt.dell.com/ifr/eql_upskill/index.asp> would then redirect me to
<https://dtt.dell.com/Pages/Login/Welcome.aspx?ReturnUrl=%2fifr%2feql_upskill%2findex.asp>
and after putting in my user/pass, I would be redirected to the page I wanted
to go to. Obviously this doesn't work because of the form based log in on an
ASP.

3. I tried changing my browser ID to that of Firefox 14 (Mozilla/5.0 (Windows
NT 6.1; WOW64; rv:14.0) Gecko/20100101 Firefox/14.0.1) and then visited and
logged into the dtt.dell.com website with Firefox 14 and saved my cookies. I
tried to copy them into the cookies.txt file for WinHTTrack but may have
messed up the cookie format. The format of a firefox cookie is different than
that of a netscape cookie.
I used MozillaCookiesView from <http://www.nirsoft.net/> and exported my cookies
to a text file. I tried to place the values of my exported cookies from
Firefox into the HTTrack cookies.txt file for this project but I obviously
didn't match them up right because I was still getting redirected to the
dtt.dell.com login page every time.

4. I installed many other website copier applications for Windows and tried
them all with no luck until I came upon MetaProducts Offline Explorer Pro 6.3.
Offline Explorer Pro has a built in browser and I was able to use it to go to
the dtt.dell.com website and log in. After logging in I was able to go to the
training materials I wanted to back up and tell the software to create an
offline copy of the page and all contents related to the material I wanted.
This all worked well and fine, however, the software does require a $150
license to be fully functional and I would pay it.. but I found a reason that
cleared my confusion up with why I wasn't able to get WinHTTrack to do the
same thing as this software was doing.

5. Realizing that the cookies and session ID values etc were what was holding
me back, I attempted to get WinHTTrack working again. The thing that made me
realize this was the I saw what the referenced "includes" were for my project
in Offline Explorer. They looked like this for one of the pages I wanted:

<https://dtt.dell.com/ifr/eql_upskill/index.asp>
Referer=https://dtt.dell.com/ifr/eql_upskill/index.asp
SetCookie=__utma=266679131.382743625.1345919052.1346002950.1346011525.6;
__utmz=266679131.1345919052.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none);
ASPSESSIONIDQEABSQSA=HJAIBCDBHEFCDDDDMLCMEEFK; __utmc=266679131;
ASPSESSIONIDSGDBSRRB=APKLOOABJMIGGDGHENGOLFCG;
ASPSESSIONIDCGBBTSTC=HOHLLKBBJBIDLCMNONMLLPCK;
ASPSESSIONIDQGBDQRQA=GDMBAEDBBPLHJGIKKLMFBGMA;
ASPSESSIONIDSGBCRRRA=LIPJACDBPCGHPGOPOMONFPAF;
ASPSESSIONIDQGDASQRB=OHHPDDDBNJNCLKEIDPAMBEGG;
__utmb=266679131.2.10.1346011525; GAAnon=f7875f88-4704-49aa-ba8e-3cd0a2e7aa61;
StormPCookie=fstn=channel_first&lstn=channel_last&pc=us&pl=en&penv=|us|corp|8cf699101914000&rpo_snp=A5712499,A5588909&4d62fff2=19&683a9f89=28&7aa8484a=04;
StormSCookie=~tidusenbiz555=Support&~tidusenbsd04=0&~tidusencorp=5&~tidusendhs19=4&~tidusengen=solutions&linkNum=17970622&lrgtxt=true&OriginSite=PP&penv=|us|corp|8cf699101914000;
lwp=c=us&l=en&ln=17970622&cs=04&fn=channel_first
channel_last&fstn=channel_first&lstn=channel_last;
snp_bn=us|bsd|SNPBaynoteEnabled.1;
SITESERVER=ID=9bd33db3d8cc496a90d79cf6116ebc26;
SITESERVER_SESSION=ID=9bd33db3d8cc496a90d79cf6116ebc26;
search_bn=us|gen|SearchBaynoteEnabled.1; eds=q0mxeo1s0ptshwnp1fewoe2i;
CartID=JTNjQ2FydHMrY3VycmVudCUzZCUyMjA0JTIyK2NhcnRFeHAlM2QlMjIyMDEyLTA4LTI2VDE0JTNhMTglM2E0Mi41MjEwODk1LTA1JTNhMDAlMjIlM2UlM2NDUytpZCUzZCUyMjA0JTIyK2NhcnQlM2QlMjIxMjk5ODklN2MxNTEwNzFhZS02YmNiLTQyMGYtOTNiYi03YWIyOGEzOWM0NTIlN2MwJTIyJTNlJTNjaSt0JTNkJTIyQnJvdGhlcitUTjQ2MCtIaWdoK1lpZWxkK0JsYWNrK1RvbmVyK0NhcnRyaWRnZStmb3IrU2VsZWN0K0ZheCtNYWNoaW5lcythbmQrUHJpbnRlcnMlMjIrYyUzZCUyMlNOQSUyYjcwNjclMmI3NTY1JTJiNTYzMCUyYjQwMTQlMmIyOTk5JTJiNzk0MiUyYjc5MzYlMmI3OTM0JTJiNzkwMyUyYjc1NjYlMmJwJTJiMDQlMmZTTlAlMmIwNCUyYiUyMitxJTNkJTIyMSUyMit1JTNkJTIyQTAxNDM1MTZyMS5qcGclMjIrcCUzZCUyMjY0Ljk5MDAlMjIrZCUzZCUyMjAlMjIraSUzZCUyMkEwMTQzNTE2JTIyK2lwJTNkJTIyNjQuOTkwMCUyMitpZCUzZCUyMjAlMjIrJTJmJTNlJTNjaSt0JTNkJTIyU2luZ2xlK1VzZStTdGFuZGFyZCtDYXBhY2l0eStCbGFjaytJbmsrQ2FydHJpZGdlKyhTZXJpZXMrMzEpK2ZvcitEZWxsK1Y1MjV3JTJmK1Y3MjV3K0FsbC1pbi1PbmUrV2lyZWxlc3MrSW5ramV0K1ByaW50ZXIlMjIrYyUzZCUyMlNOQSUyYjY4MTIlMmI3NTY1JTJiNTYzMCUyYjQwMTQlMmIyOTk5JTJiNzk0NCUyYjc5MzYlMmI3OTM0JTJiNzkwMyUyYjc1NjYlMmJwJTJiMDQlMmZTTlAlMmIwNCUyYiUyMitxJTNkJTIyMSUyMit1JTNkJTIyMzMxLTc2ODkuanBnJTIyK3AlM2QlMjI5Ljk5MDAlMjIrZCUzZCUyMjAlMjIraSUzZCUyMjMzMS03Njg5JTIyK2lwJTNkJTIyOS45OTAwJTIyK2lkJTNkJTIyMCUyMislMmYlM2UlM2NpK3QlM2QlMjJSZWd1bGFyK1VzZStFeHRyYS1IaWdoK0NhcGFjaXR5K01hZ2VudGErSW5rK0NhcnRyaWRnZSsoU2VyaWVzKzMzUikrZm9yK0RlbGwrVjUyNXclMmYrVjcyNXcrQWxsLWluLU9uZStXaXJlbGVzcytJbmtqZXQrUHJpbnRlciUyMitjJTNkJTIyU05BJTJiNjgxMiUyYjc1NjUlMmI1NjMwJTJiNDAxNCUyYjI5OTklMmI3OTQ0JTJiNzkzNiUyYjc5MzQlMmI3OTAzJTJiNzU2NiUyYnAlMmIwNCUyZlNOUCUyYjA0JTJiJTIyK3ElM2QlMjIxJTIyK3UlM2QlMjIzMzEtNzM4OXIxLkpQRyUyMitwJTNkJTIyNDcuOTkwMCUyMitkJTNkJTIyMCUyMitpJTNkJTIyMzMxLTczODklMjIraXAlM2QlMjI0Ny45OTAwJTIyK2lkJTNkJTIyMCUyMislMmYlM2UlM2NpK3QlM2QlMjJSZWd1bGFyK1VzZStFeHRyYS1IaWdoK0NhcGFjaXR5K0N5YW4rSW5rK0NhcnRyaWRnZSsoU2VyaWVzKzMzUikrZm9yK0RlbGwrVjUyNXclMmYrVjcyNXcrQWxsLWluLU9uZStXaXJlbGVzcytJbmtqZXQrUHJpbnRlciUyMitjJTNkJTIyU05BJTJiNjgxMiUyYjc1NjUlMmI1NjMwJTJiNDAxNCUyYjI5OTklMmI3OTQ0JTJiNzkzNiUyYjc5MzQlMmI3OTAzJTJiNzU2NiUyYnAlMmIwNCUyZlNOUCUyYjA0JTJiJTIyK3ElM2QlMjIxJTIyK3UlM2QlMjIzMzEtNzM4OHIxLkpQRyUyMitwJTNkJTIyNDcuOTkwMCUyMitkJTNkJTIyMCUyMitpJTNkJTIyMzMxLTczODglMjIraXAlM2QlMjI0Ny45OTAwJTIyK2lkJTNkJTIyMCUyMislMmYlM2UlM2MlMmZDUyUzZSUzYyUyZkNhcnRzJTNl;
LiveBall=uid=6430725&uky=WCP3IIOQ&rid=6901691;
RBI=usdhs19=snp:320-9334:8cf51a5064c28c0&usbsd04=snp:A0143516:8cf51a52b4585aa|snp:5HY5230:8cf51a52b107668|snp:331-7690:8cf51a52ad7b1b4|snp:331-7389:8cf51a52aa59812|snp:331-7383:8cf51a52a1d368e|snp:331-7377:8cf51a5299a89dc|snp:A0962028:8cf51a52b6172d1;
dell.config=dynamic=true

So, seeing this and realizing that the MozillaCookiesView software from
<http://www.nirsoft.net/> wasn't showing me all of the cookie info I needed (and
not in Netscape format either) I looked for another piece of software or a
plugin to replace its' role. I found Export Cookies v1.2 for Firefox:
<https://addons.mozilla.org/en-US/firefox/addon/export-cookies/> and installed
it. I cleared all of my cached data, history, cookies, etc, and went back to
the dtt.dell.com website and logged in, then went to my material I needed. I
exported my cookies using the Export Cookies plugin successfully. This plugin
exports them in Netscape format, so I would be able to simply copy and paste
them into the HTTrack cookies.txt file.

6. Now that I had my cookies, I re-created the project I was attempting to use
in WinHTTrack to archive my training materials. I copied my cookie data into
the corresponding cookies.txt file for my project and set everything up as I
saw it fit. Keep in mind that I set my browser ID to that of Firefox 14 once
again. I'm not sure if this makes any difference but I felt it would be a good
idea. I began my download session in HTTrack and everything began processing
successfully!


Here are some of the threads I searched and referenced before I ended up
finding my own fix:

Authentication FAQ - <http://httrack.kauler.com/help/Authentication>

Cookies FAQ - <http://httrack.kauler.com/help/Cookies>

CatchURL Tutorial - <http://httrack.kauler.com/help/CatchURL_tutorial>

Can't copy ASP site - <http://forum.httrack.com/readmsg/29269/index.html>

https/SSL and username/password -
<http://forum.httrack.com/readmsg/11518/index.html>

CMS on HTTPS? - <http://forum.httrack.com/readmsg/24702/index.html>
 
Reply


All articles

Subject Author Date
Unable to get HTTrack past login redirect page

08/27/2012 00:10
Re: Unable to get HTTrack past login redirect page

08/27/2012 00:21
Re: Unable to get HTTrack past login redirect page

10/09/2012 13:46




4

Created with FORUM 2.0.11