I’ve been experimenting with archiving some modern, dynamic websites lately,
and one challenge I keep running into is handling messy or parameter-heavy
URLs before feeding them into HTTrack.

For example, URLs with things like:

?session=...

?ref=...

Tracking parameters

API endpoints mixed with UI routes

Before running a full crawl, I usually clean my URL lists to avoid generating
thousands of unnecessary duplicates.
I’m curious how others here do it.

My current workflow looks like this:

Identify repeating dynamic patterns (?session=, ?ver=, utm= etc.).

Exclude them through HTTrack filters (-*?session=*, -*utm=*, etc.).

Clean the URL list manually so only the essential base paths remain.

Recently, I started using <https://texttoolz.com> because it has quick utilities
like “Extract URLs,” “Remove Duplicate Lines,” “Find & Replace,”
and text cleanup functions that help prepare clean scan rule lists before
importing them into HTTrack. It speeds up the prep work a lot.

How do you all clean or preprocess big URL lists before archiving?Any specific
patterns or tools you rely on?