As an example of what I am looking for: here is a command line program that
creates an index of ftp sites: <http://aluigi.altervista.org/mytoolz.htm>
(myftpidx)

Quote:

"tool for indexing recursively any file available inside a FTP server or one
of its folders.
it has options for verbose output, full or relative URL, fields to visualize
and HTML output.
I wrote it because sometimes happens to need a specific file or a set of files
available on a FTP server but we don't know in what folder they are located or
if exist different and most updated versions and copies of that program/file,
so this tool lists all the names and sizes of the files inside a specific FTP
directory or just in the entire server for allowing an easy searching of these
files in the local list."