Hi ,
 
This is Rajesh. My collegues were working on evaluation libraries for
Crawling. They came across HTTRACK. They found that this library has most of
functionality implemented. Code is huge and hats off for your efforts.

However, when they started using this lib in multithreaded environment, they
faced some issues. So they come up will consolidating all global variables in
single structure and instantiating this structure on per thread basis. And
this is released in un-initialization. 

One issue now I am facing with this library is there seems to serious memory
leaks. If the application using this library runs for some time we can see
memory used by this application goes on increasing till it crosses 1.8 GB and
crashes sporodically.

I have used GlowCode tool (trial version: 5.1 )from <http://www.GlowCode.com>.
It has shown error and reports are attached to this mail.

Then I had tried to scrach some of the code and found that some variables are
allocated memory and seems to be never freed. When I had tried to
match the allocation  block sizes it matches the memory leak reported by
GlowCode. This is with NOSTATIC_RESERVE macro used in most of the functions
for allocation. 

 
NOSTATIC_RESERVE(buffer, char, 8192);   > 8192 bytes 
NOSTATIC_RESERVE(prefix, char, HTS_URLMAXSIZE*2);   > 2048 bytes 
NOSTATIC_RESERVE(strc, filenote_strc, 1);   > 2052 bytes 
NOSTATIC_RESERVE(strc, usercommand_strc, 1);   > 2054 
NOSTATIC_RESERVE(strc, fspc_strc, 1); // log..  > 12 
NOSTATIC_RESERVE(rname, char, 1024);   > 1024 
NOSTATIC_RESERVE(buff, char, HTS_URLMAXSIZE*2);   > 2048 
NOSTATIC_RESERVE(buff, char, 256);   > 256 
NOSTATIC_RESERVE(strc, concat_strc, 1);   > 65540 bytes ** 
NOSTATIC_RESERVE(buffer, char*, 1);   > 4  
NOSTATIC_RESERVE(fil_noquery, char, HTS_URLMAXSIZE*2);   > 2048 
NOSTATIC_RESERVE(digest, char, 32+2);   > 34 
NOSTATIC_RESERVE(cache, t_dnscache, 1);   > 1096 bytes 


#define NOSTATIC_RESERVE(name, type, nelt) NOSTATIC_XRESERVE(name, type,
nelt)
 
#define NOSTATIC_XRESERVE(name,type,nelt) do { \
  static DWORD tlsIndex = 0; \
  static   int initValue = 0; \
  if (initValue == 0) \
  { \
    if (!hts_maylockvar()) { \
      abortLog("unable to lock mutex (not initialized?!)"); \
      abort(); \
    } \
    hts_lockvar(); \
    if (initValue == 0) { \
      tlsIndex = TlsAlloc(); \
      if (tlsIndex == 0xFFFFFFFF) { \
        abortLog("unable to allocate thread local storage (TLS) for
variable!"); \
        abort(); \
      } \
      initValue = 1; \
    } \
    hts_unlockvar(); \
  } \
  name = (type*)TlsGetValue(tlsIndex); \
  if (name == NULL) { \
    name = (type*)malloc(sizeof(type)*nelt); \
    if (name == NULL) { \
      abortLog("unable to allocate memory for variable!"); \
      abort(); \
    } \
    memset(name, 0, sizeof(type)*nelt); \
    TlsSetValue(tlsIndex, name); \
  } \
} while(0)

 
Please let me know your comments is this the really issue and how to go about
work-around or fix this issue.
 
Thanks & regards,
Rajesh